This Open Source AI Is Scaring the Biggest Tech Companies

This thing literally can take full control of your computer or even your phone and navigate it step by step on its own. If you've been following developments in AI agents like Anthropic's computer use or OpenAI's operator, let me tell you, Uitars basically took things to a whole new level. So Uitars is available in two main versions, one with 7 billion parameters and another with 72 billion. It's true trained on a massive data set of about 50 billion tokens, and ByteDance joined forces with Tsinghua University to build this model that can run on your PC or Mac and handle tasks ranging from booking flights on the web to installing extensions in something like Visual Studio code. It's not just giving you text responses, it's literally controlling the gui. Like let's say you say find me. Round trip flights from Seattle to New York. Next month uitars will open Delta's website, fill out the flight details, pick the right dates, filter by price, and even click around the site as needed. Meanwhile also has a little box on the side that explains each thought process step by step. So it's basically like having a super smart personal assistant that you can watch operate your machine in real time. It also outperforms GPT 4.0, Claude and Google's Gemini on more than 10 different GUI benchmarks Like some of these benchmarks include Visual Web Bench it got a score of 82.8 while GPT 4.0 got 78.5 for instance web src, Screen QA Short, and so on. Then when it comes to multi step tasks like opening up some office software and rearranging slides, or customizing background colors or messing around with mobile apps. Uitars kills it on a benchmark called OS World, which evaluates how well an agent can perform open ended computer tasks. Uitar 72B ended up scoring 24.6 with 50 steps, while Anthropic's Claude peaked at 22.0. Anyway, let's talk about how it's doing all this. One big difference is that it's a native GUI agent. Older AI systems often rely on text based data like HTML or accessibility trees or need some complicated specialized script to navigate. But uitars basically perceives the screen visually the same way we humans do. No more messing with hidden code behind the interface. It sees a screenshot, understands the layout and works as if it's a user physically clicking and typing. This approach is known as a pure vision based agent method. As you can imagine, it also means it can handle changes in the interface or platform more flexibly, but that's only part of the story. Another piece is something known as reflection tuning or System 2 reasoning, which Uitars uses for multi step tasks and error correction. For instance, if it tries to install an extension in VS code and something goes wrong, the model actually notices the glitch, checks if the application is still loading, and can correct itself accordingly. This reflection process allows uitars to adapt on the fly. It might go oh wait, that button didn't respond. Let's try it again. Or do a new approach. They've also run hundreds of virtual machines to generate all sorts of real world tasks for IT to practice on. Anytime it messes up, it notes how to fix the error and that data feeds back into the model so it gets better every iteration. This iterative feedback is basically how it polishes its performance. Now, how does ByteDance manage to train such a massive system? It said they're facing some GPU export restrictions, so they apparently focused on all these algorithmic breakthroughs. Rather than brute forcing with giant computational clusters, they harness a ton of synthetic data, replays from user interactions, crawled tutorials and so forth. Their pipeline collects screenshots from a broad variety of websites and apps, logs bounding boxes for each element on the screen, extracts the text that's displayed, and merges it all so the model can see how each interface piece fits together. Then it learns to unify different sorts of actions like tapping on mobile or clicking on desktop under one universal action space. The final end to end approach basically merges perception, the screenshot reasoning, like what steps to take, memory long term and long term and action mouse clicks keyboard inputs into one big model. And get this, the memory portion includes short term memory for immediate tasks like remembering which step it's on and long term memory encoded in its parameters gleamed from historical interaction. That's how it can reason about say wait, I need to open the settings panel first. Or we tried that. Let's do a different plan. They basically taught it both System one and System two. Thinking System one is that quick intuitive approach like oh, a save icon. Let's click that. Meanwhile, System two is methodical multi step reflection and planning. Let's break this big problem down, do trial and error reflect if we fail and fix it. Now you might be curious how does it stack up against the competition Anthropics? Claude can do web based tasks well enough, but apparently it doesn't handle mobile tasks nearly as fluently. GPT 4.0 is strong at a bunch of tasks, but Uitars is outscoring it in specialized metrics for something like Web SRC understanding semantic layout in web contexts. Uitars 7B gets a top score of 93.6% for screen QA short. The 72B version got 80 88.6 beating Claude 3.5 sonnet. Then there's the new Screen Spot Pro benchmark designed for high resolution more professional applications where Uitaurus 72B hit a whopping 38.1, overshadowing a bunch of older models out there. One more noteworthy piece is the evolution path they talk about in the official research. They say the field progressed from rule based automation RPA to agent frameworks that rely on big language models but still needed a lot of manual prompt engineering engineering to fully native agent models like UI TARS that do everything end to end and eventually to something called active and lifelong agents. That future scenario is basically where these models keep learning from their environment in real time without big retraining steps. Uitars is a big leap in that direction since it's already harnessing iterative training to keep refining itself. But they're looking even further to where the agent basically sets its own tasks, tries them and learns from them automatically. All of this is open sourced by the way. They have a GitHub repo@GitHub.com bytetance uitars and a separate desktop app you can snag. So if you want to have some fun letting an AI take the wheel of your computer, you can actually download it or check the code. I mean, it's kind of insane just let an AI autonomously rummage around your PC or Mac, installing Chrome extensions, booking plane tickets or even editing stuff in Photoshop. The developers joke it's like having an invisible digital human behind your shoulder, but super fast and hopefully less prone to random mistakes than your average coworker. And not to mention the name TARS is obviously borrowed from that awesome movie Interstellar. Referencing that quick witted shape shifting robot. The paper goes into insane insane detail about how they keep the model from messing up. One part is error correction data where they label a mistake, note the correct fix and then feed that into the agent. Another is post reflection where the agent tries to recover after a bad action. Anyway, the applications are huge. We're talking about OS level control, chaining tasks across different software, bridging from design to coding. People imagine hooking multiple AI agents together like Utahs, plus a coding AI to handle entire workflows from writing code to shipping it. At the same time you can see ByteDance is pushing the boundary here and basically telling the world you can have an AI that's truly integrated with your everyday computing. There's also a mention that Apple might need to watch out. We've yet to see a truly impressive Apple AI that can run natively across iOS or Mac in this manner. In short, Uitars is a big leap for AI that doesn't just chat but actually controls your computer. It dominates benchmarks in both web and mobile, outperforming, GPT4, USO, Claude 3.5, Sonnet and Google's Gemini. With reflection tuning, iterative training and a giant screenshot dataset it unifies perception, reasoning, memory and action. If you're into hands free computing this could change personal and business workflows. Plus it's open source so developers can tweak it or build fresh tasks. Bottom line, AI that can do your emailing, manage your apps or even run PowerPoint is here so check it out and let me know your thoughts on it. Thanks for watching and I'll catch you in the next one.