Microsoft New AI LAM Is the Future of AI Automation (Insane Power)

Microsoft's large action model, often called lam, is designed to go beyond just generating text and actually perform tasks in the Windows environment. It was developed to interpret user instructions, create step by step solutions, then carry out those solutions in applications like Microsoft word, Excel and PowerPoint. Its main goal is to bridge the gap between language models that only produce text and models that can interact directly with with an operating system. The way LAM is trained relies on approaches like supervised fine tuning, imitation learning and reinforcement learning all combined to give it both planning and execution skills in a dynamic environment. For training data, the Microsoft Team gathered vast amounts of material that included both task descriptions and real action sequences. They used official software documentation, wikiHow articles and Bing search queries to then relied on GPT4 to turn this raw text into structured pairs. The topics ranged from simple items like changing fonts or highlighting text in Word to more complicated tasks involving user defined styles, form filling and multi step formatting. They also used a method called Data Evolving where GPT4 adds complexity to basic tasks by introducing extra conditions or instructions. Eventually they built up over 76,000 task plan pairs, but converting written instructions into actions that run on Windows is not straightforward, so they took the additional step of creating task action data. This involves pairing text based plans with actual clicks or typed inputs in Word, such as selecting a chunk of text or clicking a button labeled Text Highlight color paired with a function call to handle that click. Their training pipeline had four main phases. In the first phase they taught a base model called Mistral 7B how to write coherent plans for different tasks. That model became LAM1, which could outline for example how to insert images or choose fonts in Word, but it didn't yet handle the actual interactions like clicking or typing. In the second phase, they used 2192 success successful examples that GPT4 had labeled, each showing a sequence of actions for different states in Word's interface. By imitating these examples, the model evolved into LAM2, which could now generate action steps and replicate what a user or an expert AI would do, such as picking the correct menu item or entering text. In the third phase, they let LAM2 try tasks that GPT4 had not completed successfully. LAM2 found new ways to solve some of those tasks, contributing 496 additional successful action sequences. So again, less crime, less danger. After retraining with this fresh data, they got LAM3. Finally, the fourth phase brought in a reward model that assigned one to successful steps and minus one to unsuccessful ones, and they used reinforcement learning to optimize decisions. That last phase produced LAM4 which systematically captured lessons from both successful and failed attempts. Now, quick heads up. This video is in partnership with Growth School, but honestly, what they're offering is something I think you'll find genuinely useful. 2024 has been a whirlwind. Jobs popping up everywhere, but layoffs are just as common. It's a wild ride and even if things feel secure now, it's you never know what's around the corner. That's why I think having multiple streams of income isn't just smart, it's essential. Here's where AI comes in. With the right tools and skills, you could seriously start earning an extra $10,000 a month. Now, if you're wondering how to get started, Growth School has something really cool. They're offering a three hour hands on AI training where you'll learn to use over 25 powerful AI tools. Normally it's paid, but but the first 1000 AI revolution viewers can join for free using the link in the description. On top of that, you'll get $500 worth of bonus resources just for signing up. The training covers everything job hunting tips, salary negotiation, mastering Excel, even content creation. And it's not just for tech experts. Whether you're in finance, sales, marketing, HR or even still studying, this can work for you. Growth School has already helped millions of people level up and this could be your turn to stay ahead in an AI driven world. So if this sounds like your kind of thing, hit the link below to grab your free spot. Plus, don't miss joining Growth School's WhatsApp community. It's a great place to connect with others. Diving into AI too Alright, now back to large action models. So they tested LoM's performance offline with about 435 word tasks covering everything from opening a blank document to more advanced formatting tasks involving headings, inserting tables or pulling data from a file. They measured success rates step by step, accuracy and whether the model could pick the correct interface object and operation. The results showed improvements at every phase. Limone, trained only on textual Instructions, achieved around 35.6% overall success in simulated tests. LAM2, which learned from GPT4's examples, rose to about 76.8%. LAM3, boosted by the self discovered solutions, climbed to roughly 79.3%. LLAM4, with reinforcement learning on top reached about 81.2%. In comparison, GPT4 in text only mode was around 67.2%, while a smaller GPT4 mini model was about 62.3%. GPT4 with visual input got closer to LAM4 at 75.5% but still lagged behind it. This highlighted how targeted training combined with iterative data collection helped LAM excel in Windows tasks more than a general model like GPT4 could. They also conducted an online evaluation in a genuine Windows environment where Microsoft Word was running on dedicated virtual machines. In this live setting, LAM relied only on text input and reached around 71.0% success across the same 435 tasks, usually taking about 30.42 seconds each. GPT4 with text only input managed around 63.0% and took about 86.42 seconds. GPT4 with visual input did better at 75.5% success, but it needed roughly 96.48 seconds and its latency per action step sometimes passed 19 seconds. LAM typically completed tasks in about 5.62 steps each, each step averaging 5.41 seconds. GPT4 in text only mode was much slower, and while GPT4 mini was somewhat faster than the larger GPT4, it still fell behind LAM. These outcomes underline how specialized training and a narrower domain can make an action model both more accurate and more efficient. Microsoft researchers then integrated the large action model into a Windows based agent called ufo. This agent looks at GUI elements on screen and executes the tasks a user wants. It gathers details like each control's name, coordinates and purpose, then gives that info to lam. LAM determines which control to select, which arguments to use for a click or a type entry or which text to enter. UFO then carries out that plan by performing the clicks or making the necessary function calls in word, all while remembering which steps it has completed. Because LAM relies on real time environment feedback and follows the plan step by step. It differs from older models that only produce text based instructions, but a system points to a promising direction for AI driven automation. Someone could ask Lam to open Excel, copy some cells, go to a website and paste those cells into an online form. Instead of just explaining how to do it, LAM actually performs each step. This capability extends across different apps and can handle more advanced workflows than models that only generate suggestions. The researchers see it as a step toward broader general capabilities, maybe even controlling other desktop programs or robotic platforms. However, a model that runs applications with minimal supervision can pose safety concerns. If the AI misunderstands a command or or drifts off track, the consequences can be significant. The development team insists on having strong error checks and possibly requiring certain steps to be verified before execution. They also note that bringing LOM to other operating systems like macOS or mobile devices means gathering new data sets. Each environment needs enough labeled examples of tasks and controls for the model to learn them well, and that process can be time intensive. Techniques such as transfer learning or multitask learning might help, but they remain active research areas. The overall project is described in a technical paper called Large Action From Inception to Implementation. The authors compare regular language models to these action based approaches, explain in detail how they collect and refine data, and describe each training phase. They also outline the architecture of the UFO agent, which includes one component that parses user requests, another that breaks them down into manageable tasks, and an app agent containing the LAM that directly interacts with the environment. Both their offline and online evaluations show that a model trained specifically for a set of tasks can beat general purpose models in that particular domain. It usually runs faster too, since it doesn't have to produce completely completely open ended responses. Experts worry about accountability, bias and ethics when AI controls digital or physical environments, especially in finance or healthcare. Lam achieves about 81, 22% success in offline tests and 71% in live settings for Windows tasks, surpassing older text based automation. Microsoft wants to expand beyond officers, though issues like safety and scalability remain. LAM signals a new era of AI acting on instructions rather than just responding in text. Do you think things are moving too fast or is this all just part of normal AI progress? Would be great to hear your take and as always, thanks for watching and I'll see you in the next video. Sa.