The COMPLETE TRUTH About AI Agents (2024)

Insane buzz going on about the phrase AI agent. Now, I do want to make this video because it's actually going to give you a complete rundown of what are AI agents and the actual shocking truth about them, because a lot of people don't know this and the buzzword is getting a little bit out of hand. So let's actually take a look what agents are, what they can do, what projects you can use today, and where we are on the grand scheme of actual real world autonomous agent applications. So when we look at what are AI agents, I want to simply define this to make this video a lot easier. What we can simply call an AI agent is basically just an advanced AI assistant. Think of it as someone that you're giving a task and you're saying, hey, I need you to go out and autonomously execute this task either by yourself regarding your environment and being able to use certain tools, or I need you to work in a team that and then of course accomplish your goal. Now, the reason I'm making this video is because there's a lot of information that most people don't know, some good information and some bad information that's going to give you guys the true rundown of actual agents. I'm going to first start with this video of Andrew Ng describing AI agentic workflows. This is another terminology that is used when you are describing an AI that is allowed to use a certain workflow to achieve a certain goal. So this is pivotal in terms of how you can use AI today. And I think this is one of the most important pieces of content that you'll ever see regarding actual AI agentic workflows that are applicable in today's use.

What are they? Many of us are used to using large language models with what's called zero shot prompting. And that means asking it to write an essay or write a response to a prompt. And that's a bit like if you imagine going to a person and saying, could at least write an essay on topic X by typing from start to finish all in one go, without ever using backspace, you know, and despite the difficulty of writing this way, I can't write that way. Despite the difficulty of writing this way, OMs do pretty well. In contrast, an agentic workflow is much more iterative. You may ask an OM please write an essay on a write an essay outline and then ask, do you need to do any web research? If so, go search the web, fetch some info, then write the first draft, then read your draft to see if you can improve it, and then Revise the draft. So with an agentic workflow, it looks more like this, where the algorithm may do some thinking, do some research, then revise it and do some more thinking. And this iterative loop actually results in a much better work product. And if you think of using agents to write code as well, today we tend to prompt an LLM write code and that's like asking a developer, could you type out the program and, and have it just run? You're typing for the first to last character.

So what Andrew Enchi just described there was the difference between a non agentic workflow and an agentic workflow, as he said right there. And I'm going to reiterate, first off, many people using ChatGPT just zero prompt it. For the average user, this is going to be their main experience with LLMs and that's completely fine. But if you're someone that's watching this channel, you're definitely going to want to benefit from an agentic workflow. If you're trying to achieve a any difficult, complex task, you're always going to want to make sure that you have multiple steps in order to get the AI system to perform at a much higher rate. Now the reason you're going to do this is because this dramatically improves the AI's output and actually allows it to achieve a higher degree of reasoning. Now if you don't believe me, take a look at Andrew Ng when he's able to do this with GPT 3.5.

And it works surprisingly well. But agentic workflows also allow it to work much better. So my team collected some data that was based on the coding benchmark called human eval. HumaneVal is a standard benchmark released by OpenAI a few years ago that gives coding puzzles like this, you know, given the non empty list, integers return the sum and that turns out to be the solution. And it turns out that GPT 3.5 on the evaluation metric parsec got it 48% right with zero shot prompting, prompting it to just write out the GPT 4 does way better, 60%, 67% accurate. But it turns out that if you take GPT 3.5 and wrap it in a gentic workflow, it does much better. And with GPT4 that does also very well. And so to me, one thing I hope you take away from this is while this there was a huge improvement from GPT 3.5 to GPT 4, that improvement is actually dwarfed by the improvement from GPT 3.5 with an agentic workflow. And to all of you building applications, I think that this maybe suggests how much promise an agentic workflow has.

So overall, just there, we even saw on the benchmarks that using an agentic workflow, and all of these right here, these small different dots, they basically describe different types of agentic workflows. We can see that Zero shop is just prompting the model with one kind of prompt. And then of course, we do have all of these different ways like language, agent, tree, search, LDB and reflection. All of these different things are ways to prompt an LLM to essentially make it more effective in some kind of feedback loop that gives you greater results. And these are things that you can try today. Now, of course, there are papers on these detailing how to do them. And sometimes they range from things that are quite hard to things that are quite easy. For example, there was this that I do really want to show you guys. There was a recent paper, and I think it was one of the most interesting papers because of what it actually showed us. So there was this paper called Mixture of Agents. If you are someone that's been paying attention within the AI community, this is a paper that has been popping up on your radar. So basically what we have here is that this is a agentic workflow that actually utilizes different AI models to basically refine your response many different times using different layers of agents. We can see here that there are essentially three layers. And for those of you who are non technical, including myself, and you don't want to get super confused, just think of it as you're using three different LLM systems and you're basically saying, I want you to judge the response and evaluate it three different times. So basically think about this like some kind of competition where, where you input your prompt and then three different LLM agents basically evaluate it every single time. We have round one, round two, and round three. Now you might be thinking, okay, mixture of Agents sounds cool. You have three judges. Every single time they refine the prompt and it continuously gets better. But the best thing about this is the actual effects of Mixture of Agents and this agentic workflow. So you, you can see here that they discuss that remarkably, this improvement occurs even when the auxiliary response provided by the other models are of lower quality than what an individual LLM could generate independently. So basically what they're stating here is that mixture of agents is something that improves ridiculously on a base model, even if the models combining together and talking together are dumber than the initial big model. You can see right here, you can see that these single models are different Open Source Models Uizard 8x22B Llama 3. You can see that with responses from other models they're able to improve on a single model that has more parameters and has more smartness. Now there was also something that was in this paper that I forgot to include, but it is really important. Basically in this paper they actually state that using mixture of agents using only open source LLMs and is the leader of the Alpaca Evaluation 2.0 by a substantial gap, achieving a score of 65.1% compared to 57.5% by GPT 4.0. Basically you know how OpenAI's got GPT 4.0 and they touted it as the smartest model. They managed to get a better benchmark result and increased reasoning capabilities by using open source LLMs that were substantially dumber than GPT4O. So overall using this agentic workflow managed to out compete GPT4O as a single model. And this is why the collaboration effect of agentic workflows actually are remarkably important. Next, we have AI agents that you can actually use today. Now this is a product called CrewAI. Crew is a collaborative working system designed to enable various AI agents to work together as a team to efficiently accomplishing complex tasks. Each agent will have a specific role resembling a team composed of researchers, writers and planners. And the main function features of CREWAI include role based agents providing each robot with a clear defined role and teamwork capabilities to enable agents to communicate, share task information and assist each other. Now this is a tool that you can run locally and many people can use this to. But this is something that I don't like because it's not easy to teach for non technical users because sometimes there are bugs and of course there are different issues that you can actually run into. Now some people do use this, but I don't really see anyone actually using CREWAI for too many real world application use cases. And that's one of the main, I'm not going to say issues with agents, but main things with agents is that there aren't many real world use cases that you can use them for today. Now, now I personally use Cassidy AI because I'm going to show you guys how with no code and with a simple prompt you can get a simple agentic workflow that you can use on a day to day basis. For example, I built this thing that's going to discuss business ideas. So my workflow input was discuss a business idea. Then the agent one considers the business ideas and its strength. Then agent two discusses it and then of course the agents come to a conclusion. But you might be thinking, okay, how on earth did you build this entire thing? How did you chain together this workflow? How hard was this? And this is why I love Cassidy AI. I'm not just promoting this for any kind of reason, but this is the only thing that I've found on the Internet that allows you to build instant no code agentic workflows. So for example, if I come to here and I want an agentic workflow, I can simply describe this in my own words. So I can say I want an agentic workflow that utilizes three agents that discuss my business idea and then come to a conclusion as to which one is the best long term. You can see right here. I want an agentic workflow that utilizes three agents that discuss my business ideas, then come to a conclusion as to which one is the best long term and why. So now I can simply click Create workflow. There are no fancy, you know, coding applications. I don't have to install anything, it just takes a minute. And the reason I like this so much is that we can use this in natural language. Now the reason I'm, you know, telling people about this is because I believe that this is how the majority of software applications are going to be built in the future. And whilst yes, that sounds pretty crazy, get getting better at prompting with natural language is not a bad idea at all. So right here you can see within a couple of seconds it managed to build this agentic workflow. You can see right here it managed to get agent one's analysis, agent two's analysis, agent three's analysis. And then of course the final agents discussing are going to be discussed by GPT4 and then it's going to give the best business idea long term and why. So for example, I can actually test this, this workflow right now. So in order to test this workflow I have a business idea. So of course it's going to analyze my business idea for long term success. So let's discuss a business idea that I could have for the YouTube channel. Let's say I have a YouTube channel channel in AI think I should launch a private community that helps people learn AI. So here I'm going to show you guys exactly what I've just done. I've got a YouTube channel in AI and I'm thinking about launching a private community that helps people to learn to prepare for AGI. And I do have this community. I'm just using this as an example, but I'm saying Should I do that or should I help, you know, launch a skincare line? And I'm wondering which business idea is the best? So essentially right now you can see the workflow is finished. So I'm basically just asking this AI system, look, do I launch this AGI community or do I launch a skincare line? Now for most of you the answer is obvious. You should launch a business that is related to AI. But of course in certain cases the answer might not be so obvious. That's why we use an agentic workflow. Here you can see Agent 1 analyzed the problem, then Agent 2 analyzed the problem, brought up many different things. Agent 3 also analyzed the problem. And then of course we have the agents discussion and, and we can see after carefully considering the perspectives of market demand, brand alignment, innovation, scalability, yada yada yada, the conclusion is here, you know, focus on your AGI community. Okay? It says the best business idea for long term success seems to be launching a private AGI community Preparation based on the following reasons. And now we have all of these different AI agencies, you know, all of their different opinions and we now have that AI information and advice in a very quick manner. This is why I like this software and this is why I'm telling you guys about it. Simply because this actually allows non technical individuals to immediately get access to building AI agent workflows and using them day to day. Now that was Cassidy AI, of course you can use that. Click the link in the description if you want, but just watch the rest of this video because of course there's a lot of stuff to continue to do. So now let's take a look at some of the other AI agents that are actually autonomous and I guess you could say do things on your behalf. And of course this is where things start to get interesting. So if we take a look at some of the AI agents that you can actually use today, you're going to see that things aren't as good as you think they are. So right here, this is Multi on. Multi on is an AI agent that essentially can be used in your browser to do very basic tasks. And this is not like an LLM based agent, this is more kind of like an agent that is able to scrape the web and do many different things. You can see that on the right hand side you have this agent that is able to book a restaurant reservation and is able to do certain things. Now these are the kind of agents that are essentially the most futuristic because they can actually go out into the real world and achieve real tasks based on what a user would want today. Not really a fancy demo, but the problem is is that Multion don't doesn't have that much usability in terms of what it's able to do. It's only able to do really, really niche stuff and it doesn't seem like they've made too much progress in that area. Now I'm not stating that to hate on Multeon. I'm just stating that because what you're about to see later on in the video is just how difficult AI agents are. What they've achieved is actually amazing. But in the grand scheme of things there are fundamental hurdles to to getting AI agents to run around the web clicking and achieving certain tasks. You can also see here that this is the multi on browser. You essentially click this right here and you can see that if you ask this AI agent, for example, you can see in the chatbot right here, book a one way flight to NYC for SFO for June 10th for the multi on hackathon. You can see that it is able to use Google and it is able to sometimes do that. You can see that it's your AI web copilot and it's able to perform a Google search. Now you can see right here that this actually allows you to have a button that allows to, you know, allows you to basically edge the AI on to be able to say yep, that is correct, you can go ahead and do that. And sometimes you're able to do this completely autonomously. Now I do want to say that this is as far as this AI stuff goes because sometimes the AI will make mistakes and it isn't essentially Multion's fault. There is just like this huge infrastructure problem that like I said I'll get to later on in the video, but this is basically going to be something that is the future of AI And I do think that will change things. Now, of course, in terms of AI agents, we also have something else. We also have AI agent devices. Now AI agent devices are a little bit stranger because they are basically AI assistants that are wrapped in hot hardware. So right here I'm pretty sure everybody knows what the rabbit R1 device is. This is a device that is heralded as the best currently AI agent. Now I do have to say hats off to Jesse Liu because when he created this, I do think that there is a remarkable level of scrutiny that he, you know, suffered from, albeit, you know, very warranted. But the point here is that this AI agent is not without its faults. This video basically showed this AI agent doing a remarkable number of different tasks in a remarkable number of different scenarios. The only problem was that the reality of this AI agent when it was released, didn't actually live up to the hype. If I want to plan a trip to London, I can just describe what I wish in full to R1. I want to take my family to London. It's going to be two of us and a child of age 12. We're thinking of January 30th to February 5th. Can you plan the entire trip for me? We like cheap nonstop flights, grouped seats, a cool SUV and a nice hotel that has WI fi.

Exploring ticketing options to make your trip a reality. For your trip, I've found various flight options, a range of hotels to choose from, and car rentals available. Please confirm each option individually for further details and booking.

So it's all been planned out. I just confirm, confirm, confirm and that's it. Could you come up with a schedule for fun things to do while I'm over there?

On it? I have prepared a detailed travel plan for your trip. In summary, you will be exploring London's iconic landmarks, visiting museums, enjoying delicious meals at local restaurants, and experiencing the vibrant atmosphere of different neighborhoods. Navigation details are also prepared and I can help you book tickets for any events you'd like to attend.

R1 just planned the entire trip for me. That's awesome.

But it seems like this is a.

Little bit too intense. Can you plan now? I'm not going to play the entire demo for you. You've probably already seen the rabbit R1 numerous times and like I said, the only problem about the R1 device is that was that there was a ton and ton of early criticism. Now like I said, if you're going to start a business, this is something that you of course have to be open to, but this is something that of course shows us what AI agents are going to be like in the future in terms of a hardware device. Now we need to get to the point to where big companies are actually starting to implement AI agents into the real world. One of the questions that I've seen floating around on Twitter is that nobody can seem to find a real world use case of AI agents doing something reliably. So here is Google's recent demo where they're actually showcasing their customer service agent that is able to help a customer with certain inquiries in real time. I'm so close to having this shirt for the concert. Let's give the store a call. Hi there, this is the Symbol Fashion.

Customer agent at South Las Vegas Boulevard.

Am I speaking With Amanda? Yes, this is Amanda. Great. Thanks for reaching out.

Amanda.

I see you had a session on another device. I've sent you an SMS message with a link to our live chat companion. If you would like to switch to chat, please click the link. How can I help you today?

I'd like to purchase the shirt in.

My cart with the card I have on file. Absolutely. I see you're also a Symbol Fashion Rewards member. Looks like you have a 20% off.

Voucher available to use.

Would you like to apply it to this purchase? Yes, please.

That would be great.

The shirt you're purchasing goes well with these items. Also available for pickup in your preferred size. Would any of these be interesting to you? Absolutely. Please add the white shirt and the boots to my cart. Great.

Your total is $203.73.

Okay to proceed with the cart on file? Yes, your purchase is confirmed. Do you need anything else today? No, I'm all set. Thank you. Incredible. Thank you for shopping with Simple Fashion.

You'll get a text when the items.

Are ready for you. In less than five minutes, I was able to find and order the shirt I wanted and an outfit to match. I'm ready for the concert. So this is currently what Google is cooking up. There is a longer video that showcases all of Google's agents in another extensive video. But let's take a look at what some of the other companies are doing. One of the main demos that you may have seen before is of course Devin, the main software engineer agent. This is essentially an agentic workflow wrapped around GPT4 that allows you to essentially benefit from the cognitive capabilities of GPT4 and get yourself hands on with an AI software engineer agent. Okay, I mean, I know they don't use the term agent, but this is essentially what it is. You can see that the workflow is here where he has a planner, he has a shell slash workspace and is able to go around and do tasks autonomously. So essentially take a look because this is going to keep coming up in certain videos. I'm going to ask Devin to benchmark the performance of Llama on a couple different API providers. From now on, Devin is in the driver's seat. First, Devin makes a step by step plan of how to tackle the problem. After that, it builds a whole project using all the same tools that a human software engineer would use. Devin has its own command line, its own code editor, its own and even its own browser. In this case, Devin decides to use the browser to pull up API documentation so that it can read up and Learn how to plug into each of these APIs. Here, Devin runs into an unexpected error. Devin actually decides to add a debugging print statement, reruns the code with the debugging print statement, and then uses the error in logs to figure out how to. Finally, Devin decides to build and deploy a website with full styling as the visualization. You can see the website here. All of this is possible today because of the advancements that we've made in both reasoning and long term planning. It's a really hard problem and we've only just started. So that was Devin. Cognition Labs Software Agent now other companies are working on really big stuff. For example, OpenAI is working on a form of agent to automate complex tasks effectively by taking over a customer's device. The customer could then ask the ChatGPT agent to transfer data from a document to a spreadsheet for analysis, for instance, or to automatically fill out expense reports and then enter them in accounting software. Those kind of requests would trigger the agent to perform the clicks, cursor movements and text typing and other actions humans take as they work with different other apps. According to a person with knowledge of the effort. Essentially, OpenAI is working on an agent that can, you know, control your computer and do your work for you. That's basically the logistics of what's going on here. So far we haven't actually managed to see OpenAI do anything just yet because it is very hard to get this done, as I'll show you guys in a minute. But this is essentially what they are working on. OpenAI recently released this right here where you can see their investment areas are in multimodal AI agents.

And last, we'll continue to invest in enabling agents. We're extremely excited about the future of agents and we shared a little bit about that vision back in November at Dev Day. And agents will be able to perceive and interact with the world using all of these modalities, just like human beings. And once again, that's where the multimodality story comes into play. Imagine an agent being able to kind of coordinate with multiple AI systems, but also securely access your data and even, yes, manage your calendar and things like that. We're very excited about agents. Devin, of course, is an amazing example of what agents can become like. Cognition Labs has built this awesome software engineer that can code alongside you, but he's able to break down complex tasks and actually browse the documentation online, submit pull requests and so on and so forth. It's really a glimpse into what we can expect for the future of agents.

Now, if you're wondering what Meta are up to, Meta are also developing AI agents that can complete tasks without human supervision, according to an internal post. They include an engineering agent to assist with coding and software development, similar to GitHub Copilot, according to the internal post, and two current employees. The post also cites monetization agents and that one current employee said that that would help the businesses advertise on Meta's apps and these agents could be for both internal use and for customers, the employee said. So essentially, Meta is also working on the same thing. We also have a private company called Imbu or Imbue. They are actually also working on AI agents and just recently raised another round where they are getting more funding in order to do this complex task.

We are writing an essay. We're not just generating an essay and then turning it in. That's not how our best essays get written. We look at our essay, we critique it, we say, hmm, this section needs more research. I'm going to go do some research and come back and then rewrite it. That iterative process of looking at something, reasoning through what the issues might be, figuring out where I need to ask questions, what the goal is, maybe changing my approach. Those are all required to actually accomplish goals in the world. At Imbue, we're training large foundation models optimized for reasoning. On top of those models, we build agents that we use to accelerate our own research. And these do more than just output something. They also iterate and reflect and figure out what's the next step to do and then take that next step. We're starting with agents that code because it requires complex reasoning to be able to code well, and because that's the work we do every day. And it's only through the serious use of these systems that we can really deeply understand how to improve the underlying reasoning models. What we're trying to do is actually get to reasoning models that we can ultimately build agents we can trust on top of all sorts of agents. It's hard to see because we're so in IT today, but AI models are like the very first electronic computer, which was just a calculator. What we're going to see is this explosion of enablement over the next 50 to 100 years. If we do things well, we have a world where we don't have to be glued to our screens anymore, where computers can help us thin the barrier between idea and execution.

So that now, after Imbue, I do want to talk about the difficulty of agents, because many people are, you know, I guess you could say wondering when AI agents are going to come. And by the looks of things, AI agents are going to be a lot further than people do think. This is a recent clip from Dario Amade speaking about the difficulty of AI agents and why they probably won't be here anytime soon.

If you want an agent to act.

In the world, usually that acting requires you to, you know, engage in a series of actions, right?

You talk to a chatbot, it only answers and maybe there's a little follow up. But with agents you might need to.

Take a bunch of actions, see what happens in the world or with a.

Human and then take more actions.

And so you need to do a long sequence of things. And for that long sequence of things to actually work, the error rate on each of the individual things has to be pretty low, right? If I'm a robot and I'm like, you know, okay, I'm going to pick.

Up this thing and walk over there.

And I'm going to pick up that, you know, I'm building the house or something, there's probably thousands of actions that go into that. And so all of this is to say the models need to get more.

Reliable because the individual steps need to have very low error rates.

And I think part of that will come from scale. Like we need another generation or two.

Of scale before the agents will really work.

There is also Mustafa Suleiman, who is the head of Microsoft's AI, former lead of Google DeepMind, stating that this also is the thing.

It's still pretty hard to get these models to follow instructions with subtlety and nuance over extended periods of time. I think that they can do it, you know, and there's a lot of cherry picked examples that are impressive, you know, on Twitter and stuff like that. But to really get it, to consistently do it in novel environments is pretty hard. And I think that it's going to be not one but two orders of magnitude more computation of training the models. So not GPT5, but more like GPT6 scale models. So I think we're talking about two years before we have systems that can really take action.

So this is why I was stating earlier that companies like Multi on and Imbue do face a very difficult problem because whilst yes, you can code an agent or whatever, if you're a startup, it's relatively hard to do that considering the fact that a lot of these foundation models that you're using simply don't have the scale to support the long term agentic workflows, task planning, whatever you want to call it that is needed to successfully do these things in the future. Now you can see here that he talk about, he spoke about GPT6 being the date that we could get agents. And this is actually further iterated by the fact that when we look at GPT6's trademark, one of the newest things that we do see is the fact that they say that GPT6 has the purpose of testing artificial intelligence agents. And this is not something that is in the trademark description for GPT5. So this further bolsters the claim that in the future it could be around two and a half years before we do get very effective agents that work at scale. And the reason this probably is most likely the case is because when you think about it, there are only a few companies that can really train models that are that big and that are that large, that are going to be able to competing in the agent space that really does work. So this is going to be a really interesting area for growth because this is going to be something that, you know, is quite difficult to do.

So when you ask a model to produce, you know, to, to complete a sequence of action, let's say it's like three things in, you know, to basically let's say to book, you know, a restaurant that you and I can go to on a certain day. Yeah, the first action would be check the availability in both of our calendars. So that's a correct function call, reconcile the correct moment. So that's the second action. Make sure that it's a restaurant that has availability. So that check is another one. And then you know, go and sign in so that you can basically, you know, use the correct tool to book the right restaurant at the right time. Put your credit card details down, having obviously also checked that it's a restaurant that we both like, etc. So it's like four or five or six different steps of, you know, just to produce that one quote unquote action subcomponents. Right. In order to get that right, you're basically saying that the model has to produce perfect function calling for each element and do so in sequence. So it can't just be arbitrary, it has to be in sequence. And that's like saying it has to write a four page document in response to one question that is exactly that document and it can't be something that is approximate or similar to that document. So we all think that obviously these models are magic at the moment and they write beautiful poetry and creative copy and text them, give you good answers and sometimes they're grounded and blah blah, blah. But when, when for each one of those answers, there's a wide range of correct answers that it could have picked. Right. Tens, hundreds, thousands maybe. So it isn't producing a specific perfect. Every single token that is outputted is correct answer for each one. It's not there yet. Right. So to get that level of precision, we have to scale up these two orders of magnitude. That's what's happened so far. The last, you know, five orders of magnitude of transformers. With every 10x of compute and data we get more precision. It's not just emergent capabilities that's wrong. People say, oh, it's surprising we had these emergent capabilities. That's an anthropomorphic projection. They're not surprising emerging, but it is just more precise attention to the correct mapping between a prompt and an output. Right. So you're just honing in on something more more specific now.

Do you think we'll be able to.

Get narrow forms of actions in specific domain before we get to GPT6?

Yeah, definitely. I mean there are some good actions today, right. You can definitely. You see these orchestrators making good API calls at the right time. The question is, can it do it with 99% accuracy? Because if it does it 80% then you know, one in five times it getting it wrong. It's not usable for a consumer. Right. So.

So the reason I included that clip is because it's the first time I truly saw it broken down how difficult it actually is to make AI agents actually work in different scenarios. Now one of the things that we also need to discuss is of course other companies future vision for agents. So we have Demis Hassabis talking about their vision for AI agents that I think will probably come a little bit earlier than OpenAI's.

For me, the highlight was probably this thing called Project Astra that we showed, which is our vision for what I'm calling a kind of universal assistant or universal AI agent could be like and how it could help in your everyday life that the main key thing about it is its multimodality ability to understand all the different modalities we operate in and beings in the context. So I think that was what's missing from the language agents is they didn't understand the context, the spatial context you were in and the environment you were in. So they can only be that that limits their use. We always had this vision of something that could understand the world around you through vision. You know, eventually audio and all the other sensors as well. And that's why we built Gemini, our large model to be series of models to be multimodal from the beginning, able to cope with any put and eventually and, and that's, that's that I think the vision that we have for maybe the next year or Smart Assist, which I think, you know, you could see.

Google there talking about how it's going to be a year or two which is much shorter than the GPT5 timeline. And of course Bill Gates also speaks about agents in this recent interview where he discusses the future of how it's going to work. Now previously I did a 30 minute video on Bill Gates blog post where he actually discusses how AI agents are truly going to change everything. And if you know in around two and a half years when we do have fundamentally different agents that are able to do things for you long term on your computer, that is truly going to change the way things go.

Agent mental therapy, agent brand, agent girlfriend, agent expert, all driven by deep AI. It seems like it will be useful in proportion to how much it knows about us. And I imagine at some point in the not too distant future probably all four of us will be asked if we want to turn on audio so our AI assistant can effectively like listen to our whole life. Right. And I would think that there'll be benefits to do that because we'll get good counsel, good advice. Do you think that's true and do you think will you turn it on when invited to turn on the audio? Well, computers today see every email message that I write and certainly digital channels are seen, you know, all my online meetings and, and phone calls. So you're already disclosing into digital systems a lot about yourself. And so yes the value added of the agent in terms of summarize that meeting or help me with those follow up, you know, be phenomenal and the agent will have different modes in terms of which of your information able to operate with. So there will be partitions that you have but for your essentially executive assistant agent, you know, you won't exclude much at all from that, that partition.

Rufus, before we.

So basically Bill Gates is just stating that your main AI agents there's not going to be much details they won't have because the more detail you give it, the more useful it is going to be. Now of course we do have Nvidia's CEO Jensen Huang basically stating their vision for AI agents and how the future is going to be. I think this is definitely one of the most insightful because he has a lot of connections and I can't imagine some of the conversations he's had.

All of these nen are experts that are now assembled as a team. So what's happening? The application layer has been changed. What used to be applications written with instructions are now applications that are assembling teams, assembling teams of AIs. Very few people know how to write programs. Almost everybody knows how to break down a problem and assemble teams. Every company I believe in the future will have a large collection of NIMs and you would bring down the experts that you want. You connect them into a team and you, you don't even have to figure out exactly how to connect them. You just give the mission to an agent, to a nim to figure out who to break the tasks down and who to give it to. And that central the leader of the application, if you will, the leader of the team would break down the task and give it to the various team members. The team members would do their perform their task, bring it back to the team leader. The team leader would reason about that and present an information back to you. Just like humans, this is in our near future.

So this is of course Nvidia's future vision for agents. And this is another video where he actually speaks about, you know, how agents are going to be collaborative in the workplace, how agents are going to be working with other companies agents and how they're going to be doing things.

Today Most of the AIs are one shot. You prompt it with something and it generates instantly the recommendation. In the future AI will be multi shot. It will be reasoning based system. Just as we plan through various complicated scenarios, it will do some planning itself. And so you're going to have fast thinking AIs like we currently have. You're going to have multi step reasoning AIs that will be coming along. These types of AIs will become increasingly agentic. And you demonstrated some early examples of this as well. AIs that takes, that uses tools, AIs that works with other AIs AIs that access information, they have privilege and access to access control to. And so in the future AIs are going to be like employees in our companies in the sense that our employees work with your employees. And so our AIs will work with other AIs and we'll have consulting AIs, we'll have specialist AIs, we'll have generous.

So this is basically a comprehensive guide on the kinds of AI agents that kind of exist and basically the holistic picture of AI agents. Now there's one last thing that I do want to actually show you guys, because it is a controversial opinion. Well, not really a controversial opinion, but it goes against what the public believe about how we're going to get AI agents. And one of the main things that many people have discussed is of course, autonomous agents. Now, autonomous agents are basically agents that, you know, do things on their own for quite some time for, you know, you know, any kind of reason. But here Mustafa Suleiman breaks down why autonomous, you know, agents aren't exactly that good.

I think, first of all, I don't think we're on a path towards fully autonomous. And I think that's actually quite undesirable. I think fully autonomous is quite dangerous. And you know, I got a lot of stick after my TED talk because I said that the autonomous capability was dangerous and so on. And that's one that should be regulated. And you know, I don't really care. I mean, I still think that, I think that if you have an agent that can formulate its own plans, come up with its own goals, acquire its own resources, just objectively speak, act completely independently of human, I'm objectively speaking, that is going to be more potentially risky than not. So I think about it as these like narrow veins of autonomy where you give it a specific goal and it has limited degrees of freedom to go off and act in some specific environment. So like making automatically calling some API to check some registry, to collect some information, to observe some state, maybe writing something into a third party API that is not yours, but is again restricted with some specific degrees of freedom. Because I think the security risks here are significant. So I just think we should tread carefully on the autonomous piece, but in terms of, on the actions piece.

So yeah, I think that essentially gives you a decent overview on where we're going to get with agents.

Huge.

One of the most common ones that we see being built are agents. And we've heard a lot about agents from a variety of speakers before. So I'm not going to go into too much of a deep kind of like overview, but at a high level it's using a language model to interact with the external world in a variety of forms. And so tool usage, memory planning, taking actions is kind of the high level gist. And the simple form of this you can maybe think of as just running an LLM in a for loop. So you ask the LLM what to do, you then go execute that, and then you ask it what to do again and then you keep on doing that until it decides it's done. So today I want to talk about some of the areas that, that I'm really excited about, that we see developers spending a lot of time in and really taking this idea of an agent and making it something that's production ready and real world and really the future of agents, as the title suggests. So there's three main things that I want to talk about and we've actually touched on all of these in some capacity already. So I think it's a great roundup. So planning, the user experience and memory. So for planning, Andrew covered this really nicely in his talk, but we see a few. The basic idea here is that if you think about running the LLM in a for loop, oftentimes there's multiple steps that it needs to take. And so when you're running it in a for loop, you're asking it implicitly to kind of reason and plan about what the best next step is. See the observation and then kind of like resume from there and think about what the next best step is right after that. Right now, at the moment, language models aren't really good enough to kind of do that reliably. And so we see a lot of external papers and external prompting strategies, kind of like enforcing planning in some method, whether this be planning steps explicitly upfront or reflection steps at the end to see if it's kind of like done everything correctly, as it should. I think the interesting thing here, thinking about the future is whether these types of prompting strategies and these types of cognitive architectures continue to be things that developers are building or whether they get built into the model APIs. As we heard Sam talk a little.

Bit about, the reason that that was so interesting for me is because these papers are essentially tree of thoughts, which is basically to basically get an AI to think a lot better. Reflection is basically where you get an AI to reflect back on its thoughts. For example, it might write something out. Then you say you just wrote this out. What do you think about this? How could you improve it? That's something that you can do. And basically he's stating here that future models might have these systems baked into the actual model. So it's not something that we have to then prompt the AI to do again. Which is why of course in the future this is going to be something that is really intriguing on how they managed to get the reasoning capabilities improved. But essentially further on in the video, he talks about how user interface is. You know, the user interface is going to be pretty, pretty nice. Of course he talks about Devin. This is why I spoke about how Devin is going to be something that is brought up quite a lot in this video. And of course, this is essentially the proposed Alpha Codium flow, which is a new workflow to kind of get better coding results. And what he talks about here is that this is a new agentic workflow that kind of delivers a lot more results in this essentially workflow, which is kind of different to before. And then of course, this is where he talks about agents for the future, that they're going to have memory. So this one right here as well, this kind of workflow was pretty crazy. Some crazy, crazy stuff.

Another kind of aspect of this is just the importance of basically flow engineering. And so this term I heard come out of this paper, Alpha Codium, it basically achieves state of the art, kind of like coding performance, not necessarily through better models or better prompting strategies, but through better flow engineering. So explicitly designing this kind of like graph or state machine type thing. And I think one way to think about this is you're actually offloading the planning of what to do to the human engineers who are doing that at the beginning. And so you're relying on that as a little bit of a crutch.

So basically they're just saying how we're going to get the AI to think in what steps, like what are the best steps it can take to generate the most efficient code. You can see, it's got all these different steps. It's got 1, 2, 3, 4, 5, 6, 7, here a 9, 10, like 10 steps before it generates a final solution. And it's really interesting how different steps and different ways that you interact with a base model can increase the capabilities, which is why a lot of people think that these future models are going to get so much more better. Now, let me know what you thought about this video. If you thought this was good, if it's actually updated your timelines on agents, if this helped you understand fundamentally what AI agents are, what they can do. Because there's a lot of information about AI agents this, AI agents that. And I just wanted to clear up the air on that stuff that we do see, because there's a lot of hype, there is also a lot of substance, but I think it's important to differentiate between the two and understand what's coming in the future.