Nvidias NEW "AI AGENT" Will Change The WORLD! (Jim Fan)

So there was a recent TED talk that actually talked about AI agents, and it was a very fascinating TED talk presented by the senior research scientist at Nvidia and the lead of AI Agents Initiative. So this is Jim Than, the senior research scientist at Nvidia AI. And in this fascinating talk, he gives us a breakdown of where the future is headed with AI agents. He talks about something called the Foundation Agent, which would essentially seamlessly operate across both the virtual and physical worlds. And he explains how this technology could fundamentally change our lives, permeating everything from video games and metaverse to drones and humanoid robots, and explores how a single model could master these skills across different realities. Now, this found agent agent is not to be confused with AGI itself, because AGI refers to a level of artificial intelligence where a machine can understand, learn and apply its intelligence to solve any problem in a manner comparable to a human across a wide range of domains. Now, this idea of the Foundation Agent seems to be about creating a versatile multifunctional AI that can operate both in virtual and physical environments, mastering skills in various realities. Now, in this video, I was lucky enough to be part of a private discussion with Jim Fan himself, where he discussed the real future of foundation agents and discussed some of the research papers that he worked on which are going to help to contribute towards the development of the future research and development of the foundation agents and the industry as a whole. So I'm going to show you guys just a few seconds from his TED talk because it is one that shouldn't be missed at all, especially if you want to stay up to date on where everything is headed in AI. And then I'll share with you guys the conversation that we had about AI agents and some of the papers that Jim Fan worked on himself.

As we progress through this map, we will eventually get to the upper right corner, which is a single agent that generalizes across all three axes and that is the Foundation Agent. I believe training Foundation Agent will be very similar to ChatGPT. All language tasks can be expressed as texting and tax out, be it writing poetry, translating English to Spanish or coding Python. It's all the same. And ChatGPT simply scales this up massively across lots and lots of data. It's the same principle. The Foundation Agent takes as input an embodiment prompt and a task prompt and output actions. And we train it by simply scaling it up massively across lots and lots of realities. Yes. So the first work I want to cover is Voyager, and Voyager was one of the was the first LLM powered AI agent that can play Minecraft professionally. So I suppose most of you are familiar with Minecraft. It's got like 140 million active players, that's more than twice the population of UK. So it's kind of insanely popular and beloved game. And it's open ended, doesn't have a fixed storyline, you can do whatever your heart desires in the game. So we want an AI to have the same capabilities. And when we set Voyager loose in Minecraft, it's able to play the game for hours on end without any human intervention. The video here actually shows snippets from a single episode of Voyager. This is just a single run that lasted for four to five hours and we took some of the segments out and made this montage. You see that Voyager explores the terrains, mine all kinds of materials, fight monsters, craft hundreds of recipes, and it's able to unlock an ever expanding tree of skills. What is the magic behind it? The key insight is coding as action. Minecraft is a 3D world. But our most powerful alarms, at least at the time of voyager's writing, was GPT4 and it was text only. So we need a way to convert a 3D world into a textual representation. And thanks to the very enthusiastic Minecraft community, we actually have an open source JavaScript API that we can use. It's called Mindflayer. We use this code API and then Voyager is an algorithm designed on top of GPT4. The way it does it is to invoke GPT4 to generate a code snippet in JavaScript and each snippet is an executable skill in the game. Then once it writes the code, it will be run in the actual game runtime. And just like human engineers, the program that Voyager writes isn't always correct. So we have a self reflection mechanism to help it improve. And more Specifically, there are three different sources of self reflection. One is JavaScript execution error, where you know the agent's current state like hunger, health and inventory, or the world state like escape resources. Enemies nearby are fed to Voyager from, from, from the, from, from the agent state. And then given these state, the agent will take an action and then observe the consequence of the action on the world and on itself, reflect on how it could do better, try out more actions and rinse and repeat. And once a skill becomes mature, Voyager stores the program into a skill library so that it can quickly recall in the future. You can think of it as a code base authored entirely by GPT4. And in this way Voyager is able to bootstrap its own capabilities recursively as it explores and experiments in Minecraft. Because now we're talking about coding, and coding is compositional. Voyager can write a bunch of functions, and in the future, the future functions can compose some of the older functions in more and more complex skills and programs. So let's go through a working example together. Like the agent in Minecraft finds its hunger bar dropping to 1 out of 20. So it knows it needs to find food. And now it senses four entities nearby. A cat, a villager, a pig, and some wheat seed. So now it starts in a monologue, right? Do I kill the cat or the villager for food? That sounds like a bad idea. How about the wheat seed? I could grow a farm, but it's going to take a very long time. So, you know, sorry, piggy, you are the chosen one. And then Voyager checks the inventory, retrieves an old skill from the library to craft an iron sword, and then starts to learn a new skill called fun to pick. So that's kind of a working example of how Voyager would go through this loop. And. And the question still remains, how does Voyager keep exploring indefinitely? So all we did is to give Voyager a high level directive, obtain as many unique items as possible, and then Voyager implements a curriculum again by itself, to find progressively harder and novel challenges to solve. So I want to highlight that none of these are hard code. This progression of skills are discovered by Voyager itself as it explores. And also the curriculum that Voyager proposes is conditioned on its current capabilities. If you only know how to use wooden tools, then you probably shouldn't propose to solve some tasks that would require diamond tools. There's a progression of it. And Voyager is able to find this curriculum automatically. And putting all these together, Voyager is able to not only master, but also discover new skills along the way. And we didn't pre program any of these. It's all Voyager's idea. We simply took some snapshots from its playing session and that's what is shown here. And we call this process lifelong learning, where the agent is forever curious and also forever pursuing new adventures.

Have you guys considered putting more than one agent in the same server together and seeing if they can learn to interact with each other and complete tasks cooperatively?

That's a great idea. So we thought about it, but back then, like the. Well, I think that the framework doesn't really support multi agent. At least, you know, the framework we implemented does not quite support that, but it is on our kind of a future. So yeah, yeah, it is a very interesting question. And I do Think having multi agent would have new emergent properties.

Right? Yeah. Because my whole thought process was like long term we could see, you know, maybe we could have 30 plus agents all in a world building villages together and stuff like that. We could really see how they could develop different maybe ideals or goals over time and see what kind of separates them. I just thought that was interesting. Thanks for answering.

It is, it is very interesting. Yeah, yeah, it's such a great idea.

I remember in your TED talk you mentioned that how foundation agent is the way to go. From what I understand, Voyager is very successful thanks to Mind Dojo. So how are you and other Nvidia researchers going to overcome the data set curation barrier and able to have foundation agent to be able to play on one 10,000 other simulated realities and maybe Terraria per se?

Yes. So I think there are a couple of dimensions here. In my TED talk I talk about three axes. So the first axis is skills, the number of skills the agent can master. And the second one is the number of embodiment that it can control. And by embodiment I mean things like robot bodies. So you can have humanoid form factor or you can have a robot dog or agents. In Minecraft you have kind of different ways, like different bodies that you can control. So that's what we call embodiment. And the third axis is realities, basically the number of simulations that the agent can master. And here for Voyager we only tried it in Minecraft because it is an open ended world. It is like one simulation, but it's kind of like a meta simulation. Right. Like in this one simulation you can do so many different things. In fact infinitely infinite number of creative things. And we have seen humans doing crazy things in this world as well. Like some were actually built a functioning CPU circuit within Minecraft because Minecraft supports something called a Redstone circuit which apparently makes the game touring complete. It's like a programmable game. And Minecraft is just one kind of simulated reality. But also there are thousands of games out there, right? There's Legend of Zelda, there's Elden Ring, all of the open ended games. And there are also simulated realities for robots. And we also have the real world which is by itself, right, Our OG reality. So the way I see kind of the future of foundation models for Agent is that we need to scale across all the three axes I just talked about. We need to scale over the number of skills embodiments you can control. A single model can control all the robot bodies and then it can master all kinds of different Rules, mechanics and physics in different worlds, virtual and physical worlds alike. And here the idea is if a model is able to master, let's say, different simulated realities, then our real physical world could simply be the 100,000 first reality. So some of you might heard about something called the simulation hypothesis, right? Like saying that, oh, our real world is actually a simulation. I mean, we can talk about metaphysics and philosophy all day, but I actually think that idea is great to build AI, because for AI, our real world is just another simulation to it. We can actually use this principle to guide the design of our next generation of embodied AI systems. And that is kind of a quick recap of the main idea called foundation agent in my TED talk. Yeah. Does that answer your question?

Yeah. I was more curious about how it's going because data is probably going to be a key and how it's going to learn skills is depending on. I remember mind dojo. I forgot which one is relying on YouTube to learn all the Minecraft movements or Minecraft skills. So is it basically had to rely on all pre existing data or the whole data creation problem or would you have agent to simulate or naturally learn skills by itself in the future?

Yes, let me switch to the mind dojo slide and let me reshare it. I think you're right. We need some data to bootstrap the process. And for Minecraft specifically, this game is one of the most, maybe the most streamed games on YouTube. So there are like hundreds if not millions of hours of Minecraft play videos online. And in mindojo we explored this process, we explored this data set. So we collected a lot of YouTube videos where both kind of the gamer is playing the game and also narrating what they're doing. And these are like real segments from a tutorial video, right? Let's say video clip 3. As I raise my ax in front of this pig, there's only one thing you know is going to happen. This is actually some youtuber said this and we put it in a data set. So the way we use this model is we train something called a mind club. And to skip the technical details, what the model does is that it learns the association between the video and the transcript that describes the actions in the video. So let's say for this example, you know, this transcript, I'm going to go around gathering a little bit more wood from these trees. This transcript aligns very well with the activity in this video. So this score will be close to one. And this part is talking about pig. It's not aligned with this video and the score will be close to zero. So the score will always be between zero and one. And one means perfect description. Zero means the text is irrelevant. And you can treat this as a reward function. So concretely, how you would use it is you have an agent simulation, and then you have a task prompt asking it to share sheep to obtain wool. And as the agent explores, it will generate a video snippet, right? And then this video snippet can be compared to this language embedding and then output a score. And you want to maximize this score because that means your behavior is aligned with what the task prompt wants you to do. And this becomes a reinforcement learning loop. So it's actually ROI trek, if you look at. If you squeeze at it, it's ROI Trek, right? Learning reinforcement learning from human feedback in Minecraft. And just that the human feedback is not learned by annotating the data set manually, but from kind of getting the transcript and videos from YouTube. So that's how in the My Dojo paper, we're able to leverage this YouTube video data set. And moving forward, there are also other ways that we can use the video. I kind of briefly mentioned a few things on the slides as well. For example, you can learn encoding. You can learn encoding of the visual representations from the video. This work is applied in robotics, but it can also be used for things like Minecraft. And you can also even directly learn stunt behaviors from video by pseudo labeling the actions. So there are many ways kind of on how to use the videos to bootstrap embodied agents. And minddojo is a very particular way to do it.

So that.

Jim Daniel, I know you have a question. Like the action space was human annotated from different YouTube clips. I think you guys set up like a label studio set up and was like labeling. This is mining something. This is doing XYZ. But in Voyager, those actions were extracted by GPT4 and then saved in a database. So my question is, did you notice any actions that were found by the AI? Kind of like AlphaGo, one of your recent tweets, where it found moves that a human wouldn't do. It stored moves that a human wouldn't do. And like, I guess this is sort of an aside because I'm now realizing that the video data was all human actions. So I'm guessing that might not be the case. Yeah. So a few things to note. One is in mind. Dojo, the labeling part is about curating a set of tasks that could be possible in Minecraft. And we curated that set of tasks from some YouTube videos. But those are not the actions and are not used to train the model. So we only train the model using kind of transcripts from in the wild. And the manual curation is only for kind of. These are the interesting tasks that can be done, but we did not use that as actions. And for Voyager, coming back to your question, so it's able to kind of learn all these skills necessary to survive and to basically find new objects. Because we gave it. Yeah, let me kind of this one. So we give it a high level directive that is to maximize the number of objects you can obtain. So we would tell Voyager, your task is to maximize the novel objects that you can obtain. And so what Voyager does is trying to meet that kind of unsupervised objective. Because we are not telling it that you need to find diamond, you need to find stone first before you need to find iron, or you need to find iron before diamond. We did not tell it that. We just say you need to find as many novel objects as possible. And we actually have a way to measure it. Right? We can look at its inventory and then count the number of diverse items it's able to obtain through its lifespan. So we can actually quantitatively measure it. And let me show you a figure here. So we actually have like a comparison with some prior works is this one. Basically this is like react reflecting on some kind of baselines and auto GPT. And this is Voyager and this is Voyager that. The blue one is Voyager without the skill library. And here in this figure, the X axis is the number of prompting iterations and the Y axis is the number of distinct objects it's able to uncover or craft. It doesn't matter as long as you're in inventory. We see a new object, we count it towards the progress. So it's got this high level objective programming tool and mostly the skills I would say a human will be able to do. And Voyager is not able to build crazy things yet because that would require vision. And in the original Voyager, we did not have computer vision. Right. It's not doing the task from pixels, it's converting the world to text. And that will be a limitation. So if you want to build Castle, you got to see what you're building, right. Otherwise it's really hard to kind of tell you the 3D coordinates and try to reason in your head, even for humans. Really hard. So Voyager doesn't do building tasks because we, we, we didn't ask it to. And also it's not quite capable of, because of the limitation of of its perception module to you, what is the strategic value of a corpus like YouTube, right, for training these type of open ended embodied agents? Like are these agents going to be able to make sense of the different.

Rules of the world?

You know, as they vary in simulations versus real world data? Physics, for example, varies drastically. So what is your thoughts? So I think the way to build one of the components to build foundation agent is really good video models that can understand not just Minecraft video, but maybe videos from many different games or even videos of the real world of people doing different tasks. We want to train on as many videos as possible because what videos encode is something that we technically call intuitive physics. So when humans, when all of us, right, go ahead to do our daily tasks, we don't solve physics equations in our head, right? Like if you kind of, if you drop a cup on the floor, your brain cannot compute exactly where the water is going to spill or you know, how the cup is going to be broken. Right. You cannot simulate all of that, but you roughly know that you're going to make a mess like the water is going to spill and the cup, if it's a glass cup, it's mostly going to be broken. You have a rough common sense of where things are going and that is the predictive model in our brain and what we call intuitive physics. It's not physics, it's intuitive. Right? We cannot compute every trajectory. And I think for the current embodied agents, they lack this common sense. They can't really predict, you know, what's going on next. They don't have this intuitive physics built into their brain. And to learn intuitive physics, I believe the best way is to learn on lots and lots of videos. And once you have that common sense model, it's still not enough, right? Like you can predict what's going on next, but you still don't know how to act. So just like if you watch, you know, tennis champions playing tennis, you can watch it all day and you know what's going to happen next. You have a predictive model in your brain, but can you play tennis as well as the best players? Right. You still need a lot of practice to actually ground the common sense that you learn from the videos. And that's how I see the simulations come into play. So you need both the videos and a lot of pre training and also the simulations, be it Minecraft or Physics Sim or some other games, to really ground the knowledge through trial and error. And that's how I see, that's how I believe we should build the Next. Embodied systems. I hope that answers your question. Yeah, it does. Is that how you see Omniverse fitting.

Into all of this?

Right, Like Tesla's like noisy data at scale. But we're going to need sort of synthetic training data or like, you know, I guess open ended agents trying stuff in the in simulated environments too. Yes. How about this? Let me share screen. This is Eureka. It is a five finger robot hand that's able to do pen spinning tricks in Nvidia simulation. How we are able to train this is actually using something called ISAAC SIM that is built on top of Omniverse in terms of abstraction levels. You can think of Omniverse as a base level graphics engine. It runs on the latest GPUs, it's able to get acceleration hardware, native acceleration and it does rendering physics and all of that. It's in Omniverse. And the Isaac SIM is a library built on top of Omniverse for robotics specifically. So it's able to import things like robot hand models, input objects, compute the contact physics of the fingers with the pen here. And most importantly and probably the most unique feature of i6 Sim is its scalability. So you can run 10,000 environments in parallel on a single GPU, which means you are basically speeding up reality by 10,000x. In the real world you are bottlenecked by real physics. Right? You simply cannot collect data with this level of throughput. But in simulation you can if you throw compute at it. And with parallel computing you can simulate 10,000 robot hands doing pen spinning at the same time. And in this way you scale up the data stream and you can train very complex policies like pen spinning that would otherwise take maybe a decade of real world time. If you want to do this directly on a physical robot, it's very slow. So yeah, that's how ICMD simulation comes into play for embodied agents. And since we're talking about Eureka, I will just quickly cover this work. So how is Eureka trained? Basically, I see Eureka has two loops. The outer loop is a language model here, GPT4 writing code in a physics simulation API. And this code will become the reward function. And reinforcement learning requires a reward function so that you have something to maximize, right? Something to work towards. And that is the second loop. Inner loop is that given a reward function. We do reinforcement learning to train another neural network that controls the robot hand. And then this dual loop system is what makes Eureka quite unique. You can think of this as system 1 and 2. Thinking from the book thinking fast and slow. The LLM loop is a system 2 loop because it's doing high level reasoning. It's looking at how the hand is performing and then proposing change in code. So it's like a system two, deliberate, slow reasoning. And the loop on the right is a System 1 loop. It is like fast unconscious. Like, you don't do reasoning when you're spinning, Pat. Right. It's more like a feeling of it. It's muscle memory. So the loop on the right hand side will be the system one where we have a smaller neural network, but it's much higher frequency than the LLM and it's able to control the hand to do very dexterous tasks. And we are able to do not just peg spinning, but like a few other kind of manual manipulation tasks for the robot. I don't think I'm showing this here, but basically this method is general purpose and it's not limited to just pen spinning. Okay. And I'll open up for like five minutes of questions.

Thank you very much. I'll try to make this quick. So I know you said. Yeah, I think in the paper it says that the reward functions can essentially be updated in somewhat real time. Is that correct? You don't have to retrain the entire model.

So for the reward function, like, it's updated every time the inner loop or the loop on the right hand side has finished. So you can think of like this loop as a full reinforcement learning training session. We train it to go to convergence, and then it will have like a performance metric which it can report back to GPT4, and then GPT4 will propose the next reward function.

Okay, so the future that I'm seeing with this here is that we could have a bot that actually exists in the real world and we could potentially, with similar architecture, train a bot by actually showing it, you know, an example and then. And then it practice. Practices that same example itself. So I'm just wondering if you guys seem to be very focused on robotics. So is this the future that you guys are looking towards?

Yeah, I think there are many ways we can scale Eureka even more. Right? Like one is, you know, it skills, assimilation skills. And here we are learning like one skill at a time, right? This pen spinning skill is like one Eureka rock. But you can imagine that we can do maybe 1000 different skills in parallel if we throw a lot of GPUs at it. So that is something that we are thinking about doing. And actually in this video, you can see there are a lot of other tasks that we tried, but each task is like a separate neural network. We're not training a single one that does multitasking, but it is an obvious next step that we can do. And the other thing is to actually make it work in the real world. And that will involve sim2real. How do we transfer the neural network learning simulation to the real world? And there are many techniques to it. One is called domain randomization, which is basically the simulation hypothesis I just mentioned. If you're able to master 10,000 different simulated realities or different physical configurations in SIM, let's say if you're able to work with Earth's gravity and also Moon's gravity, and also Mars gravity in simulation, 10,000 of them, then you are very likely able to generalize to the real world, which will be very complex and not quite the same as the simulation. The simulation is always going to be inaccurate portrayal of the real world. But that's how we can overcome the same real gap.

I feel like Eureka is a very underrated research paper for last year. It's probably my favorite. And is it the first of its kind? Sorry, is it the first of its kind where LLM trained robot, like, is it fully LLM trained robot? And if so, is there like a bridge being built right now where skill learned in ISEC gym can be applied to real world robots?

Yeah. So first, thanks for your kind words. So there are a few works on kind of combining LLMs and robotics. There are also some works from, you know, Berkeley, from Stanford, from some of the universities. But I think Eureka, at least to my knowledge, should be the first on using LLM to kind of instruct how to train robot. Right. You can see Eureka as automating the development of robotics, right? Because typically the reward functions are written by human engineers. Who's like robot developers. Robot engineers. And not like every developer can write reward function. It's actually very specific. You got to have like domain expertise on how to use physics simulation. You got to be familiar with the whole framework to be able to do it. It's not even easy for any programmer to do this without training. But here we found that GPT4 is so good at zero shot at understanding documentation. So we simply feed Nvidia's physics API documentation to GPT4 and then it writes these reward functions and it can actually write it better than the human developers. So we see Eureka as a first step towards automating the development of robotics itself. Right. If you think of robotics, it's basically a bunch of code. Ultimately it's just coding. Right? Like, can an entire robot stack be programmed not by us, but by GPT4 or whatever is coming next and then can do it iteratively? That is a fascinating question.

So would it be reasonable to describe it as to the first AI agent trained robot in a simulation, then the.

First kind of like ll instructed AI agent. Yeah, the first LL trained kind of.

Like a full AI agent agent concept. And there is there for robotics?

Yeah, sorry, go ahead.

Is there like a possibility of. Have you heard of Mamba, like the architecture?

Mamba, yeah.

Is there a possibility of Transformer being replaced for Mamba in robot Sims or robot learning, like in Vima? Sorry, that might be off topic.

Vima, Yes. I think it's an orthogonal question, because the architecture part, I mean, it will be useful, but it is not the pain point of robot research. We have not even exhausted the potential of transformers yet. I think the hard part for robotics is data. Right. How do you get data for it? The data can come from Internet videos as we just covered. It can also come from scaling up simulation. And for simulation, it's a little bit special because the data is generated by the agent itself.

Right.

It's kind of an actively collected data versus Internet will be passively connected data. So data is the bottleneck. We can use whatever architecture we want. And if Mamba replaces transform in the future, we're happy to switch, but it's not kind of the pain point right now. Got it.

Thank you so much.