The Worlds First FULLY AUTONOMOUS Robotics System Is Here (Physical A.I)

So there is a new company called Physical Intelligence and they are really innovating the game of robotics. In this video, I'll dive into exactly what they're doing and what makes them so special. On their blog post it starts by saying that we are living through an AI revolution. The past decade witnessed practically useful AI assistants and AI systems that can generate photorealistic images and videos and even video models that can predict the structure of proteins. But in spite of all these advances, human intelligence dramatically outpaces AI when it comes to the physical world. And to paraphrase Moravec's paradox, winning a game of chess or discovering a new drug represent easy problems for AI to solve. But folding a shirt or cleaning up a table requires solving some of the most difficult engineering problems ever conceived. To build AI systems that have the kind of physically situated versatility utility that humans possess, we need a new approach and we need to make systems embodied so that they can acquire physical intelligence. Over the past eight months we've developed a general purpose Robot foundation model that we call PI zero and we believe that this is the first step towards our long term goal of developing artificial physical intelligence so that users can simply ask robots to perform any tasks they want, just like they can ask any LLM and chatbot assistance. Like LLMs, our model is trained on broad and diverse data and can follow various text instructions. And unlike LLMs, it spans images, text and actions and acquires physical intelligence by training on embodied experience from robots, learning to directly output low level motor commands via a novel architecture. And it can control a variety of different robots and and can either be prompted to carry out the desired task or fine tuned to specialize it to challenging application scenarios. So this is where they talk about the promise of generalist robot policies. It says today's robots are narrow specialists. Industrial robots are programmed for repetitive motions in choreographed settings, repeatedly making the same world in the same spot on an assembly line or dropping the same item into the same box. Even such simple behaviors require require extensive manual engineering. And more complex behaviors in messy real world environments such as homes are simply infeasible. They state that if we could train a single generalist robot policy that can perform a wide range of different skills and control a wide range of different robots, we could overcome this challenge. And such a model would only need only a little bit of data from each robot and each application. Just as a person learns a new skill quickly by drawing on a lifetime worth of experience, such a generalist robot policy could be specialized to new tasks with only modest amounts of Data. And this would not be the first time that a generalist model beat a specialist at the specialist's own tasks. Language models have superseded more specialized language processing systems because they can better solve these downstream specialist tasks by drawing on their diverse and general purpose pre training. In the same way that LLMs provide a foundation model for language, these generalist robot policies will provide a robot foundation model for physical intelligence. Now, to get there, they're going to need to solve major technical challenges. They say that their first step is PI 0, which is a prototype model that combines large scale multitask and multi robot data collection with a new network architecture that enables the most capable and dexterous generalist robot policy to date. And while they believe that this is only a small early step towards developing truly general purpose robot models, they think it represents an exciting step that provides a glimpse of what is to come. And I personally do believe that this company is doing something absolutely incredible. Taking a look at the majority of these demos in real time, I can see that the tasks that these robots are faced with are genuinely, really hard. How do you manage to train a robot on how to fold a shirt when there's a million different ways to fold a shirt? Shirt. When you're picking up the shirt, it could crumple, it could fall this way, it could fall that way. There's a million different scenarios that these robots do have to face. And watching these robots perform these actions and complete them successfully time and time again shows me that physical intelligence, their first generalist policy, PI zero is steps ahead on the other robot demos that I've seen, especially some of the ones that have broken the Internet. What you need to understand is that this is something that right now is fully autonomous. And none of the demos that you're currently seeing are tele operated. It's all fully autonomous, being done by their one single generalist policy. Here you can see their cross embodiment training mixture, which is where pi0 the brain uses Internet scale vision language, pre training, open source robot manipulation data sets and their own data sets consisting of dexterous tasks from eight distinct robots. And then the model can perform a variety of tasks via either zero shot prompting or fine tuning. Entire dataset contains diverse tasks, with each task exhibiting a wide variety of motion primitives, many different objects and various scenes. And the tasks in this dataset exercise different dimensions of robot dexterity while covering the range of real tasks that these robots might be asked to perform. From bussing dishes, to packaging items into envelopes, to folding clothing, to routing cables, assembling boxes Plugging in power plugs, packaging food into go boxes and picking up and throwing out the trash. And their goal in selecting these tasks is not to solve any particular application, but to provide their model with a general understanding of physical interactions, which is basically the initial foundation for physical intelligence. Basically stating that look, we're not training it for anything specifically, we're going to train it so that it understands a huge variety of different tasks and that way it becomes smarter overall. Inheriting Internet Scale Semantic understanding says that beyond training on many different robots, Pizarro inherits semantic knowledge and visual understanding from Internet scale pre training by starting from a pre trained vision language model and VLMs are trained to model text and images on the web. Widely used VLMs include GPT4 Vision and GPT Gemini and we use a smaller 3 billion parameter VLM as a starting point and adapt it for real time dexterous robot control. Now VLMs effectively transfer semantic knowledge from the web, but they are trained to output only discrete language tokens and dexterous robot manipulation requires PI 0 to output motor commands at a high frequency up to 50 times per second. And to provide this level of clarity, the they state that they've developed a novel method to augment pre trained VLMs with continuous action outputs via flow matching, a variant of diffusion models. Starting from a diverse robot data and a VLM pre trained on Internet scale data. We train our vision language action flow matching model which we can then post train on high quality robot data to solve a range of downstream so next we have the post training for dexterous manipulation which is basically where they just want to train the model and in a very specific way for certain tasks that are really difficult. They're basically stating that fine tuning the model with high quality data for challenging tasks like folding laundry is quite similar with how post training processes are employed by the LLM designers. And pre training teaches the model about the physical world, while fine tuning for robots basically just makes it perform at a particular task really really well. Now they did one task which is of course laundry and and they fine tuned PI zero to fold laundry using either a mobile robot or a fixed pair of arms and the goal was to get it to get the clothing into a neat stack. This task is exceptionally difficult for robots and of course some humans. While a single T shirt laid on the flat table can sometimes be folded by repeating a pre scripted set of motions, a pile of tangled laundry can be crumbled in many different ways. So it's not enough to just simply, you know, move the arms through the same kind of motion and to their knowledge, okay, no prior robot system has been demonstrated to perform this task at this level of complexity. Basically what they're saying is here is that, like, look, this robot is currently state of the art. Robots can be folded in a variety of different ways and ensuring that you have repeatable success, which is what they are currently demonstrating here, is something that we simply haven't seen before, and it's particularly hard to do now. They also talk about table bussing. They said, we also fine tuned the model to bus a table. This requires the robot to pick up the dishes and trash on the table, putting any dishes, cutlery, or cups into a bussing bin, and putting trash into the trash bin. Now, this task requires the robot to handle a dizzying variety of items. One of the exciting consequences of training PI 0 on a large and diverse datasets was the range of emergent strategies that the robot employed. Instead of simply grasping each item in turn, the model could stack multiple dishes to put them into the bin together, or shake off trash from a plate into the garbage before placing the plate into the butting bin. This is rather fascinating. We often talk about how robots and AI models have these emergent capabilities. But having this robot being able to stack multiple dishes to put them in the bin together or, or shake off trash from a plate to put it into the garbage before placing the plate into the bussing bin is actually pretty cool. It's quite similar to what humans would do in that scenario. Now, of course, this is where they have something that seems absolutely incredible, which is where they got this robot to assemble a box. So it says here the robot has to take a flattened cardboard box and build it, folding the sides and then tucking in the flaps. And this is very difficult because each fold and tuck might fail in unexpected ways. The robot needs to watch its progress and adjust as it goes. It also needs to brace the box with both arms, even using the table so that the partially folded box doesn't come apart, which is pretty incredible when you see exactly how this robot is able to do that. There's a lot of subtle information that goes in to training these robots. So seeing them do this firsthand is truly rather fascinating. I know a few people that struggle to fold certain bottles. Now, of course, they evaluate this model. So it says. We compared the PI zero to other Robot foundation models that have been proposed in the academic literature on their tasks. They've got OpenVLA, a 7 billion parameter VLA model that uses discretized actions, an octo, a 93 million parameter model that uses diffusion outputs. And these tasks are very difficult compared to those that are used in academic experiments. For example, the tasks used in the open VLA evaluation typically consist of single stage behaviors, which is, you know, putting an eggplant into a pot. Whereas our simplest bussing tasks, which is pretty difficult because it consists of multiple objects and sorting those into either a garbage bin or a bussing bin. And their more complex tasks might require multiple stages, manipulation of deformable objects and the ability to deploy one of many possible strategies to given the current configuration of the environment. And these tasks are evaluated according to a scoring rubric that assigns a score of one for a fully successful completion with partial credit for partially correct execution. For example, busting half the objects leads to a score of 0.5. Now the average scores across five zero shot evaluation tasks are shown in the graph that you can see on screen. Comparing the full Pi0 pre trained model to a small Pi0 small which is a 470 parameter model that does not use VLM pre training. So OpenVLA and Octo, although OpenVLA and Octo can attain non zero performance on the easiest of tasks of Pi0 is by far the best performing model across all of the tasks and the small version of PI 0 attains the second best performance. But there is more than a two times improvement in performance from using the full size architecture with VLM pre training. Now they also state where they go from here. They state that the mission of physical intelligence is to develop foundation models that can control any robot to perform any tasks. And their experiments show and their experiments so far show that models can control a variety of robots and perform tasks that no prior robot learning system has done successfully, such as folding laundry from a hamper or assembling a cardboard box. But generalist robot policies are still in their infancy and we have a long way to go. And the frontiers of robot foundation model research include long horizon reasoning and planning, autonomous self improvement, robustness and safety. And they expect that the coming year you'll see major advances along all of these directions. But the initial results paint a promising picture for the future of robot foundation models. Highly capable generalist policies that inherit semantic understanding from Internet scale pre training. They also incorporate data from many tasks and robot platforms and enable unprecedented dexterity and physical capability. And they also think that succeeding at this will require not only new technologies, but a ton more data. And of course a collective effort involving the entire robotics community. They've already got collaborations underway with a number of different companies and robotics labs both to refine hardware designs for teleoperation and autonomy, and incorporate data from their partners into their pre trained models so that they can provide access to models adapted to their specific platforms. Overall, I think physical intelligence is absolutely incredible. They've managed to demonstrate that their models are able to successfully perform a variety of different tasks that we previously would have thought would be teleoperated. But now we can see entire operations from start to finish, completely autonomous in a zero shot setting. So it will be interesting to see how this company develops over time.