Metas NEW Secret Robotics Project Is Finally Here!

So you know that company Meta, the one that done Llama three and was absolutely amazing, They've actually done something even more amazing with their company. So they're actually releasing new research artifacts that are supporting their goal, which is essentially AGI, but they're calling it Advanced Machine Intelligence or AMI for short. And this is basically about how their new division is going to be working on robotics. And it is pretty cool.

Hi, I'm Mike.

And I'm Mustafa.

And we're working on touch perception at Meta Fair. Our Meta Fair team is excited to release several cutting edge developments in robotics and touch perception that mark a significant step forward in our understanding of the physical world and our ability to interact with it in meaningful ways.

Let's begin with Sparsh. Sparsh is the first general purpose touch representation that works across any tactile sensor and any task. To achieve this, we train Sparsh using 460,000 tactile images with self supervised learning in benchmarks we find that sparse outperforms task and sensor specific models by an average of over 95%. Sparsh unlocks pre trained backbones that allow us to measure properties like forces and contact which are imperceptible through vision. Our aim is to empower the community to build and scale these models towards novel applications.

Which leads us to digit360, a breakthrough tactile sensor that perceives the world with human level touch sensing capabilities with on device AI. Digit360 processes information locally and makes decisions inspired by the reflex function and people and animals. This allows a sensor to respond quickly to various inputs such as the poke of a needle or the flex of a tennis ball. It responds quickly to various inputs mimicking the performance of a human finger, making it an innovative solution for multiple applications. We've also paired Digit360 with DigiPlexis, a platform that standardizes robotic sensor connections and interactions mimicking the human hands touch information processing for embodied AI. DigiPlexis provides hardware and software components for easy integration of touch sensing technology into robotic systems.

Finally, we are partnering with gelsight and Whonnik Robotics who are leaders in the industry to develop and commercialize these touch sensing innovations. Our hope is to bring the community along and work together in driving the progress in AI and robotics beyond robotics. These advancements can help bring the physical world and the digital world closer together. Imagine picking up an object in a game or scanning a piece of fabric during online shopping. This could even lead to advancements in prosthetics.

Our hope is to build a driving community and ultimately make touch sensing practical out there in the real world. This work brings us one step closer to understanding what it means to digitize touch. And that future is almost within reach.

This is actually pretty fascinating because we actually got to see for the first time, one of these companies actually focus on something that isn't being focused on in terms of the mainstream robotics. What we're seeing here is a company that is focusing on touch perception. Now this doesn't seem like much at a glance. Like you might not think that this has wild scale industry use and applications, but this is one of the things that I think is going to connect us to robotics in a completely different way. And always, I would say that Meta is always focused on areas that most people just simply overlook. So the touch perception thing is absolutely crazy because it's going to allow next level robotics and human interaction that understands exactly what it's interacting with. A lot of humanoids we see today, cool. They have nice hands, they're able to touch things, but they don't know exactly what it is that they're engaging with unless they're of course using a vision system. And if you have this kind of touch sensitive area, you know, quite like human hands, we literally are able to prick ourselves with needles and we're able to grab things and feel the kinds of sensitivities which is really important with engaging with the actual physical world. It's all well and good having a robot arm that's able to, you know, grab things, but if it doesn't understand exactly how gentle it should be, or how much strength it needs to be use, or how much pressure, or how much sensitivity, or even for example, how hot something is. Because there are many different times that, you know, when we touch things that are completely hot, we immediately remove our hands. And do robots do that? I would argue that that isn't something that's currently baked in now, it's pretty crazy. The Meta are focusing on. They're actually focusing on this as well as a lot of other areas when it comes to AGI and I'm truly excited about this stuff. Now. One of the things that they actually spoke about in their blog post, which is something that I didn't even know Meta had, but it is a sign that they are going to be moving forward when it comes to their robotics department in a way that we haven't seen report. So in the video they actually talked about how they're going to be partnering with some companies. And one of the companies they actually spoke about was this company called Partner. So it says as we move closer to a future with intelligent robots and advanced AI models capable of performing everyday household chores, it's important to consider their interaction with humans. And that's why we're releasing a benchmark for planning and reasoning tasks and Human Robot Collaboration partner, which is designed to study human robot collaboration and household activities. And basically they said here, okay, what they're going to do is they're going to be training and testing social embodied agents on physical hardware with actual human partners in this environment. So of course you know that like training robots is actually very hard. And a lot of the times what we do do is we actually train these robots in a simulated environment and that's exactly what what they're going to be doing. So they're going to be using something called Habitat 3.0, which is a highly realistic real time simulator that supports both robots and humanoid avatars, that allows for human robot collaboration and home like environments with the goal of testing physical world scenarios, which is really incredible. So essentially Habitat 3.0 does seem a little bit weird, but this is a essential scenario that companies need to place themselves in. Because if you're going to be deploying human humanoid robots at scale, you need to have a system that allows you to test and ensure that these robots are safe. So these environments are basically similar to real households. And the researchers are going to be using this software to study embodied AI agents that can navigate, interact and assist in the same environment as humans while addressing major challenges in the real world. And of course this is something that I think is really, really important. Now, once again, okay, having Meta on this area where they're going to be looking at areas for realistic embodied agents in virtual environments for humans is going to be something is really essential for us to advance in that area. There's also something that I think this enables. One of the things that I think this kind of technology actually does enable is of course prosthetics. Now most people aren't going to be thinking about prosthetics because the majority of us aren't disabled. We do have all of our limbs and we are fortunate enough to be able bodied. Now of course those who are unfortunate enough to have to wear a prosthetic, one of the things that I would say would definitely help them is of course the ability to have digits360 baked into a prosthetic limb, because this is going to allow them to have realistic touch sensation. With over 8 million taxels capturing detailed tactile information, the Digit 360 can actually provide these individuals with nuanced touch sensation. And this actually means that prosthetic limbs can relay information about pressure, texture, and even the subtle contours of an object directly to the user. And of course, this will have temperature detection, so the ability to sense and perceive temperature changes, which is crucial for, for safety and performing tasks that require temperature sensitivity. And you're going to be able to have vibration awareness. Detecting vibrations helps users interpret environmental cues, such as the operation of machinery or the texture of surfaces while running your hand over them. And of course, this is going to allow for improved motor control and dexterity. When you have an on device AI that's baked into this, that enables the prosthetic to process sensory information and react in real time. This allows for immediate reaction speeds, quite like how humans have it, and allows for immediate adjustments in grip or movement when interacting with objects. And this is also going to allow fine manipulation. Detailed tactile feedback enhances the user's ability to perform delicate tasks, such as picking up small objects, typing or handling tools, by providing precise control over these movements. And think about it like this, okay? Right now, we're not in the craziest place when it comes to prosthetics or BCIs, which are brain computer interfaces. But imagine, when combined with BCIS, how the Digi360 is going to be able to facilitate more natural control of the prosthetic limb by directly interpreting neural signals and providing sensory feedback to the nervous system. Overall, this seems to be something that's going to be absolutely incredible. And I can't wait for this to be baked into the prosthetics so that individuals can have a better quality of life and of course, that we can advance.