Not Slowing Down: GAIA-1 to GPT Vision Tips, Nvidia B100 to Bard vs LLaVA

For AI progress to slow down, it would need to run out of data, compute and algorithmic efficiency. But developments this week suggest that the field isn't running out of any of these things, let alone all of them. I'm going to give you a glimpse of what this means in robotics, audio and vision, and end with some practical tips to help you use GPT vision, as well as comparing it to Bard and Lava. But let's start with Gaia 1 from Wave, which is generating the synthetic video that you can see now. And no, I'm not just bringing it up because it looks cool. The CEO this week said, I believe synthetic training data is the future for AI because it's safer, cheaper and infinitely scalable. That's my point. When synthetic data gets this good, we're not going to run out of data. Many of you may not know that GPT4 itself was trained on some synthetic data. And if you're interested, do check out my videos on Orca and PHI one to see how much synthetic data can actually help smaller language models. And the synthetic video data you just saw came from a scrappy outsider training on fewer than 100 Nvidia A1 hundreds. Now imagine the kind of synthetic data that Tesla could come up with with the equivalent of 300,000 A1 hundreds. And of course, Tesla already has billions of hours of real world data that compares to the 4700 hours that Gaia One was trained on. Now, many of you might say that, yes, it's crazy that things are improving this quickly with synthetic video data. And yes, it's cool that a model like this can generate unlimited data adversarial examples. What does that mean, by the way, in this context? Well, for example, people walking across the road, jaywalking in the fog. Even Tesla, with its billions of hours of real world data, probably only saw that scenario a limited number of times. But impressive as it is, isn't this just for autonomous driving? No, not even close. This is also for real World Robotics. Just two days ago, we got Unisim from UC Berkeley and Google DeepMind and MIT and the University of Alberta. You can play with the demo yourself, but can simulate a range of things, like unveiling toothpaste, picking up the toothpaste in multiple steps. Now, you probably don't need me to tell you why unlimited training data for robotics might be useful. I'll let you watch this imaginary demo of a robot closing the bottom drawer and opening it and opening the top drawer. The authors themselves give many suggestions, such as controllable content creation in games and movies. Point I was trying to make is that this might be way more significant than just controlling cutcenes and generating your own movie ending. As the authors say, the true value of UNISIM lies in simulating long episodes to enable optimizing decisions through search planning or reinforcement learning. You can imagine a robot planning out a series of actions and being able to visualize them all internally. It can even visualize human activities and imagine how a human might respond to its robotic actions. And yes, those shower thoughts or imaginary plans do pan out when they try it in the real world. It learns to do something in UNISIM and then is able to do it in real life. And what's crazy is that the accuracy of all of these simulations follow the same scaling laws as apply to large language models. In both cases, the authors of Gaia say the task is simplified to the prediction of the next token. Now, limited data was one of the main things holding back humanoid robotics, so expect more such demos like this one in in the near future. This is of course the famous Tesla bot called Optimus. Now this is not to say that it's coming to a home near you anytime soon. First of all, you've got production capacity. Second of all, you might have prohibitive costs so that only really rich people can get something like this. And thirdly, we would need more task specific data, real or synthetic. I'm talking data of it. Folding laundry, maybe walking the dog. I'll let you type in the comments. Any other ideas we need data for, as Disney showed this week, doesn't mean we can't have robots focused on entertainment. This little robot is not only incredibly cute, but it can withstand having a carpet pulled from underneath it. It can also be nudged and not fall over, and was mainly 3D printed. This is the kind of robot trained on real and synthetic data that I can imagine coming fairly soon. Now, I don't know if I personally would let it roam the woods, as you can see in this video, because I'd be worried that a dog or a bear maybe if you're living in North America might just come and rip it. And given how crazy expensive it's going to be at first, that would be quite the expense. Now of course many of my viewers might be a bit more cynical and thinking who needs a dumb little robot? But what about one integrated with GPT4? Or let's be honest, by the time this comes, GPT5 or Gemini, wondering if.

I could dictate to it to say something specifically, what is your battery level?

Battery level is currently at 53%. What is the voice coming from?

It's a Google text to see speech.

We give the JSON to ChatGPT and explain what the structure is and how to read that JSON. Now, ChatGPT can answer questions about that, Jason, but how many inspections in your next mission? My next mission involves 20 inspections.

Now, at the moment, the only criticism of this, you might say, is that the voice sounds a little bit synthetic even there. Total realism is probably less than a year away. Here is my voice after a quick clone on Elevenlabs. I think the voices on this, these robo pets will sound a lot more realistic in the very near future. Now, I promise I'm almost done with robotics, but there was one comment in my previous video on RT2X which said I'll be impressed when it can clean a toilet. Well, just for that person, here is a new video from August of this year from the company Somatic. This is how well robots are doing without mass synthetic data. Imagine when they can simulate millions of hotel rooms, or even harder, your bedroom. And if synthetic data shows us that data won't be a bottleneck. How about compute? That's the raw hardware needed to perform all of these computations. Nvidia typically drops its big new data center GPUs every two years. So we've already gone from A1 hundreds to H1 hundreds this year. That will train the next generation of models like GPT5 and Claude 3. But according to this week's exclusive report in semianalysis, that cadence will now become yearly. That means that in 2024 next year we will get the B100 series of GPUs and the year after 2025, the X100 series. That should mean that new models are faster to train and cheaper to use. Now, of course, all of this means that AI won't be slowing down. Even if companies like OpenAI can't find algorithmic efficiencies, that's basically doing more with the same amount of compute. But of course, with the kind of revenue that they're generating now. This is a report information Yesterday. Companies like OpenAI can reinvest in more staff and generate better efficiency, apparently. At the developer conference which is happening on November 6, OpenAI plan to announce improvements to ChatGPT features that speed up the performance of its language models and a product to lower the cost of using its AI. But I was particularly intrigued by this from Reuters, the access via API of GPT Vision. They say the company also plans to unveil new tools such as vision capabilities that will enable developers to build apps with the ability to analyze images and describe them. This will enable feedback loops like the one I'm about to demonstrate. I asked Dall? E3 to create a coffee mug that says let's think sip by sip. And I gotta be honest, I thought what it created was pretty exceptional. It doesn't always do this well, but look at the range of styles and genres. But here's the feedback loop that I mentioned. I took four more outputs from Dall? E3. These weren't quite as good as the ones you just saw. I said scrutinize the text in these images and provide a metric for each based on how closely the characters displayed match the text. Let's think sip by sip. It then did extremely well. It said the first image matches perfectly 100%. The second used periods instead of hyphens and it gave that a score of 95%. And even on the harder third one, it said the SIP has been changed to SIBCA and it gave it a 90%, which was a little generous. But the point is, this entire process could be be internal images could be created by Dall? E, internally rated, and then only the best ones put out. Apparently the only reason they're not doing this now is because of compute costs. But to cut a long story short, I think text generation in images is about to be a hell of a lot better next year. And it seems like others agree. CCS Insights say that image generation and voice synthesis is getting so good that arrests will start being made as early as next year. For deepfakes, we are rapidly climbing up the other side of the Uncanny valley and I agree with their prediction. In fact, it was only four days ago that we heard audio of Keir Starmer, which I can't play here. Basically effing and blinding and we can't even agree if it's fake or not. The evidence strongly suggests that it is fake, but isn't it kind of scary that we can't 100% tell anymore? But anyway, I promised some GPT4 vision tips and some comparisons with Bard and Lava. Just yesterday I got access to GPT vision and I've been playing about with it for hours since. I've noticed for tables it often makes minor mistakes where it gets one number wrong in a table that can cause it to get questions wrong. Like this one. Which country has the highest percentage of population visiting the zoo aquarium? That would be China according to this table. But it said the US despite getting the percentage right 48%. I even tried to help it out with visual pointers giving two examples of successful analysis. But even with those pointers it still got the question about the zoo aquarium wrong. This time it picked the third highest, Japan at 45%. Final failure. I asked it to list the bottom three countries in terms of the percentage visiting the Science Technology Museum and this time it skipped over both Japan and South Korea to list the EU as the one with the third lowest percentage. So what is my tip and let me know in the comments if any of you find this helpful. Well, drawing a bit on few shotting and self consistency, I gave it three different angles of the same chart. But even more crucially than that, perhaps I asked it recreate the data from the tables. I then said check for any dissimilarities and resolve them by majority vote. The reason I did this is that I noticed that sometimes it could output a correct table and still get the analysis wrong, even though it's simple mathematics. So what this was doing was splitting the task up into two first it was reducing the chance of minor errors by giving it three different angles and second it was getting it to do the analysis only after it had already recreated the tables and look at the difference. This time when I asked it about the bottom three countries it got it right. And then I asked it again what was it about the zoo aquarium. That was the one it got wrong twice before as you saw. This time it correctly picked out China at 51%. If you're wondering by the way how I got different versions of the same image, it was by pressing Windows Shift and S and then just just highlighting like this. Anyway, I think that's a cool tip. Try it out. Let me know in the comments if it's at all helpful. But finally, let's compare Lava and Bard to GPT vision on text. Lava didn't do as well. It wasn't able to notice that this coffee cup missed out the B in sip by sip. Bard not only noticed, but even came up with an amazing metric to find the distance between the two texts, the prompt and what came out. Another difference I found between the models was when it came came to faces. I asked what was the fate of this character Saruman. GPT4 successfully said the character is Saruman and gave the fate of that character Bard flat out refused saying sorry, I can't help with images of people yet, while Lava was kind of helpful saying the character in the image is Gandalf. What about some of those table questions like I was giving GPT4 earlier? Well Bard kind of flopped, saying that the answer for the Zoo was the US, but it at least got the percentage correct at 51%. Lava did less well saying that the answer was Brazil. Now maybe this demo doesn't reflect the full capabilities of lava, because I read the paper that came with the announcement of lava 1.5. Apparently it got 80% in visual question answering version 2. That's less than 1 of Google's models, which I've talked about before parley 17 billion, but apparently better than GPT4. So you guys can let me know if I'm missing anything about the capabilities of lava. Just before I move on though, I can't help but say that I was really impressed by GPT4's analysis of this image. I asked what is poignant and unexpected about this image and it picked up on the contrast between the devastating event that's unfolding and the seemingly calm demeanor of the observers. It picked up on almost every detail of the image and it was a fantastic answer. I saw that yesterday someone had the idea of putting the Mona Lisa into GPT vision, asking it to describe the image. It then got Dall E3 to generate an image based on that description and put it into a recursive loop. And this was the result of that recursive loop. And with the explosion in synthetic data and compute, I predict the world will get equally crazy quite soon. Thank you as ever for watching to the end and have a wonderful day.