OpenAI's upgraded ChatGPT can now see, hear, and speak. It identifies parts in images, reads manuals, generates images with DALL-E 3, and narrates stories. The update rolls out to ChatGPT Plus and Enterprise users within two weeks.
I was surprised to learn that GPT4ChatGPT, OpenAI's amazing assistant, was only trained on text information, yet it somehow still learned to see and to draw too. That was amazing. But today it gets even better. It actually learned to see and hear and speak at the same time. Wow. So how does this work? Well, let's ask it to help us lower our bike seat and find out. That should be easy. Right here I was thinking. Wait a minute. Why does this need an image? Is writing the text question not enough? Well, look, here are the instructions. Aha. Step number one. We need to locate the liver or. Or a bolt. Now we can post an image, mark it and ask, is this the lever? It says, nope, that is a bolt. And then it tells us we would need to loosen it. That is cool, but not mind blowing cool yet. And now comes the magic. You know what, little AI? Here is the manual. How about you read it? And here is the toolbox. Please help. We don't even know if we have the required tool. So do we. Look at that. Yes, we have the required tool. And not only that, but I love the fact that it even tells us exactly where it is. And bam. We are done. Excellent. We can even celebrate together with this little AI. Now let's. Let's get it to talk. With their new DALL E3 text to image system, we can now create adorable little imaginary animals like the Amazing Larry and even illustrate their lives. And we will now be able to ask ChatGPT to not only create, but also narrate a bedtime story. Here goes.
Once, in a tranquil woodland, there was a fluffy mama cat named Lila. One sunny day she cuddled with her playful kitten Milo under the shade of an old oak tree.
And we can also ask for recipes, poems and explanations as well. And I found something amazing here. Listen.
In a pocket or under a tree. Oh, where could my little keys be? I checked in the fridge behind the tv. Even the cat looked up as if to plea. They jingle, they jangle, they open the door. Yet they always end up on a different floor.
Did you notice? I think it is reading this not as a simple continuous text, but it reads it as a poem should be read. I love it already. The included paper system card reveals their safety considerations, which is highly appreciated as we know a little more about the inner workings of the technique. Not a full paper yet, but more information on when and how it should or should not be giving out medical information, because sometimes it is a bit inconsistent. And now it tries to resist jailbreak attempts where Users try to fool it into doing unsavory things. So when do we get to use this? This is coming to ChatGPT plus and enterprise users and it is possible that these features are already online by the time you see this video, but at most two weeks from now, according to OpenAI. But let's stop for a moment and appreciate how absolutely incredible it is that we are now here. Why? Well, here on Two Minute Papers we talked about GPT2, ChatGPT's predecessor before. Yes, such a thing existed and not even that long ago. Four years ago, in 2019. Then it could generate text and would be able to answer questions. And the fellow scholar with a keen eye noticed that things were starting to take off. This is an absolutely miraculous graph. Look, as we grew the neural network, it slowly started closing in on human level reading comprehension. Then for GPT3, just a year later in 2020, I said that it was good at almost everything. Not excellent good, but it could already code up websites, create graphs, help us create a population report, and oh, look at that. Reading comprehension just kept on increasing and increasing up to nearly the level of humans. Then in 2023, GPT4 appeared. This was a historic paper. Hmm, what is this? Yes, you are seeing correctly. This is the USA Biology Olympiad exam. This needs next level capabilities and a little help too, because otherwise it would require visually inspecting and evaluating electrocardiograms. So it got a little help in the form of a text description. So how correct was it? What, are you kidding? 99th percentile better than almost all humans. And now we can use it as an assistant and it can actually see and hear and speak. And all this progress from GPT2 in just four years and a couple papers. It almost feels like we are living in a science fiction world. What a time to be alive. And you all saw this history on two minute papers with Dr. Zsolai. And if you would like to see how this amazing story unfolds in the future, make sure to subscribe and hit the bell icon to not miss out on it. Weights and Biases provides tools to track your experiments in your deep learning projects. Their system is designed to save you a ton of time and money and it is actively used in projects at prestigious labs such as OpenAI, Toyota Research, GitHub and more. And the best part is that Weights and Biases is free for all individuals, academics and open source projects. It really is as good as it gets. Make sure to Visit them through WNB ME2MP or just click the link in the video description and you can get a free demo today. Our thanks to Weights and Biases for helping us make better videos for you.
