Dr. Zsolnai shows four uses of GPT-4: robots handling everyday tasks like arranging sushi, building a playable video game from scratch with multimodal feedback, mathematician Terence Tao using it to generate expert research questions, and automating file organization. The key is zero-shot capability and chain-of-thought prompting.
Chatgpt today has incredible capabilities. It is like an engineer who helps you write a video game. It is like a scholar who helps you write a paper. And it is like an assistant who organizes your life and files. Dear fellow scholars, this is Two Minute Papers with Dr. Zsolnai today. Large language model assistants understand text and they understand images. They have for quite a while now. However, connecting the two still remains a challenge. But with GPT4's vision model, we can dump all this information into a robot and it will be able to take commands from us and make a plan and perform it. And it gets better. It promises that it also has some common sense, something that is not common at all in today's world. So can it deliver on these promises? Let's see together through three experiments. Experiment one here we would like to get some Coke. But look. Haha. There is no Coke anywhere to be seen. Now's your time to shine little AI. And then. Nice. It opens the fridge very gently, gets the coke, closes the fridge and there we go. Excellent. It can also do similar other gentle tasks such as get some Pringles and pour it onto a plate. Experiment two, pick up the marvel model. It is of course behind a strategically placed glass and without additional instructions here it puts it away safely. And now we can play. Great. Now experiment three. This is a bit harder perhaps organize the sushi pieces to be like they are in this image. A tricky multimodal question. Can it do it? Oh yes. Well done little AI. Now two good news and one bad news. Good news number one. Yes, the this one is also doing one of my favorites and that is learning safely in a computer simulation. First, and only when it demonstrated that it is smart and safe can it move around in the real world. Good news number two. This can perform all of these tasks in a zero shot manner. This means that it can perform a broad spectrum of these everyday tasks without being specifically trained for it. What does that mean exactly? Well, if you give it a task that it hasn't seen before, it can lean on its knowledge of the world a bit like a human would and have a good chance of still being able to do it. If you ask me, that is one possible definition of intelligence. Now, less good news. Not even this technique is perfect. Look, Chain of thought. This means that with the prompts it gets a little help. And at the risk of simplifying, this help comes in the form of a few exemplars that outlines the reasoning process for a few tasks for the AI to lean on. It's not that bad. And I bet that just one more paper down the line and this might not even be needed. But today ChatGPT can do so much more. For instance, here it helps this fellow scholar write a video game without the user having to write a single line of code. First, we need to briefly describe the game and it gets to work. And then the first version is already playable. Excellent. Wait, the missiles are not exploding. So is the dream over? Of course not. Look after taking a couple screenshots of the game, don't forget GPT is multimodal. It can look at images and then we can explain the issues. We need some explosions, collision detection. And look at that, a significantly better game. In a matter of seconds. Explosions are now happening. Now add a scoreboard to the game, a start screen and a damage health system. It now explains what it is about to do. And there we go. A proper little arcade game. This is like having a super fast, super smart engineer at your fingertips. This is absolutely incredible. This is not only an engineer, but it can explain mathematical jokes. This is one of my favorites. This one is about explaining the difference between discrete and continuous mathematics. Love it. But wait, it's not only good for explaining mathematical jokes, but also for doing mathematics and writing papers. Take this from one of the best mathematicians in the world today, Terence Tao. Even he is using GPT4 and he says, as I could feed GPT4 the first few PDF pages of a recent math preprint and get it to generate a half dozen intelligent questions that an expert attending a talk on the preprint could ask. Also quote, I plan to use variants of such prompts to prepare my future presentations or. Or to begin reading a technically complex paper. Whoa. One of the finest mathematicians of our generation is asking it for help and gets meaningful questions and sometimes answers. But wait a second. Does that not sound like a helpful colleague or co author to you? Well, hold on to your papers, fellow scholars, because here comes the bombshell. When integrated with tools such as formal proof verifiers, Internet search and symbolic math packages, I expect say 2026 level AI, when used properly, will be a trustworthy co author in mathematical research and in many other fields as well. Wow. And we are not done yet. Not even close. I have one more amazing application for you. It can also take a bunch of your files and take some work off of your shoulders by once again using common sense and chucking them into the appropriate directories. Clutter is now gone. Now, of course, this workflow is not as convenient as it could be yet, but just imagine if this kind of useful tool could be integrated deeper into the operating system and not have to work outside of it. That would help us spend less and less time on organizing things and create and learn something new instead. Love it. So good. So fellow scholars, what you are seeing here is perhaps the birth of a new tool that might become more useful than any other tool I have ever seen. Wow. What a time to be alive. When I was a PhD student, my experiments created lots of data for my deep learning projects. But I was not looking for more data, I was looking for insights. This is what weights and biases does and it is the best everyone is using it. Try it out now at WNB Me Papers or click the link in the description below.
