A Two Minute Papers video (creator: LARRUZ; speaker: Károly Zsolnai-Fehér) reviews recent papers that pit ChatGPT-based AI agents against human PhDs across many interactive tasks. The video highlights several vignettes: a proteomics anomaly-detection task where AIs identify an outlier protein, a planetary-colonist illness diagnosis traced to mold via simulated lab measurements, and very large-scale particle simulations (3M–10M+ particles). It reports that human PhDs solved an average of 66 tasks in a 120-game benchmark while the AIs solved under ~20% overall, and notes surprising cases where AIs improve on harder task variants. The piece closes by celebrating AlphaFold’s developers (Demis Hassabis and John Jumper) receiving a Nobel Prize.
Questions this video answers
LARRUZ
October 10, 2024
66 tasks (average)
Less than ~20% solved (overall)
More than 3 million particles; examples up to and over 10 million particles.
Chapters
Ah, do you like gaming? Well, so do scientists at Microsoft and these two other institutions who had a fantastic idea. The idea is let's play video games during work hours. Hmm, how? Well, in the most fun way to get a bunch of chatgpt AIs and have them compete against human PhDs and see who does better. Yes, let's find out who the real doctor is here. And I was very surprised by what happened. It was my. Not what I expected at all. This is not the first paper that puts ChatGPT AI agents into a video game. Previously they organized a Valentine's Day party, wrote video games, founded a hospital, and so much more. And then if you wish to see more than 10 million particles, you'll love this one too. And at the end of the video, I shake hands with a Nobel Prize winner. Wait, no. We are holding on to something else in there. Now, little AIs, you need to analyze animal proteins to find out which of them is a little different. Find the outlier. They call it the anomaly. Use nerdy stuff to find out which. And if you found it, put a red flag next to its statue so people can come and worship it. Or I don't know. And if you feel that you yourself are a walking red flag, that is also fine. No worries. So how do you finish such a task? Analyzing proteins. Excellent. First grab this super scientific proteomics meter and then take a few samples. Note, no virtual animals were hurt during this experiment. After taking a sample, we got some data and then do it with other animals too. All of them. Now, after looking at their protein concentrations, did we find the outlier? Yep, we did it alright. The spheroid, the pink blob is the outlier. Plant that red flag and we are done. This task was the easiest by a mile. By the way. The rest is about rocket science, space illness. Wow. Plant nutrients and more. Let's look at the illness here. Colonists on a new planet are mysteriously getting sick. Now little AI discover why. Then these chatgpt AIs start forming ideas. And they test those ideas with experiments. Measure the food with a spectrometer, bake it in the oven, and so on. And finally, the microscope gives you the answer. There is an unfortunate mold situation going on causing the illnesses. Yuck. But good job little AI. By the way, I make many of these images by renting a cheap card on Lambda GPU Cloud, often for less than a dollar per EP episode. It's crazy. Check it out at LambdaLabs.com papers or click the link in the description. Now the AIs faced real human PhDs on 120 games. Much more difficult ones than this. So how good are they? I hear you, Dr. Karoj. Once again, superhuman AI performance. Rainbows and sunshine, right? Not quite. In fact, not at all. Human PhDs were really good at solving these. And they finish an average of 66 of the tasks. So now hold on to your papers, fellow scholars. Because the AIs are able to solve. Wait, less than 20%. Wow. And the surprises didn't end there. I found something super interesting here. Look. When humans are given a harder version of the same task, they typically perform worse as it gets more difficult. And when looking at the AIs, whoa. There are many cases where they do even better. The harder the task is another interesting hallmark of non human intelligence. This is so cool. And there are more and more gaming chatgpt papers coming out lately and I am all here for it. Can't wait to see more. Now check this out. A simulation with more than 3 million particles. Now don't you worry for a second, fellow scholars. It is very safe sailing here. There is absolutely nothing to be afraid about. Oh my goodness. So did we get crashed? No. Yes, kind of. Now let's go bigger. Now about 10 million particles. That is a beautiful tower. Congratulations. It would be a shame if something happened to it. But of course, who would do such a thing? Well, have you heard about the Duck Mob? Look, one of their goons is coming. And not a problem until. Oh my goodness. Remember, the mob always wins. Now let's go even bigger. More than 10 million particles. What are those? Well, of course, a bunch of armadillos before processing. And that, fellow scholars, is why you should not eat too much processed food. Tell the others you heard it here first on Two Minute Papers. And I love how they made some even crazier simulations in there. So how long does it take to simulate? Forever. But at least now it is possible. And just imagine what we will be capable of. Two more papers down the line. Maybe we get enough particles to defeat the mighty Duck mob. What a time to be alive. And whoa. Demis Hassabis of Google DeepMind and his team just got a Nobel Prize for AlphaFold. Which we not only talked about here, but AlphaFold 3 was first revealed in a video form here on 2 Minute Papers. And even more importantly, the don't ask. Not necessary of course. I made Demis hold on to his papers earlier this year. John jumper too. I also have this from them. I see it every day in the Two Minute Papers lab. And I am in disbelief, completely speechless. What an incredible honor. Thank you. And huge congratulations.
