Dr. Zsolnai demos Devin, an AI that codes like a software engineer: it plans tasks, edits code, debugs, and extends projects. It can build a Game of Life app or fix open-source bugs, but solves only about one in six hard GitHub tasks.
We just talked about the first published AI virus and what do I see? The first AI software engineer has appeared. Now wait a minute. We always talk about AlphaCode, ChatGPT and other AI assistants being really good at coding. So what is new here? Oh boy, look at that. The difference is that this is an AI that that is designed to take a task from us and then start working as a real software engineer would. It will make a plan, use the command line, a code editor and it will check for references within this browser. And we can supervise it like we would supervise a real person. It is called Devin. So how good is Devin? Let's see through four amazing examples. Dear fellow scholars, this is two minute papers with Dr. Zsolnai. Well look, it had found something behaving in a way that it didn't expect to. So it will try to pop the hood and find out what went wrong. Very much like a real human would. Really cool. But it gets better. So much better. One, we can for instance ask it to create a browser app where we can play the game of life, a cellular automata based simulation. Okay, that's cool. But this is something that has tons of implementations online, so I would like to see a bit of creativity here. For instance, start a new world with the letters of your own name and fix a bug where the screen is freezing. And since it works similarly to a real human, it is happy to show the work it has done. We can take a look and decide what whether we accept claims to have fixed the bug too. So let's see the result. Now we're talking. This is something that in this form has not been done before. Ok, so it can fix bugs and do some creative stuff. That's great, but can it extend and help with already existing code bases? Let's see through this open source project. 2. Here several processes were run at the same time and unfortunately they all failed. But they don't really say why. Fortunately this case doesn't even need a proper description because other people have also requested it. Just chuck the link in there and now little AI, you figure it out. It writes a plan, installs the dependencies, looks up the website and the coding can start. Hmm. Now it claims to have finished and oh my, look at that beauty. It now shows the status codes so we get more information as to why these processes have failed. Finally we can inspect the code and accept it. So it can contribute to already existing projects too. That is incredible. Now three, it can even take a real paid computer vision project. It starts out fixing this seemingly never ending list of issues with great patience and speed. And the incredible thing is that we can see through each step it has taken and it shows very human like behavior. It is understandable for us what it has done and easy to evaluate. And now finally it writes a report and marks the damage in the road on these images. For now, hold on to your papers, fellow scholars, because it can also be asked to train a different AI. Once again, issues need to be fixed. They do get fixed by reinstalling a few packages and off we go training an AI. So this is an AI training a different AI. Wow. What a time to be alive. Now before we get too excited about this, not so fast. This tool also has its own limitations. When tested on a difficult dataset that contains real software bugs from GitHub that need fixing, previous techniques had a one digit percentage success rate. 4, 5% is considered good and this one. This one is way better. Wow. However, this is still lucky to solve one out of six of these difficult problems. It is given an incredible leap forward, but there is still so much work to do and of course ultimately we are in charge here. It's just an assistant, but a super helpful one at that. If you're looking for inexpensive cloud GPUs for AI, Lambda now offers the best prices in the world for GPU cloud compute. No commitments or negotiation required. Just sign up and launch an instance and hold on to your papers. Because with the Lambda GPU cloud you can now get on demand H100 instances and they are one of the first cloud providers to offer publicly available on demand H100 access. Did I mention they also offer persistent storage? So join researchers at organizations like Apple, MIT and Caltech in using Lambda Cloud instances, workstations or servers. Make sure to go to lambdalabs.compapers to sign up for one of their amazing GPU instances today.
