Zapier Co-Founder on AI Agents and the Path to AGI | Mike Knoop (Zapier)

I think there's like two kind of broad categories of automation. One is API based automation and then there's another type of automation that is what's called like robotic process automation is the industry name for rpa. It's basically keyboard and mouse automation if you want to think about them. And they have trait, they have like pros and cons. The API based automation, this is what Zapier does. It's very permission oriented or you have to go get API keys, but it's also way more reliable. The API space is much smaller so it's simpler to create automations over. And this is why Zapier without any AI to date or was able to sort of get as many users as we did. The downsides of it are that the API is limited and this is where RPA kind of like gets, you know, especially exciting just due to the sort of nature of the technology. Without any AI it tends to be very hard to set up because you have to like literally like you know, oh, follow my mouse and keyboard and copy this sequence of patterns. And so I think that's why generally like Zapier and the API based has sort of won the day so far. Now importantly I've sort of described the world without AI. I do think that like this RPA based sort of future is a much better one if we can figure out how to get it to work because you get anything that a sort of human could do with a given piece of software. My belief today is that we need weak forms of AGI in order to get the reliability and the setup, ease of use high enough on let's call it the RPA style of automation in order to get bring consumer products to market on that but certainly within the realm of possibility now. But the full version I think we all want, you know, can work with any app, learn anything I can give it.

All right, so my guest today is Mike Noop, co founder of Zapier. Mike led Zapier's push into AI agents that can automate daily tasks and he also recently founded the ArcPrize to accelerate development of open AGI. So welcome Mike.

Yeah, thank you Peter. Excited to chat about AI here today.

Yeah, so let's talk about Zapier first for people who are not familiar, like can you describe what is a zap is and like you know why AI might be a great fit?

Yeah, Zapier is an automation platform. Our sort of mission is to put automation into the hands of everyone, including non technical folks. So folks who don't know how to code you know, aren't an engineer developer. In fact, indeed, that's like the majority of Zapier users are just like line of business folks, you know, marketing, sales support, product management, design, being able to like learn and figure out how to use Zapier to automate workflows that they use between a lot of the apps that they use day to day. And one of the things that I started realizing a couple years ago as I was getting into AI is that sort of the promise of automation and AI are kind of the same thing actually. You know, it's both like the desire for computers just to do more work for us. Right. So we don't have to do the grunt work and do the lame kind of like grindy work that, you know, a lot of our jobs sort of entail. And once I realized that like, ah, that's you know, both why our like, best users love Zapier and also why people are so like, enamored and love at the promise and dreams of AI. It's the same thing. And so, you know, I think I sort of realized, you know, oh yeah, Zapier is like an AI company. It might not know it yet, but it is.

Yeah. And like I wrote this whole blog post on Zapier AI because I really believed, like, after you learn how to use like Claude and chatgpt, like, I feel like Zapier is the next tool.

We should really learn how to use.

Because, you know, it can actually do work for you when you're not there. Like with ChatGPT, I still have to talk to it all the time. And like you, let's talk about how you let the push of Zapier AI, or just AI in general inside the company in the first place. How do you get people to even start using AI stuff?

Yeah, yeah. So I mean, I guess it sort of the genesis comes, goes back to my time in college. I did mechanical engineering, sort of focused on doing finite element analysis and optimization. You know, imagine sort of like trying to optimize the structure of like a, a race car to optimize weight versus performance. And it turned out, and I didn't know this at the time, but all the math that you need in order to be really good at that, modeling those types of systems software, it's all linear algebra and statistics. It's the exact same math that underlies deep learning. And so when I sort of realized that maybe 2015, 2016, I got kind of like into AI, but mostly as a curiosity. Like, you know, I had a day job at zapier that was my main focus. But I paid attention to a lot of the developments in deep learning and sort of the language model stuff. And so I kind of, you know, had been paying attention to GPT 2 and then 3 when it came out. Heck, I gave a whole, I remember giving a whole like presentation to at an all hands at Zapier on like how cool GPT3 was and how it worked and trying to get people like figuring out use cases for it. And you know, I feel like I had a pretty good model of what these large language models could do and what they couldn't do. And then in January 2022 there was this original like chain of thought paper. The sort of let's think step by step paper got published and I read that and all these like reasoning benchmarks that people had created to test language models were all spiking based on this like technique from I don't know, 30% accuracy up to like 70 or 80% accuracy. And that was my sort of first like, whoa, moment of hey, this stuff can do something that I didn't price in to what I thought it could do. And that has a really material impact both from like Zapier customer perspective on automation, but also just like as a human, like I'm curious, like Shooter, if we're on path for, you know, artificial general intelligence in a short timeline, like, like I want to know that and I would like to do what I can to make that happen because I think there's a lot of benefit from that. And so that I gave. I was still an exec at Zapier. I was running half the company at all product engineering reporting it for me. And fortunately Zapier has multiple co founders and I was able to give it all back over to Wade. I said, wade, you got to run this so that I can go just be an AI researcher. I think it's the most important thing I can do here at this point. And fortunately he said yes. And so that's what I basically did for the next six or six months, 12 months, me and my other co founder Brian is the cto. We just wrote code pretty much all day trying to prototype with these like language model products to kind of get a sense of what they could do, what they couldn't do. And the very first thing we identified, and this was, remember like summer 2022, so before even ChatGPT came out, we had sort of found that like tool use was one of the most important things that Zapier felt like we could like help out with because Zapier has a lot of tools, right? All the integrations on Zapier you could think of those tools, all the actions and like these language models were frozen in time, right? Because they were sort of frozen from a training perspective on their weights. And so we started prototyping with tool use and that's kind of ultimately what led us to getting involved in the OpenAI original ChatGPT plugin launch. And I think really sort of accelerated Zapier's awareness into AI and from there kind of just snowballed on awareness and users were coming to us and telling us what they wanted to do. And you know, now we've got like, I don't know, four or five. I mean we got AI pretty much across the entire product. We have several dedicated AI products as well, but all kind of started from that original kind of moment of got it.

And, and how did you encourage the average employee as Zapier to actually use Zapier AI or like any other AI tool? Or like you're just going like, you know, yeah, how does it do it?

I was looking at the numbers. We. So again we, we are over like 50% of the company still is using AI tools on a daily basis in Zapier. And I've been interviewing a lot of like executives, or at least I probably did this last year more than I've done it this year. But when I was meeting a lot of executives on this AI stuff, a lot of them sort of had like a sort of hand, you know, a finger in the air. They're like, I think a lot of people are using AI tools at our company. I'm not totally sure and I actually know like for sure because I can look at the usage of AI features on Zapier and that is actually what I just cited. So it's like 50% of Zapier employees use Zapier AI like on the platform on a daily basis. So they have these automations that AI is like a part of the middle of. And so I can say that with some degree of confidence now. Yeah, I think your question is still valid of like, well, how did you get that? And one of the things that we did, and this was the week between when GPT4 came out and the plugin store came out. There was like a one week window, if you remember, back to like March of 2023 and Wade, myself, Brian, we went to the company and just said, hey, we're going to have a code red moment on AI. Like this is Zapier's future. I think we all saw it at that point, like, you know, Zapier's automation company, but like man AI, like we're an AI company, we just aren't acting like it yet. And we went to the company and said code Red. Now we don't expect everyone sees this yet, but we think this is going to happen. And if we want to have a chance to sort of help our customers and help our users learn how to use these AI tools and build them into automation and get a lot of business value and productivity value from them, we need to be the first adopters. And so we went to, everyone said, code Red, we've got this upcoming hackathon and we're going to change it. We had this like week set for like early April I think, and we said we're not going to do that anymore. Instead everyone in the company, like you get the week off basically from your day job and all we want you to go do is learn how to use AI tools. That's it. And it doesn't need to be a zap, just go learn how to use these tools and figure them out. Because if, you know, in order to benefit our customers, we have to be, we have to have the knowledge of how to use them ourselves. And I can see that. I can actually chart this adoption of usage on Zapier AI and it's literally an S curve where the steepest slope of the S goes right through the week where we have that hackathon. And it really forced a lot of internal figuring it out. I think these are things most companies are adopting tools fed to figure out over the last 18, 24 months of what's our security and privacy policy, how do we allow usage over private data. Right. All the considerations folks went through, we kind of went through in like a very focused, finite, like one week period. And we just got it all figured out because we said this is like must happen now. And that's really, I think accelerated the sort of ability for us to build, you know, useful, hopefully useful AI tools for our users and customers.

Yeah, I love that because most companies like the security department, like super risk averse and it takes like a while for AI tools to get adopted. I mean, I, I think the reality is that like all the employees are.

Just using existential, it's existential opportunity, existential risk. So I think when you' for staring into the abyss, you can get a lot done.

Yeah, yeah, got it, got it. Makes sense.

I think more companies should probably think that way. Actually I think not enough probably do. I think that like the advent of AGI, which we don't have yet. But like when we do like, I think people, most folks still underestimate the degree of progress we will see in purely like digital realms. So things where, you know, the world of bits. And I think people still kind of way overestimate the degree of progress we're going to see in the world of atoms once we have AGI. Because the first version of AGI, like is going to be information constrained and like, what are the systems where you can go create, generate or collect information fast? It's going to be in the world where you can model it digitally. Either it's digital software, you're primarily rate limited by, you know, cpu, clock cycle, network bandwidth, things like that. Whereas in the real world you're sort of rate limited by very different things. Right. You're sort of rate limited by like, you know, can I convince a venture capitalist to invest in my company and you know, buy some equipment from, you know, overseas and ship it here and buy office space? Like, there's just very different sort of rate limiters in play. And I think that's one thing most folks don't realize or don't fully acknowledge that I think once we have weak and strong forms of AGI existing.

All right, Mike, so let's talk about like, let's take a tour, quick tour of the different functions in tech and like how they can use AI to really like save a lot of time.

What's it.

Let's start with like product managers. Like how can product managers use AI?

I'm going to answer, actually I'm going to give you like a high level of like, here's the general form of how Zapier, like across Zapier is using it and then I'm going to ground it to be specific because I think you're going to see here a similar answer across all and it's worth just like getting the general answer. So I think the like most useful thing that we've found internally for language models across Zapier that's like the, you know, majority of folks are using is to parse and handle and work with unstructured text. So wherever your business is getting inbound data where it's highly unstructured is really, I think one of the like newest opportunities to use language models, do analysis, feature extraction, summarization, drafting responses to. These are the kind of use cases that are not possible with Zapier four years ago. Right. These are the types of unique use cases that AI can do. There's lots of Zapier can do that just like AI, like kind of is just an alternative way to do. But this like, handling of unstructured text is one of the really special new use cases. So you asked like product management. One of the cool ones we had here was we when we first launched this product called AI Actions, which is a developer toolkit for allowing you to plug the Zapier like tool library into your own like LLM products. When we first launched this thing, we had some evals that we had built for it and you know, we didn't even know really know how good it was when we launched it. We kind of launched it originally on just vibe checks, getting a beta out there. And it turned out the first version was actually not very good. In fact, I think the accuracy was something like like 30%, 40% accurate. It was like really bad and low and. But we had scoring on it. We had thumbs up, thumbs down with like text boxes attached to every single sort of transaction. We got feedback from users on. You know, was this a good sort of translation of a user request or utterance into a tool or not? And we had a. All that data that like thumbs up, thumbs down was just getting plumbed into like a database table somewhere, you know, maybe a Google Sheet or Django Admin, something like that. And then we built a zap that picked up all of that, each transaction, each record. And we looked over it nightly and we said, let's collect in the last 24 hours of all of our feedback. And then we had an LM summarize all of the issues that we had collected over the last 24 hours into basically like priority and summary and then give the raw links attached as well. And we dumped that into a Slack channel along with the overall eval score that we had because we had this eval running against all the feedback we were collecting. And then we pumped that into a Slack channel that was like, you know, the engineering and product management on this project. And over the course of about 2, 3 months, based on sort of that workflow, just literally just looking at the feedback we were getting on a nightly basis and trying to react to it as best as we could, we were able to get accuracy from somewhere around like it was sub 50% all the way up to like 80, 90% accurate in just a few months. And so we're like, yeah, that's a job that like totally could be done by human. It's just like a really, a large data processing and understanding job that certainly was like much easier and happened. I Think happened with a higher degree of quality because we were able to sort of use this like AI summarization step in the middle there.

Yeah, and I guess the same AI summarization can be applied to like you know, sales and marketing, like you know, transcribing sales, sales calls.

Oh yeah. I gotta tell you about our like go to market stuff here because it's really cool. So we, this is probably the one spot we're like making the most direct dollars from actually is on the go to market side with AI. So we use HubSpot, probably no surprise there, as our CRM. We also use gong, which is a sales call transcription tool. And so when our sales reps, here's the sort of, you know, the process of old was sales reps would wrap up their calls and they would spend the next 10 to 15 minutes reviewing back over either their notes, the transcripts, their memory and trying to do extraction over to these like custom fields that we had set up in HubSpot that we thought were important to know about, you know, assessing how likely is this particular lead to buy. What are their blockers, what information do we need to get them? What are the next steps, you know, all these kinds of sort of meta signals. And every morning our sales managers wake up and they're looking over our HubSpot funnel trying to answer who are the customers that I should go spend time with today? And they're trying to find the folks that are sort of the most likely to purchase that have the biggest, easiest blockers to resolve. You know, you want to book the revenue that you can book generally in sort of a sales funnel. And so this process is sort of dependent on the quality of the data that you actually get into HubSpot. And unfortunately this manual transcription job, like the 10 to 15 minute wrap up documentation from every sales rep to do every call is one of the like least favorite things that sensors do because it is like the most error prone, it's the most grindy. Like you want to talk, be talking to people, right? Like that's the fun part of the job is figuring out how to solve their problems, not like transcribing junk into a database table. And so we've got this whole process now where every time a sales gong call wraps up, we automatically get the transcript coming off of the gong side feed that into some LLM steps where we do feature extraction. Over it we grab out, you know, lead name, email, company propensity to buy blockers, what integrations and apps do they care about? Who else matters on the Team all. All these like signals automatically fill in HubSpot sales reps don't have to spend any time after calls then sort of manually entering that data and our sales reps, our managers then in the morning get a much higher quality signal and like up to date accurate view of the funnel and to sort of go mine over. So you know, you're probably given like sales reps back 15% of their maybe time even just actually spend more time with customers. And then you're also booking additional rep because our sales reps are actually knowing what are the leads that are actually the right people to spend with.

That's crazy. So you probably like save six figures, seven figures just to do at one point.

I calculated it last year. I haven't gone back and updated the number yet, but yeah, it was like hundreds of thousands of dollars of our in additional revenue with this setup.

Yeah, I totally agree man. I think that's like my number one use case. Like just like, you know, just like copy and pasting. Even like long slack threads, internal slack threads. Like I don't want to read all this shit like come visit. Hey, what's the. Why did this person tag me on this like 15 matches of slack thread? Like tell me what's going on. It's really good for synthesizing everything. Yeah.

Coming back to the other point, we all just want computers to do the grinding work for us. They don't have to. Right. So I think that's the problem with this stuff.

Do you have. Do you also use it in your personal life? Like do you AI tools?

Yeah, the main thing that I use like a lot of our AI tools that I use, you know, across like ChatGPT and Claude and some zapier tools built in here is primarily around learning new things. I find that these like large language models are one of are extraordinarily learning and teaching tool, particularly like learning new code base. For example. I was picking up Jax this past summer, which is a sort of flavor of Pytorch where it's kind of a. I don't know how much details you care to know about this, but suffice to say it's a very different way to think about how to do machine learning and training models and it requires a mental reshift and I, and I didn't get it at first. I kind of knew the promise of it. But it's a completely sort of different API surface area and has a lot of quirks and catches that don't exist in sort of the, you know, the underlying Pytorch style libraries. And so I spent a lot of time with tools like with ChatGPT, with Claude to get a handle on the sort of API shape and space, understand the gotchas. Now it can't actually write code for you. I think this is like, you're going to see this in every case. Like, you're not actually going to get like, if all you want is like code that matches a tutorial that's somewhere on the Internet, you'll be able to get that. But as soon as you sort of step one degree of like complication, like away from it, generally you're going to have to, you're going to start being more skeptical about whether it's going to be valid or not. And in all my cases it never was, but it gave me the right sort of like style sense and shape and kind of help me understand. Okay, here's the next important post, important question to ask and kind of get to the frontier of knowledge there. And that's probably the. My personal number one use case for all these AI systems and tools is just accelerating my learning into domains that I am unfamiliar with.

And you just like have a conversation with it or like you paste in the documentation or just kind of chat with it.

For me personally, it's more targeted than that. Like more surgical. Right. Like, I have a very narrow, specific question. Like I'm generally going to read documentation directly from somewhere on the line online, but I will often then once I've like sort of sat down and parse the documentation and I now I have like a project plan in my head of like, okay, well here's what I'm actually trying to do. I will then sort of have a very targeted initial question of like, okay, how do I show me an example of how to do this and that, like initial stop, stop like going from zero to one. I find, you know, it's much easier to like work within an existing code base to when you're learning than it is to write 0 to 1 code. 0 to 1. Writing 0 to 1 code requires a really quite deep fundamental understanding of like what you're doing and the sort of programming toolkit you're working with. And if you don't have that, it's much easier to work within an existing code base. So like, I think you can accelerate your sort of initial like, you know, productivity by, you know, let letting these things at least go 0 to 1. And then even if it's not quite, you know, it's maybe like three quarters of one, it's not all the way there. You can sort of start using it as a starting point to sort of figure out, okay, how do I get from that to one? And, you know, along the way, you're sort of learning something to keep concepts, and you're getting familiar with the syntax. You're getting familiar with the shape of the API libraries. It gives you, sorry, spots for all. It.

Yeah, it makes sense. It's like, you know, it's much easier to edit a draft blog post than to, like, write one from scratch. It's the same kind of principle. Yeah.

Yeah. For what it's worth, I'm very different from my co founder, Brian. Like, he is a very prolific, you know, like, cursor visual studio completion, using language models to, like, do autocomplete and stuff like that. He has a very different workflow than I do in terms of how to link, like, leverage language models for, like, programming and coding and stuff. And I've tried to get do his, and it hasn't, like, worked as well. He's tried to do mine and, like, he hasn't. So I think there's actually some, like, different personal preferences of, like, how people learn and work. But, you know, they find different tools, so it's kind of worth experimenting with different patterns.

I mean, that. That's also like, kind of one of the big benefits of AI. You can give, like, really personalized education based on your preferences. It's not like a classroom where, like, everyone's learning the same. Same thing. Yeah.

Figure out what works for you. Right. You kind of have to have tried enough stuff to know. Okay, this is what I like. But yeah, I definitely agree with that statement.

Yeah. I was at a panel with Kevin and Mike from, like, Anthropic and OpenAI yesterday, and there was a really good quote that they said about how AI is already intelligent, but you got to teach it how to do things. It's already smart. You just got to give it the right instructions so it can actually do stuff for you.

Yeah, we can debate that definition, which we should, but I'll accept it at face value for the purpose of this conversation.

Okay, all right, cool. Let's talk about AI agents. Like, you know, like you said, the promise is like, just get AI to order a pizza for me or get AI to do stuff for me without me having it there. Do you think that's almost here? Or, like, I mean, in some ways, Zapier is kind of like a mini AI agent, but, like, how close are we to AI agents that can actually do stuff without supervision? Like, Do a lot more complicated tasks, I guess.

Yeah, I think I could give like a, let me give like an overview because I think Zapier is kind of like interesting to consider in the sort of agent world or not. Agents are photos with a very overloaded term even in Zapier. Like I think back in 2023 I said don't, we're not. You can't use the word agent anymore. You have to be more specific because like everyone was using it to and had a different sort of context loaded up. So let me talk about from an automation standpoint. I think there's like two kind of broad categories of automation that exist in the world. One is API based automation. This is what Zapier does. There's quite a few competitors at this point to Zapier, but I'll categorize it as like API based automation. And then there's another type of automation that is what's called like robotic process automation is the industry name for rpa. It's basically keyboard and mouse automation. You know, if you want to think we're computer use, if you're going to, you know, that's a more friendly name that Anthropic came up with recently I think for the quad model. But these are very distinct types of agent automation if you want to think about them. And they have tra, they have like pros and cons. The API based automation, this is what Zapier does. It's very permission oriented. Right. You have to go get API keys. So you mean you're sort of working with the API that third party vendors have given you. Right? And said hey, here's how we want robots to work with our service. And so it's like very, you know, upfront there's a permit, there's a partnership agreement, you get API keys. But it's also way more reliable. The API space is much smaller so it's simpler to create automations over. And this is why Zapier, without any AI to date or you know, up until there's 20, 22 or 3, was able to sort of get as many users as we did because like the shape of the API is relatively small. We can build it into some pretty nice UX I think to let non technical users work with it. The downsides of it are that the API is limited, it can't. The APIs you get from say Salesforce or HubSpot or Slack, these are not perfect APIs to everything you can do with those products, they are actually narrow. And this is where RPA gets know Potentially exciting. You know, RPA historically has been generally quite fragile and hard to set up. You know, the sort of main UI path is like an example of like one of the more famous, I think RPA companies in the past and their primary audience they sell to is like legacy IT and engineering organizations who are trying to automate, you know, like Cobalt code from the 90s, you know, that's like running on a mainframe server somewhere that no one wants to touch because no one knows how it works anymore. And easier to like slap a UI automation layer and like literally have a sort of software drive a mouse and keyboard around the interface and like click this like legacy software instead of trying to like re engineer it in a more modern, you know, language or framework or anything like that. And so it tends to be just due to the sort of nature of the technology without any AI, it tends to be very hard to set up because you have to like literally like you know, oh, follow my mouse and keyboard and copy this sequence of patterns. It tends to be very fragile because you imagine automating like a Windows Box or a Mac laptop or something. Well, what happens when you like close the lid? It's like okay, well it sounds working. Or what happens if you get like a system, you know, permission pop up at some point or it says hey, you need to update your software like well if you like. Generally those like RPA based automations are very, very fragile and they're just going to break. And so like it's generally it's like an order of magnitude harder to set RPA up than API automation. It's generally an order of magnitude like harder to keep maintaining, like maintain them compared to API based automation. And so I think that's why generally like Zapier and the API based has sort of won the day so far. Now importantly, I've sort of described the world without AI. I do think that like this RPA based sort of future is a much better one if we can figure out how to get it to work. Because you get sort of the full, you know, you get the full interface, you get the full like the anything that a sort of human could do with a given piece of software, whether it's in a browser or on a desktop or on your phone. Anything where humans could do it is actually part of the API, you know, or the sort of like the interface back basically you could have it operate over there. Whereas you know, with more API based it's very permission oriented and you know, you're never going to get the sort of full version that you can do. So I think like that's kind of the world today. That's absolutely the future. My belief today is that we need weak forms of AGI in order to get the reliability and the setup, ease of use high enough on, let's call it the RPA style of automation in order to get, bring consumer products to market. On that I think you could look at Rabbit, that product that came out last year, the little orange, you know, thing.

Yeah, yeah, yeah.

So like I think that that actually was a, that was a consumer RPA product. Right. They were trying to create an automation layer, a point click automation layer over Android apps that were running on a remote server and that's what they were trying to do. And it's just a really hard problem to do when language models like require a ton of like still domain specific prompting. They're not truly generalizable because they don't truly understand any of the underlying concept they're working with. They're just pattern matching. And so to the degree that you can engineer pattern matching even in somewhat of a narrow domain, it can work, but it requires a lot of effort, but certainly within the realm of possibility now. But the full version I think we all want, you know, can work with any app, learn anything I can give it. Well, frankly like learning is actually not one of sort of language model specialty sections today. So it's not very good at the current AI is actually quite bad at learning new skills.

We talk about AGI now. When you say weak AGI, like what do you mean? Like, like have you tried the computer use thing from Claude?

So I, I think it's probably like definition is actually really quite important here. Yeah, I think the two most commonly accepted definitions of AGI are either one, I know it when I see it. I'm not even going to worry about defining it because it's like undefinable in some way or, you know, it's not worth debating. I sort of disagree with this. Definitions are important because we can turn them into benchmarks and benchmarks we can measure for progress. So I actually think definitions are useful now. One definition that is used right now quite often in sort of a industry is the one that OpenAI and Microsoft like put in their contract from a couple years back where they said AGI is a system that can outperform humans at the majority of economically used work or something to the thrust of this. And I also think this is not a great definition of AGI because I think it is actually one that narrow AI systems can do Today without actually having AGI. In fact, there was an article a couple weeks ago where they were trying to OpenAI. There was reported that OpenAI was trying to like renegotiate that contract by, you know, threatening to claim they already had AGI. And it's like, because it's not act like the thing we have today is like, it's undefinable in that sense. So it's like not a very good definitional contract because it's a, perhaps you can accomplish that goal of like automating a lot of economic use floor without actually having intelligence. I think you can do it through purely this or language model, you know, memorization or near time generalization regime that you get from them. So I think I'll cut to the answer here. I think the right definition of AGI is one that Francois Chollet coined back in 2019 in his paper on measure intelligence. And it is a system that can efficiently learn how to learn new skill and apply it to new output ended problems. So efficiency is a really important definition and this idea of learning new skill is a really important problem. And here's the thought experiment to ground this. So AI versus AGI. We've had AI systems for years now that can outperform humans. OpenAI definition and games of skill like GO or chess or poker. And how do researchers and engineers accomplish this? Well, they have to start over from zero every single time, right? They have to go think of invent new architectures, they have to go invent new search algorithms often they have to go collect new data. They have to sometimes reach new levels of scale in order to accomplish those feats. The important thing is every time they're starting over though on each domain. And this is in direct contrast to you, Peter.

Right?

I mean I could sit you down here and probably teach you a new card game in probably a few hours, maybe a new board game in a few days and get you up to human level proficiency on it. And that, that I think is really the hallmark of what makes you generally intelligent is your ability to rapidly and efficiently learn that new skill and learn any new skill. It's not just, you know, card game or board. It's like any new skill I could sit you down and teach you and you'd be able to sort of pick it up quite quickly and apply it towards new problems that you had never sort of encountered before in your life. And that is what no one knows how to get computers to do today, that is AGI. And so when I say weak forms of AGI, I Think the first forms of AGI we're going to see are likely going to be quite inefficient. You know, it will be on the spectrum of efficiency of like learning new skill and apply it. But I think it's going to be very weak. It may take days to learn new tasks, it may be very slow, may need still like a sort of large degree of, you know, training data relative to what humans do. But I suspect the sort of spectrum of efficiency is going to be the one that like gets better and better over time and is probably the main gradient of how we'll get from weak AGI up to stronger AGI.

Yeah, that's a really good point. Like, yeah, like with the game of Go, like it was just trained on like millions of Go games or something.

Right. You know, there's also search. This was how chess is beaten that I use. Right. It's a large, largely a search algorithm. That was how those games initially got defeated. There was a paper that came out just a few weeks ago that I think maybe from Google or someone who trained just a really large AI model on just pure pre training alone that was able to beat chess. So like, you know, I think this sort of fits within the same regime of how large language models work, which is, you know, the paradigm of deep learning, even more fundamentally than just language models is we improve system accuracy by showing them more situations and more answers as opposed to. That's right, you know, the orthogonal dimension of actually measuring their ability to learn that skill with efficiency and apply it towards these open ended problems.

So you have this ARC prize thing going on. You went to many different schools and I went to the website and the tagline is, the tagline is AGI progress has stalled. New ideas are needed. Can you explain why you think AGI progress has stalled or at least open AI progress?

Yeah. So I think this claim was accurate coming into the summer and I think that we might not be in this regime anymore and I think there's a bunch of reasons why. So we're going to need an update. I think that for ARC Prize 2025. But I do think it was an accurate claim coming into the summer. And the reason is because coming into the summer, basically the sort of, you know, largest, you know, mainline system that we could use was right GPT4O and this is just a large scaled up pre trained language model doing, you know, a pass through the network weights, generate tokens on the outside. This is the same regime. You're basically scaling up the same regime all the way going Back to, you know, GPT2, which came out in 2019 actually around the same time that Francois published the On Measure of Intelligence paper, which is where the benchmark that Archiprise is testing against was also introduced. And so the claim is that we really haven't seen an improvement in generalization power since 2019 at best, probably going back to 2017 with the invention of our discovery of the transformer, but we really haven't seen any generalization power improving. I think the claim would be that, you know, we have, these systems have had the same amount of generalization power due to their fundamental architecture, but we've improved their accuracy on benchmarks by giving them more training data. But the generalization power has been fixed and constant and quite low. And that is going all the way up to the summer where we launched SharkPrize. And a couple of things happened. We launched SharkPrize. There are I think four or five startups that I'm aware of that are all now working on completely orthogonal research directions to deep learning to try and beat ARC Prize. And they're starting to like make pretty impressive progress along these directions. In fact, I, I would make the claim that I think the top approaches on both the private and the public leaderboard for ARC Prize are sort of not just. They're just not pure models, they are models plus some extra search or something interesting, something distinct and different from the deep learning paradigm that's allowing them to get these results they get. And this is quite similar to 1L actually from OpenAI. Right? OpenAI kind of made this big claim when they released it that 01 is not just another large language model. In fact, it's the new paradigm to scaling towards AGI. And so we did, we actually tested it on ARC Prize and it got about 20%, 21% on par with Cloud3.5 Sonnet. You know, this is about half or so the state of the art actually still on arc. So but even despite its kind of like middling score, I do believe that 01 is truly the biggest improvement in generalization power we've seen in a commercial model. Going all the way back to GPT to this idea of being able to do test time search where they're allowing the model to not just generate one chain of thought, but multiple and search over them. Do backtracking allows these models to more sort of fully like kind of navigate the situation space around the prompt that the user was given. Now I think there's a debate of like, well, is it AGI? And I, I would claim no, I Think there's one thing that's really missing from it that conceptually limits it from reaching AGI, at least by this definition. That way I just shared, which is it's still operating over the pre training distribution that it was fundamentally trained on. And you know, the way they did this was they went and generated lots of synthetic change of thought over, you know, formal domains like math and code and programming and things like that. There were slightly informal pre training as well. Things maybe like gold sub goal breakdown that got sort of scored by humans. They use that signal as a reward signal for the pre train. But you're still fundamentally limited by what data got put into the pre train care. And, and I think in order to at least conceptually relax the constraints on your architecture sufficiently, you both need some form of like test time search, like what 01 is doing, like what a lot of the top scores on Enterprise are doing. And you need the ability to incorporate information from test time back into your model and carry it forward. Right. You need the ability to sort of make contact with reality and learn from it instead of having to learn from it sort of every single time. And this is something I think alpha proof from DeepMind actually does. They have a combination of both test time fine tuning plus this like search mechanism and inference time. And so all of this landed like this summer. We got 01, we got Alpha proof. We've got a ton of like frontier approaches now that are completely, that are way more than just language models sort of at the top of the ArcPrize leaderboard. So I think it's actually a really exciting time in sort of the field of like generality research. I think we've, I think arcprize has hopefully played a small role in helping catalyze some of that to happen as well.

That's amazing, man. Like, what are some companies that we should check out as part of Arch Price? You said there's like a couple of companies that are leading this research.

Yeah, two that are on the top of the leaderboard that are worth talking about probably or that I know that I can talk about are the top of the public leaderboards. It's actually this guy, his name is Ryan Greenblatt. It's a really interesting approach where he's using GPT4O. The public leaderboard doesn't have the prize money attached, but let's use commercial models like OpenAI. And he's using GB4.0 to prompts it for like 2000 Python programs for each puzzle. And then he searches over the Python puzzles or programs to like look for ones that match the puzzles and then he applies it. And then the other team on the private leaderboard, the one that's actually in the lead right now for the prize money, is this team called Minds AI. It's actually a super team. This was a team of three people that were individually working on ARC for the last couple of years separately. And after we launched ARC Prize they sort of joined forces into like one team to kind of, you know, pull together all their insights and thinking together. And they've actually made a ton of progress this summer from doing that. And they're doing something quite different to Ryan. They're doing this test time fine tuning thing where they pre train a. I think it's like a code gen model from Salesforce, like a very small model, 200 million parameters. They pre train it on a bunch of ARC puzzles and then inside Kaggle, which is the competition platform we're using, they fine tune their pre trained model with the private puzzles that they get exposed to. And so they're making their system sort of has found a way to sort of know, relax or they find a way to make contact with that reality, incorporate some understanding of reality back into the model, into the distribution for what, for when they're prompting it. And you know, they're doing of course a lot more bells and whistles than just that. But that's like the sort of thrust of, of what they're doing. And I should mention, you know, the goal of the contest is that in order to win any of the prize money you have to open source your code. And so by the end of this year we expect to have, you know, open new open source code from at least five, maybe seven different teams that are working on ARC and hopefully use that to re baseline everyone working on it going into the next year to make it more likely we can figure this thing out.

Yeah, it's really impressive to see like just some random person like Ryan. Ryan beating out Owen Preview on some of these big models. That's great.

Ryan is a fantastic prep engineer. He's probably one of the best people in the world at figuring out how to like get language models to do interesting things.

And these are college students or.

No, they're like Ryan looks at like a research firm, NSF and then Minds AI folks all have like pretty different backgrounds. I think one of them has like a psychology background, you know, like not sort of like at you know, big AI tech lab scaling the next, you know, trillion parameter model or something like that. They're all folks that have pretty. Pretty interesting, unique journeys.

Yeah. So that's a great segue into, like, you know, like, how can really smart kids or these people just want to switch careers, like, get to get ahead of this, like, AGI thing that's coming or, like, you know, actually have a career, like, should I know how to fix toilet toilets? Or like, is it. What are.

Yeah, you know, I think the, like, I mean that sort of tactically, if you're interested in AI, it's easier to answer this question, I think, from someone who's, like, kind of interested in getting into AI in the first. Yeah. You know, one of the things that I spent a lot of time talking with, like, college students over the last year or so with. With our prize. And one of the more disappointing things that I sometimes heard, and this was more true going into the summer was like, man, I like, it feels like AI is already all figured out and I don't really want to go work on it because I don't. There's nothing else I can contribute. We're just, like, scaling this thing up. We're just going to reach AGI and, you know, like, almost sort of this mindset of defeatism. Right. And you know, on one hand, like, I don't kind of blame them because they're getting beaten over the head by this narrative from all these, like, major hyperscalers. Oh, like, now we got it all figured out. Don't compete with us. We're just going to, like, scale our way up. You need data centers with nuclear power to be able to, like, build this stuff. And, like, sure, if that's true, like, yeah, okay, I get it. But I think things like ARC the benchmark, demonstrate there are tasks that are extraordinarily easy and simple and straightforward for humans to do that AI cannot do. And it's not a random happenstance that this ARC benchmark just isn't beaten yet. It's because it was fundamentally designed to subvert the very thing that deep learning is good at, which is this memorization regime. And so, like, sure, while we made progress on our this year, we're up 20% now from the start of the contest in June. We don't have the core ideas yet to get to the grand prize solution. So I think there is still a lot of fertile ground for individual people, students, researchers, early graduates, people that just want to, like, try and make a. Make an impact. Like, we are still, like, idea bound and conceptually constrained to get to AGI. Again, no one in the world knows how to build a computer that can Efficiently learn new skill and solve open end problems that has not been trained on. So if that's an exciting problem for you and you kind of, you want to get into it, I guess I would just say like jump in because yeah, like you know, we're not just on the path ahead is not all figured out yet.

Yeah, that, that's actually like, I'm really glad you're out there telling that story because yeah, like I, I think the general public just thinks like you know, you gotta buy like a shit ton of Nvidia chips and like you gotta spend a lot of money to even compete here.

Like I think like it kind of. Yeah, I think if you zoom out enough on the public consciousness, like progress is progress, right? You know, you just zoom in. Awkward. I might check this thing out once every six months. Maybe I use it, maybe I don't. Hasn't had a sort of material impact on my day to day life but like businesses seem to use it and like that's kind of the vibe check. You know, you sort of get over this stuff and you know, on that side I don't know that the technical details on the architecture, the benchmarks really matter that much. You know, I think if you zoom out on a long enough time horizon, this stuff's going to just feel like it's going to keep getting better and better as long as we actually keep focusing on the right problem to solve though. And I think that's the point that ARK is trying to make is that like most of the industry is oriented in the wrong direction or at least it has been. I think we're finally starting to see some course correction here. But over the last like three years of scaling, four years of scaling a particular paradigm around deep learning, like I mean it was, it's really over rotated. Like in 2023 there was, I don't know, something like 200 billion by my count invested into language model startups and maybe like a hundred million into AGI startups doing new stuff. Like it was like 101 sort of attention, funding disparity ratio and you know, so I think that's kind of the reaction we're trying to have is like hey, we want to, we want ARC Prize to be a sort of North Star for AGI and have it always be pointing in the right research direction towards it to increase the chance that we actually get this thing. And that's my, one of my final beliefs is like we should seek to try and accelerate to discover it's one of the most important ones. I think we can ever sort of get to. And you know, it's just kind of frustrating to see like you know, man. Well we have this like counter example that's really important and most of the world isn't aware of it and like we're all over rotated on an approach that's never going to solve it. And that's the, I think one of its. That's the core mission of ARC Prize.

Do you think like OpenAI, some of these other companies are like doing it doing the same thing internally? Like because they're trying to reach AHI too, right? Like I think they have the same.

Approach or same approach in what sense?

Like they're not just trying to scale the existing LLM, they're just trying to.

Like I know this definitively. I mean you can just go read like no Brown's tweets ahead of reasoning and planning and opening. I like no one's an interesting fellow because he was one of the people who is probably most individually responsible for building those really great AI systems that can outperform humans. I think he started out on poker. He might have done a couple other games. He did diplomacy at Meta when he was there and I was listening to a podcast with him, I think it was last year before he joined OpenAI and I think he had this quote, something I'll paraphrase, which was like I don't think games of skill are any are interesting anymore for AI. It's a solved problem and you know, what we don't know how to do is this other sort of, you know, general form of like reasoning and working over many different domains. And you know, now he's working as the head of reasoning and planning at OpenAI and so like I know for sure that a lot of the labs are trying to like figure this out. I think DeepMind's another pretty impressive lab. I think Alpha Proof is maybe one of the systems in the world today that's closest to like AGI just in terms of its, you know, has ability to do search plus test time, fine tuning still limited to like a formal domain with Lean, the math proving Theorem. But you know, it's kind of funny like you know, Alpha proof and sort of O1 kind of have some different, very different pros and cons. Right. Alpha Proof is like, it's sort of figured out some really important things around generality, but it's very constricted to like a formal domain of reasoning with lean, whereas O1 works over unstructured text but maybe is not quite as good at doing the formal reasoning. And so they're both trying to like, get to each other's spot today. So I do think that there, there's certainly a lot of like frontier research on all these. I don't think. I think we are at the spirit tip of research. We're past, oh, we're just going to go train a model bigger than GPT4.

That's great. So just to close on this, like, when you say just jumping, like, what do you mean by just jumping? Like, like just start contributing to ARC Prize or is there a prerequisite not.

To sort of, you know, talk my own book too much here? But like, yeah, I think ARC Prize is not a bad place to get started if you want to, like, if you are curious, here's the frame I'll give for it. If you are curious what this benchmark looks like that has resisted every single language model that has been invented over the last five years. Remember, this benchmark was created in 2019, before LMS even existed. It was created as a response to deep learning, not lms. And it has resisted every, like, every amount of scale that we've been able to throw at with lms. If you're, that's just not a little bit like curious, you know, if your curiosity is sparked and you're like, man, what is that benchmark? Go to arcprize.org There are some puzzles in the benchmark. It's a very visual benchmark that you can do as a human to get a feel for the data set. Go check it out. You can take some of these things and just like I think you might be surprised at the degree of simplicity and the degree of straightforwardness of, of the actual, you know, tasks that are in the benchmark. And if you're curious to then go further, there's a really good technical guide on the website and get started to kind of start reading about the benchmark and even some open source code, you can get started running it on Kaggle and on your own machine to get started on the contest.

Awesome. Well, I mean, it's good that there's at least I've seen this graph. It's good that at least in this area humans are still way more advanced than any kind of AI system so far. But yeah, hopefully if ARC practices succeeds, it will catch up quick, quickly. Yeah. Yeah, Cool. Well, thanks so much, Mike. Yeah, this is just super awesome conversation.

Yeah, thanks for having me, Peter. It was always exciting to chat about this stuff, Sam.