Eric Siu breaks down Andrej Karpathy's AutoResearch, an open-source agent that runs thousands of marketing experiments automatically. It can test cold email, ads, and landing pages at a fraction of what junior marketers cost, forcing teams to rethink how they hire and experiment.
Chapters
Most businesses, when it comes to marketing, maybe they're running 50 experiments a year. This autonomous agent allows you to run 36,000 experiments a year. That's a 720 times increase, and this is completely free. Andrej Karpathy, super AI researcher, used to be at Tesla, used to be at OpenAI. He released this into the wild. It's called Auto Research, and Auto Research is incredible because it helps train up LLMs and it helps them run these experiments every 5 minutes. Let's say while you're sleeping, you can run like 100 of these over the evening, and that is extremely powerful. So I want to talk about some of the practical implications behind this and how it completely changed your business and your marketing. So let's get into it. All right, so let's jump into a little insight here. So Karpathy, he built an AI agent that autonomously self-improves while you sleep and made it free for anyone to use. We're talking about an AI agent that gets smarter overnight and runs itself, only requires one GPU to run. This is basically— this is the very beginning of— think about it, you're setting up your marketing to become an autonomous experimentation engine. It's it's no longer about having employees on the team running these experiments. You can have the AIs running these experiments for you. It could be on the advertising side, it could be on the SEO side, it could be at the email marketing side, whatever it is exactly where there's a lot of button pushing that's required, you can now repurpose that, right? So the way I like to break this down too, if you look at this chart over here, you can see at the very bottom, this is the number of experiments that, you know, this AI can run over evening. And you can see of 83 experiments that were ran, 15 were kept. So 15 on 83 is 18% or so. Now in marketing, maybe your success rate is about 12.5%, just depending on what you're doing. And I like to be a little more conservative. And so if you do the math on this, if we want exact numbers, it would be 4,562 divided by 6.5. So that's 701 times. So that's 701x. Okay? So if you're getting a 700x multiple on anything, ultimately that means your business is going to grow a lot faster. And that's how I want you to be thinking about this. That is what the true payoff is behind something like this. So even though Andrej Karpathy is talking about it from an AI researcher standpoint, you got to be thinking about how you can apply this to your business, how you can apply this to your marketing. Okay, so I actually wrote this piece over here, and I want to bring your attention to the left side over here. So this is the universal growth loop. So this is how you think about marketing in general. So generally, you're going to deploy an experiment, you're going to send it out, then you're going to measure. So let's say you're doing an outreach campaign, you're going to deploy it, Then you're going to measure, and then you're going to decide which parts you want to keep, which parts that you want to discard. And then from there, you're going to continue to modify and you're going to continue to deploy. So you're basically— you have this universal growth loop over here. Now, the challenge that we had before as humans is that it takes a lot of time to run these experiments. But now let's say you have 17 different channels. You can have one engine and this compounds, this knowledge all compounds together. And so whatever you have working for cold email can then feed what you're doing on the social DM side, can then feed your ad creative. And then by the way, your time to signal on all these is going to be different. So if you're sending a cold email, maybe the time to signal is maybe 48 hours or 72 hours or so. Maybe with ads, it's a little faster than that. Maybe if we're doing— we're talking about, you know, community, right? We're talking about building a community that's going to take even longer. So you just got to think about all these ideas that you have, you're going to have to set a different time to signal. Okay. And the whole idea again here is, again, you are building a compounding experimentation engine that helps your business grow faster, it's not necessarily you're building scaffolding around an agent, right? Because I think the agents are going to— that's not going to be the most important thing. It's going to be the knowledge layer that you have that's going to be the most important thing. Now then, when you think about this as the auto researcher of marketing, you now have something that will chase deals for you while you sleep. Before, you know, with your business, you couldn't do that and your team had to go to sleep too. And so when you look at, you know, the AI versus, you know, human manpower before, Maybe a junior marketer might cost you $5,000 a month, and maybe they can cap out. An autonomous AI engine, we're talking about $112 a month, and you may be running, you know, 200+ experiments a month or so, right? Um, and the whole idea here is that you have this compounding engine, um, that, that grows with you. That's how you want to be thinking about this. So let's talk about the strategic implication for businesses here. Businesses in the past, you might have to hire a bunch of different roles. Let's say you're an agency, maybe you might throw a bunch of people at the problem. The way I look at it now, at least for my business, single grain is we are thinking about how do we have AI automation engineers or AI engineers. So you might have a stack over there and then you have super smart marketers that are running these programs. So they're controlling the agents, they're controlling the agentic workflows. They're also just coming up with different experimentation ideas. And through that, again, if you're able to run, what, 4,600— you're not only running, if you're able to have 4,600 experiments that are successful in a year, How much faster is that business going to grow? How much more money are you going to make that business? And how much are you going to get valued more as a business partner? And that partner, by the way, being that successful business partner, that company's going to want to pay you more money in the long term. Practically, you're probably thinking about this right now. Okay, Eric, this all sounds nice. How do I go about setting this up? Here's how to think about this. First, you have to live in the world of X. Okay, so X is where all the action happens. This is where Karpathy shared this. Literally all I do is I go to my Claude instance and I just take the GitHub link that he shares. He's like, hey guys, I shared this over to GitHub. And I'm not technical, by the way, okay? So I just take that link, I go to my Claude instance, I'm just like, hey, how does this relate to the goals that we have? And if you're using Claude Cowork, you can use Claude Cowork because Claude Cowork understands your goals. Maybe you have a sandbox that it plays in, and then basically you say, hey, I want to understand how this applies to my business goals. And most of the time that you share things with your Claude Cowork, it's going to say this is a dumb idea. But in this case, when I shared it, it's like, hey, this is a 92 out of 100 leverage program. So Oftentimes you're looking to kind of just make idea babies with the ideas that are out there, and you're just saying, hey, like, how can I apply this? And again, like, like I said, most experiments are not going to work. But when you find something like this that potentially is going to work, like, I just have this running already. I already have it experimenting on a couple of the agents that I have, and I'm curious to see what's going to happen. I'm going to run it over the next 2 weeks or so, and we'll see what the results are. I mean, but this is really fresh right now, and I do believe that this is the future. I think this is the future of business. If you have something that learns on its own, experiments on its own, Without needing the intervention of a human. And this is just the very beginning of it. It's only going to get a lot stronger. And just to get the idea juices going for you, how does this apply to what you're doing from a marketing standpoint? Okay, so cold email, landing pages, ad creative, YouTube thumbnails, discovery call scripts— these are all things that follow loops. Okay, anything that has a process, anything that's looping, that touches computers, that's how you want to think about it. And by the way, it doesn't even have to just be for marketing. You can apply it over to business as well. Maybe you're running different experiments when it comes to finance, AP, AR, right? There's a lot of different things that you can do here. And So what I want to reinforce again is not necessarily the tool itself. It's amazing, you know, when you have, you know, Karpathy releasing this, but it's more so is now about how can you make AI the best experimentation engine for you to grow your business faster. Real quick, if you want to acquire customers faster and more efficiently this year with the latest strategies and tactics, then check out singlegrain.com. That is my ad agency. Again, www.singlegrain.com. Check it out. And if it seems like a fit, we'll get in touch and help you with a free marketing plan. So let me just say this. If your competitor is running 50 experiments in a year and you're running 10,000 in a year, you're 200 times faster. That is math. They cannot beat that, especially if experiments are learning on themselves over time and everything else is learning like an orchestra. I think that's it for this one. Let me know if you have any questions. Let me know what else you'd like to hear about in the comments below. Drop a comment below. And drop a like as well, and we'll see you in the next video.
