Andrew from Klaviyo explains how AI agents work as superusers of software, running thousands of A/B tests on small audience slices that would exhaust humans. Klaviyo built an autonomy framework with three levels, requiring all 2,300 employees to reach level 3 proficiency by June.
that an agent wants to do very repetitively. They don't wanna do it once or 10 times, they wanna do it thousands of times. Another good example for us is we've dramatically leveled up the underlying experimentation infrastructure in Klaviyo because now agents are like, well, if I have an idea, it's so cheap to test, I'll just throw in a test and give it to 1% of my audience and just see what happens. Versus a human user would say, okay, I would love to do that. Yes, that makes sense, but it's just very hard to do. So So last thing, as you're building agents and you're using these dark factory type patterns to build, I think you should expect that for your software, right? Whether you're building agents, they're going to be the superusers, or if you're building the software that you expect either humans or you're going to plug into MCP or other people are going to expose to algorithms and agents, expect them to be superusers almost right away. And I would say that whatever you're doing to get customer feedback, I do the same thing with agents. I'm with that. Thanks, Fred.
Sure. You want, you have time for a few questions? How many employees do you have, Andrew? We're about 2,300 Klavians.
Yeah. Okay.
I know they're all great, all, all the Klavians, but all the product folks and the designers, whether it's through Dark Factory, others truly can commit code. They've all stepped up to the challenge.
Yeah. So look, we've, so we've, you know how there's these different levels of autonomy of driving, right? Yeah. So we've defined, you can say like, okay, An L1, you know, this is not unique to an engineer or product person. Like you say, same thing for sales and marketing. Hey, they might be like, I use it to search or whatever, right? Yeah. Level 2 might be like, okay, I spin up a little Claude session or whatever, and I kind of run an agent. We tell everybody, it's like, look, by the end of June, you need to be an L3, which is if you're not constantly running either end sessions or a team of agents, right? You're just not, you're not going to be skilled to survive in this next era.
