Agents Are Your New Power Users: How Klaviyo CEO Andrew Bialecki Is Remaking a $1.4B Business
Klaviyo CEO Andrew Bialecki is rebuilding the $1.4B marketing platform around AI agents that handle campaign design, audience selection, and performance prediction. The company built an internal "Dark Factory" system that takes prompts and breaks them into production-ready code with human feedback. Customer-facing agents will work headless and API-first, handling storefronts, support, and payments without a UI.
Chapters
How many employees do you have, Andrew?
We're about 2,300 Klavios. Yeah.
Okay.
I know they're all great, all, all the Klavians, but all the product folks and the designers, whether it's through Dark Factory or others, truly can commit code? They've all stepped up to the challenge?
Yeah. So look, we've, so we've, you know how there's these different levels of autonomy of driving, right?
Yeah.
So we've defined, say like, okay, an L, say an L1, you know, this is not unique to an engineer or product person. Like you say, same thing for sales and marketing. Hey, They might be like, I use it to search or whatever, right?
Yeah.
Level 2 might be like, okay, I spin up a little Claude session or whatever, and I kind of run an agent. We tell everybody, it's like, look, by the end of June, you need to be an L3, which is if you're not constantly running either N sessions or a team of agents, right? You're just not, you're not going to be skilled to survive in this next era.
If you guys don't know Andrew by like your— Andrew will describe it, but far and away. One of the most successful players in B2B for e-commerce. Um, number one in the Shopify ecosystem, IPO'd. Boy, you were like the first IPO after the death of the 2021 era, right? Something like that. Were you the first one to go public?
Yeah, maybe like 2 or 3 of us in that like 2023 cohort.
Yeah, almost nobody, right? Incredible rocket ship. I should know the metrics cold today, uh, $1.4 billion in revenue or something like that. And For those of you that don't know, Klaviyo really dominated marketing for e-commerce. And this is probably, if you don't know Klaviyo, I'm going to be overly simplistic, but it is HubSpot and more for e-commerce. It's a brutal oversimplification, but if HubSpot, there's a parallel universe of B2B e-commerce and Klaviyo had 80% market share. And it was when I started to learn about the ecosystem through some investments. And I said this when Andrew was here a couple of years ago. It was the single most beloved app I'd ever heard too. I would talk to e-commerce merchants. They would like folks in the ecosystem. They would like Gorgias and these other apps. And then they'd be like, Klaviyo, I effing love Klaviyo. And I wanna tell you why that's important. That was interesting pre-AI. And Andrew did a couple, a podcast and came here a couple years ago. And the pre-AI aha moment was, and I didn't learn this till Andrew was here before, is one of the reasons, one reason they loved it is it worked. for marketing. But Andrew and his team hadn't had what now is maybe a Captain Obvious insight. Instead of logging into Klaviyo and seeing how many marketing opens you had and how many sends, it was able to connect with your cart and Shopify the other day and say, not only did you send 11,000 emails, this campaign made you $18,372. And if you look on social media or LinkedIn, you'll still see people showing screenshots of Klaviyo, not showing email sends, but how much money that made. So That changed the industry. They destroyed the competition. Everyone loved it. It wasn't that expensive, so everyone used it. Couldn't do much better. Now we're in an AI world. So now Andrew's gotta rethink owning this pre-AI, the magic of email being dollars. And I wanna learn what he's doing because in some ways you probably have to remake the whole company, right?
Totally. Yeah. All right. So if you don't know Klaviyo, The best way, like Jason said, the best way I think about it is like traditional CRM. When I was growing up, around 2010, 2012, when we started Klaviyo, CRM meant like you were really like for sales teams. But we wanted to build the CRM for consumer businesses. So obviously retail, e-commerce, big chunk of that. And, but consumer teams don't have salespeople. They just, they have to deliver every experience automatically. It has to be done through software, not through people. You have way too many customers. So we started Klaviyo because we want to build software that could act as a digital brain and then the actual actions. It's funny to think about now because everybody's like, okay, well, agents, of course I'm gonna use an agent to be my like surrogate. But that has been our thesis all along. How do you build the infrastructure and then the automation to make all that possible?
You're listening to the official SaaStr AI podcast brought to you by— Hey everybody, starting a business can get expensive fast. Website here, email somewhere else, business phone with another provider, Northwest registered Registered Agent gives you a complete business identity in one place with free tools, resources, and built-in privacy from day one. Get more at northwestregisteredagent.com/saastree. That's northwestregisteredagent.com/saastree. S-A-A-S-T-R-F-R-E-E. Hey everyone, if you're listening to the agents, then you need to be at SaaStr AI Annual. We'll be back next May in 2027, the world's largest B2B and AI event, May 11th and 12th in the San Francisco Bay Area. Look, most conferences and content these days are about what's possible. As you know from The Agents, we're all about what's happening now, the latest and greatest, and that's what we do even more so at SaaStray Annual. See the actual work where we bring together 10,000 founders, revenue leaders, and VCs, plus 200 of the very best speakers showing you the playbooks of what's already running in production behind the scenes, deep dive demos, and so much more. Plus over 5,000 scheduled one-on-one meetings so you can actually meet the people, Doing all this, not just to hear from them. If you're a founder, a revenue leader, or you're the one who has to make this all work with agents, this is where you want to be. You already know this from listening to the agents. That's SaaStr AI Annual 2027. Go to saastrannual.com and use our code THEAGENTS for a special discount on your tickets. See you there.
Yeah, what we pulled together for today is 2 things. One is we'll talk a little bit about Click on what we're doing with agents. It's interesting, interesting use cases, but spend most of my time on— I'm an engineer by trade. So we'll talk a little bit about like actually building AI applications at scale and at our scale where it needs to work for a lot of long tail use cases, be rock solid. So that's what we've got. Maybe just to start. So I mentioned I'm an engineer. I'm actually before that. I was a physics undergrad and thought about going and getting a PhD. When people talk about first principles thinking, I'm like, yeah, it's like we just measure everything from the bottom up. So a lot of this LLM stuff, to me at least, is look, at the end of the day, whatever's going on in here is probably, it's just a bunch of neurons, a bunch of actual biological weights. This idea that LLMs work is like not too surprising. So we should expect that they can probably do a lot of the things that We can do. And so we've been thinking, we think a lot about that at Klaviyo. And the biggest one for us is, as Jason talked about, because we provide the infrastructure to know who your customers are and then actually reach them across various digital channels, right? Whether it's marketing, email, text messaging, mobile, your web, customizing a web experience, customer service, et cetera. We realized with AI, there was a big problem our customers had that we could now address. And that for us was this. We have 200,000 customers, but I have so many of them. Every conversation I have basically boils down to one of two things. Either one, tell me what to do, right? Which is what this question is. What, I have these products, I don't know how to market them. Um, a lot of people get a little anxious, a little nervous about marketing. A lot of you have experimented, but maybe they don't have the confidence to know what's right. They don't have the time to do the analysis to prove it. I wanna grow and I wanna just deliver an awesome experience. So customers ask us, hey, how do I use your software? I think one of the cool parts about AI is I think we built up all this software, this infrastructure. There's a big gap between the best users actually using it versus what the software can do. And AI allows us to close that. And the other part was, it's like, okay, well now you've told me what to do. Can you just help me get the darn thing done?
Yeah.
So a lot of folks say, hey, I don't know if I'm creative enough, or I don't know how to do that analysis, or I just literally run outta time in the day. Can you point and click and mash the buttons for me? So that's what we set out to solve. We started on this probably about a year ago. We've gone through dozens of iterations. But one of the things that we have is at Klaviyo is everybody, you know, we have a lot of engineers, but everybody codes. From myself all the way down to some of our like co-ops and interns. For the summers. So when we started out is, this is the prompts, or probably part of the prompts, that in January when the models got really good, one of the things that we have as a pattern is like, hey, we'll often, we meet every Friday to review progress and usually over the weekend or early next week, we'll say, if we've got some ideas, we're not gonna say, hey, let's wait till the next sprint or next month or whatever to prototype 'em. Let's just get it rolling Friday afternoon and see where we are early next week. So one of the things we tried out was we had first built an agent that was using a lot of the state machine type of architecture, but we wanted to move to something that was just more open-ended. A lot of us are building agents now where you just give it a bunch of tools, give it some guidance, and let it rip. That Friday night, I said, all right, let's see how far we can get, and fired up our first prompts. And so I want to show you a couple of the lessons and tricks and techniques that we've been using. To make stuff like this actually work, because it's pretty easy to build a proof of concept, but how you actually get it into production. So a couple of lessons from us, right? One of the first ones is that getting the agent going is pretty easy, but figuring out where it gets stuck is hard. We've all seen some of the examples of, hey, it accidentally dropped my database, et cetera. One of the things we've done inside of Klaviyo is we've architected our agent environment so that it has very tight access to various like tools and controls. In terms of what it's allowed to build on our infrastructure, and then good sandboxing staging environments so that you can build up prototypes, then ship them live so that we can actually look at them, but we don't risk any of our internal infrastructure. So we don't have any of these 9-second failures. The other thing is that while demoing it is good, we talk a lot about, hey, start out, draw me a couple circles, and then just draw the rest of the owl. That's actually quite hard. And this, this gap is something we obsess about, is this prototype to how you get it all the way to completion. How do we make sure that at our scale, when we're dealing with 200,000 customers around the world, multiple languages, very different kind of workloads, businesses that might have 100 customers or 100 million customers, how do we make sure that it scales for all of them and that it's going to be performant? This is one of the things where we've actually written agents that I'll show you in a second that once you go past prototyping will actually validate the actual software and then will run load tests for us so we can figure out where things are going to fail at scale. So one of the things that we've done is we've taken the entire software toolchain, right? Things I'm used to doing by hand, one by one, is once we've got a prototype, whether it's testing, load testing, testing against various edge cases, thinking about, hey, what are the features or the functionality that we're likely to need next? How do we then build an agent that can plug into our agent loop to then go run all of those tasks? And the reason we did this, so as we did this, we found that when we started, a lot of our code, like, I don't know if you're an engineer, We found there's a lot of stuff that looked like on the left, right? You build out these prompts and you have one by one, and you kind of, you add on and you get this, like, if you're, if you've been an engineer, like, you end up with a lot of, like, spaghetti code, a lot of stuff that's, like, just an unmaintainable mess. One of the things we started building last fall is, is a project, an internal agent that we have called Dark Factory. If folks don't know, in manufacturing, there's a concept of lights-out manufacturing, which is basically, hey, if you have all this machinery and there's people around that are using it, then you keep the lights on because they're the ones that are actually like going in there and fixing machines and all this kind of stuff. Increasingly, there's this concept of the more you can automate the factory, you can actually just turn the lights off because that's not what the machines are after, right? They don't need that. So one of the patterns we built is this. What does it take to build a good— to build more complex applications A software dark factory. And so for us, what this loop looks like is that prompt that I showed you up at the top. We run this through our dark factory, which you can access through a standalone repo, through Slack. You give it a prompt, and what it will do is it will take that, it'll write out, like it'll do the PM for you. It'll break it out into a whole bunch of different specifications. It'll then break that down into, it'll decompose the prompt into various engineering subsystems.
Mm-hmm.
It'll write actual contractual API interfaces between those systems. So for our example, for our composer agent, which is this one that solves the, what should I do next for marketing? It breaks down into an agent that can design, do creative, pulling from Canva and Figma and your assets. It can then do the actual orchestration, the building of, hey, what segment should I send this to and when and why? It'll do analysis to figure out, hey, what does it predict will happen? We use our dark factory pattern to basically take something that's that broad and break it down into those component pieces and then have sub-agents run against each one. And one of the critical things I'd recommend you do is having these very clear contracts and interfaces between different parts of the build really makes a big difference. And so we, as we call this pattern now, we have in some cases we're able to run for the whole weekends on projects where we're prototyping something as big as this agent. We say, hey, here's a new architecture we want you to go after, decompose it. And then when it comes time for us to actually human review the code, which we obviously still do, it's actually a lot easier because the system is not this big tangled mess. It's nicely laid out, right? And it's been done by now agents at scale. So that's our pattern for building and how we did Composer. And then with that, that first prototype I mentioned in January, we built this more freeform agent that could do all sorts of stuff, analysis, creative. design. And now that's what we ship to our customers. And that first prototype took a weekend to run and build through our dark factory pattern. And now we've been able to iterate on it on top of it. And so now we use this when we're building, really, it doesn't matter whether we're building agents or some of the core infrastructure of Klaviyo. This is a common pattern we'll use is to give it initial prompts, let it go hunt. And then one of the things we've now built into dark factory is this idea of human-in-the-loop feedback. So we ask the agent to think about, hey, When you get stuck or you're not sure about a decision, we want you to raise it up. So one of the things that will come back over the course of that build on Friday, Saturday, Sunday is it'll say, hey, here's some questions where your requirements weren't quite clear. I need you to specify it harder. So instead of doing this all-in-one kind of plan mode where everything comes up front, we actually break that apart where you get these kind of repetitive little questions over the course of a couple of hours or a couple of days, depending on the size of the build. I think this is the future of how software is going to get built, where very much like when you do a normal product review with actual, like, humans, with people, like, you meet and you discuss the key product decisions and you harden what you're after. We can now do the same thing, but really having agents drive it and then just raise the flag when they get to something that seems uncertain. All right. One other tip, lesson we've learned from building our marketing analysis composer, Run Your Business agent. Is we're Boston-based company, so we're big fans of this guy, Tom Brady, also a San Mateo native. I think one of the things that's happened with LLMs, we're now very firmly in this world of, hey, it's all about building the software that goes around the core LLM, the harness as we speak, as we talk about it. So one of the mental models we have is I often tell folks like, look, you should treat the underlying, especially if you're building agentic products, You should treat the underlying LM as like a very athletic middle schooler or high schooler, right? They're really good at a lot of sports, but if you want to make them great at one thing— so for Tom Brady, it's funny, he almost— he got drafted to go play baseball to the now defunct Montreal Expos, so he might have had a great career there. But instead he went and played football, and we all know how that ended up, and we're very thankful for that in New England. But the reality is, is like it took a lot of coaching and tailoring to turn that general purpose athlete into something that was great at a specific domain. And so one of the things that we built into our marketing agent Composer to help differentiate, help it stand out, help it be really good at the sport that it's playing, is we've given it a feed of data, like real-time data of what's happening between all of the businesses and consumers Across the globe that Klaviyo can see. And what's cool is we make this analogy that for Tom is probably, look, for him to be a great football player, he needs great coaches, people that would put him through, show him film of maybe Joe Montana and some of his idols, but also put him through his paces of like, these are the kinds of drills you need to run to be a great quarterback. The same exact thing I think is true for agents. And the way that we go about doing that is we took our Dark Factory software building system and said, hey, I want you to build into this a feed of how consumers are reacting. And so what our agent will do is every time it has a proposal, like, hey, we think you should market to these customers in this way, you should send them this email, it will go check its work kind of against its coach that will say, hey, how good will this— how good do you think this will actually perform? And that kind of a little bit like coaching agents will give it feedback. It'll give it a numerical score of, hey, here's how much engagement or revenue What kind of outcomes will this produce? And then also give it some feedback on how it can tune and optimize. And we found that like when you combine these 2 things, that's what really makes our, uh, an agent harness really special at a particular task, training that general purpose LLM to be better at the domain-specific sport that it's playing. And finally, I got 2 more things for, one, we are very big believers you're building agent products. So one, you should use agents and teams of agents to build agents. 2, Use that LLM harness as like a general purpose good athlete. 3 is this new category on top. What we refer to inside of Klaviyo is this like autonomy layer where all the agents live, where intelligence is like, it is the unit. What we tend to think of as software, I now think of as it's infrastructure. It's just a means to an end. So if you build some amazing UI, that's great. Like there might be, there are a lot of humans that will still use that. Some AIs, but the reality is, is like headless is now the default. And so in Klaviyo's case, we look at ourselves as like, we're the database for all of your customer consumer data. We know everything about your consumers. We then provide the infrastructure on top of that to market and message to them. The opportunity that agents provide is then to be that actual like intelligence on top. So all those customers of ours that come and say, hey, I know Klaviyo is a great product, but I actually, as good as it is, I don't know how to use it optimally. We're now populating that intelligence layer with our composer agents so that it can interact with our customers. It can actually teach them or even do their work. And we think that's gonna be the future of frankly, like how marketing and just consumer experiences in general are delivered. It's gonna be much less like each of us deciding every single decision and more, hey, having really good agents that will bring to us A feed of, hey, here's the campaigns you should run next. Here's the experiments I ran last week. These ones worked. I'm gonna keep doing those. These ones I'm gonna throw away. All of that's gonna get automated and happen at much larger scale. Last thing, a mental model that we have when you're building agents and infrastructure for them. So we think about Klaviyo and like all the stuff that we've built around data marketing, customer service. One of the things we found from our users that are leveraging our Composer agent is normally in software, you have this power law of like how good your users are at your products. So you got a couple of really advanced users, maybe you put them on a customer panel together, you bring them in, they're really the ones pushing the limits. But then you have a lot of users that are novices, and that's no different for Klaviyo, right? We have a lot of, we have a, we work with a lot of businesses where people have maybe an hour or 2 a week. To use our software and that's it. They just don't have the time and the bandwidth to get super advanced, no matter how good our software is. So you can imagine there's kind of this distribution, right? We have very few really great users and then a lot of users that are kind of novices. Agents change that entirely. With agents, they go straight to the most advanced users, and we actually think sit to the right of them. All the agents from day one. Or power users. And so it's a different way of thinking about building the underlying infrastructure of, of what is your software. So I'll give you 2 practical things that we think a lot about. One is this idea of needing to onboard users. We just don't worry about that as much because agents will onboard automatically if there's good enough documentation, they'll reason through it. And then obviously we'll give them some hinting of what we do to Composer of here's some guidelines of general marketing best practices.
Yeah.
But onboarding actually matters less because these agents will go 0 to 60 much faster. The second thing is because they'll push the limits so much, we've actually started asking the agents we've built, what functionality is holding you back? And I'll give you a very concrete example here, right? In the world of email, there was probably about 10 years ago, maybe 7, 8, 9, 10 years ago, there was this big push to make email more interactive. It's something that I've cared about for a long time. Why is it that if you wanna, I don't know, if you wanna buy something that you see in a message, you can't just buy it straight from that, from your email client, right? Now we're limited by what email clients provide, et cetera. So there's a big push to make email more interactive. Google has a big push, AMP, all this kind of stuff. Turns out, well, Google released this AMP specification for email. And it's out there and you can use it. A lot of people didn't know how to build for it because now you're effectively building a web app inside of this email payload. So for a long time, we're like, man, this is tough. And we'd give people basically just a raw HTML editor, but a lot, very few people could figure it out. Almost on day one, our Composer agent figured out, they're like, hey, if I want better conversion, I can make this email more interactive. I can include some carousels of product imagery. I can make it more dynamic. I can include some JavaScript calls, all this kind of, you know, callbacks to some end servers. I could do all of that. But hey, Klaviyo, we have a problem. I'm missing some APIs that would allow me to edit and specify my content according to the AMP spec. Can you fix that for me? So we've started to ask our agent, hey, what are tasks that either it's hard to do, that we just don't have enough flexibility? On like that. So we've now built that out, or that an agent wants to do very repetitively. They don't wanna do it once or 10 times. They wanna do it thousands of times. Another good example for us is we've dramatically leveled up the underlying experimentation infrastructure in Klaviyo because now it, agents are like, well, if I have an idea, it's so cheap to test. I'll just throw in a test and give it to 1% of my audience and just see what happens. Versus a human user would say, okay, I would love to do that. Yes, that makes sense, but it's just very hard to do.
Yeah.
So last thing, as you're building agents and you're using these dark factory type patterns to build, I think you should expect that for your software, right? Whether you're building agents, they're going to be the super users, or if you're building the software that you expect either humans or you're going to plug into MCP or other people are going to expose to algorithms and agents, expect them to be super users almost right away. And I would say that whatever you're doing to get customer feedback, I do the same thing with agents. I like that. Thanks for having me.
Sure. You want, you have time for a few questions? How many employees do you have, Andrew?
We're about 2,300 Klavios.
Yeah.
Okay.
I know they're all great, all, all the Klavians, but all the product folks and the designers, whether it's through Dark Factory or others, truly can commit code? They've all stepped up to the challenge?
Yeah. So look, we've, so we've, uh, you know how there's these different levels of autonomy of driving, right? Yeah. So we've defined, say like, okay, An L1, you know, this is not unique to an engineer or product person. Like you say, same thing for sales and marketing. Hey, they might be like, I use it to search or whatever, right?
Yeah.
Level 2 might be like, okay, I spin up a little Claude session or whatever, and I kind of run an agent. We tell everybody, it's like, look, by the end of June, you need to be an L3, which is if you're not constantly running either end sessions or a team of agents, right? You're just not, you're not going to be skilled to survive in this next era.
Your average PM at Klaviyo pre-AI who's working on wireframes and specs now has to become an L3 in, in your hierarchy?
Oh yeah, absolutely. Yes. And then there's—
I believe you. I'm just learning.
Yeah. There's no, there's, and I, what we, I wanna tell you, there's been very little pushback on this. It's a, look, it's a little intimidating cuz it's okay. We're, the slope is steep.
Yep.
But what we've said is, look, the, the reality is, and yeah, one of the things I think about that I'll share. There's so few people that are gonna get there in the next year or two. If those skills are gonna be so valuable, if you truly are not talking about AI, right?
Yes.
But if you're the kind of person that's, I can put an agent to work and actually have it produce value. If I can put a team of agents to work that could decompose a problem, right? And do each of the pieces and I can find a way to check those and validate those, you're gonna be able to take on bigger projects. If you can do that, Klaviyo or otherwise, you are gonna be enormously successful in the, the next 3 years.
Without, without question. Right. Do you think Maybe just 2 questions, 'cause I know you talk so much about Dark Factory. For the non-developers, are they getting close to programming in English, to coding in English? Do they still need to understand code or do you think they can almost code in English?
Yeah. So there's a little course we teach internally about system decomposition. By the way, this one I'll give you just a quick analogy. I was talking with our people team about this. I should, we showed them this, this on the software pattern we did, and this, this idea of contracts between software systems.
Yeah.
Hey, what does the front end do? What does the backend do? They took that and said, oh my gosh, we have to think through what the jobs of the future are. And they said, I said, like, why don't you use the same program to tear that apart and figure out what does like a marketer or a salesperson or engineer actually do in real contractual terms, right? I know this feels sucky. I think, look, a lot of humans, we're used to these kind of fuzzy interfaces and then we'll meet, what, meetings to kind of figure out. What he's first gonna do. I think that it's not a great way to work, honestly. And it's, it doesn't work with agents because agents aren't that fuzzy. They want, they want hard and fast rules. So what we've done is we've actually took the same pattern and said, hey, go think through this team, this function, this job, map out the actual work as in almost like software interfaces. And then tell me about how would you scale that team for 2x, 5x, 10x growth, right? Or scale or volume, what they're doing without Any, uh, without adding any, say, more people, but more agents, right? And just ask it to reason through that. That has actually worked really well.
I don't know that level of depth of the whole organization committing is that common today. I think it will be, right? It can also lead to slop because if every PM can now build features, even if they're not committing to production code, if they can build features and they can build a basic feature in an hour or 2, not one prompt, instead of maybe getting a feature out a quarter, you could build 100 features a week if you wanted. And it rises, even inadvertently arises the level of slop. How does the organization process 100 features built by a non-developer? If you have 50 people on your PM team, that could be 2,000 features a month that are half-baked work and are intellectually interesting.
So taste still really matters a lot. And I'll give you one thing we do. Okay. Every, every Thursday, part Thursday, like I'll do this after, after we wrap up here. And then on Friday, I spend all day with our product team just reviewing work. And so one of the common patterns that we found is for folks that are new, they're like, okay, gosh, what kinds of feedback are you gonna have? And so one of the things we used to write this down, but then sometimes you'll read it or, oh man, we didn't update this. The, the kind of the guidelines change or whatever. I took all the crits, all the feedback that we'd given over the last couple years, right? That these like, whether Zoom meetings or raw notes, all this stuff, we just dumped it in a database. Right. Gave it to an agent and said, look, from here on out, before we even get to those reviews, here's what's good. Here's what's worked. Here's what won't. And so that has helped us, like when people have ideas, cuz we, there's an explosion of ideas. That's great. They're not all worthy of shipping. What we've said is, hey, this needs to pass this bar. And by the way, if you think there's something wrong with our rule set or feedback, push back on it, right? We found that actually that cuts down on a lot of stuff really quickly. In terms of like, ah, yeah, it's kind of a good idea, but it doesn't meet certain conditions. And then obviously now we have a team that's like, are actually pushing back on our first principles of like, hey, what makes a great product? I think in the future, look, we're gonna be able to give agents the idea. I think, well, taste matters. I think we can codify taste too, of our product should work this way. It should feel this way. These are the outcomes we're after. The trick for us is that decomposition. It's like, can you have it explain its work? And then for now we have to train it up on, okay, how, where, where there's logical, Failures.
And does, does, and then I want to just talk briefly and then we can take a few questions before we run out of time. I want to talk about your customer agent next, but since you've built Dark Factory in-house, right? Maybe just 2 questions. How do you encompass that taste organizational-wide? Do you have an organizational-wide memory? Do you have guidelines, Klaviyo guidelines for product development? How do you do that? And why do you, are you going to maintain it in-house? Is it worth, in a year, is it worth maintaining the art factory in-house versus finding a replacement? Because the rate of paint changes so fast and you have to maintain it.
So look, this, this idea, but I think obviously our taste, yes, now lives in this database, which, you know, instead of it being institutional knowledge where it's like, you know it or I know it, that's silly. And I also say it's like, look, well, it's, well, I have a lot of input into it. It's a living, breathing thing. We'll work on it together. So that, that now having AI interface on that is a big Is a big plus up because now more people can access it. It's not like, oh, I wasn't there for that, or I missed that, or whatever. Now, in terms of the software pattern, look, our belief is the gains are going to go to companies that can stay on the bleeding edge. And the bleeding edge is hard because you have to invent a lot of stuff, figure out the patterns. I tell you, for us, I think this, this team of agents pattern obviously exists. It's going to get more codified. We're going to keep investing into, in that. Until we see some other open source or other product that we think can really do this well. But as I go back to this, like the idea of this LLM as like kind of this very smart or very good athlete, I think a lot of these patterns that we're building around it are just leveraging things that the labs are building into the models, but you still have to build around it. And that, that part they may build, but also like it could just as easily be somebody outside of the labs.
Yeah.
Yeah.
All right. Let me, let me just switch gears for folks. So that, that was, I, that was super cool. I didn't even know we'd talk about internal. On your customer-facing agent, just how much— so you said you don't really need to train the customer agent, right? So let me just learn from that. When a lot of the folks that have agentic products here in GTM, if you don't have an FDE or an SE, it sucks. Okay. Point 1. Point 2, when I watch what Zendesk has said, which is adjacent, right? Basically what Zendesk has said, and it's an older platform, this, their self-service agent for support. It can get about 20% of the way there, but to do what Zendesk needs to get to 80+, you need FTEs and humans. So how does that tie to the fact that for your SMB customer base where you can't even afford FTEs and SDEs anyway?
Yes.
How do you actually make the agent great?
Yeah.
Okay.
So for Klaviyo, we talked a little bit. The prompt I showed you in that product is, we think of it as our inside agent. It helps you run your business. Our Composer agent helps you do marketing, understand your customers. Then Jay's talking about, we also have this outside agent. It's an agent that's a digital representative of your business to the world. I think these are going to be the future of like how consumers and businesses are going to interact. I think in the future we're going to have these customer agents that actually act as like a proto web server where it's like you just query into it or you visit a URL, you provide some context, and it will just render output HTML based on who you are, what you want to see, right? So we're very much far down this path. So I think this idea of these customer agents as Purely for customer service is way too narrow. Now, one of our beliefs here though is like, look, I remember growing up in the late '90s and, uh, GeoCities and then building my first websites in the early 2000s. This is all like doing, and I remember seeing WordPress and be like, wow, this is magic. 'Cause I don't have to, I don't have to, I don't have to code HTML anymore. And then obviously see the great website builders we have today. The reality is agents are going to come into our lives, like for every business, much faster than I think we're gonna be able to build up a population of people that know how to build them. Yeah. Because agents are just, they're more technical. We're talking about things like tools and API access and all this stuff. And as Jason said, we have some very, very talented small businesses, very technical founders, owners of those businesses. Even for them, they're like, look, I don't have time. I'm not gonna be bleeding edge on this stuff. I, our team stays up nights and weekends just to stay on top of all the latest things that you can do with the models. By the end of this year, I think most businesses are gonna have their own agent they deploy either on their website or behind a phone number or through email or to ChatGPT or Claude that you can just access. But to get all those businesses, right, take our 200,000 apps, get every single one of them using, building that agent. There's no way we make that happen this year if we have to teach all of our users how to build agents. And obviously it's, it's cost prohibitive for them to hire somebody to do it. So we are just down the path of agents can train the agent. And I think the trick there is we're just obsessive about when you hire the, you hire an FDE, I like, we look at the, what is the person actually doing? Right? What is the actual literal process? And so I'll give you an example of what we do. We take, we can take a feed, right? And we can synthetically generate this if we have to, of all the different kinds of cases and experiences or conversations, right? That somebody might have with an agent that represents a business. We can take a real feed, we can generate a synthetic one. We then classify that into various use cases. We take a couple representative examples. We then give an agent that has full access to our platform, 'cause it's all API-based, right? And we say, great, I want you to train up this agent. Here's, here's an eval, here's a use case. Don't stop, little Ralph loop. Don't stop until this comes back and it works.
Yeah.
And similar to our Composer agent, what it will do is it will interrupt when it says, hey, I've hit a logical point where I just don't have the instructions on how to do this. So in e-commerce, hey, somebody wants to process a refund, let's say, right? Different companies have different rules around that. If something's below or above a certain dollar threshold, they might say, you know what, we actually don't want you to return it. It's too expensive. Other cases they'll say, okay, we do want you to return it. There's also all sorts of little rules, right? This agent will figure out, they're like, look, I can build this. But I often find that businesses like you, and this is where we get value from scale, have rules like this. Can you tell me what yours are? Or it'll say, hey, there's, I know there's a backend system that say, let's take the case of returns. We'll print that shipping label and send you, say, a box or something to return something in. But I need to know where that system is, right? So we'll ask and request from the user to basically go use those. And we have a lot of integrations, so it's a little bit point and click, right? In the future, we see a world where it's like, I don't know, maybe you could give it access to just go hunt on your behalf. We're already seeing great success with this where we can take a standard business and find 5, 10 use cases, have the agent train itself up, prove that it works. And then what we deliver to customers is we say, okay, we built you this agent. We took all of your use cases and hey, it's already at a 50, 60, 70% resolution rate.
Wow.
That changes the game. I think this, a lot of these agent products, this idea of You go through a demo and then you decide you're going to implement it. There's a lot of deciding. You don't get to trial it really. It's all going away. I think for a lot of agents, if you can't try that product out of the gates and it requires some implementation, you're dead on arrival. And so for both of our agents, both Composer and our custom agent product, we think that's— no, the wow experience day one should be you get to use it, see it working, and then you can either immediately turn it on and then obviously you can optimize and tailor it to your own needs.
Maybe one last follow-up question, just to get your— in an agentic world, whether it's— if your internal APIs or your external APIs are state-of-the-art, it can make up for a lot of dated software.
Yeah.
We use— my epiphany was when we went headless on Salesforce. I had not logged into Salesforce since 2 companies ago. I didn't know, it turned out, Amelia told me I didn't even have a seat. Now that we run it headless, I can query Salesforce in real time all the time. There's other internal use. So I actually think, and for some folks, if your software was almost hopelessly dated a year ago, just focus on great APIs, internal and external, and you may be able to make your product. And to me, Twilio is another example. It almost died 18 months ago. It was growing 4%. Now it's growing over 20% because the API worked with agents. So I don't know, you can make old software good if your internal APIs are great, because that's all the agents care about, right?
It's the first thing that I think, yes, first advice I give anybody is if you're working on older or new codebase, Yeah. Amazon had this concept, or, or, and others in the early 2000s of service-oriented architecture. We've been through this many times. That's why I think if you previously, if you were building software, anything that a human would log into and use, I think the right mental model is think about it as infrastructure. And we think about infrastructure as things developers use, and therefore they have APIs. If you think about it that way, then yes, you'll naturally lead to, okay, what are the interfaces, the actions, the APIs I expose? And actually things get a lot easier from there. And you get the benefit of now, You don't necessarily need a user to level up. You'll get these PhD-level agents that will just figure it out and prefer your stuff because it's accessible and it's really good at what it does.
Yeah, I think it's almost like even if you slip far behind, if you make your internal and external APIs, you might get parity again. I think it's the simplest thing you can improve in your product internally, externally, is have state-of-the-art agent-friendly APIs because you just don't know. All of a sudden you can have your product guys come up with magic that they couldn't do a year ago if your internal APIs are great, right? Yes.
One last thing we've done for infrastructure is we've also gone down this path of how do we let somebody sign up? for our software with their identity, right? Email, phone number, whatever it is, and use our product without ever touching it. So I think the future of, say for us, right?
Headless player.
CRM is you'll, you might be in a, with an agent or just say, hey, I'm starting a business this weekend and I need a place to store everything. I wanna spin up an email, text message list, just go get it done. We have a team, it's really a team of an engineer whose only mission is to make sure That is an awesome experience, right? And we expect in the future there may be something where you never even, you know, of Playvio, you got a bill or whatever, right? You pay agentically, but you never have to touch our interface and it's a great experience.
They have this Playvio, right?
I'm all in.
All right, everyone, big hand up for Andrew for staying on. Appreciate it.
You're listening to the official SaaStr AI podcast brought to you by— costs can add up quickly when you start a business, so stop paying for 5 different services. Northwest Registered Agent gives you a complete business identity in one place with free tools and resources to actually launch your business for real. Get more at northwestregisteredagent.com/saastrefree. That's northwestregisteredagent.com/saastrefree.
