LLMs are like athletic teenagers talented at many things but needing focused coaching to excel at one. The speaker's marketing agent, Composer, gets real-time consumer data from Klaviyo and a feedback loop that scores each proposed action against predicted engagement and revenue, turning a generic model into a specialized performer.
One of the mental models we have is I often tell folks like, look, you should treat the underlying, especially if you're building agentic products, you should treat the underlying LLM as like a very athletic middle schooler or high schooler, right? They're really good at a lot of sports, but if you wanna make them great at one thing, so for Tom Brady, it's funny, he almost, he got drafted to go play baseball to the now defunct Montreal Expos. So he might've had a great career there. But instead he went and played football and we all know how that ended up. And we're very thankful for that in New England. But the reality is, is like, it took a lot of coaching and tailoring to turn that general purpose athlete into something that was great at a specific domain. And so one of the things that we built into our marketing agent Composer to help differentiate, help it stand out, help it be really good at the sport that it's playing, is we've given it a feed of data. Like real-time data of what's happening between all of the businesses and consumers across the globe that Klaviyo can see. And what's cool is we make this analogy that for Tom is probably, look, for him to be a great football player, he needs great coaches, people that would put him through, show him film of maybe Joe Montana and some of his idols, but also put him through his paces of like, these are the kinds of drills you need to run to be a great quarterback. The same exact thing I think is true for agents. And the way that we go about doing that is we took our Dark Factory software building system and said, hey, I want you to build into this a feed of how consumers are reacting. And so what our agent will do is every time it has a proposal, like, hey, we think you should market to these customers in this way, you should send them this email, it will go check its work kind of against its coach that will say, hey, how good will this— how good do you think this will actually perform? And that kind of a little bit like coaching agents. We'll give it feedback. It'll give it a numerical score of, hey, here's how much engagement or revenue, what kind of outcomes will this produce? And then also give it some feedback on how it can tune and optimize. And we found that like when you combine these 2 things, that's what really makes our, uh, an agent harness really special at a particular task, training that general purpose LLM to be better at the domain-specific sport that it's playing.
