Jason M. Lemkin discusses SaaStr's shift from building a general AI agent called 'digital Jason' to specialized agents for events, sales, and customer support. The team discovered that horizontal agents hallucinate and fail unpredictably, but improved dramatically after feeding them specific content like the prospectus.
And so in the run up to that, famously we both were like, okay, this is pretty cool. And people are starting to use it more often. They like it for advice. Again, classic like. But now it's like, how do I hire a VP of sales? They have the conversation with the AI and the AI can have follow up conversations. So now we'll see people use it as like a copilot of oh, hey, I'm thinking about hiring this person for my VP of sales. What do you think of their resume? Or I asked them this in an interview and they asked me for this comp. Should I push back on that? Like, what would, what would real Jason do? And so that piece was cool and it's really good at that. And it still is to your point. But where I kind of saw some failings that we both saw. I was like, it's not as good at answering our actual like Saster needs of. Okay, we have this event coming up, right.
So that's a specific use case of basically trying to clone Jason JSON. But the interesting part to me for sure, it's a horizontal. You know, there's talked or Dreamforce say about horizontal versus vertical agents. Right. Delphi is as horizontal as it gets. There's almost no workflow. There's maybe a couple little triggers. It can't do things qualified can, but it can do a little bit of everything. And so it was supposed to just be digital, Jason. And what was interesting is, and this is an issue with a lot of agents, people would start using it for support. Okay, hey, I got a booth at Saster annual. Can I have popcorn? And we didn't expect that. And this is an issue so many will have with AI agencies. If it's good, if it's bad, you'll have no roi. If it's good, people will start using it. I'm sure you hear it all the time. People will use it for things you do not envision. If they trust it, you got it. So instantly people wanted it for issues with attending our events and for sponsors. And we're like, oh, well then it started to hallucinate. Okay, it was pretty good on my content. And so we had a magical moment which is we uploaded our prospectus.
Yes.
And it only took a certain amount of context and training. But then it started to get good. So then it was pretty good at doing maybe 20% of what qualified good. But here's the thing, and the people get this wrong. 20% is a lot better than zero. Like if we never used qualified and only use this General agent, we wouldn't have the 7 appointment because it can't set meetings. Yeah, but even 20% is better than zero if it's good. Sure. Just the fact that instead of it being ghost, like we used to use intercom and it'd be a weak response time. Okay. So we switch to our agent and it could instantly tell you about 90%. Right. Your, your, your, your customer question. And so sometimes 20% is enough. That's as far as we got. And then when we had time after annual, we're like, okay, let's do better. Yeah, let's go from horizontal to vertical as a phase two. And I don't think that's a bad approach for some folks. Like get one thing working that's horizontal, gets confident and then go, okay, now let's do sdr. BDR marketing dedicated this, dedicated that. Right. So we went from horizontal to, you know, 12 verticalized agents in a sense.
But we knew in that process it set us up for, for where we are now, like successfully with multiple agents. Cause I think otherwise, if it had failed or if we expected too much out of it, I don't know, we probably would have stayed on the one or something or dumped it. Right. So I think that's where we're like, okay, post annual and we have more time. I started to look for vendors like a qualified, an artisan, a Delphi that we could bring on that would make sense for us. Right. So Saster has big scale, but internally we're very small. So it's like, okay, the number one thing that was driving me crazy was two fold. Like two things were driving me crazy. The first being that when you came to the website to become a sponsor or buy a ticket is very antiquated. Like fill out this contact form, which I hated. I was like, not only is this the worst of filling out a contact form, it's very dated. And also half the time I round robin it to our sales team, they don't follow up.
Right. So that at least half the time.
At least half the time. So like I'm like half the time I know if I'm giving it to like one of the better reps, it's going to get followed up with instantly. But I give it to the other rep because it's their day on round robin. So I'm like, they're either getting a response half the time or not at all.
So have you blown all that up and you just do it all with agents now?
Yes.
And who manages the agents right now?
I do.
