The Ai Strategy That Increased Our Email Conversion Rate By 82%

We have a banger of a show for you today. We are talking about some real life AI marketing experiments, particularly how you generate the world's most personal email with AI that will increase conversion rates over 80%. For your email, I'm joined by Emmy Johnison, who is a VP of marketing here with me at HubSpot, to walk through some of the work we're doing. I'm Kip Bodnar, CMO at HubSpot and this is Marketing against the Grain. Let's get to today's show. Hey, welcome to another episode of Marketing against the Grain. We have a great show today. We are breaking down some real life AI experiments. Emmy, thanks for joining us on Marking against the Grain. Thanks for filling in for Kieran today.

Yeah, thanks for having me, Kemp. Looking forward to talking to you and looking forward to being on Marketing against the Grain too. I'm a big fan.

Oh, that's, that's. Well, we're fans of the work that you do, first of all, and I know everybody's going to be very excited to hear what you and your team have been up to. So what we're going to talk about today is some real world examples of experiments that we're doing with artificial intelligence in our marketing, what we're doing, how they're working. But first, I mean, I wanted to talk about how we got there. How did, how do we even figure out what experiments to run? Because I imagine there are a lot of people out there watching and listening to the show today being like, cool, like, I want to get started. It feels like I could do like 100 different things. How do I actually figure out where to get started? So could you maybe walk me through how you thought about that, how you got us from doing absolutely nothing with AI to where we are today.

When you think about AI and how you can apply it in marketing, really the sky's the limit when it comes to ideas. And we really found that as AI really started coming on the scene fast and furiously, there were no shortage of ideas on the team. But of course, we have a finite number of resources that can really work on these ideas. So we had to come up with a way to prioritize the use cases, the ideas that people were bringing forward, so that we could make sure that we were putting our finite resources toward those use cases that we felt could have the biggest impact. And so we really had to define how do we want to measure impact, how do we want to prioritize these things? And so, as we typically do at HubSpot, we turned to a framework and we used our trusty two by two. And I know that we love a two by two. I know you and Kiran are big fans of two by twos. And so we put together a two by two that really plotted what we felt were two very important criteria to measure these use cases against. On the why we had the impact that a use case might have on top of funnel metrics. And so there we looked at really could this use case meaningfully impact demand generation? So acquisition metrics, or could it meaningfully impact brand awareness metrics? And then on the X we had breadth of use, meaning what size of the marketing team could really leverage this use case and how could it benefit? And the reason we had those two criteria was because we were measuring use cases that could impact revenue against use cases that could really impact internal efficiencies. And that's why we had breadth of use on the X and kind of demand brand awareness impact on the. Yes.

Yeah. And as a kind of a clarifying question, because I'm going to try to play the listener here in this discussion, many times, breath of use, were you thinking about just as internal employees taking advantage of the experiment, or were you also thinking about number of potential customers or humans out there in the world that might get touched or interact with this experience?

For this first two by two we really thought about internal employees. So we were really looking at efficiency gains internally. You know, productivity. Are you going to get exactly productivity? Yep. That's a great way to look at it. Productivity. Yeah. And so after we kind of had that defined, we started running a big intake process across the marketing team. So we opened it up to every single marketer across marketing and allowed people to submit ideas. And then we had a central team kind of collecting these ideas and stack ranking these ideas, looking at these two criteria that we were judging everything by. And then we kind of plotted the top 10 to 15 use cases within that 2x2. And we put together this really agile process where the team that was running the intake and also plotting these use cases against the 2x2 would meet on a regular basis with you and me, Kim, to show us kind of where these plotted against the two by two. And we sort of made the ultimate call around, okay, how do we want to stack rank these based on where they're plotted, where do we want to start? What do we want the order of operations to be? And then based on that final stack ranking, we started applying resources to take on each of the use cases and really started having dedicated effort and resourcing behind each one to start executing against them.

All right, a couple of questions because again, I'm playing list here. You say you opened it up to different ideas. How many ideas did you get and how did you tactically collect those ideas?

We got. I don't know what the final number was, but it was a lot. Maybe 100, I think.

I think it was well over 100.

It was over 100?

Well over 100, yes. Okay.

Yeah, it was a sizable amount. And I think that we collected them just through. We sent out a message over Slack to the marketing team and then I think we collected everything through a Google form, if I'm remembering correctly.

Yeah, so very. You weren't focused on the most perfect process.

That's right.

Is the point I'm getting at is you wanted to move quickly. You did a basic way of collecting ideas and you wanted to move as fast as possible.

Yeah, that's exactly right. And actually, after we did the initial stack ranking, we decided to continue to keep these prioritization meetings every two weeks. And the reason for that was, let's say that something new came up. You know, some change in the macro environment or a new technology came to light. We wanted to have the ability to agilely switch which use cases we were prioritizing so we could take advantage or respond to that macro change in a really fast way.

Okay, next question that I think people are probably going to have. You have over 100 ideas that come in. And I heard a couple minutes ago that you plotted like the top 10 to 15 on your. On your two by two that you walked through for us. Like, how did you get from the 100 plus to the 10 to 15?

We really leaned into that two by two. So we started. When we looked across all the different ideas, we started applying just rough estimates to. All right, if we look at this idea, how much impact do we think it could have on top of funnel metrics? Do we think it could potentially have a high level of impact, a medium level of impact, a low? Again, we were doing this kind of down and dirty because we wanted speed to be the thing that we really leaned into. And then we did the same for breadth of use across the marketing team. Do we believe the majority of the marketing team could take advantage of this medium amount, a small amount? And that's what really led us to be able to sort out which of those use cases were really plotted at the high ends of both axes. And those were the ones that we really focused around.

Okay, so you basically used your two factors, your two axes on the 2x2 as basically like rudimentary like points and grading systems. And you kind of went through and kind of gave a rough grading to each to kind of get that unstructured hundred plus idea list into like a structured hundred plus idea list. It sounds like. Which is, which is awesome. And now the last question before we go and talk about what we picked and what we're doing and everything is who did that? Because people are going to be like, oh, was this a big committee? Did you just do it by yourself? Like, how did you actually figure out who needed to like do that scoring and make that those decisions?

When we started this whole process and really as, as AI started coming onto the scene, we internally got together and realized, okay, if we want to take advantage of AI, which we definitely, definitely do, because we think it could be a technology that has outsized impact on our marketing motions, we need to really centralize, start building out a team that can take this on and centralize it somewhere within the marketing team so that we can start moving very quickly and agilely on it. And what we decided was our marketing technology group made a lot of sense to start building this team out on and centralizing it within. And so we started building out a team within marketing technology. And we had that team, along with the head of marketing technology, our friend Mark, really put together the process, own it and run it for us, which.

I think is the other thing I wanted to really make sure people understood. It's like me, you and Mark, and the team did a great job of prioritizing those use cases. But in parallel to collecting and prioritizing those use cases, you made sure we had the right skills to start actually attacking and doing those use cases. And we brought on Dave G. To kind of on our team to kind of start leading that path. And then we eventually built out a bigger and bigger team. But we got started, we kind of had like one person who is focused and spending time on our top things, Right? Yep, that's right. And whether that person's internal to your company, external to your company. Yeah, kind of the formula here is some priority rating mechanism for you. It was the two by two. You outline some process for collecting the ideas and then in parallel making sure you have access to the right technical resources to actually go and do the work.

Yeah, exactly. And I think you're hitting on some good points here, Kip. Nothing needs to be perfect. Perfect is kind of the enemy of good. Right. Like, part of this is we knew that if we wanted to see meaningful Progress quickly. We needed to move quickly. And actually one of our key learnings around AI is that it is, it is very. You can try to train something as well as you can. Until you actually release it into the wild, you're not going to be able to get it to perfection because an AI model really needs to have that feedback from real users. You need to be able to see how real people are going to be interacting with the thing that you're working on so that you can start training the model and perfecting it to really interact with real people and how they're actually going to be using it. So I think the big learning is really get things out, get it to a certain point, get it out quickly and then iterate and train and perfect. Iterate, train and perfect. Until it's at a place where it's really optimized and acting in a way that you really want it to.

Exactly. You and I both worked with Andy Petrie who runs the product team at HubSpot for a long time and Nike says, just do it. Andy says, just ship it. What you're kind of saying is shipping it is critical in AI more than anything else because you have to get that experience out there, start collecting the data so that you can fine tune and iterate and improve it.

That's right, yep.

And so if you wait too long, you don't really gain anything from waiting. If you're trying to make it perfect, that's impossible without shipping it and getting that feedback.

Impossible. Yep.

That I think that is a critical thing for everybody listening today to understand, know it's not going to be great when you first launch it, set the expectations and know that you're going to get it to great as you iterate on it. Okay, what the heck did we actually do? What did we pick? Like, let's get into it a little bit.

Yeah, there were a few use cases that really shook up at that top quadrant. So high potential impact on demand. And the two that really come to mind are leveraging AI for chat on our website and also leveraging AI in email. And I think we could start talking about the email example because.

Let's do it.

Okay, great.

Everybody wants to talk about AI email. A lot of people out there shitting on AI email. We love AI email at HubSpot. Let's talk about what we actually did.

Yeah, great. We're very excited about this one. Okay, so first, just to give some context around this emails genesis, I think it's important for the audience to understand kind of what this email Is and the context behind it. In a past episode, you and Kiran were talking about our demand engine. And one of the things that you were talking about is these varying levels of intent that we go after so that we can generate more volume. The cream of the crop demand at HubSpot that we are trying to attract is people who are interested in buying software. And we call that here Ql qualified lead. The way that we define that here is somebody is actually raising their hand to talk to sales and they're interested in buying software. Of course, if we were just focused on trying to attract that kind of demand, it would limit us as far as the volume that we can attract, Right? Yeah. Because there's only a finite number of people in the world who at this present moment are looking to buy software. And so what we do to try to grow the amount of volume that we can we can attract is click out a few levels of intent. And so you and Kieran were talking about how we have different motions to really go after those different levels. One being a click out is looking for people who are interested in trying software. And we have our freemium acquisition motion in place to convert that level of intent so people can create a free Account and try HubSpot software. If you click out another orbit, you have people who are interested in learning or educating themselves about a topic that is related to software. This is more educational intent and represents the largest group of people within the funnel. And you can imagine why that is. There's a much bigger swath of people who are interested in educating themselves about something versus trying to, you know, wanting to try software versus wanting to buy software.

It's, well, more than 10x right. To give people kind of a rough rule of thumb. If you're like, hey, what's the ratio between people evaluating versus learning? There's at least 10x more people in your market out there learning than they are evaluating right now.

That's right. Yep, that's right. And so that cohort of people represents kind of the largest cohort within our funnel. Of course, because their intent is further away from wanting to buy software. We have mechanisms in place that are trying to nurture them to the point where they're more interested in trying software or buying software. And that nurturing path we call first conversion nurturing, and it's an email flow that we have here at HubSpot. And everyone who converts on an educational offer, like an ebook, a template, a guide, we call them leads, and they're put into this first conversion nurturing flow.

So you're a stranger to HubSpot. You come in for the very first time because you fill it out a form, downloaded some content, and there's a series of emails we're sending you. Right. The emails you're talking about with us today that are AI generated are from that kind of series of emails, right?

Yes, yes, yes. So that's the kind of the context for this.

Very helpful context.

Yeah. Okay. So for a long time we've had this email in place and we have always used personalization on this email to try to increase conversion rate. And when I say personalization, what we're really doing is segmenting people who are coming into our database, so leads that are coming into database based on similar characteristics and then creating emails that tailor content toward those those characteristics. So it's really personalization based on segmented cohorts.

And those cohorts are largely segmented both by the kind of demographics of those people and the interests of those people.

That's right. That's right.

And that's what they're segmented on. And there are many segments, but they're not like hundreds of segments.

That's right.

You're not talking about these endless number of segments. It's fairly finite, right?

Yes. And I'll give you an example. So if someone converts on a marketing related offer, we might send them in first conversion nurturing a piece of content that is more geared toward a marketing Persona than a sales Persona. And so that's an example of how we're cohorting people and then sending relative related content. So enter AI. And we're all very excited about how AI can help first conversion nurturing, because this is a very high volume nurturing workflow that we have in place because it is serving that giant cohort that I talked about. And we had been iterating and testing on this first conversion nurturing flow forever and ever, and we had seen some success. But you know, Kip, when we talk.

About success, very little success. Like very like incremental gains.

Right, Exactly. I think you've talked about before on this podcast how CRO works. When you first start doing optimization testing on some use case, that's when you see your biggest gains. And then after that it kind of gets to a point where it's diminishing returns. So we had been testing on this for a long time. We've seen some success, but it's been a while since we've seen really meaningful, impactful success. So we're all very excited about what AI can do. And I want to Go into now kind of like where we are with this email. So our hypothesis, by introducing AI and our hypothesis around how we could leverage AI to really improve first conversion nurturing is it would be. So right now we have these segmented groups and we're trying to present content that is relevant to their Persona.

We're doing group guessing, we're putting people in groups and we're trying to guess what they like in those groups, Right?

That's right.

Nothing more fancy than that. I wish it was, but it's not.

Our hypothesis was if we could use AI to actually guess very accurately what that job was and then surface the best content in our library of content to actually help accomplish that job, we think we can meaningfully increase conversion rate and help nurture that person further toward being ready to try software or buy software.

Guess at the individual level versus the group level.

That's right. And then use AI to actually do, instead of cohorted, personalization, one to one personalization at scale.

Exactly. Which is impossible to do without AI because you can't have somebody write thousands of emails a day, which is what you're talking about here.

Yeah, I mean, we could, but we'd have to hire thousands of marketers to do this.

It wouldn't be cost efficient based on the net upside of that work.

Right. So this was our hypothesis and this was the vision that we had. We believe that AI can actually do one to one personalization at scale. And so that's exactly what we set out to prove. What we did was when a person signs up or fills out a form in exchange for an educational offer, they provide us with a few pieces of information. One piece of information that they provide us is their business URL. So the company website. Once we have that information, we're able to scrape that website and get a good understanding for what the company does. The person also provides some other information about their company, for example, what the size of the company is, their email address, first name, et cetera. The other thing we know when a person enters our database is what offer they're actually converting on and other things that they might be doing on our website. And with all of that information at hand, we are able to get a good understanding for what they're actually trying to do. The first step in this process is after a person converts and we see the website, we know what they're converting on, and we're able to see some of the other steps they might be taking. After they convert, the LLM puts together a summary of what we think they are trying to accomplish based on all of those things from that summary, we then our LLM then takes a best guess at, okay, if there was no holds barred, what would be the course that would help this person accomplish that thing that they're looking to do, Whether that course exists in HubSpot or not. It puts that together. We then take that course, that perfect course, and we send it to our vector database, which has a list of all of our courses, as well as the relationships they have to one another, et cetera. And the database looks at that perfect course and says, okay, we don't have that course, but we have these 10 courses that are somewhat similar, that is passed back. And the LLM looks at that and says, all right, based on what I know about this person, the summary I typed up what they're looking to do. I think this course right here is the best course for this person to help them do the job they want to do. And I'm going to plop that in an email along with some text that speaks to how they could actually use this course to accomplish that thing that they're looking to do. Personalize it to them using their name, using their company name, and send that back to them. And that's what.

Which is magical, by the way, to think magic exists in the world. And this is all happening. We are using OpenAI for this. Right. Just to GPT4, because people are going to ask in the comments on YouTube, so want to be clear, GPT4 within OpenAI to do all this work. And I think there's one other thing I would point out, Emmy, before I kick it back to you, is like, we do have an advantage in doing this because we do have a lot. Of course, it's a lot of information, a lot of content we've already created. So we have more things that can be matched by the LLM. Right. Which is so. So the more you are focused on educating, creating value for your customer, the easier working with AI to deliver that value is going to be.

Yes. Right.

Right.

Yeah, yeah. So I, I think to your point. Yes. That that is our advantage. And, and it also is magical. Like, never in the history in marketing have we been able to do something like this.

No. It's incredible.

It's incredible. It's ext. And it proved our hypothesis right. Our conversion rate increased by 82%. Huge.

I love a big conversion rate gain. There are few things better in life than a nice big conversion rate gain.

I know, I know. It was, I mean, it was so big. We took a Double take, triple. Sure that it was actually accurate. But not only did that improve, our open rate improved and our click through rate improved too.

All pretty dramatically.

Very dramatically.

You're not talking like 10% of clicks. No, no, no. You're talking 1, 2, 3x improvement across most of these things. Right.

Open rate, I think was close to 30%, maybe just a hair under. Click through rate over 50%, conversion rate over 80%. Meaningful steps.

Those are all improvements, not absolute numbers, but those are real, real improvements.

Yes, real improvements. Very. And the other thing I would say is, I think when you see improvements like that, especially the engagement kind of improvements, I think what that says to us is this is not only helping with conversion rate. Yes. But it's actually delivering something that is way more valuable to the contact who is actually signing up.

I totally agree. Okay. I think it might be helpful, especially for the people on the YouTube, to walk through, like the output of this magical process that you just talked to us about so people can get kind of a tangible sense for what we're doing.

So here we have this person. They downloaded a content offer and the offer they actually downloaded was about influencer marketing. And then they went on to do a couple of other things within the HubSpot ecosystem. They have a small coffee online coffee company. And so when the LLM sees this happening, we know the offer that this person downloaded, we know what their URL is, their business URL. So we're able to understand that this is a small business online coffee retailer. And then the LLM with that puts together a summary around this Persona and what they may be trying to accomplish. Yeah, this person associated with this coffee company, an organic cold brew coffee online retailer, has demonstrated a consistent interest in enhancing her marketing strategies, initially engaging with influencer marketing resources. So that's the original content offer that she downloaded. She's recently shifted her focus towards content organization and planning. As the winter season unfolds, her interest in content calendar suggests a drive to streamline marketing efforts, possibly in preparation for seasonal promotions or new product launches. Her journey from influencer marketing to content planning reflects a strategic approach to brand growth and audience engagement. And so you can see there we're trying, we're starting to make predictions or assumptions around what this person is trying to do. And then we start to put together what like a perfect course could look like for this person.

So serious question, what you just read came from the LLM.

Yes.

Okay, so just so everybody is clear in kind of what we're breaking down, the LLM took this person's behavior and what they knew about their business and basically inferred what this person was trying to do so that it could then come up with the best next step to help that person solve their problem.

That's right. Okay, exactly.

I want to be super clear with everybody because this is a different way to kind of think about marketing.

Yes. And so from that hypothesis around what this person is trying to do, we're also querying our database to try to understand what courses or content does HubSpot have that could help this person actually accomplish the thing that they're trying to do. And that was the process that I explained earlier. And so what the LLM decides based on the courses that we have that could accomplish this is that our content strategy course could serve this purpose. So this person, we're coming into the winter season. It's clear that they're taking some actions that show that they're interested in content marketing. Perhaps they are preparing. They're an online retailer, so perhaps they're preparing for an upcoming promotion. And so the course that's recommended is the content strategy course.

It's all pretty mind blowing, isn't it?

It's mind blowing.

What we're talking about is wild. And if you're listening, it's really remarkable that you go from guessing about these large groups of people to you have this LLM that can make such a personalized recommendation as to what this person in this business might need that the conversion rate improvements seem rather obvious.

Yeah, right.

Like, you would actually be shocked if it didn't work way better.

I know, right?

You're not like, surprised that it does work so well. You'd be like, wait, based on how much better this is, I would be just shocked and appalled if it didn't work way better.

Yeah, it's amazing. It really is.

I just try to not under like, we've been doing this for a while, so I'm trying to not, like, be desensitized to it. It's rather amazing.

It's amazing. And I love too, how it's returning, how it's returning copy. So it's not just the recommendation recommendations that are very personal. I mean, the recommendations. That is the most amazing part because. And I can talk a little bit about the iterations that we went through to get here because we had a lot of important learnings throughout that. But really the recommendation is the most important thing, or the thing that's really most important is being able to accurately guess what the job is this person is looking to do and then give them A piece of content that will help them do that job. That's the most important thing. That's the key learning here. But the thing that really blows my mind is in addition to doing that, we're generating some copy that just truly speaks to this person and what they do. For example, turn every sip into a story that captivates and converts.

You can't do this without AI. You can't have that level of personalized copywriting at scale without AI. When people are like, well, I don't know, I'm not sure about AI. I think AI creates a bunch of crap. Crap that's not crap. That's really good.

Yeah.

Yeah, that's really good.

It's really good. It's really good. It takes some time to get here. You know, this is. This is where, going back to our earlier conversation, this is where you really need to. You get. You get something trained to a certain point, and then you need to really start testing its interaction in the way.

It took us a couple months, right, Emmy?

Oh, yeah, yeah. This took a couple months to get.

To this point of tuning and iterating. Right?

Yeah. In fact, the first iteration that we had, our hypothesis was around more of not the recommendation itself, but the output. If we personalize the output, the email itself, that would result in step function conversion rate improvements. And what we found was really where the step function conversion rate improvements came is not necessarily in how personalized this text is, but really in are we able to guess accurately what the job to be done is and then surface content to really help that person do that job to be done? That's where the magic is perfectly said. But you can see, I mean, imagine seamlessly crafting narratives that revenate with every coffee lover's morning ritual.

It's good copy. It's good copy. I'm sorry, I know I couldn't have done anything better than that.

Yeah, I know. It's. It's really. It's just amazing. Again, like we said one to one, personalization at scale, this has never been able to be done before.

Yeah. I mean, look, internally, we've always talked about going from personalization to personal, and that feels very personal to that person. Yeah, right. Like, if I got that email, I'd be like, oh, this company, like, actually understands my business, acknowledges what my business is trying to do, and is trying to give me the right next step to take that business forward.

Yeah.

I think if you're watching this, you probably have a bunch of questions. You can drop them in the comments on YouTube. And Emmy and I will answer them. But at the high level, the reality is it took an OpenAI subscription, one person who is spending the majority of their time for a few months building the vector database, training the LLM, making sure we had the right data and annotating that data to make sure the fine tuning works worked. And this is, and this is where we got to. We got to an experience that is amazing for the prospect and conversion rates that are amazing for the marketer.

Yes.

Right. And that's the value Kip.

One other important piece that we didn't talk about but quick shout out to Jordan Douglas and Josh Bliskel, both on my team who are doing the day to day operations around this. But I'd say the other important thing that we've learned is. So Josh is kind of handling the AI piece of this. Jordan works on our email automation team and she really has a very in depth understanding around first conversion, nurturing our Personas. Pairing the two together has also been.

Really important, I think.

Yeah, really important. So you have a marketing someone who is just a subject matter expert in the automation work, the Persona, the email and then you have an AI expert working with them to really bring this vision to life.

I love that. Emmy, I have a proposal for you.

Okay.

Ready for this?

Yes.

I imagine that the folks listening to the show so far have thought it's been pretty good. I'm biased but I thought it was been a great discussion. I think they're going to want to hear more and we've been going for a little while. I think we should do a part two because there were a couple other things that you potentially were to share around some other work that was live. I wonder if we should make this AI experiments part one and we come back and we've got some chat experiments we want to show. We have a little website content recommendation we want to show and walk through with people. What do you, what do you think about making this a part one of two?

Sure. I'd be down. We have, we have no, like we said, no shortage of ideas. No shortage. Well, we have lots of other use cases we could talk to that we didn't get to. Yeah, yeah.

I think we have a lot to talk about. I think this is a topic people really want to dive into. I'm glad we took the time up front to set up the framework to set up like how you actually get started, what is actually entailed in getting started. But I don't think we're done yet. So what we're going to do is we're going to bring Emmy back and in short order, publish a part two to this show to go deeper on some other examples and results. And maybe we'll even be able to read some of the questions from the first episode that we can then apply to the second episode, which would be awesome. So, Emmy, we'll see you real soon on marketing against the Grain. Everybody out there watching and listening today. Thanks for joining Emmy and I, and we'll be back real soon.

Sounds good. Bye, everyone. Thanks, Kip. This data is wrong every freaking time. Have you heard of HubSpot? HubSpot is a CRM platform where everything is fully integrated.

Whoa. I can see the client's whole history. Calls, support tickets, emails.

And here's a task from three days ago I totally missed HubSpot Grow better.