B2B Ad Campaigns: Real Test Results from 7 Channels with Pranav Piyush

Why do people get into trouble? Because here's what happens. And they take that 10,000amonth spent and they spread it. They spray and pray across five different channels at the same time. When you do that, you cannot get clear signal if you don't have sophisticated analytics. So in our case, we don't need any sophistication. We're doing one at a time. We're proving out that exercise sponsorship works for us. And so now that's baked into our, our strategy. And the next investment would be coming in the second half of this year where we're starting to invest much more in events and dinners. And we're going to look at, are we able to increase our traffic or leads or whatever our metric is, by another 50%. We're taking big swings. We're not trying to go for 1% lift, 2% lift. We're taking, hey, how do we double, triple our traffic? Because guess what, we're a startup, right? We can't live with 1 or 2% increases. That doesn't work when you're on the VC train. You just have to go for big swings. Foreign.

Hello, everybody. Everybody's here, so. All right, can you send us some Ben and Jerry's then? Hi. Hi, everybody. So I was a minute late because we're getting all this new studio gear for, for my, for my house here, for my podcast studio. And I had to run up and get signed for the packages and everything. My favorite Ben and Jerry's flight flavor. There really isn't one like all of them fish food, I like the mint. The mint cookies and cream is amazing, so. And Danielle's high school job was scooping ice cream. I love that. All right, so, hey, welcome. This is a 8 to 5 live session. Once a month or so we do these to bring in an expert in B2B marketing to come in and go deep on a specific topic. Our guest today, who I'm going to bring up in a minute, is Pranav. He's the founder and CEO of Paramark, and he has an awesome point of view on how to think through marketing strategy, how to measure things, which is a big part of doing this job. And pitched us on a really cool topic for today, which is digging into some actual paid campaigns. And we're going to dive into what worked and what didn't with facts only, no opinions. But before we get into that, just let's go in the chat real quick. A bunch of people are already doing it, like right in, coming from Dave from Vermont and then also with, with where you're coming from. Like, I want to know why did you take an hour out of your day to come to this thing live? You certainly could have just got the recording. But what is it that made you want to come here? Come here live today? Danielle said she'd get fired if she wasn't here. Not a me. Not. Not initially, but I mean, like, if it became a thing. Absolutely. Carrie's in Raleigh, Jen from London. Carl's in Newton, Mass. Shout out to Newton, Mass. Beautiful Victoria, Vancouver Island. Look, man, we got some sun out here in Vermont today. We're doing great. We got a fig tree, I got a salt lamp. It's all good. So, all right, we're here. We're going to go for an hour. You'll get the slides, you'll get all that at the end. We got Q and A and everything. Yeah. Why don't you bring in the great. The great Pranav. Who. Big day for the crew over at Powermark. Funding announcement dropped today. I have a funny story for you. So when I was first time head of marketing, I was working at a company called Drift and we raised like $60 million Series B or something, which today is like a. Today is, you know, pence.

No, it's still, it's.

I mean, no, it's changing. Sorry. Sorry, that was, that was a bad. That was a bad joke. I. I think what happened is like, I think the 2021 funding environment, like re. Changed everything and now we're, we're back to like, you know, I think your. Was this. Did you know, was this your c. Would you announce your seed?

This is our seed. Yeah.

Seed. That seems more right than like, you know, two or three years ago.

I mean, you, you ask any of the, the real OGs, right, they will say no, no, this is an A, right. So there's definitely some inflation in, in our round names and round sizes, for sure.

Okay. So anyway, so I'm symmetric and we had this big funding announcement and it was obviously the most traffic, the most inbound, the most ever thing to the, to the site. And then, you know, for, for three years after that, basically that, that thing became the bane of my existence because it was the anomaly in all of the charts. Like every board meeting, every presentation, we'd always. Wait, can you go back for a sec? What was that spike? That spike?

September.

September 2017. What was that spike? And I'd be like, is when we raised six. And look, look, I said, I said that. I was like, look, if you guys want to raise $60 million every quarter. Like, I'm happy to get you that.

The ROI is perfect, Dave.

The ROI is perfect. Yeah. So anyway, and then this is just a, just a riffing on here on a side note. And then I just had a crazy conversation with my UPS driver and I need to know if this is true. So he was saying that UPS is the best and safest. They have the best. They have the best hiring process. He said they don't hire. They don't hire felons. He said FedEx and Amazon, they hire felons and they drive around and deliver packages. And then he said, Amazon drivers. He said, Amazon's driving around and it's legitimately spyware. They have cameras and everything in all of their cars. And so they're, they're delivering your packages but filming your house so they know what ads to serve up to you.

Oh, wow. I was wondering where the story was going and I was like, okay, this is a weird tangent, but, I mean, that's believable to me, right? Like, why wouldn't you collect that data? You're not, I guess you're not breaking any rules. I actually don't know if you're breaking any rules.

Yeah, well, anyway, there's, there's, there's my. There's my. There's my ad story for the day. So. All right, I'm going to kick it over to you. I know your audience, right?

Absolutely. Yeah.

Let's see if we can get you to focus on marketing for about an hour. I know it's a lot. It's very hard to not check your email. My dopamine would be depleted right now if I were you. But so we're going to talk about. You guys have pulled a bunch of campaigns from, from Paramar customers, which is cool. And we're going to dive into, like, some truth here, which. One of the hardest parts about marketing is, you know, you see people write these articles about breaking down the strategy behind X Brand, and like, you actually don't really know how the ad ever actually performed. Did it, did it do well or not? So we get to criticize these things on social media, but we don't actually know. And I think this is an awesome idea for the session. So I'm going to let you drive and I'll be here. I'll bring in the chat, we'll take questions at the end. And let's go.

Awesome. So. Well, first, hey, everyone, I'm Pranav. Dave. Thanks for having us. This is exciting. We get to talk about all the learnings that we are sort of receiving in the last whatever number of months with all of our customers. But I want to start with something just kind of a disclaimer. We are very sensitive around customer privacy. And so I'm not going to share any names. I'm not going to give you like, hey, this is the customer, this is their ad, this is what they did. But I'm going to talk about generally sort of what's happening and you'll see some statistics and how sort of they measured it. So just kind of a very clear disclaimer there. If you have questions that you want to go specific into certain strategies, you can always reach out to me. I'm happy to have a one on one conversation around specific things or even put you in touch with those customers who ran those tests in case that's helpful. But just want to kind of say that now do you want to put up the slide deck and we can kind of start to. Yeah, there we go. All right, so beautiful imagery here. Good job. Exit 5 team. I just copy pasted that. Okay, I am going to start with. We always have this debate. We use the word attribution all the time. And I went to Dictionary.com this morning when I was creating the slides for this and just took a screenshot of what does it mean when people say attributed? And there's actually two different definitions and I think this is an issue for a lot of people. One of the definitions is the attribute itself, right? The quality or characteristic of a thing, a person, what have you. But that's second on the list. The first definition is this idea of quality is something, a cause of something else. And so when people use the word attribution, when people use attribution measurement. Are you trying to answer the first question or are you trying to answer the second question? Are you just looking for some type of a relationship or are you looking for causality? If you're looking for causality, well, guess what? There's no better way than experimentation. I just went into ChatGPT and I asked the question this morning, what's the best to find causality in marketing? Guess what it says. Randomized controlled trials. A, B testing. This is not new. This is the basis of the entire modern society, right? All of science is basically through experimentation. If you don't do experiments, there is no science. And so I start with this topic because if you are really serious about attribution and measurement and you're not thinking about cause and effect relationships, if you're not thinking about experimentation, what are we even doing?

Here.

So I'm going to get off of my pedestal here. Talk about some examples. But I did want to start with this, right? It's like you don't have to be perfect, but like, know what you are doing. And the other thing that really annoys me, we all say in marketing sometimes, let's test it, let's put it out there, let's test it. Guess what? If you don't have a control, there is no test. You're just throwing spaghetti at the wall and calling it a test.

90% of the things that I say, test it, test it, try it. Just try it.

Sorry, Dave. And it's okay, right? Like, I'm not against just doing it. So my, my problem isn't with the fact that we all want to do things because yes, you can just do things. There's nothing wrong with it. I do things all the time on, on social media and emails. Like, we don't have to test everything, but the words have meanings, right? And if we say that we are running an experiment or running a test, that means something. And I'm just asking for all of us to level up our sort of seriousness when we use the worst test and experimentation. That doesn't mean you have to experiment everything. You don't.

I think it's a good foundation because I think it is. I think it is like, you know, Victoria in the chat said, when people run an A B test and change the headline, body and subject all at once, it's like part of this, part of the ability to measure what's working in marketing is the ability to take a step back and slow down and pick one thing to go and test so we can learn. There could be multiple factors. Can you school me really quick though, on I wonder if a lot of things that I say often are correlation versus causation. We could ask ChatGPT if you don't have a pithy line on this, but just could you explain the difference between those two?

Yeah, it's a good question. So correlation is essentially saying, are these two trend lines moving together? So my impressions on LinkedIn go up, my demos go up. When you map both of those charts together, they seem to move in tandem. That's correlation. Causation is saying that because I move my LinkedIn impressions up, my demos go up. That because is very important because it's very possible. It's not really cause and effect. It just happens to be the case that you're doing all these things, people are going to your LinkedIn and people are going to your website. And booking a demo. Oh, guess what? You launched a podcast ad that you're not counting in this correlation. And that's the real reason why people are going and checking you out on LinkedIn or going to your website. So that cause and effect relationship is what you're truly after if you want to go from correlation to causation.

Nice. That was a perfect example because it is a trap that we fall into when we see the correlation between those things. Awesome.

Okay, now one other thing before I go into examples and I do want to get there as fast as possible. Chat. Repeating these terms, right? So Conversion Lift, if you haven't heard that, oh my God, that is a problem. Meta offers this as a native experimentation tool. LinkedIn is launching it. It's in beta right now. You can call your LinkedIn reps and launch a conversion lift ab test today if you wanted to, but it's going to be GA in like three or four months. So go check out Conversion Lift again. Run. Go to ChatGPT and just type geotesting. How does it work? Time is testing. How does it work? Holdouts. How do they work? These are just like keywords that I'm dropping to you, but you just self educate yourself. I don't have to spend a ton of time giving you sort of basic education now that we all have an amazing AI assistant at our fingertips all the time. Okay, here's the basic.

That was crazy to me. Like, I know this is so obvious, but just to hear you say chatgpt, it talks about the world that we're in. Which even a year ago saying that would have been like, whoa, this guy's suggesting this cutting edge thing to find this information. This is the world we're in now. Yeah, I love it. All right.

You know, we were having this conversation. My co founder and I were figuring out if we can sponsor somebody's visa for an interview process. This is like live from last night. And I'm like, he sent me a question like, do you think this is possible? I copy it paste in the chat GPT. ChatGPT is, yes, it's possible. Here are the three sources. Send it back to my co founder. We're like, all right, great. So if you have questions about incrementality measurement, ChatGPT is your best friend, really. Okay, so I'm going to talk about a bunch of incrementality tests, geo tests, holdout tests. These are all names for the same thing. And these are real examples from real customers over the last whatever number of months that we've been Doing these, how do they work? Very high level. We pick a few geographies where we test a specific channel or campaign and that means that that particular campaign only ran in those GEOs. And then we look at how the aggregate metric in those GEOs performed compared to the GEOs where there was no ad campaign. And then we look at, was there lift, was there no lift? What was the cost of any incremental lift that we saw? Okay, first example, I want to start with the easy one. This is like taking candy from a kid. Branded search. If you haven't run this test for your business, what are we doing? Like go hop off this webinar right now and run that test. Two examples. There's a serious C company, SaaS company. There's a series E company, SaaS company. So these are well funded, you know, venture backed businesses. They're spending millions of dollars on marketing. They're spending hundreds and thousands of dollars on branded search ads both on Google and on Bing. So what do we do? We said, hey, we're going to take a few geographies and we use our platform to determine what those geographies are. Where we are going to cut your branded search spend down to zero. So you know, California, Texas, Massachusetts, whatever the algorithm and the machine learning tells us like here are the states, here's how long you need to do it. Usually that's like four to six weeks and you don't have to cut your spend down to zero. But in this case we were like, let's just cut it down to zero because we had some other data that you know, pushed us in that direction. Now I want to see maybe in the chat how many people think that there was no impact or there was, you know, a ton of impact on the business. Like just put it in the chat. I don't think I can see the chat.

I can. Oh yeah, yeah, click on the right. Oh, maybe not. I'll, I'll.

Oh, I can see it. No, no, I see it, I see it.

Music, music, Some theme music. Kelvin says no impact. Megan. Minimal impact or low impact? No impact. No impact.

Yeah, all the people who are saying no impact, spot on. And I'm not talking about tens of thousands of dollars here, you guys. Like the collected savings here are in the millions on an analyzed basis. Millions. So let that sink in. Right. Like, and I'm not suggesting that, oh, branded search is horrible. You should never do branded search. All I'm showing is real examples of two customers where branded search really did not impact their business. And if you haven't tested this in this fashion. The, oh, we're talking about the metrics here, Luke, are MQLs, these are Pipe, these are hand raisers, demos. These are like real business metrics. And we're saying that there's all of that stuff got shifted to organic. This is literally the customer said it even before we shared the results with them. Like, hey, we looked at our own internal metrics. Everything shifted to organic. I'm wondering what powermark's results are going to show. It's like, yep, didn't do anything. Right? So this happens.

Megan says, so does this mean they were still seeing the same click and impressions in that Geo without paying for branded search?

Correct. It all just shifted from the branded search, the sponsored ad, to your organic SEO.

So literally the end user, the consumer behavior there is look for the term skip over the what would have been in the ad anyway, find the organic listing and click that.

Exactly. And here's the thing that happens, right? Talk to these customers. Even when you see these results, it's still scary because what happens is these brands are thinking, well, what if my competitor starts bidding on my terms and start taking attention from us? Well, guess what? In those states where we turned it off, right, we're now running a test for a longer period of time to see exactly that. What is the competitive observation there? But even if there are competitors who are bidding on your terms, think of it this way. The person who is putting it into search and saying, I'm looking for Paramark, and then just because of a competitor ad, they're like, oh no, I'm not going to do a demo with Paramark. I'm going to go to their competitor. That's a little bit odd. Like, our customers generally are quite savvy. They're talking to two or three people. They understand all the competitors in the ecosystem. So just the presence of a competitive ad is not the thing that's going to swing the deal 100% towards your competitor.

There's a bunch of. This is clearly. It's so funny because everyone in chat was like, obviously, you know, they, there was no impact. But then there's a billion questions on this. So let's just run. And this is like, I don't want to take up all the time on this one topic, but this is such a hot topic, like this branded search or not thing. So maybe let's just see if you have opinions and answers to some of these. Judy said, were these customers in competitive spaces and were these customers in a competitive space?

You know, yeah, they're all competitive.

Jeanette says, are these well known brands?

Yes.

Brands though.

Yes, very well known brands.

So does that have it, do you think? They're asking because that has some impact on the results here.

Well, does it, does it not? Like look at the numbers. Right? Like my, again, my point is the answer is going to be different for each brand. So don't take that at face value and say branded search is dead. That's not what I'm saying. All I'm saying is like, if you don't test it, how will you know? And guess what? These same customers had high impact, high efficiency channels where they were spending a pittance. A pittance. And so taking this budget from branded search where there's no incrementality and switching it to channels where there is incrementality and doing the opposite of this, which we'll talk about in a couple of examples, right, where you can increase your spend on YouTube on. It's going to sound crazy. PMax on Twitter, on out of home, on billboard boards. And those are showing positive lift. So we'll talk about those examples. But this is not an isolated conversation, right? You have to think about the whole mix of where are you allocating between different channels?

And then there's, there's three comments in a row. Was there SEO already strong? I feel like you need to know, you need, you need to be strong for that to happen. This is dependent on good SEO. This assumes an organic strategy and the site is rather, is optimized rather than playing for traffic. Any reaction?

100%. Yep. SEO is good for both of them and they invest in SEO and you can measure SEO in their models to see that, you know, obviously SEO is doing very well. So there, if you haven't done any SEO, then yes, this could be for sure. Cool.

Thank you. All right.

All right. Now this one is really interesting. This is billboards. Fun little backstory. The CMO is like, hey, we're heavily reliant on Google Search. Most of it is like non branded search, but it's like we're heavy, heavily reliant on, on, on Google Search. And they inherited a new team and a new business. This, this CMO is like, yeah, our leadership does not buy into all of these brand things that we do because they've been burnt by it before. We did all these things three years ago. The previous leader did it but it showed no impact and so we just had to move on from that. What's really happening here is that there was no good measurement in place. So when that Previous leader did all of that brand stuff. It just didn't show up because it doesn't generate any clicks. So we set up a geotest where they bought a bunch of billboards but in a very specific part of their target market. And guess what? I'm showing you the data, right? So you're looking at and it's anonymized so hopefully you can't see who it is. But you look at the conversion metric and you look at this observed value which is essentially their success metric, how that performed over the course of when the test was running. And the control, the control is basically saying what would have happened if you did not run this billboard campaign. Now you have clear causal evidence of the lift generated through billboard. And guess what? None of this would show up in any type of qlik attribution tool and not even their post purchase survey tool. Right? Because that's just people don't put it in. People are lazy. So this is an example and this is a company that's saturated their Google search. They have nowhere more to go in Google search. They're looking for additional growth vectors. And this is a prime example of how you can do an AB test in the real world with billboards. And guess what? These were not just the old school billboards. This was digital out of home. If you haven't heard of what digital out of home is, you really do need to go learn. There are a bunch of companies that do this ad quick onescreen. There's a bunch of others that'll let you buy digital real estate, right? These are digital sort of bus stands or even sort of digital sort of displays on buildings, what have you. So they actually went down that route, not just the physical billboards.

A couple follow ups on this. Which metric do people want to know if this is clicks as the measured metric here?

This is actually so I can talk about. It's a fintech company and they do some type of a fintech product. And so you have to apply to get qualified for that product. And so we're talking about the applications that customers are going and filling all the way through. You can think of it as like credit cards or loans or. Right, those types of things. But this is B2B. So it's B2B FinTech and it's a high consideration process and purchase like we're not talking about you know, a sort of a transactional sort of thing that's happening.

Cool.

All right, bunch of questions. I want to take this one. Is this correlation versus causality? This is causality. Right. Because we are literally saying that in the test locations we ran billboards and everywhere else in the control we did not have billboards. And we're looking at the aggregate lift in your applications between your test and control. So that's why this is causality and not just a simple correlation. There are so many questions. They do have a large tan. So this is like focused on sort of SMBs, long tail SMBs. So yes, that is something that is relevant here for a business. I'll show you some examples of how Paramark does our own testing, which will be fun. So we'll talk about that. Okay, here's another one that might be a little bit more accessible. So we're now talking about YouTube advertising for a couple of different companies. One is doing hundreds of millions of dollars in revenue. And then there is a series C company. I think this is the same series C company that we talked about earlier. And the first one is very enterprise ish. Right. You have to use this company's product like as a developer. You have to integrate it into your product. There's a high consideration cycle, all that fun stuff. And so they look at MQLS as their measure of success. This might be spicy, but it's fine because for them MQLS is literally a hand raiser. This is somebody coming to their website filling out a form to say, I want to get in touch with your team to do X, Y and Z. So two very different examples. And this is why I say that, you know, any channel being dead or not is not a thing. In the first example there was very clear lift for YouTube for that particular business. We're looking at the pre test period and then we're looking at the test period and we're seeing a clear lift in their measure of success compared to the control in the second case. It's identical. And again, in that second case we're talking about a collective spend of close to a million dollars a year in YouTube advertising. And zero impact, zero lift. Now why is that the case? That particular brand had massive, for whatever reason, YouTube SEO. So they had organic YouTube content that was doing really, really well. And so their advertising was really not breaking through beyond sort of what was already happening on organic. For this other brand, they had no organic content. YouTube did really well from an advertising strategy. So yeah. Questions on this?

Yeah, I have a question for you. This is. There's a bunch of comments that kind of echoing the same thing and I think we can help people. Maybe we just like kind of think about what the first principle Here. So people are saying like, how do you. Okay, these are great, but these often require huge budgets to run. There's some folks that are on here that are. Have smaller, smaller businesses, smaller budgets. Any fun ideas for what experiments? So I think no matter what, that's always going to be the question is when you see someone spend money, you're going to say, okay, well what's the version of that? And so I don't know if we have a particular example, but Pranav, let's help with just a first principles type of thing. Thinking if I was a smaller brand and we wanted to maybe test into doing something like this, like how. How would you think about that?

Yeah, great. I love this. And I do see Savannah's question there. Right. Many of the experiments you're talking about require huge budgets to run. What, what are medium sized businesses going to do now?

So wait, just, just to build on that though. Like, like for me also as a first time marketing leader back in the day, I think one of my wrong beliefs though was like, oh, we don't have the budget for this. This is scrappy. We're going to be scrappy. And I actually end up screwing up later because I never did enough testing into the bigger budget. And so like, I do think it into like, I was like. And I saw a comment earlier about like, well, yeah, we can't do the test. We're not spending 120 grand a month on this channel. We can't do the testing. Well, I went from no budget to a lot of budget and I didn't have any middle steps. And so I think maybe we can talk about like how to test and learn so that you can feel good about going to spend more money.

I love that question. It's fantastic. I put a post out on LinkedIn a few days ago where I had this like hypothetical conversation with a hypothetical customer. And it happens sometimes in reality as well where I'm like, okay, let's do a holdout where we cut. They're like, no, no, we can't risk that much. You know, we can't cut spend. Like, what do we mean? Like, how can we not advertise to our audience? We need to be going after growth. I'm like, okay, that's fine. Let's increase spend in a specific geography. Oh, no, no, no. We can't go get additional budget. Like, that's not going to happen. I'm like, okay, so you don't want to cut spend, you don't want to increase spend. What are we doing here? Right. So first thing, if you want to get growth, if you want to run experiments, you have to take risk. There is no alternative, zero alternative. If you wanted everything to be done, deal, A plus, guaranteed, proven, before you even did anything, we would all be sipping pina coladas at the beach. Why are we here? Right? Like, that's not how marketing works. That's not how reality works. Now, let's take a specific example. For a small brand. When Paramark starts advertising, guess what? We're not going to run a national campaign. We're going to build an account list. We know the thousand companies that we want to go after, and we're going to take our advertising and split that account list into half and say for half of that account list, that's going to go into my LinkedIn audience, and the remaining half is not going to go into my LinkedIn audience. That's a very simple test. It's not done by GEOs, it's done by account lists. I could structure it differently. I would say, hey, we're based in California, we're based in sf. Our entire team is here. We're only going to advertise in sf. Or let's flip it. We're only going to advertise in New York to see if we can lift our inbound demos from New York. There's nothing stopping you from doing that. We could run that test for $500. You could run that test for $1,000. You can run that test for $100,000. So this idea of you need big budgets to do this type of testing is plain.

Savannah said to be fair. We're not talking about not being willing to take risks. We're talking about businesses who have 10k to spend on ads, which eliminates most of the meta experiments that you mentioned. Okay, maybe in that example. Yes.

And yeah, 10K is a great example. Right?

Yeah. I was going to say there's actually. I bet you there's a bunch of people on this webinar who would be pumped to have 10k a month to spend.

That's right. And I would just not spread that out nationally. I would do it either by geography or some cut of your audience where you can see your end metrics move along with that audience. And, you know, so that audience where you can see your end metrics move along with that audience. And you know, Savannah, happy to do some live experimentation, sort of brainstorm with you as well. Like, I do this all the time with people who are also not customers. So we should, we should do that. Okay, here's Babura CTV. So this is Connected TV. This is a series of SaaS company. They, you know, again, we're reaching a point of saturation. They're like, hey, we want to try some connected TV ads and you know, see what that does. And same exact thing. The budget was fairly small. It was only about a 30 or $40,000 budget. And that means that the noise is higher, right? Because when you're spending lesser across a fewer set of geographies, you're not going to get perfect precision. But in this case we did see very clear lift. And so for them, this is the first time that they're going into sort of more of a video, audio type of format. Most of their prior work had been very direct response, bottom of funnel. But they're reaching a saturation point. They're seeing a little bit of a decline in their business. They're trying to find new avenues of growth. So paid social, paid search, those are not your only avenues. Podcasts, ctv, direct mail, we've seen brands get success with those types of channels. And your budgets don't always have to be in the millions. Even with a 20, $30,000 budget, you can run experiments. Okay, here's a campaign that didn't work. So this is Sequoia backed company. They launched a multi channel test across YouTube, across Out of Home. So Billboard, CTV, Meta, all @ the same time concentrated on a specific set of geographies. And here's why it didn't work. We saw clear lift. Don't get me wrong, it worked, but it was very expensive. The cost per incremental conversion that they were tracking was almost 10x what it was the case for the rest of their media investments. And so yes, you have Lyft, but it's not efficient. And so what do we do now? We're going back and testing individual components within their media plan rather than bunching it all up together and doing smaller tests. And this goes back to that point of like you don't have to run a massive test, you can run smaller tests to get signal before you ramp up into that bigger budget that you have. Okay, I'm going to skip pmax because I want to do PMAX is performance max. I think somebody asked that question. This is an ad format on both Google and on Bing now, so you should look into it. Many people sort of swear off of PMAX and say it's all trash inventory and it doesn't work and all that fun stuff. But in this particular case we did see a very clear signal that worked and it's become part of the strategy for that specific brand. All right, I'm gonna switch because I think I'm sensing people are like, hey, you're telling us about these big ideas, these big brand campaigns. We don't have the budget for that. We're trying to just, you know, execute with 10k, 50k. I'll give you our example, Paramark. We are a 10 person company. We just announced our seed fundraise which has been long overdue. But we make some real money, we generate revenue, we have real customers, as you can tell, and we have a little bit of investment in marketing. So what did we do? We actually for the first time did a pretty big sponsorship and that was with Exit 5. So this is very meta conversation here and I'm showing you our real stats. How do I run this experiment? Very, very simple. Our consideration cycle is anywhere between three months to 12 months, which means somebody becomes aware of us, somebody does a demo with us. It might take three months to close that customer into closed one, or it might take 12 months to close that customer into close one. We don't know when we are engaging with them. So when we think about success, I look at the aggregate traffic to our website, this is going to sound blasphemous to half the audience or maybe 90% of the audience here. Now, you know, I'm fortunate that I'm a marketer and I'm also the CEO, so I get the skepticism that you might have around that idea. But for us, if we increase overall traffic and we're not doing in the spammy way, that's a good, you know, success outcome for us. So you can see this red line. That's basically our traffic from July of last year to December of last year. It was pretty standard, pretty basic, like it was just running. And most of that was because of all of my LinkedIn activity. That is the first channel that I broke into. That's what I did every day. Showed up on LinkedIn, added, you know, context. And we were not doing any other advertising, we weren't doing any events, we weren't doing anything other than LinkedIn one channel. Because we knew, knew that it worked. We were like, all right, we're ready to make some investments into marketing. The first investment that we decided to make was into Exit five. So we're going to sponsor Exit five. We're going to do this, you know, bunch of things together. And guess what happened? Our traffic literally doubled the moment we started working with Exit 5. I don't need any UTM codes I don't need any click IDs. There's nothing else that we did. So it was a very controlled test. The only experiment that we ran at that point of time was Exit 5 sponsorship. And you can see the direct impact. We didn't have to do any holdouts. Right. That's not always necessary. But this is a very simple pre post type of an experiment and you can see that it's staying elevated because of the way that we ran this campaign. Now you can go a little bit more detail. I'm showing you month by month numbers. Now I'm going to show you the week by week numbers. This is directly. All of these spikes are perfectly timed with every single campaign that Exit 5 ran for us. Perfectly.

So it's a perfect example of earlier. Also throw out exit 5. You could do this with anything. When I was a CMO at Privy, we wanted to test podcast ads and we basically found 10 smaller micro, you know, niche influencers that were running podcasts. And we ran the campaign over a set of three months, like, wish I was a little bit smarter. And this is actually a simpler way to look at it. And just let's, let's, let's map those two things together and let's look at the spikes in awareness and interest in people hitting our site since we started running those podcast ads. This is cool.

Yeah, that's precisely it. So when you think about how you're going to now, why do people get into drop? Because here's what happens and you know, they take that 10,000amonth spend and they spread it. They spray and pray across five different channels at the same time. When you do that, you cannot get clear signal if you don't have sophisticated analytics. Right. So in our case, we don't need any sophistication. We're doing one at a time. We're proving out that exercise sponsorship works for us. And so now that's baked into our, our strategy. And the next investment would be coming in the second half of this year where we're starting to invest much more in events and dinners. And we're going to look at are we able to increase our traffic or leads or whatever our metric is by another 50%. We're taking big swings. We're not trying to go for 1% lift, 2% lift. We're taking, hey, how do we double, triple our traffic? Because guess what, we're a startup. We can't live with 1 or 2% increases. That doesn't work when you're on the VC train. You just have to go for big swings.

So on the, on the point that you mentioned earlier, like when you started writing on LinkedIn, was there a time frame? Like, did you look at the, did you know instantly? Because, like, obviously I love this topic of, you know, founder, brand and having the founder right on LinkedIn. And this is exactly how I'd want people to look at it. But I feel like oftentimes what happens is people give up too early. And so they were like, yeah, I was writing on LinkedIn, I wrote two posts a week for like a month. And you know, we didn't see anything from it. Or what they do is they only post with like UTM links and, you know, sign up for this webinar and do this thing. I, from my observation, it's like I saw you kind of commit to, all right, I got this company, Paramark. I have a strong point of view on the market. I'm going to commit to posting to LinkedIn with my strong point of view a couple times a week for like six months and I'm not going to look at the short term results. And then now a year later, you're able to look at it and see those spikes, like, did you know that going in? I think just like with some of these channels, like these more brand channels, that you can't go in there thinking of it from a direct sales standpoint. But when you give it time and then you get a year out, you look at it like we see the number one driver for us of Exit 5 memberships as an example, Beyond LinkedIn is our podcast. There's no links on the podcast. It's just people over time, listen to the podcast, hear about the community and eventually join.

Totally. I'll give you a counter to that, Dave. I'll answer your question directly. But like, look at this example. This is billboards and it's having short term impact on the outcome metric. Not long term, short term. So this idea that brand campaigns only work in the long term, completely flawed. That's bullshit. So if it doesn't work in the short term, Highly unlikely. It's going to work in the long term. Highly unlikely. Right. Like saying that somehow you're just going to sit with that idea and it's going to give you some crazy information 12 months from now.

So it's either going to work right away or it's never going to work.

Very unlikely that it will work down the line. Right, so exactly. Now let me answer that specific question to you, right, which is why did I do LinkedIn the way that I did. I don't know why I thought LinkedIn was a good idea. So, you know, I don't have the benefit of, like, going back in time and thinking exactly about it, but I started, and for me, the results were actually quite immediate. What that means is our ICP was engaging with my content right off the bat. That was like, one engagement. I was like, oh, another marketing leader liked my post. That's good. I just, you know, got a warm way to connect with that individual. I would go connect with that individual. And so it was kind of immediate, even if that was not sort of the, oh, I'm getting closed one revenue out of it. But the feedback, the engagement, the activity, the leading metrics were all very positive right off the bat, and so never stopped doing it. I'm like, oh, great. Like, let's just keep growing this thing. And by now it's reached a point where people literally take my LinkedIn posts and email me and say, hey, love this post. Can we have a demo? Right? It's like, I don't need to prove the efficacy of LinkedIn anymore to myself because, like, we know it works. Like, so I don't know if that helps.

No, it totally helps. I actually. I actually think about this a lot. When. When. When we were at Drift, we had a podcast, and the podcast ended up being the. If you talk to the sales team. Our podcast, which was not about the product that we were selling at all, became the number one driver of Pipeline. Like, until we made basically our first to ARR. And we used to joke, how are we going to measure the podcast? How are we going to measure podcast? Well, it turns out on all the sales calls, they had basically first found out about us. They started listening to our podcast and became fans of the podcast. And eventually, over time, they went to our website. They were deciding to shop Drift versus Intercom versus other thing, and they went with us. But if we had Gong at the time, we could analyze that. I also find myself quoting something you said to me early on at some point in an interview or something, you're like, we need to stop treating our customers like they're morons as part of this attribution puzzle. It's like, if I see Pranav online and he says something interesting, I don't need to click on this perfectly tracked link and get credit for that. It's like, oh, no, what was that guy's name again? Oh, yeah, Pranav.

What was the company?

I go to LinkedIn. Oh, it's Powermark. Boom. Then I go to The Powermark website. We need to give more people, we need to give our customers more credit in that if you do something that is memorable, people will find you. And let's focus more on the part of getting people to be memorable. Right.

100% repetition is key. And there was an interesting point in the chat about what about if your content isn't working? Yeah, absolutely. When we are talking about tests that didn't work, this YouTube example that didn't work, this particular customer, it's a great brand and they do something that's very relevant for YouTube, but YouTube ads didn't work for them. All that means is that the specific creative offer at yoution didn't work. Doesn't mean that it will never work. It just means that whatever they were doing at that particular time was not working. And so they have to change something about their creative, about the offer, about the execution of that creative. And that's, you know, absolutely sort of critical for any marketing team. Don't give up on a channel just because your first test fails. Okay, this is a fun 1B 2C example, but I think it's relevant. They worked with us for about seven months. They did about seven tests, all failed. Not incremental, no lift. Not incremental, no lift. The eighth test worked. So it's very easy to give up when you do experimentation. It's very easy to say ah, shit, like none of this stuff is working. But it takes a long term commitment to growth because not everything is going to work. Think about it from a sales perspective. You have to hear seven nos to get three yeses. I've lived that life for the last two years. That's the same exact concept in marketing. Seven campaigns are not gonna work, but the three that work are gonna make your business amazing. All right, I got super excited there. Dave.

That's great, man. This is great. This is what I want these sessions to be. This is is. This is perfect. So keep. Do you have more stuff or do you want to take a Q and A?

Let's do Q and A. Like happy to take qa.

Oh wait, there was comment about. Somebody did want the PMAX example. If you want to.

Let's talk about it. Yeah, performance.

Then let's do Q and A.

So this is a pretty big company. They have, you know, lots of different channels. Classic, right? Heavy reliance on paid search. And they're running out of sort of, you know, real ways to kind of grow that part of their allocation and performance. Max is this new thing that Google launched. And I do think that this is the future by the way. So any media buyers who are in the chat, every ad platform is moving towards an algorithmic buying process, which means you just tell us your brand, tell us your copy and messaging and give us your website, we're automatically going to generate a whole bunch of ads. That's like the generative AI part of like ad creation. And then we're also going to use our own machine learning and AI to figure out who to serve those ads to. So you don't have to worry too much about giving us too much detail. We'll just figure it out for you. Performance Max is kind of like that where they're looking at all the Google properties, whether it's the Discover feed on Google, it's YouTube, it's search, it's shopping, it's whatever else they have access to. And it gets a lot of hate because people say Performance Max is like bad inventory. Google is just trying to increase their monetization by giving you bad inventory and just putting ads there. So we ran this test. The target metric is MQLs which are basically hand raisers for this brand. And we only ran it in about 10 states in the US and so PMAX was performance max was limited to those 10 states and we observed what the conversions the MQLS were from those 10 states. This is an important point, I think some other people ask. So we're just. The blue line is just showing you MQLs from those 10 states where performance Max ran and the control is everything else. The other 40 states where performance max did not one. I'm simplifying the explanation a little bit, but I don't want to get too geeky here. And then you look at what is the lift or the lack thereof and you look at the pre test period to kind of inform yourself on what your test period is as well. And so in this case it worked perfectly for them. Now why did it work for them and it doesn't work for others? I think some people mentioned this here that you have to be very careful about all the exclusions. So this is a media buying thing. You probably don't want to have your branded search placements and Performance Max and you can do all of those exclusions. So you're only targeting surfaces and inventory that you're not capitalizing on, you know, your non brand search and brand search and other sort of, you know, Google properties by running it this way. So there might be people who are more, well, worse than I am on Performance Max in the chat. So if I got anything Wrong. Please, please keep me honest. But from a testing perspective, from a measurement perspective, it was very positive for that. All right, there was a good question here. How do you know how long to run the test? You want me to take that one, Dave?

You think I was going to hit that one?

The answer is actually all based on math. So when you do the experiment design, I didn't show you the process of how you do experimentation design, but you essentially get the. And this is where I plug Paramark. We essentially help you come up with the geographies, the time and the budget that you have to either increase or decrease to get a statistically valid answer. You can think of that whole process as very similar to an AB testing tool on your website. You know, you have all those AB testing calculators. You just put in some numbers and like hit enter and tells you. This is how you, you know how much sample size you need. You're kind of doing that. But it's a lot more sophisticated because you're working with much more messy numbers when it comes to, you know, actual sort of media in the real world. So it could be four weeks, could be six weeks. Typically, it's ranging between that time. If you have very noisy data, that means your data moves very up and down every day, then you need a longer period of time because you're trying to lift above that noise. So I'm getting a little bit data sciencey here.

I just said my answer is till you get bored, then just change it.

That works too.

Hardest part about turning off the branded search is the cfo. This is back in the day when I used to go to office. The CFO comes over at desk, and we're not showing up at Google right now. Oh, my God.

Yes, yes, yes. I don't know.

Hey, can I actually, I just want. This has nothing to do with what we're talking about. We should answer people's questions. But can you. Could you actually share the summary of. You hired a marketing leader for the first time recently, and you have this very thorough hiring process, and I thought that was interesting. As somebody who has in the past not done great at the hiring thing. Exit 5 team is awesome. I've hired some good people, but I think really treating it. You gave it a really important go at it, and I think it's a cool example. So if you just would refine that.

For a second, that'd be cool. Yeah, totally. No, it's a good one. And we should do an episode with Sam on my podcast because I think it's A good one.

Danny, I think you got to wait. I think you got to wait for a year on that. I just think that that's no disrespect to Sam. It's no disrespect. Like, just things happen, man. It's crazy. Everybody's in love. Everybody's in love with the new hire. Like, you know, the first three months is the honeymoon phase. And like, by month four, you're like, well, anything yet? And then this is slippery slope. So.

Well, well, so this is a really fun one.

Sam has had. Literally, Sam has entered the chat. I'm paying attention now.

I love this. So Sam. Well, first, if you don't know Sam, go check him out. He's awesome, and he's like a super connector. So I would say that if you want to learn marketing. And also, just like he spent time at Microsoft, at Shutterfly, SunBasket, he actually comes from more of a consumer background. So it was a little bit of a Sam and I kind of figuring out, how is this going to work? So I'm sharing sort of all the background here. And he did a power move in the process. Even before we started talking about the role. He sent me leads, and one of them converted.

That's amazing, Don. That's so good, right?

One of them converted, and I'm like, wait, what is going on here? And I'm like, you know, frankly, I'm like, in an alternative universe, I would go work for Sam. And so that was my thought process. And like, hey, if we're going to bring in somebody at a seed stage, right, where, you know, they're a leader, they're vp, and they're sort of pushing us in a very sort of, you know, big way. It's got to be someone that's baller, right? And so to steal Justin's language here now, the interview process itself was with Sam specifically. It was like multiple weeks, multiple months. We riffed out a bunch of ideas, but more concretely, we actually gave him an exercise that was a take home that was like, hey, take this other series B startup. So we didn't give him any cases on Paramark. We actually gave him a case on evaluate this other Series B startup because we wanted to be on neutral ground. This was an important thing for us. It's like, if we gave him everything about Paramark, we would be biased. If he gave him everything about Sun Basket, he would be biased. Right? He just has this information asymmetry there. Let's pick a neutral territory and have him do a whole sort of rundown of positioning, branding, messaging, campaign execution. And I think if I'm not wrong, Sam spent like a couple of days with us in the office. Like, he came, you know, on site and he just, you know, workshopped that whole thing with us with different people on the team. Right. We've got engineers mostly other than me. And so I'm sure it was a little bit awkward. You should interview Sam about that for him. Right. Trying to explain marketing to engineers. But that was the whole process and it worked out pretty well. And again, like, Sam was just had power moves up his sleeves.

Did you pay him to come hang out with you?

Yes, we did.

Well, Sam, I see that come up in our community a lot. It's like, hey, I'm interviewing for this job. And not that obviously, I think you handle it great. There's just something that comes up a lot which is like, hey, I'm interviewing you for this role. They want me to make a presentation and basically they're going to steal my strategy and not compensate me for it. And it's a really slippery slope there.

Yeah. So for all of our on sites, if we're doing anything beyond like a day long on site. Right. Which is pretty typical. So if we're doing a take home or what have you, we essentially structure it as a paid project and we agree on that up front.

Right.

So we're not saying like, hey, you do any free work that is expensive, but we do it like for engineers. Like our engineers come on site for three days if they can, if they can carve out the time. If they can't carve out the time, like, one day is fine. Right. So like, we, we customize it for each person.

Cool. All right, we're gonna, we'll, we'll export all these questions and we'll, we'll give them to you all. I know you all do there. There's definitely a content thing where there's a bunch of questions that we didn't get to. We got a ton of really, really timely, relevant ones in the chat. So I'm not gonna, I'm not gonna do Q and A right now, but maybe you all can answer them in some content form or another. We'll give you the export of all this. Thanks, man. This was awesome. This was super fun for me. The chat was awesome. And I messaged something to Daniel during this session and I just want to give a shout out to the people who came to this. Man, there is so much information out there online to treat your job like a craft and to be like Yeah, I could get the recording, but to actually come and hang out and ask questions and take notes and, like, this is, you know, you. You get what you put in. And I just really appreciate, like, marketers that are here trying to learn, trying to get better. Like, it's very easy to just listen to podcasts and mess around on YouTube, like, putting in the time and effort to, like, come in and do something to help you and grow your career and your business. Like, heck, yeah, those are our people. And I love that. I believe in self development and personal development in, at work and outside of work, and so I love that. And Pranav, you're always. You're always great, so thank you for doing it. Go check out Powermark. They're doing a great job. Pranav's got to go back to his inbox because this is the best inbound lead day ever. Well, for now, because they announced their seed funding, and he came out here and hung out for a little bit, so appreciate it.

A lot of fun. Great question.

Good to see you all. We'll see you later. I was supposed to ask you to rate this, but. Yeah, rate it. Rate it real quick. Where's the poll? Where's my producer, Danielle? So we got a measurement. Measurement. Measurement. We're supposed to do this. Okay, no poll. All right. It was great. 10 out of 10. See ya. Goodbye, everybody. Who's awesome. See y'. All.