Canva's incrementality playbook: Branded search & MMM buy-in | Matt Castino (Canva) | Coffee Breaks

All right. Matt Castino is the marketing measurement lead at canva. Matt's background comes from both academia as well as industry. He has a PhD in Behavioral Neuroscience and he now applies his extensive expertise in statistical analysis and human behavior to help drive data driven decision making in marketing at Canva. In today's episode we had a great conversation touching on experimentation, how to get buy in from your organization about incrementality, Canva's branded search and whether or not it' actually driving incremental subscriptions or not. The surprisingly hardest part of actually having a media mix model, communicating trade offs to the finance team, marketing measurement when operating in 15 countries and many, many, many more things. Please, please, please give it a listen. I hope you really enjoy it. Super educational both for practitioners as well as for people on the data science and statistical modeling side. Matt was super insightful guest. Hope you enjoy the conversation. Maybe just to start, tell us about your role, what do you do? Why are you on a marketing science podcast here?

Yeah, so first of all, thanks to my show, really excited show, thanks for having me on the podcast. I'm really excited. So a few years ago Canva went down this path of revisiting how we measure, how we think about measuring marketing performance and we underwent a big shift in our thinking and our approach in how we use.

Right.

So we used to think quite a lot about or we historically we've done things around using attribution like frameworks for measuring marketing performance and using that to make decisions. And about three years ago we went through a big philosophical shift in how we think about this and started to go down the path of really focusing on approaches that led us down the thinking about measuring incrementality.

Right.

And as part of that we created a designated team within Canvas marketing data science team called the Marketing Measurement Team, which I lead. And that team is really has a pretty simple goal to help Canva maximize its return for every single dollar of ad spend that we put out there.

Right.

So I lead that team and as opposed to being aligned to a particular marketing vertical like brand marketing or performance marketing or lifecycle marketing, where sort of positioned horizontally across all of marketing across Canva, Swiss run modeling and experimentation to help different verticals within Canva measure their performance and improve and improve their impact essentially. So yeah, the team's about two or three years old now and yeah, just really, really enjoy working really interesting problems and sort of like it's been a journey, right. Like we've gone from like literally the start and you know, trying to think about like how philosophically do we want to do this? How do we want to be. How do we want to think about marketing, measuring measurement, and then go through the process of actually implementing it, scaling it, refining it, getting better and. Yeah, so it's been an exciting journey.

Sounds amazing. You are right that there's no shortage of interesting and challenging problems to work on in marketing science and marketing measurement. Curious. Was there, like, a catalyzing event that caused the creation of this team or. I think, you know, a lot of our listeners are probably like, man, I wish our company had a team like this. But, like, so how did you get there? Like, what caused the team to come into existence?

Interesting. To be honest, I can't claim credit for that, which is fine. So the Callus was a person.

Right.

So we hired someone. Sam Redfern. He came from Facebook, where, you know, he spent a lot of time helping companies build measurement solutions. And he came on board three years ago, and he really drove this change within the organization. So he really fought for it, spent a lot of time advocating for it and coming up with this. This framework of how we should think about marketing measurement. Right. So he was the initial advocate for this solution.

Right.

And, you know, he's gone. He's still a canva, obviously, he's gone to bigger and more interesting things. And, you know, over the past couple of years, I've been kind of the person that's really led and driven the implementation of this way of thinking and scaling this way of thinking across the organization and getting leadership by. And thinking about all the different ways we can use this data to make good decisions. But, yeah, I guess the catalyst for this change was just a new thinker.

Right.

I think that every organization can benefit from bringing in people from the outside who think about things in a different way and then just really effective at driving change.

Sounds amazing. You mentioned the word incrementality. Well, you mentioned a couple of interesting words. You mentioned one word, incrementality and another word, framework. Can you tell us about, like, what is the incrementality framework at canva? Like, how. How have y' all brought that to life and are actually using that to drive value?

Anyone who knows me or anyone who knows the people that have been involved in this the past few years knows of what we call the pizza analogy. The pizza. The pizza shop analogy that we use. Like, whenever we're presenting this to someone new, we always show this one slide in a slide deck.

Right.

And essentially what it shows, just this visualization, is it shows two cars driving along the Road passing a billboard advertising two for one pizza, right? And there's a pizza shop in the distance. And the first car says next to the first car was already driving to the pizza shop. And the second one says the second car says decided to stop at the, at the, at the pizza shop because they saw the billboard, right? And when we think about, when we try to explain to people that we think how we think about incrementality, like the philosophy, the framework, right? It's that, you know, basically these traditional attribution style frameworks will think, will. Will tell you that that two people saw the ad and two people went to the pizza shop. Therefore we drove two conversions, right? But we know that many people just do stuff, right? They see ads and then they sign up, they. And then, you know, the, the, the marketing intervention was, was not, you know, causal and driving that person's behavior, right? People see ads and they also do stuff organically and those things just happen together, right? So we show these people this analogy that kind of like crystallizes, I think for people how we think about things, right? That like simply knowing that someone saw an ad and converted doesn't mean that that ad was causal or instrumental in driving that person's behavior, right? And that's kind of like the overarching framework, right? The overarching way of thinking about things. We want to isolate and, and reward marketing channels, campaigns, interventions, whatever you like for driving those behaviors that would not have happened or will far less likely to happen if that person didn't see marketing, right? Because that's where the real value is, right? So that's kind of like the overarching philosophy or framework in terms of incrementality, right? Is how do we isolate those marketing interventions that are causal to people's change in behavior, right? And our approach is, our way of thinking is that experimentation really is the gold standard for thinking about this, right? In terms of the best way to get to the God's honest truth of the subject is to run an experimentation. An experiment, right? And we'd run lots of experiments in our team. We run match market tests where we can. We've got another approach which we're going to talk about in a second. But the reality is that it's really hard to run loads of experiments, the kind of breadth and scale that you want to. Especially when we're a global organization, right? We're running marketing in 15 plus countries, right? So we want to get a sense of. We use modeling, we use MMM to help us get this sort of like constant heartbeat to how our marketing is performing across this massive breadth.

Right.

And then we calibrate and we optimize our model or we improve our model through iteratively experimenting and then using that to calibrate and inform our model.

Right.

So that's our basic approach. Yeah, it's been, you know, it's been very. It's been a journey, I guess.

Right.

Like, we've done things like so many different ways. It's like the journey to this point has been very nonlinear.

Right.

I want to dig into that because I talk to a lot of people who are on this journey and I think a lot, most of them are at the earlier stages of this journey and a lot of them find it quite painful. And whether that's like educating executives about this idea of incrementality when they're used to looking at, you know, touch based digital tracking, attribution methods and, you know, how to pull this together. There's experimentation, there's. Mmm. Like, can you maybe, I guess, talk a little bit more about that journey and then also, you know, give advice like, what worked well? What would you do differently on this journey if you were starting over for people who, for other marketers or measurement people who are looking to go on this journey as well?

Great question. So I think for us, the biggest, the biggest thing is to get buy in from the people in your organization.

Right?

And the best way to get buy in from these people is to, is to answer questions that. Or to solve problems that they don't have an answer to.

Right.

Or things that have been really at the forefront of their might they've had no way of approaching or thinking about.

Right?

So the biggest win for us was an early experiment that we ran and our CEO actually said one day, why am I spending so much money on advertising for my own brand name on Google search, right? Why am I pay, why am I bidding on the word Canva? So why am I paying so much for that? Right. And is it necessary?

Right?

So the first experiment that we ran was to literally answer that question posed by a CEO, right? And we said, like, how can we think about, how can we solve this problem or think about this problem and really get like that would generate advocacy and buy in for how we think about this, right? So we ran an experiment where we tested different bidding reductions for Canva branded terms and, and found that we could just turn it off, right? We ran an experiment and we just turned it off. You know, we showed that like, if you turn, if you basically eliminate branded SEM for Canva Terms that the organic search result just completely compensates.

Right.

That the. That the organic result just completely carries the. The. Carries the. The user through to the website and you don't need to pay for canva branded terms or we turn that off.

Right, right.

And that has been sort of like. That was like the cornerstone thing for us, which was like, answer a big business problem, answer a big business question, get the buy in and then get the trust, and then that will help you get advocates for your way of thinking about things. So before we had this kind of framework of experimentation and modeling had no way of answering this question.

Right.

And this just gave the business a load of confidence that this approach would hit us in the right direction.

Right.

You also need to know your lovely at once.

Yeah, no, I love that story because first of all, it resonates with me. I've gone through exactly that of like, branded search has the big question mark, you run the experiment, the results are striking and it sort of gets the whole organization aligned. And I think I've talked to a lot of other people who have had exactly, literally, exactly the same channel, same question from the CEO. CEOs hate having debate on their own for interim. I can tell you this from experience. And so this is like a really. It's a great crystallizing moment. Um, and so I would say, like, you know, if you're on this journey as well and you're listening to this, like, that is like an awesome experiment to go run often because the results are interesting and it's not always the case. I will say that, like, it's not always the case that the result is branded search does nothing. I've worked with a number of businesses where branded search actually is highly incremental for them, or at least turning it off does cause their business to drop. And so you do need to go run the experiment to prove it. It does. I think it is incremental in a number of very important cases. But this type of experiment could be so valuable for getting an organization all of a sudden excited about incrementality in really positive ways that I just love the story and I love that it was also true for y'.

All.

And I think that there might be a perception for companies, different sizes and maturity, that you need a highly specialized set of tools or equipment or people like data scientists to run that kind of experiment. You really don't.

Right.

Like, as long as you've got someone that has a vague understanding of how all these search platforms work and can orchestrate an experiment, that platform or Have a contact in the, with the publisher. You can run that kind of experiment and then analyze the result. Any data scientist can analyze those results.

Right.

So you kind of don't need this like massive suite of tools to run the first thing, right? To do the first thing that gets the buy in to get you on that journey, right? And yeah, a lot of open source packages for doing GLF experiments. The barrier is smaller than you might think it is to get your foot in the door, right. Or to dip your toes in the water, right? So yeah, that was a big step for us.

Big first step, start off with a bang, things are going great. The way that you described this journey, it sort of sounded like it was slightly more sinuous after that point. Maybe like, what were some of the rough patches that you went through or where were places where, you know, it was like, oh man, it's not, it's not all easy from here. I would love to hear again a little bit more about like, what are the hard parts of doing this?

I'll be honest with you. I think that we've been fortunate to work with great partners over the years. We outsource our mmm, right? And I would say that from, from my perspective, the hardest part hasn't been the modeling because obviously we outsource it, right? The hardest part is everything around it, right? It's like the. I would say, to be honest, the hardest part of running an MMM is now you have the data, you have an organization or a team that doesn't have the muscle or doesn't have the knowledge and experience and the, and the framework for leveraging and utilizing that data and figuring out what to do with it, right? Like, I remember when our first modeling came in and we were just like sitting around a table and going, we've got these curves, we've got these like, increment, these rises, these marginal rises. So what should we do with it? And we at least like three or four meetings with finance and going like, oh, we could do this, we could do this. And obviously we got there, right? But like, took quite a lot of time to figure out, like, what exactly do we do with this, right when we got it, you know, so this is an interest.

This is an interesting topic because I am. People have described me as insane many times when I see this and I see like, okay, we see the marginal roses. In my mind, I'm like a robot and I'm like, okay, immediately. All right, we just changed the budget and we optimize it so that we find the profit maximizing Point on the curve, you know, done. And then I like walk away. But it's very clear that like organizations, that that's not actually feasible for organizations. So like I guess my question to you is like why like what was the blocker there to like looking at the curves to actually changing the marketing budget?

Yeah, great question. To be honest, I would say that when I, when I first started out I was like you, I was like we'll just create this feedback loop between the, an automated feedback loop between our MMM and the platforms that just optimizes towards whatever the model says, right. And we can just go home after that, right? It just takes care of itself, right. I've learned a lot since that, that first time. And I would say like the first thing is, is that I kind of view our role as less about going to finance, going to marketing and saying do this, spend this. Our role is about communicating trade offs, right? And MMM is really, really great for communicating trade offs, right? Because you, you don't know whether spending for the profit max is necessarily the avenue that your business wants to go down, right? You want to be able to communicate the trade offs between growth and efficiency, right? It's like how much do you want to grow and how much do you want to pay for that growth and being able to clearly present those trade offs to your finance team, right? Because maybe that maybe you're in an organization that has cash to burn and just like really high, like has really high growth aspirations, right? And how that company spends is, might be very different to how a company that's like we're really constrained. We've got to try and do more with less. Got to really focus on profitability and profitable spend, right? So I think that like, and to be honest, Canva has fluctuated since I've been there to like different, different positions or different like attitudes of how we think about things over the course of the last three years, right? So to me it's about, you need that connection with finance to understand, you know, your company's financial situation, its growth aspirations. And you know, there are also situations where what the model says to do might feel counterintuitive or might feel contrary to what the market is on the ground are telling you. And that's still valuable information, right? They may say like the cost per trial or the cost per in the platform is really saying something quite different to what the modeling saying. Like, you know, what do we do in this situation? We don't want to blindly trust this model necessarily. So like, you know, maybe we want to Calibrate with an experiment. Maybe we want to, or maybe, you know, we're about to launch this new, these new creatives or this new strategy in this platform. We just run a platform experiment that really showed that like targeting in this different way has a massive lift. The model might take some time to catch up to that. Maybe we don't want to just like slash the budget or cut the budget of that channel.

Right.

So I think it's about trying to understand, to leverage these different perspectives and to really understand the business and then help the business and help communicate clearly to the business the trade offs of different investment decisions are.

Right.

And the other thing is like not all marketing is necessarily designed to drive short term roi.

Right.

Like you've got brand branded spend which like is it primarily KPI'd on short term revenue or user growth? So there's more nuances than simply just spend. Exactly. To the dollar, what the model says, right. I think that you need those different perspectives. I think that was, that's been the hardest part, I would say is figuring out how do we work together and leverage our different perspectives, inputs, subject matter, expertises to figure out how we make the most of what we've got. And I think that's been the hardest part, but also where the real impact's been.

There's a lot to love about what you just described. I love the framing of using the model or the models. Right. Like there could be different sort of aspects of modeling that are happening here, but really it's about describing trade offs. Right. And helping the business understand and really making concrete the, the trade offs that the business might be facing. I mean, I've seen this, I'm sure you've seen it before as well with, you know, less sophisticated businesses. They're like, well, you know, we spent $5 million last year, we're going to spend $10 million next year and the revenue is going to double because you just, you know, move the formula across in Excel and that ends up causing a lot of pain in those organizations when that doubling doesn't just magically materialize. And I think getting really rigorous about incrementality really helps give you a better way of thinking, at least thinking about those questions as opposed to just like again, you know, the finance people love to just drag the formulas across, but if you don't actually understand incrementality and what the levers of the different marketing channels are, can end up getting backed into a really nasty corner. One of the other things that I, it sounds really interesting is the relationship between like analytics slash data science and marketing and finance in your organization, can you talk about like what's, what does good look like there in terms of collaboration across those teams and how do you help those teams, which at other organizations at least are often at odds? You know, I talk to a lot of CMOs who constantly complain about the finance team, the CFO, and a lot of CFOs who do the same about the marketers. Like how do you promote good, healthy collaboration when it comes to these trade off questions across those, across those different groups?

I think like we've gotten better at this over the years as well. And I think that, I'll be honest with you, Canva is one of those places where because people really want to work here, we have the luxury of being fairly selective with who we hire. We can hire for culture, right? Not every organization has the luxury of being able to hire for culture, right? So I think that it helps that here you can always assume positive intent for everybody, right? And I think that that starting point is really powerful, right? It's like we all want the same things, right? So that makes conversations kind of easier, right? It's like we all want the same thing. We have different ideas of how to get there, but we all want the same thing, right? I think that it's sort of like when we get go time to do budgeting for the following year, we kind of, we sort of have the data team provide a. If we just spend to exactly what the model said, right? Or you know, this is what the, this is what the data suggests that we should do, right? And then that provides sort of like a starting point, right? And then simultaneously, you know, finance and marketing, preparing, you know, sort of like what are the strategic principles or what of the, what are the strategic things that we value? What are we trying to achieve? Like what are our goals, our aspirations, both long and short term? And then we sit down, we just kind of put them together and say like, where do these, where do these things work well together? And where do we need to rethink from both sides? And we just kind of iterate through that. I'd love to tell you that it goes smoothly. We just sort of like wrap it up in half an hour. There's usually lots of back and forth, right? And I said we're getting better. Like for example, we started this process in October this year instead of like end of November the year before, which made the end of the absolute chaos. But yeah, it's, it's kind of like we try and figure out where the, where the strategic stuff, how that intersects with the data and then say like, and again it comes back to that trade off thing where it's like, if you really value this strategic thing, this is what it's going to cost or this is what you might give up if you were to invest this money for a profit max situation in another market. And we don't just have, that's the other complex thing is we don't just have a single market, right? If we were just a company in the US then the trade offs become, you know, the degrees of freedom are much smaller, right? But when you're operating across 15 countries, the trade offs become quite real, right? And Canva has, you know, two different, like parallel objectives. Obviously we want to make money. We're a company, right? We want to make money. But we also have this free product, right? And you know, we, the mission statement of the company is to empower the world to design. So we want to get people using the platform for free. We want educate, we want to seed for the future by getting schools to be using the platform and teachers so that when they get older and you design platform in the future, they've had years of experience using Canva. So that becomes another thing is like how do we consider this trade off between making the most money but also trying to grow this free product that we want? And there's so many dimensions there and I just don't think you're going to be able to solve a complex problem like that by just doing what the model says, right? So you know, we just try and figure out where the strategy and the company objectives intersect, what the model's saying, communicate trade offs and then go from there. And the good thing about the, about our modeling is it allows us to be flexible, you know, in the sense that we don't lock. I think previously before we had these kinds of tools that allowed us to see how we're going. We just kind of like chose a budget for the year and just did it. And now we sort of can go, this is working well. This isn't working well. This is, you know, maybe we need to be, we can, we have a monthly meeting, we sit down and talk about like, where could we better invest, how could we be dynamic here? And over time we're getting better and better at being flexible right before it's like, maybe it's too hard to change this way or maybe we've got too much momentum in this direction to change. But one of the things that modeling has given us is, is that the courage and the power to be flexible and be dynamic and not feel locked in to a strategy that we chose at the start of the year. And that's really powerful as well.

I love that. I think it really highlights again the ability to think about trade offs and do that in a dynamic way. I mean, you said one thing of like, oh, you know, we obviously don't want to just like follow exactly what the model says because there's a bunch of other reasonable things that we need to be thinking about. But also one of the things that's really nice about having a model, and I use model sort of in the, in the generic sense here, like an Excel model does this, but like it gives you a structured way of thinking about a problem and as you said, thinking about those trade offs and being able to at least have a reasonable conversation that's based in sort of some sort of structure as opposed to just like, I'm the cmo, I want to, you know, run TV ads, I'm the cfo, I hate advertising. Like, that's not productive. But like being able to get together and look at some coherent, systematic way of thinking about the world, it can be really powerful to help break out of those deadlocks and say like, okay, we're going to try this and we're going to see how it goes and then we're going to move on, on to the other thing. So I really love all of that. I want to change topics slightly. You know, it sounds like y' all have really made the pivot to thinking a lot about incrementality and using experimentation and statistical modeling in order to, to drive the business. One of the, the things that can be challenging about making that change is the problem of uncertainty. So one of the nice things, or I'll put nice and square codes. One of the seemingly nice things about, you know, deterministic methods like digital tracking, or it's like first touch attribution or last touch attribution, is that there's no uncertainty right there. It's just like either it's the first touch or it's not, or it's the last touch or it's not, or it's evenly weighted, like very deterministic. You get to an answer on the other side. I think we probably both agree that it's probably not the, it's probably the right answer to the wrong question. But it has that very nice property with both experiments, right, where you have, you know, you're running some experiment, you have to interpret the results on the other side and Understand if it's statistically significant or what the range of compatible values are. And media mix modeling obviously comes with a lot of uncertainty because there's lots of degrees of freedom and variables that go into the model. How do you think about communicating that uncertainty to the executive? I know a lot of people that I talk to, like, they really struggle with that. They're like, oh, I want to move our company off of Last Touch attribution, but my executives hate uncertainty. And so how do you all think about that? How do you communicate that to your executives? How do you get them to move from a deterministic world into a maybe a probabilistic world? Or do you.

Yeah, it's a good question. I think, like, I think the other thing that's really valuable about, about love, about attribution based frameworks, like Last or First, hardware you want to use is it's just so tangible. Like it's almost like you can reach out and touch it, right? It's just like person clicked on ad person did this, right? It's so easy to understand, right? It's not a lot of complexity there. And, and marketers love thinking about things in terms of the user journey, right? They love being able to sort of say like, person saw ad, person came on product, person had this interaction with the product. Like you can follow them all the way through journey throughout the funnel, right? And obviously there's still value in attribution based frameworks for that kind of thinking, right? But sort of it was so hard just to get people to think about, they would still use user journey, like language and terminology to describe MMM outputs or MMM worlds, right? And they had trouble extracting it away to aggregated data, right? And that's hard. Like it's, it's hard for people to wrap their heads around, especially because MMMs are complicated for statisticians, let alone, you know, people who are not from data, science or statistics, right? Like, these are complicated things. And Another challenge with MMMs, right, is that you often get results and you sort of go, like, why, like, why does it look like that? Why does it, like, what changed here? And there's often not a very good answer, right? Because, you know, Last Touch allows you to break things down, you know, by like customer profile segment, all these sorts of things. Whereas MM's aggregated, it's harder to diagnose, you know, where things got off track, right? So that's a big challenge. And the short answer is, is like years and years of communication, right? Like it's not a. It's not like a great. It's not, it's not an answer that people like love or want to hear. It's just like, it's just like repeating the same message in different ways. That's kind of a challenge, right? It's like, how do I really tapping into sort of like how do I explain this in a way or find a way of explaining this. It's going to resonate with the people that I'm talking about. And especially like when you're working in an organization that has finance, people who love to drag the formula across or who love things like tabulated and itemized on a spreadsheet and marketers and then, you know, leadership who have varying degrees of experience with, you know, data science, with marketing, with finance. There's so many different audiences as well. And that presents another challenge. I wish I had a better answer to you than just sort of like you just really need to find a way to communicate in a language and try different ways of communicating it to help people get over the line with.

Any like metaphors or anything. Like any sort of way of framing it that at least in your experience has worked especially well to help people get their heads around this, these, these as you note, like very complicated and often unintuitive topics and ideas.

I went through the pizza analogy before. Some very helpful. Right. But I think that sort of like one thing that's helped us is being able to say if we invested in this way, if we execute on this strategy, we think that this is a range of possible outcomes for what could happen.

Right.

And sort of providing this lower and upper bound.

Right, right.

And finding out, working with them to find out. Is that within your risk tolerance?

Right.

So some examples that have come up within Canva is we really feel like we want to invest more heavily in brand. To do that, we may have to pull back a little bit from performance marketing.

Right.

Because budget's not infinite.

Right.

And we. And they'll say to you like what might be the revenue or the user growth trade off to pulling back a little bit from performance and reinvesting and doubling down in brand because you really want to get this. We really think that the long term investment's really important here for, you know, driving awareness and consideration of our brand.

Right.

And we can say one, we can tell them how they can pull back in the least impactful way in terms of, we can tell them which channels to take it from so that it has minimizes the impact. And then we can say over this period of time, we think the impact is between this and this and just providing a range of possible outcomes. And I think that that resonates pretty well with basically all audiences. Right? It's understanding the best and worst case scenario, knowing that most time is going to end up somewhere in the middle.

Right?

Yeah, I love that and I think so. The thing that you're describing, which is what I actually like often recommend to people in this situation is focus on the uncertainty in the business outcome. Like the thing that the executives care about as opposed to uncertainty in some parameter estimates. Because if you're like, oh yeah, the uncertainty around the time shift varies from 7 to 34.7 days with a 90% credible number. It's just like what are we talking about here?

It goes absolutely without saying is like, is you try and talk about ad stock timeshift priors. Try explaining prior to somebody.

Like, oh, I've tried.

Do you know how hard it is to explain a prior to somebody that doesn't have a, you know, the fake background in Bayesian modeling?

Right.

Don't. If you, if, if the word prior comes out of your mouth when you're talking to a cmo, is that a wrong?

Right?

Like no, guaranteed you've done it wrong, right? You need to abstract away from the technicalities of the model and talk about it as you said, in concretely in the terms that they care about, right? Because your CMO doesn't care if the uncertainty is because of, you know, you know, a lack of confidence in a timeshift parameter or like, you know, they don't care about that stuff. What they care about is revenue between X and Y, right? And someone also said something to me once which was really helpful in how I communicated. They said think about all the things that you need to be across in your day to day job, all the different priorities, all the different streams of work that you are across right now. Imagine what that's like for your cmo. Imagine what that's like for your CEO, right? Imagine like how light touch they can have. They have to be on every individual work stream in order to be across the full breadth of it. And that was like really crystal. Imagine how many emails you've got unread right now. Imagine what that must be like for the CEO. For the CEO, the cmo, right? They just don't have the time to connect deeply with every single thing that they're across. So you need to say what are the like four things? Like what are the what is the what, why and what should I do of this thing? And nine times out of 10, that's all they want. It's just like, what is the problem? How are you going to fix it? What is the implication for me and what do you need me to do, right? What do you need from me? You can drill abstract, all of the complicated modeling down into those things. And that's a really effective way of having good conversations because trying to explain the ins and outs of your model is just not going to work.

And I think even outside of like the some complicated statistical model, like a media mix model, even with the results of like an experiment, again, convert it taking like whatever the outcome of that is, like we estimated the lift to be between X and Y and like just convert it into. If we follow this planned budget, here's what the impact of this is, right? And like make it into that end of year revenue number or the end of quarter revenue number, the thing that everyone knows and sort of use that to create the bounds that all of a sudden makes the information way more tangible to like an executive audience than trying to talk through like, oh, the p value was 0.04 and that. And then you have to explain a P value. It's like, that's lost. Instead, what is the impact on our forecast for revenue at the end of the quarter? Like, that is a thing that people are going to get and can really, I think, latch onto in a really nice way. And so that's often my recommendation is like, get away from the parameters and get towards the outcome as much as possible.

I think that another point that's important to make there is that your ability to do that, to drill it down into things they care about, actually starts so far back in terms of making sure that the KPI that you've chosen for your models connects really as closely as possible to that business outcome, you know, because I think that where we've iterated in the past is metrics is one of the things that we've iterated over quite a few times. Subscriptions are a bit of a complicated animal when it comes to MMMs, because people, unlike E commerce, people just pay right upfront for their order, they buy socks, they just wish we were a sock company. Be way easier for just sold socks with subscriptions, right? People like, they sign up for a trial, you don't know if they're going to pay and then they're going to cancel it at some undetermined point in the future.

Right?

So we've iterated over our metrics quite a few times, right? But and the key lesson that I've learned From that is your ability to have that conversation with the CMO like 11 months down the track about what they should do, right? Starts with you getting your metric thing right. You know, it starts with you developing a metric or a KPI that's simple, easy to understand and super closely connected with the thing that you care about. Hopefully you just sell socks and you can just pick revenue.

Right.

But not all businesses are like that.

Right.

Your ability to have that conversation is critically dependent on you picking a metric that allows you to talk in a language that diverse people across your business are going to understand.

Right.

So I said that the modeling for us has been the easiest part. Again, we don't do it right, but the hardest parts, the before and after, right? It's like what do we do with it? And then how do we categorize our media spend? How do we choose a metric that's really connected and valuable to talking to the business about later? And getting those decisions right is worth iterating over and thinking deeply about before you just do it.

I mean, I think this is very sage wisdom. And honestly, when I've seen, I've seen a lot of like media mix modeling programs go wrong and not when they're using recast, hopefully, but like, you know, I talk to a lot of people and a lot of the times they make this mistake that you're describing where they just choose the wrong KPI variable to model against and it's either something that isn't connected to the business and so no one uses it, or it's like too far away from marketing. And so it's not actually impacted, but it's a thing that a lot of people overlook. You know, I think especially people who don't come from like a data science background, they're just like, oh yeah, it's the KPI and it's just like that's not the important thing. The important thing is the marketing. But actually you have to get the KPI right otherwise everything else sort of falls apart. And so I think it's really good advice and worth taking that like extra, you know, however many hours it's going to take to get alignment across the business on like what is the thing that we are actually trying to inflect and you know, is that the right thing to be measuring? And if it's not like what is the other thing and what are the trade offs? Those conversations are super, super important in order for this to be successful. Otherwise it ends up just going off the rails. This conversation about the complexity of, you know, your Business and subscriptions made me think of a sort of a more tactical question. When you're doing some sort of experimentation, whether they're geo experiments or experiments on the platform, how do you think about that in the context of like a subscription event where, or like a subscription, I guess, like again, where it's not just like a one time purchase, there's like quality to the subscriptions. But you're probably only running your experiments over some, you know, medium length to short length of time. Like how do you bring those pieces together? How do you think about that? You know, maybe with that like branded search experiment that you run. Like, is there a difference in like subscription quality or over time effects that you need to take into account as you're analyzing those results?

Yeah, really good question. So the first time, the early experiments that we ran our KPI was trials and cut and dry, diminishing returns between spend and trial volume. Right? But as you pointed out, not all trials are created equally.

Right.

And there's probably also an interesting dynamic where between how much you spend and trial quality.

Right?

So you get diminishing returns in terms of for every dollar that you spend, you get less trials back. But for every dollar that you spend, it might be the case that the quality of those trials also diminishes with additional spend.

Right.

Because you're reaching audiences potentially with less use cases or so on and so forth.

Right.

So that's where we started, where we, where we got to, where we iterated to is we have an internally developed model that estimates, a machine learning model that estimates the value of a trial on the, it used to be actually on the day that the trial was struck. We found that delaying that estimate by about a week makes a huge difference to predictive accuracy. Because knowing what people do in the first week is quite telling in terms of, do they just cancel straight away? Did they like complete one use case and then never use it again? Or are they using it consistently over the course of the trial? But the machine learning model tells us for every trial, what do we expect to return over some window of time? Currently it's six months, Right. Because the longer that you try to estimate in terms of like 12 months forever, the quality of the prediction diminishes over time with longer prediction windows. But yeah, we estimate how much each trial is worth and then that allows us to understand that relationship better than just using raw trials. So things like what countries a trial from, what device the trial is struck on. So on average, for example, we make a lot more money from, you know, $2,000 iPhone trials than we do from, you know, $7 Android phones.

Right.

So, yeah, that model allows us to, to reward channels which drive the highest value trials, essentially because, yeah, not all trials are created equal.

I love it. I think, I mean, I've seen other companies do similar things, so I think that makes a lot of sense. But it sort of highlights the complexity here. Like even a sort of simple experiment, like it requires this amount of machinery to get it right because otherwise you can get misdirected. You could say, like, oh, this channel drives a ton of trials really efficiently, but if you forget to account for trial quality, like you could end up making the wrong decision. So I think it really highlights that. The last thing that I wanted to talk about was, you know, you mentioned you outsource your media mix model. I'm curious, like, how do you do that? Like, how do you outsource it and then know that it's good? How do you think about finding the right partner? Like, is there any way that you can talk about, like, how y' all think about finding that partner, making sure that the quality is actually good and then like being able to really use that for the internal trade offs and decision making that you're, that, that you're actually doing practically with those results?

Yeah, I mean, I mean the alternative that you left out is doing it yourself.

Right?

Like building it internally.

Right.

And you know, there's, there's obviously Robin like meta's mmm package. You can build pretty rudimentary mmm in Excel if you're really keen. I guess. Like in terms of like the build versus buy, first of all, it's like, what do you want to spend your time doing?

Right?

Like, what do you, how do you want to invest your data science resources? Right. I find it much more impactful to be connected to the business case.

Right.

Which is like, let somebody else worry about this incredibly complicated statistical problem which could easily absorb like all of our time and energy into just making it run. Or do we want to be spending our time talking to people, connecting it to business cases and actually driving impact for the business.

Right.

So we thought a lot about that and we're like, well, where do we want to be spending our time? Like, and if we build an mm, we're going to build it once and then we're just going to run it till the end of time.

Right?

We're not going to like invest significant, we're not going to be able to invest significant resources into making it better, to optimizing it to like reading the Current research, keeping up to date with industry trends. Like, no one's resourced for that, right? I don't think at our scale at least. So. So we thought that we'd like to be more connected to the outputs from the model than the inputs to the model, right? Or to, to machinery of running the model, right? So first thing, right, Choosing an MM provider, that's hard, right? Like, and I think you'll find that whenever you talk to an MM provider, they will tell you that their way of thinking about it is the right way and everybody else is wrong, right? And definitely, yeah. And the question is, like, can they defend that statement, right? So when we went about choosing an MM provider, we came up with a challenge exercise, essentially, which was like, we've got some data. We've got. It's all prepped and ready in terms of, like, it's in an appropriate format for you to start modeling. It's not like, you know, we can just give you a bunch of raw data and say, go ahead, like, we've categorized, come up with a high modeling hierarchy. You've got a KPI that we're thinking of using and can you, can you model it for us and see, like, image. Present the results back to us, right? And I'll tell you, in the thing, in those meetings and those situations, we went back, the thing that we cared least about was the roi, right? Like, ignore that. That's irrelevant, right? For the, for the purposes of choosing the right vendor, right? Because the question is like, how can they prove or how can they show you that these, these predictions are reasonable? So you're looking for things like, can they show you forecast accuracy? Are they willing to be transparent in terms of what they show you from the model, right? Like, some vendors are very cage about what they show you. They just show you some curves and they show you the data and they all. Everything under the hood's bit proprietary, right? We don't want to show you that stuff. So, yeah, I think, like, can, can they show you for forecast accuracy? Can they show you, and are they transparent with the rigor in which they've gone to estimate these things? And you can really tell by the questions they ask you, right? I think that's like, aside from the outputs, like, a good inventor is so thoughtful with, like, the questions that they ask you. Because the reality of modeling, right, is that you're trying to estimate like hundreds of parameters from very scant data, right? Like, you might be trying, you're trying to estimate stuff from like 800 rows of data, like just like, it's really not that much data. And, like, so. And this thing that people can find uncomfortable is, like, there's quite a lot of what the modeler does, and decisions that modeling modelers make along the process have quite a large influence over the final result.

Right.

And you want to make sure that you pick a provider that's thoughtful and structured and transparent with what those decisions are, because that has such a great. A massive influence over what the model is that you get.

Right.

And so I think that, like, for us, it was like, model validation and transparency.