How to make marketing analytics a true business partner | Dominic Williamson | Coffee Breaks

I'm sure talk about testing, but testing almost by definition has an opportunity cost associated with it. It does not help people achieve their goals in the short term because you're taking people out of the pool of people who can be impacted. So there's a kind of a negative impact that you will see through certain types of measurement. And so then the question for me is how do I balance that need to provide measurement because we know there's long-term impact value that we get from understanding things, but also balance that against the needs of a company who have to hit goals in a certain quarter in order to kind of achieve certain things.

Dominic Williamson is the director of marketing analytics at Chime. He's worked at eBay and Meta and a number of other great companies measuring marketing effectiveness. In today's episode, we had a great conversation. We touched on where does marketing analytics sit within an organization? When is measurement more disruptive than helpful? What are the marketing effects that are super difficult to measure and what can we do about it? How to think about brand measurement, weighted versus unweighted geographic lift tests, who should own the business forecast, and much more. Hope you all enjoy. Dom Williamson, thank you so much for joining me. It's great to have you here.

Thank you. It's great to be here.

So, Dom, you've run marketing science, marketing analytics type organizations, worked in them at a lot of great companies. Just to like level set, I'd love to hear from you. Like, what do you think about, or how do you conceptualize the role of marketing analytics, marketing science? Like, what does that mean to have that organization or to have that sort of sub-organization within a larger brand?

Yeah, and I think we could maybe come back to where it sits later because I think that's a question in and of itself. But in terms of what it should do, yeah, I think ultimately it's how you make the dollars work harder. And there's obviously a lot of ways that happens outside of marketing analytics. There's creative optimization. There's just things that people operate on day to day. But I think it for marketing science or marketing analytics, particularly, it's really about using the data to do that. And so there's, there's a couple of ways in which it's really obviously happening. One is how do you budget allocate? So you have various different channels, how do you make sure you're investing in the right ones and with the, with the right kind of share of spend. And in order to do that accurately, you need to know what each channel is doing for you. So you need to know the true incremental impact. Now, I know everyone will be really familiar with with what that means. But essentially, it's what happened because you spent money rather than what happened after or kind of alongside you spending money. So it's really teasing out that impact of spend. And I think that's fundamental to like, that's the fundamental base rock upon everything else upon which everything else needs to be built. And so there's the budget allocation piece. And then I think there's the within-channel optimization piece. So that's saying what signals do we have to optimize against? Can we send a signal back to our partners that gives us even more of an impact? And there's obviously targeting and other things that we could do on top of that as well. So the within-channel optimization piece is really interesting. There's lots of different strings to it and things that can be done to optimize. And then the budget allocation piece is great, but that also requires this holistic view of what's the true impact of all of these different channels.

Love it. So it's really like you sort of think about two buckets. It's like we have some pool of dollars. How do we allocate that across the different potential marketing channels? And then within each of those channels that we've decided to allocate to, How do we get the most bang for our buck using whatever tools are available within that, that channel? Is that a good summary?

Yeah. And they, and they are both essentially the same question, which is how do I get more from my dollars? But there's different paths in which you can do that.

Sort of different levels that you're making decisions at. And, and you teed this up at the beginning, which is like, where does, does marketing analytics sit, right? So like, how do you think about where this sits organizationally? And I think this relates to like this other question that I want to ask with like, who is sort of the boss of marketing analytics? Is it like, oh, the marketers ask for things from the marketing analytics team, and so the marketing analytics team is like building dashboards for marketers? Or is it like the marketing analytics team is holding the marketers to account, being like, you have to achieve this level ROI and we're the judges? So I guess, how do you think about that alignment of incentives problem and that organizational structure problem?

It's a great question. And I have to say, and like you mentioned, I've worked in various different companies. I've also worked in different structures, but for the same team, but in different structures. So it could be under finance and it could be under marketing and it could be under a third-party body like data. And so it helps the budget allocation question really requires a kind of an objectivity and that helps to be outside of the marketing org in order to take a fully objective viewpoint of where should I be spending my dollars? Because the truth is, that question goes wider than marketing. It goes to any lever where you could be spending and you could be investing and what's the spend and response that you could get back. And so it does help to kind of enforce that objectivity by placing this team in a separate reporting structure. The second part, though, the kind of within channel is very aligned. And so that often helps to be in a marketing team, because then you're really close to the day-to-day of what could we do to make this more effective. What are the models that we could create? What is targeting that we could do? What signals could we create to kind of optimize that? So that piece is very aligned. And so that means that I think the answer to your question is, as you might have expected, somewhat ambiguous. I think it depends on what you're trying to achieve as a company. I would say it wouldn't be a particularly healthy relationship if it was entirely, I am here to measure your, like, mark your homework is the term that often gets used. Like, I'm here to mark your homework. I'm here to see if you do a good job. I think as much as possible, the way I try to run a marketing analytics team is to be kind of in lockstep with the marketing team, but also the finance team and also the kind of broader strategic teams. And in order to do that, yes, it requires objectivity, but it also requires being part of that team, understanding what their needs are and kind of what their day-to-day looks like and how you can help with that. So it requires some degree of embedding into that team, even if it's not a formal reporting structure.

Can you give some like tactical tips? So everything you said makes perfect sense, but we're still left with like the ambiguity. And I think, you know, one of the ways that I conceptualize of this problem is like, in order to make change, you have to work with the marketing team. But in order to like make that universal across the business, you have the finance team still has to trust you. And so you have to achieve both of those goals. And I think that's in line sort of with what you were saying. But I'm curious if you have like tactical tips on like, oh, this works really well for achieving that or Ooh, like, you know, I've seen this go wrong if this happens, like, from your experience, anything along those lines that you can share?

Yeah, I think so. So one thing that helps with the relationship with marketing is to not get in the way. And so obviously, you need to do your role really well. But we'll, I'm sure, talk about testing. But testing almost by definition has an opportunity cost associated with it. It does not help people achieve their goals in the short term, because you're taking people out of the pool of people who can be impacted. And so there's, you a kind of a negative impact that you will see through certain types of measurement. And so then the question for me is, how do I make— how do I balance that need to provide measurement? Because we know there's long-term impact value that we get from understanding things, but also balance that against the needs of a company who have to hit goals in a certain quarter in order to kind of achieve certain things. So I think one of the questions is, how can you work with the marketing team to make sure that it's as non-disruptive as possible? There are certain types of measurement techniques that are less disruptive than others, and they may answer your question in a way that's sufficient. The other thing is like, are you going to do this again? Do you need to measure this in a way that's repeatable in the future, or is this just a one-off? If you're doing a product launch that will never happen again, is there a value in keeping a holdout when actually you want a more splashier impact? So those kind of questions, I think, really important to get aligned with the marketing team on. And then finance obviously have their own questions as well, and they come at things from a different viewpoint. But it's ultimately the same question, which is, where can I spend my money most effectively? And I think pre-aligning on all of those things with all of those different teams is the kind of the key is to say, we all have one common goal here. There's a bucket of money, we want it to be as impactful as possible. Here's all the options that we have to do that. What are the timelines? What are the restrictions? Do we need it to do X, Y, and Z? Like those things, if we lay them out beforehand and all agree to them, I think help with this kind of broader narrative.

So what are the types of measurement techniques that are less disruptive, right? That sort of keep you a little further out of the way of the marketers. And then every measurement technique comes with trade-offs. So like, what are those trade-offs and how do you think about navigating that?

Yeah, so I think, I think of, you know, a very— to go to one extreme, right? The most disruptive is to hold out half a country or to turn off an entire channel or something, something along those lines. And so that could be a user-level test that requires a 50% holdout. That's my cleanest read of what the true incremental impact, and that's something I really want to know. But is keeping a 50% control going to be more valuable than keeping a 10% control from an analytics point of view? Yes, but then maybe not enough to change the decisions that you would be making. So there's just like the percent of control that you might keep in a user-level test. There's geo tests, but geo tests often require restarting a campaign. They require changing something that, that people may believe is already working very effectively. So you have to balance that, that need. Um, I know that geo tests are great for their transparency, and they are very easy to explain to someone. I turn this place off and let's see what happens. Um, and so they really build trust really effectively. I really like geo tests for that reason, but I appreciate that they can have their, you know, impacts on performance. Um, and so geo tests I think can be disruptive too. The, the less disruptive end of things I would say is a media mix model or some regression where you're saying, let's see what happened and how does that impact our ability to predict these things. So it is a less disruptive way of measuring. It can be disruptive again when you say, well, I need to feed this model with some volatility, I need to pulse and I need to do these things. And that's not the way that I would necessarily be operating if measurement was not my goal. But I do think it's a less disruptive way of measuring some channels in particular. But then some channels do lend themselves very well to kind of platform-level, user-level holdouts, things like that, they're not very disruptive. So it does depend on the kind of functionality available per channel.

Do you have any tips for— it sounds like you've worked in a lot of organizations that are sort of fairly bought in to like measurement in general and incrementality broadly. But I speak with a lot of people who are not necessarily in that situation, who are still at the stage of needing to do some convincing or education. Especially on the marketing team side, that like this is really important and is important for the long-term health of the business. Do you have any tips for like how to have that conversation or particular arguments or illustrations that are particularly impactful or convincing?

I think laying out the path as to how this becomes valuable to them. So yes, it's often perceived as the the thing I think you described earlier where the thing I called marking your own homework, but this idea of am I just coming in and saying this is doing well and this isn't doing well. And naturally people spending money in these channels believe they are doing well.

That's why they do it.

They genuinely believe it. So you don't want to come in and say, I'm just going to test your beliefs and see if they're true or not. That's not the goal here. The reality is there are inefficiencies everywhere in any kind of investment anywhere. And the more clarity you can bring to that, The more transparency you can bring to any type of investment, the more valuable it is to you as an investor. And so if you think of it from that perspective of there will certainly be pockets of inefficiency here, no one is fully efficient. If we can find those pockets of inefficiency, then every dollar you spend from then on gets more effective. If you can lay it out in those terms and have people believe that, then I think that you are then all working toward a common goal. And I think people do inherently believe that no one believes that they are 100% efficient in everything that they do, or no one should believe that. And so it's really about spotting those gaps. I think it is very improbable that there aren't efficiencies to be had. And I think everyone in their own minds and hearts knows that. So maybe just saying, look, let's find those. It's not a bad thing if we find an inefficiency. It's a really positive thing because we're already doing this well, we're already doing X well. How can we get better than that? Well, we can only do it by changing, and we can only do it by finding the areas where we need to invest more and where we need to invest less.

I love it. And you talked a little bit about this idea of, you know, geographic experimentation, incrementality tests. I've seen you make some other posts on socials about this. I'm curious about like what, how you think about like what does good look like for this type of an incrementality testing program? And again, any sort of like highlights of like, these are the things that make a best-in-class program and like, or these are the things that like you really need to avoid or be careful about if you're trying to roll this out in your organization.

Yeah, I think, I think the word program is probably the important one. So it is, it's a process. It's, it's not just one-off tests. It's really, well, how can we test all of these different things? I think identifying where you need to test and when. So there are certain channels where you will learn far more by testing than others. And I think you can identify those fairly easily. You can say, well, if we're spending this much here, we really need to know if this is the signal, the signal that we have right now is effective for us. So I think it is about laying out a program and saying, this is what we'll do. We'll have, you know, let's say 4 quarters, we will test these channels here, these are our priorities, we will retest them if needed at these points, and just laying it out in that way and then identifying the questions that you need to answer. So one of the questions is, what's the true incremental value of of this channel. Another question is, well, what are my diminishing return curves and do they change over time? Another question is, what are the— there's more than one way to use a channel. What are the optimized ways of using this channel? If I try this program versus this program, how does it look? So, I think laying out those different questions first of all, and then saying, this is how we can execute against them. So, that's kind of a best-in-class view is to say, here's the whole suite of things that we'll do, here's how they're going to help us. You can, with every one of those tests, lay out the decision that would be associated committed to it before you get the results. And I think that's important. So you say, look, it's going to say one of X, one, I'm going to say X, it's going to say one of 3 things, right? It's going to say better, the same, worse. What are we going to do in each one of those situations? And it may be more of a spectrum than that, in fairness, it may be, well, it could be ROI positive up to this point, or it could be negative. But there are only so many things that this test is going to tell us, we can, we can agree in advance what decisions we'll make based off those results, rather than get to the results and then kind of equivocate and say, well, it could have been this, it could have been that. So I think that's how a great program would work. I think the negative aspects, one that I touched on earlier is just I don't want it to be too disruptive. I don't want to hamstring the marketing operations just so I get a signal of measurement because that's not great partnership either. So making it work within the framework of kind of ongoing marketing, I think is really helpful, making it as non-disruptive as possible. Is what will, A, have the best impact for the business, but B, have the best buy-in. Because if you can do it in a way that doesn't really negatively affect the day-to-day operations, then that's where you'll get the most agreement in terms of running these tests.

So, I love all of that, and I think there's some really good tips there. One of the things that I have found to be a blocker with these types of experiments in particular is that they're often not super high powered, um, which is to say that like you often get results that are either somewhat inconclusive or they have like really large uncertainty ranges associated with the results. Um, and I'm curious, like, how do, what do you do with that? Like, how do you communicate that to marketing stakeholders, to finance? Like, how do you actually talk about the things that you do learn, but also like the uncertainty? What do you do with that to actually keep driving the business forward?

Yeah, I think life is uncertain and ambiguous, and so it would be very unusual if these things weren't. One way, and I don't want to go too deep into the technical aspects of it because I probably won't explain them well, but I think a Bayesian interpretation helps because it does align with the way that people think. And if you're turning things into these probabilistic ranges, then you are saying, well, look, this does not need to be a binary yes/no, it was significant or not. There is a 28% chance that this is cost-effective to us, and there's a, you know, 72% chance that it's not, or some version of that calculation. I think if we start presenting things in those terms, it helps us make a more informed decision overall, because it's not a binary decision either. Whether or not we might want to take a risk depends on the kind of appetite of the company at that given point. Do you want to risk this? Do you want to test it out again? Or Is it really important that you have investments in a place where you're guaranteed returns? So, so how conservative are you planning to be? So I, I think that the company strategy, and maybe this kind of goes back to inform the testing program, the company strategy and, and the place that the company's at, at that point in time should inform the way that you test and interpret results.

One of the things that sometimes people raise when, when we're talking about this testing program and we're talking about uncertainty is, I hear that people are like, oh, you know, my executive team hates uncertainty. Like, I have to come in with an answer. If I bring in these ranges, they're going to be really frustrated with me. Have you heard that before? Have you experienced that? Or do you have advice to give to a person who like has that concern?

I think it's about the communication of it. We often hear the words directional, and directional can mean anything, right? It can just mean, well, you know, it was 0.01% higher, or it can mean, you know, that I, I feel like it's true. Um, I, I think that kind of language does, does push the idea that we just don't know the answer. Um, and, and often then maybe that's a, maybe that's a true reflection of where we are and we need to test differently. We don't really get a clear result. You usually have some view from, from any kind of, let's say, user-level or geo-level test of the ranges that you can be quite confident in. And you can say, well, look, somewhere between here and here. And here's the probabilities of it landing outside of those ranges. And I think that alone does help you make decisions. If you can, if you can provide it with that level of transparency, even if it is a wide range, you can say, well, it's like, it's going to be between here and here. This is profitable, this is not profitable. And this could be very profitable, this could be slightly negative. Like, what's the, what's the risk of making the wrong choice here, versus making the right choice here. So it goes beyond statistical significance and that type of reporting. And it ties back into your business metrics of, well, if I get this many new buyers, how much is that actually worth to me? And here's the cost. And here's the total possible negative cost. And like, just where's the risk and reward here? How would I make it like very much thinking about it as a broader investment decision, and the risk and reward? What's our appetite as a company? Here are the percentages, I think, I think providing all of those things does give some clarity. And I know that there's a frustration when it's a very wide range, And when I was a 50% chance that it might be worth it and maybe not, but then if that's the reality, I think that's what you have to work with at that point.

Yeah, I totally agree. And I think I've had this same experience where in my experience, at least executives are often very frustrated when, you know, someone else is bringing data and what to do about it is unclear, but acknowledging risk or acknowledging uncertainty in the data itself is totally fine. And so I think when I'm, when I'm talking is, and again, this is especially like sort of people maybe a little bit more junior in their like analytics career trajectory is I try to get them to split out, like we can be uncertain about the data and we could say like, look, the data isn't enough sufficient to like answer this question with 100% certainty. That's totally fine. But then your job is to separate that out from your communication of a recommendation, right? And you can take into account the uncertainty when it comes to actually making a decision or making a recommendation about what to do. Finance teams are really good about thinking about risk and understanding risk, right? That's a core part of finance training. And so you can always bring the idea of risk into these conversations and uncertainty into these conversations. They're going to get that, but you still have to, at the end of the day, make a decision. And that's the point where you have to like combine what the data say, which may be uncertain with like business strategy and other considerations. And that's what's actually yields the the decision. And so I think getting people to really like pull apart those ideas, I think, is really critical to actually being able to push the business forward.

Yeah. And I think, I think part of that is, is this quantification of what does it mean if, if this is successful? What does it mean if we get this much? Does that, does that open the door to a next level of something? And, and so there are, there are just business implications that go beyond what you will see in the test results that you want to incorporate in there as well. Sometimes it is a lot more straightforward. I just think sometimes it is a simple kind ROAS conversation. But sometimes there are bigger strategic things that build on top of it. And then like I say, if you're at a point at the company where saving money is much more important than growing, then the decision that you make will be different with exactly the same results. And so that needs to influence or kind of advise your decision-making.

Yeah, 100%. So there's, I at least believe that there's like a certain class of marketing impacts or marketing effects that are very difficult or probably impossible to measure with like the tools that we have available to us, especially something like Geolift, right? Um, and those sorts of effects are often, you know, these like very long-term potentially brand awareness type effects, right? How do you think about either measuring those or thinking about them or like building that into the marketing strategy, um, where you want to be rigorous but also like sort of have to acknowledge that there's some things that are very difficult or impossible measure, at least not very precisely.

Yeah, I, I don't think it's a hard take to say that brand impact and brand awareness are just difficult metrics to, to measure and to, to tie back to, to business metrics. I think to some— I shouldn't say business metrics, but like, uh, transactional metrics. Um, and I think, um, I think to some degree that is a strategic decision. When we say we want to drive awareness, we have a belief that the awareness is going through payback in a way that's not really measurable today, because this is a future state that we're talking about. And so that's a strategic decision now. So I think acknowledging that when it, when it is true, I do think it's not outside of the realms of possibility to tie brand metrics back to transactional metrics, and to build that full circle of I spend this much, I drive this much awareness, and then this, it does this much in the future. But I do think you need to have been in a steady state for quite a long time to get there. And you also need to have seen movement in brand awareness. So it's, for most companies, I think, outside of the realms of realistic measurability in a short period of time. I do think it's possible. And I think for those companies where it's outside because the company's growing very quickly, or because the company looks very different now to how it looked 2 years ago in terms of the way it operates, I think you do have to push that into a more strategic world where you say, well, this is the assumption that we're going to make, and are we all comfortable with it? I do think you have to articulate that. I think as a company, you have to say that in order to be able to then quantify decision-making around it. I do think you have to say, I believe that we'll do this at this point, rather than just say, I kind of vaguely believe that it's a positive thing that I will value.

Yeah, I love that. And I think it very much aligns with, with what I say and what I tell people. I think it's totally reasonable to invest a lot in brand and not expect a short-term return. but you have to sort of be deliberate about it and be thoughtful about it and say like, yes, this is part of our strategy. And then measure what you can, right? Like you might be able to say, because I see this on LinkedIn all the time where people are like, you have to invest in brand and then you're like, brand is good. And it's like, well, maybe, right? But like not every TV ad is good or builds, actually builds brand for you. And so when you say invest in brand, what do you actually mean? And how do you know if it's working? Or if you're doing the good brand versus the bad brand. And that's where a lot of like, you know, the people who post a lot on LinkedIn sort of like don't actually have great answers. And I think this is the thing that gets really hard where it's like, okay, we have to make a, we have a strategic belief that this is going to be good and we're going to invest in this for the business. We're not going to be able to measure it super precisely, but we're going to do these things, X, Y, and Z, to at least make sure that it's somewhat working at all. Right. And that might be, Look, we're going to measure the short-term impact. We think there's going to be more long-term impact, and that's okay, but we have to at least see some short-term impact, or we have to see some movement in like the Brand Lift study, or, you know, whatever. But we need to see something so that we at least have some confidence that we're in the good world and not the bad world.

I think so. And I do think it's about putting a stake in the ground and saying, yes, brand is a good investment. Potentially a good investment. But here's how we'll know, here's how we will define whether or not it's working for us. There needs to be that framework. I think speaking very pragmatically, and I'm not, you know, a brand survey expert, but I think you need to have a brand survey and you need to stick with the same vendor for a certain period of time to be able to see that tracking over time. Because every time you change a questionnaire, you change the methodology, you change your ability to do longitudinal tracking over it. So you do need that stability to really dig that out. One thing I will say though, and another tactical piece, often you don't see the movement in brand at the kind of longitudinal level because it doesn't move a huge amount day to day and your metrics might. So it's hard to then tie those two things together. But if you are able to get enough sample to look at these things at the geo level, it does often yield the kind of insights that you're looking for where you can say, is there a place where my cost of acquisition, or is there a relationship between my cost of acquisition in DR and my brand awareness are actually related to each other, which is entirely plausible. It often gets lost when you look at things over time because there are other things that influence your cost of acquisition. But if you look at it at the geo level, if you can, and then that does require a big sample size, obviously, because you might not have interviewed that many people in a geo X. But if you can do that, then do you think you can start to build out those relationships between the two?

That's a really interesting idea and very insightful. And so does that mean just like talking to your survey vendor and seeing if you can get the cuts by geo, or like, what does that look like?

Yeah.

Yes.

So can you get the cuts by geo? Probably yes for like 3 geos unless you have a big sample size. So do you need to sample more in order to do that? But yeah, I do think, I think for some of those kind of really interesting questions for people like you and I around, well, what does brand do in relation to other metrics? And it's always going to be very hard to tease out causality when these are quite separately correlated metrics. But I do think that there's something that can be yielded at the geo level that you'll never really see kind of over time.

Fascinating. Yeah, I think that's a really interesting idea, but again, going to be expensive at the levels of sampling we're talking about. But like, if it is a really important question, I think it makes a lot of sense.

I think it comes down to putting your kind of stake in the ground. It's, well, how much do you want to build around this? And if, you know, yes, it's expensive, but it's not as expensive as certain brand campaigns might be. So make your decision around that. But you may be comfortable outside of that with that strategic decision because even then, If you're a young company in terms of, you know, growth, if you're in a hypergrowth stage, for example, the thing, the value of brand in the future is very different to the brand value of brand today. And you can't know what that is yet without making big, big assumptions. And so, so you might just be comfortable saying, strategically, I'm going to make this decision that this is what it will be worth to me.

Yeah, totally. I mean, I, you know, and I say this a lot, like, at Recast, like, we do marketing, the vast majority of it is quote unquote, like brand marketing, we do almost no direct response. And I'm like, I think that that is the right strategic decision for us. And we just sort of have to accept that it's not really that measurable. And like, that's okay. Like, that's part of the strategic decision that we're making. We look at the metrics where we have them. And otherwise, we're making bets on like what we think is going to be impactful and what's the right thing for our business and our market position long term. And I think that that's like a totally reasonable decision to make.

Yeah.

Um, I, as we, when I was doing research for this conversation, I was listening to a podcast that you did recently. Um, and you talked about your experience with geographic experimentation at eBay and how that really influenced you. And you made this comment, um, about how, you know, all of those methods are part of the public domain. And today marketers have moved even beyond basic geolift tests. I sort of wanted to dig into that and understand like, what, what has that journey been like? What are the new methods? Like, how do you think about how this space has evolved?

I think measurement in general has moved into the public domain now. I think there's very little that's kind of behind closed doors. There's expertise obviously that sits with people who have the experience and all of that kind of stuff. But in terms of methodologies, you know, once upon a time, MMM would've been kind of very, not secretive, but there would be certain people who hold the formulas and some people who would then have to create them from scratch. And now that's changed. But with regards to GEO in particular, Yeah, so I think the main difference I would say is, I love the idea of having one geo or one set of geos and another geo or set of geos and just saying, look, I'm going to turn this off, and we can see what happens. Everyone in the company understands conceptually what that means, and what the value is if something changes. And so that's a really, really transparent, easy way to get everyone on board with both the methodology and then the results. I think the more nuanced version of that is where you reweight your, your geos. And so rather than just say, I'm going to compare this bunch to this bunch, half of the country, you might take a basket of geos and reweight them so that this one is worth like slightly more than this one so that it then matches better to the test market. Now, I know that that reduces your noise levels, but then also it's a couple more assumptions that you put on top and you fiddled around with something and it's a less transparent— it is transparent, but it feels like a less transparent view of—

it's more difficult to explain. Now there's another step where someone could put their thumb on the scale and— exactly, totally.

Exactly, there's more kind of that could be done to change the numbers if you're choosing to rewrite things rather than if you're just choosing to take the entire geo. So I think both are very valuable. I think the issue that, I'm sure you know this, I'm sure everyone knows this, but the issue with geos is obviously they're going to be noisier because there's only so many geos, there's not millions of them. And so that noise will often kind of get in the way of reading your signal. And so this rewriting type methodologies, they're all some version of that or they're often some version of that. Do help to reduce that noise, but also they take away some of the perceived transparency of the methodology too. So I prefer overall the reweighted version of the world because I think you do get past some of the bigger drawbacks of geo, which is just the amount of noise that you'll see with it.

Totally. Yeah. So I think what I really like about the way that you're talking about this is that you see really the value in the clarity, right? And part of what— part of the value of the, of the geo test is that it's very easy to explain. It's very easy to get like the whole company on board without having to like talk about all of these statistics. And I think this is a problem that I see a lot with like media mix models. It's like there's a million assumptions. Everyone, every single person in the company can pick one assumption that they disagree with and be like, oh, I don't trust it because there's this assumption. But, you know, a really simple geo experiment doesn't have that, right? Where it's like, okay, we can all really get aligned on, like, we all agree with these assumptions. It's very straightforward. And so we can actually have the conversation about what works and what doesn't without getting distracted by, by all of those other assumptions. And I really like that you're thoughtful about, look, as you make the method more complicated, it might get better statistically in some ways, but you're reducing that sort of the, the power in the conversation with the rest of the organization. And that is a thing that you need to be thoughtful about and be thinking about if you're a marketing analyst or marketing scientist in this world.

Yeah, I— well, one thing is that the reweighting— I mean, the, the important part of, of any test is the execution of the test. So do you actually hit these geos and not these geos? Um, and, and that, that is not always the case. I've found, um, that, uh, that certain channels are much better at identifying geos and hitting them effectively than others. Um, but, but that— once that's done Then you actually do have two versions of the, the analysis that you can do. You can have the kind of straight up unadjusted, we turned off New York and we turned on California, whatever that is version of it. But then you can have the reweighted version where you say, well, actually also if we want to reduce the noise, because the noise is the thing that we mentioned earlier, that's kind of holding you back from some decision making, or at least making it harder. And then here's, here's a way that we could do that. And I think, uh, it, it's, You know, it's, it's a good way to maybe take it a step further in terms of the analysis. But the execution is the hard bit and the execution is the important bit for the test. I love it.

And I think that's great advice. The unweighted versus weighted version, I think, is like a really clever way to sort of try to thread the needle and show like this is the impact of the additional assumptions. And again, just try to be really transparent and clear about it.

Yeah.

Pivoting topics a little bit, I want to— this is sort of the last section of things that I wanted to talk about. Who should own the business forecast? Should it be the marketing team? Should it be the marketing analytics team? Should it be the finance team? Like, how do you think about bringing together the plan or the budget on the marketing side with like the business outcomes on the other side?

Well, a very pertinent question this time of year as well. Yeah, I, I, it's, it's always going to be a combination of those three. I think ultimately who owns it is probably finance, because despite the recommendations from elsewhere, I think they tend to own those plans. So I think ultimately finance, I think they would always look for direction from the operating team who would say, well, look, there are very pragmatic things that this operating team knows that a finance team or even a marketing analytics team might not kind of put into their modeling when they're making assumptions. So for example, We are going to lose this relationship with this partner and we're not going to spend there anymore. Like just things like that, that you need to do from the bottoms up and say, look, spend and response will look like this. Affiliate programs, for example, you'll always have something there with that. That's just pragmatic management things that your model won't take into account. So I think it starts with a bottoms-up view. I do think that the marketing analytics team tends to take it from that point and say, well, what do we actually think can and will happen? Here's different spend and response curves that we might expect. Where should, where should we expect efficiency gains? Where should we pragmatically expect efficiency to get worse? Because efficiency as a rule tends to get worse over time just because inflation and reality and all of those things hit. And so, so kind of balancing all of those. And then I think ultimately it's a, it's a decision that finance will make.

And who, like, what's the right way to track that over time? Because it's like, what I see is a lot of people like You know, again, we're in Q4, so it's planning time and everybody like beats their head against the wall for a couple of months and then we lock in a budget and then like one month later it's like we're in a totally different world. Like, how do you actually like track that and make sure that the things are coming true or not over time? Like, what should that relationship look like to make sure that things are staying on track?

I think, I think there's always so many, I mean, this, this confuses me, um, my day to day, but there are various different counterfactuals that we're all holding in our heads. At the same time. There's, there's what do we want to happen, there's what do we think is going to happen, there's what did we plan to happen. Um, and so I, I think you have to pick a couple of those and be really kind of true to them. So I think the goal is your, is your first kind of North Star, which tends to be the most aggressive version. But what are the company goals? What are the things that we are aiming for? And they tend to be optimistic, and they tend to be more of a what do we want to happen than what is likely to happen. And, but I do think it has to be there as a kind of the North Star that you're tracking over time. Well, we would, for most of the places I've worked, you would have that as a monthly, let's say, or weekly metric that you're tracking against, say, see if you're likely to hit it or not. I think the other one is this kind of what might be called a kind of management plan, where we say, well, where are we tracking? Like, what do we think is going to happen realistically? It might not be a straight line from where we are today. It might reflect the fact that there are changes to come. But what do we think is most realistic given where we are right now? And then kind of track that against the actuals as they come in. So the management plan might be a moving target that keeps getting updated by reality as it comes along. But I do think maintaining both of those is important. And having a really clear articulation as to what each one of those is, because that is often ambiguous. And I think making that super clear to say, this is our goal, this is what we think is going to happen, this is what happened. And this, this kind of, you know, how do they align to each other?

Well, it's a lot of work, but thankfully we have marketing analytics teams, smart people like you, Dom, to work on it.

Yeah. Yeah. Well, I honestly, that's the easy part, right? I appreciate that the hard part is actually executing against all of these things and making all this stuff happen.

Well, Dom, this has been a great conversation. These were all the questions that I had for you today. Thank you so much for joining me. I think this was like super great conversation, super informative, and going to be super helpful for our listeners.

Great. Well, thank you very much. Appreciate it.