Building & Measuring Sustainable Growth Marketing Strategies | Greymatter

Hi everyone, I'm Mike DeBeau at Greylock Partners and welcome to our podcast Gray Matter, where we bring in some of today's top entrepreneurs and business leaders to share their stories from startup to scale up. Today we're talking with Tyler Elliston, former head of Performance Marketing Eventbrite and now CEO at Right sideup, and Kim Larson, VP Data at Keep Truckin. Prior to that, led data science functions at Stitch Fix. Their love in Uber. Great to have you here guys.

Great to be here.

So we've previously discussed product and growth fundamentals on the podcast, but the reality is paid marketing is playing an increasingly important role in today's startup environment. I'm excited to be talking to two leaders with breadth of experience, not only running these functions, but actually building differentiated capabilities around them. Tyler has a uniquely broad exposure to the growth marketing functions across a good number of startups, and Kim has worked on these problems from the vantage point of a career long data scientist. So there's a lot of ground we could cover from conceptual to tactical before jumping in. Just want to spend some time on your background. So Tyler, what are you doing at Right Sideup now and what led you there?

Yeah, Right Sideup is a collective of growth marketers deployed into high growth companies in very versatile configurations. We're helping companies develop their growth strategies and then fill gaps in their team to accelerate progress. So companies that we've worked with include Stitch Fix, Fitbit, Doordash, Instacart, Yelp, Calm, Rothy's, Wealthfront, and a variety of others. I started Right side Up because there's a lot of support companies can use that agencies aren't particularly well suited to provide.

Yep, Kim, we spent time together back at Stitch Fix. You were leading our client algorithms efforts. Part of what always fascinated me about Stitch Fix was how algos were applied in facets of the company that weren't necessarily obvious. Tell us a bit about that role and also your broader arc of your career.

Yeah. So at Stitch Fix, my team was responsible for demand forecasting and everything that had to do with marketing analytics, really, from analyzing the funnel to CRM campaigns, et cetera. I also had a small team that worked on identifying the right warehouse to ship a given package from. I don't know exactly where that team falls now at Stitchfix, but this is the kind of stuff my team did was not really what you'd hear about when you think about Stitch Fix. You're thinking more about the recommendation algorithm set up, but it's a good example of how Stitch Fix used data science across just any function whether it's inventory management or forecasting, et cetera. So that was Stitch Fix. Prior to that I actually started on my career before data science even was something you talked about. I worked at Charles Schwab for many years and did all sorts of analytics. I mean this was a time where a Fortune 500 company could have eight data scientists for the whole company and then got into a company called Market Share, later bought by NuStar. And that's where I really started learning about performance marketing and measurement and attribution, et cetera, and then went to Stitch Fix, which is just a great experience. Then I've spent some time at Uber and now I'm at Keep Trucking.

Great.

So just diving into this. I mean growth is something that means something quite a bit different. Company to company can be used to describe function across marketing, sales, analytics, product. Earlier on the podcast we had Brian Balfour and Sean Clowes talk about product led growth. Knowing your guys backgrounds, I think what would be fun to focus on today is more on the marketing side of things and the intersection of marketing and data. One of the areas that many companies, especially on the consumer side are facing and talking about is just this rising cost to acquire customers and decreasing returns to scale on marketing channels. It seems that given this, many companies are actually shifting away from doing paid marketing. Yet I know in talking with you guys you're believers in it to a certain extent. And so the question I would have is how do you think about building performance marketing as a sustainable differentiator and how might a company deal with this inevitability almost of rising CAC or customer acquisition cost?

So there's definitely risk and a lack of diversification because CACs can increase and can increase really quickly. But I think in a lot of ways you can mitigate that risk through diversification both within paid and outside of paid. And also focusing on things like conversion optimization, which can blunt some of the impact when something happens in an advertising platform. I don't know that we could broadly describe performance marketing as differentiated and sustainable, but certainly sophistication is a competitive advantage.

I agree with that. I have one comment on what you said Mike, about rising Kex Think there's two things going on here. There might be a global trend of rising cacs. And you hear this a lot. Facebook is getting more expensive. I I don't know if that's true or not, so I'm not going to like opine on that too much. But I think there's another thing going on here too Channels are increasingly more targeted. Right. Like you think about in the 1980s, you were doing TV advertising, and you would run TV ads, and they were extremely untargeted. And so it would take you a long time to find your early adopters and saturate those. But with the channels such as Facebook and their optimization algorithms, you get to those early adopters right away, which leads to very low cacs. But once you've kind of eaten through that audience, the kegs are going to rise. So I don't know how much it's whether performance marketing is more expensive or it's that the targeting capability is so much better, but that the early results you get aren't really indicative of what you should expect. So, case in point, maybe Facebook at scale should be just like TV in terms of effectiveness, once you've eaten through those early adopters. I think that's just something I've always thought about.

Yeah. I mean, I think certainly one of the macro trends we see and I've experienced in my career is that it's easier than ever to actually get these channels spun up and actually working pretty effectively. And with kind of the proliferation of just funding for these companies, it can be highly addicting to just start doing these and running them early and actually becoming quite dependent on it without even realizing it.

Right.

So how do you assess Tyler and companies you work with the role that Page should have within a broader mix? What does healthy versus unhealthy look like there?

Yeah, it's a really tough question when I think healthy versus unhealthy. Generally, because I'm a growth marketer, I say more is better unless it's harming the customer experience in some way. Now, granted, from a company standpoint, strategically, it can be unwise to be too focused on a particular channel and be, as you said, sort of addicted to a channel.

Yeah. I guess an extreme point is like, why is diversification important if it's optimal in the short term to put all your eggs in one basket, if that basket's really performing well? Why wouldn't a company do that?

The main reason is that it's not predictable. You don't know that it'll still be available a year from now. That's the biggest thing. So, like Cambridge Analytica, for example, really hurt the customer acquisition costs for a lot of companies because they lost a lot of their targeting ability. That was unforeseen. And, you know, for companies that were spending 80% of their marketing budget based on that targeting capability, that was then taken away, that is very harmful. To that company.

Yeah, I would assume that played some role in it. But you also have the dynamic of, you know, ad load on a channel like Facebook is just capped and you know, it's basic supply, demand kind of math. And so with advertisers rushing dollars in ad load being capped, Facebook trying to prioritize user experience, potentially over monetization, you just start to see rising cacs there. You would expect that at scale and I guess there's a question of whether all channels follow that trajectory. Kim, I know you've written your essay. The tyranny of the S curve, I think is probably relevant to dive into here.

Yeah. But build on Tyler's point. So I think it's inevitable that CACs are going to go up, you're going to saturate your early adopters and that next wave of potential are going to need a little bit more convincing and is going to be more expensive. I think it's simply physics. Now, if you know that's going to happen, my point is that you can't prevent that it's going to happen. You can reduce the extent to which it will happen to you. And the way you do that is through diversification. Now you could make the argument, well, why couldn't I just put all my eggs in the good basket and when disaster happens, then I'll diversify. But at that point you won't know how to diversify because you haven't built those muscles. It actually takes time to spin up all these kind of channels, even though it's easier and get good at it. Specifically offline and building a testing framework. And we'll talk about that later. So in short, the S curve is real. Unless you have some sort of crazy organic word of mouth effect that can outpace all the other negative effects from saturation, it's going to happen to you. And diversification early on is a way to not get caught when it happens. Yeah.

So this probably brings us to a broader point on attribution measurement, mix modeling, essentially that headline questioning how do you know what your ideal mix should be and what is the true impact of any given channel? I think as advertisers get started, in my experience there tends to be an over focus on last click attribution because it's easiest, it's out of box, it's actually simplest to understand payback at a channel level if you're assigning every user to one given channel. But over time, as you become more sophisticated, you misrepresents what's actually happening and so then you tend to see marketers going into fancier multi touch models which have their own kind of trade offs, potentially media mix models somewhere in there. And then what I've seen some of the more sophisticated advertisers doing is like incrementality tests on everything. And so let's spend some time talking about that. I think a lot of our listeners, I'm not sure how familiar they are with lift testing as a concept. So I guess, Kim, when you've seen measurement really done well within companies, like what does that look like?

That's a great question. I think first of all it's very easy to poke holes at last click attribution. Just because you click on an ad doesn't mean that that ad drove you to buy. And obviously it doesn't work for things you don't click on. I think at the very early stages of a company it's probably less wrong because you won't have as many people who would come without ads. However, what I've been preaching and we implemented more or less this framework at Uber, so I've been preaching sort of a triangular approach to measurement. So one is, yeah, have your last click, first click or you know, fusion of the two type of click attribution because it's easy, you can go very granular with it, you can track it over time. Just don't take the Y axis of those graphs too seriously because the level of those graphs are probably wrong, but the direction of them might actually be informative. Number two, you should absolutely do incrementality testing. And the reason being that I believe that the only way to figure out, let's say you have 100 people who come into a certain channel, the percentage of those that would have joined on their own. The only way to figure out what that percentage is is by running a test where you know, one group doesn't get any ads and the other group does and you compare the two. So that's what you got, click attribution, you got constant experimentation. The last part is some sort of top down model. And I still really believe in this. I know, typical. The popular name is media mix modeling. I think more of it as demand modeling. But at the end of the day what it is is a model that doesn't care about click attribution, doesn't care about tracking people through some cookie, etc. It looks at the world from a bird's eye view daily or weekly and it tries to establish a causal relationship between some KPI, call it acquisition and marketing levers, plus controlling for other things Such as seasonality, product changes, essentially using econometrics. And this is not something new. Right. Like people have been doing this to study the impact of economic policy, et cetera. It's kind of using that approach. If I'm studying the impact on a global economic policy, I can't do it with individual level attribution and the same thing here. So I think if you have those three together, you have a comprehensive framework to understand what's going on.

Yeah. And I think one way to think about it is also on timeframes. So in my experience, mix modeling, demand modeling, probably more valuable to do on a less frequent timeframe and last click attribution perhaps gets you some signal like relative week to week channel to channel how things are working and then incrementality testing somewhere in between. Is that consistent with your.

Absolutely. And then the incrementality test keeps it all real. Right. So when you fit these mix models or demand models, you're building some sort of time series model. It is more or less a black box. And let's say this model tells you that if 100k of Facebook ads will give you x amount of new acquisition over some baseline. And the question is, how do you really know that's right? Yeah. You can do a sort of holdout test where you hold out the last month of the data to see what the model. But that's not enough. There's too many things going on. An incrementality test is the perfect way to keep the model real to see if it's pulling out the right effects. So I think I completely agree. And then the last click will give you that very short term signal and you can also drill down to like the sub campaign level, which you couldn't do with a time series model.

Yeah, I mean I think for just to spend a little bit of time defining incrementality testing for the audience at least taking Facebook for instance, or as an example from our time at Stitch Fix together, you know, when you're spending at a certain scale, it's actually quite straightforward to do this stuff on Facebook. So just to dive into that example, essentially what a lift test would do there is Facebook would have determined what audience that you would have won in the auction. You're holding out that x percent of traffic and Facebook will go ahead and serve them the next ad in the auction and track them all the way through to see who ended up converting on the site anyway. And so the real lift that you have is control versus Expose, basically. And that's how you calculate an incremental cpa. I mean cost per acquisition. I think one common method is actually to calibrate your last click results to what a lift test like multiplier would be. But Tyler, I want to flip things to you. You're working with a number of companies that are probably earlier aren't spending at the scale that some of the advertisers that Kim or myself had had experience with. For companies that are getting started like, are they thinking about lift testing this early? And if not, should they be?

No, they're typically not. And honestly I would argue in the really early days, I think that's okay. A lot of early stage companies aren't familiar with the fact that Facebook is a one day view 28 day click, for example. And so even just looking at okay, how do we want to handle view through conversions is a big question. And interestingly, there are some companies that truly Facebook is the only way that anybody can find out about them. And so truly if they try to do a lift test, the view through window could possibly be 28 days and it could be 50, which is ludicrous to most performance marketers at most mid to late stage companies. So I think in the really early days it's about understanding view through conversions, click through conversions, getting some sort of alignment between finance, which is looking at the actual source of truth of conversions, and the marketing team which is looking at AdWords or Facebook. A lot of times those don't match up. There's often a point of maturation where there's this moment where they have to come together and say how do we reconcile these different numbers? Usually that's when they start operating in channels that have more view through conversions, when they're trying to make display work, when they're starting to spend more on Facebook and they've noted that the view throughs are driving 90% of their conversions. And so then there's this rationalization. So that's kind of the typical progression. And then once they're spinning, call it $100,000 a month or more, then they start doing some incrementality testing and then look at some of the more advanced models as they're spending in the millions.

Something that we're kind of touching upon is using self attribution from the networks. And as a general principle, I think it's best for companies to build their own measurement methods in house and use internal as the source of truth. Kim, I'm curious, even take Facebook's lift testing tool for instance and other platforms will have their own version of that when do and don't you trust self attribution?

I think of self attribution as two things. One is the lift test technology and another one is when they tell you these are the people who converted because of us. Facebook, which is the one that I have most experience with. Having talked to them quite a bit, I trust what they're doing. I think it seems neutral. And the folks there are legitimately trying to do the best they can do to come up with good experimental design and they have people that will work with you on the power analysis and that kind of stuff. So I feel fairly comfortable now in terms of attribution. If they tell you this customer joined because of us, well, they don't see what's going on on the other channels and it is what it is.

Cool. So I want to wrap up the chat on measurement in a big because I know we spent a lot of time on it, but one of the areas that's been fascinating to me that I think more people on the growth side have been spending time on is offline channels in general. And I think, you know, measurement is one of the difficulties things about running those channels. But I think there's a myth that TV actually is not measurable or it's only measurable for kind of brand lift and brand awareness studies, whereas the reality is you could get signal on it. Kim, I want to get your point of view on different methods actually to measure not only TV but offline as a starting point. What myself, a lot of other people that are more kind of on the growth marketing side of things will typically do is just burst analysis. So looking at lift and signups or traffic above what baseline should have been during that day of time of day, day of week, and get a window around when a TV spot airs and attribute that lift back to the spot that obviously ignores the tail effect of tv. It assumes that all the impact is happening during that window. But it can be useful in actually getting a relative measure of spot to spot, network to network, et cetera, but doesn't tell the whole picture about tv. I think general question for you Kim, is as you have more growth people starting to approach these offline channels and want to get the same insights and measurement out of it. How should they think about getting started with measuring specifically to TV and I guess audio channels?

That's a really interesting question. How much time do we have? First of all, if I can just make one comment, there's a perception that offline is absolutely impossible to measure and online is so easy to measure I agree with Tyler that early on in the company's life stage, online is probably pretty easy to measure because how else are you going to find this company? But as you mature, last click attribution or whatever, click based attribution gets more and more incorrect because many of those people would have joined anyway, brand search being the biggest offender here. So I don't actually think that digital marketing is easy to measure, maybe a little bit easier than offline, but I think there's a general misperception there. Now in terms of offline measurement, let's just take tv. There's four different ways I've seen this done, so I'm just going to step through one by one as fast as I can. Number one is what you mentioned, burst analysis, which is typically looking at TV impressions from one of these companies that track them and then correlating that on a very short term basis to spikes in website visits or something like that. Theory being that you don't need to control for a bunch of stuff because when you're looking at signal on such a short time frame, it's probably causal and not due to like seasonality or something like that or some other channel that magically came in in that very minute. And then of course you have to convert these website visits into some sort of real KPI and it can break down there a little bit. But I think it's a good way to look at things. The gold standard to me really is the market level testing, but that's a lot harder than people give it credit for because if you don't have enough markets and you don't spend enough, your test is going to be underpowered, meaning you're not going to have the sufficient accurate lens to measure the lift. At Uber we actually built like a whole algorithm to figure out how many markets do you need, which markets do you need to get the most precision? And I was always surprised at how many markets we needed to go into to measure lift.

It's also expensive. Right. So for TV in particular, I think you're paying like 5x the CPM buying regionally than you are nationally.

Absolutely. Three to five at least. I agree, but I think it is best practices. So that's that one. Then another one I think that's really intriguing is addressable TV here. The testing platforms are quite nascent at this point, but there you can go in and do a real A B test, a randomized test where some people get the ad, some people don't, and you can match that back to some sort of file to see what happens. That's really interesting. Last but not least, you have your good old top down model, for lack of better word, your media mix model. And again, what that is trying to do is say I'm not going to try to match things. I'm going to try to see controlling for seasonality and everything else you're doing and trend and whatnot. Do I see any movement in my KPI when TV is moving and can I even fit a curve to that to get at the diminishing returns? Now I wouldn't solely rely on that, but but I think that's another way. And I have actually gone at this problem from all four directions and often been pleasantly surprised at how consistent they were.

Tyler, one area that I know right side up has been investing in his podcast and that was an exciting channel for us early on as well. I think being somewhat early to running it at Stitch Fix, you see the gap between monetization and engagement there and actually the authenticity of host read and giving them the space to actually talk about the product is seen to be highly effective for the right businesses, but it's also probably the toughest channel in the mix to measure. How do you guys think about the efficacy of podcast ads and also just broader thoughts on podcast in general right now?

Yeah, podcast is really interesting right now because it is very effective for a lot of companies on a direct response basis using the best available techniques for measurement, which I'll talk about in a second. The burst analysis. Great for tv, really hard for podcast because consumption happens, you know, at some point after download and you don't know when it's consumed and it's usually over an extended period of time. So typically what we'll do in this case is we'll look at direct conversions either through a vanity URL or a promo code or both. And then we'll use a How did you hear about a survey in the post conversion funnel to see how many people cite podcast as how they heard about us. And so obviously there's a lot of science going in there, you know, making sure it's randomized, getting a control before you're running podcasts to see how many people set podcasts to then calculate a multiplier to say, okay, you know, for every direct conversion we saw come through on a promo code or vanity URL, we saw this many indirect conversions from the how did you hear about a survey? Certainly not perfect, but basically best available right now. And a lot of that methodology also applies to terrestrial radio. A lot of people listening in the Car. They're not at a computer. So also burst analysis is, I would argue, a little bit harder in that channel as well.

Cool.

So why don't I shift gears a bit? We spent a lot of time talking about measurement. One kind of thought that's been popular in growth circles these days is that increasingly media buying is commoditized. Whereas five years ago, seven years ago, you might have been able to find arbitrage by savvy media buying on emerging channels. Nowadays the art of making performance marketing work is really around measurement and great creative. Do you agree with that? Could you still find differentiation on savvy media buying? What's your point of view there?

I do disagree with that. I think that channels are becoming increasingly complex. And so while I certainly think that there's a risk of sort of the marginalization of the human component through AI, for example, the platforms are increasingly complex that really necessitate advanced skills. So we see a lot fewer people, for example, that are expert across Facebook and search and whatever other channel you want to add. More and more people are really specializing and need to specialize to be amazing. That doesn't mean that the impact of that specialization is huge. Like certainly the arbitrage opportunities that we saw 10 years ago or 15 years ago in search and five, eight years ago in Facebook, certainly those opportunities are not there, but there's still differentiation and advantage in specialization.

Are they there in emerging channels? So what are you guys seeing as far as companies really jumping on new channels and being first to really seize them?

I don't know that there are any, you know, at the scale of Facebook and Google, but we're increasingly seeing companies that are making Snapchat work really profitably. So, you know, first order payback, you know, ROI positive, especially for DTC ecom companies. When we first started hearing this, it's like, yeah, but how much are you really spending? And we're seeing companies spending 4 to $700,000 a month, which meets sort of a minimum threshold that a lot of other platforms I won't mention can't seem to get advertisers to that level. I think that's one YouTube, we're seeing more success in prospecting, which is exciting because obviously the inventory is vast, if that works. And then four true E comm companies, Amazon, not just listing optimization, but advertising, they're doing a lot of interesting things. And we're seeing companies really built on the back of, of unique opportunities in a particular category where they're crafting their product for it based on the opportunity in the auction.

For some of the earlier stage entrepreneurs listening, I think this is a common question is how do you balance the inherent scale of the usual suspect channels with the potential for greater near term efficiency in emerging channels? What frameworks do you have for helping your earlier stage clients? Think about this.

To Kim's earlier point about building a muscle of experimentation, I would strongly discourage early stage entrepreneurs from saying, well, I'm not interested in that channel because it'll only scale to $100,000. Right. I would say go and exploit that, build the muscle of experimentation. You might be surprised at how much depth there actually is when you get into it. Certainly larger stage companies where it just, it just doesn't move the needle. If you can't spend a million a month, that's a little bit of a different story. But for the early entrepreneurs, I would say look everywhere for good efficient conversions.

And you're also speaking to differences in budgeting processes. You will have some companies actually just index to a payback target and then spend up until that. You'll have others actually set budgets at the start of a year and even if they're wildly efficient, not go beyond that. There's some hybrids in between where maybe you have more fluid budgets, you know, setting week to week, month to month, et cetera. So you're still budget constrained, but, but that constraint is, you know, flexible over time. What pitfalls have you seen in clients approach budgeting and what do you think kind of best practice looks like here?

Yeah, I think a few of the pitfalls. One is treating ROAS as a goal.

Rather than roas's return on ad spend.

Yes, thank you. I think that's one pitfall. So what I mean by that is a lot of companies, they'll go raise a seed or series A based on a certain CAC and then they'll say, okay, what's next? We need to drop that by 20% and we need to increase volume by 300%. And that in my view is the wrong goal. I think it should be grow as much as you can within a ROAS constraint. So that's one. I think another pitfall is mismatching the ROAS guidance with the company stage. So we see a lot of companies who, you know, they say they have product market fit and they're in sort of a land grab mode. It's a race to be first in the market, but they have really tight ROAS constraints. Our argument in that case is, hey, if it's a land grab and you have product market fit, your conversion funnel is Tight. This is the time to really go hard. The last thing I would mention is guidance whiplash. So we see some companies where like on a weekly basis the acceptable CAC is changing and that makes it really hard, especially when you're in channels that have a 28 day window. I mean, how do you even predict what that cohort is going to look like in 28 days?

Kim, one concept I wanted to talk about with you, I mean you've been part of some pretty sophisticated orgs on this. And something that surprises me is that the concept of marginal CAC isn't talked about that often. And what I mean by that is the incremental impact of every marginal dollar spent. Looking at that versus just simply aggregating all spend across all volume probably tells different stories depending on where you are in your spend curve. How have you seen companies actually utilize that concept well in budgeting?

Great question. So I think the reason that marginal CAC isn't used widely is primarily because it's hard to estimate what the marginal CAC is. It's much easier just to say I spent this much on marketing and I got these many new customers, I can just divide the two and be done with it. But really what matters is the marginal cac. In fact, in a perfect world, if that world exists, what you would know for every single channel is a simple graph where the X axis is how much you spend and say the Y axis is your acquisition volume. And you get some sort of curve. And we know that curve is going to have diminishing returns. And if you know that curve, you would be in fantastic shape. Now is it easy to get that curve? No, but that's what you want to know. If you know that curve, you can take any point in that curve and you can see what is the price of acquiring one more customer, given where I'm spending right now. And that's your marginal cac. Now to get to that curve it requires a couple of things. You have to first of all have enough variation in your data to get different points on this curve. And then you have a stable enough environment that you can fit some sort of curve to this and learn it. Incrementality testing is great, that would give you one point or another point, but it may not be able to fill out the entire curve. So you probably again have to go back to some sort of modeling plus experimentation to approximate that curve. And one point that is just so important. When you draw that curve, you will notice that it doesn't intersect the Y axis at zero. Sometimes that's like a big mistake. You will have lots of acquisition if you're not spending on marketing. And that's another thing that's interesting to know.

Yep. So I want to shift gears a little bit to org construction which is tied to all this stuff. And I think one question I get asked a lot is in house versus agency. Listeners might hear this conversation and think, oh man, this is actually like pretty specialized stuff, a little bit intimidating to get started with. And I know there's different schools of thought whether you could throw generalists into this versus if you need specialists. But more broadly, you know, how do you think about for a company just getting started and knowing that there might be some inherent bias in Tyler here thinking about building an in house team off the bat versus leveraging kind of outside support and agency support to kind of get things running?

Yeah, I would argue that. Well, a few things. One, some companies performance marketing is very likely to become a core competency because paid is going to be central to their marketing strategy longer term. I think in those cases it's more valuable to have it in house from an earlier stage and more important that full time team members are really excellent at it. We often see companies start with really scrappy, smart generalists, you know, trying to launch channels and be successful with them getting 75% of the way there, going and raising money. And then there's this sort of moment of like, okay, wow, now we need to really take the sophistication up a notch and there's a shift from generalist to specialists. And that's often the inflection point where we see this decision being made, okay, do we want this specialization to live in house or do we want it to live with an agency? And it's very company specific. You know, there are some companies that we've worked with that, you know, they want to build in house, but their office is in a part of town that nobody wants to go to. And so they have a really hard time recruiting like that company, you know, may, may be better served working with an agency. And I would have to plug that. There is a middle ground between fully in house and fully agency. You know, that's what we're doing at right side up, trying to provide that middle ground.

I mean one of the arguments that being a fan of doing things in house, but also having worked with you, Tyler, I mean, I think one of the benefits of doing things in house is that you just have greater context. This stuff is integrated and so giving better context to your creative team and also having data Science being in the loop with stuff, it's just decreasing the cost of communication is important there. The other big piece is just accelerating your pace of learning. I mean I think one of the reasons you do pay it is actually to validate things faster than you would otherwise be able to organically and sense is that having those learnings live externally is not great. But I think a hybrid approach like you said, where yes, it's third party people kind of even try before you buy, but they are living in house, working with your team in house, I think that can be an effective hybrid there.

It's also a bit risky to go heavy on hiring a marketing team if you haven't validated yet that marketing will indeed play a huge role. To your point Tyler, some companies, you know that marketing is going to be a big factor, but you may not always know that. And so to go hire a big team only to find that it's not really working is not the best approach. And I agree with the learnings earlier.

Yes. One additional comment there. I think there's little doubt that if you can get the same expertise in a person in house versus that expertise through an agency and you know, you need at least an FTE full time employee internal, you know, to do that work, I don't think there's any doubt in house is better. I mean you run some risk in not having a backup plan when they give you two weeks notice, but you will get better results generally. I think the hybrid approach makes sense when you don't know that you need an fte, you know, maybe a half FTE or you're not sure about a particular channel, you want to test it before you hire a specialist. But generally we support an in house ethos.

So Kim, on org design from your end, you know you've overseen a number of different data science orgs, but you've also been at the intersection of all this stuff and data science. You also wrote a medium post on why data scientists can't yet replace human marketers. What's your point of view on the integration between data science and performance marketing? What does great look like there?

There's two components to this. There's how you work together and then there's the org structure itself. On the org structure itself, I have come to believe that data scientists should really be in an org that rolls up to something technical. I have just found that data scientists really prefer this and they can identify with this. I have been inside, I have, you know, belong to marketing organizations and I've been in data science Organizations and product organizations. But I think having data scientists being a technical team is preferable for the long term. Having said that, absolutely. And this is nothing sort of insightful, you should make sure that the data scientists are involved in your discussions earlier on. So the performance marketing people shouldn't come to them when they've figured everything out and just now need some sort of result. They should bring them in early because knowing the context, understanding the business spurs ideas and just generates better work on the post itself. I have sensed over the years that there's this notion, and particularly when multi touch attribution was a big topic, that what you really need is some single model that's just going to tell you what to do. And it is one monolithic model that has all the answers and you look at that one and it guides you as a performance marketer. But I just don't think that's realistic. In fact, I don't know if there's any industry where it's like that. Like if you're a doctor, there's not like one thing you look at. You probably have to look at 10 things and then make a decision on what to do. Same thing as a performance marketing. You might have incrementality test over here. You might have results from a top down demand time series model here and you might have some last click attribution stuff and you have to cook all this together. And then you might have some brand marketing that is very hard to measure because the impact is so slow and you have to cook all in together into a strategy. And for that reason I don't think, you know, machine learning can replace the human marketer. You need that person to cook all this together. Now it does take a slightly different skill set than perhaps 20 years ago. You have to have someone who is comfortable having all those conversations and know that My favorite quote, all models are wrong, but some are useful. It's always like that.

How would that change if you subbed in analytics for data science? I think something that I see some companies conflate is just applying the term data science very loosely and actually using it to describe things that are really just kind of basic analytics. Where is that line drawn for you?

Yeah, that's a fantastic question. I can tell you there are probably like multiple teams sitting right now debating what's a data scientist, what's a data analyst and how should we split them. The way I've set it up at Keep Trucking and other companies that data scientists really are folks who are getting into the more computer sciencey stuff or doing machine learning, etc. And they really want to do that. They're a little bit more like engineers. The projects have slightly longer time frames, et cetera. And analytics are people who could be ex consultants, they could come from all sorts of backgrounds who really enjoy being embedded every day with their business partners. And the analyses are more like a slide deck or dashboard and a set of recommendations. The funny thing is that the work these teams go through, they're very similar up to a certain point. They both have to pull data, they both have to understand the data, talk to people, to understand the context. But the deliverable for a data scientist is more of a product. The deliverable for an analyst is more of a recommendation. I think analysts are incredibly important and I think actually there's also a big room for democratizing data as well. And you don't always have to have people who have, you know, who are specialists in data analytics to look at graphs. You can publish some of that stuff in tableau dashboards and let people look at it themselves as well.

Yeah, that was a topic I wanted to spend some time on. And so I know one of the principles I've held in running growth teams is just really One of the KPIs is just velocity of experimentation. And I think one way to foster greater experimentation is just democratizing data, democratizing insights and using that to feed subsequent experiments and then just making those learnings available to everyone. There's obviously a fine line in that give people the superpowers of actually being able to test anything that could, that could really work against you and be, you know, I guess cause lack of focus in an organization. But what is that right balance there? I guess how do you think about designing an ideal experiment framework within companies? And Kim, how has your data science team played a role in that?

When it comes to performance marketing, which would be a very different experimentation framework than say pricing or something like that, when it comes to performance marketing, I think the goal is to continuously run experiments across different channels, across different regions and store all those results. So you always have that view of at least what is the best version of the truth, at least what we believe to be the truth. And that doesn't mean you have to run the same experiment every single day or every single week, but you have to just refresh it on a recurring basis and publish that and make sure everyone can see it. That would be on the experimentation side. In terms of democratizing data, you could argue that if you publish very intricate graphs based on complicated cohort views and it's very easy to misinterpret. You shouldn't just publish that widely. But that's just more common sense. Right? But anything that's fairly easy for most people to look at, I think that's no problem.

So as we're closing out, I think one more topic I wanted to cover was just how you think about acquisition versus retention. When growth was getting started, I think many teams focused explicitly on acquisition. Nowadays there's more of an emphasis on the same kind of capabilities and same types of folks working more on retention, moving increasingly down funnel and providing more leverage to what you're doing upstream. How do you think about the balance between acquisition and retention, specifically as it pertains to performance marketing?

So I would argue that the skill set of growth marketing applies really well to retention and engagement insofar as you're trying to grow lifetime value of a set of customers. And so I think it's a very similar skill set. Organizationally, I would argue that they should live separately because they're separate goals. One is to increase ltv, the other is to increase customers. So I think the application of that skill set is really valuable. But I would argue for structural separation.

I mean one of the just to push on that. I think one of the risks is that if you have a separate team gold on retention and a separate team gold on acquisition, both, I mean really the acquisition team is more likely to be able to game those metrics. And so you could send a bunch of low quality customers to the retention team and be done with your work. And the retention team's left struggling with that. Nowadays it seems like there's actually more ability to fine tune the types of customers that you actually acquire and optimizing towards quality upstream. Or also have a more integrated approach and having teams live together. How have you seen teams address that? Is that something that goaling acquisition teams more down funnel? I'll just say from my experience at Stitch Fix, we progressively move to that over time. Are you seeing that done in a lot of your clients, Tyler?

Definitely. I think looking at roas versus a CAC target accomplishes that if you start sending a lot of low LTV customers, then your CAC threshold is reduced. I think the shift to ROAS has definitely been real and definitely been positive. Though I would acknowledge that in this sort of structure, really any goal structure, it can be gamed by bad actors.

Well, just on the same topic and it shift gears to Kim for many businesses, Stitch Fix being an example, a lot of prediction actually goes into ltv. You don't always get a good sense of LTV from first transaction. And in the case of a business like Stitch Fix, you actually don't know first transaction until long after you've acquired them. How do you actually build a more sophisticated understanding of quality in an environment like that? How do you predict ltv? And also what are the trade offs between being like too predictive with that versus optimizing on some first transaction?

Yeah, when it comes to ltv, building LTV models is actually much harder than people give it credit. I think the best practice would be an LTV model that actually updates at multiple stages. So for example, at Uber we had it was an LTV model, but we had a model to predict whether a driver would ever start driving. And the way the model would work is that as soon as the driver went through the first steps of the funnel, a prediction was made. But as that driver moved progressively through the funnel, the model would update its prediction. You can take that same framework and move it to the LTV world. Like Stitch Fix, for example. You can make a not great but decent prediction off of what people put input in the survey originally. And then as you actually see transactions, you update that as you go along. So in that sense you can choose between optimizing for a number of acquired customers or you can optimize for number of acquired customers who filled out the survey in such a way that they looked like other great customers. And that's actually a better metric than just looking at number of customers. And then of course, you update that prediction as you go along and you can report on how your channels are doing. So I think that would be the ideal way to deal with ltv.

So I think the important concept there is that retention is actually a function of quality of customer you're acquiring to begin with.

Right.

Typically when you think about retention teams, in my experience, it's more kind of core product, product oriented teams working on that. Tyler, just knowing your experience across growth marketing teams, like what are examples of marketing focused teams working on retention? Maybe outside of your typical kind of email CRM stuff?

Well, honestly we don't see a lot out of the typical email CRM outreach stuff and frankly it's not particularly effective. And so I think having that be product centric is the right way to go. There are a lot of frustrated retention teams out there that are gold on a certain level that really they have very little ability to influence.

Right. I think another point around retention is I actually think that retention and acquisition, they're very much, although highly related, meaning the People you acquire will affect the LTV statistics down the line, but they're very different levers. Paid marketing is something you can flex up and down if you are short on demand. On the short term, you can run some ads. Same thing with incentives and that kind of thing. Retention is not just a lever. You can just pull and say, hey, let me just take my retention from X to Y. You can run a bunch of experiments, you can try a bunch of things, but you have absolutely no guarantees that retention will move at all. Right? So I would rather say that, like you should always have a team that's constantly trying to maximize retention and make the product experience and the funnel experience as good as possible. You should have a team that just obsesses over the funnel where they sit. I don't have strong opinion on, but you should have that. And then I view incentives and marketing as more as these short term levers you're using to grow your business while keeping the spend at a certain level so you're not spending more to acquire than your ltv. I always get a little nervous when I hear people say, hey, should we invest in retention or acquisition? I'm like, that's not apples to oranges. They're very different things.

All right, so we're up against time, guys. I think the last question I would ask, and this is for founders or operators at early stage startups that are thinking about starting to do this stuff and really want to get smarter. What resources or experiences would you suggest to people that are looking to go a little bit deeper on this?

I have stumbled into performance marketing throughout my career and so I never studied it formally as I can't mention any books or anything like that. I would identify companies that are very good at this stuff and I would reach out to people and bribe them with coffee and learn. I think that's the best way to do it. And there's this perception oftentimes that the sexy unicorns must be just extremely good at marketing. And then if you worked at this or this SF company that's hot, you're just the best marketing in the world. And maybe you are, maybe you're not, but there are a lot of very, very good marketing team in the older, more traditional company that have been at this for a long time. Back in 2003 or four, I need you to check my calendar when Talk to Chuck came out. And if you guys remember that those ads, the amount of rigor we used back then, this is a long time ago to measure the impact of this campaign before we Went national, I think is as good or better than what a lot of companies would do today. I don't even dating myself here. This is a long time ago. Right. And they are probably even better at this stuff. They've been at this for 20, 30 years. When I worked at market Share, I work at companies like Fidelity and they were very sophisticated because they had been at it for a long time. And people tend to hang around at these companies. So that is a place to go look sometimes. When it comes to performance marketing, rigor is not always your hot unicorn because guess what, those companies are young and they're still learning. Right. They really are just forming these teams.

Yeah. It's interesting how cyclical some of this is too. Like the original performance marketing channel was direct mail, right?

Yeah.

So you know, which is now kind of in vogue again. But Tyler, how about you, any suggestions you have?

Well, just to build on that a little bit, a lot of the later stage companies, the unicorn super successful companies, a lot of those are the least sophisticated because they've been built on the back of amazing products that just fundamentally grow. And that's one of the few ways that you can become a unicorn in short order is just a killer product. So I would actively discourage just, you know, being being drawn in by these amazing brands. There are a lot of great performance marketers in gaming. There are a lot of good performance marketing folks in affiliate. It can be a dangerous world, but there's a lot of good skill there. And then early stage companies that are trying to figure it out, a company that someone who's been the head of marketing at a company that has not found product market fit and they've tried everything, that person is a good resource. And I would echo that advice. Just go and talk to people and don't be afraid to try things. This is less applicable to founders, but oftentimes younger folks will ask me for advice to get into performance marketing and they always say, oh, it's a chicken egg. I can't get experience until I get a job and I can't get a job until I get experience. I say go run a Google Grant AdWords campaign. Google will give you $10,000 a month in free advertising for a nonprofit. Literally. Go find a nonprofit you support, set up a Google grant account, manage the $10,000, optimize it and then you can actually learn it through experience. Certainly for the earlier folks out there, just finding a way to do it, ideally with Google's money, that would be a great way to get going.

Awesome. Guys. This has been great. I feel like this podcast needs a disclaimer. If you're going to do this stuff, do it responsibly. And if you're going to do it at scale, do it in a sophisticated way.

And when it comes to forms marketing, prepare for the worst and hope for the best. Is that the way the phrase goes? Meaning your cost is going to go up. Don't assume it's just going to go down.

Got it. And for the last point for our listeners, where can they find you guys?

I mean, for me, just LinkedIn. I check my LinkedIn messages. Most of the stuff I've been talking about today you can find in my medium posts. That would probably be the best way to find me.

People can find me at LinkedIn or rightsideup.com thanks guys. Thank you.

Thanks listeners. If you have any feedback on topics discussed today or topics you'd like to hear in the future, just tweet us at greylockvc or mdebeaux. And thanks for subscribing to Gray Matter.