Why marketing measurement fails & how to fix | Jonathan Hershaff (ex-Stripe, Airbnb) | Coffee Breaks

Why do you think it is that marketing measurement in terms of like identifying real causal relationships is so hard?

There's a lot of really important reasons. A lot of the marketing spend that we do is on other platforms. First of all, these platforms have control of their users, right? And so we can't necessarily always get all the data that they're getting.

Jonathan Hershaw, Ph.D. is a data scientist and economist with over a decade of experience in cultural modeling across both government and tech sectors. His impressive career spans roles at the Federal Reserve, the sec, Wells Fargo, and tech companies like Stripe and Airbnb. Jonathan's research has been cited In a winning U.S. supreme Court case, and he has testified as an expert economic witness, teaching statistical concepts to a judge and jury for financial crimes. In today's episode, we had a fascinating conversation touching on causal inference versus model fitting. Very interesting branded search experiment at Stripe, how to really evaluate a media mixed model, and of course, experimentation. There's a whole lot more in the conversation and Jonathan bridges the gap between academic rigor and practical business application with remarkable clarity. I really hope you enjoy the conversation and we'll take it away. Jonathan, great to have you. Really excited to chat today. I want to start with maybe an easy question, maybe a hard question. What does causal inference actually mean? And what does it mean to you specifically?

Sure. So causal inference to me is usually trying to answer questions what's the impact of X on Y?

Right.

And it could be anything. It could be what is the impact of a marketing campaign on sales or visitors? It could be what is the impact of a feature launch? And what makes causal inference a special field is that there's usually something like called confounding variables that make simple comparisons misleading. So one of the easy examples might be, you know, what is the impact of Amazon prime on purchases? Right. And you can't just compare the purchases of people who have Amazon prime to people who don't have Amazon prime, because the people who get Amazon prime are exactly those people who have selected into it. They're people who are probably more likely to buy anyway. And so if you were just compare the people who select into a program versus don't, you're going to end up with biased results. And so causal inference is, you know, how do you get back to finding what is the role of some feature or some new campaign on actual incremental KPIs got it.

Great. So it's about really, I mean it's about understanding caudality is sort of the high level. But it's hard because there's often other effects that we have to deal with, whether they're collection effects or other types of biases.

Yeah, absolutely. And typically we have as the sort of the gold standard to this and people use the gold standard all the time. I forget where it came from, but is experimentation. And so when you hear the words causal inference, say on LinkedIn and job postings, there is sometimes this differentiation between experimentation, which is like the gold in causality and incrementality. But what is harder is the observational causal inference. So how do you mimic what would be happening in an experiment when for some reason or another, and there are plenty, you cannot do an experiment? And so when you hear causal inference, a lot of times people are specifically talking about observational causal inference, but globally.

It would also include experimentation that makes permanent sense, aligns with my understanding. Speaking of LinkedIn, see a lot of talk in LinkedIn about AI and sometimes people are talking about, you know, automating intelligence. How does AI fit in with causal inference? To what extent are these fields coming together or drifting apart? How do you think about that?

To me, AI right now has been, I almost treated as, as, as an intern, in some ways I have posed causal inference related questions to various GPTs and it does an okay job. It might eventually get there. But I think one of the things that right now is making our field a little bit safer from just being replaced by AI is that we really need to understand, like what is happening in the business, institutional details. So for example, if something comes out and there is a feature that affects a group of people, and we want to know what the impact of that is at this moment, we can't ask AI. AI might give us some suggestions, but we really have to think about who was the population affected? How was this treatment implemented? Were there any details of the implementation that could help us identify what could be a reasonable treatment and a reasonable control group? So AI to some extent, if it was fed the entire history of all the docs in the company, it could eventually start to identify groups that are reasonable comparison groups. But right now it takes like interviews with stakeholders. Hey, when you did this, what made you choose this group? And right now I think AI is great for coding things up, for testing analyses, for improving the speed. But a lot right now is still research when it comes to causal inference. And until the AI is fed the history of the company, we're still in pretty good shape.

Switching tack a little bit, you've been doing some work in marketing analytics, marketing measurement especially over the last couple of years has really been talking a lot about incrementality. And so incrementality, I think, really just means causality.

Right.

It's the word that marketers use to describe the idea of the causal relationship between marketing activity on one side and a business KPI on the other. Why do you think it is that marketing measurement in terms of, like, identifying real causal relationships is so hard?

Yeah, I think there's, like, really two really important reasons, or I say there's a lot of really important reasons. So the first is, what are you trying to measure? And so a lot of the marketing spend that we do is on other platforms, first of all. And some of these platforms, we cannot just go in there and run experiments on Google customers or on Hulu's customers. Like, these platforms have control of their users. Right. And so we can't necessarily always get all the data that they're getting. Then there's an example of some of the things that make marketing really tough is that there's not necessarily always a digital footprint. Right. So if there is a video that comes on YouTube, we might watch it, or an ad that goes through the screen, we might see it, but we might not necessarily click it. And there might not be a digital footprint. Then you start adding, even if we did click it, there might be issues with cookie tracking. Right. Or people who switch devices. So I see an ad on my phone when I'm walking around, I remember it later, I go back to my laptop and book, and the cookies might be different. And so me on the receiving the ad could be in the treatment group. And then me on my laptop might be in the control group, and that could mess up the estimates. And then you have. The other fact, and we start getting into, like, the marketing mix model is that we have so much advertising going on at the same time. So you have multiple channels, people seeing an ad on Instagram, they're seeing an ad in the movie theater, they're seeing an ad on the. The radio and on a billboard when you're walking by. And so how do you appropriately attribute one booking to any number of advertising paths that somebody might have. Might have been exposed to?

And I think even beyond that, there's also the problem of, like, targeting. And you mentioned selection effects when we were introducing Kali imprints at the top. And this can be a real problem. There's selection effects with marketing channels like Branded Search, where I'm selecting myself into searching for some particular brand. And just because I saw an ad or clicked on an ad doesn't necessarily mean that that ad caused the purchase. And similarly on some of the really large walled gardens, you know, meta, Facebook is very, very good at targeting people. It's very good at targeting people who are likely to purchase. And therefore it's very difficult to know, okay, are these ads causing incremental purchases or are we just cream skimming by targeting the people who are likely to purchase anyway? And so those, in addition to everything that you talked about, I think that that like selection effect problem also just makes it really, really difficult to start to think about how do we even construct a bump for scenario or like a control group to compare to when we don't have all of the information. And there's all of these like other mediating effects surrounding this decision making process.

The incrementality of branded search that you brought up right there always comes back to, to me, this, this famous ebay study I think was in 2015 where some economists had found that I think they were spending like $10 million on branded search ads and they found that 99% of it was just cannibalizing people who would have clicked and would have purchased anyway on the organic links. And that in a way I feel like almost started this, like this entire field of economics and tech because it was so mind blowing how much money in marketing could be wasted and that you can, you can save a lot and be more optimized if you can do some of these incrementality tests.

And just for our listeners to know, like, that ebay study is very famous, but there's been another of other slightly less famous studies that have come later with different brands that are, you know, repeating this, trying to measure the incrementality of branded search. And the results actually vary a lot. Like eBay is a very interesting specific example of like everyone knows what ebay is. It's a verb. Like there's good reason to believe that if you're searching for like ebay and then like product name, you know, you know what you're going to, you're trying to navigate there, not actually search, but there's a lot of other brands that are potentially more competitive where you know, actually the branded search ad spin actually is incremental for them. And so this is one of those things where it's not, there's not just like an obvious answer where it's like branded search is always bad. It's like we really don't know. It's going to vary. It might change over time. It will definitely be different for different brands for different products. And so this is what like Causal inference and incrementality is all about is we have to figure out some ways of going out and learning this about the world in order to make good decisions.

And that's actually a great point. One of my, I guess, biggest contributions when I was at Stripe was working on this branded experiment and branded search experiment. And I think I will speak directly for the co founders, but my impression is that it had come from very up high, was that they were just planning on eliminating the branded search channel. And I had done this work before at this, the first startup that I, that I was at after I transitioned out of government and I proposed that we do a branded search ad experiment at Stripe. And the expectations, I mean, they had already booked, you know, that they were going to save this money and reallocate it to other channels. And we ran the experiment that, and what we found was that when you considered the cost of branded search ads, because our Google quality scores were so high, and you consider the fact that there were adversarial bidders against our branded ads, we ended up having something where our acquisition cost through that channel or cost of an incremental booking through the branded search channel was very comparable to the non branded channel. And we ended up saving that through experimentation. I believe I had seen a paper where some folks did a lot of. They were working one of these big platforms and they basically said if there's no competitive adversarial bids against your branded ads that you're probably almost all cannibalizing. But once people start bidding against your branded ads, you might need to save it.

That makes perfect sense. It aligns with like, first of all our, my personal experience of defending our like recast branded terms and being against competitors, like, you know, you really have to think it through. Like there is, there's real dynamics at play in these networks and something that might be true today might not be true six months from now, depending on your actions, the actions of your competitors, how the market is like all of these things can change and have an impact.

One of the things that I actually really like doing as I'm getting into these projects, is trying to think about it from the actual customer's point of view. And so when I was putting together that project for Stripe, and I've done it similarly with Airbnb, I was typing in just into Google Stripe payment processing, right? Just something that would be a branded search. And at the time there were ads that came up, why, you know, payment B or something. So whatever, whatever this competitor was is better than Stripe, right? And so you can imagine we had our branded ad at the top and then there were some other branded ads that Y X company is better than Stripe for payment processing, and then a few more ads down. Finally the organic listings start. And so if you're thinking about from the customer point of view and you type in Stripe payment processing and all of a sudden there's an ad right at the top, if Stripe's not there, that says Y X company is better than Stripe, you might click it. And it's not hard to then see how you could have incremental bookings by bidding on those branded keywords 100%.

You been publishing a lot on, on LinkedIn lately. Your, your LinkedIn account is incredible. Everyone should go check it out. But the reason why I reached out to you is because you have started publishing some really, really great work on marketing mixed models. Can you tell us about this work that you've been doing on marketing mix modeling? You know how like what caused you to want to get into it and then what has been your approach for looking into this class of models, starting to evaluate them?

Marketing mix models to me are, I mean, must be one of the hardest problems across certainly marketing data science. They are, from what I've seen, essentially the holy grail for marketers. I mean they promise to estimate the efficiency, the ROIs of multiple channels all at once. And again, if you, you can imagine if you're a big company in Apple, a Samsung and Airbnb, you were, you might be advertising across 15 different channels and to run experiments in each one just takes an unbelievable amount of time. Just, just not really reasonable to, to be able to make decisions. And so one of the things that I've noticed is that these marketing mix models are highly in demand, they are just highly sought after. And it's a complicated problem. And you might have big companies where getting this right, getting the rank ordering of ROI and efficiency across media channels, your work could influence tens or even hundreds of millions of dollars if you're working for a big company. And so the impact is huge if you're in this space. What I found, and I didn't start by building models, I came into Airbnb and I was evaluating both our internal model and also a proposed third party agency that was coming in that specializes in marketing mixed models. And so I got to go through some of the code that we had and some of the code also that were the technical papers of this third party agency. And one of the things that I just noticed was that you had mentioned the endogeneity before, like the endogeneity of spend. There's also issues with multicollinearity across these channels and these marketing mix models, at least so far I just haven't really seen them solve the problem. So in an experiment you can really deal with endogeneity, of course, because you have a treatment and you have a control group. But I, I just have found this to be an incredibly interesting space because the challenges are so difficult. So I've started with building some of my own models, really just off from scratch and noticing little things. And this is what I've been posting about. What happens when we start with a marketing mix model that has just one media channel? Does this method reasonably get back the results? Like there's a reason we get back the roi? What happens when we add a second media channel? But there's.

Yeah, yeah, I do want to pause you. You said get back to the roi, which I think is incredibly important to your process. But not everyone listening might understand the full implications of that. How do you, how are you actually testing this? What does it mean to get back the roi?

Oh yeah, great point. So in a marketing mix model, a lot of times we don't actually have the ground truth, right? So we don't know how many bookings or how much revenue this one channel actually had. We just know what our total bookings are and we don't know what the true allocation is. And so to help test the model, I simulate data. So I create my own data, I set my own parameters. I create these common functions and ad stock, a hill function, things that you would see in these technical papers. And I generate data so that I know exactly what the roas would be for each channel. I then feed that data into whichever model I choose. It could be an internal model that I have built myself on my own time, not the, not the company internal model. It could be feeding it into an open source model such as Google Meridian. And I can test how well it's just picking up what I know as the ground truth ROI or the ground truth ROAs, because I've coded it myself.

Okay, amazing. That makes perfect sense. And then you were talking about some of the different experiments that you had been running where you simulate data in some certain way and then test it. And so I'll let you keep going with that.

So I know that some of the hardcoms is dealing with the multicollinearity, for example, or the endogeneity of the spending where spend where you might spend. More specifically in cases where there is more demand. So sometimes I will simulate data that has like a baseline demand, a seasonality, and then I'll have it where spend is tracking alongside that seasonality. And this is something that you can imagine marketers are doing if you're working for an E commerce. So you know, there's a ton of buyers out there during the holiday season. So you ramp up the ad spending, you ramp up the ad, any marketer.

And they will tell you they do that. It's not a secret. They will tell you that this is part of the plan.

Exactly. And it makes sense, right? You know, this is when the buyers are out there. But now imagine this when if you're looking at the data, if you were just naively looking at this data, you see higher advertising spend and you also see higher sales. But of course you have this omitted variable buys that there's, you know, increased demand out there in the market because it's the holiday season. And so I would start simulating these things in the code that there is endogeneity or that what happens when we add a second media channel. But now these two media channels are correlated, so we have multicollinearity in the data. How well do the models actually pick these things up? I love this approach.

And this is, this is something that.

We do a lot at Recast.

We call this process a parameter recovery exercise, which is to say you simulate data with some known parameter and then you push it through a model and you try to recover the parameters on the other side. You check how well the model does. And this is like what you're describing is what we recommend to anyone evaluating any type of media mix model. Because as you note, the core problem with media mix modeling is that the truth that we care about the true incrementality or the true causality of all of our different marketing channels is unknown and unknowable. We cannot, there's nowhere that we can go look up what the answer is and say, say like, yes, the model got it right or it didn't right with real data, with the real data that we're using for our business, we just can't know that. And so parameter recovery helps us understand where a model works well and where it doesn't under ideally different simulations. And so what I tell people is when you're evaluating one of these models, you should always try to do it in such a way so that it breaks right. You should be trying to find where does this model break down? And if you can't find it, you've screwed up Right. Because every model has limitations. Every model will break down under some set of assumptions. And so your job when you're actually evaluating these things needs to be how can I simulate data in such a way that a total that it breaks the model in a really meaningful way and then that lets you start to reason about well, is that important or not? Right. Does that matter or not? And that is, I think when you're dealing with these complex observational causal inference models that there's always going to be a lot of assumptions. The model results are generally going to be pretty sensitive to at least some of those assumptions. And so you need to figure out what are those load bearing assumptions so that then you can make a decision does that matter or not for this business? Like yes, you know, I know that we don't have Endogen in our spin, so I'm not worried about it. I don't, I would be skeptical of someone saying that but like, fine, right. At least you are honest about it. But just I see people do this other people on LinkedIn who are not, I think as good as you are, Jonathan. Like they're like, yeah, we tested this model, we ran through the data and we looked at the results and they looked reasonable and so we know it's good. And I'm like that is not. You can't evaluate these sorts of observational models that way.

Thought the, the last post I shared was a really interesting result that surprised even me. And, and actually this is something that I found interesting about posting on LinkedIn recently is that I'm do, I'm not actually just doing this purely for like content creation. I'm learning while doing and I' so because these models like mmm, they're so hard to evaluate because we don't know the ground truth. I have seen a lot of people that will use things like a mate or an MAE or an RMSE to evaluate and say this model with these parameters are better than another model with another parameters. And so I think what you have are a couple of things going on at the same time. One is the results are can be extremely sensitive to your choice of priors. I mean I've seen results that just all over the place depending on how you choose your priors and then you need to choose between them and if you use an evaluation such as like a MAPE or an mae, you might think, well this version of the model is doing better at least at predicting total bookings and all else being equal. You would probably, you would think I'd rather have a model that is at least more accurate picking total bookings than not. Right. All else equal. But what I found in my latest simulation I was testing uninformed versus informed priors. And in the first model where I had uninformed priors, it completely failed at the main business question, which was can it rank order media channels by true roi? And again I'm using simulated data. So I know Media channel one had an ROI of X, Media channel two had an ROI Y Y was greater than X. We wanted to rank order it in that way. So media channel 2 looks better than media channel 1. But what happened was the model that actually had the better mate, the better mae, the better rmse actually did a worse job at rank ordering the two channels. And so it was this effect where you might choose a model that has a better evaluation metric and gives you the wrong business decision. And again, I think that's one of these just really challenging parts of MMM that I'm still learning and trying to solve. But at least I feel like I'm asking questions. And one other thing just sort of the, the one other thing I wanted to add to this, a big reason that I've been posting, talking about mmm I've been posting stuff on some of these causal inference packages such as Causal Impact is that when people are first getting started and they're using these models, they give an answer and the answer can seem so confident. And if you almost think of the way like when you first started using ChatGPT and it would just give these wild answers that were completely wrong but it just sounds so confident like in the answer. And my concern was that you get a lot of people who are first using this and it says wow, the marketing mix model says this is, this is the result, this is the rank ordering, this is the ROI or something like a causal impact says this is the incremental effect of whatever it is that the treatment and here's this low P value and the results aren't necessarily always accurate. And so it's worth you want to kick the tires a little bit more on.

You're hitting on just so many interesting points and like really important things for practitioners to be thinking about. And I want to comment on some of these. So first, you know you're talking about this exercise where the model with the higher in sample fit metrics of Ruby squared error mean average mean absolute percent error is made if I got it right, mean absolute error. I always forget which the which one.

Is absolute error and mean absolute percent.

There you go. That, that model right. It fits really well, to the data in sample, but it actually doesn't get the causal inference correct. This is critically important. And again, it's really important because what we're doing right, is causal inference, not just model fitting. And there are contexts in which looking at goodness of fit for models like makes sense and is a fair way to evaluate those models. But in the context of causal inference, it almost never is. With causal inference, what we need to be thinking hard about is what is the parameter of interest that we are trying to estimate and what is the best way to estimate that parameter of interest. And so often things like, you know, making the model more flexible, right. Will increase your in sample fit metrics, but will tend to overfit and give you worse inferences. Adding more variables, right? A lot of, a lot of people have this intuition that, oh, we just need to add more variables to the model, we should add as many variables as we possibly can to the model. But again, actually has often has the opposite effect. You add more variables, you get better fit. Right, because the model has more like levers to pull to fit to the in sample data at least. But you get worse causal inferences for the parameters of interest. And so really, really it's a very different way of thinking about the world. And I think a lot of this is that or a lot of what's going on is that a lot of data scientists especially recently have come up through this world of like machine learning and prediction, right, where that really is what you're focused on. Like it's a black box model. Throw as much data into it as you can, let the algorithm do the thing, automate model selection by looking at the right set of metrics and you know, if you're doing sort of engineering work around that, whereas that is so different from the world of causal inference and econometrics that I think a lot of people, you know, data scientists are like, yes, I've, you know, I've been building these models I can go move into. Mmm. And I'll just take my same toolbox, I'll run the data through and I'll pick the model that has the highest R squared. But unfortunately that like fails. And so I sort of, you know, I'm wondering if you see that same pattern or if you have other like things that data scientists should be looking out for as they are making that transition into doing causal inference or working with media mixed models when maybe they don't have like a formal econometric stream.

Yeah, I think that's actually a really interesting point and something that I have been probably promoting a little bit more on TikTok. I have a lot of followers who are coming out of data science boot camps, who are coming out of a master's or an undergrad in data science. And they are telling me, John, you're sitting here talking about causal inference problems. We are not learning this in school, we are not learning this in, in our programs. And in a way, I mean, I guess it makes sense because these programs are giving people what they want. A lot of people want to work on. And the fanciest machine learning, you know, it was the deep learning maybe a couple of years ago. Now I'm sure they're all interested in anything that's related to LLMs and AI and the causal inference, at least historically, from what I've seen, had one, been in the realm of PhD economists and two, actually in some of the masters in public policy. And so I actually feel in a lot of ways that I got the very lucky. I went through an econ PhD, but my focus was on public policy. And these things fit so well because the government a lot of times cannot randomize, right? Like we, we can't, we can't, we're not going to randomize what student loan interest rates are, right? We're not going to randomize, you know, what the minimum wage is going to be or, you know, or if you're in a state, you're not going to randomize. You get a minimum wage of this and you get a minimum wage of that. And so by nature, a lot of the public policy things that we study, that we study as economists or maybe some of the MPP students study as well, necessarily need this observational caudal inference. And so we just have been trained to think, wow, we wish we could run an experiment, but we can't. What are ways that we can get a valid control group out of this? And I think when the people are coming through the DS programs that program evaluation and policy evaluation is a lesser part of it. Like you said, it's going to be a lot more of the prediction. And so when people are trying to get into this, I think the best thing they can do is first get a strong foundation in experimentation. Do you understand why we need experimentation? Like how does it solve some of these things, such as confounding variables or selection bias that we talked about earlier? Can you do experiment design? Because while we do observational causal inference, sometimes when things happen in the past. So hey, we put out a new feature. We want to know what the impact of this is, or, you know, systems went down, we had an outage for our, you know, our API. We want to see how this affected our customers. There are times that we can design non experimental causal inference in advance as well. And so it helps to have that foundation because to me, causal inference is attempting to do an experiment, but for some reason we cannot. And so you can't break the rules unless you know them really well. And so for people who are from that machine learning background, they're coming out of these DS programs. To me, step one is understanding deeply ab testing, analyzing and designing experiments. And from there you can start to build into some of these causalities.

Yeah, I think that's totally right. The one aspect that I would add is I definitely consider experimentation to be a subset of. And experiments are amazing ways of doing causal inference, but they're not perfect and they have limitations almost always. And it's very important, especially when you're doing this, you know, with millions of dollars of budget on the line to actually like really understand those limitations. What are the, with this experiment that I just ran, what are the ways that it can go wrong? Right? And having that like good intuition for that I think is super important. And it's not just, oh yeah, experiments are perfect and they give us perfect causal inference every time. It's. In many cases you can get like really good causal inference as long as certain assumptions are met. But then you need to understand, like, is this actually random? If it weren't actually random, like, how much does that impact those results? Those sorts of questions that you can start asking yourselves. And like the toolkit that that gives you, I think, I think is really powerful. There's obviously like amazing software that you can build, like just doing prediction and using machine learning. But from like a business impact perspective, being able to design a really good experiment and interpret it correctly, you can have just as large of an impact. And I think that that's sadly a little bit overlooked, I think in a lot of these, like now undergraduate programs or boot camps where they just don't put as much emphasis.

Yeah, absolutely. I mean, experiments are one of these things where they seem so easy in the classroom, right? You have unlimited users and you just randomize them into treatment and control and you do a T test. But that just, it's just not feasible in the real world most of the time. Maybe if you're doing like email marketing, right? Or if you're, you know, if you're Google or, you know, you, you're getting a ton of traffic, you can say, oh, we're going to change the font for app of our users and you're getting millions of people come in every, every day. But in so many cases there is some reason that you cannot do an experiment either. You know, you want to, you want to test something that is insurance, right? Against like a really bad outcome. Like, like an outage that happens. Well, you don't want to just, you don't want to just apply an outage, you know, randomly to your customers and get half of your customers a really bad experience. Or what happens. In many cases a marketer will just, you know, launch a feature and they come back to you and say, we launched this six months ago. Can you tell us what the impact is? Well, we obviously don't have time machines, right? So you can't go back and launch an experiment. But, but there are times when designing an experiment can have huge impact and can really change around how the business thinks about. And you know, one of the things that we talked about a little earlier was that branded search that I did this one company, we were ready to reallocate all this money to other things and we found through a well designed experiment that no branded search was actually saving a lot of users and we could have lost a lot of customers and high value ticket customers and high intent customers by getting rid of it. And we were able to make that business decision through a well assigned experiment.

No, you know, we, we've dug into marketing for a little while and now I want to like take a step back. You know, you've really been putting out a lot of content that I think that's really resonated with people. You know, you, you mentioned, I think recently in, actually in a, in a thread that we had going on LinkedIn about like, look, this is one of the ancillary benefits of all of this posting that I've been doing is just like learning stuff and meeting people. I'd love to hear, like, how is that going for you? What have you found to be like most interesting or most compelling about this like journey of producing content and sharing it with the world?

Yeah, the networking aspect has been like a mind boggling positive that I found. So I'll give a couple of examples. I had shared one post where I was just digging into, again, pretty entry level, clearly not entry level, but I was learning about Google Meridian, right? And I posted a couple of videos and one of my posts actually was shared by the technical lead at Google who was working on Google Meridian, right? And after that post I had about 10 or 15 people that were technical at Google who added me to their network. They reached out to me. I had multiple people who were data science hiring managers that reached out directly to me, say, we're working on mmm in our company. We're looking for someone. I really like the way you're thinking about. So there has been job opportunities, the connections from people who are working on similar problems at these other companies. Like, right now, I'm in a good place. But, you know, we see what happens with layoffs. They can. They can happen at any time. And my network of people who are in a position where either they're a data science manager or they're someone who is working on a team where I could be a really strong fit, these people are now first connections of mine, right? They're not second, they're not third, they're first and they've added me and they're already familiar with my work. So they're going to respond when they see my message. So that's sort of one from like a job and a networking thing. It's been phenomenal. But I've also had people, two professors, who are not just like, I'm a practitioner of some of this causal inference stuff, but there are folks on there that are right now actively publishing in some of the biggest journals out there and publishing work on causal inference, on advanced difference in difference methods, on synthetic difference in difference. I mean, these are tools that are just now in the forefront and they're commenting on my results and getting little pieces of extra intuition. There are also folks that are commenting, hey, have you seen this technique that's related? And it's introducing me to other people, other creators, other people who are working in fields that are similar that I maybe wouldn't have known because they're working at another company and maybe they're not necessarily publishing a lot or I just have to come across them. And so this has been an incredible positive externality toward posting. And it's something that when people who are coming out of the data science programs now and they're facing a really difficult job market and they're wondering, how can I network? You know, how can I showcase my projects? And some of my suggestions have been, while you're learning something, you're going through a bootcamp, you're working on your projects. If you have an interesting insight, just start sharing it online because as you've learned, there are other people who are coming up that are trying to learn that same thing you are. And you're going to start building a network of people who are in your field and you can learn from each other and you can, you know, add more connections into these companies. And I'm learning a ton. My LinkedIn feed used to just be, hey, I got a new job at this place, and now it is just full of technical papers that people wrote or simulations that they've done. And I am learning a ton. People will ask me, what are you doing to stay on top of causal inference in your field? And I'm like, honestly, I just strolled through my LinkedIn feed at this point. I mean, there are people publishing brilliant stuff, and I'm staying on the top of my game, just learning through, just going through my LinkedIn feed.

I love it. I love it. And, you know, maybe you've already answered this question, but in addition to all of the work that you've been doing on LinkedIn, I know you do a lot of coaching and sort of career guidance stuff. I'm curious, like, for young data scientists, outside of telling them to go, you know, publish more about what they're working on, which is also advice that I give. What, like, what advice do you, do you give for them? Like, if you had to give them one takeaway? I think a lot of people, at least I see on Reddit and I see in other places that, like, people are very nervous about the job market. You know, is data science going away? Like, what is the advice that you're giving to these young people that are trying to come up with this field?

I am someone who is not trying to sugarcoat people, and I try to kind of be a little bit of a dose of reality. I know that there are roles that are data science roles at big companies that are getting a thousand or more applicants in a day. But where I try and inspire a little bit is I actually came out of my, my undergrad degree. I thought I was going to be going to Wall Street. And if you were to look back at the stock market going from like 1999 to 2003, you'll see like the NASDAQ drop by almost 80%. And literally, literally right at the bottom, right at the bottom was, you know, Jonathan Hersh applying from a small liberal arts college, trying to get into Goldman Sachs. And I was just hearing nothing, but I mean, I was getting interviews and they were literally like, I've got kids from, you know, Wharton, Harvard, Stanford, Yale out there in the lobby. You're coming from Vassar. Why are we, you know, why are we wasting our time? I mean, those are literally comments and questions that I was getting it. It was Just such an insane time. And so I share, I mean some of the crazy things that I did back then. I mean there was no LinkedIn, there was no Indeed.com. i was straight up, I was going on the chamber of commerce website where I just saw almost just like a list of companies and email addresses. And I'm just writing to state by state, hundreds of people, you know, hey, I see you do this. People are writing me back like, Jonathan, thank you for writing me. I'm a one person consulting shop. Like how on earth did you find me, right? The one I, I almost got. I, I still, it's my what if was I found an econ consulting place that was in Hawaii, but they were like, we would love to have you, but you need to be a joint PhD student at the University of Hawaii to say it at the same time. And I thought about it, but I was like, I'm the exhaust my other things. So the first thing I say is that you need to just be, you know, look, some people really win the lottery with their timing, right? They come out in the hot job market and they can just get years of getting jobs. Life just isn't fair. And some of us come out in much tougher times and we just have to deal with what life has given us. And you just have to be smarter, more aggressive and think a little bit more outside the box than your peers. And it just, it's just a reality. And so I just suggest people to try and think about things that they can do that maybe their peers aren't and, and to go with it. And it's, it's going to be tough. But if you're able to get a job in this position, in this market, you know, a couple of years from now, you're going to be the one with experience and the other people won't. And the other thing I add is that don't be afraid to pivot. So I thought, I thought for sure I'm coming out. I was watching movies like the Boiler Room and Wall street and I thought I was, you know, a young punk kid who thought he was going to, you know, be making it as a trader or something. And I ended up working at the Federal Reserve. And the Federal Reserve from there opened up so many doors for me. I mean it's really what led me to eventually doing a pH. I got trained in SAS coding, coding that got me into this other, it was just such a great training ground. So I would say don't be afraid to pivot. Don't be afraid to build your own things and just, just keep trying new stuff and just try and do more than what your peers are doing.

Jonathan, I believe that is a great note to end on. Thank you so much for joining me. This is a really wonderful conversation. And yeah, I hope you have a great rest of your day.

Awesome. Thank you so much, Michael. And you.