Fact vs. fiction in causal AI for marketing & data science | Aleksander Molak | Coffee Breaks

Foreign Howdy y', all and welcome back to Recast's Marketing Measurement Coffee Breaks, where we dive deep into complex marketing and measurement topics with industry experts while you enjoy a nice warm beverage. This week I had the chance to talk all about causal AI with Alex Moloch, a machine learning researcher, author, advisor and educator and entrepreneur specializing in causality, natural language processing and AI strategy. Alex and I had a great and wide ranging conversation covering everything from fact and fiction in causal machine learning to how to identify snake oil sellers pitching causal AI. It's a value packed episode that I think technical marketers, econometricians and data scientists will really enjoy. So go pour yourself that hot cup of joe and make this a coffee break where you will really learn something. All right, Alex, good morning. Well, good afternoon to you. It's great to have you. Thank you for joining us here on the Marketing Measurement Coffee Break. I'm really excited to chat today. I've been following your work on LinkedIn and on the Internet for a very long time. I think you're one of the best and brightest voices talking about a thing that I am very passionate about, which is causal inference, and in particular talking about the intersection between causal inference and AI, which I know is a a hot topic with a lot of snake oil salesmen out there. And I'm glad to have someone who has made it their business to really tell the truth about what's possible and what's not. So thank you for joining me, Alex.

Thank you, Michael. I appreciate this beautiful intro. I'm very happy to be joining you as well. And I've been following your staff on LinkedIn for a while as well. And I think you're doing great job with building the awareness in the community that marketing mix modeling is something that is essentially in its nature, it's something causal. Right. And maybe it requires a little bit more work than we sometimes assume. So it's a pleasure to have this conversation with you.

Today's going to be a great conversation. So maybe to start, could you give a quick background on yourself just for the listeners or the viewers who aren't as familiar with your work. Just the very brief background, what you study, what you do, and then we'll jump into the meat of the conversation here.

Sure. Historically, my academic background is partially in philosophy, partially in psychology and neuroscience, but for many years I work in machine learning. And for most of those years, actually I think already now I work with causal modeling. I'm the author of Causal Inference and Discovery in Python. That's a book that has Been released last year. It's a book that I wrote for people who are in machine learning and want to learn about causal inference and its intersection with machine learning. I'm also a consultant and a speaker and an educator and I help companies go through their causal journey smoothly.

Love it. Wonderful. So maybe to start, can you define what causal inference is or at the very least what it means to you?

Yeah. So causal inference, there are two meanings of this term, one more general and one more specific. I will start with the more specific one. In the most specific sense, causal inference is inferring how strong causal relationships between variables are or what's the nature of those relationships. If it's a linear system, maybe we are interested in, in quantifying the strength. If it's a nonlinear system, maybe we want to learn what kind of function is this? So we can use this information to either guide our decisions or to make predictions about certain, certain quantities that we are interested in. In a more broad sense, causal inference encompasses causal inference in this more narrow sense that we just discussed and also encompasses another subfield that is called causal discovery. And causal discovery is this activity of trying to find out the structure of the data generating process, what causes what in some system of interest from either observational or experimental or combined data. Experimental and observational.

Really helpful. Great baseline. So when I think about causal inference and machine learning, I actually sort of think about them as being like two very sort of distinct fields and almost like warring factions at times, right? Where there's like the machine learning people who are all about prediction and the inference doesn't matter. And then there's the causal inference people who are like doing a little bit more like scientific research sort of work, trying to learn things about the underlying mechanisms. And a lot of the work that you've done and the book that you've written is sort of about the intersection. So I'd love to hear about how do you think about that intersection and sort of maybe dispel for me this myth of the warring factions. Like how do these things actually end up relating? And where is that intersection? And where do those two fields sort of borrow information from each other?

Yeah, I'm really glad you asked this question. I think machine learning, we know that machine learning can be very, very powerful. In particular, it can be a very, very powerful predictive device. What we tend to forget sometimes in the community is that although machine learning is much more flexible than traditional statistics in terms of that, we need much less assumptions when we work with neural networks, for instance, on tree based models, than with some traditional linear models in traditional statistics, despite this fact, machine learning is not assumption free. It can be a great predictive device. Your machine learning model can be a great predictive device, but it will lead to effective predictions only as long as we meet certain assumptions. And the main of those assumptions is the IID assumption. So identically and independently distributed variables, which means that the underlying joint distribution that we use to predict some variable of interest is basically static. It doesn't change over time. It's a static distribution. It's all the same, and it's multivariate static. So that's an even stronger assumption. As one of my podcast guests, I think that was Alicia Kurt that said people don't care about prediction, they care about generalizable prediction. And in this sense, I think causal inference also is something that can be useful for predictive tasks. Because in many settings, especially in open systems, open complex systems, when we talk about marketing, when we talk about policy evaluation, when we talk about promotions, advertising, and so on and so on, we have structures that we have distributions that are changing, that are permanently, permanently changing. So in this sense, causal inference can be helpful there. But when we went to work with complex and nonlinear systems, machine learning is just a perfect engine for this. So we often merge the idea of thinking structurally about data generating process, about the system that we want to describe, and this part comes from causal inference. And then for the estimation part, because in causal inference, we still estimate quantities in a statistical way, we can use machine learning so we can plug in machine learning as a building block for our causal machine learning model. And this is a place when these two ideas, when we add them, when we sum them up, they lead to something more than just a sum of those two parts.

I love it and I specifically love what you called out and I think what your podcast gets said, which is that the real task is the generalizable prediction. And in particular, I think when I talk about causal inference, I talk about what we care about is being able to predict in the face of intervention. So when we make a change to some system via a policy intervention, or by changing our advertising mix or whatever, can we predict what is going to happen in that context? So not just predicting business as usual, if everything stays exactly as it's always been, can we predict the future? That definitely has value in some applications, but what you really want to be able to do in a lot of contexts is be able to predict what's going to happen when you change something. And there's a lot of contexts in which machine Learning doesn't give you that because of the sort of underlying assumptions there that you have, the IID distribution of variables, which is sort of by definition violated when you're making an intervention. And so that's the thing that we really want to solve.

That's a really great remark. So what you touched upon is, is that sometimes we are not interested in just predicting something, even in a generalizable way. We are rather interested in predicting the response of the system to our intervention. This is related to the idea that comes from Judea Pearl, the idea of the ladder of causation. The ladder of causation, it can represent many things. In the simplest term, it's just a metaphor that allows us to talk about different levels of relationships between variables. And so the first rung is association. All predictions in machine learning are based on some types of association, right? They might be complex association measured by, I don't know, some measure theoretic devices or some more advanced correlations or just simple correlations, but these are all associations. And we have the second rank that these are interventions. What you mentioned, we're interested in seeing how the system reacts and predicting how the system would react if, if we do something. And then finally we have counterfactuals. And counterfactuals means the action, as Perl expresses this is imagining. So we can imagine how a system would react under some minimal change. So how would my life be if I had a Ferrari in my 20s? Maybe some people on YouTube are asking this question to themselves. So they would like to keep everything constant, everything that happened in the their lives just change this one thing, this Ferrari, right? And so these are counterfactual queries. They don't have to be necessarily backwards looking. We can also imagine alternative realities in the future. And the ladder of causation also comes. It's not only a metaphor. I said it's a metaphor. Or like a mnemonic device. But it's not only this. It also comes with three distinct formal languages that allow us to talk about those quantities. Different questions, different quantities at different levels. So we have different formal languages, different mathematical formalisms and different sets of assumptions to answer those questions. And maybe as one remark that I think is also important, and it's not so often discussed in the community, is that we often talk about causality as some uniform entity. It's just one thing, causality or causation. And then we can solve it or not solve it, but in fact we have a bunch of different causal queries, different causal questions that we can ask and the methods that we will use to answer those questions might be different. There are some methods that are universal that are coming from rank three. They can answer interventional observational questions as well, because counterfactual models or structural causal models actually are hypothesis about the underlying mechanisms. Very low level hypothesis. They can answer all types of questions. But interventional distributions cannot tell us anything about counterfactuals in general. In special cases, yes, but in general, no. This also shows us that there's a limit to the answers that we can get from experimentations from randomization. Randomization can tell us something about interventional distribution, but not necessarily about counterfactual. So sometimes we need to get a little bit smarter, a little bit more creative in order to say something about those counterfactual quantities.

I love it. I think it's a good call out that everyone who's like a LinkedIn influencer is constantly just posting like correlation is not causation. But it's actually like a very deep philosophical concept here that we're talking about. And it requires thinking fairly deeply about exactly what question are we asking and what are the right methods that we need to answer that specific question. And exactly how the question is framed, imposed or defined will really determine what methods are required or whether the question is even answerable at all. I want to.

Yes, exactly. That's a very good point.

I want to change topics a little bit. So artificial intelligence AI, very hot right now. Everybody's talking about it. Everybody wants to jam it into their product or solution or claim that their solution is some sort of artificial intelligence or AI. People have started using this phrase causal AI, which I think at the surface sounds great because you get causal inference, you've got AI. It's hot. It sounds like it could solve a lot of problems for us. But I would love to hear from you who spent a bunch of time in this space at sort of the intersection of machine learning and causal inference. Like, what is causal AI? What are people, what do people mean when they're talking about that? Is it just this thing at the intersection of causal inference and machine learning, or is it something else? And in particular, I'd love to start to hear your opinion on what can we actually use it for and what can we not? What's the sort of fact versus fiction here? And how should people who are seeing these claims about causal AI on LinkedIn or Twitter or X or wherever start to think about evaluating what's real here and what's just salesmanship?

That's a really great question. Causal AI, the term is used essentially as an umbrella term for anything that is an intersection of causality, some type of causal inference or causal discovery and machine learning methods. So we discussed briefly in the beginning, we started today with discussing the definition of what causal inference is. We said that there's this broader definition that encompasses causal inference, just inferring certain quantities and causal discovery, so trying to learn the structure from the data. And there are also some other subfields to this. So there is a subfield called causal structure learning. We can think of this as a generalization of causal discovery to data that might be high dimensional, for instance, videos. Right? So we might want to understand not only how one variable is causally related to another one, which in video would be pixels, and it doesn't make too much sense, right? How one pixel is causally related to another one. We actually want to discover what are the objects in the video and how can we track these objects consistently over different frames and how we can see how interaction between one object and another object to what outcomes this interaction leads us and maybe predict outcomes of such interactions in the future as well. So we have these methods from causal structure learning, causal inference, and all of those methods, also methods from econometrics that we sometimes call quasi experimental methods or yeah, I think let's just call them quasi experimental methods. They can also be combined with machine learning or they can be combined with some devices for, let's say, sensitivity analysis that draw from machine learning vocabulary and they will also be included in this broad, broad umbrella under the broad, broad umbrella term of causal, of causal AI. So that's regarding what causal AI is now, what works and what doesn't work. That's a very broad question. And we could probably spend a lot of time discussing this in very general terms, trying to give an answer that could be helpful for somebody who's interested in this field, but maybe they don't have too much experience already. I think a very important first step is to set your expectations realistically. So thinking that causal AI is some magic box that you can ask a question, you can throw a bunch of data on this and just ask any causal question and you will get answer that's not a realistic expectation. Causal inference. And with or without machine learning, we usually require us to also share with this machine a little bit of, a little bit of our knowledge about, you know, what we know about the system, how it works, you know, say something, say something from the expert perspective. So setting expectations, I would say it's very, very important. And what doesn't work and where sometimes I see people come with unrealistic expectation and they do something and then they are surprised that things didn't work, is for instance, using quasi experimental methods with an expectation that we'll get causal identification, which means that our results will be causally unbiased just as given. Okay, so this does not come for free. Which means that even with quasi experimental methods, even if we take time into account and maybe some structure of the design that the data comes from, we take those things into account, we can still have some hidden confounding, for instance, that can bias the results in more or less arbitrary way. But. And so there is no free lunch, as we like to say in machine learning. There's also no freelancing causality. Usually if we want to do a valuable project in causal inference in causal machine learning, we should be ready to put some work into this to get the value out of this. And I would say, based on my experience, I worked as a consultant with, with many companies, Fortune 100, Fortune 500 companies, as a machine learning consultant. It's also true for purely predictive machine learning. The most valuable projects, the most valuable solutions that we've built always required some work, understanding what we're doing, understanding the business context, and so on and so on. So I think in very general terms, that would be my answer to your question, that there are many methods that can work in many circumstances, but assuming that something will work just out of the box and answer a pretty complex question for us just like this, this is not realistic. So we should be very cautious if somebody comes to us and tells us like, okay, this is this box. You just put your data here and you will get all causal answers to your questions.

I think that is a great summary and is right. And there's a lot of people who are selling those magic boxes. And I think it's important to be skeptical and to be thoughtful. I'd love to talk a little bit more, get sort of one or two levels deeper on this idea of causal discovery. So my background is a little bit more of the traditional causal inference approach. And when I think about doing causal inference, I sort of put, you know, the scientific understanding as the starting place. Right. We're going to start by doing literature review or talking to experts and then write out our beliefs about the causal DAG is and then try to design some sort of experiment or model to try to estimate different parts of that and understand those relationships. But very much like the starting from the scientific belief is the first place, when I see something like causal Discovery, it sort of seems like that maybe gets flipped on its head a little bit. And this might be a misconception of mine. So I really want to understand, you know, what's actually being proposed here. Like how does causal discovery work? Is it we're going to throw out the DAG and let the machine learning model find the dag. So and for the listeners, DAG is directed acyclic graph. It's one of the tools for describing causal systems. So are we getting rid of that and using machine learning to tell us what that is or sort of where does causal discovery actually happen? And how does it fit into this sort of traditional causal inference process of starting with the science and the research and then using models to estimate that?

Yeah, that's a really, really great question. So I love that you started with starting with a scientific understanding of a problem. And I think this is a very important perspective. And I would like to close my answer with coming back to this topic. But before we can get there, let's define what causal discovery is. Causal discovery is essentially a set of methods that aim to retrieve the information about the data generating process from the data. It can be from observational data. That's the most, let's say, extreme case. Then we can have a combination of observational experimental data, randomized experimental data, and then we have a purely experimental data, right? Maybe. So the goal of causal discovery is to actually build a dagger based on the data and a dag. Going back to what you said about the scientific understanding of a problem, a DAG is essentially a hypothesis about the structure of the process that generated the data that we observe. So we say maybe we have three variables, A, B and C, right? And we observe some values for all of those variables. Now we can pass it to a causal discovery algorithm and this algorithm will return a directed acyclic graph that's the output from the algorithm. So it will say, for instance, that A causes B and B causes C. And that's a hypothesis. Now I say that's a hypothesis. Probably we should always say that. We should always use language of hypothesis when we talk about machine learning, even if it's predictive machine learning, as we used to do this in statistical learning theory, but today we tend to think as those hypothesis with more certainty often. Sometimes it justifies, sometimes not. But here I think it's very, very essential that we think about those outputs from causal discovery methods as hypothesis. There are a couple of reasons for this. And the first and the most important reason for this is that in general, causal Discovery from observational data, from purely observational data is not possible without additional assumptions. So if we don't know anything about the data and we just take off the shelf algorithm for causal discovery, we essentially have zero guarantees that this hypothesis, this DAG that the algorithm is returning makes any sense. There's zero guarantee for this. Now, if we add some assumptions to this. So for instance, we add assumptions regarding the nature of relationships between variables. We say like, okay, the relationships are linear in this data set. Now we can get some guarantees for identification. Or maybe we say they are not linear, or maybe we say they are not Gaussian. The distributions of variables are not Gaussian. So this is like a little bit of a reverse what we have in traditional statistics, right? In traditional statistics, we love things when things are Gaussian because it makes everything super easy for us. We can do a lot of stuff just by calculating stuff and doing deductive kind of inference. But in causal discovery, it's reversed. We can leverage asymmetries in the data, distributional asymmetries or some functional asymmetries to detect the traces of causes in the data. And so this is very helpful. But this requires us to make certain assumptions about the data or the data generating process. And then we can say something. Now we can say something, but not always with very high certainty. And that's also something we should remember about. For instance, we have an algorithm called PC. This is a classic algorithm in causal discovery. And it comes from with some very strong guarantees. But it turns out that these guarantees are in the infinite data regime, right? So every time we go to a finite samples, our guarantees will shrink, but we don't know how much they shrink exactly unless we understand something about the data generating process. But we don't do it. We don't understand this because the goal is to actually learn something about this process. So that's causal discovery in a nutshell. And I think that the most useful way to use causal discovery today is to combine it with expert knowledge. So we might have different scenarios how to do it. We can see those algorithms. We can in a sense inject expert knowledge within these algorithms and then let them learn from the data. That will make the task easier for them because they know already that there are some, some variables are connected or some variables are not connected in the system according to expert knowledge. Or we can also use large language models to, you know, to share some insights about which variables can be connected, build some hypothesis, initial hypothesis, and then maybe triangulate this with real human experts. And then Seed the model with this knowledge. So that would be an overview of what we can do. But of course, there's much more to this.

I love it. That was a great explanation. I think there were two, I think, points that really jumped out at me, and at least one of them I want to follow up on a little bit more. So the first was that we should think about the DAG as a hypothesis. I think that that's a really succinct way of framing of how we're thinking about structuring causal dags. This is a hypothesis. These are assumptions about what we believe the world, how we believe the world works. And then we want to maybe test some of those assumptions, like, that's the point. And so what is coming out of causal discovery is a hypothesis, right? It's not an answer. And I think that this is a place where, again, if you're not, if you read the literature and you're deep in this, you probably won't make this mistake. But if you're sort of a more passive consumer of this information, that might not be immediately clear to you, right. It might seem like, oh, this is a magical box. I don't have to think about causality anymore. But that's not right. We can use these algorithms to generate hypotheses, but then we need to go do something with them in order to actually have something that we feel confident in making decisions based off of. And then the last, I think I just want to really want to double underscore this point. With these methods, we get basically no guarantees without additional assumptions. Right. If we just take some observational data and pump it through these algorithms, we don't get very much out on the other side. So we have to think about what assumptions we can make, what assumptions we can't make, and then be very careful about what guarantees are we getting back from these algorithms. It's not just like, pump the observational data through, get guaranteed to work causal graph on the other side. That's not what this is about. So those are the two things that I really took away. I want to think a little bit. I want to ask you the question about, okay, so we do this causal discovery positivity process. We get a DAG on the other side, as you noted, that's a hypothesis about how the world works. What do we do from there? Like, what's the, what's the next step after we've generated this hypothesis or potentially sets of hypotheses from these processes? How do we, how do we take that and do something else with it.

That's a great question. So I would answer, you know, I would start my answer with saying that it really depends on the question that we started with, right? So I would say that in causal inference, it's perhaps even more important to clearly understand what question we want to answer before we start a project, because maybe we are interested in something very, very particular in the system of interest. And it might be the case sometimes that we can answer this particular question with only a fraction of effort that would be needed to say something more about the system. So I think there is a very, very strong incentive also from the resources point of view, or resources management point of view, to ask very clear questions before we start projects like this. Now, assuming that our question and our original question was regarding some quantity of interest, maybe we want to understand what would be the result of changing our media mix, right? So maybe I want to go like 90% digital today and what would be the effect of this? So now if we discovered a DAG and we think that this DAG is a good hypothesis, then we can build a model that will allow us to estimate the causal effect of our actions, right? So maybe changing the percentage that we invest in whatever, some digital channel versus some traditional channel, we change it from 40 to 60 or from 80 to 90. And then we can query a model like this. And assuming that our dagger is a valid hypothesis about how the world works and we have enough data in order to saturate our estimators, that would be, in most cases, machine learning models. And assuming again that the distribution in the future where we want to make the intervention, perform the intervention, or carry out the intervention will be the same as in our historical data, we can expect that we will get a reasonable estimate of the outcome of our intervention on the system.

Just to restate that, because you gave a very precise answer that used some complex verbiage, correct me if I'm wrong. But restating this is basically like, okay, we can use causal discovery, generate a dag, but then the thing to do with that is to go test, will have that hypothesis. That DAG hypothesis should have some testable implications. And what we want to do, especially in the business context, is really go try to, like, actually see if that works through an experiment or through just leveraging those insights. And then once we do that, now we have more data, now we can sort of verify and say, okay, was this hypothesis correct or not? Was the results of the experiment consistent with that hypothesis? And then keep doing that. And so correct me if I'm wrong. But what I'm sort of hearing is that the idea is sort of get out of the analysis mode as quickly as possible and get into the experimental or interventional mode as quickly as possible, because that is what is going to start confirming or disproving these hypotheses that we've generated. Is that right?

Yeah. So that's a great question. I even haven't reached the point of testing the model, but that's actually the next step, something we should always do. And I loved one thing that you said, that we expect that a model like this, a hypothesis like this will have some testable implications. That's a really, really important term. So let me just broaden this a little bit and also bring it back to the original point. When you started talking about science, the connection between causal inferences in science. I often get this question. Just yesterday I had a call with one company and I got a question. Okay, so we have this DAG that we got from the Causal Discovery algorithm. It's a huge DAG. It's over 1,000 variables. How do we make sure that it's correct? So I think this is also a very good point to remind ourselves about the importance of explaining expectations. Sometimes when we look at the progress in AI, we might produce this expectation in our heads that we are somewhere very close to the world when we can just go query a large language model and it will discover the physics, answer any possible question and tell us what to do, what's the optimal decision, and so on and so on. But in fact, we are very far away from this and we are very close to the moment where we actually started using, or we invented the scientific method. So we build hypotheses, we can test those hypotheses, and then based on knowledge from many different hypotheses, we build a theory how the reality works, right? And then from this theory, we infer more hypothesis and we test them. And so in a sense, and then the only thing we can do about this hypothesis is not to confirm them in some sense, to obtain certainty, we can just falsify them. So this idea of falsifiability comes from Karl Popper, right? That's one of the fundamental ideas of modern science. And so with causal inference, we are essentially in the same place. We are generating hypothesis. We can try to test those hypotheses either by experiments or sometimes if we are lucky, if we are lucky, we can see in the observational data that maybe something happened historically in observational data that actually proves that this hypothesis is wrong. Something that would be impossible if the world looked like in this DAG that is suggested by our model. And then we build a theory, and the theory would be a structural causal model. So we built a model that tries to reflect the structure of reality. We have some variables and we propose what are the connections between these variables? What is the nature of those connections? Are they linear, nonlinear functions? Are those variables categorical, continuous, and so on and so on. So we build a model of reality and then going back to what you said, Michael, before, we will have some testable implications from a model like this. So then we can run experiments, we can compare observations, we can maybe take observations from another environment. So maybe if you work with a number of companies and you have the same underlying model about the reality, you can actually inform your confidence in your model by combining the data from different environments, maybe different markets or different cities, or different channels and so on and so on. All of this can be helpful in validating causal models. But we don't really have an equivalent of quote unquote, like a cross validation device for a dag, right? So there is nothing like this, that we just have one device that will always work for any dag, any hypothesis, and will just tell us under the IID assumption this DAC is correct or it's not correct.

So to some extent it's really that the work has just moved a little bit. Maybe we don't have to do as much thinking upfront to generate the causal DAG and the hypothesis, but we still have to then think about how do we falsify that or at least try to generate data that is or is not aligned with that. And that is the part now that takes the creative thinking and the scientific thinking of how do we structure an extra experiment, how do we interpret it, how do we verify that it does or does not align with the dag. So I love that. The last topic that I want to talk about is you had mentioned these quasi experimental methods from econometrics and how people are starting to combine them with machine learning or causal AI. I have been spending some time with the synthetic control literature recently, which has some of this flavor. You've made some points that you have to be very careful with these methods about what guarantees you get around causal interpretation. I'd love to hear sort of your perspective and your overview on what do these methods look like and where should people be being careful in terms of their interpretation and their use of these tools.

Yeah, so that's a great question. I'm actually a fan of synthetic controls. I Think that's such a. Such a clever idea. The danger, though, I will keep the example of synthetic controls, not to confuse people, right? Because there are many different quasi experimental methods, and for most of them, a similar argument will hold. Even if not identical, it will be structurally similar. So in synthetic control, we assume that there is some relationship between our variable of interest that we can intervene on. Maybe there is our investment in Facebook ads. Our investment in Facebook ads is something over an extensive period of time. And we have some sales. Our sales correlate maybe with some other variables, maybe with inflation or some other macro economic variables. Maybe our sales correlate with the weather. Maybe our sales correlate with whatever micro macroeconomic variables or other variables that can be relevant for the system. And now we say correlated, which means there doesn't have to be any direct or indirect causal relationship between our sales and those other variables, but they should be correlated to an extent that we can build a model that predicts our sales based on those variables. And then at some point in time, we decide that we will increase our investment in Facebook ads. Maybe we increase our budget by 20%. And because we are doing this, we know when this happens, Precisely when this happens, or even if this happens in we do not control it, but someone else in our organization or our client, we can observe it. Sometimes people refer to this as a natural experiment. Even if we cannot randomize the treatment, the treatment happens somehow, naturally. Maybe our client decides, based on their intuitions, when to do this increase. We can observe it, and there is something that resembles an experiment. That's why people say natural experiment. It's not randomized. But in some sense, maybe it resembles an experiment. So there is this change, and then we are interested in the effect of this change on our sales. So what we do in synthetic controls is that we are keeping this model that predicts our sales based on those other variables. Let's agree that for the sake of example, it's weather and inflation. And maybe for other variables, and we are trying to predict this, we land this model before our intervention, and then after our intervention, we kind of predict the sales with an assumption that the sales would be like this. The relationship between our sales and all the other variables would be the same as before our treatment, right? And we compare the predicted sales with the sales that we observed after actually changing our investment, our budget for Facebook ads. And so this difference between the predicted value of our sales and the actual value of our sales after our intervention would be the causal effect of our intervention. And now this looks really really great and amazing. But there might be certain factors that are impacting, for instance, both our treatment decision and general sales. So this is something that's an example of hidden confounding. I have a blog post where I describe one of the approaches for synthetic controls, and it's based on searches on Googling habits of people in response to Elon Musk tweeting about his acquisition of Twitter. And so the conclusion is that these Googling habits has changed, at least temporarily, in response to this tweet. But we actually don't know if tweet was the treatment or the treatment was maybe just media information about the acquisition. Right. So we should always take into account the context and that there might be things in the context that might not be obviously connected to our treatment, but they might be nevertheless connected to the treatment. Right. And if this happens, then our conclusion will not be valid because we cannot distinguish between the impact of. I will go back to the blog, the example in the blog, because I think it's maybe just intuitive. We cannot distinguish between the impact of the tweet itself, Elon Musk's tweet, and about the media coverage of this tweet and of the event. So that's one of the scenarios where we don't have causal identification in synthetic control. And we would need to think really, really hard about. About possible factors that could influence the treatment decisions in the real world. Right. So when you work with people in marketing, people in marketing will often be very reluctant to randomize the treatments because they have a theory about how the world works. And they say we should advertise here or maybe we should run this campaign here because it's the summer, and during the summer people are more, you know. Yeah. Have higher propensity to convert. Right. And so they also produce the data that is an outcome of their world model. But going back to what we spoke about a second ago, if we want to validate a causal theory, we need to actually falsify our world model. Try to falsify our world model. And if it survives our repeated attempts to falsify it, we'll say that this theory, this world model that we have today has a good fit. But it's still. It's not confirmed. It doesn't mean that it's 100% correct. It was just able to survive our attempts to falsify it. So when we talk to clients and they insist that we should still keep doing things the way we do because we believe that this is the world, it might be a difficult conversation sometimes to convince people that it's really worthwhile to change the way we do things. And maybe our main task then, if we really deeply care about the client and their outcomes, is to convey the value and show the value to them that if we find the courage to change the way we do things today, we might get a benefit that exceeds what we think is possible.

Alex, I think that is sort of a great point to end on. I'll quickly summarize a few of the things that I heard in that last section, just because I think a lot of them were really powerful. So the first thing we sort of came back to this point that you've now made a couple of times in our conversation, that with all of these methods you really don't get guarantees without assumptions, right? The synthetic control method and other sort of blends between quasi experimentation and machine learning, they're great tools. They can potentially help you get additional precision within the sort of constrained, assumed world that you have. But you have to make those assumptions. And that requires deep thinking about are there potential plausible co founders, what is the data generating process that we believe, and are those assumptions consistent with the methods that we're doing? There is no free lunch. There is no magic box that can give you guaranteed causal inference without thought and without making assumptions. And so I think it's really just important to hammer that home for people. And then the last point that you made about having the courage to actually run an experiment, I think that that's a thing that really resonates with me and the work that I do. And I think that that's a thing that us sort of practitioners really can't say enough, which is that all of these sort of observational methods are really interesting. But the more that we can actually encourage people to change what they're doing by running experiments by a t tempting to invalidate the hypotheses that they're generating, actually, the more that we can learn faster, right? And you get sort of more, you don't have to make quite as many assumptions. The more that you can actually make changes and drive interventions, and the more that you can do that consistently and frequently and across a range of different possible interventions that you have, the more that you can learn without necessarily needing these really complex measures, methods that have these sort of difficult to sometimes reason about weak points or lack of guarantees. So all that is to say, Alex, this has been a really, really wonderful conversation. Thank you so much for joining me. Hopefully this has been interesting to our listeners, listeners watchers. We are going to include links to a bunch of things from Alex that you should all check out is a book called Causal Inference and Discovery in Python. He has his own podcast, which is wonderful, and you should also definitely give him a follow on LinkedIn. That was how I learned about Alex's work. He produces just consistently wonderful content, so please do that. Alex, thank you so much for joining us and hope you have a great rest of your afternoon.

Great, Michael, thank you so much. And I hope also that this will be useful for our listeners. Thank you. Appreciate it.

All right. Thanks, Sam.