042. Running Experiments What You Need to Know

Foreign. I think we are rolling. Hello everyone, this is Emilio Garcia from the Mangen Studio. I'm hoping that you're having a fantastic day and today at the studio is with me, Andres Glusman from Do what Works. Andres, welcome. How are you today?

I'm doing great. Thank you for having me.

Awesome. Fantastic. And Andres, we were talking before recording on the topic for today and I feel that it's going to be a very interesting one for audience B2B marketers that are working on demand Generation is around experimentation and there's so many things that can be said there because as we were talking before, there's a lot of best practices, if you want to call that that might or not apply for what people is doing in the field, especially the testing. And you know, I'm very about the topic. Tell us a little bit about yourself, your company, what you do and how it relates to this topic, please.

Yeah, so my name is Andre Schlussman. I'm the co founder and CEO of Do what Works. I've been running experiments in some form or fashion online since 1998. So in the very early days of the commercial Internet, I've been running experiments there that were on ads. I became an early pioneer in the lean startup movement because I used to lead product and growth. I meet up, I helped make their first $14 of revenue helping them get off the ground. And I've just been passionate about experimentation pretty much my entire career. My new company is launched basically to solve a pain that I felt in running experiments. And the pain is that when it comes to growing, experiments are one of the most powerful drivers you could possibly have. One of the most important things you can do. And if you can do it right, you can grow from like this to suddenly hockey sticking upwards. It's amazing. However, it's really hard to do right. There's so many ways of doing it wrong. And there's a lot to learn. And as an industry we're learning a ton. I've learned I was fortunate to get to learn some of those lessons early. And what we built is a technology that allows our customers to see the experiments that are being run by other people in their space in order to not reinvent the wheel, not recreate the experiments that don't work and really hone in on the things that do disregard conventional wisdom and instead focus on the things that are more likely to move the needle. So we built this really cool platform initially as a toy and then it snowballed in this thing that's much, much bigger now being used by some of the world's largest SaaS companies, six of the world's largest streaming brands, banks, et cetera, et cetera. So it's a really fun product and mostly because I just get to play around with experiments all day long and see what everyone's testing and learn without actually having to do all the work of running success myself.

I love that part because, yeah, I, I agree with you that, you know, this space has evolved tremendously. Right. My largest experience with experimentation overall is around landing pages, of course, and in the world of advertising, particularly with copy. But I follow, I follow probably, you know him, I think, Pete Capula or Capula from Databox, which is, which is, you know them, right? Like a dashboard tool and all that.

Yeah.

And they have recently been working on these benchmarking process or groups where, you know, you say, well, I am a marketing agency or I am a B2B company and I want to know what my metrics compare to the rest. Right. So I'm really drawn to what you were talking about, the tool, the fact that you can say, well, if I can see what other people or companies are doing, that will be very helpful because sometimes you can't. And I think we were discussing this before. For me, the two reasons why you might leverage what someone else is doing instead of, you know, waiting for to see what happens on your own field is because sometimes you don't have the volume to run those experiments or the time. Right. Eventually you will get the data. It's just that if you have very low volume, it will take a long time. And sometimes also I think is better to start from a ground of best practices and move from there on the testing rather than, as you said, Right. Rediscover the wheel all over again. Well, start with whatever is working and then improve and customize for the situation that you are in instead of investing all this time to just come back to the same conclusion that other people in the field as already, you know, experimented and walked through. So that's fantastic.

And yeah, that's exactly right. So I love the people over at Databox. They're great people. Jeremiah and I had a conversation the other day. He's awesome over there. And it's crazy because you mentioned that all you have is people feel constrained on time and the number of experiments. They get around time and traffic. I told you earlier, right. We work with six of the top streaming brands. They feel constrained by the number of experiments they can run. We work with the top SaaS brands. You would think they have all this traffic, they feel constrained about the number of experiments they can run. In general, what we see on our platform is that it typically takes about a month for people to run an experiment.

Yeah.

And if you don't have enough traffic to be able to run multiple experiments on the same page, if it takes you one whole month to run an experiment, it means that on any given surface you're working on, you're gonna get 12 shots a year.

Yeah.

Which is so little. Just think about how few shots that is. It's so little. It's a lot of work, but it's not a whole lot of shots. And so every one of those is precious. And then according to Optimizely, guess what percentage of experiments don't move the needle. Like do not have an impact?

I have no idea. But I will, I will guess that definitely more than 50%, but I don't know.

You're right, you're right. It's 80%. 80 to 90% of experiments don't move the needle. So think about that. You get 12 shots a year and 80% of those are not going to move the needle. So you're really, really lucky. After a whole year of working, you've got two to three wins to show for it. Yeah.

And I'm not surprised because let me tell you the other thing that I feel is overlooked when you're running experiments and probably your platform run experiments across all the funnel or all the journey experience. Maybe not. But what I, from my point of view, what I have seen, for example, imagine that you're running an experiment on a landing page and your goal is to improve conversion rates. Right. You want to get more form fields. To put an example, there are things that you can do to improve that number. Right. You change the number of fields that headline all those things that you can do. But I have seen that even for experiments that move the needle in the metric that you are measuring, you still have to wait to see if the metrics that follow on the path are really impactful, being impacted in a positive way. Right. I. I've seen experiments or you know, strategies where you increasingly volume but quality goes down and then total new business doesn't go up or it takes longer to close those opportunities. Right. Because the quality is not there. They are not. And, and you, you know, for me, quality is code for are you ready to buy. Right. Not. Not every lead is bad because they're not the right buyer. It's just that they don't have still the intent. And if you lower the barrier of Entry, you pretty much what you're saying is accepting low intent leads or conversations with people and you are just working more for, you know, less impact. So I agree with you that what moves a little, I, I didn't know that it was 80%. I'm not surprised that is more than 50%. But I will even argue that you still have to follow up what happens after the metrics that follow because usually on experimentation, you will be experimenting with leading indicators, but the lagging indicators are going to take another three, six months.

That's right. It's like air in a balloon. You can squeeze the air from one part of a balloon and it kind of moves to the other part for sure.

Yes.

And so it kind of is bad news on bad news in that regard in that it means that it takes even longer to get results than you can, that you can work on or act on it or work on. So it's not good news for the conversion rate optimization community or for the growth community. The good news though is that when you do get a win, you're like, why are people crazy? Why would they be dumb enough to do this thing? And they'd be insane to try and run all these tests if they're constantly going to fail. When you do get a win, the impact has a pretty profound effect on your overall metrics. And so for the same amount of money that you're spending, for the same amount of traffic that you're getting to your site, you're suddenly getting more customers, you're getting more leads. And then your cost per lead, your cost per customer went down and now you have a more profitable customer. Now you can outspend your competitors. When you're, when you're bidding for different words, right, or for different places on, on the different platforms, you can now spend more because you're more effective than your competitors, which gives you a leg up. And so it's a little bit like the VC model, like venture capitalists who, you know, they're lucky if they can get 1 in 10 things to be a big hit. And that's sort of what the conversion rate optimization community needs to be doing too, or how they need to be thinking about things. And when they do, they get big, big wins. And so, you know, it's the wins definitely outweigh the losses. It's just a question of, well, hey, how can we just tip the odds to make it so that we win more often along the way?

I love that you're talking about that point because I, I couldn't agree more. That we need to see experiments and overall mark new and marketing initiatives as what VC do, VCs do or what R and D departments do, right? We, we say it all the time, you gotta be okay failing, you gotta be okay, you know, and, but in, in practice, we truly do not behave in that way. Right? For the few. I haven't run, you know, like you were saying, hundreds of experiments or thousands of experiments because at least in the, for the clients that we work and for the industries that we work, the volumes, you know, rarely justified or you know, bring enough data. But I can tell you one thing that I've experimented especially for the point of contact, which is we go out there with. Experiment with the assumption that the trial or the test versus the base is going to work, right? We, we don't go with the assumption that this might fail, like usually is. I feel strongly that this is going to work and for that for me matters, because then you don't do what you are describing. Having that idea that don't think that you run one experiment and that will be the one that hits. And that's it. It's got to be like bcs. It's got to be like R and D is. We're going to, we are going to work on all these initiatives, right? To develop new features, new products, new technology. Most of them will not progress. And we will have stages where we will say, well, our first initial idea is to get this change to this point. If that works, we continue to put more resources and if it doesn't, we stop it and we learn from it. Right? And there is value in investing time on trying stuff that doesn't work. But that's something that is very hard for organizations because they want, as you said, right? They want the stick head, the stick growth jockey, stick growth. But they are not willing to put the time for all the failures.

Yeah, yeah, it's, it's extremely humbling. And you're right. And don't get me wrong, it's terrible to lose. Like, nobody likes losing. Losing is painful, especially if you just spent a month doing it. Nobody feels good about it. And it's a really hard job if you get four losses in a row on your way to your fifth and your fifth one, which is a win. Man, that's a lonely half year right there. The year you're sort of feeling like, oh, I really don't want to share my results at the next all hands meeting. We're looking really bad here. It's part of the process. There's a great quote that I love, which is that experience is what you get when you don't get what you want.

Yes. Right.

And I think it's sort of in the growth community, it's sort of learning is what you get when you don't get what you want. Our point of view though, and kind of what I think the world needs is exactly what you're saying, which is one, you need to have the right mindset that you're going to learn a lot. And then the other question is, well, shoot, how can I learn faster? How can I learn from others? You know, somebody said the best kind of test, the best test is the one you didn't run.

Exactly.

And so that's, that's really what we're all about. And so it's really sort of about how do you get hone in on a signal? It can be qualitative signal, it could be a data driven signal using our platform, it could be conversations with it. How do you get a signal as early as you possibly can in order to inform the decisions you make to do better and get better odds?

Now let me ask you this. Now you're mentioning that part in your experience, maybe based on what you have already on your platform or overall in the marketing, you know, marketing things or things that markers can do for experimentation, where do you see the most need on this? Right. Is where do marketers experiment the most? Or where do you think they want to experiment more? But they, right now, because of the constraints of tools, volume, a challenge is to get signal as you are describing it. I love that because that's the other thing, right. Is that you truly, as you said, moving this, the needle for me is are you getting the proper signal or it's just noise. That's right.

That's right. And so usually traffic, sorry, experiments follow traffic. So people run experiments where there's the most traffic. It's a good thing too, by the way, because where there's the most traffic is where there's the most impact. So improving a, a 10% improvement on a page that gets 10 people a month versus a page that gets 10 million people a day at 10% means a lot more. So the places where there's the most traffic is generally a great place to focus because that's where you get the impact. In fact, the head of Asana's growth program, I love this term. I wish I could take credit for it, but. But I promised him I'd give him credit for it. The head of Asana's growth program calls these golden pages and so there's a handful of pages where most people spend tons of time and money getting your customers to. If in your case.

Right.

For your. For the folks that typically probably listen to this podcast, they probably spend a lot of time and money getting people to a landing page. They get them spending time to request a demo form to a homepage, to a product page. Those places, those are the places where there's a lot of wins that can be had. A 10, 15, 20% improvement, you know, or stack, start stacking those up over and over again. That's a lot of improvement on this place where they're spending a lot of time and money. So that's where you, that's where the experiments should be happening. And the good news is that's where you have the volume to learn. Bad news is that there's a lot of experiments that also people wish they could run kind of deeper in the funnel or in places that are super high value, but just much smaller traffic. And ultimately there's just not going to be a lot of experimentation today. It might take half a year, a year to get enough data in uncertain experiences.

Yes.

And in which case you're not going to be able to get the kind of data that would make your high school biology teacher proud. You're not going to get the kind of data that's going to make you comfortable launching a rocket up into space with people in it. Right. It's going to be data that's going to be very, that's going to have a large error margin. Margin of error margin.

Yeah.

Unfortunately, the, that's, that's the world you live in. That's the world you're. You're playing in and you just have to make the best of it. And you still want to gather signal, you still want to gather the data you can, but just know that there's going to be more error and that's okay. And by the way, that if somebody had told that to me when I was 25 or 30 or like early in my career, that would have felt like a weight would have been lifted off my shoulders that, like, oh, it's okay for there to be a little bit of error because you're just trying to minimize the error as much as you can. You're never, ever, ever going to do away with error entirely. Even when people run scientific studies to like put pills in people's bodies, like, there's still error involved. It's just small and manageable and under control, I would hope.

Yeah, we are, we are improving confidence. But it's not 100%, as you said. Right. It's, it's, there's always a level of error there. And now, now that you're saying that the things that you, as you said, right, someone have told you years ago, what do you feel are some of the, the pitfalls of the typical, you know, misconceptions about experimentation in general? That, that markers are starting out experimenting with, you know, with landing pages, how they drive traffic from social media platforms or search engines typically do that. They, you know, it's ideal that they'd avoid. And maybe, you know, your platform helps a little bit on that too.

Yeah, it's a really, really good question. The types of errors that people make that are most common. Well, there's a few different kinds. One is, is not running experiments for the wrong reasons. And those are things like saying, oh, well, we don't have enough time, or let's just get it out there. If we see a big impact, we see a big impact.

Right.

Or we'll figure it out with our analysts. We'll have our analysts figure it out pre post. And we're not going to run it like a split test when invariably happens on that one. And I've seen poor data scientists then spend months of time. Everybody's excited, everybody works really hard to get the thing out the door, but they don't actually run it as a test. And so the next day after it launches, the executives all come in, they say, okay, how are we doing with it? How's it going? Are we winning? Are we losing? And you're like, well, I don't know, it's too early to tell. And then a week later they're like, do we win or do we lose? What's the impact? And we're like, well, it might have had a lift, might have been a little lower, but last week was Easter, so we just don't know. And so the amount of work that goes into trying to approximate a truth, it means that months after the release, poor people have spent months or time, like tons and tons of time, hours and hours and hours trying to unpack and figure out what's going on. And it's so hard to discern that the desire to move fast often just means you're going to pay the price later down the road. So you might get it out the door, you might get it live faster, which feels good, but you're not actually learning faster. In fact, it's probably going to take you longer to learn than if you had just run it as a test. Is Sort of the unfortunate truth. So that's kind of one. The other big mistake that people tend to make is to want to be very precise and to want to be extremely scientific about every pixel on the page. The good news with that is it's like, why wouldn't you want to be that? Because it's so fun. You suddenly have an opportunity to learn. Does red make it? Is a red button better than a blue one should? Is the copy on the button more important? Is the shape of the button more important? Is the size of the button more important? I'm gonna run an experiment a month on each of those things and you're like, I'm really gonna learn what works or doesn't work on buttons. Great. Now five months have gone by, you've learned, you learned five things. That is it really worth spending five months on, like being very, very, very precise in what you've learned? And what we'd love to see people do is to be a little bit more holistic and use other people's experiments as building blocks. But, but even if you don't do that, is to be holistic in terms of thinking through like the ultimate delivery vehicle or what you're trying to change and put things together that sort of solve for the end user problem more dramatically or more in a more interesting fashion and assemble pieces together to run tests. Obviously it helps to do that when you're leveraging like the individual building blocks and you have good certainty on what you know. But the other flaw that I see people make, especially people with not enough traffic make, is to be very small in their experiments in order to maximize learning. But the learning doesn't necessarily maximize impact and you need the right mix of learning and impact.

I was going to ask you what can, were you about that? Right? The small changes versus you. More, more meaningful changes. You answered already because that was, that was kind of a question that happened. And I agree with you. I'm also all for, you know, especially when, when you say, like what people have done in the past, when you have precedence for things that marketers know already or, you know, previous tests, human behavior and all that. I think that only in a small cases makes sense to focus, as you said, on the placement of a button in the image or a video or, you know, one or two fields. I'd rather go for larger changes that have a point of view behind them. Right. And so you can compare those two different ideas. I like, I like, for example, that on web pages, right, when you are thinking about a different design for a For a page or for a whole website, you are not truly worried about the placement of specific elements rather is, you know, this is how we look like, this is the message that we convey right now and this is how we want to present ourselves now. And is the combination of all the, all those elements that the experience that users will have, what will make the change? So, yes, I definitely feel that that needs to be considered when you are doing those experiments.

For sure, yeah, there's a sweet spot again, because I think there's going too far in the other direction is also the other kind of error. You'll see a lot. And so anyone who's ever been involved in a full scale redesign will pretty much tell you that in many of those cases, the new experience that's kind of a massive redesign usually will end up with a worse result because you've taken something that was optimized and you've gotten rid of all the parts that were working and you've just replaced all those with new things that are based on assumption and not data. Right. And so I, I think there's a sort of a sweet spot. And the way that I always sort of think about it is sort of saying as you're rethinking a sales page, for example, in a B2B SaaS product, you could redesign the whole thing or you could sort of look at, say, okay, are we doing an effective job of conveying how it works? What are the approaches we could take to conveying how it works? And you're not just testing a button, you're not testing a sentence, you're not testing one singular image. Although don't get me wrong, those things do work. They make a difference. And you, you want to have the right combinations of those things, but at a certain point you're going to exhaust those and you want to sort of like look at the sort of adding, what are the approaches you can take to how it works? Should you have a section on the page related how it works, where on the page should it be? How should it be laid out? How do you, how do you work on that thing? How do you convey it? Do you convey in the first person? Do you use testimonials? Do you convey different elements? Those are the kinds of things you should, you can be working on by assembling the pieces together. So it's sort of think of it like a camera with the right aperture. And I think a full scale redesign is a bit like 52 card pickup if you have older siblings. Have you ever played 52 card pickup. My siblings introduced this to me. 52 card pickup is when your older sibling says, hey, do you want to play 52 card pickup? And you say, yeah, that sounds cool. And they, they basically take a deck of cards, they throw it in the air and they say, great, go pick up those 52 cards.

I haven't heard of that one.

You might be an older sibling if you have older brothers and sisters. You probably. At least my older brothers and sisters did that to me.

We'll do that.

So, so when it comes to playing 52 card pickup, that's what a redesign is like. You basically take everything in the air, you throw it and it's like, okay, now we're start over. There are times to start over. But it, the, the better approach is to sort of go like building block and with the right level of effort.

No, it is true because we, we need to don't, you know, remember that we work within constraints. Because for me, websites in general, not only landing pages need to balance the act between being different and attractive and you know, like improving workflows, but being also a little bit familiar to what's, you know, the rest. Because we just like with any other interface, right, Users expect things being in some places, right. Your menu is on the lateral part of, you know, like just because of the sake of. It makes no sense, right? So there are, there are certain behaviors that users expect from their websites and when you violate those usually will have the, you know, worse performance. So it is a balancing act of introducing new elements, but at the same time still being familiar for sure.

That's right. That's right.

I think especially applies to apps, right. Where I have seen designs where so, so different and exciting, all that, but you don't know what to do with them and you get a little bit of frustration out of well, how much use this app or, or any other, you know, like design or something like that.

I've seen that.

Now I want to cover with you also. What do you make of, you know, tools like yours are also probably improving the landscape on that sense. But I feel years ago for marketers to perform most of those tests was very complicated. Right. The technology requires someone that had a pretty good knowledge of even programming to set up some of those tests when you want to split the traffic or send, you know, a split test on emails. I feel that nowadays marketing automation tools and landing page tools are making more and more easier to. For people to run experiments. So I think that barrier of as you were saying, right. We don't have the time to Run the experiments is going down. I don't know what, you know, impact or changes. Do you see in the next few years on people being able to run more tests in a more easy way with less technical knowledge?

Yeah, it's a great question. When I started running experiments in the early 2000s online, it was a massive pain in the neck. It was a huge pain. You needed a team of people building the whole solution in house. Thanks to the likes of the optimize leads of the world, the unbounces, the Intellimize Adobe, like all of these different folks, it's now easier than ever to launch experiments. It's never been easier. It's not trivial, but it's easier than ever. So the good news is that launching is not the hard part. The thing you're going to see in the next little bit is that people are going to say, oh, we're going to auto, we're going to use AI to make things better. Right. And which I think is great. I we have a technology, we have a tool that we're using AI to help people write headline copy and to make it effortless. And it's really cool, it's really, really powerful. That's going to work on ads, that's where we have it, which is in a search interface, which is when people are running search ads. You can have the volume of traffic where you can use AI to write new headlines for you based on what's winning and losing for other people that you can do that. I think really, really positive about the challenge on websites and it's one that I think will be solved more and more over time with smart statistics. But it's a little bit of a challenge, is that it takes a certain amount of traffic in order to be able to run an experiment and to let those tools work their magic. You also need a certain amount of traffic. And so it gets really hard to do at a smaller scale. And so I think it's still going to be the case that it's easier than ever to launch, but it's still harder than ever to launch things that work. And so you still are going to sort of ultimately sort of bump up against the win rate is going to become the most important metric that you're going to see in the space, which is what percentage of things that you try are moving the needle. Because the most important gating function in all this is the number of experiments you're going to be able to run. And for just about everyone, it's going to be capped out in some way. Unless statistics can solve for it in a more profoundly interesting way.

Yeah, I agree. I tell my team, right. And clients all the time there's so many things that you can improve. But I oba, you know, I oblige by the Pareto law. Right. You want to improve, you know, your campaigns. What is, what is the campaign that is, you know, spending the most or driving the most traffic and within those campaigns or that within that campaign, what is the ad group or the keyword that is, you know, where you are spending the more, the most or driving the most traffic and the ads are getting the most traffic or impressions. Those are the ones that you want to start testing because one, you have the volume and two, if it works, it will have the largest impact. So I. And that extends to the rest. Right. So as you were saying on the website, probably your website will have one or two, three, four pages that either have a ton of volume or they are so relevant to the business in terms of, you know, moving the revenue impact that justify that. You start experiments even if the volume is not as high. Right. Like what you were saying about the, the demo page or the pricing page or the contact us page. If you have a landing page, even if it's not the, the number, the number one page, like the homepage, it is too relevant because that is the gate to your funnel. Right. So I think that as long as you go with that, then it will make the most of those very liminal, limited experiments that you can run in a year for sure. At each of the stages.

That's right. And you generalize the learning from where you have traffic to where you don't have the traffic. You can sort of try and draw the lessons to make up for the fact that you don't have as much traffic. You're right, yeah.

And extends to the rest, automation in general. I think in that part, for me, experimentation is not different from the problem or the challenge that some marketers have around automation. Right. Sometimes we get too excited about the technology itself and we want to automate everything that we do. When in reality you need to automate the things that are really consuming a lot of time or that are, you know, like consistent enough that you can use automation to provide something very, very close to what a person will do. Right. But if you have very low volume or you really need to add that human touch like will happen with AI, I think those are the cases where it doesn't make sense and you have to be smart about it and choose carefully.

Problem with automation is that you lock in on a system very, very quickly in a process. And so you're just making a process efficient, which is great. We as a company, when we're launching new things, it's, it's extremely painful. But we will do a lot of things that technology could do. But we'll do it manually first. Yes, we'll suffer through it. To really experience all of the different edge cases. It's very painful. It's, it's not fun. It leads to much, much better product and much better user experiences because you're running into those things in real time. Solving them manually and then figuring out how to automate them or how to solve for it in an automated way. That's what creates really cool experiences. But I do not. It's not the fun way to live. It's not the fun way to make product. I think it's the effective way to make product. But, but it's really painful.

No, but, but it's true that element of manually doing things. And I know that we are talking not right now about automation, but applied to experimentation. I remember one thing that I want to mention is that I draw from how science works around experiments. I. In my past life, right before being a marketer, I was, I was, I have worked for many years on manufacturing companies as an engineer. I'm a mechanical engineer by trade. And I remember in those days that for the, for all the product production line, it was a man electronic manufacturing company engineers wanted to run experiments and, and they fall into this trap of just testing something without a hypothesis, right. Without an idea of what is exactly what I'm trying to test. So I, I used to remember you like, well, there is physics around this, right? And I know it's a little bit silly comparing that to, to ma, to marketing, but in marketing there's psychology around human behavior, right? So it's the same with testing on, on you know, experimentation, on marketing that you will do in physics. If you, if you are going to run an experiment, there's gotta be some physics background there that you can do. You can use to at least create a hypothesis that will have that experiments within a acceptable range. Rather than just say well, I would just run different temperatures because I want to find the optimized spot. Well, you can, you can do some calculation and estimate where it's going to be and just test around. So I think that's another pitfall that is very important for markers to prevent, right. Is that we want to test with a purpose and also remembering that while you know, tools and machines and AI will render some recommendations or results. At the end of the day, what we do is based on human psychology and there are things that we know about there, right. That, that, that we need to remember before running experiments blindly.

I guess that's right. That's right. My friend Eric Reese, who is the guy who wrote the Lean Startup, I think it was he that, him that said that, you know, if you don't have a hypothesis, you're going to run a test and there will be an outcome, but you don't want, you're not necessarily learning anything.

Right.

So, you know, you're launching this thing. Did, did the temperature go up? Did the temperature go down? Did it, did it explode? Did it not explode? The real question is sort of like, what are you trying to learn? And then working backwards from that, from generalizable principle. So I agree with you.

Very good. I think this has been fantastic. Andres, I really appreciate you spending time with us and sharing this knowledge with our audience. I had a fantastic time. I appreciate it. And you know, if people want to know more about experimentation and your field of expertise or about the, the platform that you are building where they can learn more.

Yeah. So they can find our website at do what works IO and they can find me on LinkedIn and I, I like to pontificate and share, share knowledge there too. So you can find me on LinkedIn. I'm Andres Glusman.

Fantastic. Well, Andres has been fantastic. I appreciate it. I hope this is not the last time we speak. And yeah, thank you for being here.

My pleasure. Lots of fun.

Thank you, everybody.