#23 Online Experiments at Booking.com (with Lukas Vermeer)

In this episode of Data Framed, a Data Camp podcast, I'm speaking with Lucas Firmea, who is responsible for experimentation@booking.com in the broadest sense of the word. From infrastructure and tools used to run experiments, methodology and metrics that help people make decisions to training and culture that help people understand what to do. We'll be talking about how Booking leverages data science to help empower people to experience the world through the three pillars of of exploratory analysis, qualitative research, and quantitative studies. We'll also take a deep dive into the fact that data science isn't actually anywhere near as objective as you may think. I'm Hugo Bauen Anderson, a Data Scientist at DataCamp and this is Data Framed. Welcome to Data Framed, a weekly datacamp podcast exploring what data science looks like on the ground for working data scientists and what problems it can solve. I'm your host, Hugo Bown Anderson. You can follow me on Twitter HugoBownt and DataCamp at DataCamp, you can find all our episodes and show notes@datacamp.com community podcast. Hi Lukas, and welcome to Data Framed.

Hey Hugo.

Great to have you on the show. Really excited today to be talking about online experimentation@booking.com, talking about the role of the scientific method in data science and diversity in tech. But before we get into all of these issues, I want to first find out a bit about you. And I want to take a slightly sinuous approach by first finding out, what do you think your colleagues think that you do?

I think if you ask them, they would mumble, something, something, experiments, something, something related to experimentation. I love it at least. But I think practically speaking, people come to me when they have questions about experiments or about metric design or when they have had difficulty analyzing the results of their own experiments. So the way we're set up, we allow lots and lots of different teams to run experiments using our shared infrastructure. So I'm responsible for that infrastructure and the training and documentation around it and the methodology metrics that we use. So that's my real responsibility. But the end result is that people are doing these experiments by themselves and then when they get stuck, they somehow end up with me. So I think if you ask people what does Lukas do? Then they would say he answers my questions about experiments.

Great. So hence the something, something, experiments, something. But as you said, it's really the basis of what you do is structured around infrastructure and tools, methodology and training.

Yes, yes. So I try not to get in the way. I try to let them do these things as self service. As much as possible. And we're not a central authority, not an ivory tower in that sense. So we really try to enable and empower people to do these experiments themselves.

Great. And I'm really looking forward to delving deeper into what experiments look like@booking.com, but first, are there other things that you're responsible for?

Yeah. So more broadly speaking, I work with the wider analyst community that's very spread out within booking in the research community. So analytics and research are a little bit different from experimentation in the sense that analytics is usually more open ended, it's more exploratory and user research, I mean, more qualitative studies, trying to understand what users are thinking, what our customers want. And I work with both of those communities as well, trying to support them, seeing how they can collaborate with experimentation, trying to understand how with these three pillars, we can improve the customer experience.

Cool. Speaking to the research aspects, it sounds like, as we'll discuss later on, that you do really think about data science and the scientific method in the data scientific approach.

Yeah, I mean, so I'm not doing data science for fun. Right. We're doing this for a purpose. And our purpose here is to help, we say, empower people to experience the world. And so this is really about helping end users and trying to figure out what it is that they need to make a decision. And the other aspect of that is we're doing that by building a product. And so all of the analytics and all of the data science work that we do is all geared towards helping product development. So that's either through exploratory analysis, trying to figure out where to go, or a qualitative understanding why people want or need a certain thing, and then quantitative, trying to quantify what the impact of something is, trying to confirm that something has actually solved a user issue. So I see all of these things as being related to product development and customer experience, not as fields in and of themselves.

And I know you mentioned already that the hierarchy@booking.com is relatively flat, but how many people do you work with or are responsible for?

So I have no direct reports, so no one directly reports to me. But we have hundreds of people who are using the experimentation infrastructure and I do feel responsible for their behavior, so to speak. So the group I work with is, I think 1500 people, 2000 people, something like this.

Amazing.

It's a pretty large organization.

Absolutely. And you're also in that respect, involved in hiring, right?

Yes. To make this work, you need good people. Right. If you want people to have their own responsibility. If you want them to be accountable for their own results, you want to give them this power, then you need to have people that you really trust. So I spend a lot of time on hiring as well and making sure that we get the right people with the right mindset into this company. So that will be people who are really thinking about how does their work impact customers. So not doing it for the sake of the work, but doing it because they want to make the product better and they're also able to function independently. So not people who like to be told what to do, but they're also vocal and can explain why they do things and why they do them in that way and are willing to change their mind when someone challenges their approach. So I spend a lot of time on hiring.

Fantastic. It sounds like a really interesting culture. And are you hiring at the moment?

Oh, yeah, always.

Great. Well, we might circle back to that. But any listeners who interested and enjoy this conversation, please do get in touch with booking. So now I know what you do at Booking. I'd like to know a bit about your journey there. How did you get involved in data science initially?

Well, in high school, I was good at two things. I was good at computers and I was good at biology. And I was interested in biology because of the systems aspect of it. So I like to figuring out when a patient has a particular diagnosis, how do we figure out what part of the system is failing? How do we fix that? So I like computers and complex systems. And this was back in 2000, and I found the studies here in Utrecht, at Utrecht University, that was computing science with a machine learning or AI minor. It was called artificial intelligence. And this was before data science was cool. Right? This was when these conferences were still referring to machine learning or statistical analysis.

Yeah, I think it was before data science was even a term.

Yeah, yeah. It wasn't coined yet. That's where my journey began, with a fairly technical background in computing science. And then after I finished my studies, after eight long years, because I'm slow, I joined a consulting company. And I was working for a consulting company that was trying to get marketing departments to implement recommendation systems. And as part of the system, there was a control group. So there was the way for the system to figure out for itself whether it was being effective or not. You want something to compare against. If you're making recommendations, you want to know, do those recommendations actually add value? So they had a control group built in. And one of the questions I would get a lot was, can we please turn off the control group because we just want to give all of our customers the best thing. And that spoke to me as such a fundamental misunderstanding of the scientific method. I thought that was very striking that many of these companies, they're trying to apply these machine learning methods but they don't really understand the core foundations of data analysis. And so then at some point I ran into a man who worked for Booking at a conference and we started talking and he was really interested in my work and I was very interested in the culture that they seemed to have. So I joined Booking five years ago, mostly because I was fed up with companies that didn't really understand this idea of being open to change, being open to being proven wrong, and trying to build a product that actually provably helps people rather than going by gut all the time and just using data to confirm your initial beliefs. So I started building ranking systems and recommendations here at Booking. That was the first thing I did. And then I wasn't really good at that I think wasn't my forte. But one thing that I noticed is a lot of people were running experiments and I had opinions, very strong opinions about how they were doing that. And I was so vocal about that that at some point people just asked me, do you want to do that for like a full time job? That's about four years ago. And so since then I've been responsible for experimentation. Just because I was so loud. Yeah.

And even at a prior point in the trajectory, your trajectory that you described is realizing that experiments with respect to the recommendation systems for marketing weren't actually being done correctly. Right. With respect to the control group.

So the cool thing about that is that it' the tool was working right in terms of execution, it was executing that experiment correctly. The method was actually very sound. It was just that people weren't listening, they weren't using that data to then inform their decision. And I think when I go to conferences about data science, I'm always worried that all these fancy methods that are being proposed that they don't end up with anyone actually changing their mind. Then what's the point of doing data analysis if you're going to do the exact same thing that we're going to do anyway?

Right. And we're actually talking about making business decisions and improving our customers experiences and customers lives. As you stated earlier, you're not in data science only for fun. You're also doing, you're facing a lot of business challenges. So I'm wondering what are the biggest business challenges faced@booking.com?

Oh boy. So my role is quite technical, so I'm not really on the business side of things. But I think the gist of it is that we're the world's largest accommodations provider, so the biggest seller of rooms. And we want to help people pick the right place for them to stay. So we have 1.7 million accommodations on our site. That's a lot of places to stay. And we want to give people the information so they can then make a decision where to stay. Now that's already a challenge, but I think the other thing that's going on is that we also see that consumers want more than just a bed to sleep in. Right? We want to power them to experience the world. That means we need to help them find things that are not just a bed, but places to see things to do as well as places to stay. So those are two sort of business challenges. And I think on a technical end, we've been growing faster than Moore's Law for about two decades now. You can think of us as the biggest hockey sticked startup in the world.

And this is in term of number of transactions or number of customers or number of hotels or all of those.

All of the above, everything. So growth is growth, right? We're a marketplace. So all these things are related.

And how do you scale then?

So one part of scaling is a technical scaling, right? So it's a constant challenge just to make sure that stuff works. Just making sure that we survive summer and that all of our systems keep working is an interesting technical challenge. But for me, the bigger challenge is the organizational challenge. Our size as a company has also grown a lot. And like you said, we're a very flat organization. We like to give people a lot of power. We like to let them make their own decisions at the lower level. Right? So it's not the boss telling them what to do. The boss is telling them what to aim for. And then they execute. They decide how to get there. And scaling that is tricky because it's not just building more layers to the org chart. And you have to find ways to help people communicate. How do we share knowledge? How do I find out what other people are doing? How do I challenge them? So if I disagree with what they're doing, how do I escalate? How do we do training? When I started doing experiments, I could talk to all of my users. We were all on one floor here in Amsterdam, in our headquarters. I could walk up to everyone who was running experiments and ask them what was on their mind. And if I disagreed with any Decisions I could walk up to them and go talk to them. We're running so many experiments now, I can't even read them, all of them anymore. It's just going too fast.

And as you said, you have 1500 or 2000 people running experiments, right?

Pretty much, yeah.

Incredible. So you've identified three, perhaps four really interesting challenges. I mean, there's the matching problem of getting your customers the inform or your clients the information they need, the hotels they need. Then on top of that, there's actually giving them, as you say, empowering them to experience the world. Then scaling up, not just technically, but organizationally, which involve a whole set of sub challenges. What I'm interested in is how data science can help you and booking to solve these challenges.

How not? I don't think there's any part of booking that doesn't use data science. I want to shape or form of another. So the obvious ones are where it started, right? Working on ranking and recommendation systems to help people choose from this 1.7 million properties, trying to make sure that the stuff that we show them is relevant to them. That's a sort of prediction problem, right? Trying to predict which things these people like. And then on the marketplace, there's lots of interesting marketplace dynamics. Things like fraud, trying to prevent fraud, to protect our customers, but also our partners. Fake credit cards, of course, are an issue. And then we have our own customer service in house, so we don't outsource our customer service. All these people really work for booking. That also helps the sort of the feeling of belonging. And there, I mean, our CS operation is so huge that just predicting how many phone calls we're going to get in one of these 24 different languages that we support. And scheduling staff so that we have the right people available at the right time is an interesting scheduling and prediction problem, with the added problem of many of our agents speaking multiple languages. So that scheduling challenge is actually a very nice one. And then of course, analysis and experimentation, sort of more supporting decision making at all levels. So I'm more concerned with the infrastructure that allows people that do product development to make decisions. But obviously leadership also needs data analysis to figure out where to go. Yeah. So I can't really think of a place where data science doesn't in some way affect how we work.

So, Lukas, that provides a great segue into what I'm really interested in talking about next. And it's the explicit role of online experimentation in decision making at booking. So can you tell me a number of concrete ways in which online experimentation is Used to make decisions at booking.

So in the end, we're interested in making the product better for customers. Right. We're in it for improving the customer experience. And experiments are a great way to do that in a measurable way. So essentially you come up with a hypothesis about a customer problem that you're trying to solve, then you propose a solution. You say, I think customers are struggling with this, but I can fix that by doing something. And then you use an experiment to validate that these solutions or features help customers find what they need. So that's where experimentation comes in. It really comes in at the end where you're trying to figure out, did this thing actually work in the way that I hypothesized that it would work. And so what that does for us as an organization is that it ensures that these individual teams are empowered to function autonomously, that they can make independent decisions, because everyone can look at what their intent was and what the outcome was. And as long as they trust the shared infrastructure, they can also trust the results of that experiment. So in some sense, this experimentation helps us validate that the changes that we make really help users. But also it helps us have this flat organization where these individual teams are empowered to do that autonomously.

So if I'm a customer, what type of changes could I expect to see if I'm a subject in one of your experiments?

Nothing that you wouldn't see if we were just not running experiments and changing product. Right. I think any product on the Internet needs to evolve and needs to adapt to customers needs. So you do things like, it looks like this button has low contrast. For instance, we have a lab here, user research lab, where we bring people in and sometimes we notice that a certain piece of information or a certain button is really not obvious to people. You can see this in eye tracking studies, or you can see this when you ask them, did you notice this? And they say, no, no, I didn't see this. So we have some indication that this is not clear enough. And then you hypothesize that this information is important and people are not seeing it. So to help users, I must make it more obvious. And so you do things like increase the contrast or make it bigger. I always joke that we should make very important stuff blink, but apparently that's more like 90s Internet. When I was doing web design, that was cool. Marquee and blink were the new tags at the time. But you come up with some proposed solution for how you would help users find this information better. So then you implement that but that's no different from implementing a normal product change. The only thing that's different here is that now we're going to validate that when we made that change, users actually responded differently. So that when you made the button more obvious, did more people actually hover over it? Did more people click on it? Did that help them make a decision? Did they appreciate that decision more than decisions they were making before?

And also you want people to do more of the thing that makes business better and the customer more satisfied.

Right. Well, so for us, luckily those two things are very much aligned, I think, business wise. And this, I don't want to get into the philosophy of the value of the capitalist society. Right. But in some sense all companies and their consumers are at odds because the business wants to make money and consumers want to spend as little of it as possible. But we are a travel company. People want to travel. People like to go places. And I think it's in our interest that they go to places that they like so that they come back. And so we are selling a product that makes people happy. We want them to find the product that makes them the happiest so that long term both parties get the most value out of it.

I know you're very passionate about experiments. I think you've even told me other people may call you very opinionated about experiments. A lot of the time when people first talk about online experiments, experiments, they talk about a red button versus a green button.

Oh God.

And I'm wondering what your take is on this, you know, inadverted commas paradigm of online experimentation.

Yeah, I think it's all this all started with the 41 shades of blue article that Google shared where they found that one shade of blue was actually better than the other shades of blue. I find that such an uninspired way of thinking about product that's so narrow.

Tell me more about that.

So I mean, look, I think when people talk about website optimization, they're thinking of this as there's a website that has a very clear purpose, I can very clearly measure it, and I'm just going to optimize that space. And even the term website optimization suggests that it's an optimization problem. And with a background in machine learning, optimization problems are a very specific thing. But for product development, none of these things are true. It's very difficult to really get down and measure customer satisfaction, for instance, or long term value. These are things that are not easily quantifiable. So thinking that you can use this to optimize is an illusion. I Think. But moreover, when you're building product, you're not operating within a confined space. This website has lots of different things on it. And if you think only about changing the color of a button, then you're not really expanding your product, making it do better things that customer needs. So, for instance, there might be information that people need to make a decision about which hotel is the best for them. Let's say people want to know whether they can bring their pets. And I'm just making this up, so people want to know, can I bring my pet to this place? Yes or no? Now, if you think of website optimization as changing the color of the buttons, there's no color that's going to help people understand whether pets are allowed or not. Just doesn't work that way. What you need is, you need to try, probably using qualitative methods first, try to understand what it is that people need. Then think of features that you can build that serve that need, and then you validate whether that need is served. That's a much broader view than just changing the color of buttons. And it also requires that you start writing down more clearly in the form of a hypothesis. What is the need that you're serving? What is the problem that you're solving? And so the example with the two button colors, I use that a lot. And I said, look, website optimization is saying, I'm going to change the button from yellow to blue and then see if the magic number goes up. That's it, that's website optimization. But it doesn't help you at all understand what's going on. And if I said, well, instead of changing the button to blue, let's change it to green, you would be guessing which one of these two would be more effective. But if I said, we know from eye tracking studies that people don't notice this button, and we believe or we suspect that is the case because the contrast between the white font on the button and the yellow background is too low for people to distinguish the text. And we think that we can resolve this contrast issue by changing the background color to be darker so that the white text stands out more brightly. And so we will make the button blue, and then we will check whether people actually hover over this button more, whether they click on it more, and whether they eventually find what they need more. Now, the implementation of this is exactly the same, right? I'm still changing the color of a button, but the intent is so much different.

In that sense, you're really testing a hypothesis that you have for other reasons as well.

Yes, yes. So there's some data to back this up. Right? We already have qualitative data here, and we really need it because we're trying to understand, we have some theory about how this implementation might actually work. So the thing that changes, not the implementation, but the thing that changes the things that you can now do as a human. And so one of the things that changes is that if I said, well, if instead of blue, let's make it light green. Before, with the website optimization approach, you would be guessing which of these colors was better. But now they've told you that the intent is to increase contrast. You look at these two colors and you go, what? Light green does not at all increase the contrast, so it doesn't even get to the point. So then you reject this potential implementation by understanding what it is that it's trying to achieve in the first place. And another thing you can do is actually come up with alternative implementations, so not just reject them, but you can also say, well, what if instead of changing the background, we change the color of the font? What if instead of changing the contrast, we change the size, we make the button bigger? What if we make it blink? So you can propose alternatives based on understanding what it is that the thing is trying to achieve?

Yeah, absolutely. So what you're really speaking to is making sort of your experiments more robust and being more mindful about your assumptions and why you're doing what you're doing.

I'm talking about I'm in it to help customers. And to help customers, I need to understand what I'm doing to my product. And this is not an optimization problem.

So I'd like to keep our focus firmly on online experimentation, but I'd like to actually take it from a slightly different vantage point. And I'd like to know people who use the infrastructures that you build, people running online experiments, what they actually do, do they need to have a strong statistical or computational foundation, or do you productize everything to hide a lot of the things that are happening in the backend?

So here you mean. Yeah, so we try to attract away a lot of the elements of the method. There's similar products out there. You can think of VWO or Optimizely or Google Optimize. These are all Conductrix, one of my favorites. These are all products that do very similar things. They allow you to set up an experiment without understanding a lot about what's going underneath the hood. One ways in which we are more embedded or more tailored towards the book in the comm culture is that we the way our product development is set up is we have lots of small heterogeneous teams. So teams that have different skills in them. We don't have like siloed it and marketing. We put those people together to build a product and so we don't really need to have these sort of self service features for setting up experiments that a lot of these companies provide. So it's not point and click, it is an API because every team will have developers on it anyway. So that's not the challenge. And on the flip side, we spend a lot more time on thinking of ways to democratize the decision process itself. So how do we make sure that people have all the information that they need to make a decision? So that's providing them with not just statistics but also guidance on method. So one thing for instance, power calculations, figuring out how long an experiment needs to run. That is something that we try to really make self service. Let people do that independently, but also just let them stick to it. So it's one thing to do power analysis, it's another thing to then run your experiment for exactly that duration. So the tool is very much geared towards self servicing the method.

We'll jump right back into our interview with Lucas Vermeer after a short segment. Now it's time for a segment called Blog Post of the Week. I'm here with Spencer Boucher.

Hugo, what's up?

Hey Spence, I've been thinking about data science in the cloud a lot recently and I found an interesting post by Kunal Jain on the website Analytics Vidya.

Okay, what's it all about? Hugo? What's this post called?

It's called Running Scalable Data Science on the Cloud with R and Python and we'll link to it in the show Notes. Now I have to warn you, the post is from late 2015, so it's not necessarily up to date with respect to all the details, but it provides some great motivation for why we should all be interested in and paying attention to the possibilities and realities of cloud based data science.

Do tell Hugo.

So the post opens with the point that complexity in data science is rapidly increasing.

In what ways exactly is it rapidly increasing?

Well, it makes the point that complexity in data science is really driven by several factors. The first is increased data generation, that the world around us is collecting and generating more and more data as time goes on.

Absolutely. We talked about this in our segment on Big Data back a couple of months ago. In addition to having more data, the cost of actually storing that data is going down. So there's Just no downside to collecting more and more.

Exactly. And that's the second point that this post makes, the low cost of data storage. And the third point is cheap computation, foundational power that as we go on, we can do more and more computation. And this gives rise to more and more complexity in the data scientific process.

Yeah, I know some of my favorite algorithms. There's a lot of Bayesian methods that, you know, just didn't used to be very practical, but now we're starting to see them get used more and more just because of this cheap computational power that's available to everybody.

Exactly.

But so, okay, even given all this increased complexity, why else would we want to run data science on the cloud instead of on your laptop?

Well, one really important reason is to have your data science scalable. And what I mean by that is, as your data grows and grows by potentially orders of magnitude, if you're doing it on your own system, you'll need to be buying more and more hardware and you'll need to keep changing your hardware setup. If you're running it in the cloud, your data science work is actually scalable as your data grows.

Yes. And this is exactly what you want for like a big one off project, for example, where you don't have the resources to make a huge capital investment, but you need a lot of resources for just, you know, one quick project.

Absolutely. And another great reason for doing data science in the cloud is collaboration. If you've got a lot of people working on a particular project or working on the same data set, whatever it may be, if it all exists in the cloud, you can all have relatively straightforward access to it.

Yes, exactly. So sharing code goes a long way towards reproducibility as well. So you don't have to get requirement files and versions of software synced up on everybody's machines. One great example of this, I think, is like the Jupyter notebook server, where you can actually have a whole team of data scientists doing data science in the cloud on a remote server with the exact same environment across an entire company or even multiple institutions.

Absolutely. And RStudio server does a similar thing for the R ecosystem as well.

Yeah, exactly.

And this also speaks to the last point that the post makes, is that when we have cloud based data science infrastructures, we really have this larger ecosystem then for machine learning system deployments. So some cloud services, like AWS Azure, provide complete ecosystems to collect data, run your models and then deploy them.

Okay, so I'm sold on this. What are the options for getting started? If I've never done this before.

So the big players, at least in late 2015, were AWS and Microsoft Azure. In this blog post of the week, Kunal also mentions IBM, Bluemix, Sense IO and Domino Data Labs, among others.

But 2015 was a long time ago in data science years. Hugo. I know this is really just the tip of the iceberg compared to what we have today. So what else is out there?

Well, Spencer, I hate to be a master of suspense, but you'll have to wait until next week's episode to find out. In which I'll be speaking with Paige Bailey, Senior cloud developer advocate at Microsoft. I'll be chatting with Paige about data science in the cloud as it is today and as we move into the future.

Okay, well, that sounds pretty exciting, Hugo. I will definitely be tuning in next week.

I definitely hope so. After that interlude, it's time to jump back into our chat with Lucas. So we've spoken a lot about the data science capabilities that you guys leverage@booking.com and the role of online experimentation. I want to step back a bit. People think of data science as a. As a modern superpower. And as with all, all superpowers, there's a lot of hype and a lot of buzz and a lot of things that a lot of expectations that won't be met. So I'm wondering, to your mind, what can't data science do at booking or elsewhere? Or what is the limitation, the event horizon of the possibilities of data science?

Huh. I think a lot of stuff you read about data science concerns itself with prediction and correlation models. There's this famous Wired, I think it was Wired said that at some point we won't need theory anymore, we just need more data. Wow, that's just so at odds with my own experience, my own worldview.

Sounds like nonsense.

Well, that's a bit extreme. Say that. I think there's lots of things where predictions are actually fine and you don't really need understanding. If you're trying to predict what a consumer will buy, then I don't really need to understand why you like this particular type of hotel. So for recommendations and ranking, I don't think we need understanding.

Not necessarily. But you do need some sort of theory. You can't just throw data at a question without understanding how to interpret the question, interpret the data, interpret the answer, and build some sort of cognitive model as well.

Yeah, absolutely. And so this is what I call, or have called in the past, philosophers and telescope builders.

Right.

So the thing that I think is missing from the data science field or Thing that's not emphasized enough is the need for people who think about the causal connections between things. Understanding rather than just predicting, constructing theories, thinking about what it is that I can learn with this method and what are the things that I cannot learn. What are the blind spots? What are the. What is the white space? So those are the people I call philosophers because they think about what is the meaning of what we are finding. So the other group I think are undervalued are the people who build telescopes, the people who work on the telemetry, people who think about how to measure things. It's sometimes difficult to even think about what data do I have, what metrics can I build? And this is a. A field or challenge in itself, especially on the Internet, where a lot of the stuff that we want to know is actually happening on your computer and we don't have access to that we can't see you. You have. A lot of the consumers feel that they don't have a lot of control over their own privacy and such, but they do. We are a remote party, right? We can't really see that much. And so telemetry is hugely important. The example I often use is that the heliocentric model, the idea that the Earth was revolving around the sun, that was 2,000 years old before someone built a telescope powerful enough to make some observations that actually corroborated that idea so much that people rejected the geocentric model. So it wasn't that this theory wasn't around that people didn't think of this before. It was just that you need someone who can build the infrastructure or the telemetry that allows you to measure the things that you need to support that. I think that's something that's hugely undervalued. People who can really think about what data do I have, what data is missing? Missing data is a huge issue.

And do you see yourself as a philosopher or a telescope builder?

I should have seen that one coming. Yeah. Well, considering that I came up with those two roles, I would say I'm more of a philosopher than a telescope builder. Wouldn't you?

I would, but the rest of our conversation has firmly revolved around your role in telemetry and building telescopes and building product and infrastructure for online experimentation. But this is great because I think actually where this discussion is going is a movement from the first half of this. This conversation really was about the telemetry. Now we're moving kind of into the philosophical implications of data science, which I think are very much missing from the conversation. And I'm not really talking about metaphysics per se, but I'm talking about, you know, thinking about the scientific method, the implications of everything we do and working through all of our assumptions and just things we do on, on a daily basis. And I think that's a really nice way to frame. I know there's a paper that, that you love and you've spoken about, called many analysts one, one data set. And I'd love for you to tell us a bit about this paper because it makes clear how small variations in an analysis at different points can actually affect results. So that different analysts can get totally different results. So I'd love to use this paper as a starting point to really talk about the role of diversity in data science and tech as a whole and to get your opinion on that.

I love that paper. It's so good. I recommend it to, it's on an open source foundation and we'll put it.

In the show notes as well.

Oh, excellent. Yeah, yeah. So I recommend it to any analysts that I speak to, people who are used to doing data analysis because I think one assumption that people make a lot is that data is objective and data points are already theory related. We can talk about that later. But the process of analysis itself is a subjective process and I don't say that lightly. This paper actually shows that this is the case. So as an analyst you feel that you're following the data, that all of the small decisions that you make make sense and that they're inconsequential to that objective decision that you reach in the end. That's not true. There's this concept of the garden of forking paths, which is this idea that while you are making all these small decisions, you're actually meandering towards a particular decision. But that's not necessarily the only decision or the only conclusion that you could have ended up with. If you had made minor different decisions along the way, you could have ended up with a completely different conclusion even on based on the same data. So in this paper they actually tried to show this by taking a fixed data set of referees whistleblowing for different players in the soccer league. So we have data on for each player and for each referee whether they were given a yellow or red card, which is what happens when you make an offense. And they also included the ethnicity of the player. And then asked 30 something groups of analysts, so individual groups of analysts, they asked them, do you think that referees or is there evidence that referees are ethnically biased? Yes or no? So can you find any evidence of bias and now the interesting thing is that even though the data set is fixed here, so every group was given exactly the same data. And the question is what seems like a pretty straightforward yes, no question. You get 30 something different answers because each of these groups decided to analyze the data in slightly different ways, bucket it in different ways, model it in different ways, use different techniques. So they end up with I think a 60, 40 or a 70, 30 split between yes and no, not quite 50, 50, but you know, close enough. And then what they did is they, they said to these groups, now that you've come up with your answer, please discuss amongst yourselves the pros and cons of each of your analyses. So they were actually allowed to discuss amongst themselves and between groups the different approaches, and then they were allowed to adjust their analysis, change their, change their approach, and relatively little changed. Even though people now knew that it was possible to come up with a different conclusion, they didn't really change their own.

And there are a lot of steps in there that are actually incredibly scary for us as scientists and data scientists and people trying to make business decisions, right? Not only, of course, that you can give the same data set and have what a lot of people presume are objective analyses occur, but then when you get them in a room together, that you're going to get people being pretty stubborn, which is something I can relate to. Don't get me wrong, I'm not claiming to be the most flexible.

Everyone has their own confirmation bias, right?

Everyone.

Once you draw a conclusion, it's difficult to change someone's mind. I think one way to avoid this kind of thing is one way to avoid this thing that made me think a lot about this for data science in general is that in experimentation it's actually common practice to pre register your analysis before you see any data. You describe exactly what you're going to do, you describe how you're going to analyze the data. And so this is what a hypothesis essentially is. Then you run your experiment and analyze the result, and by pre registering that analysis, your analysis is not influenced by any data that comes up after the fact. So essentially you avoid the garden of forking paths by laying out the route you're going to take before you start your journey. And that's not always possible, right? For confirmatory analysis and when you know what the data is going to look like, that's a possibility. But sometimes you do exploratory analysis and then it's a lot more difficult to try to sort of describe what steps you're going to take. So another approach that I think people in risk analysis have already adopted is this idea of bootstrapping the analysis by having many analysts perform independently the same analysis. So you get not one analyst or one group of analysts to try and answer this question, but you get two or three or four trying to answer exactly the same question and you compare the answers you get. Now you mentioned diversity, but I think that's one of the reasons that diversity is so crucial, because if all of these analysts are like minded, then you won't actually get a very good bootstrap. It's like you get a biased sample. Right. So you want that group of analysts to have different backgrounds, to come from different parts of the world, to have different trainings, different experiences. Yeah, different experiences, different views of the world, maybe even different political positions. Right. So that you get a very broad sense of what potential answers to this question could be. Now, of course, that's a very expensive way to answer the question, right? Because now you need more than one analyst. So I'm not saying you should do this for all questions, but certainly very, very important business questions. I would be very hesitant to get one group of analysts or one person answer these questions because you want to have some sense of how, how sure am I that this is the only correct answer to my question. And when I propose this to analyst, I almost always get pushback when I push this, suggest this to data scientist. The idea that data is objective is so persuasive and the confirmation bias is so strong that people don't generally accept this. Now maybe this is true for other people, not for me.

Yeah. And people generally don't like to admit perhaps that they've done something wrong or thought about something in a manner which then looks like it could be something different.

Right? Right. And in some sense this gets to their skill. Right. It gets to their, you are effectively challenging then their ability to objectively assess data. And I think there's other disciplines where this is less of a problem. Think about developers, right? If you say to a developer, we want to make sure that we test all the code before we roll it out because we want to make sure there's no bugs. There's very few developers that would argue that they don't ever write bugs and it's because they've been confronted with stuff that they wrote that doesn't work several times and made them realize that they're not perfect. And so that leads to acceptance. And I think the same some somehow needs to happen in, in data science where unless we show people that their analysis is not the only way to come to an answer, and that maybe their answer is not as objective as they as they thought. It's going to be very difficult for people to accept the idea that they might be wrong.

So I want to focus for a second on your statement that the more diverse the group of analysts, the better. And I'm just wondering, is this something which is stated explicitly and implemented at booking?

Oh, yeah. Diversity gives us strength. It's one of our core values. We very firmly and explicitly believe that we want a diverse as possible group of people to work on this product because it will help us make better decisions, because it will help us understand customers better. We're a travel company, we're global, so we want as many voices as possible to be represented. So as an example here, the office where I'm sitting here in Amsterdam, it's not a very large building. It's in. It's right next to Rembrandt Square, which is really in the heart of Amsterdam Smack mill. In this relatively small building, there's 100 different nationalities, which I think is enormous. And that's not accidental. That's on purpose. We want as much diversity as we can because we think that actually is a core strength.

That's really interesting. And I think that type and magnitude of diversity, coupled with the flat hierarchy that we spoke of earlier, would make for an incredible conversations and dialogue within the company as a whole.

Oh, yeah, our lunch conversations are awesome. I mean, every day you meet different people from different parts of the world who think about the world completely differently. And it's also very humbling experience. Right, because you think of your own upbringing, your own environment makes you who you are, and you think of that as the only right way almost to be. So it's almost like the garden of forking paths. There's also a garden of forking paths in upbringing, in education, in how you live your life. And actually, that diversity is something that really gives me a lot of energy. I love this group of weird people that are so alien to me. It's great.

Awesome.

One other thing about diversity is. So we wrote a paper about our experimentation infrastructure, the underlying stuff that we use to run these experiments. And one of the points that we make is that we actually have two completely independent data pipelines that are trying to measure the same things, but in different ways. And they are maintained and built by different people. And those people are, of course, allowed to talk to each other, but they hardly look at each other's code. And that's exactly this idea of bootstrapping Right. We acknowledge that the experimentation infrastructure we have will, and this is not a maybe, but it's a definite. There will be edge cases and bugs, and it's very difficult to uncover them unless you have something to compare against. And so we have these two independent data pipelines and we compare the results of those for every individual experiment so that we can say with more confidence, here are the results of your experiment as corroborated by two independent sources.

That's fantastic. So I think this has been a phenomenal whirlwind tour through all of the online experiments you guys do. At booking.com, the infrastructure you've built out. The role of science in data science, how you consider diversity, particularly in a developing, I suppose, ecosystem of roles of data science, which requires such a. Such diverse opinions and backgrounds and experiences.

Yeah. So if you're a factory and you want people to do the same thing over and over and over, then you don't want diversity. Diversity is bad. You want people to be the same. You want homogeneous group. But if you're doing innovative things, if you want to learn things, if you want to expand, if you want to grow new things, then you need diversity. Homogeneous group in that case is actually harmful. There's lots of science to back this up. I can send you another paper, one of my favorites, where they looked at the decisions that are made by groups. When you take lots of people who are really, really, really good, but they're really good at the same thing, versus a group of people who are not that great, but they are more diverse. So their expertise is not as centered as the first group. And even if that expertise is centered on the topic at hand, the diverse group still does better.

Amazing. I'd love to see that paper and we'll share it in the show notes as well. So before we close out, I'd like to know if you have a final call to action for our listeners out there.

I think we talked about this a lot already. So if you want to build a better product, if you want to help customers, then you have to learn and understand what they need. It's not an optimization problem, it's a learning problem. And you and the people around you are probably not representative for the audience that you have. Facebook has this saying of failing fast. It's a bit of a misnomer because we're not really after failing. Failing is something that happens along the way, but it's not what we're after. We are after learning and in. And improving. And we acknowledge that in that process, we will do things that are not working because we are learning. We have to learn from our mistakes. So when I joined booking, all of this thinking hadn't evolved as much. And so I just jumped headlong into the first problem that I was given. And that was that on the main site of bookings, when you type in booking.com and you land on our website, we have a few destinations that we recommend to you. And under each destination, we had five hotels for that particular destination that we thought were relevant to users. We're trying to help people want to explore, find places without having to type. Because the other thing you can do on this page is type where you want to go. But if you don't really know where you want to go, we want to help you explore. One of the things that my team that I joined was, was working on was finding the optimal ranking for the hotels that were being shown. So, for instance, you show as a destination, you show Paris, which five from the 3,000 hotels that we have in Paris do you show? And so for about six months, we tried different ways of ranking those hotels based on academic papers that I found. And what we tried to do was things like, should we increase the diversity of the hotels we show so that people get a better understanding of what the city is like? Or should we use context about where they are from so that people from China get different recommendations than people from Europe? Or should we use your past history so that if you're logged in, we can use hotels that you've stayed at before to change? So you can think of many ways to improve the ranking. But what we were using at the time as a success metric was the classic website optimization metric of how many of these people then eventually book and stay at the property, so how many people actually stay at the hotel? And we were not measuring any of the in between steps because we considered them to be irrelevant. We didn't really look at the behavior on the site itself until at some point, I found a paper that talked about a ranking algorithm that required us to know how many people clicked on each individual hotel. So that's not something we had measured up until then because we didn't think it was important. But then we started measuring how many people clicked on each hotel and surprise, surprise, it turns out no one clicks on the hotels. So that immediately challenged my core assumption that these hotels are helping people find hotels that they like and then book them. Because all I had been looking at was whether they booked them and stayed, not whether they were getting to those hotels through Clicking on them. So now that sort of challenged the notion that this ranking was actually important. And it made me realize later also that there are multiple layers of assumptions being made here and that I was only challenging, like the superficial top layer, which is what A B testing, which is what website optimization, which is what changing blood color seems to be about, is challenging those superficial assumptions, when what you should really be doing is looking at what are some of the core assumptions that are being made that I can challenge to make sure that I'm spending my time on things that customers value. And so in this case, the layer was I was assuming that I can improve the ranking. And that assumes that the ranking is actually important, that it matters which hotels you show there. That assumes that the feature itself is important, that actually showing hotels there is important, that assumes that anything that you show on that page is actually important, because if people only go there to search, then everything else you show there is irrelevant. So these are layers upon layers of assumptions and I'd have been only challenging the outer layer. And as soon as I realized this, we challenged one of the lower layers. And what we did is we removed, we hid the hotels themselves. So rather than trying to optimize the ranking, there are now no more hotels. And what happened is that nothing changed. No one cared. Users were not clicking on them. Well, now they're not there, they're still not clicking on them. There was no impact whatsoever.

That is a wonderful demonstration of how many assumptions we actually implicitly and subconsciously make. I mean, it's assumptions all the way down, right? And you need to go down and figure out.

Yes, exactly, it's turtles all the way down.

Yeah, yeah, exactly. And it's the garden of forking assumptions.

And I wasted six months of my time, right? Me and my team, we wasted six months of our time, like scraping the superficial assumption. And so the irony is that actually hiding these hotels was the easiest thing we had ever done in those six months, right? All these ranking algorithms, they took days or weeks to build. Whereas just hiding these hotels, challenging that core assumption was like a five minute change. And the way we do continuous delivery, like the moment we decided that like an hour later this test is running, you can already see it's not doing anything. I think I could have come in at booking and within the first day, I could have shown, if I had thought about this upfront, I could have shown that there was no value to this product and removed it and then spent six months doing something else. But instead, because I wasn't thinking about it, Because I wasn't challenging those assumptions. I wasted six months of our time, they didn't fire me. So that's good.

That's something. Otherwise we wouldn't be talking right now.

Yeah, exactly. That's something. But it did make me realize that oftentimes when we're trying to improve a product, we're making many unspoken assumptions and actually writing those down can be super, super important. And that's, that's a philosopher that was talking about.

Yeah, exactly. And I think it's very telling that you've learned from such a mistake, you've been improving from it, but also that you're communicating about it so that we can tell all the listeners out there to try to uncover what assumptions you're actually making and to challenge them.

Yeah, you know, the, the weird. Like sometimes people say, well, you can only use a B testing to, to test small things. And, and I think this is wrong for, for two reasons. Right. One is that you can actually ab test large changes, and we do. But the other is that it's not necessarily about how big the change is, it's about how important that assumption is. So in this case, hiding those hotels is a one line change. It's a bunch of CSS that hides them, does nothing else. That's a very small change, but it has ginormous impact on how we allocate our time and how much we can help customers. Because in the end, we're still trying to improve this product. I'm not just clocking my hours trying to not get fired. I'm trying to make changes that help customers. And to do that, I need to challenge those core assumptions first so that I can figure out which things to focus on. I think this is true in many businesses, not just for booking.

Oh yeah. And it's even true in basic science research. I think my listeners are probably sick and tired of me telling this story now, but I've seen grad students in biology, which is my former incarnation, pipette for four years and do 10 times as many experiments as they needed to if they'd realized how to actually perform the experiments to test the hypothesis that they'd made.

Right, Right. And many cases, these core assumptions, they are not just important for you to decide whether to proceed, but you also expect to see much larger effects there. And so the irony is that once you uncover these big, big assumptions, and it's easier to challenge them and it's easier to detect an impact. So it's not like it's. The difficult part here is the philosophy part. The difficult part is thinking about what are the assumptions that I'm making and then challenging them and being open to being challenged.

Very much so. Lukas, it's been an absolute pleasure having you on the show.

Thank you. It was great. Love to talk about this stuff. You mentioned at the beginning you were excited to talk about experiments@booking.com to me. That's my bread and butter. I do this every day. I love it.

Fantastic. That's why I invited you on.

If you have any listeners who want to come join me and come work for booking and talk about experiments all day, then by all means send them my way.

Absolutely. Thanks for joining. Our conversation with Lucas about online experiments@booking.com we saw how data science impacts every part of business decision making at booking, from ranking and recommendations to predicting marketplace dynamics and fraud to customer service operations. Lucas made it clear that the common trope of data science for optimization can miss the point, and that building better products, for example, is not an optimization problem, but rather a learning problem. And if that's not enough, we saw that data science and its results are far from objective and that the output of analyses is also a serious function of all the micro decisions that an analyst makes. Two concrete ways of dealing with this are pre registering your methods and bootstrapping your workflow with several analysts as we see happening at Booking. Make sure to check out our next episode, a conversation with Paige Bailey, a senior Cloud Developer advocate at Microsoft. Paige and I are going to talk about all the wild new exciting developments concerning data science in the cloud. I'm your host Hugo Bown Anderson. You can follow me on Twitter Hugo Bown and datacamp at datacamp. You can find all our episodes and show notes@datacamp.com community podcast.