What a former Booking.com product leader says about testing | Kristina Gibson (Director of Product, Booking.com)

Like, if I already have my dates, why would I necessarily show something that's sold out? And I think there's a couple of reasons for that, but one is it showed that if you show sold out properties, it in the end gets people to convert more in the end. So I think very early in the funnel, it's creating a bit of a scarcity effect that New York City during your dates. You know, this is a property you can't have. Kristen.

Oh man, I want it more.

You want it or you do, but you can't have it.

Hi everyone, I'm Kristen Berman, behavioral scientist and CEO of Irrational Labs, and this is the Irrational Mind. This podcast is for product marketing design people, people who want to know more about how their customers think and get inspired to build and grow products that work. In today's episode, we talk to Christina Gibson, former Booking.com director of product. Christina is an experienced PM. She was at Intuit, then Mint, Eventbrite, Lyft, Booking.com, and and now leads product at Micromobility Company. Booking.com is one of the largest online travel agencies, but in the product and design community, they are most famous for running experiments. In this episode, we get tactical with Christina and talk experiment results. Do emojis vs images work better? And why? What's the best description for a hotel breakfast? What weird amenity is listed on the site that drives page over page conversion? We'll also chat about the pros and cons of relying solely on experimental results to drive product innovation. Quick aside here, booking.com mainly relies on your revealed preferences to optimize their site. What are revealed preferences? So yes, you can say you like salads, but you eat pizza. Pizza is a revealed preference. You say you like to read, but you're on TikTok. This is your revealed preference. Click Data is a revealed preference preference. Real preferences are helpful, but they do have a blind spot. The blind spot is called latent demand. Latent demand is kind of the opposite of revealed preference. So latent demand is basically people who want to buy, use, click on something but it's not on the shelf or in this case, the website. You never know about their demand. For cities without E bikes, there is latent demand. People don't know it's an option, otherwise they may choose it. When Airbnb launched a section of their website called Experiences, they were guessing there was latent demand for travelers to find unique and quality activities like beekeeping and making local mochi. Travelers didn't say they wanted to learn beekeeping or how to make mochi. But Airbnb saw the opportunity. Building for and around a hypothesis of latent demand requires product teams to take a bet, take a risk. Airbnb experiences hasn't been a grand slam, but but it is an interesting business. My hunch is that when a company is doing a thousand concurrent tests, it's hard to launch something without data. Big swing pushes will always have less data than incremental innovation, and thus possibly harder to justify. And that's why I'm so excited for today's episode. We give you an inside look into the fascinating and complicated relationship between data, latent demand and revealed preferences for booking.com what we don't talk about in the episode is ethics. As you listen, you'll likely be whispering to yourself about how this doesn't feel quite right to use manufactured scarcity to drive our behavior. And I kind of agree, but also want to acknowledge that travel is a big decision. Personally, I am terrible at optimizing my travel decisions and I take forever to do it. My husband now does 100% of our booking and even for him I have to put a timer on the exercise so he doesn't rabbit hole into all the options. So it's not unreasonable for a website to include tactics in their design to help you make a decision. It can feel icky if you haven't requested the help though. So it's kind of okay for me to give my husband the timer because he has consented by marrying me, but it feels weird for booking.com to give it to me. The messenger matters, but it doesn't take away that big decisions with high uncertainty are hard and we hire companies like booking.com to make them easier. Ethics, by the way, is really an incentive decision. The incentive for booking.com is to improve our conversion. They want you to book, and thus their UI decisions are a reflection of this incentive and maybe some of our underlying human tendencies. If we want to improve ethics, we need to change their incentives. I'd for one pay for a membership fee to have a travel site know all of my preferences and routines and serve up personalized offers. And this would help booking.com have a longer term view of my lifetime value. But regardless, this episode is a masterclass in showing how small changes are drive real impact. It's not one big change, it's multiple darts thrown over the years that really move the needle. And on a more personal note, my day job is CEO of Irrational Labs. Irrational Labs is a SWAT team of behavioral scientists. We help companies design and build products that customers Use, love and pay for. This podcast is an exploration of behavior change across different domains. If you like it, you may also like working with us. Give me a shout@hellorationallabs.com and mention the podcast and also throw us a bone. I mean, review on Apple or Spotify podcasts. Thank you in advance. It really does help. So welcome to the Irrational Mind. We are going to deep dive into all the things that Christina Gibson has done in her career, but we are going to focus primarily on the booking.com segment. Let's start off with the main thing in behavioral science is that small changes have big effects. And booking.com seems to basically this is the small changes have big effects idea in practice. So can you talk about some of these small changes@booking.com?

Yeah, and you're definitely correct. Booking is very known for almost these small, micro tiny changes. And that's one of the reasons why I was excited to join Booking, because it felt almost so different than some of the work I had done previously at product companies where it was much larger strategic initiatives. So booking has a very deep experimentation focus and this kind of culture within the product and engineering teams and really across the company as well as some in house tooling has allowed them to really be testing at any given time thousands of different small changes and larger changes on the booking.com platform. So for me, when I was working at Booking, I was responsible for looking at different verticals in seeing how there could be additional growth in the accommodation space. Booking started with hotels. Now they have something like 30 million different accommodations across hotels, campsites, specialty housing in Japan called Royokons. For my team, we were looking at a hostel segment. So hostels, they definitely have a reputation. And so our team was trying to understand, okay, what is it that the hostel booker is looking for and how can we actually increase the share of hostels, perhaps getting people away from some of our competitors or helping people find the right accommodation for them. I think a lot of people associate hostels with cheap, like just the price point. But it actually from our research, it was more than the price point and the hostel space was expanding. It was less about a bunk bed, it was more about the lounge and meeting other people when you were traveling. So one of the things like small thing that our team was testing was how we can create the entry point for the hostile vertical. So one of the hypotheses the team had was younger audience, like, you know, very into more visual content, highly mobile, like let's use emojis in some of the copy. And I Think. There was definitely a belief that this hypothesis would lead to the outcome that we wanted, that people would get to the next stage of the funnel. But we were wrong. And emojis did not resonate as well as the base. So through another round of okay, why did this fail? What we realized is it was not the emoji that was selling people. It was understanding the imagery about the experience. And so booking at the time, like, you've probably booked many hotels, you usually see a view of the room, right? Like the bed. If you're booking a hostel, it's like, sure, you want to see what the bed looks like, but it's. You're not booking it for that bunk bed, you're booking it for the downstairs area, the events that they might be hosting. So by changing away from the emojis and rather updating the carousel of images to lead with the first hero image being that experience downstairs and not, you know, the bed, it actually led to the outcome that we were wanting. Amazing.

So basically booking.com is the first vibe check that we've had. Princess vibe now. And I feel like that's, that's a really nice. You let people vibe check the hostels. So how did that process go? Like, somebody says, look, I want to try an emoji and that like, you guys like duke it out and say, yeah, you're right. And like, no, like, how does, how does that internal process go to get to the emoji and then to get to the, to the vibe check at booking?

It's very, very celebrated and baked into the way of working on product teams that you're constantly testing. So the idea that someone had a hypothesis of, you know, let's test emojis in the headline copy on the entry point for the funnel was not at all wild or something that needed debate. And because of the in house tooling that booking has been able to develop over the years, you're able to do so in a much more safe and controlled environment because you can understand, okay, what's the right runtime for this experiment? You know, what's my willingness to have a decrease in bookings? You can really set these parameters to make it a more safe environment for doing these types of micro tests. I don't think, I don't recall there being a huge debate. This seemed almost a safe bet. I think what usually happens afterwards is most of these types of tests fail. And I think that's where having a culture where you're actually celebrating the failure and learning from it to get to that next step so hey, emojis failed. Like, okay, why did they fail? Like, let's go back and look at some of the qualitative user research that the team had done. And from there it was like, oh, you know, people are really, they're into images. Yes. But the image is not in the form of like, you know, the flame emoji. The image is in the form of show me this experience, because that's what I'm interested in booking. So having that really rich hero image of the vibe at the hostel downstairs, rather than that, you know, very like cold looking bunk bed picture, that was the imagery that really got to the outcome of increasing the conversion.

That's great. And what percentage of tests fail?

I think internally we tend to use like 9 out of 10 will fail.

Wow.

So a lot. Right. But sometimes that you just need that one. Right. To get to something that is going to lead to a good outcome. And again, when you have some controls in terms of the exposure that you'll get from that risk that you're taking, it makes it a lot easier to take some of these bets and be okay with, you know, a 90% failure rate.

Totally. Can you talk about some more of the tests that worked and maybe some of the ones that didn't?

One thing that we also saw from kind of qualitative research and also quality quantitative research around the people who are booking hostels was definitely a lot more on the go, more mobile bookings. So we were looking at, okay, how can we take the mobile booking experience and increase the conversion for the hostel vertical? And one of the things we saw was around search. So even when you're going to booking on the web and searching for a hotel, there's like a ton of things you can search for. And a lot of people actually have an idea of the parameters that they, they want to search by. But for a hostel, it's actually even more important that you have the right search filters because with a hotel it's like, okay, I want a room for two adults. For a hostel booker, do you want a private room in the hostel? Do you want a shared room in the hostel? And also do you want a female only room is also a type of room that you can book and that almost is like a, a higher level heuristic in the search. That was super common. And if you're booking on your mobile with all of the search filters that booking offers, it was pretty cumbersome. So through experimentation and small micro changes, the team was able to realize, let's actually do a shortcut. So if we think that you're on the hostel vertical, let's actually present, you know, do I want a shared room, do I want a private room, am I interested in female room only? And then drop the user into a already pre sorted search result. And that I believe was actually pretty successful for the team because it took out a lot of the cumbersome. I'm, I'm on my mobile phone, I'm in the airport. Like let's look at radio buttons. Like just getting people to the result that they want faster.

Yeah, this seems to be like the problem in the ham or any, you know, search list is you have to figure out how to present choices. And so in one world you can just give a full list and another you're basically saying you actually have to pre filter. Help people pre filter the choices so they get exactly what they want. Lots of questions here. So one would be, if you're asking me like, do you want a female only room or do you want a shared room? We assume people are coming in with a point of view about what they actually want. How much do you believe that? Like, do people come in with like, I must have this or could they be like, I don't care, I don't know, I never thought about it. How do you reconcile this idea that, you know, maybe people just aren't that confident in their answer?

I think that's a tough one because it's very depending on the use case and the situation. Because I think the same person booking for different, like if I'm booking a hotel for business, I might have different criteria than if I'm booking a hotel for leisure or if I'm booking, you know, with a friend for a longer term holiday. So I think the key from, and this is not specific to experimentation, but the key is to make sure that your product is flexible to allow for both of those cases. So for booking they're constantly, we are constantly testing the main webpage and the entry point to allow for a lot of that discovery. Like here are all the different options, but perhaps there's a different hero carousel showing you different types of things that we think you might be interested in. Whether that is because you're a logged in user or some other characteristics that we think, you know, this is why we'd be presenting this to you. With the hostel example, this was actually people who are coming in through a, an SEM landing page. So we'd already knew that they had an interest in booking the hostel vertical specifically. And the idea was how can we get this person who already indicated that they're interested in a hostel and booking.com to actually book their hostel faster and with a higher degree of confidence. And that's where that kind of shortcut filter really was successful.

That's nice. And I get asked this question, our team gets asked this question a lot is just how many options are correct? So should we offer 3, 6, 20? You know, there are pros and cons to each. What do you. How do you answer that question?

So at booking.com, the question would be, what's our hypothesis? Our hypothesis would be, you know, we think that six is going to be the right outcome to lead to conversion. Let's test six versus a base variant. You know, three. For whatever reason, we all love three. Designers love three. I'm not sure why. So let's test three, verse six, and then you can very quantitatively show which one is more successful. Great.

And so when people are making this decision, sometimes there are heuristics or, like, decision aids that they would use. Like, let's say you did give them an option of six, but, like, the six seem to be either very close to each other or very far apart. And it's a difficult decision either way. How do you think about the decision aids that when people are scrolling, scrolling to help them, like, actually make the call?

So your. Your question is how you actually decide between the three and the six in a world where the result is close?

Yeah. So if you have a list of five amazing hotels, there could be something that says like, you know, Airbnb does rare find. And now it's like, ah, rare find.

Ah.

And so these types of heuristics could help people, you know, make quicker decisions.

Yes. So I think booking.com also has a lot of those types of treatments to really encourage you to have confidence in your booking and to confirm your booking. So I think if you were to go to, you know, booking on your phone right now or the website, you'd really quickly see if you were searching for a hotel in New York City, there would be messaging around, hey, for your dates, Kristen, like, this is 85% sold out. And there'd be like a little, you know, indicator of, you know, 85%. Like, we're almost sold out on your dates. And then at each stage in the funnel, there would be different types of messaging or labeling to help you get to the next stage of the funnel. So some of these are sold out messaging. A lot of them are also, like, almost confidence inducing, I think, in a way where, you know, New York City, there's so many hotels. Even if you go and search for, I want, you know, four stars, this price range, Midtown, you still have so much to pick from. So by showing you, you know, 13 other people are looking at this hotel right now, whether or not, you know, those 13 people or there's 13 people are really like you, I think it does somehow trick us or give us confidence that other people are looking at it. It must be good, right? And that coupled with all of the other traditional product features around, you know, reviews and stars and things like that, I think really help push you forward. And I think one of the things with booking that was a huge competitive advantage was their free cancellation policy. So at this point, like, you're looking at your hotel in New York, but, you know, do I want to confirm, you know, it's the kind of that moment where in all product journeys, like putting out the credit card, but by messaging free cancellation, no risk, it really can encourage people to put down their credit card. And more often than not, I think we all know, like, once we buy something, the return or cancellation, it's not as used as frequently as just being like, okay, cool, check off my list, I have my hotel in New York City. Move on with the next item that I want to tackle.

This is. This is nice. So vacations or booking hotels and flights are just big purchases for people. And so there's high uncertainty and we procrastinate the decision. And so the idea of a free cancellation or the scarcity of other people are looking at this are actually just decrease the level of procrastination that we have to push us to make a decision. And so it's interesting with big decisions that we need a little bit of nudging, a little bit of help to get us to actually make the call. I could imagine, though, that especially for the free cancellation, there is a cost to this. So the funnel conversion has to outweigh the support costs of cancellation. How did you guys internally debate this?

I don't actually think they had a huge internal debate about this. It's not something that reached my team. I'm sure it was baked in, but so many of the properties already offer free cancellations. If you look at other OTAs, like, you know, Expedia, even if you were to book direct, they do offer this system. And a lot of times, yes, there is a support cost to it, but it didn't really weigh that heavily. And I think the free cancellation felt like a tried and true experiment where we, we know that this works. Like, we don't need to necessarily revisit. Should we have free cancellation?

Um, okay, let, let's go back and mention something else you said about kind of showing people sold out, showing people that other people are looking at it. Can you talk more about this experimentation set and like what you guys learned?

Yes. So this was, to be clear, not one that I worked on because this was one that has been over, you know, 10 plus years of booking experimentation. They've done micro experiments around some of these, but they are at any given time always being experimented with. Again, so small things, even like the color of the copy, the placement of the copy, the iconography of the copy would always be updated to really make sure that driving for that highest level of conversion that is possible. I think for me, one that coming into booking, as someone, obviously I've been in product management, I'd done lots of a B testing and experimentation, but booking was so driven with experimentation that to me, some of the things on the website almost felt like, why would you have this from a UX point of view? And one of the ones that even today when I still book, I have to remember to go and change is showing sold out properties. Like, if I already have my dates, why would I necessarily show something that's sold out? And I think there's a couple of reasons for that, but one is it showed that if you show sold out properties, it in the end gets people to convert more in the end. So I think very early in the funnel, it's creating a bit of a scarcity effect that New York City, during your dates, you know, this is a property you can't have, Kristen. You want it more, you do, but you can't have it and you're probably not going to change your dates. I think more often than not, especially for city bookings, people know their dates. It's not, it's not as common of a use case where you're just browsing, like, oh, maybe I'll go to New York. It's like, no, you're usually going and you have your dates in mind.

Got it. And so when we're showing basically these sold out concepts, this is like setting a mindset that things could be sold out and kind of pushing our decision to be a little bit faster. So if I, if I wait until tomorrow or the next day, it could be gone, it could go on. I will never have a vacation again. I will not. I'm not going to get what I want. My family's going to be disappointed in me. I should just buy now, which I Think this is like a really interesting example because the rational perspective here is not to take up your precious website space with things that people can't have. And the idea here is that you're putting people in a frame of mind that is about making the end decision. It's about basically converting in the end, not necessarily the current, you know, like how do you optimize this webpage in this frame? So it's a nice example of like one team. You know, when we work with teams, sometimes it's like you put one team on one page and another team on another page and this is really about the full decision making process of, of travel. You're helping people kind of overcome, overcome that hump. Yeah.

And I think these types of messaging, like you see this all the time. I think you see it on e commerce websites like Ticketmaster, I think is classic with this, with the, you know, the fear countdown. Like you only have 7 minutes to go to fill in this form and like hi, we need to quickly type it in. I think another one for booking that I recall being very successful was this idea that it's already reserved for you. Like hey Kristen, you're already logged in. We already have this hotel, you know, it's reserved for you. We hold it for you.

Oh, you're holding it for me.

Yeah. So if you, if you choose not to go to the next step, I mean, you're lost. We'll give it to the next person because we're 85% sold out on those dates. So again like the subtle messaging of like it's for you, we're holding it, it's a reservation. At this point you haven't given your credit card, you don't have a reservation in the, in the sense of a confirmed booking. But it is that slight push to get someone to the next stage which is really the outcome of a confirmed booking.

Right, Exactly. We've tried some of these experiments in health where basically in the employer in the US market, you basically get a benefit. So you're getting some benefit from your employer and if you can reframe that model to be it's your benefit. You already have this benefit, like go use it. It's a different mindset, a different mental model of how to engage. I think inherent in there is this reducing friction. So in a world where you have to go book something, that's a lot of friction in a world where you already have it, it's reserved for you. It just doesn't feel as heavy is a task to do. So great. And yeah, the urgency countdowns. This is interesting to think about kind of how mechanisms like this fade over time. So now we're kind of accustomed, like our web browsing process may be more accustomed to this than it was two or three years ago or five years ago. Do things change over time? So you, you do one experiment and then like you see an effect. Are you retesting or how do you think about the changes over time?

I think you definitely need to be retesting because not only does the industry change, your content changes, how people interact with your website will change like introduction of mobile and that's really where you get to with booking know thousands of different tests being run at any given point in time. So I do think if you were to take a look at maybe companies who have less of a in house culture for experimentation, I still would be recommending the approach of keeping things fresh. So maybe you've tested a referral program, for example. It's like okay, we tested the, you know, refer your friend and you get €5 off your next ride model. Maybe you should test. Is 5 Euro the right incentive? Is the framing of this the right incentive? How's the UI doing? Especially if it's something that has been a proven growth channel or something that's been working successfully. I like to make sure that teams aren't almost letting things kind of hang out on the branch that they're constantly looking at how we could maybe improve. Because even if you're getting like 1 to 2% extra, that translates into a lot over time.

Yep, that's great. The kind of the refresh questioning, we hear a lot from teams that we did that before and it didn't work. It's like well one, how did you do that? And maybe the world has changed. Wonderful. So let's go back to the funnel because I think this is a question that is just like debated in circles is are we trying to open up the funnel, get more people to do a thing and this could maybe in the hostile world you're kind of trying to open up the hostile market and. Or are we trying to. Should resources be at the end? So. So that when you get to the end of the funnel of the hostile booking flow that you actually convert because otherwise we're just getting a lot of bad traffic and you're losing them. So how do you think about that conundrum?

I think conundrum is a good word for it. I think you need to take a look at the, the stage of the company and your willingness to have risk in terms of Losses. So are you okay with pouring water through a leaky bucket? You know, if you open up the floodgates, you're most likely are going to get more at the end, even if you aren't changing anything in terms of the funnel and your drop off rates. But you could be losing some of your valuable audience along the way. Or do you pause, fix the leaky bucket, patch it up and then pour in the water. So I think it really depends on the appetite for losses and the industry that you're in. For example, like newer products. Sometimes I really want the team to have confidence in the health of their funnel. Like if you're launching something new and you're going to be spending a lot of your limited marketing budget to announce this, you certainly don't want to drive people to a landing page and then find out that your sign up now button's broken. Like that would be terrible. So that's like a very targeted example. But I think taking a step back and looking at what's the stage of the company and what are we willing to gain lose with different approaches. And if you are staffed and you have healthy amounts of team, like you could actually do both at the same time and with booking because people would come into the top of the funnel. Maybe they were coming into the hostile vertical and we had like a leaky bucket down below because I mean no bucket's perfect, right? But because of the, the nature of booking, we could still have somebody, maybe they get out of the hostile vertical but they end up booking something else as well. It's not like a. They don't completely detract from the, the bucket. Maybe I'll stop with the bucket analogy.

Or at least fun this one. This is something I think about a lot, which is you generally sometimes have to. I think you do too. It's coming into booking, you know, whenever I'm doing the A B testing and you're kind of doing more bigger swings is how paternalistic should products be? Because you may have a point of view about how people should travel and yet, you know, if the data shows that they're clicking on tours and kind of other maybe normative things, but you want to like surprise them with something like a fun hostel with a cool vibe check. How, how paternalistic do you be? Because basically they may not, they may.

Not buy your vibe check.

They're going to buy the vibe check. But yeah. What do you think?

So if I understand your question correctly, it's. I think you're asking a little bit around, you know, people maybe As a company, you want to offer one thing and your customer thinks that they want another thing. How do you actually present a few different options? Was that what you're asking?

Yeah. So let me try that again. So basically, so in product management, there are worlds by which if you continually optimize the funnel, you will deliver what people want. And it seems like even for booking.com, there's just a long list of tours. People want tours, it seems, but also by clicking on the tours, you're just going to signal to booking that you want more tours. And so it's called revealed preferences, where if you do something, it shows that you want that. But what we forget is that if there was something else on the page, you may actually want that too, but we don't know because there isn't that in our environment to actually choose from. And so at some point you kind of have to take a risk and say, I'm going to add something completely new and hope that people like it and not just follow kind of a click path.

When I joined booking, one of the things I was looking to do with the team is to actually help them take some of these kind of larger, more incremental step changes. So I think you can certainly like micro optimize and do small tests and end up at the end result. Or you can kind of take a leap of faith or a strategic bet that this is an area that we're going to go into. We're actually not going to, you know, run a bunch of AB tests to determine whether or not it's true. We're going to do it and we're going to see how it performs. And I think a lot of times this is a combination of different things to help you have confidence in taking that larger step. For me, I sometimes like to think of it as a, like an overlapping Venn diagram. So as a product person, it's like you have your quantitative data, so maybe that's experimentation sets that have been run in the past in this area at your company, maybe it's industry data, you know, app reviews, whatever it might be, then you have your qualitative data. So what you're hearing from customers always, always, always be talking to customers. They might not be telling you exactly what you should go build, but it's going to help more inform your product decision making process. And then the kind of third piece of the overlap is a little bit of this, like strategic insight or instinct, if you will. Like, okay, I have some quantitative, some qual data and like instinctively I just Think that this is something that would make sense. So at booking, an example of this was prepackaged tours. So booking is actually part of a larger company that has access to airline inventory. So there's this concept of prepackaged trips in North America. Really common example of this is I want to go to Cancun, I want to stay in an all inclusive resort, I want my flight there, I want my transfer, I want my hotel, I don't want to do anything else, I want to buy one thing and that's my, that's my two week vacation. And booking wasn't really playing in this space. If you look at the quantitative data side of our diagram, it's lucrative, people spend money here. But the work to actually integrate the flight inventory into bookings funnel was pretty big. You kind of had to just take a leap of faith that hey, this is going to make sense, like, let's do it. And I think within the first six months there was something like a million euros worth of bookings from introducing this type of prepackaged travel on the site.

Wow. If this hadn't worked, let's say like total bomb. This just doesn't, doesn't fly. You've just invested a ton of the time in it. How, how would you reconcile that? Or like, what would you do in that case? That's the risk. Like everyone's thinking about the negative impact of taking a big paternalistic jump, maybe not the upside.

Yeah, I mean, I think the risk could be, yeah, you fail, but also what did you, you learn in that? So one, maybe you've for this moment of time been like, okay, prepackaged travel didn't work in this iteration. Well, why didn't it work in this case? We actually had built, we'd forked and created our own entry point funnel to move a lot faster with the complexities of the airline travel. And I don't think long term that was going to be the way that it needed to be built on the booking platform. But I think you would still have a lot of learnings along the way. And whenever you're taking one of these larger, more strategic product bets or launches, I think it's really important to set with all the different stakeholders the expectations upfront. You know, we expect that we're going to have a check in or a review of performance at this point in time. These are the things that we expect to see, these are the things that we'll measure upon because it creates a little bit of a culture of checking in on the performance. And I think that Almost. In some cases, it removes the fear that we're going to invest all of this and then we're not going to know what happens or we'll find out too late that this was a waste of energy and we had this team working on this. So almost creating a constraint of, hey, this is how we're going to stage gate this new feature and there will be an opportunity that we can make the decision to kill it. And I think as product teams, sometimes we don't kill things enough. It's like you just let it kind of hang out there and then you just end up with this complexity sitting in your product stack for far too long. I think it'd be a stronger push to say we're going to try for six months. You inform that six months based on, you know, your own industry, your own patterns, and then make a decision when making decisions.

Anything else that booking did to, to help push that decision along or give people like different heuristics to make it.

Yeah. So I think booking, from having so many accommodations also on their platform has a lot of data that can be presented at any given point. There's always that design balance of. At what point are you sharing all of these details within, you know, a seemingly large table of information? And through optimization and experimentation, there are a few that over time really stood out. So breakfast hotels. A lot of hotels include breakfast, but actually gathering reviews specifically around breakfast after someone's checked out and then creating a little bit of a classification of like, it's a good breakfast or it was a very good breakfast or it was an excellent break breakfast. And actually showing that description led to an increase in conversion. If someone was looking at the, the property, it's like, oh, there's a, a very good breakfast. I think I'm going to book versus if it would just said breakfast included or good breakfast, very good breakfast.

We like our breakfast. You know, it's, it's part of the experience of, of hotels. How you gonna, how you eat. That's great.

Yeah. And I think also similarly with the, the hostel vertical, when you're going to a hostel, there's a lot of different considerations that you had where there are outlets. So in a bunk bedroom. I don't know if you've ever stayed in a traditional hostel. I didn't want to assume, but I figured you, being a good traveler, had stayed in hostels at one point. So WI fi now is like by large and far, like the thing everyone's searching for is there good WI fi. So in the same way that People care about the quality of the breakfast. Reviews are on the quality. Quality of the WI fi is really important, especially as we see more kind of digital nomad types, but also for hostile bookers. Some things that we ran experimentations on and were successful were better surfacing around if your bunk bed had an outlet. So could you charge your iPhone and thing overnight near your bed or were you going to have to be, you know, going in the dark to a shared outlet? And also toilet paper, because sometimes that's not included. Something that I learned. Wow.

So basically showing people things that you would kind of expect it to have maybe increases the certainty of their decision. So you're basically saying, look, toilet paper is included, which we should expect. It probably gets me to think that some places toilets don't give you toilet paper. And if I. And this may be better because the other places just don't have it. So that's. That's interesting to highlight something and then maybe the implicit assumption is, is it's different, better, new from other things.

Yeah. And I think it is something about that. If you're getting down to that, like, level of fidelity with the details, it's like, okay, this, this hostel has what I need. It has toilet paper and WI fi. Let's go.

Yeah, let's go. Vibe check. Vibe check. Sold. Okay. Any advice that you would give to teams that are trying to do more testing and learning?

Advice for doing more testing and learning is like, yes, you definitely should be doing this. I really like teams to make sure that leadership is bought in before you go and push for testing in a way so it gives you more of a budget or clearance to have those failures. Because a good experimentation culture is going like, you're going to have failures and you need to somehow figure out within your own company what's the right way to celebrate that or have tolerance around the failures. Maybe it's you celebrate and share learnings and like a public Slack channel when something fails. Or maybe you have like a budget for failures. Like you can. You can hurt our conversion funnel by, you know, this amount through some of your experimentation. And then wherever possible, invest in education and tooling for the team to make sure that you are having a very solid hypothesis going in. I think a lot of experimentation, there's good intent, but if you have a poorly formed hypothesis, you're not going to be able to really measure the outcome in the end. So then it's like your whole stream of data experimentation is to a degree a failure and a throwaway because you can't use it. And I think there's a lot of publicly available things on LinkedIn and all the different online courses around how to run good A B tests. So if it's something that's new to the company, I would really encourage kind of level setting the education around this and getting buy in to have failure.

Thank you for listening. If you like the show, go to irrationallabs.com to say hello to us or post a review on Apple or Spotify. Big thank you to the show's two Jacks, Jack Buer, our writer, and Jack Steger, our genius editor. Until next time.