#7 Data Science at Airbnb (with Robert Chang)

In this episode of Data Framed, a Data Camp podcast, I'll be speaking with Robert Chang, data scientist at Airbnb and previously at Twitter. We'll be chatting about the different types of roles data science can play in digital businesses such as Airbnb and Twitter. How companies at different stages of development actually require divergent types of data science to be done, along with the different models for how data scientists are placed within companies, from the centralized model to the embedded to the hybrid. Can you guess which is Robert's favorite? This is a hands on, practical look at how data science works at Airbnb and digital businesses in general. I'm Hugo Baun Anderson, a Data Scientist at DataCamp and this is Data Framed. Welcome to Data Framed, a weekly Data Camp podcast exploring what data science looks like on the ground for working data scientists and what problems it can solve. Hugo, I'm your host, Hugo Bown Anderson. You can follow me on Twitter HugoBown and DataCamp. At DataCamp, you can find all our episodes and show notes@datacamp.com community podcast. Robert, welcome to Data Framed.

Thank you for having me. I'm super excited to be here.

I'm super excited to have you on the show to discuss all types of things about data science, but in particular, data science in tech and data science at Airbnb, where you are now, and Twitter, where you worked previously. But before we get there, I'd like to discuss you.

Okay.

What are you known for in the data science world?

That's a good question. So I think for me, I'm just like every other data scientist. But I guess when I was at Twitter, I wrote a blog post on medium called Doing Data Science at Twitter. And that was a reflection of my experience working as a data scientist at Twitter for two years. And I just thought that I learned a ton about data science and I was really grateful for learning all these stuff and I wanted to kind of share it. Originally, I actually wrote it as a reflection for myself and so I didn't really want to share it. But then in the end of the day, I thought that a lot of the stuff there could be very relevant for other people. So I shared it. And very serendipitously, it got picked up by Pete Skomaruk, which was a former data scientist at LinkedIn and he's kind of an influencer on Twitter. And so he shared my blog post and I got shared about two other people. And then, yeah, somehow the post got viral and that kind of got my, I guess, blogging career going. And since then I'VE written a few other posts all related to data science. So I don't know, I don't think I'm famous, but I mean, many people have told me and that they found my blog posts helpful.

Well, I've definitely found your blog post very helpful. I mean, more recently, you've written 12 years on reflecting on your time at Airbnb also, which we'll get to. But I do think your blog posts have opened the eyes of a lot of people working in data science, or working as data analysts, or working as data engineers, or even people from other disciplines into the daily ins and outs of what it means to be a modern data scientist. And that's one of the really exciting things about having you on this show.

Great. Yeah, I agree. It's really important. I mean, when I started writing the blog post, I didn't really have anything. I didn't have any specific goals in mind. But then over time, I realized just how useful it was. I thought I was just stating the obvious, but then a lot of people gave me the feedback that they found the blog post very helpful because it helped them to demystify what data science is really about. In an environment where everyone talks about modeling, machine learning, deep learning, it's really hard to understand what data science really is and what it really is, especially in tech, which is our current. The subject of our discussion today. And so I think really just sharing those that information with people is a very useful thing for the community.

So before we get into a detailed discussion about data science in tech, I'd like to know a bit about how you actually got into data science initially.

Sure. Yeah. So when I was in school, data science was not really a thing. It was not really a profession. So I study operations research as an undergraduate, and then later I studied statistics in grad school. And, you know, generally and broadly speaking, I have always been very interested in statistics computation and also data visualization. I was very deliberate, you know, even, I guess, early on, I was kind of very deliberate about how I want to build my skill sets. And so I actually very much enjoy planning all my courses and curriculum. And so I started taking a lot of classes in those three areas. And then one summer I came across a blog post by Nathan Yao, who is the author for a very great blog called Flowing Data. And in. In that blog post he mentioned about, I think he cited this, like, Harvard Business Review article on, oh, data scientists, you know, sexiest job of the 21st century or something like that. Something really like eye catching and even like click Baiting. And so I just click into it and start reading about it. And then I read through his description and I realized whatever he was describing, the data scientists, this particular profession or role that he was describing was very similar to kind of the skill sets, involves the skill sets that I have been building. And so I thought that, know, maybe there are interesting industry opportunities that would allow me to continue to work on that domain after school. And so that's kind of how I got started.

And so these, these three specific disciplines or skill sets were statistics computation and visualization.

That's correct.

And what, what type of research were you thinking about or what type of data sets were you looking at or visualizing or doing computation on? What interested you at that point?

At that point, I think it was, to be honest, I don't think it was anything specific. There are many, well, there are many areas that I was interested in doing, but I was quite interested in understanding using open data to investigate some of the more social issues. And so I didn't do much public work when I was in school, but I was very interested in like, basically taking these like governmental data that is like buried in the, you know, bureaucracy and just kind of dig them out and then just trying to visualize these basic statistics in a way that, that, that paints a picture about, you know, how our society work and it, it really. Yeah, so that, that I would say that's probably the, the area that I'm most interested. When I was in school, I, I mean, that's probably one of the reasons why I later joined a social network company. Just because there's so many data that's being generated by people and it really gives you a glimpse into kind of how people share information, how they use information. That's kind of what I was interested on.

Great. And so essentially you were engaged in these practices of statistics computation and visualization and then noticed that these, among others, had been rebranded or remarketed as data science in some sense. Yeah.

Oh yeah. I should bring one extra example. I think when I was taking a visualization class, one of the things that, one of the examples, and there are many examples, but one of the examples that really came to mind was, I think there was the Enron scandal and then somehow they like, collected all the emails. Yeah, the emails. And then someone like, geniusly enough to like, basically parse out all the email communications and then put it into a graph and it shows you how like, basically how people have been communicating up until the scandal broke. And to me it was just like so fascinating to See how you people leave these like digital traces that allows you to see that really just, it's another way of capturing the history that is, that is otherwise not, not possible.

I've actually gone through a bunch of those emails because they are public and there's all types of wacky, you know, office romances. There's all types of really wacky stuff in there. And a couple of my old friends, they're digital artists and they actually did a project with the Enron email set where you can sign up to their mating list to receive these emails in perpetuity for the rest of, you know, for the next thousand years or whatever it is.

That's great. That's great.

So my last question about, about your history is when you say computation, which is such an important part of data science today, we're not talking about pen and paper, we're talking about using a computer. What languages were you using back in the day?

Yeah, that's a great question and I have a lot of thoughts to share on this, this very topic. But to answer your question, when I was an undergrad I actually really hated anything computer related. When I was undergrad I did pure math, so I did a lot of proofs. So I use a lot of pen and pencils and I try to avoid as much as I can on doing anything computation related. It's not until. So I learned, I think I learned Matlab when I was an undergrad. I didn't really use it and then it was not until grad school I think. And there's one class where I think it's a stochastic optimization course where we were learning some probabilistic models and I think it was Metropolis Pasting Algorithm. I think I still remembered it. Besides, that's kind of besides the point. But like we were asked to do some simulation and I was, I was just, I just got stuck. I didn't know what to do. And then that's when I really started to realize there's actually a very huge gap in my education where we are moving into a world where there's just a plethora of data and there's so much data that can be leveraged but you can't do it by hand. You have to really leverage computers to do these processing. And that's when I started to realize, okay, I really need to get better at data manipulation or computing computation in general. And so I started, I pick up R to start with and then over time I moved to Python. Nowadays I've been using a combination of R Python, Unix, basically tools are relevant to solve the problems at hand. So yeah, it's definitely a really important area if you want to become a data scientist. Learning how to deal with data and how to do computation in addition to statistics is a very important skill to have.

And you found R is the way you entered though?

Yeah, R was my first language. R to me was kind of an interesting language. It was the language that taught me. And if you talk to other people, you probably will hear similar stories where a lot of the statisticians and I guess scientists, a lot of their intro introductory like programming language is not really an actual programming language, not a gen, like a general purpose programming language, but more a domain specific, usually scientific computation like language like R or matlab. Over time I basically learn more about computer science in general just on my own to pick up other stuff.

Yeah, my trajectory is actually quite similar and we're just about to get into conversation about data science in tech. But I just wanted to take a slight detour and talk about the fact that people used to do stuff by hand, right? I mean Pascal, Fermat, I think it was Daniel Bernoulli who came up with the binomial distribution and the Bernoulli distribution. He sat around and he. I think the story is he flipped a coin 6,000 times and wrote and saw the distribution appear, then figured out the closed form solution to it. But we do, Reverend Bayes did the same actually. But we do, we do have such incredible computational power these days. And thank goodness, because of all the data that's coming through, we couldn't, we couldn't do any of that. That by hand these days.

Exactly. And you see a lot of these patterns emerge or reemerge in the statistical and even computer science world. In statistics, Bootstrap was around for a long time, but its real power wasn't able to be realized until there's really enough computation powers for people to run these resampling schemes. And then you have the world of neural network where you know, those invented like as early as I think the 670s. And you know, it's not a new idea but it's just like the computers wasn't able to do it. Nowadays it's like growing really fast because all the GPUs and all that stuff and so it's definitely the interplay of statistics and computation. I think it's going to be increasingly important.

So let's jump in. From your experience at Twitter, your current work at Airbnb, I'm going to ask a pretty general question. I'm just wondering what, to your mind are the biggest challenges currently facing digital business.

This again, kind of, this relates to what we discussed earlier, which is computation specifically, I think as a startup, you know, any business, digital businesses, like if you want to eventually scale, you have to start from somewhere. And it's actually very important to be able to build a data foundation for, for you to do any sort of analytics. And so by data foundation, I mean, you know, setting up the data infrastructure. So that might involves things like, oh, making choices of like, what database are we going to use, are we going to use, you know, put things on, on Amazon, on Amazon, on aws, do we want to use hdfs, all that kind of stuff, which I'm less familiar with. But like, building a strong data, data infrastructure is very important. And then second, it's important once you have the infrastructure, then to build your data warehouse on top of it. And so this involves in basically taking your raw data, organize it, aggregate it, cleaned it, and then put it in a form that is analytics friendly so that other downstream users can use. And so data infrastructure and data warehousing think is really the bedrock of all kinds of analytics. And only after then can you move on and to do more fancy stuff like business intelligence experimentation or building data products. So Data Foundation, I think is a very big challenge for many different companies and that is something that everyone has to go through and it's very important.

At a business like Airbnb, Data foundation will be incredibly important in terms of storing information about properties, users, all of these types of things, right?

Yeah, exactly. And this work might sound trivial, but it's actually not. And a lot of problems would pose as a trivial problem, but it's actually not. So I think it was Monica Rogatti, she was also a former data scientist from LinkedIn. She had a famous tweet and said that basically the majority of data science is just doing counting, but doing counting in a smart way. And I feel like building Data foundation is very much like that. In addition to Data Foundation, I think, or I should say once you have the data foundation, then if you look at the trajectory of a company, once you have the product market fit, that's where you really want to start growing your company. And so helping your company to grow and drive more adoption and find more users who love enjoy your services is really a big part of what data scientists typically do to help companies to, to, to scale. So that's something that I work very closely when I was at Twitter and you know, if you look at all the top companies nowadays, there's inevitably a growth team that's focusing on driving growth, sustainable growth, beyond growth. Then once you really have the scale, then I think that's when you want to start optimizing your business. And I think that's where that's really where machine learning and then data product personalization can help. They can make the product smarter, they can make it more personalized and they can make the business more efficient. I'm laying out in a way, I'm laying out a challenge, I guess kind of aligned with the trajectory of how a company will grow itself in the data space.

I think that's a really intuitive approach and it leads to a number of questions. We've discussed the data foundations aspect briefly. I'm interested in growth and what type of data science questions or techniques arise when thinking about these growth challenges.

There are various ways to answer that. One is that I think growth is very much data driven because a lot of the growth work is. Well, there are kind of good growth work and bad growth work. The bad kind of growth work or the so called growth hacking where you're very short term thinking, you're just trying to provide short term incentives for people to like kind of sign up for your services and then they come and then they use it for a little bit, but then they don't really retain. Those are not the kind of. I don't personally, in my opinion, I don't think those are the good kind of growth problem to think about. I think the good kind of growth is you really think about the business, you really think about what is the values that your service is able to provide and then thinking about what are the product changes that you can make to expedite the process in which your users can start enjoining the values that your service provide. And so these are typically more longer term thinking. But I should say that either whether it is for short term growth hacking or long term growth work, I would say experimentation. It's a very huge part of doing growth. And because you can't really verify whether your hypotheses are correct or not unless you run a randomized control experiment. And so when I was on growth, that was the period where I learned a lot, a ton about online experimentation. That's an area that's very important. I think another, there are a lot of other, I would say techniques that Facebook use which, well, both Facebook and Twitter use that I learned a lot. So I think this is probably invented at Facebook. But there's this framework called growth accounting where you broke down your active users into different segments of users. So new user, resurrected users, users who churn and users who retain. And then if you kind of break down your business by, if you break down your active users by these segments, you can observe over time, oh, are you actually increasing the number, the share of new users or do you have a leaky problem where your share of churn users are increasing? And that's a very informed view, informative view, telling you what is the composition of the users who are using your services.

So could you give me an example of an experiment that you did at Twitter that you found interesting or exciting?

That's a great question. I've ran so many experiments at Twitter. I can talk about the first project that I was involved when I first joined Twitter. It was a project that is less growth related, to be honest. So let me describe what it is. So basically it was a project where we as a social network people get notifications and emails as they interact with our platform. And we had a hypothesis that a lot of our users are receiving all kinds of notifications and they're getting fatigued by these messages and some of them are more useful than the other. And so one of the things that we were trying to run is, is there a way in which we can identify a subset of email or push notifications which user don't really need to get because they have low value. But and we have a hypothesis that if we are able to identify the combination of that email and user set, then we can remove those emails from them. That will one, help them to have a cleaner inbox and two, it will still not hurt our bottom line. Like people will still find value, they will still come back. And so there are some data science work going into it and figuring out what is the right combination. And then we ran an a B test where we, we have a control group where users will continue to receive whatever notifications that we have in the system. And then we have a treatment group where we hold back certain emails from the users so that we respect the user's inbox. And then the end result for us is to compare the result of the engagement metrics between the control group and the treatment group and to see if we are able to maintain the same level of engagement for users who did not receive those notifications. And luckily it turns out that we kind of did the analytics right, so that we realize basically the users who are most active, they're receiving, for example, if I get a, if I get a direct message or a tweet from you, it's very likely that I will get the same notifications on my push notification as well as my email inbox. So they're in some sense kind of redundant. And so if we can just basically remove one or the other, then that will make their inbox less clutter, but at the same time they'll still get the message from other channels. So that's kind of one example where it wasn't really used to drive growth, but it was using a B test to respect our users space and inboxes.

That's a wonderful example and it's a great experiment. But one of the reasons I think it's so wonderful is because it's something we can all relate to, right? With with the world of having push notifications come in. I mean, you know, while I'm recording this podcast with you, I've got my phone on airplane mode for the obvious reason. Up next, we have a segment called Data Science Best Practices with Justin Boyce. Justin's a lecturer at Caltech, where he teaches lots of courses for biologists and bioengineers and helps them to work with their data. Hey, Justin.

Great to be with you, Hugo.

I understand you want to talk to us today about how you share data.

Yeah, I want to make a simple but important point. If you want to share a data set, make an informative plot that displays as much of your data as possible.

I totally agree with that, Justin, but that can't be the whole story.

You're right, it often is not the whole story. And there can be lots of statistical modeling and inference to be done. But the seemingly obvious notion that you should generate a plot that displays as much of your data as you can is often overlooked. All too often I see research papers, figures in news articles and when I was working in industry, internal memos that do not do this.

Really?

What do they do instead?

Most commonly, people will report a summary statistic like the mean or median, and sometimes a confidence interval. This is often done in text, not with graphical plots. And sometimes the confidence interval is not even properly described. There are often implicit assumptions of normality of the data and symmetric confidence intervals, that sort of thing.

What's wrong with that?

Well, there's nothing wrong with reporting summary statistics, provided you clearly define the statistical model and assumptions under which they are calculated. But when you don't plot the data, you can end up over distilling the data to just a few numbers. An effective graphic tells so much more of the story. In a graphic, you can explicitly see what the measurements are and importantly, how they might be distributed.

Can you give me an example of a good display of data?

Sure. One of my favorites was used to show how dominant Steph Curry's 2015-2016 basketball season was in terms of three point shooting. This graphic was made by the Upshot for the New York Times. We'll link to it in the show notes or you can find it by googling Steph Curry upshot. Anyway, the people at the Upshot collated all of the NBA players who were in the top 20 in three point attempts in each season since 1980. They could have computed the mean number of made threes over all players and the standard deviation and then said that Curry season was three or four standard deviations above the mean. That distills it down, but does not really show the whole story. Rather, the folks at the Upshot plotted the cumulative number of threes made for all of these players during the respective seasons. This graphic is so much richer. It shows how the number of threes made by league leaders has been steadily increasing over time. And it effectively shows how completely anomalous Curry's season was.

Yeah, that's a great graphic, but it takes up so much more space than reporting a summary statistic.

Well, that is true, but your data are important. Show them some respect and give them space in your reports. I also think space constraints, at least for graphics versus text, are irrelevant. Reports, research papers, even news articles, are distributed digitally these days. Space is essentially free. Beyond that, if you want to make a point with your data, it is worth it to make a plot. If you don't have the space for that, you might ask yourself how important the inquiry for which you acquired the data actually is.

So then, as a rule, plot your data.

Yes, it's a simple but effective rule. Give your data the space they deserve.

This has been a lot of fun, Justin. Let's chat again soon.

You bet. I talked today about the importance of just plotting your data. In future segments, I hope to give some examples of effective plotting techniques such as B swarm plots and empirical cumulative distribution functions. I'll see you then.

I'm looking forward to it. After that interlude, it's time to jump back into our chat with Robert. We talked about the three different phases, data foundation growth and machine learning. Where is Airbnb at now?

Oh, that's a great question. The answer is it depends. The business is organized into different product verticals, and some verticals are very mature and some verticals are quite new. To give you an example, the homes business, which is kind of how Airbnb got started, is very mature by now. And then the trips business, which is a new product vertical that was launched just last year, where in addition to finding homes, you can find local guides who can help you to enjoy local, like experience. That particular business is quite new. And so depending on the face of the product, depending on where it is, the focus on data foundation growth versus machine learning might be different. So on homes, the data foundation has been built, and it's still being built, but. But the data foundation for, by and large is there, and we'll continue growing the business. And there's a lot of machine learning and personalization data products that's being worked on for the homes business versus, if you look at trips, really, because it's a new product, a very big focus is to focus on growth. So it really depends on which product vertical we're talking about.

One thing that changed in your professional life when you moved from Twitter to Airbnb was a far stronger focus on machine learning, right?

That's correct.

And what type of machine learning are you doing or involved in at Airbnb?

To give you a bit more context, when I was at Twitter, I was very much involved in product analytics, experimentation, statistical inference. And so when I transitioned to Airbnb, I made a pretty conscious decision to learn about other areas of data science. And I want to do more machine learning and I want to do more sort of data, basically modeling, statistical modeling. And so I was very lucky. In the past, I would say, two years at Airbnb, I was able to involve in some MIDI modeling project. And the one that I have been working on recently, early this year, is a project where we try to model the lifetime value of a listing when they first get booked on Airbnb.

And what's lifetime value?

Yeah, so lifetime value, you can think of it as, you know, if you have a listing and you put it on Airbnb, you will hopefully start getting bookings. And then over time, you will accumulate. Each time you host someone, you will get paid. And if you accumulate these payouts, then over time, these are the total sum of money that you're able to earn by participating on this marketplace. And so if we look far ahead enough, we will be able to calculate your lifetime value of your listing on Airbnb. And so that's what LTV Lifetime value stands for as of why it's important for us. It's very important because in addition to having very strong organic growth, we're also trying to grow our business in other ways and sometimes growing our business, which involves costs and so what we want to do is to be able to optimize our acquisition growth, acquisition channels, supply, acquisition channels. And what we want to be able to do is to calculate for each channel, what is the return on investment. So if we put in $1 to acquire listings from channel A versus channel B, how much money will we get back in return? In theory, we can just wait for a year and then, or wait for a long period of time and then we will get to really observe how much money each listing is able to make, and then we can calculate a return on investment. But from a business standpoint, in practice, we want to be able to make that decision of optimizing across different channels as early as possible, because we want to be forward looking. And so this is where the predictive model for predicting the listing lifetime value can be very, very handy. So by taking into account a bunch of signals like the location of the listing, the availability, what kind of amenities they have, what are their historical performance, and all that stuff, then we can start to build a historical data set and then use that training data set to train a model and then to predict for future listings that are on board with similar characteristics, what are the likely returns for them. And then when we have those predictions, then we can start making these important decisions of optimization across channels a lot easier than just waiting for it for a year.

That's a really good example of having a machine learning algorithm or challenge or task feeding right back into business development. Right?

That's correct.

And so that kind of leads me into my next question is, you know, data science, especially for a company such as Airbnb, where there is so much data, data science can have such an influence on any decision anybody within the company makes. So I'm just wondering how data science is integrated into the business as a whole.

Yeah, that's also a great question. In tech in Silicon Valley, typically, companies organize data science team in different ways. And as far as I can tell, there are generally three models. And this might not be the only three, but these are the three broad categories or approach that I've seen how people organize data science teams or how they integrate into the business. The first one is called the centralized model, where you have a team of data scientists basically sitting together, working very closely, trying to solve very challenging problems. And then once they kind of solve it, then they will kind of push it, try to try to sell it to the rest of the organization to adopt them. And so it's a very centralized approach where people are really taking the data scientists are taking Command and driving the roadmaps on how they will provide value for the organization. That's one extreme. The other extreme is what we call the embedded model. And embedded in the sense that you have a distributed data science team where data scientists are embedded with the product team. So just like a full stack product team, you will have perhaps a designer and a team of engineers, engineering managers. You will also staff a data scientist on the team. And the data scientists on the team on that particular product team is very much responsible for all the data related work for that product. So it could be doing opportunity sizing, it could be doing product analytics, understanding how the users use the product. It could be designing experiments so to help and iterate a product. It could be building machine learning models for the product. It could be a wide variety of things. And so these are people who are very, very much embedded into the product and they know very well the domain of the product, but not necessarily on other parts of the product verticals. One extreme is centralized, fully centralized model. The other extreme is fully embedded model and it has its pros and cons. And we can talk about it a little bit more. But the last, the last model is kind of the hybrid of the two. Where you have data scientists continue to be embedded with product team, but then you still have this functional team where data scientists do not report to like a product manager or engineering manager, but they report into a data science manager. And so that way you have data scientists all contributing to specific product areas and they're really the product owner and experts on these specific product areas. But at the same time they are able to enjoy the right level of guidance and career growth from their immediate functional manager. And that's called a hybrid model.

The hybrid model. So the answer to my next question may be business specific, but, but in your experience, which, which is the most successful both on a business side, but also for professional development for data scientists.

Speaking from the experience of someone who have experience all three, I actually think the hybrid model is the best. The issue with centralized model is that a lot of time, as a centralized organization, you have to, despite doing very good work, you really have to push your work to other people. By people here, I usually mean like the product teams, but you can imagine it being very challenging because a lot of time, especially in technology companies, they went through, you know, either sprint planning or cycle planning, where they are always constantly developing on what to build. And that's a very rigorous process and it requires a lot of attention and a lot of planning. So if you're not part of the team and you're trying to push your insight or even ideas or agendas for the other teams to adopt. Unless you have that right level of organizational relationship, it's really hard for your vision to be realized within the product. And so I think I find that generally very challenging if you're on a centralized team. On the other hand, if you're in a company with an embedded model, I think that's the best way to, for your data scientists provide immediate impact. But then I think the downside of that is there could be times where data scientists actually do not report to a data science manager. They report to either mostly into like an engineering manager, in which case the engineering manager might not know how data science work because it's such a new field, unlike engineering. And so to me, that end of the spectrum, the embedded model really maximize the utility of the data scientists, but then sometimes at the expense of their career growth. So I think that's why the hybrid model was created, which is to balance the two, like business impact and career growth. And so I personally really enjoyed the hybrid model.

It also, as you say, balances the trade off between having a manager who can speak the language of data science with you, but also feeling like you have ownership over the decisions that are being made with respect to.