#258 Machine Learning for Ride Sharing at Lyft | Rachita Naik, ML Engineer at Lyft

The most important and major use case of machine learning has been the ETA prediction problem, which is the estimated time of arrival. That's basically the duration that riders see on the screen telling them when they can expect the driver to arrive. But while riders just see that time estimate on the back end, there are like routing algorithms that need to crunch through like historical ride patterns, even like real time traffic and weather conditions to be able to predict this critical estimate as accurately as possible.

Rashida Naik, it's great to have you on the show.

Yeah. Hi Adele, thank you for having me. Yeah, it's great to be on the show as well.

It's great to have you. So you are a machine learning engineer at Lyft and you've worked on many use cases in the ride sharing space. I'm excited to see how machine learning is impacting ride sharing. So maybe could you walk us through how machine learning is currently use in the ride sharing space?

I've been a machine learning engineer at Lyft for about two years now and I can confidently tell you that it's widely leveraged for several use cases across ride sharing. I think like Uber, Lyft, even Doordash, they have several technical blogs about the different use cases that teams are working on. Even at Lyft we have teams working on some really cool stuff. So I guess the most important and major use case of machine learning has been the EPA prediction problem, which is the estimated time of arrival. That's basically the duration that riders see on the screen telling them when they can expect their driver to arrive. But while riders just see that time estimate on the back end, there are routing algorithms that need to crunch through historical ride patterns, even real time traffic and weather conditions to be able to predict this critical estimate as accurately as possible. And as you can imagine, these algorithms need to adapt as dynamically as possible, be it to long term pandemic induced disruptions that we saw a couple of years ago, or even more short term. On a daily basis you have road closures or traffic accidents, et cetera. I would say that's the most important use case, which we're still actively working on making more accurate. But then there are also teams working on dynamic pricing and demand prediction problems to be able to leverage search pricing during peak hours to balance the marketplace demand and supply conditions. We also have teams working on safety and security enhancements on the ride sharing platform because with AI we are able to power easy verification for riders as well as drivers or even when riders are on their ride, like completing their ride. We can flag certain deviations from expected paths so that we ensure that the riders feel safe and comfortable as they are getting to their destination. Yeah, I think these are some of the distributed use cases across the company. And with the recent surge in Genai, we also have a lot of NLP powered use cases coming up, especially with respect to automating customer support.

Okay, wonderful. And we're definitely going to unpack a lot of those use cases, as well as how generative AI can be used at Lyft. Maybe let's deep dive into ETA prediction. I think it's kind of related as well to route optimization. How to think about that. Right. Can you take us maybe behind the scenes and explain to us what a typical pipeline looks like for an ETA prediction algorithm? I'd love to see what type of data goes in, what are the nuances that we should think about? Because I think for a user, what they just see is, you know, five minutes, six minutes. Right. Something along those lines. But yeah, walk me through kind of the complexity behind the scene that comes up with coming up with such a number.

Yeah, so I think there are different definitions of eta. Like when you consider the time series of a ride. So before the rider requests a ride, they also see an eta. When they like, enter their destination, they see a bunch of options with the price and time estimate. So that ETA is actually an estimate estimate on our end when we don't even have a driver assigned to the ride. So in that phase of the ride, we are basically trying to estimate, okay, if you were to request a ride, what would be the expected time that you could expect the driver to arrive in? So at that point, since we don't have any driver assigned, we have these dynamic marketplace conditions. That prediction problem becomes like, a little more complicated since we have like all these variables, given that the rider has not even requested the ride. But once the rider requests the ride, that's the like, once we assign a driver, that's the ETAs that like, riders are like, most concerned about. Like, you know, like, is my driver moving towards me? Like, can we expect them to arrive on time? So we actually have a very strong mapping team at Lyft that works on this whole stack, right? From like, routing to the EPA prediction problems. So if I were to like, walk you end to end, Then at the bottom layer we have like, we also like, have our own navigation system called Lift Nav. So the mapping team basically curates all of this data in house using signals from driver GPS sources or even like open street maps, which is like this open source community that curates like these mapping databases. So the mapping team basically is in charge of managing all this data, which contains all the points of interest and road segments that Lyft needs to be able to build routes optimally. This is the mapping layer. Then on the layer on top we have more real time signals associated with drivers, their exact locations, real time speeds, any traffic and road closures, real time events which help us analyze the congestion on these roads. Once we have this data model, we're able to convert it to a graph model to run a Dijkstra search to build routes optimally from say, point A to point B. So once we have computed these routes, we have the ETA layer on top of that. So the mapping team has built really sophisticated ETA models to be able to predict this time accurately along the routes that we calculate given a certain driver. And as you can imagine, like we show the driver a route and we provide them the epa, but you know, like, the driver could end up taking another route. Like they may take a turn that's not that they're not expected to. And so like, to be able to handle these like real time updates that we like, we need to recompute the route and like recalculate the ETAs on top of that. So we have like a final bias correction layer on top which basically trains dynamically like every 15 minutes to be able to capture this error between our predicted estimates versus the factuals. So that helps us to incorporate quick feedback loops into the system which will account for any sudden road closures or traffic accidents.

Okay, wonderful. So there's a lot of nuances here to the pipeline that go beyond just calculating the distance from A to B. And I think this kind segues into my next set of questions, which I think pertain to the challenges really of building machine learning systems that are used by millions and millions of people. You mentioned here the ETA prediction algorithm, but also dynamic pricing, demand prediction, kind of thinking about supply and demand conditions and how to allocate drivers to riders, et cetera. These algorithms are all operating within the Lyft application. But Lyft is being used by millions and millions of people. So these algorithms are being called in any given day, millions of times, which must present a unique set of challenges. Right. So maybe what are some of the biggest obstacles you faced in building these systems at scale? How do you approach solving these challenges to ensure reliable performance and accurate performance of these algorithms?

Yeah, for sure. Like you're right. At the scale that we work at at Lyft, where there Are so many services interfacing with each other to be able to serve nearly like hundreds and thousands of requests per second? I think latency and scale are primary concerns. We want our system to be as reliable as possible, so that's our utmost priority because it impacts user experience and loyalty. We want riders and drivers to return to our platform, so we don't want any single points of failure in the system or at any point in the pipeline. So since we have a microservice architecture at Lyft, we ensure that we have proper fallbacks at both client and server side across all services, so that we can deal with transient errors or blips, network blips that occur from time to time. Even whenever we roll out any feature or a new product. We ensure proper shadowing and testing. No deployment is directly rolled out at 100% to the users because we don't want to hamp their experience in any manner. Talking about modeling specifically, if I were to speak about building ML models, I think one of the common challenges that we faced is teams develop and prototype models offline, test out a new feature or an entirely new model for a certain use case, and they observe really great performance on offline data. But once you deploy the model online, there's some disparity in these metrics and it does not translate to the same performance gains in the online world. So trying to debug and investigate that, that's like one of the challenges even like we also face like eventual model degradation over a period of time. I think that's very common in the dynamic marketplace that the ride sharing industry provides because you have these data drifts all the time, I guess like solving these problems, I would say is an iterative process because like you encounter these problems and then sort of brainstorm on how to come up with solutions. But we also have solid infra and ML platform teams at Lyft that are able to help out with a lot of these issues related to deployments or ML infra related to our services. And just talking to colleagues who have worked on similar problems in the past, that helps you gain perspective.

Wonderful. So let's maybe unpack those. One thing that you mentioned here is the disparity between online and offline data when training a model. Right. I think this is something that's actually not talked about a lot as a challenge in machine learning. When you are training your models offline, experimenting in a notebook on your local machine and then you deploy it into the wild and you start seeing quite different performance in how the model is Behaving. So maybe first unpacking, why does that happen? And then two, how do you approach solving it?

Yeah, so I think debugging that can take a while because I think you need to take it step by step, like unpack several layers to identify where the root cause might be. But if I were to say, one of the common problems is that you basically compute your features offline for training your model. You do a bunch of feature engineering stuff. Steps use SQL to pull data from different data sources and then feed that into your model. But you basically need to recompute and recalculate all of these features in the same way during online inference. So you need to have that sort of parity between your feature sources. Because if your feature source online has a different data distribution as compared to your offline analysis, then that I can definitely tell you will impact the model performance. At least when you're deploying the model initially, you don't want any disparity in your feature computation steps. I guess just having regular feature validation and data validation during online inference helps because sometimes a lot of these pipelines, which are computing features online, can break and result in null values or just all zeros, for instance. So need to be able to intercept that in time during serving these online requests. The way Lyft tries to resolve a lot of these issues, we want to avoid this disparity between online inference and offline analysis to a great extent. And so the data platform team has actually built this centralized feature store at Lyft for serving ML features. So we have both streaming and batching pipelines to serve like real time and historical features. And then we also write these simultaneously to offline data sources so that even during offline analysis, like, you're basically using the same set of features computed using like a single source of truth.

Wonderful. And then you mentioned something as you deploy online and you realize that the results may not necessarily match with the offline training. How do you ensure as well that you're able to roll back any update that you've done in the machine learning system? So I'd love to see kind of how do you. What are kind of the safeguards, the guardrails set in place to be able in case you deploy a machine learning model that behaves incorrectly, that you're able to roll it back and not harm the user experience?

Yeah, I think rolling it out seamlessly is like a priority for sure. So what we do is whenever we deploy like offline prototyping, then we deploy the model to production, we don't enable it at to all users. So we usually shadow the model in the online world, so we have the production path and then we have a shadow execution path so that we are basically able to analyze the performance of the model before we deploy it at 100%. And that helps us compare the parity between the offline analysis that we had and the online model using production data. Once we've confirmed the model performance via shadow analysis, we usually set up AB tests to have another layer of validation to test this new model and treatment versus an existing system in control. Even once we decide to ship based on business metrics, if we end up improving them, then we don't deprecate the existing stable version of the model immediately. We usually have it around for a month so that we are basically able to roll back to the previous stable version immediately if the new model creates any issues. But usually by then we're pretty confident that this model will work in the online environment once it's rolled out to 100%.

Yeah. And when you mentioned that you've done a B test, and I think that's wonderful, but when you look at a B test here, what's the main objective? What's the main metric that you're measuring? Are you measuring some form of metric that relates to users are depending on the use case, of course. But is the metric usually an accuracy metric or a technical machine learning accuracy metric, or is it necessarily like a business metric, like activations of users, customer satisfaction? What's the main metrics that you look at in terms of the effectiveness of a machine learning model?

I think yeah, we ultimately want to see how it translates to business impact. So while model accuracy is important and we usually monitor that as we shadow during these experiments, we basically monitor business metrics and test the impact. Suppose we have a model that improves this ETA prediction problem. Then we would want to monitor rider cancels or the number of completed drives should be statistically significant. We have a bunch of guardrail metrics, like the top line metrics that we did not want to degrade at all. So and depending on that, depending on those guidelines, we sort of make the shift decision.

Okay, great. And let's maybe jump to another challenge that you mentioned here is data drift, which I think is really common in the online inference space. But also I think every organization that had a machine Learning model in 2020 When Covid happened, encountered data drift, for example, because a lot of norms changed in how we approach so many different things in the world. Right. As a species after 2020 happened. So maybe. How do you approach data drift both in detection, but also how do you approach detecting data drift within your model, but also how do you alleviate it and solve it?

I think data drift is a given since these commuting patterns and ride share variables change so dynamically. And that's why we want to be able to intercept any anomalies as quickly as possible. So we leverage a third party tool to set up these feature checks on the model inference site to ensure that we can manually intervene and investigate as needed. So these checks could be simple checks ensuring feature values always are between a certain min and max threshold or within three standard deviations from the mean. So simple checks like that, depending on the feature distribution, can be leveraged. But since this is a given, we usually set up periodic automated retraining with the most recent production data, because that's like the most recent relevant distribution that our model should be trained on. So we set up dynamic data collection of the production rides, data essentially in offline tables, so that we can then run some scheduled cron job to retrain the model periodically to mitigate any of these performance declines before they even happen, I guess. We also have model performance monitoring dashboards and alarms in place so that these alerts trigger whenever the performance drops below a certain threshold over say a two day or a three day period.

And then I imagine another challenge here, which I think we didn't touch upon yet, is, especially when an application is used by hundreds of thousands of millions of people at the same time, is latency, the ability to provide almost real time inference at scale. It must be quite a big challenge because it can make or break the user experience. If I need to wait three minutes to get the ETA of a prediction, I'm probably going to cancel the ride before I get the prediction. So what are the main challenges associated with latency? How do you make sure that you're able to deliver real time inference for so many users?

Yeah, I think so. Latency is definitely a primary concern at Lyft. And since as you mentioned, a lot of these golden parts need sub second inference, we need to be mindful of that right from the prototyping phase. So you can imagine that when we're having these discussions around model architecture and what sort of model do we want to to build in production, we need to make that trade off because you could have a complex deep learning architecture for your models that can result in really high accuracy. But then if you want that sub second online inference, loading these heavy models and memory per request, that can get expensive, then you have discussions around, can you cache some of these requests, can you have asynchronous model calls to be able to serve these requests in real time without impacting any user experience? So, depending on the latency budgets, we make our modeling decisions. And even when it comes to the features that we use for the model, I think we often forget that feature computation in the online inference request also takes some time. So if you have real time features that you want to use in your model, then these streaming pipelines that compute these features every 5 minutes or every 10 minutes, that can get expensive very quickly because of course these streaming pipelines need to run SQL queries on that volume of data every five minutes. So being mindful having that consideration in place when you're prototyping, I think that is also important. We have some strategies in place. Of course, it depends on the use case as well.

Okay, and you mentioned here, terrain really depends on the model architecture. I'm curious to see what are kind of, you know, of course the old adage remains the best model for the job is the model that performs the best. Right. It's not like there's a preference on the model. But what are the class of models that Lyft usually uses for these types of use cases such as ETA prediction, et cetera? Is it deep learning based architectures? Is it more time series forecasting and Luftski can of the class of models that are used here?

Yeah, so I think we use a lot of tree based models for a lot of our use cases because we've often noticed that for these structured. Right. Sharing data sets, tree based classifiers or regressors, they are super powerful, they're able to capture complex nonlinear relationships and at the same time they are more lightweight and interpretable as compared to deep learning. But then the mapping team I think uses a lot of deep learning models, like some computer vision, even for ETA predictions. I think they're migrating to deep learning now. So we have a blend of both. But I think we also have some reinforcement learning in fact at Lyft for some of these pricing problems. So I would say yeah, it depends on the use case at Lyft.

Yeah. I'm curious to see now how is reinforcement learning used in pricing? I'd love to kind of of learn how because you don't hear a lot of reinforcement learning use cases in production. So yeah, I'd love to see how you've approached it.

Yeah. So since we need to be able to optimize for multiple objectives in a dynamic environment, they basically employ Reinforcement learning to be able to learn through trial and error and through constant feedback. So basically they try to maximize overall objectives like profit, customer satisfaction, even like driver availability. So the actions like in the reinforcement learning system would basically involve like selecting these pricing multipliers because you're trying to implement search pricing. So do I, you know, double the price or do I half the price? Like what are those multipliers that I need to apply to be able to adjust the base fare in response to current conditions. And then the environment would like basically be factor factors like the demand and supply conditions, the traffic conditions, time of day, etc and basically the rewards which we talk about in the reinforcement learning system would basically be the effectiveness of these pricing decisions depending on metrics like revenue or right completion and acceptance rates. So the reinforcement learning model is basically able to learn a pricing policy that is like maximizes long time rewards and is able to balance between like exploring new pricing strategies versus like exploiting existing successful ones. And yeah, I think it's basically able to adapt and continually learn from new data to be able to refine that to improve efficiency and responsiveness over time.

You know one thing that comes out from a lot of what you're saying is, you know, there's the modeling layer, there's the infrastructure layer, there's the embedding it in the user experience, the A B testing layer, right. And I'm sure there's also a design layer, right? Like how, how is the prediction served within the application in a way that the user immediately internalizes. And I think that requires a really strong collaboration between different teams from machine learning, engineering, product design. So maybe could you walk us through what that collaboration looks like in practice? How do you make sure what does ideation to production look like across these different teams?

What I really like about Lyft is we have a bottom side up culture. So like a lot of ideas bubble up from like individual contributors within each team and then like move from ideation to productionization. So like we usually have these quarterly or annual like brainstorming sessions where like the teams get together to sort of brainstorm a bunch of ideas that they like envision themselves working on during the next quarter or the next year. And depending on like some rough impact sizing, we then like proceed towards work with product managers to sort of understand the requirements of the product so that we can like come up with like model requirements and like sort of outline the development and execution plan. And each project usually like requires cross functional collaboration because you could like have dependencies among multiple teams for A given product. So we also like work with PMs and engineers from other teams to sort of build like a small working pod for that particular project. And then in the prototyping phase it's basically like more iterative where like machine learning and data science like work together to sort of offline prototype and work through like the model requirements, but also like talk to engineers because is you need to understand the feasibility of actually putting this model into production as well. So that's sort of an iterative process. And then once we have a sort of developed system in place, then we talk to stakeholder teams, understand dependencies like train systems on their end as well. Because as you can imagine what happens a lot of times is like like we update our systems like launch some product within our service. But then if there are like multiple dependencies on downstreams and they don't update their models or their systems to like start consuming these new predictions then there's like like during a B tests that reflects because it does not impact like any business metrics or sometimes even impacts it negatively. So I think like proper cross functional collaboration like and communication that's also key so that you're able to basically shadow the pipeline end to end and launch experiments with all parties on board.

Yeah, that's wonderful. And you mentioned here one thing that you kind of alluded to at the beginning, working offline and then working online are quite different. I think one thing as well given your background working in academia versus working in real world machine learning is actually quite different as well know you've made that transition from academia to real world machine learning. So I'd love to know what are the biggest challenges within, you know, machine learning, the biggest challenge you face during that transition. Right. And what are common advices that you would give someone coming out of academia going into a machine learning engineering job?

Yeah, I think after I completed grad school and started at Lyft it was a little challenging since I started started peak Covid. So I transitioned into like the industry in like a remote work environment and I couldn't like meet with my mentor or manager to be able to like ask questions, collaborate with people. So that was a little challenging having to like rapidly adapt to a new technical stack, learn the internals of like the company's infrastructure, even like ML pipelines. So there was a significant shift I would say from my academic experience where I was working with clean curated data sets in control jupyter environments. I was in that local optimal mindset focused on improving model accuracy or just tweaking model architectures. But in the industry, you need to take a step back because the model is generally a smaller component providing value to a much larger system. So you need to be able to balance model accuracy with real world constraints like the scalability, latency and some of the challenges that we spoke about earlier.

Okay, wonderful. And you mentioned here the business skills and understanding, asking the right questions and being able to kind of navigate the environment. What are the kind of soft skills and business skills you think you need to have to succeed as a machine learning engineer?

Yeah, I think just like not having any inhibitions when you're starting off with a new job when it comes to asking questions or questioning the why behind a lot of the design of systems. When you're starting out in a new company, it's very easy to get overwhelmed. Even if you join a team that owns multiple services, you try to dive deep and then you have 10 different architectures that you need to onboard onto. But I'd say your colleagues are usually more than happy to help and just being able to talk to them and understand why certain systems have been built that way. Reading documentation, I think is also really helpful. At least at Lyft we have a really rich culture of maintaining documents for whatever decisions that we make. So reading up on the history of why certain services were built or models or prototype, that's also super helpful. And I'd say for machine learning specifically, it's really important to stay up to date with what's happening in the industry as well, because once you start working, you're engulfed by your own project, the tools that the company is using. But the machine learning in the industry, the deep learning world, that's changing rapidly every day. Every day there's some new model that's being released or that's surpassed the previous state of the art. So just being aware of the trends that are in the industry so that you don't find yourself stagnated or stuck after a period of time, I think that's important as well.

And then you mentioned this at the beginning, like you're investigating generative AI use cases at Lyft as well. As we look into the future, what are genai use cases that Lyft is working on at the moment? And how do you see this technology also shaping the future of ride sharing?

Yeah, I think there's a lot of buzz even within Lyft and ride sharing broadly to identify ways that we could leverage Genai at scale and to basically streamline the ride sharing app experience. I guess within Lyft There is a strong effort to integrate these tools to improve internal productivity. The ML platform team has basically been trying to integrate ChatGPT into tools like Slack and GitHub to improve employee productivity because now we can ask questions against internal documentation or data tables to of get answers more quickly than having to parse through a lot of threads as we used to earlier. Even using it for summarization and sentiment analysis of writer driver feedback for gaining insight that has reduced manual analysis time to a large degree. As I mentioned, we've also been leveraging these advanced chatbots for complex query resolution to be able to generate to generate personalized responses for common customer concerns without having to step in or provide human support for each ticket that customers raise. But I think going forward there's still a lot of scope. You can imagine hands free platform usage when it comes to Lyft where you're using AI powered conversational booking systems to like request rides and manage your trips just by like speaking into the app or even like leveraging real time language translation when you're traveling globally and like don't know the language of a certain country but still want to be able to communicate seamlessly with your drivers.

Okay, that's wonderful. That's really exciting Use cases and you know, beyond generative AI as well. What machine learning trends are you excited about the most and what are kind of of emerging technologies or methodologies in the machine learning space that you think will have a big impact on ride sharing as well?

I think ride sharing could definitely benefit from continual learning. I think that's something that's more isolated at Lyft. Today I spoke about the ETA bias prediction layer where that sort of model updates rapidly real time. But I think there are a lot of use cases across Lyft that could where the models could basically adapt to changes and improve more dynamically because these robust continual learning frameworks basically will help improve the efficiency and responsiveness of our machine learning systems and basically more rapidly adapt to traffic and weather changes which influence these rider driver matching decisions. I guess I'm also excited by reinforcement learning. I'd say I've had the chance to read about some of the algorithms that pricing uses, but I feel like even other use cases across Lyft pair an agent learns to maximize multiple objectives learning from feedback that could benefit this dynamic ecosystem. Because right now we're using a lot of of supervised algorithms where the model is trained on a fixed data set in time. But basically since we have constantly changing variables, I think RL would be more suited for this type of environment you could imagine more efficient rider driver matching, for instance, by considering factors like demand and supply conditions and acceptance rates, et cetera, to be able to better match riders and drivers, to reduce wait times and efficiency, for instance. So yeah, these are some of the trends I'd say that I look forward to leveraging in the future.

That's exciting. And then maybe before we wrap up, Rashida, do you have any final advice or notes you would share with listeners? Whether it's about machine learning, your work at Lyft, or anything that that aspiring machine learning engineers could benefit from, I.

Think I'd just re emphasize on the fact that try to engage in continuous learning. I think it's important to build your skills by either taking courses off these sites, like Coursera or Udemy, for instance, or reading weekly newsletters on the latest news. Whichever industry you're most excited about it need not just be machine learning. But I guess at least in ML, I think experimentation is key. So outside of your work projects, also trying to explore new ideas to build smaller prototypes that leverage these new technologies and trends. Because I think that's how you start thinking outside the box and like try to apply those algorithms to your work as well. Like that's how you sort of try to link use cases and like come up with new ideas that you can also like maybe turn into a tangible product in production. I think it's also like important to like find what drives you at work because I think that's like important if you want to be able to work in a sustainable way over a long period of time. Because for me I think the most fulfilling aspect is knowing the direct impact of my work on people lives. Like every feature we roll out or like product we launch has like we can see the impact on a user, even like through these A B tests for instance. And like seeing those metrics move tells us, okay, the users are liking these features. So that really motivates me and like, like pushes me to sort of come up with new ideas to like resolve some of these pain points. So yeah, I think like, hopefully like our listeners had something to gain from from these insights, but it was really great being here.

Yeah, they definitely did gain. I gained as well. Rashida Nag, thank you so much for coming on Data Framed.

Thank you. Thank you so much. Adele.

It.