#134 Building Great Machine Learning Products at Opendoor

Hi, I'm Maggie, VP of Curriculum at datacamp and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey on datacamp.com.

First, I think you need to have good cross functional partnerships. You need a data science and an engineering team that work very closely together. An anti pattern here that I think is relatively common is the throw it over the wall approach where data science prototypes a model, sees that it works in backtesting and then throws it over the wall to engineering to productionize. We try really hard to avoid that at Opendoor. And so in the very early kind of model prototyping discussions, before any code has been written, we try to make sure engineering is involved and conversely, we try to make sure data science stays involved as we get into productionization and is there at every step in the way to help engineering think through all of the nuances and all of the challenges that inevitably come from moving a model from prototype to production. We also think a lot about the collaboration cadences and rituals that bring together our operations, our design team, our commercial teams, and get them to work with our data science and engineering teams around ML, product ideation and creation and monitoring. And product managers play a big role in bringing all of these stakeholders together. But ideally, it's not all going through product managers that you have engineers who are directly talking with people from customer support, or data scientists who are working directly with people from operations.

Hello, welcome to Data Framed. Hi, I'm Adele, Data evangelist and educator at DataCamp and data framed is a weekly podcast in which we explore how individuals and organizations can succeed with data. Today we're speaking to Sam Stone, director of Product Management of Pricing and Data at Opendoor. For those who are not aware, Opendoor is an operator of an online real estate marketplace used to streamline the sales process of home buying and selling. And they do that by using machine learning models to instantly price homes and provide recommendations to its millions of users. You know, oftentimes when we think about building great machine learning products like Opendoor, we get fixated on the nuts and bolts of machine learning, which algorithm to use, what are the features to be engineered, and so on and so forth. But to build truly great machine learning products that reach millions of users, you need to marry great data science expertise with strong attention to user experience, design thinking, and a deep consideration for the impacts of predictions on stakeholders and building great machine learning products. The type that I'm speaking of is something Sam Stone has a deep understanding of.

So I wanted to have a deeper.

Conversation with Sam on what he thinks a great machine learning product looks like.

Throughout the episode we spoke about his.

Principles for great ML product design, how to think about data collection for these.

Types of products, how to package outputs.

From a model within a slick user interface, what interpretability means in the eyes of customers, how to be proactive about monitoring failure points, and much more. If you enjoyed this episode, make sure.

To subscribe to the show.

And now onto today's episode.

Sam Stone, it's great to have you on the show.

Thanks. Excited to be here.

So you are the Director of Product Management at Open Door. You've overseen the development and maintenance of machine learning products that are extremely integral to the Open Door value proposition. So maybe let's start at the very beginning. What do you think makes makes a great machine learning product?

First, before I even get into that, really excited to be here. Thanks for having me on the show and excited to talk about what we're doing with machine learning at Opendoor. I think there are three things that makes for a great enduring ML based product. Number one, building a product that stems from deep understanding of user behaviors. And that's true for every product. Machine learning products are in no way special in that regard. Number two, optimizing for the accuracy in your ML product in a way that's aligned with what users actually want. And number three, building feedback loops into the product from the very beginning so that that product can be continuously improved.

That's very great and I'm excited to unpack all of that in our discussion today. But before, let's maybe focus on how machine learning drives value. Open Doors so maybe to start off maybe, what is Open Door's mission? What is the business of Open Door?

Open Door is an e commerce platform for residential real estate that allows you to buy, sell and move with a tap of a button. So if you consider the traditional process of selling a home, at least in the US you hire a realtor, you have a bunch of open houses, you maybe get some offers, they fall through, you negotiate, you do repairs, you get more offers. It's highly uncertain. There's a lot of friction, A lot of Inconvenience and the whole thing might take weeks or months or maybe even more. With Opendoor, you can go to our website, type in your address, answer a few questions, and very quickly get a real and competitive offer for Opendoor to buy your house. And so that is much simpler, much more certain, much more convenient. Now, the way that we actually deliver on that certainty, that speed, that convenience, is that we buy the home, we use our own balance sheet, our own money to purchase the home, which means that if we misprice that home, that mistake comes out of our own pocket. And so that probably starts to give you some indication of why algorithmic accuracy in ML products are really important at Open North.

That's really great. And the second half of that question is, where does machine learning come in? You mentioned the importance of algorithmic accuracy. Where does machine learning provides value for Opendoor?

So I already mentioned pricing, and that goes all the way back to the creation of the company. The very first lines of code that our technical founder wrote were a pricing model to value homes. And this is before we were even operating. They were just like testing out or thinking about the business model. One of the really interesting things about residential real estate in the US is that eventually all the transactions become public record. And I'll probably talk more about this later. But what that meant was that we could build a pricing model before we had bought any homes on that public data and see how accurate would we be and then flow that through into a simulated P and L and say, do we think this business model will work? And obviously we did, and we launched the company and it has worked. Pricing has continued to be a very large focus area for our ML products. And we've taken the pricing problem, we've broken it up into discrete parts. So one part is what is a home likely to sell for if it's sold on the open market today? Another part is if we buy the home, we'll hold it for some period of months. What's likely to happen to the price over that period? So that's very much a forecasting problem. Whereas the first problem was a now casting problem. A third area of pricing would be how do we run an effective auction when we go to resell that home. Beyond pricing, we've added ML models in a number of areas over the years. For example, we do recommendations and personalization. So for home buyers, based on their preferences, what homes are they most likely to be interested in? We use it for trust and safety reasons, to detect signals that may indicate potential incidents in the homes that we own where we need to go investigate and assure that that home is clean, safe, functional and safe. We use ML for allocating our marketing budget and there's a whole bunch of other use cases as well.

That's a very holistic overview and I think it really begs to unpack a lot of what you think are best practices in delivering great machine learning products. So maybe let's start off with the first component that you mentioned, which is developing products that are really based on deep understanding of user behavior. Of course, it goes without saying data makes or breaks a machine learning product, but you mentioned how in order to create build machine learning products as well, it's really important to not just know the data to really understand the user. Tell us maybe why that's so important. In the context of a machine learning.

Product, there's often something that is lost in between the real world and the data exhaust that comes out of the real world or the digitization of user behaviors. And I think it's really important to start with the users and directly observing them and then using that to inform how you think about data. So an example of this is that we have lots of data on homes. Oftentimes we have multiple different sources that are telling us about homes, and those sources conflict. This is a problem for us. A very common pattern that we see is that we will have data on the square footage of a home from a listing on what's called the Multiple Listing Service, or the mls. This is essentially a database maintained by agents for listings with at least one or more MLS per market. And we'll have data on the square footage of home from county, tax assessor or tax recorder offices. And those numbers will disagree. One will say this home is 1,000 square feet, and the other will say this home is 2,000 square feet. Now, if you look at that from a purely technical perspective, that's pretty perplexing. But if you think about what users are doing when they are creating this data, it actually makes a lot more sense. When you are selling your home, you have an incentive to position your home in the best light possible. You're probably going to describe the square footage as the largest number that it would be reasonable to use. But when you're reporting data about your home to tax authorities, you want to minimize your tax bill, and you're probably going to report the square footage that is small as could be reasonably construed. And so when you start to understand those behavioral elements, it gives you a lens in which to understand the data and to say, okay, I've got These two different sources, they disagree. But that actually makes sense to me and I understand the bias in the data and how to start to reconcile that.

That's really great. And I love that example that you mentioned because it really is as well the two sided marketplace that Opendoor works in in a lot of ways from the perspective of the seller and for the perspective of the buyer. I want to understand maybe in a bit more detail for you as a product leader and as someone who works with data teams as well. How do we reconcile these differences and how do you reconcile the user behavior with the data that' fed into the machine learning model?

There's a couple different strategies here. I mentioned understanding the way that data is generated. That's strategy number one. The second strategy that I would think about here is understanding what users are really paying attention to that might not be easily available as data, and then going out and figuring out how to get that data or how to work with that data. Sometimes the problem is getting the data. Sometimes the problem is you have the data, but you're not sure what to do with it. An example of this would be photos. So a lot of the time when people start to build models related to real estate pricing, they start with the data that's easy to work with. They start with structured fields like bedroom count, bathroom count and square footage. Right? That's kind of like Data Science 101. I've got a couple of scalar values and maybe I'll throw in some categoricals. But for anyone who has ever bought or sold or rented a home, you will know that no one is going to move into a home without seeing it. Now, pre Covid that probably meant seeing it in person Today. Sometimes people only look at photos or videos, but it's a hugely visual decision, right? People don't make that decision just based on the structured data. You absolutely need to pay attention to to the unstructured data. Now that's a very hard modeling challenge and there's different ways to go about it. But any good solution to that problem, be it you're using computer vision or human in the loop systems, any good solution to that problem is going to stem from the principle that a model, to be highly accurate, needs to make use of that difficult unstructured visual data that users pay so much attention to.

Have you felt that throughout your time at Opendoor and collecting feedback from users, did that user feedback alter and required updates to how you collect data and your data collection strategy and how you structure data that goes into machine learning products? At Opendoor Absolutely.

I think for any ML product there's an evolution of you need user feedback loops, you need ways to show your predictions or your recommendations or whatever you call the output of your ML product to users. And at that point in the user journey, the user needs to be able to tell you something about what they think about your prediction. And I think part of that needs to be unstructured. Right. Not a lot of users are going to do this. Right. Unstructured feedback is higher cognitive load, but it will point you in directions that you never thought of. And then it's normally good to accompany that with some structured fields which are easier for users to fill out. I'm selecting from a dropdown of I think this is off and here's the top three most common reasons. And that can be turned into analytics and that can be processed by ML development teams based on aggregated insights. But I think you want both of those strategies for feedback loops that are embedded in the product at the point where ML outputs are exposed to users.

That's really great. And we're going to discuss feedback loops a bit more because you mentioned that as an important aspect of building machine learning products. But first let's talk about how you actually deliver a machine learning product within an application for a user. Right. I think a lot of data teams tend to think about Mach machine learning systems that go into production as just an algorithm, something that needs to be packaged into an API. But there's also a really important aspect of how the machine learning system is going to be packaged in front of the user within an interface or a software. It's one thing to produce a prediction, but it's another thing to produce a prediction that is understood by the user in a way that creates user trust and where the prediction is presented cohesively as part of a user journey. So maybe walk us through example of where that's been important for machine learning products at Opendoor and how you've approached packaging machine learning systems systems as part of the open door application.

This is an area I'm really interested in. It's this kind of fast growing field of model interpretability. And I think you hit the nail on the head when you talked about user trust and interpretability is key to user trust. And there's a couple different benefits from that. First of all, users who trust ML outputs are more likely to use the product or continue on in that journey. Second, if they understand something about the model output, they're more likely to provide feedback that's useful to the team. And then third, I think interpretability is really important for development teams. As you think about the long term arc of iterating on an ML product. Understanding why the model output is what it is at an individual inference level is really key to generating good research hypotheses that turn into model improvements. So at Opendoor, we spent a lot of time thinking about model interpretability and how that goes hand in hand with long term accuracy improvements. When we show a price to a home seller, we don't just show that price to the seller. There's two broad parts of what we accompany that price with to make it interpretable. The first is actually relatively basic. It's what are the inputs that we used around that home to generate the price? Now some of those inputs the user, the home seller has told us. But there's a lot of home attributes and neighborhood attributes that go into our pricing model. And we don't want to make the user answer 100 questions. If in a particular neighborhood we're using 100 different inputs to value their home, especially because we have a lot of those inputs from public data. But it's important to show the user what we used so they can say, oh yeah, this looks wrong, or maybe this was right up until a month ago, but this thing changed. And when they can provide us that kind of concrete feedback, we investigate it. And a lot of the time they're right and we go and we raise their offer. That's a very kind of deterministic, I would say, relatively straightforward path. They can be implemented for pretty much any model. Right. To show the inputs to the user that you are using. The second part of our interpretability strategy is showing how those inputs were transformed into the price. Now this is where it gets much more domain specific and there's a lot of creativity and nuance that is required. So I want to step back and talk about how do actors in the residential real estate marketplace produce prices? How do sellers come up with their willingness to sell? How do buyers come up with their willingness to pay? Most people follow a three step process and they might do this explicitly if they're an agent or they're an appraiser, or they do it implicitly just in their heads. The first step, they select comparable sales, homes that have sold nearby that they think are similar. Second, they adjust the sale prices of those comparable sales. So maybe the comparable sale is your neighbor and your neighbor's home is identical to yours, but you have a pool and your neighbor doesn't and your neighbor sold for $500,000, you're going to say, okay, I think my house is probably worth $600,000 because I have a pool. And then the third step is they take a weighted average of a couple of adjusted comp prices. So this looks very different, by the way, than just doing like a linear regression, right? This is really a three step process. And maybe you would do a linear regression at each step or maybe use a neural network at each step. But the core part is that there are comps, you adjust them and then you take a weighted average and we expose information about each of those steps to our end user. And that allows us, that allows them again to trust the process more and to provide us really useful targeted feedback where they can say things like, hey, this comp that you used, it's actually not very similar to my home and here's why, or you missed a comp my neighbor literally just sold yesterday, and maybe it wasn't in your data set yet, but you should certainly consider this. If we look beyond Opendoor, I think we're starting to see some really interesting applications of interpretability for especially complex model types, especially deep learning models. For example, in computer vision, we're seeing really interesting work where computer vision models can highlight what part of an image they're paying the most attention to. And in large language models, increasingly the models are able to seamlessly explain their reasoning just by including the request to explain yourself in the prompt. Or you can say something even more specific, you can ask for citations, and a language model can tell you specifically what sources it used to.

That's awesome. And you mentioned at the beginning talking about interpretability and user trust. There's also an emotional aspect to this, right? People don't want to feel that their home is undervalued. It's a place that they've been in for decades maybe, and they've been really working hard on making sure that their home has high value. And I love how you leverage model interpretability data science metrics in a lot of ways to package it as part of a product to create that trust. Are there any lessons that maybe you can share with other product leaders or data leaders around incorporating product management design thinking with data science metrics and data science interoperability as part of a mechanism to create more trust and to create a stronger positive experience for users within a product?

First and foremost, don't assume that what is interpretable to you or your data science team is going to be interpretable to end users. The people who are developing the model are so close to it, they understand all of the nuances all of the pros and cons. And a lot of the time you need different interpretability strategies for end users versus internal users.

And what are ways you've been able to translate the findings of the data team in terms of model interpretability, to make it and package it in a way that is much more empathetic towards the user needs and problems.

One strategy that we do a lot of is we evaluate individual ML inferences that we have some reason to believe were really, really off. Either a user provided direct feedback to us that they thought it was off, a member of our operations team flagged something, or and this is a great aspect of the residential real estate market, we can compare our price estimate versus what the market actually transacted at. So we might have offered on a house, that seller didn't take us up on our offer, they sold a month later. We can look at that price and say how did it compare? And so we have a sorting process for all these different sources where we are identifying individual instances and then digging into those and saying how do we produce that prediction? Which model inputs were leaned especially heavily on? Was there any that the model didn't weigh heavily that we think maybe it should have? What was the sign, what was the magnitude, what were the interactions in the model? And all of that is being done at the different steps in the process.

Right.

I mentioned one model that had a three step process. We're really going to that level of detail.

So let's jump to talk about metrics and evaluation. This segues really well here. As a product leader, I think it's very intuitive for you to think about North Star metrics and there needs to be one single metric to optimize for, for the success of a product in general. But I think a lot of data teams that may not be intuitive for them because when you design a machine learning system, right, you have dozens of metrics from like recall, accuracy, precision, auc, et cetera. So maybe to start off, how do you define a North Star metric for a machine learning product?

I think it's good to be aware of and to track many metrics. And at a product level, I think it's also very common that you have paired metrics. One really common set of paired metrics would be something focused on revenue or transactions or some notion of volume versus some notion of profitability. Most businesses have that trade off and it gets reflected in most products. However, for most machine learning models in the actual training process, you have to pick some loss function that you are seeking to to minimize. And I Think this is where it's important for an MLPM to understand the details. I think that in a well functioning ML product team, researchers or data scientists are the ones who are going to propose what that loss function will be, maybe articulate a couple different options, lay out the pros and cons and say this is the one that we would recommend. But I think a PM should be engaged enough with the details to understand those pros and cons, to ask probing questions and to really be a sanity check there. No loss metric is perfect. Anyone is going to have pros and cons. And some of those are pure data science pros and cons and you need data scientists to weigh in on those. But some of those are more kind of business related. And that's where I think a product manager can be really, really helpful. An example of this at Opendoor was that when we first built our recommendation product for home shoppers, people considering different listings to go tour and ultimately to put in an offer on, we tried to minimize loss in terms of listings that buyers clicked on. What this resulted in was a model that showed highly visually unusual homes. A lot of them were mansions or they were homes that had something else in the photos that just really caught your eye. They were a lot of fun to look at, but they tended not to be homes that our users home buyers would actually go to or much less offer on. And so we had to shift to a different loss function, a different kind of core metric that our model was trained to, to minimize, well, minimize the loss. But what it was doing was we actually trained it on offers that buyers submitted. This was a big challenge for our data science team because there were a lot fewer offers in our historical data than there were clicks. So they had to deal with a bunch of data sparsity issues. But at the end of the day, this resulted in a model that was much more aligned with actual user intent.

That's really great. And maybe deep diving into that last example, what are common ways to mitigate data sparsity issues, especially when you're trying to optimize for outcomes that are not yet recorded and you're trying to build out an initial product right as a product leader to be able to get that type of outcome that you want from particular users, provide them with that value and you're still building it out early in the journey of the product life cycle.

First, I think it starts from having a strong perspective on what matters to users. To translate that into data science terms. If you have a set of inputs, you should probably have A strong a priori perspective on the sign and magnitude of the relationship between each of those inputs and the output. Maybe not for all of them. And there might be interaction effects, but you should know which factors you think are most important and how you expect those behaviors to occur. And if your data is really sparse, you might start with a set of heuristics and release a product that is purely heuristic based and use that to gather operating data or real world kind of user interaction data. I think the second level when you move on to an ML product is to use a relatively simple, highly parameterized model. So take those priors that you have about how the features should work and actually encode them to some extent in the model and maybe allow your model to update as new data surfaces. But don't force the model to learn the entirety of your belief set just from data, because if you don't have much data or the data is currently sparse, that's just going to lead to a weak model.

Okay, and related to Northstar metrics, what do you think are common pitfalls? Maybe when you're trying to define a North Star metric and as a product leader, what are systems that you can put in place place to ensure that you're always optimizing for the correct outcomes?

I think there's two parts to that question in terms of common pitfalls on North Star metrics, I think at the more technical level, choosing metrics that are overly academic, maybe it's really cutting edge. But if it's difficult for your team to understand, or very few people deeply understand it, that probably outweighs the benefits of using a loss function that might be slightly better than the more common thing. You also asked about how do you actually monitor? I think you said monitor inputs and outputs, is that right?

No, it's optimizing for the correct outcomes here. What are systems that you can put in place to ensure that you're always optimizing for the correct outcomes? When it comes to developing a model.

I think they tend to be more process and people oriented than technical because user behavior is changing the real world world. And you want processes where you are soliciting internal and external feedback about particular predictions or inferences that look wrong. And you're doing individual investigations. You're also looking at aggregate metrics and seeing how those are shifting over time, be it from backtesting or aggregations of user behavior.

So the third aspect here of building a great machine learning system is understanding the product really well. In creating feedback loop systems where you're able to get immediate feedback from users and to be able to improve the product, right? In a lot of ways, our conversation so far has touched on the nuts and bolts of building an excellent machine learning product. But let's maybe take a step back and think about how to continuously improve it. What does getting feedback look like in practice? What is the mechanism of getting user feedback look like? And walk us through some of the best practices that you've learned there.

I think part of this comes from getting user feedback. Part of it also is about understanding your blind spots where you're not getting user feedback and reacting in those areas. On the explicit user feedback side, I think it's important to embed opportunities for users to provide feedback at the same place in the user journey where they are seeing or interacting with with model outputs. And I think you should provide users the opportunity to provide structured feedback which is generally easier for them to provide. We tend to have higher completion rates around that if it's just say a dropdown menu and unstructured feedback so they can tell us about things that maybe we've never seen before or never considered.

In a lot of ways, what you mentioned here requires a lot of overhead, right? Monitoring metrics, understanding what is going on in the wider world, soliciting user feedback, maybe walk me through what you're doing. Learned to make the machine of managing machine learning products more predictable, right? What can you share about becoming more efficient, about soliciting feedback, about understanding the world around you when building and improving machine learning products?

First I think you need to have good cross functional partnerships. You need a data science and an engineering team that work very closely together. An anti pattern here that I think is relatively common is the throw it over the wall approach where data science prototypes a model, sees that it works in backtesting and then throws it over the wall to engineering to productionize. We try really hard to avoid that at Opendoor. And so in the very early kind of model prototyping discussions, before any code has been written, we try to make sure engineering is involved and conversely, we try to make sure data science stays involved as we get into productionization and is there at every step in the way to help engineering think through all of the nuances and all the challenges that inevitably come from moving a model from prototype to production. We also think a lot about the collaboration cadences and rituals that bring together our operations, our design team, our commercial teams and get them to work with our data science and engineering teams around ML product ideation and creation and monitoring. And product managers play a big role in bringing all of these stakeholders together. But ideally it's not all going through product managers that you have engineers who are directly talking with people from customer support, or data scientists who are working directly with people from operations.

Yeah, that's really great. And maybe expanding on that. What are great rituals that you find to be effective that improve collaboration between data teams, engineering teams, design teams, and that has been able to create this cross disciplinary function with an open door that is responsible for machine learning products?

We have a set of rituals around sharing work in progress. And that can be uncomfortable because a lot of the time people like to share work, especially on the data science or research side, that they feel is polished. They feel like it's ready for peer review. Especially if you're coming from an academic background. I think that's a very common inclination. And instead we push to say, no, we're going to discuss your work before you have a strong opinion on what the right answer is, where you're truly receptive to feedback. And that's very much like a kind of learned organizational skill because it runs counter to a lot of the things that we've kind of been trained in from the corporate world or academia. I think another set of rituals that's really powerful is hearing directly from customers. So for us that will mean getting out to our markets, working with our operations team on the ground, that is inspecting and renovating and maintaining homes, directly, listening from customers, joining calls, watching videos. I mentioned before, we're profiling individual instances of where our ML output seems to have been off. So there's a whole set of rituals around that. And then there's planning. And I think planning processes need to differ by the team and the organization. But one thing they all share in common is that they provide opportunities for a wide variety of people to submit ideas. So an engineer should feel empowered to submit an idea that might not be pure engineering, but might be customer facing. And our researchers should feel empowered to submit ideas that are not just research ideas, but might be about tech debt, pay down, or again, something that's more commercially oriented.

Okay. And related to the example that you gave on regime change. Right. And these types of tectonic events that change the performance of a model. An additional wrinkle that I found here is a lot of the learnings that improve machine learning model systems comes from a reactive mode, right? There is a decrease in a North Star metric, a failure in a metric, and data leaders and product leaders have to react to that. And they tend to improve their product as a function of that I'm definitely not saying that leaders or product leaders or data leaders are able to predict Covid or are able to predict a regime change and how the Federal Reserve approaches interest rates. But what are strategies that have worked with you to be able to go from to be able to be more proactive about improving and stress testing machine.

Learning systems in production first, setting aside time in sprints or planning processes to dig into the things that are unexpected. So it's digging into bugs. That one's relatively obvious. Most teams have an on call, but maybe taking that concept that's common for engineering and applying it to data scientists as well, it's really difficult if you plan to spend your whole week as a data scientist working on model feature X and you feel pressure to ship it at the end of the week. And then there's this really interesting bug that comes in, but your week is booked. That bug probably never gets investigated. I would also say dig into bad customer experiences that may or may not be bugs. Sometimes there is a desire to triage, such that only definite bugs get routed to technical team members because their time can be so scarce and valuable. And there's certainly merit to that. But there's also merit to forging on some of the most uncertain pieces of customer feedback. And I've seen really interesting hypotheses come out of digging into those areas. Finally, I would say in terms of becoming more proactive, go on listening tours with your operators, with your end customers, and also pay a lot of attention to what other companies in your space are doing. Both your direct competitors, but also companies that may be in a different industry, but maybe doing technically similar things. So at Opendoor, we're not doing anything related to trading equities or bonds, but there's a lot of relevant technical work around those types of trading algorithms that we pay attention to. On the other hand, we're paying attention to a lot of innovation in the real estate startup space, even if technically it doesn't look as similar to our stack.

Okay, that's really awesome feedback, Sam. As we close out our episode reaching the 40 minute mark here, I wanted to talk to you a bit about your career and how kind of advice that you can give for other people maybe looking to be in a similar position to yours. In a lot of ways your role sits in the intersection of product and data, right? What do you think is needed to succeed in this type of role? Maybe walk us through some best practices that you can share with the audience here.

I think the things that have allowed me to really enjoy being an ML product manager. And as I look at my team, the things that have enabled them to be successful are one, they have a deep understanding and desire to understand users. Number two, they love data science. There's a kind of fundamental nerdiness and excitement about a high quality data set or understanding the intricacies of a model. And number three, they have deep empathy for engineers. I think engineering is a difficult role in general, and if you're an engineer working on ML products or the outputs are probabilistic, it's especially difficult. And so as a product manager, you need to really understand and have the trust of your customers, your data scientists, your engineers, and a whole bunch of other functions. I think those are three key components to having that and allowing you to be successful and ultimately enjoy the role.

That's great. And what are maybe what level of data science should you know to be successful as a machine learning product lead?

It will depend on the organization. I think you should have passion for it. And one test for passion is you've probably built some models yourself. So I think there is a level of ability to code and implement things and learn from that firsthand experience that is necessary going a level deeper than just reading about it and knowing the concepts. And then the second litmus test here is you probably want to know more. Whatever your level of knowledge is, there's probably a handful of things that you would be really excited to go deeper on if you had the time.

Okay, that's really great. Finally, Sam, as we close out our episode, do you have any final call to action to share with the audience?

Yeah, if you are buying or selling a home or you happen to live in a home, so pretty much everybody out there, you should check out Opendoor. And even if you have no interest in real estate, you should also check out our blog, which is called Open House, where we have a ton of great posts on data science, engineering and product management topics.

Okay, that's great, Sam, thank you so much for coming on Data Framed.

Thanks for having me.