#65 Preventing Fraud in eCommerce with Data Science (with Elad Cohen)

Hello, this is Adele Neme from datacamp and welcome to Data Framed, a podcast covering all things data and its impact on organizations across the world. I think something we can all relate to during 2020 and the confinement it brought upon us last year is the rise of E commerce and just how digitized retail has become. E commerce is known for its data science use cases, whether market basket analysis, recommendation engines, marketing optimization techniques, fraud detection, and more. This is why I'm excited to have Elad Cohen on the show. Elad is the VP of Data Science at Riskified, a startup specialized in machine learning based fraud detection solutions for E commerce. Elad is responsible for the ongoing improvements of the machine learning algorithms that power Riskified. He has over 12 years of experience managing data science analytics teams across research domains. He holds a Master's and Bachelor of Science in Applied Physics from Bar Ilan University and an Executive MBA from Tel Aviv University. In this episode, Elad and I talk about the state of e commerce and the plethora of data science use cases. Therein we deep dive into fraud detection and what are the best practices riskified learned when creating bleeding edge fraud solutions. We also talked about his best practices leading and organizing data teams such as balancing long term research with short term goals, keeping business stakeholders engaged, how the data team could own an organization's data literacy and the future skillset data scientists need to hone in order to be successful. If you enjoy today's conversation with Elad, make sure to also register for our upcoming webinar with him where he'll deep dive into much more detail on the ins and outs of scaling fraud detection solutions and other e commerce use cases in data science. We made sure to include the registration link in the show notes. If you want to check out previous episodes of the podcast and show notes, make sure to go to datacamp.com community podcast Elad, it's great to have you on the show.

Well, thank you very much first off for having me here. Really excited to be here.

I'm really excited to talk to you about data science and e commerce, your experience at Riskified, fighting fraud with data science, and more. But before, can you give us a brief introduction about your background and how you got into the data space?

Sure Del. So my background started with a Bachelor's in Physics. I continued to Masters and was working actually with lasers which was really interesting and really cool. From there I went to the military focusing in the field of signal processing and statistics and algorithms. And I remember at one point when I was working with lasers I thought you know, I consider myself a laser guy. And then afterwards in the military, I had the same kind of thinking where I was like, okay, maybe I'm a signal processing guy. And when I thought about it a lot harder, I came to the realization that, okay, you know what? The domain isn't actually the most important thing. What is connecting these is my love for data and for research and for being able to look into what's happening and understand it. And that was back in late 2013 when I finished my military service. Data science was really starting to come to life and I decided to shift there. So I had already had some background in a bit of machine learning and statistics, but really jumped in with a lot of online courses with all and more machine learning algorithms and actually I learned a lot on datacamp. So it was a great process.

That's great. So before deep diving into the data science applications in e commerce, I want to first start off by asking you to share your thoughts on the state of e commerce today. The past year has definitely forced many organizations to start thinking digital when selling products and consumer to buy products using digital channels. How do you view the state of e commerce today and what are the forces that make it such an interesting space for data science use cases?

So e commerce is definitely in a super interesting space right now. So you think about last year. Covid catapulted us forward by several years. You have scenarios where people physically can't get to shops. You've got quarantines and lockdowns and they have to continue buying whatever they want. And all of these brick and mortar companies realize that they have to go digital, they have to start selling in e commerce. Everyone joined the show and on top of that you're seeing all these sorts of new flows. So things like buy online, pick up at curb or bopak, all these scenarios where you're trying to make the customer experience as safe as possible for customers. And we saw huge transitions. You've also got in the same time lots of new shoppers. So people who are maybe a little bit older aren't as tech savvy, who were not as accustomed to buying with e commerce before they had to make this transition and they had to start buying online as well. And this happened across lots of different verticals. So we saw across really all the different industries a significant uptick in the purchase patterns as well as changes in shopping behavior. So things that may have been riskier or safer all of a sudden are changing. Lots of people are buying home fitness goods and sweatpants and you know, Lots of food and whatnot. I think the most important thing to take from this is that it wasn't a temporary situation. What happened was that customers who may not have been buying with E commerce as much beforehand understand that, okay, there's actually a really significant advantage of the customer experience that you can get when you're shopping online versus when you're shopping in a store. And E commerce is here to stay, and it's here to stay in a very, very big way. We really shifted a few years, fast forwarded a few years during this experience and today when you're looking at it, you can see a situation where customer experience is phenomenal and the competition is getting really, really fierce. So the companies that are doing this as best as possible are trying to make it as frictionless as possible. They want to have the best possible customer experience and give the customer everything they want. They want to go omnichannel and allow you to start buying on scanning with your mobile and then finish the purchase with your computer and ship to a completely different location and so on. And data science plays a huge role in all this. Being able to connect all of that data, understanding who you're working with and who you're talking to so that the customer can switch between devices, they can switch between channels. You're still able to understand who is this entity and who's this person that you're working with. So for them it becomes a completely frictionless, almost magical experience. That's all due to data science and how we can help.

That's really exciting and I'm really excited to talk to you about different use cases of data science and E commerce. E commerce tends to be an umbrella term that sometimes comprises multiple different activities. So when we talk about data science use cases in E commerce, there could be a host of different use cases in different domains. Can you walk us through some of the use cases and which ones you think are the most interesting for organizations in this?

Sure. So if you think about it from the E commerce perspective, okay, you're a business that you want to sell and give the best experience possible. So at the end of the day, you want to make sure that your customers are able to find the products that they want. They're going to have a really easy time to purchase them and have an overall great experience. So there's a few use cases that are going to fall within this. The first, probably one of the more basic examples, is you need to have a good recommendation system. You want to know your customers, understand what were their buying patterns and what are the things that they're interested in? What do they like? And based off of that, give them good recommendations that will actually captivate them and they're going to want to be interested in buying those products from you and not from a competitor. Some additional use cases that are really important, for example, pricing optimization. So I'm sure that you've already seen cases where if you don't buy something for a while, you're going to start getting emails with more discounts. And this is extremely personalized and tailored. On the one hand, you have this capability of giving out discounts to certain customers so that you'll increase their likelihood to buy. On the other hand, if you give too many discounts, then you aren't going to be as profitable as you you could have been. So the way that merchants are able to optimize this and to really maximize the revenue is extremely important. They need a lot of information about who it is they're working with in order to do that effectively. Another third really important use case is marketing optimization. So when you are using different campaigns, either email campaigns or through social media and so on, you always have this exploration, exploitation, trade off that you want. On the one hand, you want to try different messages and you want to try to see what's going to create the best impact, but you don't want to wait too long because if you found something that's going to be useful, you want to start exploiting it. And a lot of the messages, they might be transient. There may be a holiday right now, so you've got two, three days coming up in order to make the most out of it. You can't wait for the experiment to go through. So utilizing things like multi armed bandits or Thompson sampling, those are extremely interesting use cases in order to maximize the effectiveness of your marketing as well.

One of the use cases that Riskified specializes in is specifically in tackling fraud in E commerce. Can you walk us through how fraud works and why this is such an important use case to tackle? And how does riskified solve this problem?

Excellent. So the sad truth today is that a lot of the credit card information is out there on the dark web. Fraudsters are very sophisticated. They actually specialize just like we do. So you have hackers who are going and pulling this information. They go to the dark web, they sell it to other fraudsters who then buy that information and they're going to go ahead and create purchases. Now if you ask the average person, when you buy online and there was fraud, fraudulent activity, who is liable for that? And you might think, well, you know, credit card company obviously, right. Well that's not the case. When you're making a purchase where the credit card was not present, then the liability is on the merchant. So if, you know, you go to the supermarket, you actually show your credit card. Sure, if that was stolen, fraudulent, the credit card company will pay. But if you're doing this online, then the merchant is the one that's liable for that. And this creates a really big problem because if you go back a few years, almost all the systems were legacy based and rules based and merchants were really conservative because when you have this fraud happening, there's a process called a chargeback. And this chargeback, whoever is the victim is calling up their credit card company, they're saying, my information was stolen, I didn't make this transaction. And the merchant pays back the full amount. So they might have relatively small margins, but when they lose, they lose all the transaction and whatever they already sold has already been shipped off or to whoever. So what Riskify does is we use machine learning algorithms based on our network of data across all of the retailers that we work with and we provide a real time decision for merchants. We'll either say they should approve or they should decline a transaction and we'll offer a chargeback guarantee because we have that kind of confidence in our models. So that means that if we got it wrong, we're going to pay for the entire order and we're only going to be charging a fee if we approve. So we are highly incentivized to approve as many orders as possible, otherwise we don't get paid at the end of the day because we have really accurate machine learning models and we have a lot of people working on this problem and this very large network effect. We're able to get more accurate than what a single merchant could do on their own, which means that we're both able to reduce their operating costs around fraud, how much they would be paying for chargebacks versus how much they would be paying if they worked for us. But on the other hand, we're also able to, to make them or to enable them to approve more orders, which means that they can actually increase their revenue, their top line, by a significant amount. If you look at some of the legacy systems, they many cases are going to be super conservative at the point where specific countries could be black labeled. You know, if you want to buy from a country that might have notoriously high fraud, they could just blackmail the entire country and you know, you just can't buy from there. They're also very inflexible and very slow to adapt. So if there are changes like with COVID where some things that historically would have been risky and now they're safe, you've also got the same problem. We saw this in Covid, where all of a sudden you have these last minute flights. People are trying to get out of specific countries where the pandemic is starting to become a huge problem. They know that there's going to be a lockdown and last minute flights, historically super risky, now all of a sudden it's a legitimate pattern. So you need to be able to adapt to those kinds of things. And when you have a rules based system, it just won't work.

That's great. And I'm sure as someone who is, you know, working full time at scaling and improving machine learning based systems for combating fraud, you've developed a plethora of best practices and tricks up your sleeve to create these state of the art predictive models on fraud use cases. While I won't ask you to divulge all these tricks, can you walk us through some of the best practices you learned when designing fraud detection solutions?

So if you're going to be building a fraud detection solution from scratch, and I will warn you, I highly recommend not to go down that path. You know where you start, you don't know when you're going to end. There are a few important things to consider. So one of the most important ones is your ability to link to historic orders. When, let's say I'm, you know, I'm a merchant, I'm a data scientist working in a merchant. And Adele, you're my, you're my customer. And I've seen that you've made these past purchases and they all came back as legitimate. Okay, I've got a good history around things that you've done. I know where you ship, your billing address, maybe IP ranges that you come from and so on. So I can always link to that and understand is there enough history behind this customer and does it make sense with a new order that's coming in, I see all the information is suddenly changing in this new order. But it's your account that's going to trigger some questions around that. So you definitely need a very, very rich history with very accurate labels so you know which orders were good and which orders were bad and have features that are able to understand that information well and pick up on what's important. With that linking further on, when we're talking about some of these features, you need strong domain Expertise around how do fraudsters actually operate? If you look at the raw data points between an order today, can you tell fraud from not fraud? And this is something that takes time. So I'll give you an example. At Riskified, we actually have an internal fraud academy. And before someone is going to be looking at orders, they go through 140 hours of training to understand what fraudulent orders looks like. It's a lot. I've passed through only some of it, and I can tell you there's a ton of information there. Once you have that domain expertise, you can start understanding how would a risk expert actually look at a specific order and what's the thought pattern behind this? Now, if you can take that thought pattern and you can actually create it into very good, very accurate features, then you're going to be able to help your model now, and it's going to have a much better opportunity and possibility in order to recreate that same accuracy next. I'd say you need a lot of data points and it's from all over. You're going to be looking for things and information around the device that it's coming from, information around anything that you can get supplied. You want enrichment from additional data sources. So, for example, you know the ip, you'll want information like, what is the geolocation of this ip? It's not going to be super accurate, but at least you want to understand that it's the right proximity and it makes sense and it's not from a completely different country. You want to look at things like, is there a proxy or not? There's dozens, if not hundreds of data points that you're going to want to look at from anything that might be helpful. And there's three specific data science challenges in the domain that you're going to have to try to overcome. So the first is partial labeling. If you look historically, anytime you approved an order, you're going to get a feedback. You're going to understand whether it was a chargeback and it was fraud or not. Even here, I'll say that that's not even 100% accurate because we know that historically there can be cases where there was fraud, but someone just didn't pick up on it in their credit card information afterwards, so they didn't file a chargeback. So a chargeback is not necessarily fraudulent. There's also the other case as well where someone actually did make an order, but they call their credit card company and say it wasn't me, which we call friendly fraud. And that's another situation. So you've got that part of the problem. In regards to the label, the really big issue is what happens if you declined an order. So if you declined an order, you can't naively just assume that it was fraud. Because if you assume that it was fraud, the next time you retrain your model, what you end up happening is that you're basically going to be in a reinforced state where you're teaching it the same things from last times. So you're never able to correct based off of the model you had earlier. So you have to be very, very suspicious around the declined orders. And were they really fraud or did I get it wrong? Second big problem in the fraud space is that you've got a long feedback loop. Now, this is all relative. If you're working in an insurance company and there's an insurance claim on a, I don't know, you know, house mortgage 10 years later, then you probably have a little bit worse than we do. But if you look at like ad techs, you know, someone clicks on the button basically within a few seconds and you know, if your experiment is working instantaneously, you've got a lot of traffic and so on. If you think about when was the last time you actually looked at your credit card statement? Not everybody does it all the time. Sometimes they don't do it every month. They might check it every two, three times or whatnot. And we see that it can take a couple months, three months, even longer. The credit card company will actually in many cases allow you to file a chargeback even up to, I think it's nine or 12 months after the fact. So the maturation time is long. If you've got a problem in your model and you're not able to pick up on it quickly, by the time you're going to start getting back those chargebacks, it can be too late. Basically. The third problem, which is maybe a little more standard and I think a lot of data scientists are going to be able to relate to this, is the imbalanced balance that you have here. So obviously, you know, luckily fraud is not as common as legitimate orders. It's much more rare. So you're going to have to work around that and be able to either up sample or down sample or think about additional tricks into how you're going to be able to do this so that your training set is as effective as possible. And just going back to what I opened up with, if you're thinking about doing this, you know, it's a Pandora's box. This is something that you can probably do an okay job if you're working on it for a few weeks or a few months. But if you want to get really, really good, and we're talking about scales of merchants that are selling in the billions of dollars or more every year, where every 10 basis points of approvals is going to be significant in their top line, you want to do a really, really good job. And that is going to take really not even many years. It's man decades or more.

This is fascinating specifically when you talk about the machine learning challenges in the fraud space. In a similar vein, what do you think are the main challenges when designing and deploying these AI systems at scale and how do you go about allevia?

So I think there's some challenges that you're going to have to overcome. Whenever you're working at a large scale and you've got a big organization of a lot of people who are trying to tackle a similar problem, you need to start standardizing everything. So that means that all of the tests are going to be standard. Anytime you're doing a comparison, you're doing a champion challenger between whatever you have right now working in production versus your alternative model. Everyone has to be using the exact same tests. It has to be agreed upon and then you can actually compare things and know pretty well what's going on. You've got to decouple the work between different data scientists so they can work independently. If you think about, you know, I love to do Kaggle projects, you take on a competition, you've got your data set and you can do anything you want. But if you took a Kaggle competition and you let 20 people work on it, it's not going to be really effective. If you let each of the 20 people just say, you know, have at it and let's see what happens. Okay, okay. So, you know, one of them might be doing a little bit better than somebody else, but the other 19 is basically waste work. So you've got to think about ways where you're going to be able to split the work effectively. Sometimes we do this by domain. We can say for specific teams, okay, you're going to take our gradient boosting algorithm and you're going to try to optimize the algorithm. In other cases, we're going to say, okay, these teams are going to be focused on feature engineering for specific families of features, and if the families or features are orthogonal enough, then they're going to be able to make their own independent contributions and incremental contributions without stepping on someone else's toes. Now if you go basically back to the theory, someone might say, wait, you can't really do anything that's going to be suboptimal in specific places because everything is related. And you change the algorithm, maybe you have to optimize and change your feature engineering and your sampling methodology. It's true, it's true. But this is the constraint we live in. So at the end of the day, you have to try to find how to do that in some manner. So we try to create our teams in ways that they each have areas of affordable, of responsibility and authority, and they can work there as autonomously as they want. They can create their own roadmaps, but they know that these are the guardrails, this is the area that they're going to be trying to optimize. And then the last challenge, which I think a lot of people are also going to be able to connect to, is this is a mission critical real time system. So think about it. As a merchant, you've gone through so much pain to send out your marketing campaigns to get these shoppers on. They've gone through the cart, they've clicked, you know, everything, they're clicking on the checkout button. And now that data is going to riskified. There's no way you want it not to work. Okay? It has to be up all the time, no matter what. This is super robust system and has to work in very, very fast timescales as well. So that means we got to collaborate very, very closely with engineering. In some cases that means there, you know, we might have a really cool model that performs very, very well, but it's going to be hard from a technical perspective to put it into production today. Sometimes we can definitely work with product and with our engineering and we'll see that the value is big enough that we are going to have to change part of our tech stack or find different ways in which we can implement it. In other cases, you might show value, but the value is just not going to necessarily be enough in order to want to implement that. And that is something that people need to comprehend. It's not just that I've been able to improve the model, it's that the improvement is big enough to justify additional tech debt or the incremental investment that we're going to be doing on the tech side.

I want to zero in on one challenge you talked about here. So you mentioned the need to work really closely with engineering, especially in order to operationalize and deploy some of these data science models into mission critical systems. There's also A lot of different schools of thought in data science today where one argues data scientists should be specialists and work purely on data related problems and others arguing that data scientists should learn engineering skills and should learn how to deploy and maintain models in production as well. Do you think data scientists should learn these engineering skills and do you think that they will make the difference between a successful and not a successful data scientist in the future?

So that's such a great question. I think that it really depends on the company you're working with and the ecosystem that you have. So it's going to be really hard for me to say there's a one size fits all. If you work in a culture where, you know, the engineering has created magnificent infrastructure for data scientists to focus on data science and they can go ahead and implement things in production and they might not necessarily need very, very heavy engineering skills. That's one way to go at it and that can be great. I personally believe, and this is probably also a lot to do with where we are at riskify today, that data scientists do need some engineering skills. We have to be able to talk in the same language, we have to be able to work with similar tools. So we don't want to get to a situation where we're basically saying, okay, this is a object that represents my model, go ahead and take it from here. Or I don't want to get to a point where I am engineering a feature and I write pseudo code and here's the logic behind it, go ahead and implement it some way and basically throwing it over the wall. We do want to have some engineering skills. Where I am more troubled and concerned about is scenarios where engineering is a practice on its own and data science is a practice on its own. And if companies are at the point where, and you can, it's a slippery slope, you can very easily get there. You do some of the research and now you're working on implementing it in production and now you maintain it and you own it. And the maintenance is starting to take up so much of your time. You can get to a point where the majority of the data scientist time is only around how do I implement what I do in production. And at the end of the day, data scientists are trying to do, or you know, should be doing research. They should be looking at the data, they should be improving the models. That is what is really their specialty and what's different from a ML engineer or data engineer. So in my mind they should have these skills, they should be able to do it. But I think it needs to be a situation where there are other people who can take on more of the burden and you're able to create that separation more effectively. Now, again, that's very much specific to us. If we were in a much, much smaller situation with maybe one or two data scientists working in a smaller startup, then everybody's going to have to do everything. And yeah, you're definitely going to need engineering skills. If you ask me, long term, for people listening in right now, if you don't have engineering skills, it's really going to limit where you're going to be able to work. So you need to have the fundamentals. But on the other hand, if you have those fundamentals, then there's always areas within data science where you can continue to improve your craft.

Now, speaking of challenges, data science and machine learning are at the heart of Riskified's business model. And I'm sure with that comes many challenges. So I'd like to segue more into the challenges challenges you face when leading data projects and data teams. I'm always excited to discuss these since they are use case or industry agnostic challenges. And any data team can learn from your challenges and your best practices. One thing that I've seen you write about is the common pitfalls data teams face with any data science project. Do you mind walking us through these pitfalls and how you go about solving them?

Sure. So I've worked in a few different places in the past and I have to say actually at risk if I were lucky enough, where the main product is machine learning models. That's really the core of the, of what we're doing. So we're in a relatively good situation in regards to these pitfalls. But some of the things that I've seen first is bad data. If you are starting up a data science practice right now or you're going to work on data science in a field that may not have been using machine learning before, the quality of data can be horrendous and in many cases there might be an executive stakeholder who's going to try to bring in data scientists to work on this. You're going to have so much time just cleaning up the data that you're not really going to be able to show business value for a long time. If your stakeholders understand that that can be okay, in other cases they may expect instantaneous magic. And you know, why are you still cleaning this data after six months? And that can definitely create a lot of problems. So if you're going to be starting, I would highly recommend that someone who's experienced in the field actually start digging into the data and doing exploration and say, is there something to work with right now? Or alternatively, you can create a backlog of these are the things that we need to do in order to clean the data. And only then should you, for example, hire someone full time or bring someone into this project. But we've got a backlog before we can do that. One of the pitfalls I've seen is lack of buy in and trust. And again, this can happen many times where you have someone at the top, someone who says, okay, I want to bring in data science here, I want to change it. And then you might have a middle manager or someone you're working with who just basically received this from the top. They don't necessarily have the buy in. In many cases when they work with data scientists, they can see this as a threat. And I can completely relate to that. So if you're thinking about someone who's a domain expert in some field, let's say you're working in operations or fulfillment or marketing and you're an analyst and maybe you're doing forecasting models in Excel and you're creating whatever, it's not working necessarily well enough in its scale. And all of a sudden you have this data scientist come starting asking you questions about how you go about your business and the data you do. In many cases, the immediate response can be, we don't have a problem, or why are we even working on this? Or creating criteria that are ridiculously hard for any data scientist to meet them. And you know, they want you to come up with a solution that is way better than anything that they can do right now. And they're the domain experts and they're doing this manually. So they'll usually have an advantage, at least on any MVP model you can create here. You have to work really, really closely with those stakeholders. You have to make them understand that, you know, you're not coming here to replace them. And usually that's the case. I mean, in many cases what data scientists are going to do is enable other people to do better at their job. You're not, you know, maybe you're going to take some of the more manual work and make that better and now they can use those models to do something better. There's other cases where there's stakeholders who are engaged, they're interested, but they might not necessarily know how they're going to be using machine learning. So it resembles a little bit of what I said in the first one, where they'll bring Somebody in and basically they'll just want them to bring the magic as soon as possible. And here again, you really need to work with the stakeholders to formalize what's the problem that they're interested in what is going to make give them the business value they want. In a lot of cases I've been, the stakeholders may just want you to start exploring, start looking at the data, try to create something and it can be problematic. You start looking in that data, you'll start bringing some kind of nuggets, some insights. You might have something that's going to be helpful and it just goes on and on. There's an education process where you have to work with your stakeholders and tell them, you know, guys, let's work on these KPIs. Let's understand what is going to be important for you. What do you want to see from the model. So I understand you're engaged, but you need to know how to utilize our time. Then finally, one of the fourth pitfall I've seen is that stakeholders change their minds. So, you know, business is very dynamic. You might be working with one stakeholder today, they may change. It's going to be challenging to make sure that whenever you're working with the stakeholders, they understand the value of what you're doing. They understand the research process as well. The buy in is longer term and they're committed to this process. It's not a situation where you know, especially if you're coming into something new, a new field, new data, data types and whatnot, you're going to start giving them immediate value. It'll take time. And I think if we try to look at all these different pitfalls, one of the most important things I would tell any data scientist anywhere is there is no such thing as over communication. If you think that you are meeting with your stakeholders, product, whoever it may be too much. Basically, I would guarantee that if you talk to them and ask them questions that you probably think that for sure they know. Not so sure. Talk to your product if they really understand some of the fundamentals behind the KPIs or the research that you're doing or what kind of risk this entails and whether it's likely to succeed or not, how you see things. There's a lot of these fundamental questions here. And time and time again I see cases where data scientists naturally like working with the data. They may not see the value behind keeping the stakeholders up to date all the time and discussing things, especially when they don't see these big milestones. But you can't wait for the big milestone. You have to continue having those conversations.

That's great. And I want to zero in on the human side of this. So you mentioned one of the common pitfalls being a lack of stakeholder buy in or lack of trust. And I'd like to zero in on that. There's a lot more discussion nowadays around data storytelling, the importance of presenting insight and how to maximize value. Value when bridging the gap between business stakeholders and data scientists and analysts. Do you have any best practices you can share there about how to maximize the value there and to bridge that gap?

Yeah. So there's a few things that you need to keep into consideration. I'd say one of the most important ones is to try to have a good, honest reading of the level of engagement that you have with your stakeholders. So you want to look for things like are they asking you questions? Are they actually interested in what you're doing? Are they really interested in this? Or are they coming up to sync meetings just because their boss is there and they have to be there. I've seen situations where, like one of the pitfalls we mentioned earlier, the senior management decided to go on with a project. Some of the people who are actually the domain experts may have felt threatened, they may not have felt that it's as necessary as possible. They arrive to the meetings, they don't necessarily ask many questions. They are, you know, you could say a bit passive aggressive. And the data scientist is almost trying to prove to them that the model is better than what they're doing. But at the end, you know, it's not necessarily a rational conversation. People are really good at deflecting even when they see proof. And it can be hard. It's a cultural issue. So you want to be sure that they're engaged as soon as possible and work with them. Another really important thing is to ask a lot of what ifs. So before you go ahead and create the model and present the final results way, way, way back before, even before you started, you know, working extensively into this, you should start asking them, let's say, imagine, hypothetically, tomorrow morning I had the model ready and it reached 90% precision, 50% recall, whatever metrics you want, it worked within 100 millisecond latency. Would you use it? How would you use it? Is it good enough? And then you start seeing the thought process. If they start asking lots of additional questions or they start giving you more excuses around this, well, the worst thing you want to hear is, let's see, create the model and then we'll talk about it. You want to avoid that at all costs. You basically want to try to get to a commitment saying, these are the KPIs, this is what I need to try to reach. But if I can do this and I work within all of the constraints that you've given me for the problem, you're actually going to be using it. Because if you're not committed to using this, then really I need to question whether I want to put in the work to try to research it and keep doing this. It's not just a one off. As you go along, you and your stakeholder may understand that, okay, there's more constraints than you originally understood. Maybe it's not just the latency, maybe it's important. What's the memory footprint of the model, what technology it's going to be running on or whatnot. So keep asking these questions and you'll get a much better understanding of what do you need to do in order to actually make this be valuable for the business. Finally, you want to make the stakeholder and the expert part of this process. So we talked earlier about the fact that you have to update them and keep them in the loop. You want to ask them for the feedback as well. When you're working with a domain expert who's not a data scientist, they have a lot of body of knowledge. You don't want to get to a scenario where you're setting up a one hour meeting, you're trying to debrief them from everything they know and then basically say, okay, thank you very much, fellow. From here, this is now a machine learning problem and I'm going to run ahead. You want them to feel part of the process and be involved and get their feedback. They're going to be super useful. Every time you see quirks in the data and you're probably going to be seeing, you know, all these anomalies and little problems in the data and you want to ask them, is this real? Is this an outlier? Should we be removing this? The more they're bought in and you know, the more they feel that they were part of this process and it's their model too, they were really contributing. Then they're going to be wanting to use this model and it's not going to feel like a competition.

This is wonderful. And you mentioned here in your response around the element of culture, as well as relaying back to the other human element of this, which is stakeholders who are engaged but not necessarily know how to leverage these machine learning models and their day to Day tasks. So how important do you view the role of organization wide data skills and data literacy when creating that buy in and that culture Culture and where business stakeholders feel like they own the data solutions that are built for them.

So data literacy is super critical. It is extremely challenging and personally can be very, very frustrating when you're trying to have a conversation with stakeholders who aren't literate at data. In many companies that is the situation and you have to understand it. And it's very important that you do have a good understanding of which stakeholders are more data literate and which are not. And I would go so far as to say that if you're data scientist and you're going to be working with stakeholders who aren't data literate, the first thing you should do, and you should, you know, for your own success, take this as your personal responsibility is to educate your stakeholders, make them data literate. It doesn't have to be extreme, but they really need to understand what are the meaning behind the KPIs, how is this going to work, where does it make more sense? And you know, in some cases they need to understand the algorithm to an extent where, how explainable is it going to be? Is this going to be a black box or are you going to be able to actually have knobs where you can tweak it on your own or change it once you have that data literacy, like I mentioned before, you want to have basically an informal contract if you're able to reach X precision, Y recall, whatever latency or your RMC is going to be underneath some threshold now you're the stakeholder is going to go ahead and integrate this. And it can't be vague, it has to be very, very specific. You also should work with a stakeholder to understand what's the baseline right now. So I've seen this as well, where in many cases you're coming in and you're trying to develop a solution that might have been done manually before and the stakeholder might set this very, very extravagant threshold and you ask them, okay, so how are we doing right now? And they actually have no idea. So is this like a hundred percent improvement or a 10x improvement or 100x improvement? Like I don't know what this threshold really is. So it's also very, very important to try to work with them and understand what's the baseline that I'm competing against. So I understand that this actually makes sense.

Another important aspect of leading data science at riskify that I've seen you write about and speak about is balancing between longer term research goals and short term product. And this is something that I'm pretty sure many data science teams also have to balance. Can you walk us through that balancing act and how Riskify chooses to prioritize its research goals?

So this is one of those cases where it's going to depend on the scale of the organization, the organization you're working with. So at Riskified, we're pretty big. We've got about 40 data scientists right now working many of them on chargeback, guarantee and a lot of other different products. And when you get to that kind of scale, you start thinking about this more statistically. It's almost like a portfolio you can imagine you've got, okay, I'm going to have 20 different research projects. How do I want to allocate them? Very, very different than if you've got one or two. And you know, it really is going to depend if you are successful in this one. You don't want to take a lot of long shots that aren't going to pan out here. You want to have a trade off between some of the long term bigger bets. Typically those might have the potential huge payout. So your expected value might be higher, but you've also got a very, very large standard deviation. If you can run a few of them, your expected value, as I mentioned earlier, may be higher. So it makes sense, but you don't want to overdo it. You definitely don't want a situation where you know, you're trying out a few of these, maybe none of them work out, and all of a sudden you didn't bring enough value for a few months. You also got lots of shorter term projects that the business is more interested in and you always want to be delivering value consistently. So we definitely want to continue working on shorter term stuff and there's always going to be more lower hanging fruit. You also don't want low hanging fruit as a strategy. If you continue doing that, then over time you're basically going to be drying up the well, you're not going to have the next big thing that you can build upon. And once you typically do that, then you can continue optimizing it more and more and it's going to be generating more lower hanging fruit. And we try to think about what is going to be the portfolio that will bring us to the best place maybe six 12 months down the road. So that means that we're mostly going to be doing some short term projects or a lot of short term projects that are going to be improving the current features that we have and the models, and we carefully choose a few handful, handful of moonshots that could take on several months, hoping that one or two of them are going to be successful. And this is a very delicate balancing act that you always want to do closely with your stakeholders. And we work very closely with our product so they understand how are things looking, they give us a lot of input and insight and try to get to something that we feel is going to give us the best possible outcome. I will say that try as you might, when you're trying to decide on a research project before you've initiated it, there aren't a lot of really good ways in order to understand it. The best two heuristics that I can come up with is one is the Delphi method, which is basically where you try to take experts who have good intuition. This can be your senior data scientists, maybe some of the team leads, or anyone who's already done this before. And you want to ask each of them, how valuable do you think this direction could be? You let them do it independently. You compare the results, you talk out the differences, if the differences are very, very big, and you have another couple of rounds like that, and at the end you're going to have a more or less group consensus using all those experts to try and understand, okay, the way we understand this, this is going to be hugely impactful, or not so much. You can do that process and try to get a better estimation using more people. Again, built onto intuition. But this intuition is based off of a lot of expertise and experience that they've built over time. And then the second thing that you can do instead of just using expertise is looking at historic returns from specific areas. So I mentioned earlier, we can break down and we decouple where we're doing different research projects into families of features or different areas within the algorithm and how we're tackling it. If I see that over the last year or so, and obviously not one project, but I look at a lot of projects, they have it in one domain and I see that they gave fantastic returns relative to what I did in other domains, it's definitely not going to be a guaranteed win for the next project, but there's a higher likelihood that there's still more potential to be utilized there. So it will help us and say, okay, probably a little more promising and we're going to be putting more resources there, but definitely a very challenging area to try to work around. And prioritizing research goals is one of the more challenging things that we face.

Thank you so much for this insight, Elad. Finally, Elad, before we let you go, do you have any call to action for listeners before we wrap up?

So if you're listening to this, you're familiar with datacamp, and I've got to say, you really need to never stop learning. AutoML is real. It may not be as big as you think, but there's a famous quote where people tend to over predict what will be coming in the next year and they under predict how things are going to change 10 years down the road. So auto may not change everything, you know, in the next year or two, but five, ten years from now, a lot of the problems that data scientists work on are likely going to be automated. And you have to continue to improve your skill set and become an expert in what you do so that you're always going to be ahead of the curve so that the value that you can bring is more than, you know, fit predict. Secondly, I'd say focus on bringing the most value to the business. At the end of the day, that's what you're getting paid for. You know, if you aren't able to bring value, it happens sometimes. Maybe you were on a project that didn't work out and you know, it's really important to decouple good research from, you know, the outcome. Sometimes great research didn't work, but you still need to try to focus on that because that is what people will be looking at over time to see what kind of contributions are you able to bring. So while working on the most latest technologies and last generation algorithms, state of the art can be really exciting and can teach you a lot. Always think about the value. And then I'd say the last one is if you can, especially if you're in a management position and this is possible, you have to choose your projects well, wisely. Data scientists that I'm familiar with, especially the really good ones, they will just love to dive into a problem and they're going to start exploring the data and they'll create these models and continue to prove them and it becomes addictive, you know, can I get better features? Can I change the model and the hyperparameters and improve it over time? Like I mentioned earlier, if you're going to go into a new domain and it's a really big problem, you should carefully consider whether you want to buy instead of trying to build something in house. If you're going to be creating something where, you know, maybe you can get a good enough solution, that's fine. If the bar that's being set is very, very high. There's situations where you have to make the sometimes, you know, more difficult considerations. Say, okay, this is super interesting. I love this project. I think I could do a good job here. But I'm going to think about my time as a really precious resource because it is there's not enough data scientists and I'm going to go ahead and look for a buy solution. I can tell you for ourselves, we have a lot of data scientists, but there are data science problems where we're also looking outwards. Things like monitoring some of our models. We look for areas where these are components that aren't specific to the domain. And if we can get an outside solution, that's going to save a lot of our time and we're going to do that. So you have to pick your projects well when you're able to do that and think of what are the core competencies within your company where you're going to be able to bring the best possible improvement and not try to solve every possible problem that could be solved with data science because there's just so many of them.

Indeed. Thank you so much for the time today, Elad. I really appreciate your insights.

Excellent. Thank you very much. And thank you for all the listeners who joined us today.

That's it for today's episode of Data Framed. Thanks for being with us. I really enjoyed Elad's insights on the state of data science and E commerce and his best practices leading data teams. If you enjoyed this podcast, make sure to leave a review on itunes. Our next episode will be with Sudaman Thopan Mohanchandral, Regional Chief Data Analytics Officer at Allianz Benelux, on the importance of building data cultures. I hope it'll be useful for you and I hope to catch you next time on Data Framed.