#76 Providing Financial Inclusion with Data Science, with Vishnu V Ram, VP of Data Science and Engineering at Credit Karma

Hi, I'm Maggie, VP of Curriculum at datacamp, and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey on datacamp.com.

Hello, this is Adele Neme from datacamp and welcome to Data Framed, a podcast covering all things data and its impact on organizations across the world. One of the most exciting aspects of data science and technology in general is how it reduces the barrier to previously difficult to acquire information and services, whether Google Maps providing navigation, the ability to immediately watch a tutorial on YouTube, or ordering a taxi via Uber. In that light, the spaces where data science can have the most impact in lowering the barrier to information and services are high stakes industry like finance and healthcare. This is why I'm excited to speak to Vishnu Ram, Vice President of Data Science and Engineering at Credit Karma. Credit Karma is a fintech startup founded in 2007 with the aim of providing financial inclusion to individuals by allowing them to see their credit scores for free. Free. Ever since, it has been producing a suite of products that leverage data science and that provide users more certainty around their financial future. Throughout the episode, Vishnu talks about his background, Credit Karma's mission, the inner workings of the data science products powering Credit Karma, how he led his data team through a phase of growth from 20 million to 120 million users, building data cultures, the skills data teams need to have, and and more. Now, speaking of the skills data teams need to have, we're also happy to announce the launch of a 14 day free trial for Datacamp Professional. It's designed for teams of any size with access to all of datacamp's courses. Make sure to check it out and invite your colleagues. Vishnu, it's great to have you on the show.

Thanks a lot for having me here.

At L. I'm excited to discuss with you your experience managing data science and engineering at Credit Karma, the best practices you've developed, leading data teams and operationalizing data science at scale. But before we begin, can you give us a brief background about yourself and how you got into the data space?

So the way I got into data was like, I've heard a lot of your Podcasts and I know a lot of people have also talked about this. It's a little bit like roundabout Getting into data science in my undergrad I did a couple of projects which were related to using fuzzy logic for control of dynamics of systems. And that's how I kind of like got into AI. After my undergrad I had an opportunity to actually do more in neural networks, but I ended up choosing a different path. If I had chosen neural networks, I might have come into data science and come into the data world much earlier is the way I think about it. But my path ended up being more of a early stage startup engineer, build things from scratch and then moved into taking on a few early stage startup CTO roles. Ended up wearing multiple hats when you're doing early stage startups, as you can understand, wore a bunch of different hats. And I think in the beginning stages in these startups it was more about leveraging data for analytics, leveraging data for make or break business decisions, and then over a period of time started doing more because these are more consumer tech startups, doing more of data pipelines for user behavior understanding and making big product decisions. And then along the way got into E commerce, product recommendations along the way in one of the startups, and then finally in Credit Karma. I feel like it's come full circle where I didn't join the company when it was in its early stage, but I feel like I've been involved in data across all of these aspects. Analytics, data science building, recommendation systems, user behavior, understanding, machine learning, all of the above.

That's awesome. And for those who are not aware as well, can you give a brief background about Credit Karma and how it works?

Yeah. I have to go back to say 2007 when our founders started the company. I think in 2007 when our founders started the company, it was primarily about making it free and open for everyone. Credit scores from TransUnion, that's how it started out. Then around 2015 or so, kind of the time around I joined. So I'm going to bring that up as a big landmark year. It was a landmark year because we added Equifax as well as TransUnion. And then if you look at today where we are, we have done like 4 billion credit score and credit reports across US, UK and Canada. So the journey for Credit Karma has been it started out as a credit score provider for our members and then it started helping our members understand their credit reports really well. But along the way we have really added some really important features to our product like identity monitoring, and today, our members use Credit Karma as a platform for shopping all their financial products, whether it is credit cards or personal loans or auto insurance or auto loans or mortgages. And over the last couple of years, we've also added free banking products like checking and saving accounts, which allows our members to really leverage their Credit Karma for everything that they need to do as far as their personal finances are concerned.

That's great. If I'm not mistaken, you have about 120 million customers, right?

That's right. And between 2014 and 2019, we added around 70 million users. And it's just amazing to see the growth that we have had. Part of the growth has just been really, really amazing.

For me, that must have been very exciting and really sets the stage for today's conversation. What's really exciting about Credit Karma's mission is really the use of data to democratize financial information and to equip everyone with the ability to make healthy financial decisions. So I want to set the stage for today's conversation by really trying to understand how central data is for Credit Karma's success and operations. Do you mind giving us an overview of different ways data science is used at Credit Karma and some of the use cases you've worked on personally?

So probably take something that is like, more relevant to all of us. Let's say you open up Google Maps and you're trying to see get to somewhere, the first thing that you actually will probably need to enter into the system if you do not have your GPS tracking turned on is where are you right now? What's your starting address? So for a lot of our members, the starting point really is where are they today in the eyes of all the financial institutions that they go to to get their financial products from? And not just financial institutions. If you want to get a home to rent, if you want to get the new Apple iPhone from Verizon or T Mobile or whichever is your provider, you need to know where you stand. I think that's been the starting point for all our Credit Karma members as well, to understand where they stand with respect to their credit history, credit report, and credit score that I would say the single data point of what is my credit score kind of is the genesis of the entire company. So I would say that's just such a powerful data point. When you put it in front of an individual, when they have a strong sense of, then there is no more thinking about it, right? In terms of like, hey, do I have a good credit score? Do I have a bad credit score? Can I afford to get the latest iPhone? Can I afford to go and upgrade my rental? Can I afford to go get a new home loan? What home loan rate would I get? You are no longer at the mercy of any of these people. You have a good sense of what you're allowed to do at that point of time, right? So I think that is like kind of like the starting point for all of our members. And then along the way you get into using data for more automation, using data for more advanced use cases like leveraging machine learning to understand users better, understand the products better, understand the user product interactions better. And then you put it all together. So to come down to actual applications of data and machine learning in Credit Karma. So the way we think about things, the way things are split up is we have something called certainty. When a user is applying for a financial product, a lot of times they are putting themselves in a vulnerable position of potentially getting declined for that product. So there is that uncertainty in their mind whether they are going to get approved for it or not. So one of the biggest, biggest use cases that we have is providing certainty to our members. What are the chances that you're going to get approved for this product? Once you're able to provide that certainty to our members, it just provides them with like complete relief. It's the kind of certainty that you know when you're leaving home and you want to go to your friend's place or you want to go to a new location, you just type in the destination address in your Google Maps or Apple Maps and you're going to get the information back. This is the route. Follow this route, you're going to reach there in 25 minutes. And more often than not, it doesn't matter whether you switch lanes along the way, whether you're going at 70 miles per hour or 75 miles per hour on the highway, you get that certainty in terms of when you're going to reach and that you're going to reach there. So that is a big application of data that we have within Credit Karma. And then certainty is great. But I kind of want to have control over what financial products I want to get. I might want to get financial product from ABC versus xyz. What is my own control over that? What is my own intent? Do I want to get a home loan which is a 15 year or do I want to pay my home loan or a 15 year time frame or do I want to pay a home loan over a 30 year time frame? Those are things that I want control over. And it's Something that me as a user, I want to understand, I want to be able to do that. Well, what is the data that the system has and the company has to help order things or rank things in a way where you are taking into account what the user wants and what they really want to get out of Credit Karma when they're using Credit Karma, that is where our ranking application comes from. And then the third thing is like we all have needs where we want a credit card or a personal loan or auto or a home loan at various points of time. But the various points of time is the key phrase there. The timing of when you are getting something and when you're not looking to get something else is where our propensity models come into place. Then when you put all of these things together is when you have an opportunity to provide really strong, relevant recommendations to our members. When we provide those relevant recommendations to our members is when you are able to take all the things that the user has provided you in terms of information and data and be able to help them get what they want out of out of Credit Karma and out of all the other partners that Credit Karma works with. So that I would say is like bread and butter in terms of what we use data for. But apart from that, we also leverage data for doing important things like anomaly detection in our systems, anomaly detection in our business metrics. And then we also want to understand how well are we doing, how well are we going to do? Ask an answer like what if questions, which is where things like forecasting come in place. But I would say my bread and butter and a lot of the bread and butter for Credit Karma as far as data has been in the recommendation.

Space, that's absolutely fantastic in the way you frame it. Especially when you mentioned the ability to provide relief for people. I don't think a lot of people consider the psychological stress especially underprivileged folks experience when they try to interact with a bank or a financial institution. So the ability to provide a financial compass is so important. Obviously, recommendation engines are at the heart of Credit Karma's business model and you're one of the key people building this. So in a nutshell, please correct me if I'm wrong. You provide distilled insights from people's credit data. They get provided recommendations of financial products from different banks and lenders based on their credit insights. And if they take that recommendation, Credit Karma gets paid by the lender, so the customer always gets free insights. And the inclusion that you discussed, do you mind going into detail over the first iteration of the recommendation engine and how has it evolved ever since?

Yeah, I think you got it almost right there, Adil. So I'm just going to add some fine nuances here. So when I joined the company in 2014 and before, there were a few things that was already set and I don't think we've really changed any of that even now. One thing that was said was more of a win, win win business model where we wanted to make sure that anytime Credit Karma gets any incentives, it gets incentives in a way, when the user has benefited from the interaction with Credit Karma. And then on the other side, a bank or some other financial institution has also got benefit from the interaction of the member with Credit Karma. So we have this construct of win, win win for our members, for us as well as for our partners. We think that's how the ecosystem is really well, self sustaining and also sets all of us up on a growth path. Our members, us as well as our partners. But at the same time, the ways we have gone about it, the ways we have approached it has changed over a period of time. When I started, I would say there were rules everywhere. There was limited data collection in terms of what the user wants and there was a lot of rule based decision making. And then I would say the first year or so the job was to centralize the rules in one place. Keep in mind that the company was doing really well. Our members were getting a lot of really strong benefits. The certainty models that I talked about were already in place in its earliest avatar. So there was a lot of things that were working really well at the point of time when I joined the company. But there was an opportunity for us to look at an operation that was serving more than 20 million members at that point of time and to be able to make it like far, far better than what was there. So I would say the first couple of years it was more about centralizing all the rules in one place, looking at models as the centerpiece and have an approach where we were able to replace a lot of these analytic insights which were more point and time based and might also be specific to certain populations, and then replace them with models that could be evolving and learning. Then to be able to support that, we had to make really important investments in our data collection infrastructure. And also we made a big bet of moving on to BigQuery. At a point in time very, very early on when GCP was evolving and GCP was very early stage, we made a big bet of getting into BigQuery. I still remember a time when we were sitting in our CTO's room, where we were thinking about BigQuery was like other products in the market and thinking about like, hey, why we want to go into BigQuery? I think BigQuery was a huge, huge win for us when we made that investment. What it allowed our data scientists was certainty, actually in a way where they knew that when they come in on Monday morning, the data set that they need for them to operate on to be able to start training their models, as well as the data that they need to look at to understand if the models that are currently running in production are working well or not, that it's there, they can access it and they don't need to worry about being able to access it. I think four years or so back, around 2017 or so, our model started to take over. We realized that we needed to make a major improvement in our experimentation system, which was kind of like little archaic and was making a lot of weird decisions. So we wanted to make sure that we were able to build out a robust experimentation service. And that's something that we made an investment in 2016. And that's the year we also started building out our early ML Infra team, based out in AWS initially, which we later ported into Google Cloud. I think at that point in time we were also looking at whether we want to be like a all Scala shop because a lot of the engineering, a lot of the systems were all based in Scala and we were thinking, hey, there's like more engineers than data scientists maybe, and data scientists, like write fewer lines of code. Maybe we can get our data scientists to learn Scala because some of them are coming from R, some of them coming from some other ecosystem. So we said like, hey, why don't we just like figure out how if we can get them to do Scala? That didn't last long. And I would say in 2018 is when we moved more towards Scala for engineering, Python for data science and allow our data scientists to really bring in everything that they want and also be part of the community. The entire community, data science community is based on Python. So allow them to also interact with the community, learn from the community and then bring it into credit grammar in an easy fashion while allowing our engineering systems to operate at scale on a stack like Scala. This is also the time where we had been following TensorFlow since it's like 0.1 days. And we definitely thought that it was going to be something that we could leverage in some point of time in future. And 2018 is when we said, okay, let's start moving towards TensorFlow. And either in 2018 or so is when Google also published the TensorFlow Extended Paper. And then when we looked at the extended paper, we knew that these are exactly the way we were thinking about our own machine learning infrastructure. So it kind of like made a lot of sense for us to go big on TensorFlow, continue to go big on GCP, especially given our early success with BigQuery. And today I would say we do 35 billion model predictions a day. We collect all the data about what the user is seeing. The recommendation system drives everything that the user sees when they get a notification. It just has a hand in every interaction with the user. Go back and reiterate the point that throughout all of these evolution the constant has been the win win win business model. Our objective functions have not really changed. Our objective functions have stayed true to what the business really wants us to deliver.

I'd love to unpack some of this. So there were some infrastructure level improvements that were done, data collection improvements that were done, and more. In terms of the data being used to power this recommendation engine, obviously key partners are credit bureaus like EC, Equifax and TransUnion. But do you also use external data to supplement your solution? If so, what type of data are you using to supplement the recommendation engine?

Yeah, I think definitely the bureau data from our bureaus are like kind of like the main driver for everything that we do. But along the way we've also realized that while it's important for us to provide a lot of certainty to our members and also to understand what they want, the application process of applying for a financial product and applying for different financial products is kind of like complex and it requires you to go through a lot of steps and long forms. And a lot of that is also repetitive. There are few data points that we realized were just required in all of these applications. Keep in mind that when someone finds it easy to apply for a product and if they have certainty, then that's just going to kind of like provide that flywheel for our win win win. Right. So with that thought process we have definitely brought in a few data points to help our members. And some of the data point is actually what the member themselves provide us. Just make sure that if you're asked for an income for applying for a particular product, then two weeks later you're going to apply for some other product. Guess what, that same income is used in the next application flow. Also, it's just like make it easy for the user to go through the process, then there are other things that might seem trivial from a data perspective, but from a member ease of use perspective, it just makes it so much better for us. And we all take this for granted nowadays, but you still need to do the work to make it happen. It's just like when I'm entering my address, am I getting my address right? Have I screwed up something thing? And a lot of this, if you are just like putting this in Doordash or putting it on Uber, if you miss it, you miss a ride or you miss a food delivery. But if you do it in a financial product, you have the opportunity to waste a lot of time. You might be applying for a home loan, you might have put a bid, you want to get the home loan, you don't want to get this wrong, you want to get it right. So to help our members get things right almost all the time, we kind of like invest in some of these areas where it allows us to just like correct the address that the users are entering and that way they don't get it wrong and it allows us to just like provide that ease of use for our members. And with a lot of our vendors as well adopting cloud and us being in the cloud as well, a lot of these data integrations have just become cheaper and easier. So it's kind of a no brainer to do some of these things to help our members get like a much better ease of use as far as their experience is concerned.

That's great. And you've mentioned so far the win win win model and how you want to optimize for this. Now there's a lot of angles by which we can approach this discussion, but given how central empowering users is for Credit Karma's mission, how do you ensure you're delivering recommendations and products that align with the key principles of the company, which is ultimately providing users with the ability to make better, healthier financial decisions.

Yeah, I think that's a really hard problem. Right. So great, great question. So it's a really, really hard problem of being able to just consistently do this on an ongoing basis. And anytime you need to consistently do something that's really hard, you kind of have to start with what you do with your culture. And I think as part of our culture as an organization, as teams, as individuals within the organization, there is a lot of collaboration that we have. There is like by definition, while we are doing things in data, we are collaborating really strongly with business to understand what the business needs are, to understand what the business constraints are better. We also Collaborate very heavily with legal and marketing to understand what are the regulations that we need to be able to follow properly and make sure that we are understanding our user needs better and understand like what's working well and what's not working well throughout all of this. It's also just having the right empathy for our end users to really understand who our end users are and what they get out of using the product and what feedback do they have to help us that we can leverage to get better done. A lot of different things here, but I would say probably as far as culture is concerned and what's really worked for us is just like staying humble and curious throughout the entire process when you're talking to any of your stakeholders, because especially when you're leveraging data to make decisions for 100 million people, you're working at very, very large scale. So it's really important to stay humble in this process and curious in this process and be able to get into a lot of these conversations where someone might ask you the question, like, hey, why am I seeing this? Why are you sending this push notification to me? Why are you sending this email to me? And the way they may ask it, it might hurt you a little bit. You might think that, oh, like, of course I'm doing this for 100 million users. You can expect me to get a few users here and there wrong. But if you stay humble and curious, you want to get something out of that conversation. So just, just part of how you're able to inculcate a culture of just staying humble and curious and listen with empathy, that goes a long way. And then to be able to support it, you really want to have the right processes and you keep setting up the right processes along the way. Things like set up everything as an experiment, measure everything, make sure that your launches and ramps are well designed. And you know that you have to look at these metrics before launch and even after launch, you are looking at the right metrics to make sure that there are times when we have said like, hey, if it's revenue neutral, we're going to launch. But then if the revenue ends up being negative, then still being thoughtful about like, hey, find the revenue as negative. But there are other important metrics as far as the users are concerned, it became better, so let's just go launch it. And then as far as again can talk about all of this. But there is change all around us, especially in Covid times. We have seen all of this change come through and hit us in a big way. You have to build Systems in a way where you're protecting the downside well so that you're able to support innovation by the team all the time, to be able to keep up with the changes, to keep working with the changes, to get more value out of the changes. I would say to deal with the hard problem like that, you need to have like strong investments in culture, process and systems, is how I would put it.

And how do you make sure you're able to understand what the downside is? Do you employ any explainability techniques on your models currently, or do you do any retroactive analysis on how users have been impacted by a particular model, for example?

Great question there, Adil. So I think starting 2017, when we kind of like ended up rebuilding our recommendation systems from scratch, we were very clear that we had embarked on a process of building out a very complex system. And the system has models, the system has rules, the system has other business constraints that might be applied outside of the recommended system. We might be running hundreds of experiments at the same time. So we knew that we really needed to invest again in data collection. And in this case what we really did was we invested in data collection at scale of the inner workings of the system. So what that allowed us to do was to be able to diagnose, troubleshoot if we see any anomalies or discrepancies, if someone reports a problem, or if some segment of users were seeing something that they should not be seeing, or not seeing something that they should be seeing. We really wanted to make that investment available for our analysts and for our data scientists. So that I would say was like a very, very big driving factor for us to be able to understand how the end to end system works. Because when you build models, you have like a. More often than not you want to say, hey, I'm building certainty models. Is my models providing good certainty or bad? Are they doing a good job of providing certainty or not? But you could have the most highly certain models, financial products, but if you never show it to the member, they're never going to apply for it or never going to interact with that. You don't know what's happening under the hood. So we knew that we had different sets of models, different rules, different business constraints, as well as new data flowing in. So when you put all of that together, it's a very complex system. And when you think about explainability, you really want to think about end to end system explainability. As far as the final outcome for our members are concerned, rather than looking at like piecemeal, the piecemeal is also something you get out of it, but you have to start from the member outcome and then work downwards. So the biggest investment that we made there was to just like make sure that we are collecting a lot of data about how the system is working and that allows us to then keep building on top of it to be able to. Because explainability techniques, they also keep changing so that you have an opportunity to be able to keep building on top of what you already have.

I think this is a great segue to discuss your overall experiences being a data leader and keeping data teams on track at Credit Karma. I'd love to dive deeper into the ways data drive success at Credit Karma and obviously steering the overall direction of the data team is a key component of it. How do you ensure that at any given time you're always creating data solutions that are beneficial to Credit Karma customers and that add value?

Yeah, I think it starts out by understanding the value that you're creating for the customers and the business. If you have a really good understanding of the value that you're creating, then you are able to do a good job. And the way I look at things is that you need a lot of different things to come together to create that value. You need really good data, high quality data. You need to be able to collect all of that in scale, make that available in a timely manner for both model training as well as analytics as well as model scoring. Then you need to be able to bring the right modeling techniques to place to solve which are appropriate for the problem. Over the years we've gone through various iterations. We still have a variety of different modeling techniques that we leverage. Both neural networks as well as trees and other models that we use, then you need systems and processes. When you have data, the right data, the right modeling techniques, the right systems and processes, when it all of that combines really well, that's when you create the best end user value. And if you're thinking about all of these aspects and you think about like what are the gaps in each of these areas and then you are able to fill the right gap at the right time, then you know that you have the capability to consistently create that end user value and you're going to get it wrong and how do you catch when you're going to get it wrong? You just have to reflect on what went well and what can be improved. Because there have been times at which we have over invested in data or we over invested in systems or we were interested in modern techniques at all. All of these things we've got it wrong. So the idea really is to make sure that you have all of these things coming together. And if you get one of these things wrong, guess what? You probably not deliver the user value and you have a real opportunity of just complet killing that initiative. If you're being very thoughtful, if you reflect on these properly, then you know, hey, if I got my system right, then these models would have worked. We would have done a much better job of solving this problem for our users. So it's thinking about like each of these areas on its own and how they come together and how do you reflect on it and how you can improve on that. That's how you really consistently provide value for our customers.

So of course there's a lot of priorities that can compete here. How do you go about prioritizing different tasks and projects over time? Of course, expected ROI is important when prioritizing projects. And how do you measure that roi and how does this feed into your prioritization framework overall?

Yeah, there really is no silver bullet to this. Right. So I think you kind of want to do the things that you mentioned, for sure. The other thing that I feel that we have learned and I feel like we do a good job of it is just like I'll just use the data science name, just be Bayesian about it, right? So it's really valuable in having sound priors in terms of which project is going to provide you with an ROI and which project is not going to provide you with an roi. And again, some of it is also timing, right? So it's not just getting the ROI right, it's also getting the timing for the ROI right. So one of the things that I feel that we've really got done a good job of is historically Credicrum has always done great job of attracting industry veterans into the company. And what can they help with? They can help us in coming up with sound priors and then you're able to build out a setup where iterations are cheap. So then when you have sound priors and when your iterations are cheap, then the ROI decision making process, you can afford to get it wrong. Once you get it wrong, you run through a few iterations, the iterations are cheap. That allows us to be able to revert back and then take a different decision, go in a different path. And some of this also comes from our ability to be able to build and assemble platforms that can help reduce the denominator of roi. So it's cheaper to make some of these investments. And again, that's where you also have to have a good mix of build and buy. If you have a good mix of build and build versus buy, then you have an opportunity to reduce the investment so your ROI decisions are not set in stone. It's cheaper to revert back saying all of that. I did talk about some of the data collection that we have done in the past of how the systems have worked. It allows us, for us to be able to kind of ask interesting what if questions like what if I had this data? What if we were able to launch this model or build this model? Would we have been able to do a better job of making better recommendations? So there is some science to it. There is also a lot of art to it.

Now, given that you've been at Credit Karma for about seven years now, I'm sure you've seen massive explosive growth of the company over that time. And what that means is also making sure that you scale the data and engineering teams as well as the infrastructure that supports data analytics at scale. Going beyond that though, how have you been able as a data leader to not only scale data science for the data scientists, but to scale the ability for everyone in the organization to make data driven decisions at scale? What was the role of the data team in enabling that?

Yeah, when you operate at 20 million users and then when you go to a grow towards like 120 million users, the team size naturally grows along with that. And when the team size grows along with that, you have to constantly keep evolving the processes, constantly have to be thinking about how is my organization structured? How is the organization structured for delivering what the future is going to ask out of Credit Karma and what the future is going to ask Credit Karma to deliver for the members rather than like staying focused too much on historical context. And then you're also constantly evolving the infrastructure to be able to support all the data. And it's an ongoing challenge each of these areas how we evolve the process, how we evolve the org structure, how we evolve the infrastructure. For example, when I joined in 2014, we had a few data scientists who were spread across few teams. There was no well defined data science team at that point of time. And then till 2017 we didn't have like a machine learning infrastructure team. So you have to be able to pick the time when you're going to create these new teams out of basically you're asking a few people to stop doing whatever they're doing in different parts of the organization and force them to come together and become a cohesive unit. And then you're creating a charter for them. And then you also have to be thinking about how is the business evolving over this time frame. Our business has evolved in a way where we have moved towards more of a verticalized structure. And when you move towards a verticalized structure, each of these verticals have really important, strong goals that they're going after. And as a data organization, you are in a position where you can help each of them achieve their goals. So that again comes down to like how is org and how are our systems evolving with the business needs? Then you have to be able to think about what do successful partnerships look like. You have to understand which are the partnerships where you need to be spending more time on to help them become more successful, and which are the partnerships that you can afford to kind of let it go in its status quo, at least for certain amount of time. And then sometimes you want to look at something and say, hey, that's a project. And sometimes you're going to look at something and say that that's not a project, that's a long stand, long standing team that needs to go through seven or eight different iterations in terms of building the system to get it to a good place. And then even after that, they want to be able to keep adding more nuances and keep making things more complex and more effective. And throughout all of this, how you communicate, how we prioritize, everything changes. Like we probably changed our prioritization and our planning and execution mechanisms multiple times over the years. Honestly speaking, there was a point of time I was focused very, very heavily on execution and there was less towards planning. Especially when we were putting together the initial building blocks of our systems, it really required us to be able to execute strongly. But once you have the base systems in place, then you get into more of a heap. You need to work more with the verticals to plan properly how they're going to leverage the systems to get more value out of the investments that you've made. So I think it's like you are going to keep evolving your org structure and processes along the way so that you start getting really strong in execution and you get really strong in planning. And the plan planning process allows you to have these successful partnerships where you are helping multiple business units succeed by leveraging data.

Now connecting back to scaling the data team, how have you made sure to evolve your data scientist skill set? So in terms of direction, how do you value more, do you value more generalist data scientists or more specialists? And what are the ways in which you've made sure to evolve the data team skill set.

Yeah, I talked about the blooper of trying to get our data scientists to start doing Scala at one point of time. Right. So I think it also, it goes back to what we want to deliver for the business and for our members to have a good sense of what are the current gaps, to have a good sense of how we can leverage our data really well, how we can model our business problems really, really well. When we started out, we never had the systems, the systems to be able to help our data scientists leverage something like deep learning along the way. Once we made our investments in TensorFlow and GCP, then we were able to get into a place where our data scientists were able to look at deep learning as like another tool that they added to their toolbox. So I think a lot of these modeling techniques and being able to look at data, being able to explore data, a lot of these skill sets are, I would say you keep adding it along the way as you face the right business problems. Sometimes people are going to go and do like, hey, I want to do like Bert or I want to do GPT3 or whatever, but you might not really have the use case in the business to be able to, able to leverage some of those things. I think when we started out, we definitely needed a lot more generalist data scientists, especially when we were building out our systems. And we need our data scientists to also get into the trenches with our engineers to make sure that our systems are designed well. Over a period of time, we have moved into more of a world where we need our data scientists to understand in a much deeper way the problems that they are solving and the implications of some low level decisions that they may be making, which can change the way in which you are solving the problems. And then also try to understand what, how are these problems evolving. So the partnership with more on the business side, the partnership with product understanding in specific areas, how we are doing things. I'll give you an example to kind of understand that better. Historically, credit cards and personal loans have been like a predominant driver of revenue and transactions for the business and for our members. And there are a lot more credit card transactions that happen that can happen than like home loan transactions. So what that means is you have a lot of data there, whereas when it comes to like home loan, you have like lot less data there. So to be able to build your skill sets, not just dealing with like large scale credit card transaction data, but also being able to deal with like smaller, relatively smaller scale home loan home purchase data becomes Very, very valuable. So a data scientist needs to be able to understand what techniques that they're going to bring to play. They can't take the same technique and plonk it over here. They need to be able to understand what techniques will apply better here. And then they have to continuously keep understanding the problem better and try to simplify it so that they can, maybe some of their existing techniques can still work out here.

That's really awesome. And kind of expanding it into the organizational culture and skill set. How would you describe data literacy at Credit Karma? And what are the steps you take to set sustain it? And what are the systems that enable data access for everyone and the ability to make decisions for themselves?

Yeah, I think ensuring that when you're building out an organization, you are consciously, deliberately thinking about how to make your team a diverse and inclusive team. I think being able to build a diverse and inclusive team allows you to be able to have a much broader understanding of some of the problems that our members face and also really hit it out of the park in terms of how you can have successful partnerships with your stakeholders. Because believe it or not, you want to be working with a diverse and inclusive set of stakeholders. You really need to be able to engage with them. You really need to be able to understand their problems well. And we've had, I would say, especially in the data science team side, we've had a really early success in, in hiring and developing like key women leaders who have really helped us be very strong in how we operate here. And that goes back to the data culture of setting up these key people, setting them up for hiring these key people, setting them up for ongoing growth. Growth is something that we hold really, really dearly. I can tell that you honestly, because this is the longest I've been in any company. And for me, I just feel like every year, every six months, every quarter, every month, I feel like there's always some new things that is going on that is helping me grow. And when you want to, when growth is a big part of how you operate, then automatically what happens is you can't hold on to all the decision making, you can't hold on to all the prioritizations. You need to be able to allow your teams to also do bottoms up thinking in terms of what they think is important for us to be able to solve our problems really well. And when you go through that process, then the plan that you end up getting out of that process is like a much, much richer plan. And I won't say we are perfect at This, I still feel like I'm constantly finding ways in which we can improve there. And in terms of data literacy at Credit Karma, I think it's we, I mean, like when I told you, right, like when I started, there were a lot of rules, rules all over the place. And the fact that there were rules all over the place meant that we were already looking at data and bringing these rules in and which were like more analytic, analytic driven insights where we said, like, hey, this kind of a product works really well for people in this credit score range. That kind of a product really works for people in that credit score range based on data. There were a lot of analytic insights that we had captured in our rules. I would say one of the biggest challenges that we have had is to be able to slowly kind of like remove some of those rules and allow models to learn, allow models to evolve. And that's a really hard process because models can't do everything for you. Models are going to get some things really, really wrong. And rules are the protective mechanism that you build into place to make sure that models and rules work together really well. So if you want to take one of those rules, which has been such an important factor of delivering the right value to our members, and you want to replace that with models, what you really need to be able to do is to really, really invest in transparency of how the models work and make sure that all your stakeholders have a strong sense of how do these models work. When you iterate on these models, are you following the right process? Are you getting to a better place every single time you go out and do those iterations? And if you not doing a good thing, if you're doing something which is like kind of in a gray zone, do you come and have a conversation with us before you go and ramp these models up? So I would say those have been the things that continues to evolve and continues to keep our lives interesting and challenging. But at the same time, we know that's what's really required for us to deliver what's best for our members, and not just what's best for our models or the data driven system, but what's best for our members.

I think that's a fantastic take on data literacy, specifically in how it arms people to have difficult conversations about machine learning systems and production. I'd love to pivot to discuss a bit more around the future of Data Science and FinTech and what you're most excited about. What are some of the most exciting things that you're looking forward to in data science? And specifically, what are the most exciting things that you're looking forward to that will provide value for Credit Karma customers?

Yeah, I think our CEO and leadership have talked quite a bit about autonomous finance where we are able to help our members do the right thing. Finally that's what we are about, right. We want to make sure that our members can kind of like leverage Credit Karma to be mostly on autopilot on most of the mundane day to day pieces of their financial journey and be able to take the right decisions at the right time, all the time. Right. I think if you need to renew your auto insurance, if you have, if you have like, if you have like if you just had a kid recently in the family, you might want to change what you have in the auto insurance. It's not just like lowering your rates, it's. It's also about making sure that you have the right protection for your family to be able to use the data to optimize our consumers lives in a way where it's like rich and a lot of the mundane stuff gets taken care for them and then a lot of the important things, they are still kind of in control of their journey, in control of their destiny. I think that I would. How do we go about solving those problems is probably going to take like probably the next five years as far as I'm concerned in the company. And then that's when you get into things like hey, how can I leverage causality here? Because if you're not able to do things like causality, you're not going to be able to solve these problems. Well, how do I. And do we need to make investments in something like a knowledge graph to be able to understand what the users want out of certain financial products or what are their motivations, what are their goals, what do they get out of these financial products to solve their regular day to day things in their life. So we need to make the right investments along the way in things like causality, things like knowledge graph to be able to understand our users better, to be able to understand what value financial products provide and then that will allow us to get into more of execute more on our vision for autonomous finance for our members.

That's really exciting and really awesome. So Vishnu, before we wrap up, do you have any final words before we finish today's episode?

Yeah, I think operating in the finance space and fintech space, historically the space has been really opaque, but I would say there are a lot of innovations, there is a lot of transparency, a lot of new players who have definitely embracing transparency and I would really want to make sure that all of us are doubling down on that. It's always easy to just take all this data and make more money for the business, but let's keep in mind that the data that we leverage actually belong to the users and belong to our members. And we really want to be able to make it work for helping them succeed, succeed, and the business success would just naturally follow from that, is what I would want to say.

Thank you so much for coming on Dataframe Vishnu, and for sharing your insights.

Thanks a lot, Adam.

That's it for today's episode of Data Framed. Thanks for being with us. I really enjoyed Vishnu's insights on the data science powering. Credit Karma. If you enjoyed this episode, make sure to leave a review on itunes and we'll see you next time on Data Framed.