Data science for intuitive user experiences

I like to point back to 2008 where you think it's like a once in a generation event and then like 10 years later you have another once in a generation event. These events, like you can't write them off as one time anomalies. You actually have to figure out how to fold that in and be able to essentially like automatically regress against them, learn from them, and then figure out kind of how to apply it forward. Like you said, we're at the tail end of the pandemic right now. Hopefully. Fingers crossed. Hopefully, right?

Yep.

But what happens if, you know, another large worldwide event happens 10 years from now? We can't just clip that chunk of data, remove it and act as if it never existed. You need to build that into the methodology of your models.

Big thanks to our partners Linode, Fastly and launchdarkly. We love Linode. They keep it fast and simple. Check them out@lino.com changelog r our bandwidth is provided by Fastly. Learn more@fastly.com and get your feature flags powered by LaunchDarkly. Get a demo@launchdarkly.com this episode is brought to you by our friends at O'Reilly. Many of you know O'Reilly for their animal tech books and their conferences, but you may not know they have an online learning platform as well. The platform has all their books, all their videos and all their conference talks. Plus you can learn by doing with live online training courses and virtual conferences, certification practice exams and any interactive sandboxes and scenarios to practice coding alongside what you're learning. They cover a ton of technology topics, machine learning, AI programming languages, DevOps, data science, cloud containers, security, and even soft skills like business management and presentation skills. You name it, it is all in there. If you need to keep your team or yourself up to speed on their tech skills, Then check out O'Reilly's online learning platform. Learn more and keep your team skills sharp@o'reilly.com changelog again o'reilly.com changelo.

Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changeaw.com community and follow us on Twitter. We're racticalaifm.

Well, welcome to another episode of Practical AI. This is Daniel Whitenack. I am a data Scientist with SIL International and I'm joined as always by my co host Chris Benson, who is a Principal Emerging Technology Strategist at Lockheed Martin. How you doing, Chris?

I am doing very well. How's it going today, Daniel?

Oh, no complaints. Lots of work to do. But all interesting things and fun things. You know my world. Lots of natural.

About the same. Just lots of work and trying to enjoy spring weather. So I have been since I'm living in the South, I live in Georgia. I have a standing desk. I think I've mentioned this to you in the past.

I didn't know you had a standing desk.

I do. I have a regular desk inside which I need to replace, but I have a standing desk in my sunroom.

Oh, okay.

And so it's an open air sunroom.

For those of you that aren't from the American South, a sunroom is basically just like a room. It's like a patio on the side of your house that has like actual glass walls instead of being open.

I just realized that sunroom is jargon maybe.

So I don't know.

You know, I don't think of sunroom as jargon.

But you know, clearly, obviously in certain parts of the country, sunrooms aren't as maybe appealing as as other parts of the country.

I'm showing my bias right there. So I didn't understand that. So. Okay, we're good. We're good.

Yeah.

And just doing work and enjoying the weather. Nice way to combine the two. I highly encourage anyone out there. If you have an outdoor area with a roof over it, put a standing desk there. You'll love it.

Put a desk out there.

That's right. It's great. Yeah.

That's a good idea. Yeah. Today's topic I'm really excited about and the guest we were talking before the show. This is actually a topic that's really close to my heart because as our listeners will know, my wife owns a small business and has tried to navigate, I think with a lot of success, that business through these Covid times. And of course that involves a lot of complicated stuff, especially her business is a manufacturing business. So there were all sorts of supply chain things and cash flow things and planning things that were just really difficult. And I think Daley having very interesting financial discussions with her team to try to make it through that time, from, you know, PPP loans to all sorts of things.

She needs good tools, doesn't she? That's what I'm getting from that.

She needs good tools and good ways to make sense of that financial data and plan accordingly. And I'm really excited on that topic because today we have Intuit's Director of Data Science, Nung Ho, with Us. Welcome, Nung.

Hey, guys. Thank you for having me.

Yeah.

Before we get into all of that great stuff about customer experience and working with financial data and cash flow planning and all of those cool things, could you just give us a little bit of sense of your background and how you ended up as director of data science at Intuit?

Yeah, definitely. You know, it's interesting is seven years ago when I joined Intuit, I came from. We would then call a non traditional background. So if you look at my background, I actually have a PhD in astrophysics. And before I started into it, I was doing a lot of data reduction, image processing, signal processing, and studying dark matter dominated dwarf galaxies. Right. A whole complete other world away from like what I do now in applied AI. But what was interesting thing was that at that time, right, the field was still fairly nascent. And a lot of us who came from non traditional backgrounds kind of came over into data science and AI as a field because we realized, like, holy heck, we actually were doing using the same techniques. We just had completely different names for these techniques in our respective fields.

That's so true. Right. I remember. So I came from physics too, and I was like, had this experience where it was like I was looking at all these data science methods and I was like, oh, like when you're saying you're doing this, you're just using ordinary least squares. Like, how many times have I done that in my life? Like a million times. And like you're just using a different word for it. That's kind of weird, but okay, I'll do that.

Yeah, exactly.

What is it with physicists? I'm telling you, I think that there are more people in data science and AI from physics than any other field that transfers over. You guys are ruling the world here in that way.

Yeah. Well, I think also physicists have this. Whether it's. And maybe Nung, you have a perspective on this. But I feel like in physics, for whatever reason, it breeds this sense of like, I'm not qualified to operate in this area, but I do feel like I have the expertise to work in this area. So I'm going to go ahead and jump in and do it. I don't know if you get that same sense.

Yeah, for sure. I mean, there was, you know, kind of like figuring out the language of data science and how to translate the skills that you have and the things that, you know into this, like, new industry and this new accepted skill. But just knowing that, hey, you know, I write a lot of code, I know a lot of math, I Do a lot of matrix math in my daily life previously. It seems like these are all the things that we do in data science, so let's just go for it. And exactly as you said, Chris. Right. Like now, if you look at the composition of the backgrounds of folks who are coming into data science, it's all over the place. And the umbrella has really expanded. But when I started, I actually had, I had to go through a program to help me repackage all the things I knew, like, oh, ordinary least squares. Oh, actually it's regularization and regression. So just being able to speak that language to convince folks that, hey, we had the skills. Because in the past I would write these massive programs to take raw data from telescopes and then actually convert them into meaningful information and then write papers. A lot of the day to day work is exactly the same. That's a bit my journey towards data science. I specifically wanted to come into Intuit because when I was in grad school, I used Mint. A lot of like every dollar counts and you have to kind of be pretty obsessed with it. But I had a really poor user experience with Mint and I'm like, I'm pretty sure I can do better than what exists right now. And so when I came for my interviews and they said, hey, you could totally work on Mint. You can come in and fix this. Like, yes, please sign me up. It's just this openness to allowing us to innovate. And I found it kind of refreshing that I told my boss, hey, I actually hate one of your products. Can I come and make it better?

He's like, yes, that's great. It's great to have that environment where you can sort of be comfortable sharing those things and brainstorming solutions. Of course, I mean, there's always boundaries within a workplace, but it's really cool to hear that you found that environment.

Yeah. And you know, it's been great ever since. So as I mentioned, I've been at Intuit for seven years and one of the things that I specialize in and my team specialize in is we specialize on the applied AI side. So building a lot of production models, features and systems that go directly to our end customer, because that's something that I'm personally super passionate about. I really like to take a look at some of the core problems that our customers have right now and figure out, well, how do I bring my skill set in helping solve that for you and make it easier for you? Because I previously was a customer and I think similar to you, Daniel, is that I have 10 siblings and half of them own small businesses. So it really is a space that really speaks to me and it's really fun to be able to actually build things that you can see your family using. I can really see that benefit.

I just want you to know you have my sympathies. I thought I have four siblings, there's five of us and I thought we were or so. Yeah, lots of respect there.

Do your siblings own businesses, Chris?

Actually, yes. Yes, several of them do relate in.

That sense, I guess.

Yeah, yeah, yeah, I can, but she's done way better than I have. I have a bigger family in the world doing that, so keep going. I'm sorry, I didn't mean to interrupt.

Yeah, no, I think that most people are like, how do you deal with 10 siblings? It's, you know, any one time you, you have a high probability of liking at least one of them. Right?

There you go. That's a good attitude.

And I think also like the small business owners, they, they have to, like we were talking about in physics, like this sort of comfort level with like operating in a space that you're not necessarily qualified to operate in and just like relying on the skills that you have in your past and picking up what you need going forward. I feel like small business ownership is, is very similar. It's like, oh, I've never cleared a container from China through customs with my raw materials in it. It's like there's some information on the Internet about that, but I'm going to figure it out. And maybe some of that you're going to talk about the cash flow of things and other things, I think on the financial side, but a lot of that too. It's like maybe in. I've learned a lot of terms from my wife, which I'm not a financial person, but she learned, you know, how to do like profit and loss statement and all of those things in her school, which is great. But there's all of these other like, things and like, for example, forecasting and, you know, forecasting sales and raw material need and all that. That's like, it is a very data science problem. It has to be data driven. And you know, if you get it wrong, then your business goes under. So it's very high stakes too. But maybe before we jump into some of like the individual tools that you've worked on, one of the things that you mentioned in terms of the sort of general space that you're working in within Intuit, is customer experience and AI for customer experience. So when you're thinking about customer experience at Intuit. What, what does that mean? Maybe for data science people out there that are trying to understand what does it mean to do data science for customer experience? How do you look at that?

Yeah, that's a great question. The way, you know, we, as a company approach it is we have a system called Design for Delight and Customer Driven Innovation. And the approach is to really get into the minds of our customers and understand what are the most important problems that they have. Because you can only build great solutions if you really understand what the problem is and not just like what it is that you want to build, but why it's actually important. And so we use this, the system called Design for Delight, where we do a lot of customer interviews. You work with a customer experience researcher and, you know, you just kind of see how your customers are using your products. Where are they getting stuck, right? What is confusing for them? Where are the tasks that they're spending a lot of time on that they wish they were not doing? And then there's also the other piece, which is what did they wish that you actually had that would make their lives a lot easier? And these are things that, you know, like there are some traditional methods where you can infer it by looking how a customer uses the product, but honestly, it's just faster to talk to a wide variety of customers and have them tell you directly. So starting from that base where you really understand what the problem is, then we take it back and we say, okay, what are some of the hypotheses that we have in creating the solution? And the solution could be done by data science. It could be done via an experience design, it could be done via some very simple rules. Right, but what is the solution that you want to build to solve that problem? And then you go into the mode of decomposing it into, okay, like there's the experience piece, there's the machine learning piece, there's the engineering piece. Then we go off with these hypotheses. You know, build a couple of quick MVPs, minimum viable products for our listeners who are new. Once we have these MVPs, the thing that I learned, which actually took me a little bit to appreciate, is that when you build an MVP for a customer and you show it to them, you're like, tell me how you would use this. Would this be beneficial to you? In the past, we would put in mock data and then the customer would say, yeah, if this was my data, I would react this way. I would see this, and it was actually not useful. At all, because you're asking them to imagine what they would do. And so we pivoted and moving forward from about three or four years ago when we found this insight, we actually just loaded the customer's data in, we ran it through our models, we made the predictions, whatever, and then we showed it to the customer. And then immediately, like, the feedback that we got back was so much richer. It actually helped us like, narrow in on the right solution. And it was just like, you know, there's like, aha moment for me that I guess it's obvious now, but was not obvious back then.

I love the. As you've gone through, I've had like three or four questions and you keep answering them before I ever get, no, no, no, that's good, that's not a bad thing. I'm really curious, I was going to ask you about kind of like, how do you know when you get to the quality of those inputs so that it can translate to data? And you were kind of saying, well, now we've, we're just going and taking the data. What is it about when you said, I took the data. Can you dive into that for a moment in terms of saying, how do you know that you've just ended up with the right data set to innovate in such a way that your customers are going to be delighted at the end of the process? How do you recognize that you've achieved that moment?

Yeah, it's a little bit of an art. But I would say that, you know, as a scientist, one of the things that we do is we always have a couple of hypotheses and a couple of approaches on how to solve the problem. And so at any one time we would have multiple solutions and we would beforehand identify what the success criteria is first that we want, then we show it to the customer and we compare them. So this is like a super lean, very baby early AB test. Before we go full into AB testing. What I find that gets in the way of this is a lot of times folks build models, but they don't think about, well, what is the success criteria for me, what am I trying to get the user to do? Like, if I talk to our marketing teams, it's like we want to get a user to convert, but that's like maybe like 50 clicks before you even get there. But what is the thing that I'm going to actually physically affect and ask the customer to do? That's my target. And so, like having that very robust target and measuring it across your different hypotheses and experiments is one way to do it, but it can be a little bit challenging to come up with that.

I can imagine. I know that I have participated in similar processes before and speaking from experience, not as much as you have, it was really hard to do. And so I'm very impressed with that.

Yeah, I'm curious. One perspective on this is that, like, you look at the customers and this customer experience survey and then you identify where they're getting stuck. Like you said, what if it's like scenarios where maybe it's something that you need to develop that's outside? Like the customer doesn't know that they need this yet, or it's outside of their scope of maybe, you know, what they even have imagined? Like, oh, I didn't imagine that I could have this, but how do you come up with a way to validate that those are sort of good ideas, the ones that might not be driven by the specific customer pain that you've observed in an interview, but maybe sort of new data product development is a good way to put it.

Yeah, that one. It's very interesting because for me, I tend to be fairly customer backed. And so I go back to the roots of like, well, why is this thing of pain? And not focusing so much on the what. I think a lot of times folks stop at the what, like, what is it that the customer wants? It's really more about the why piece because, you know, otherwise you can't innovate, right? You can't wait for someone to say, I really want this thing. They might not know that they actually want that thing. And so getting down to the why, and then working with the teams on coming up with, okay, well, what are some ways that we could actually solve this? In experiences that are new, in my team, we do a mix of replacing really rote existing things and backing it with machine learning. And in other instances, we're building tools that just the customers didn't even think they wanted at all. One example of that is in the cash flow space. Overall, when you think about cash flow, for me, naively, I initially said money in, money out, delta between money and money out. That's what I care about. And then you talk to a small business and they're like, well, actually, you know, I have like invoices, I have bills, I have, you know, loans that I need to pay, I have assets, and I care about all of those things and I need to know where they all are. But when we were talking about our, to our small businesses, we said, well, okay, like you want to know your invoices, you know, what are outstanding invoices, so that you can get paid. They said, yeah, we pushed and we said, well, what if we can tell you not only when an invoice would get paid, but like the date range within which it would get paid? Would that be something you know you would want? And they were like, you can do that. I didn't know that was possible at all. Right. For them. They were really thinking about just send me a reminder if it's going to be due soon. But we expanded that and we said we can actually build this new capability that allows you to go much further and have more control. And so I think that that's like kind of one example of starting with a why, which is like they need to understand their cash flow.

This episode is brought to you by our friends at rudderstack and we're calling all data engineers to check out rudderstack cloud and start building smart customer data pipelines. Rudderstack stack is warehouse first. No more silos. Rudderstack builds your customer data lake on your data warehouse, not theirs, Enabling all functionality of a CDP with more security and retaining full ownership of your data. It's open source and API first. Rudderstack can be easily integrated into your existing development processes. And because they're open source, you can see all their code so you don't have to worry about vendor lock in or black boxes. And best of all, they have transparent pricing. Stop paying your CDP a premium to store your data. Rudderstack is free up to 500,000 events and pricing scales transparently. From there, learn more and get started@rudderstack.com again. Rudderstack.com that's R U D D E R S T A C K dot com.

So I'd really like dive back into that even more because I'm very intrigued by you. You know, knowing the data that you have and figuring out how do you start that process? Because, you know, we talk to people all the time, they're doing all sorts of cool things with data. But I'm really fascinated by your creative process that's kind of encompassing that. And so when you're thinking about I'm going to innovate for the customer in a way they have no idea that I can. How do you approach that? How do you know what you're going to do with the data that you have or know that you need to go get that data in order to achieve that? And using data science as a mechanism to get there, how do you put it together before you do the do. How do you conceive of it before.

I do the do? I like that. Yeah, sorry. That's something that I think a lot of us in the field are still working through because every new problem is a little bit different. Some are much more straightforward. You know exactly which data source you're going to get, you know exactly where to start, and you even have an idea of what the initial features are that you want to start with. For others, you have to get a little bit creative. Right. And trawl through the data a little bit. One of the ways that I've tried to fast track this is that, you know, as data scientists, we don't work in silos a lot of times. We work with our analytics teams, we work with our product partners, and we work with our design teams to jam together and say, okay, these are some pie in the sky ideas that we want to work on that could really solve a big customer problem. Once we narrow in on that, we actually source it from the broader team because they have an initial starting point. For me, I would rather not do a random walk through parameter space to figure out where to start. It's easier to just crowdsource it from the team. Sometimes it's actually from our engineering partners who say, you know, I'm aware of this data that we're capturing right now. Do you think that that would be useful for you? Aha. Okay. Actually never thought of that. Let's go and try it out.

So it's almost kind of taking the creativity of the entire organization, the entire larger team, and then figuring out where to allocate resources against that creativity based on what's rising to the top.

That's right. That's right. And so when we start at the problem first, and then we figure out whether or not we have the data, because I have never worked on a project where all the data we ever needed was available. Sometimes you actually need to go and capture that data. You need to go and buy that data. Right. Having everybody in there together kind of opens up the door and more avenues for you to kind of see what you're not seeing right now.

I'd really like to dive into this idea. There's an article that we'll link in our show notes about some of the things that you've been doing during the pandemic time, and I'd love to dive a little bit into those. One of the intriguing things that I kind of noted was that pandemic hit. And then, of course, as any business, including intuit, it's like, there's this kind of natural inclination to be hesitant to do a lot of different things because, you know, we have no idea what's going to happen next. We're really unsure about the future. But it seems like, you know, your team and Intuit really took the vision that, like, hey, now is the time to innovate and now is the time to try something different. It seems like that really paid off well. But I'm wondering if you could go through that mindset and how you kind of came to be motivated that this sort of cash flow tool and things were things that you should be focusing on during this time, during the pandemic.

Yeah, Ben, you know, a year ago, like, the. The world kind of went upside down, right?

Yeah, I remember. Yeah.

And I think we all kind of had plans for ourselves and businesses had plans for what growth was going to look like. You know, for us, we. We had a roadmap of what we were going to build for our customers, assuming that the growth was going to continue. And then kind of overnight, you know, when the shutdown happened, like, you can look at it in our data for some of our small businesses, like, revenue dropped 99%. And like you, you don't see step functions like that in the data ever. So this was just a global, worldwide, massive event that everybody was feeling simultaneously and through every single new locale or state where a lockdown occurred, this drop happened. And we said this like, we have to do something for our small businesses because this not only is such a scary time for them, but it's also a huge time of uncertainty for everyone across the world.

And your data is sort of a history book for a monumental event in world history, when you think about it.

Yeah.

You know, seeing that financial data do that, that is a powerful story right there that you're describing, seeing that step function.

Yeah, it absolutely was fairly shocking, right, to be able to see it so immediately. And it was so fast as well. And so, you know, we really came together and we said, what can we do to help our small business customers? And, you know, like, we kind of sourced across the entire company a whole host of different solutions. So if you can recall, early on in the pandemic, there are a lot of GoFundMes for small businesses. We hosted quite a few. We partner with GoFundMe to enable our small businesses to raise money. But that's a little bit of a temporary solve for that moment because we didn't know how long the pandemic was going to last. And there's the PPP's loan program. Right when that was announced, we realized actually a lot of our small businesses and a lot of customers in general, including my sisters and family, who I helped through the loan process, eligibility was really confusing. Like you needed to know so much more about your business than most people were equipped to be able to do at all. And so we rolled out another program using like our knowledge engine to build like the ADA system, which helps small businesses quickly say, am I eligible for these loans? What are the loans that I'm eligible for? And get that. And then for my team, specifically, because we're so close to the data, we looked in there and we said, wow, cash flow right now, the historical data that we have about your cash flow is wrong. Those of us in this field, the historical data is where you build your models off of. And if you have this drastic shift and this drastic change, you need to go in and make huge modifications to your model to accommodate for that. Not just that, the level of granularity of details that you provide to customers, the information they want to know is different. Some businesses did super well and some are really, really suffering.

When you're looking at that data and you're going through the middle of this enormous event, one which we're still in the tail end of, presumably, hopefully the tail end. And you are, you're looking at how so many organizations are trying to cope and you have that real time data and the history developing on that and you're, you're figuring out how your own organization is going to address that service customers and deal with the same set of circumstances. Did that give you or the company? I'll let you answer it any way you want. Any sense of insight or revelation in terms of how to approach the business of finance in a different way than you ever had before, something that might actually be sustained past just the pandemic. So truly a lesson learned, not just a coping mechanism for the moment. Is, is there anything that you and your colleagues talked about about, like now that we know this, nothing will ever be the same in how we look at this and, and we're making changes.

Yeah. So I'd say that there's two parts I can speak for my team and how we build models. But I'll start with how we as a company realize and really reinforce one of our beliefs from before the pandemic and made it much stronger, is that we have our product called QuickBooks Capital and so we provide small business loans to our customers. And I would say the magic or secret sauce there is that we have a lot more data than a standard lender would. Right. For our customers, we have all of their accounting data. And from that we're able to actually offer loans to customers who traditionally would not be able to get loans. What we saw during the pandemic was that there's a large population who are underserved in general, who would not have been able to get loans, who we were able to offer PPP loans for because we knew their data very well, we knew their risk profile we could actually underwrite. So I think that really reaffirmed our decision that, you know, we need to continue to invest in figuring out ways to continue to serve the underbanked and underserved population. That's at the company level. And there's a lot that you can do in the data to figure out whether or not, you know, you're capturing the population or you're actually doing the right thing there. But on the other side, the more technical side with our teams, is that we've seen just with these large worldwide events. I like to point back to 2008, where you think it's a once in a Generation event. Then 10 years later you have another once in a generation event. These events, you can't write them off as one time anomalies. You actually have to figure out how to fold that in and be able to essentially like automatically regress against them, learn from them, and then figure out kind of how to apply it forward. Like you said, we're at the tail end of the pandemic right now. Hopefully. Fingers crossed. Hopefully, right?

Yeah.

But what happens if another large worldwide event happens 10 years from now? We can't just clip that chunk of data, remove it and act as a vacation. It never existed. And so you need to build that into the methodology of your models real quick.

And Daniel, I saw you about to say something. I just wanted to say something before you left that point. In the different industry that I'm on in defense, I noticed because it's directly to your point there. The intelligence community, which is the industry I'm in, has just released its annual assessment of world events. And one of the big things that goes directly to your point is the fact that they're talking about with the pandemic over, there will still be fallout politically and economically in various geographies and such for years to come. So the outcome of today will lead to other events that are unforeseen, some of which may be significant. I just wanted to note that's a great point and it really spans across Industries in different contexts. And Daniel, I apologize for cutting you. I saw you just about to speak.

Oh, I forgive you, Chris. This, I'll always forgive you. So you were talking about like this idea that models that you build need to be robust against these sort of once in a generation events that you can sort of no longer expect that this is a once in a generation, but you have to be ready for them again. We'll link to this blog post in our show notes, but I'm sort of looking at the cash flow planner and maybe you can talk a little bit about the goals there. But, but one of the things I see is like developing a personalized forecast. And when I think about forecasting like you were talking about, it's like, oh well, how much good historical data do you have to develop that forecast? So how is it even possible to develop a sort of robust forecasting system that's robust against these sort of, you know, really crazy events that we're dealing with?

Well, I can tell you it's dang hard is the short answer. One of the things that we got creative with in the forecasting example is, well, let's take sales as one type of input forecast, right? Traditionally you take your sales in the past and then you can project that forward standard time series forecasting. What we were seeing during the pandemic is that there are actually a lot of customers who have extremely sparse data because what used to be daily sales are now turning into maybe once a week you're getting a sale that you didn't have and that was the totality of your sale. And so incorporating that in and being able to deal with sparsity of data is one way that we folded that in. There's also actually learnings to be had. This is where you can actually transfer the models that you build from for one business to another. There are certain businesses who essentially all of their sales and all of their events happen once during the year. Like my favorite example is a Halloween store. There's a very short lifetime for a Halloween store where they have nothing, they have everything and then they have nothing again for a while. And so like for those, we actually can take the models that we've built for them, learn from them, and actually apply them for companies who are going through somewhat similar events right now. So that's where it's been advantageous to have that broad view across the industry. And being able to transfer across.

Changelog is the best way for you to directly support practical AI. Join today and unlock access to a private feed that makes the ads disappear, gets you closer to the metal and helps sustain our production of practical AI into the future. Simply follow the Changelog link in your show notes or point your favorite Web browser to changelog.com Once again, that's changelog.com.

Changelog.

It's better.

So Nung, I'm kind of interested to I guess get a little bit more into the weeds of this response to the, these changing times in your data and your forecasting. And we don't have time to get into all the specifics, but if you take and you look at, you know, some of the trends out there in AI right now in terms of, well, we've got all sorts of things. We've got, of course the trend to use, you know, neural network based models. We've got sort of semi supervised things, we've got reinforcement learning and plus this sort of, you know, quote more traditional type machine learning models or statistical models. As you're thinking and you're looking about this scenario where like looking to the forward, we don't know what sort of events are going to come up and we need to be able to adapt our models as our data increasingly changes over and over. So if you sort of match that landscape of AI techniques up with this problem that you're looking at, which of those sort of become most relevant and have you experimented with a variety of those or what have you found most useful in your context?

Yeah, that's a great question. We've actually experimented with quite a bit. It wasn't until very recently that deep learning kind of beat your traditional time series methodology in terms of performance in the space that myself and my team work in. One of the considerations that we have to make is model performance and efficacy compared to in our case, explainability and being able to actually help our customers understand why we're making these decisions. And that's specifically because we're dealing with financial data and it can have very large implications if you don't really trust the output of the recommendations we provide for you. And so, you know, in most cases where it's not as customer facing and the decision is not directly to your dollar. Right. We use, you know, more black box techniques as I call them. But for this case, specifically for time series forecasting, we tend to stick with models where we can explain why these decisions are made simply because like for our customers, like you have to be able to build trust. They won't use it unless they know what goes into it. I'm sure you know, your wife may have comments and feedback about this Daniel but when you're dealing with money, right, you have to be fairly sensitive and really understand what it is, the tolerance level from the customer in terms of like how far you can go and how black boxy you can go.

How might that evolve over time? And I'm asking a speculative question, and I recognize that, but there's a lot of concern about anything that's sensitive. I'm in a sensitive industry. You're in a sensitive industry dealing with people's money and their finances. And as these technologies evolve, both on the capability side and the, the need for explainability a for the purpose of satisfying regulatory concerns, for the purposes of gaining trust so that you can continue to innovate the way that you and your team have been doing. How do you envision, you know, when you're thinking about it, how do you envision explainability evolving and, and the level of importance? Do you think we're just going to reach a level and it kind of plateaus or how does that weave its way through the industry over the years ahead in the way that you're thinking?

Yeah, I mean, I think like where we are early now, especially in our space in Fintech, is just getting customers comfortable with having an AI system work on their behalf and making decisions on their behalf on a daily basis. But if you look at some of the other areas where AI has kind of had a bit of a head start, people get comfortable real fast. Right. And it's about, it's about building trust. And once they trust in the decision making power of the system, what I've seen is they tend to need to know less about how it works because there's that like large baseline in time that they've had to build trust in the system.

Yeah, it's a great insight.

Like I expect, you know, in the near future that we'll be able to use much more sophisticated techniques because we've built up that trust. But you know, trust takes time and it's very easy to violate that trust as well. So you have to be fairly careful and make sure you bring your customers along. But sometimes also you just need to push it through because folks don't like change. So there's that balance as a quick.

Follow up when you just talking about folks liking change, you've made these innovations that you've talked about through the COVID period because your customers really needed you to do that and stuff. And I guess there's that old saying about don't let a good crisis go to waste and all. And I realize that the sentiment on that is probably not appropriate for Covid time. And I'm, I'm trying to, to acknowledge that. But at the same time, it's during times of change that great things can really come about and happen. As you're thinking about that and knowing that that is necessary and the explainability can be achieved to some degree, gain that trust and do that. How might that is you, you have amazing tools going forward in the years ahead that are being developed. Deep learning goes farther. Reinforcement other branches that we don't yet know about that could come about at some point from current research. How might you approach some of those in terms of, you know, how do you make that judgment? On now's the time to push because they really need it and I can really help them and I know that they'll trust it quickly versus being more conservative and saying, you know, I'm afraid to. How do you assess that?

Yeah, I would say just ab test it. Right. A small a B test. Oh, good point. Costs very little. Right. And there are certain portions of the population who are okay with it. Right. So doing a random test with a small population on a very bold idea that pushes it far ahead, you might get actually really, really great results. And then you can kind of step your way up into opening up more is how I would, you know, think about approaching it. Because like, I can tell you, like this population we serve for some of our products with accountants, they're not necessarily, you know, the most open to trying out new technologies at all. But you know, if you go through just with an A B test and test without them really knowing it, right. Then you really get the like a true reaction and then you can actually start like bringing the people along as you have that data and that evidence. Gotcha.

Chris knows I'm always the one like asking super selfish practical questions because I always struggle to do practical things myself. And I want to make sure I know there's people, listeners out there that are all the time thinking about practical data science things. And I noticed in reading about some of your work, you're talking about having, you know, individual models for different transaction streams and combining, you know, aggregating those together. And I'm just thinking as like in a practical sense, you've got, you know, models per transaction stream, per user and you know, how many users does this QuickBooks have? My question, I guess is like, when you're dealing with this many models and this many predictions, do you have any sort of practical advice for in terms of model management and like model debugging and that sort of things. You know, I think a lot of people can sometimes get something into production, but then managing it after that, especially when you're managing for multiple customers, becomes really difficult. So any practical advice on that front?

Yeah. Oh, man. The operational component of the work that we do cannot be understated because it's not just about building the model. It's actually taking to production, dealing with the etl, grabbing the data, serving the customers. So you need to really make sure that you have fairly robust monitoring in so that you can be proactively alerted if something goes wrong. Because the worst case scenario would be a customer contacting you and saying, hey, you totally messed up my cash flow. Now I can't pay a vendor this month, right? That that would be disastrous. So building and monitoring, but you can't, you know, feasibly monitor for every event, for everything. And so doing a sampling and monitoring along the way, I will also just put in a plug, which is that when you're building for us, we have 3 million customers in the US alone for QuickBooks. And then multiple transaction stream grabbing in all that data, we quickly get into the realm of over 10 million models being built and served. Start in batch mode. There's this obsession with building models in real time for everything. Every single team I talk to, they're like, when am I going to get the model in real time? Well, let's start with batch. The complexity for a batch model is just much easier to handle. You validate your hypothesis, you validate the use case, and then you can say, you know, is the ROI enough if we move it to real time? That's one area. There's also the thing that we've made an investment in that has really helped is investing in a robust machine learning platform that allows us to do distributed training and distributed scoring. So marrying, you know, the best of essentially the MapReduce framework, right? And doing that distributed computing with our models and the cash flow is not the only use case where we have individual models for users, for transaction categorization, which is where any transaction comes in. We figure out which accounting books it goes into. We have individual models serving in real time for customers in that case as well. But you know, you also have to figure out like when is the scenario where you actually want to get to that level of complexity. So for us, we always start with one base vanilla model for everyone to start with and then see how far we get. And then we figure out, okay, how do we blow this up in terms of scale and complexity and how much headroom is there in order to solve the problem.

Well, gotcha. You said something as you were going through there that grabbed my attention as well as you were going, and that was you talked about platform. And you can answer this one any way that you want, because not every company wants to talk about the details of their infrastructure and stuff like that. And I totally get that, but how do you approach platform? Because that is one of those practical questions for practical AI that everybody has to contend with. And we often ask guests how their organization is tackling that. So I'd love to hear how Intuit is approaching a data science and AI platform and, and what that means to you, because it means different things to different people.

Yeah. So I've been at Intuit for long enough where I've lived in the world, world before we invested in a platform and the world after we invested in a platform. And I can tell you that like, you know, for us, a platform should enable us to be able to ship machine learning models that are high quality, that have scale built in faster. So those are essentially kind of like the requirements of what a great platform would be able to allow you to do, essentially. And so our platform solves for being able to deploy models quickly, train models in a consistent way, and then monitor the performance of those models when they go out the door. And you can find this. You know, we have a close collaboration with AWS and underneath we use SageMaker as one of the core components of our platforms. Different companies have a build versus buy. In our case, we partner specifically for the. But for me, as a data scientist, what I want out of a platform is to make the job of actually the maintenance, the monitoring, the scaling easy so I can focus on the algorithms. That's what we've optimized for.

Awesome. So at the beginning of our conversation, you mentioned how your engagement with Intuit really started from a felt need that you had and a project problem that you saw and you brought that problem to Intuit. As we kind of close out here, I'm wondering, as you look to the future, what are those things in your mind like. Oh, I, I really like, I see this problem and I really think AI or data science has a great solution for it. I just haven't, haven't done it yet. Or maybe it's things out there that you just like you're really excited to, to try, but you haven't tried yet. When you're looking to the future, what kind of, of keeps you up at night and runs through your mind?

Yeah, man, I am constantly chasing for me, one of the holy grail problems, and I would be happy to retire if we could solve this, which is actually, how do we go from the state right now where if you have to actually run your books every month, if you're a small business right now, a lot of small businesses are doing it manually. They're entering things by hand. They're like figuring out the complexity of accounting. Like as we talked about early on, like a lot of small businesses don't go into the business of small business thinking about books, right? They think about, wow, I really want to do this thing. They don't. It's not, no one gets joy from doing accounting, to say the least, right? But it's a really difficult problem, right, to go from raw transaction data to a state where we can actually tell you, this is what your books are like, this is your cash flow, this is how much taxes are owed, and this is how you should run your payroll. So for me, I'm always chasing after, how could we actually fully automate the entire accounting process for our customers so that they can actually spend time doing the things they like and we can have the machine learning solve this problem? And it's, it's really difficult because it's so multifaceted. And that's where, you know, in the future we're really going to be, you know, innovating quite a bit, both on the algorithmic side, but also on potentially making fairly major changes to our product to get, you know, folks comfortable with this. So I'm super jazzed about that. I know not every kid grows up saying, you know, I want to be an accountant and I want to build machine learning for accounting. It's such like an old problem, but it's, for me, it's a super exciting problem.

You know, it's fascinating to hear you say that because as you say that, I'm thinking back to what you were talking about, customer experience, and holding that as high as you do. There is a self awareness in you and your organization when you recognize that while your product is doing great things for someone's business, that's not what they want to be thinking about, that's what they want you to handle. So you're basically saying the highest thing is for them not to have to worry about us at all. We're just there doing the magic, but they never have to deal with us. And that is almost like perfect customer experience when you think about it, because you're optimizing their own experience so they can do the thing they love. I loved hearing that answer.

I've learned a lot in this and I think that really appreciate the insights and thank you so much Nung for joining us. As I mentioned I'll point our listeners to some links in the there the description so make sure you check those things out and check out what Intuit is doing in this space. But really appreciate you taking time. It's great to talk to you Nung.

Yeah thank you for having me guys.

Thank you for listening to Practical AI. We appreciate your time and your attention. Follow the show on Apple Podcasts, Spotify or your favorite podcast app. Your neural networks will thank you you we are also on the web at PracticalAI FM. There you'll find recommended episodes, listener favorites and a free sign up to join the community. Practical AI is hosted by Chris Benson and Daniel Whitenack. It's produced by Jared Santo with music by Breakmaster Cylinder. Thanks again to our sponsors Fastly Linode and launchdarkly. That's our show. We hope you enjoyed it and we'll talk to you again next week.

Sam.