#73 Scaling AI Adoption in Financial Services with Chief Strategy Officer and Head of Financial Services at TruEra Shameek Kundu

Hi, I'm Maggie, VP of Curriculum at datacamp, and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI Transformation Journey on datacamp.com.

Hello, this is Adele Neme from datacamp and welcome to Data Framed, a podcast covering all things data and its impact on organizations across the world. Arguably the most data rich industry out there is financial services, whether banking, insurance, investment banking and more. Data science and machine learning have a variety of use cases. However, that doesn't mean that machine learning is being adopted to its fullest potential as there are a lot of different obstacles in the way. That's why I'm so excited to have Shamik Kundu Chief Strategy Officer and Head of Financial Services at Truera and former Group CDO at Standard Chartered. Shamik has spent most of his career driving responsible adoption of data analytics and AI in the financial services industry. He is a member of the bank of England's AI Public Private Forum and the OECD Global Partnership on AI, and was part of the Monetary Authority of Singapore's Steering Committee on Fairness and Ethics, Accountability and Transparency in AI. Most recently, Chemik was Group Chief Data Officer at Standard Chartered bank where he helped the bank explore and adopt AI in multiple areas and has shaped the bank's internal approach to responsible AI. Throughout the episode, Shamik discusses his background, the state of data transformation in financial services, the depth ver breadth of machine learning, operationalization and financial services today, the challenges standing in the way of scalable AI adoption in the industry, the importance of data literacy, the trust and responsibility challenge of AI, the future of data science and financial services, and more. Shamik, it's great to have you on the show. I'm excited to discuss with you the state of data science and financial services, your experience leading data science at major organizations, and your current role at Truera. Before we begin though, can you give us a brief introduction about your background and how you got into the data space?

First of all, thank you so much Adel for the opportunity. It's great to be on your podcast. To your question, I'm an engineer by training. I followed it up with an MBA in Finance and systems. First job was to help build an online retail brokerage, believe it or not, two decades ago back in India, then joined McKinsey, spent eight years advising financial services clients in Europe on technology and operations topics and then in 2009, right in the middle of the financial crisis joined Standard Chartered bank which is an Asia, Africa, Middle east focused international bank. And I was there for the last 11 years focusing on data and technology roles before joining Truera this year.

That's great. And you're someone who has a breadth of experience at the intersection of financial services and data science and AI. As you said in your previous role at Standard Chartered you've had multiple CIO roles and were the group CDO for the organization for about 6. Given your breadth of experience, I'd love if you can describe how you view the state of data transformation in finance today and how it has evolved throughout your time as being a data leader in the industry.

So financial services has always been a data driven business, right? Everything you do, there is nothing in some ways nothing real in financial services. It's all movement of data from one place to the other. Your bank balance is a piece of data short of physical cash and the few checks and demand drafts that are there. Everything in financial services is data. So it's always been a business that is all about processing, storing, protecting and moving data around. What has changed is two things. First, over the last six years, first I have seen the questions around data move from being a pure defensive play to something that has also got a strong offensive leg or a business facing leg. So defensive meaning I need to take care of this data in order to run my business. Such as? Well, I need to know the balance of the customer, I need to protect their information, I need to send the data safely. All of this is I need to because I am required to and that is core to my business. From that to a more offensive which is actually using all this information I am able to do a lot more for my clients, for my business, for my partners and so on. So that's been one shift which is pure defensive to let's call it a defense and offense play. Now just before I move on from this, I do want to stress the offense and defense bit. It's not that the defense part has gone away. In fact arguably it's become more sophisticated as regulators and citizens have started looking beyond data quality which what most people are worried about earlier and data retention, to privacy, data ethics, algorithmic transparency, fairness, all the bigger concerns about bigger big tech and so on all of that has made the defensive angle much more interesting, I suspect to CDOs, but it has also been in addition to a strong enhancement on the offensive side. So that's one big shift I've seen. The second thing is who talks about data? When I started being Chief Data Officer, everybody in my bank at least was happy to leave me in my corner and deal with data all on my own. By the time I left everybody from the group CEO down, actually I lied from the group chairman down, understood and wanted to be involved with the data journey. Right? So there's a much broad, a very big broadening of the set of people in a bank or a financial institution or who care about data and who think about their job being primarily about data, well beyond the people who have data in their job titles. So of course these are the two big shifts I have seen. Of course there have also been a whole host of enabling changes on the side. Everything from strides in data analytics, machine learning, related technology in adjacent technology spaces like APIs that has made data interchange so much easier. There's a lot of broadening and deepening of data related talent in the organization. Regulators have understood the data and analytics related challenge and opportunities more. So all of this has happened, but ultimately these have made these two shifts possible. From defense only to defense and offense, and from being a specialist geeky thing left on its own to something that everybody from the CEO down understands and wants to be involved with.

I find this shift from defensive to defensive and offensive to be very fascinating. You mentioned that in the past few years the offensive part has been about enabling value through data science, analytics and machine learning. Do you mind describing where are these areas of value in financial services are today?

Sure. I mean, maybe instead of speaking solely in terms of use cases, maybe we can start with what it does to the financial services business model. So the way I think about it, the first category where you'll see many of the early use cases of data and analytics, and particularly of machine learning is on improving the effectiveness of risk management inside financial services firms. Now you might say, well that sounds very boring. But actually, as you well know, banks and insurers or banking and insurance is fundamentally about managing risk, right? In various forms. So being able to do that more effectively either through better anti money laundering solutions or new ways of assessing and monitoring credit risk, or new ways of detecting claims fraud in insurance, or even detecting cyber threats, that is clearly one of the most important and frankly the earliest areas where advanced analytics has been having a lot of impact. Right. So that's the first one. So that's risk effect, risk management effectiveness and risk management. The second, which is arguably not new at all, is incremental revenue increases. So again, you can argue this is not new because the first analytics use cases in banking were around customer retention and cross sell, but these have gone massively turbocharged today on on the back of advances in analytics and data. Right. So that's the second block of area use cases, if you will. The third, of course, are efficiency improvements in the middle and back office process. It's not very glamorous, but actually can save quite a bit of dollars when you automate stubbornly manual processes such as those in trade, finance and insurance claims assessments. The fourth, closely related to the above and one that is not always visible. But a lot of times when you and I as customers, we think about the end to end digital experience. Whether it's the time and effort it takes for us to onboard as a client or to complete a transaction or to get information about where my payment is, all of that is actually underpinned by a lot of data and analytics, particularly by good data and data interchange, if not advanced analytics always. So there's a big role that data and analytics is playing in shaping the customer experience on your app or on your website, whatever. And finally, and most importantly, there are transformational business model changes that data and analytics is enabling. So this is where you're not making the existing business less risky or more efficient, or slightly more revenue producing, or when you're not just improving the customer experience, but you're actually building brand new businesses. So what do I mean by that? Well, this is the whole holy grail around improving financial inclusion. Not just in Africa and Asia and Latin America where not everybody might have a bank account, but even in supposedly more developed economies where the access to financial services is uneven. And by being able to use data and analytics effectively, you're going to be able to expand that access and thereby create brand new businesses for yourself. So that's a huge area of opportunity. There's opportunities around building brand new businesses like banks that are tying up with e commerce platforms to be the bank underpinning them. They're calling it the platform as a service kind of strategy. Right where you are the bank underneath many E commerce partners, that whole business model is based on that. But also if you're not a bank or insurer, if you're an insuretech or a fintech that is either trying to steal some of the business from a bank or insurer, or trying to support A bank or insurer. Again, you're able to do things in dramatically different ways. For example, you know, you're able to take, you know, in insurance, for example, there's entirely new ways of underwriting insurance that is based on IoT data and analytics that was just not possible. So it's almost like a continuum from more effective risk management to incremental revenue to efficiency improvements, to better customer experience, to finally large scale changes, transformational changes. In business model. There's this whole spectrum and of course adoption levels are different in different areas.

And how would you then describe the state of adoption for many of these use cases? It seems that there's a breadth of use cases being operationalized, which is extremely exciting. In terms of scaling value, where do you think the industry stands today?

So I think it's important to distinguish between what I would call just data and analytics, which includes some predictive models but not machine learning, and on the other side, the machine learning end of it. If you just refer to traditional data and analytics, meaning the use of extensive use of data, descriptive analytics, visual analytics, predictive analytics, not using machine learning, I would say four out of these five categories are pretty advanced. The only one that is not that advanced is building transformational new business models. But whether it's better risk management or being more efficient or transforming the customer experience, all of these things are being done quite a lot at scale. I'm talking about tens, hundreds of millions of dollars that banks and insurers are getting in terms of incremental value today, not tomorrow, today. Right. This has changed significantly in the last four to five years. But when it comes to machine learning in particular, the story is different. I would call it the broad but shallow story. Right. So, and I put three different sources for you. One is a Bank of England survey from 2020 and 19 which looked at the level of adoption by banks and insurers which are in the uk. Another is a study by Temasek, which is a Singapore sovereign wealth fund, which was recently released on the adoption of AI in financial services. And the third is my own rather unscientific assessment based on 35 interviews in the first half of this year. Now, all of these three things suggest two things. One, that between 50 to 65%, so between half and two thirds of even the traditional financial institutions have started using machine learning in a non trivial way, meaning beyond a pilot or a proof of concept, they're actually using it. There's some value coming out of it, but between just 10 to 20% of them. So only 1 out of 10 or 2 out of 10 depending on which source you choose, are actually enabling AI at a level where it makes a real difference to the bottom line. So it's not just pilots and proofs of concepts. People are adopting AI. But if you ask the CEO of a bank or insurer or even most fintechs, will a failing ML model become a big source of worry for you? In the vast majority of cases, it's not reached there yet. Right, and that's the story. It's broad but shallow. The only exceptions, I would say are in areas like marketing and fraud analytics, where of course machine learning adoption has been there for a while. But generally speaking, traditional data and analytics adoption, quite high, quite impactful in many organizations. Machine learning adoption, broad but still mainly shallow in most. But in all. But like the 1 in 10 or 2 in 10 organizations, what do you.

Think are the main barriers to that deep adoption of machine learning and AI in the industry?

As you say, there's groups of these barriers, there's technological barriers, there's organizational barriers, there's talent barriers of business that you guys are in, there's barriers related to data itself and then there's regulatory barriers or things that can be construed as barriers. And finally, of course, there's barriers around trust. So let me go slightly deeper into each of them. I mean technological barriers, Banks and insurers have built up their systems over decades, sometimes in the case of my previous employer, over a century. And they will have the proverbial silos and spades, right? I mean they will be. It might sound like a cliche, but it is really a very big challenge to get all your data together about a particular customer in a way that can both make it meaningful to do analytics on that person. So I really understand Shamik Kundu as a customer because I've covered everything from there wealth management holdings to their last interaction and complained with us to their external data. All of it are brought together in one place and I've done it sufficiently in real time to be able to influence a frontline business process. I mean you can do one or the other. You can either give real time interactions with a customer which use some limited data, or you can build a very nice picture of the customer. But it's not possible in real time. I mean people will tell you it's possible to bring everything together, but I'm yet to see that. Right. So that's a very big technological barrier, by the way. It's not just a problem with legacy banks. A common mistake is to say this is why fintechs or neobanks will win only if they're playing in one area. The moment a fintech or a neobank tries to build out its profile, you have no option. You will need to get deeper and deeper into specialized areas. Of course it won't be as bad as a traditional bank or insurer, but this challenge of saying actually I've got systems for different parts of my relationship with the customer, I need to bring that together. It's quite a big one. So that's probably the biggest barrier that you can see. Visible barrier. There's organizational barriers partly because of lack of trust, which we'll come to later on algorithms, but more broadly about lack of trust about how other people will treat my data. There is a certain degree of silos within organizations where different teams might be concerned about sharing their data more broadly in the fear that it might not be handled properly or even not knowing whether this data that I've collected for XYZ purpose can be used elsewhere. If you're coming from a tech first organization, frankly these barriers are the reverse. I mean they start by assuming everybody can access everything, then you've got talent related barriers. I think an area that you guys will be well aware of. The only thing I would say is it might be useful and we can talk more about this. To talk both about the core data related skills, but also about how to increase the data quotient of the rest of the organization. It's not enough to just have a pool of data specialist or analytics specialist, you need the entire organization's talent, AI or data quotient to increase. Otherwise you can't get the full value. I mentioned there are barriers around data itself. I mean the fact of the matter is we sometimes overestimate how much data financial institutions have or how much they're able to use. And that means in several areas there may simply not be enough data to build an effective machine learning model. One example is money laundering. Right? It's one of the best areas anti money laundering is one of the best areas where you'd want to use predictive analytics to dramatically improve your ability to identify money laundering activity. But here's the problem. When a bank, after a lot of pain, after a lot of false alerts, I'm talking 99% false alerts. When a bank takes those 1% where it believes it's got real money laundering situations and reports it to a law enforcement agency, they don't actually get back positive confirmation often because it won't be known for years. So what that means is one of the most important factors in any supervised machine learning, which is the ground truth is missing. You don't have positive instances of machine learning, sorry, of money laundering highlighted. So how are you going to train? Of course you can do unsupervised learning, but all you generate with that is unusual behavior. Not. Yeah, so it's a really difficult. So this is just one example. But there are many areas like this where it's not that easy to say I have past behavior that I can use. I mean if you look outside banking, for example, one area, Aurora banking and insurance. One area where this became very obvious was, I mean, look at how poorly machine learning performed in most areas of dealing with the pandemic. Whether it's predicting who's going to be hit by it or predicting which medicine might work or predicting how the vaccines. I mean, I think some models got the numbers right, how many people will have it. But in terms of helping authorities with either what medicines to apply or who is more likely to build it, it's just not done well. I mean there's been a very major review in MIT tech review of course for that. Right. So these are quite serious concerns. And then there is of course regulatory concerns, this is increasing over time around data sovereignty, concerns about the power of big tech and big data concerns about unfairness, I'm sorry, unfair behavior with algorithms. These are also impacting. And then lastly there is trust. Now what I mean by this, this is particularly around the trustworthiness of algorithms in an industry that has a lot of specialization and there are subject matter experts who've been kind of, you know, you can argue actuaries are the original data scientists. They've been there for a very long time. In insurance, for example, you now suddenly come in and say, move aside, my algorithm will do a better job than you because it's been trained on historical data. Well, first of all, it might not be true. Indeed, as experiences like Covid show, it's a valid concern that past behavior is not always a good indicator of the future. But secondly, even if it was right, even if it was right, you are entering a business for the vast majority of banks and insurers where there is an existing way of doing things. You cannot just turn everything off one sudden day. At least if you're a traditional banker insurer, you do need to do the work as a data science team or as someone who's using, wanting to use data science. You have to do the work to carry along the rest of the organization. And of course you have to Carry along your clients, your regulators and so on. And I think this has been a major barrier to the adoption of machine learning in regulated financial services.

Very excited to unpack trust and explainability with you, but before we do that, you mentioned here talent transformation, which is something that datacamp focuses on. What are your thoughts on the talent transformation challenge required in financial services to have scalable value from data science, machine learning and AI? You mentioned a lot of technical skills in data science and machine learning, but going beyond that, how do you imagine a data literate organization and the industry to look like?

Yeah, I mean, to be honest, you guys are probably the specialist, but I've been thinking about this and I think it's an area where I'll be fascinated by how you guys continue to contribute in this space as well. But as you said, there are these two blocks of work. In the first block, I think of course as data scientists, data engineers, but some of the other areas are somewhat neglected in my view, like data risk management, which people broadly will call it data governance, but it's, that's a very big term. I think you need people who are data librarians. I mean, if you go into, I think the original Google piece was, I suspect they hired many people who literally did information management, right, which is how to structure data properly. So I think there's more specialization even within data. Beyond people who build predictive models and people who run the pipes for data, there's a lot more. Whether it's data risk management, data governance, data visualization, I think it needs. There are people who can do data visualization, there are people who cannot. And I by the way, firmly in the second category. And if you can't, you just cannot tell the story in a way that works. So here the focus is both on quantity. I think we don't have enough of these skills, as well as quality. But it's also very importantly about making sure that these specialists understand the broader context of the industry when they're joining. So to take a non financial services example, just going back to that Covid report in the MIT Tech Review, you know, one of the examples quoted there was how a particular hospital had very high cases of COVID perhaps because of its catchment area or whatever. And what did the model learn from that data? That hospitals whose printed reports used a particular English font or Roman font font were the ones most likely to be having high levels of COVID Right now any basic subject matter expert would say, of course this is problem, but if you don't combine data science expertise with some very basic understanding of the domain you're talking about, then it doesn't work. So one area that I think even for the specialists who are working in this field is actually to increase their financial services quotient. So just as I'm going to talk about the rest of financial services organization increasing their data quotient, I think the data specialists need to increase their financial services quotient or any industry that they're working. So that's one block of work where I think that will be quite helpful for data scientists, particularly those coming from outside the industry. But the second, as we both discussed, is how do you take the rest of the organization, CEO down to the facilities person? And I really like the facilities or real estate example because you don't think of that person as the most data driven. But actually before COVID came in, a lot of the optimization of, of the facilities in a bank or insurer, right? Which rooms, how much energy are they consuming? It was very data driven and I was super impressed because this particular area was part of my CIO functions thing. And I was super impressed by the fact that at Standard Chartered at least the real estate folks were one of the most advanced in thinking about it. And it's a great example because you don't think about the real estate person as the first person when you think of data literate. But guess what? They were extremely because their entire value proposition around how to make businesses more eco friendly, our footprint more eco friendly, how to reduce the space in a, you know, in a sustainable way. All of that was so fundamentally dependent on data. And I just take that as an example. Every single role, from public relations to HR to of course core banking and insurance roles, to even regulatory interaction, every single role needs to become much more data literate. You need to have people who can understand the opportunities and risks arising from data and algorithms. You need to know how to best use it to your advantage in your job, not in somebody else's job. You need to be able to ask the right questions when you're promised the moon by an internal or external salesperson. And look, it's a tough one, but my sense is this is a tougher challenge. If I take an organization like Standard Chartered Bank, I would expect, I don't know, data specialists would be probably in the low thousands of staff, right? Any organization, any bank of that scale, like let's say out of 100,000 people, you probably will have 5,000 people who are data specialists at best, right? But it's the other 95,000 people. If you do have 100,000 who need to become data literate. So arguably that's an even bigger challenge. And to be honest, I don't know if there's a well proven template for it. I think it's a mix of some basic education with hands on ability to play with data and being able to make it relevant for them. I think too often all of us data folks have gone into this attitude of don't worry, leave it to the experts. Wrong attitude. Yes, the experts will deal with it, but we also need to get the rest of the organization engaged so that ultimately, even if it's not the best 100% best way of dealing with the problem, Maybe it's a 90% best way, but it has much more chance of success because it's got more.

And to add to your point, data literacy, it's one of the few ways that you can increase the adoption of data products that data teams are working on. Because ultimately data teams are creating data products for the functional business experts and there needs to be a common data language to enable that conversation between both of them. I think there's an interesting connection as well between the trust element and the data literacy elements and acquires the ability to understand and have intelligent conversations with the experts building these data systems. So I'd love to deep dive more into your current role at Truera and the importance of trust in AI in financial services. You've mentioned trust as a major barrier to AI adoption in financial services. Do you mind walking us through why building trust in AI is such a difficult, such a difficult task and for financial services organizations, sure.

Maybe I'm going to, I want to pick on your words and say, actually it is not building trust which is a difficult task for financial services firms. It is the fact that they have a huge amount of existing trust level to defend. Right. And losing that trust is both very easy and, and catastrophic to them. I mean, you could see how bad it could be when the previous financial crisis struck, when if people lose trust in financial services, then that's the end, right? I mean, other than your doctor, your bank or insurer probably knows more about you than almost everyone else, right? We've had instances in the past of people complaining about why their credit card statements went to their home address because unfortunately it contains certain expenses with another partner that the wife was not or the spouse was not meant to know about as an example. So sometimes your bank or insurer might know more about you than almost anybody else, other than maybe your doctor. If you can't trust your financial service provider to do the right thing by you, to treat you fairly, to protect your data from misuse. Where does that leave your relationship with them? Right, so it's not that building trust is a new act, it is actually that there is a huge amount of trust to defend and if you don't watch it, it can quickly, quickly fall on your head. Very quickly. And particularly with large organizations, the cost, this is almost the one advantage compared to more agile newer firms that are coming in it is that while it's a well established bank or insurer and if I lose that trust is just particularly if you're using it for something high stakes like deciding my health insurance premium or deciding whether I can get the loan or not, or being advised whether I should make this investment, significant investment or not in those areas. If I can't trust it, it just becomes extremely complex. And so it's at two levels. One, as an organization I want my customers and indeed my regulators trust me. But actually there's a DNA of this kind of trust culture inside traditional banks and insurers at least, which means that even inside the organization there are multiple layers of people who are obsessed about this, maybe sometimes too much. But you have to convince all of them that yes, this thing can be trusted upon and that is why this is such a big challenge.

The task of creating responsible and trustworthy AI is even more exacerbated by the evolving regulatory landscape in financial services. Do you mind walking us through how the regulators are responding to the emergence of AI in financial services and what are the major concerns that need to be addressed?

So the interesting thing about this question is how much it is evolving. So in the Weekend I've had two new LinkedIn posts on this topic. Just because two regulatory things have happened over the last week, frankly, one in the US and one in China. But yeah, I think it is a fast evolving space. Now financial regulators as against broader technology, data or competition regulators, they've actually been quite reasonable. They've been cognizant of both the risks involved and the need to continue to encourage innovation in this space. In fact we in TRU era have been closely engaged with both regulators in the UK and Singapore and many other countries on this topic. So several regulators started off with early guidance minded guidance, not regulation. Regulatory guidance as against regulation. Right. Means not mandated. So regulators in Singapore back in November 2018, Hong Kong one year later Netherlands Canada, bank of England, sorry, bank of England hasn't published but these others that I mentioned, they've all published specific guidelines on the use of AI in Financial services, the bank of England and Financial Conduct Authority in the UK have formed a consultative forum of which we're a part. US banking regulators have sought massive major industry comment on a wide ranging set of questions around AI risk. That was in March and then of course over the weekend or just before the weekend, the securities Exchange Commission asked a specific question about how online retail brokerages and how that kind of investment, online retail investments, the use of algorithms to encourage certain behavior there, that's been another area of interest. So so far largely the focus has been on guidance and on consultation rather than new binding regulation. But I do think in the next six to 18 months, depending on the geography, we're probably going to see regulators becoming more explicit in their expectations. Industry players now this may not be a bad thing because frankly leaving a lot of guidelines and lots of things to discuss is in some ways uncertain. Getting regulatory uncertainty need not be a bad thing right now in terms of what they care about though it's actually quite clear now explainability, fairness, explainability both internally meaning if the organization wants to understand what is happening as well as customer facing or external facing transparency. So that's clearly big area fairness. So preventing unjust bias or unfair outcomes. Stability of the model, so making sure that the model won't break the first time data changes dramatically and of course overfitting, which is a particular issue with machine learning models. So these are quite clearly key areas of focus. There's also some broader aspects which is worth spending a little amount of time on actually often financial regulators are not just worried about whether the model and the data is being managed well. They're worried about back to that point about rest of organizations AI quotient improving. They're worried about whether the rest of the organization has fully understood the risk and reflected it in the actual business. So for example, most recent piece around the around the use of digital engagement in online retail investments. The question is not whether the use of AI to encourage someone to buy a product is illegal. It's not, you know, as long as the product itself is not illegal, you can use any kind of algorithm to encourage them. What is potentially bad is to sell someone a stock or bond that was wrong for them given their risk profile. Now if you're a human advisor, you know this, this is your day one piece. You know that you are, if Shamik is a medium risk person, you do not sell that guy an a complex structured hedging product. But the algorithm, if you don't tell the algorithm, the algorithm doesn't know it. So there's a difference between governing a model, an algorithm, or sorry, an AI, a machine learning model for fairness, for explainability, overfitting, etc. That is actually relatively well understood. What is more complex is how do you make sure that every element of what a bank or insurer or any fintech is doing, they are thinking about the indirect implications of using AI in that space. That is the more complex one. I think that will evolve even further over time.

How do you see it evolving? What's the charter you can propose for a bank, for example, to evaluate its risk correctly?

That's a really good point because actually many banks have struggled with this. Banks and insurers have struggled with this. But I think the easiest way of thinking about is you don't actually need a large number of data or AI experts, you need some data experts, but most banks and insurers have them. What you need is, frankly, at least my personal experience has been if you spend a lot of effort educating the rest of the risk folks, whether it's a credit risk or market risk or compliance risk or reputational risk person, if you make them aware of how the machine learning models work, what could be the risk, etc. Actually they're much better at working out what it might mean. So the first time I spoke to somebody who was managing financial markets risk, this guy immediately picked up on what I was saying and saying, oh, okay, hold on, there might be an impact of this on anti competition policy. I said, what does that mean? How is that possible? Said imagine if my model and another model in another bank are engaging in rigging Libor prices. That could be a problem. It's like I hadn't thought about it as a data and AI person, but he immediately thought about it because he's thinking about the intrinsic risk. So I would say that's probably the approach. Don't try and get one central team to try and predict everything that will happen. Disseminate the knowledge of both the opportunities and the risk from AI across the broader community who are well placed to deal with each of these risks and then let them internalize it and figure out how AI will impact them. I think that is a better, more federated approach than trying to say one person or one team in the bank or insurer will somehow become the master of all AI risk at the bank.

And this is where data literacy comes into play as well. Because if the business expert does not understand the limitations of machine learning or how a data system in general operates, they won't be able to make those assessments. So Something TRERA specializes in is model interpretability and explainability. Can you give us an overview of the state or what is possible today in model interpretability and explainability, especially given how niche it is of an aspect in machine learning research?

Yeah, talking about how it's a niche aspect, one of the reasons I joined Truera, I mean initially I became a client and then joined them, was I heard that the founder, Professor Anupam Dutta, first started researching this in 2012. I don't know about you, I had not heard of model explainability and interpretability back in 2012. So I was thinking, well, the guy has been researching it this long, he must know what he's doing. So anyway, so to a question, where is it? I think first of all it's worth thinking about the difference between what some people call inherently explainable models versus the post facto ones, the ones where you can only explain after the event. There is a school of thought which says maybe in certain very high risk areas you should only use inherently explainable models. However, there are two concerns that I personally have with them. One is there are several areas including image and text related pieces, where non structured data, where inherently explainable models just have not got that far yet in performance. It might change over time, but I think the bigger problem is it might sound sometimes create a false sense of comfort because you might get a very well explained but extremely complex inherently explainable model. But if you haven't synthesized that to a level where it actually makes sense to human beings, it's actually giving you a false sense of comfort. So therefore I think there is a role for inherently explainable models to play and there is a role for what is crudely called post hoc explainability. And I think there, there's been a lot of progress with reasonably accurate levels or at the level of saying this explanation will be x percent accurate Y percent of the time. It's certainly something that is possible now for most types of machine learning models, particularly in the space that we are not just Truera's own technique called qii, but that whole game theory based approach to explanations. They're reasonably powerful and they've been seen to work. Particularly if you the science is not behind the explanation of, sorry, the implementation of science is trickier. How you implement those explanation techniques, which can be resource intensive, etc. Is more complex than the science itself. The science is kind of reasonably mature now. Now there is still some work needed when it comes to deep neural networks and we at Truera have actually recognized this and we've created an open source initiative called True Lens which takes it's kind of truelens.org it's not part of the Truera product at all. It takes what is there in open source today, adds our own and kind of. We've just kicked off. It's got some fantastic reviews in the last 10 days precisely. People recognize that in deep neural networks there's more work to be done. So we are opening it up. It's not a commercial product but I would argue that the area where we need the most attention because we did some review of this on behalf of MAS led consortium here in Singapore, is actually the form of the explanation that is presented to the end user. So whether it is you or me as a customer or whether it is the vast majority of people in a bank or insurer who are not deep data science folks, just giving them the raw explanations is useless. You have to find a way in which it resonates for them. And when we look to the research in that, which is, let's call it the human machine interface, for want of a better word, there is certainly more work to be done both in terms of how you explain a decision to an end customer that makes a difference. But something as simple as oh, there's a human in the loop for a decision, so we shouldn't worry about it because every, every AI decision is a human being is reviewing it and deciding whether to accept it or not. Well, has somebody looked at the psychological effect of if someone continuously is pressing yes, yes, yes, yes to the, to the. To the algorithm's recommendations then the one time it should say no, it probably overlooks it because he's seen the last 99 cases where all algorithm was right. The human got it wrong the hundredth time when they should have intervened. Now this is not a theoretical point, as you know some of the issues with the Tesla self driving car. I mean exactly that the human has kind of assumed that the. So this area which is at the. It's not so much directly the science of the underlying explainability, but it's the form that that explanation and that human machine interface works, I would say that needs more work than the underlying science. Except in deep neural networks where I think there's more work to be done.

This is a great overview, especially when you mention the packaging of explainability. Do you find that there are categories of AI or machine learning use cases where trustworthiness and interpretability are more important than others within financial services? If so, what are the different measures organizations need to apply from a trust or risk Perspective based on these different levels of risk.

I think your first question is easier to answer than the second one because one of the things around the current state of regulation and regulatory initiatives is people have not yet gotten around to creating standards for. I will use X for Y category yet. That's one of the areas where I think financial institutions need to do more work over time. But on your first question, yes, there are very clearly areas. Not every area is equally risky when it comes to trustworthiness and interpretability. So the ones that tend to have the maximum attention are those that result in somebody being denied a service of any kind and particularly in this instance of course, a financial service. Right, so what could this be? Well, you're not getting an insurance premium, sorry, an insurance package cover, or getting an unacceptably high rate because of AI not behaving well or because of some question about the AI or similarly not getting a loan or even not getting accepted as a customer because your KYC risk was seen to be too high. So anywhere when you are in a position of being denied access, access to a financial service is probably the highest, right. Next highest would be where there is an opportunity for, sorry, there's a. There is room for you to have a poorer experience or a poorer outcome than some other group. That is discrimination. But the first one is not discrimination, it is absolute lack of access. So if an algorithm could be used to, for example, deny someone the right to come and collect money from their own account in an atm, that is facial recognition, then you better have very good, you know, safeguards there where the person has an alternative right because otherwise they're just not getting access. Never mind discrimination, I'm just not getting access. So I would say right at the top is lack denial of, risk of denial of valid access. Next would be risk of discrimination, unfair discrimination between parties. I think another one that is very high is where you have to prove to a regulator that you have done the right thing. Now when you have to prove, most famously in financial crime and you know, anti money laundering sanctions related stuff as well as, let's say, insider trade behavior, etc. This is not about customers, but you have to prove the onus is on you to show that you've done the right thing in terms of detecting and catching and investigating, if you can't prove it to the regulator in this case, not to the staff member or the customer, but to the regulator or to your internal risk management teams, that becomes difficult as well. So these, I would say are quite easily things that you have to care about because you're impacting a customer or it's impacting your direct regulatory obligations. Now the tricky ones are the ones which don't fall in these areas. Right. So I was really interested in, in how the Chinese regulator, not financial services regulator, but they came up over the weekend or on Friday, I think with a bunch of regulations on what they call algorithmic recommendation systems. And it's not specific to financial services, but normally we don't think of. Sorry, there was this and there was the SEC thing in the US about Robinhood style online investment tools and both of them. It struck an important thing in my head, which is normally I would treat marketing use cases. Well, what's the harm it can do? Worst case, it will sell something to you you don't want to buy. Fine, go away. It's fine, it's not a big problem. But actually you can of course Ms. Sell. You can inappropriately target a financial product at somebody who should not have been sold that product in the first instance. Imagine selling a very high interest loan to somebody who the machine has worked out is desperate for money. Is that fair? Is that ethical? Right. It's not nothing to do with denial of service or discrimination, but it's just wrong to offer that person that loan. You might want to offer them some other kind of help, but not a high interest loan to address that.

It's predatory.

Yeah, it's predatory, exactly. So easy ones are the ones I mentioned at the start. Denial of service discrimination, explicit regulatory. I think others where it's kind of, well, I'm offering a service, but actually where is the line between offering a service and offering something inappropriate or predatory? That is a trickier one, but I think most banks are working that out. So one of the first things most banks and insurers are doing is coming up with a materiality framework with a bunch of conditions. Now to your second part of the question. Have we got to a stage where people have said for this level, we need this explainability for this level? I haven't seen it yet. I think that's part of work in progress.

I think for a lot of these use cases, having a human in the loop will also be very important for this decision making. It's all about packaging that information as well. And this is a great segue to discuss Truera since it packages explainability and interpretability very, very well. Do you mind sharing how Truera solves some of these fundamental issues you're discussing?

Yeah, sure. And thanks again for the opportunity. So, two things. First of all, we just do two things. We provide software that does one, it allows somebody who's just built a model to assess that the model's quality, the quality of that machine learning model. Now, when I say quality, traditionally, as you know, data scientists often and their business stakeholders will often think of quality as accuracy against test data. But we're talking about that. But also a whole host of other things, such as the potential for unjust bias, such as overfitting instability, and the comparison with other models to show that it's not just about how accurate is it against the train and test population, but also what are the known weaknesses of machine learning models around overfitting instability, unjust bias, et cetera. And have we checked for that? That is like call it a diagnostic tool. Now, its primary purpose has indeed in financial services very much been about not doing the wrong thing. But actually when you use it, it also helps you do the right thing, if you see what I mean. I mean, no one goes out and says I want to build an overfitted model or I want to build an unstable model. So actually, even if you improve the quality for defensive purposes, in reality you're improving it for just for the sake of it. You're actually improving the quality of the model. So that's one part, it's like a model diagnostic kind of tool. The other is monitoring. So once you've gone live with the model, we allow you to monitor the output as well as the input into the model in a way that there's a meaningful connection. Meaning when you see a model drift or data drift, you're able to quickly go back to that diagnostic module and see whether this is a material drift and if so, what is causing it. So that connection between monitoring back to the diagnostic is quite important in terms of how we are deployed. We are not an end to end MLOps. So you don't develop models on Truera, you don't use Truera to train models, you don't use Truera to deploy models. And that is intentional because our whole proposition is it doesn't matter whether you used platform X or Y or Z or you just did it out of a notebook or you bought a model from somebody. It doesn't matter how that model came about, we just need access, or rather you just need access to the model's output, like a pickle file or something like that, a serializable output and the training and test data. And with that trueer, I will be able to give you all these AI quality diagnostics and monitoring for that matter, as long as you have access to that model. So this is quite important because it means that you're able to for example, compare an in house model with something you bought from outside or a model that one team has built using one platform and another has built using another platform. So if you want to build consistency and standardization over across an organization, short of forcing everybody to build models on the same platform, this is a good way of kind of ensuring some degree of consistency. Right. So that's how we work and we work on the client's premises largely at this point at least certainly for large enterprise clients, meaning either on premise or on the client's own cloud environment, not it's not as a service. And that is intentional because for the kind of clients we are talking about right now at enterprise level, nobody's willing to share confidential IP like models and data with us. So it's entirely on the client side. Nobody from Truera will ever see what you are doing with the software.

It's very exciting to see the evolution of the software stack around explainability evolve. I'd love to pivot to discuss the future of data science and financial services with you and how it intersects with explainability. Right. How do you view the industry for example, accommodating large language models such as GPT3, especially given its black box nature and the fact fact that from a packaging perspective it's ready to use plug and play API rather than a traditional machine learning model that follows the fit predict paradigm, for example.

So I haven't come across financial services client already using GPT3 obviously, but there are examples similar to it. For example, facial recognition models are used not for emotion recognition, but certainly for authentication of identity, etc. And I would say there's a whole range of techniques. At one extreme, the lowest bar extreme would be, well, I've got the testing results of the person who's built the model and created it and I can see that they had X percent accuracy, X Y percent, X percent precision, Y percent recall and I'm happy with those kind of odds. Right. So that's kind of don't need explainability because I'm happy with the guaranteed performance. And why am I happy with that? Because particularly for facial recognition, as you know, there are kind of regular tests done with NIST and so on where you are able to get that kind of information. So, so that's one level of explainability where you understand how the model works and you have a very good understanding of the training and test data as well as the most recent performance of that model out in the wild, so to say. So you could accept that. And some use cases is accepted, often with significant human intervention. So in this case you might say, well, it's okay, but we want 100% of the refused identifications to go to a human being straight away so that no customer is unfairly. Even if it's 0.1%, I don't want any customer to be unfairly rejected. So whenever you're rejected, it goes to a human being. That's one way of addressing it. That's back to the human intervention piece. The next level would be to say, actually, no, this is not enough. I want the provider to at least provide me a level of generic explanations on how the model is. Sorry, specific explanation, how the model is working. Maybe not for my client's data, but for the data you've trained on. Now, this is where some of the deep neural network kind of explanation comes in, where you're not saying, okay, for XYZ bank, this is how the model is working. You're saying as a whole, this is how GPT3 or this particular facial recognition model is working. Now, I do think this will become important over time because, you know, while I was testing this at one of the banks or while somebody was testing this at one of the banks where I became aware of it, the model providers explicitly said, if you wear a mask, the model will not work. Which in the middle of COVID was a, was a relief because, you know, if it started working with mask and you know, without proper authentication, it would be a problem. And lo and behold, somebody found a picture of somebody being authenticated with a mask. Right. The model provider had no idea how that happened. This points to the fact that simply depending on past test and training data will not be enough. People might be forcing you to say, okay, you don't have to explain to me how GPT3 or this facial recognition model will work in the context of my bank or my insurer. But you do need to give me a reasonable level of explanation more broadly. Right. And that leads me to the third level. I think the only way this kind of model can be governed safely is, frankly, if you think there are precedents of this in banking, for sure. If you think about a FICO model in the US Credit model or even some of these big Moody's and so on. A bank or insurer or investor doesn't go and challenge every single Moody's rating. What does happen is that a regulator is checking whether Moody's is doing his job. So I do think that might be A sustainable way where you say if a model is used very broadly across the industry, right, whatever industry that is, then that industry's regulator might regulate that model. So in other words, the model provider will privately with the industry's regulator, explain their model and continuously convince the regulator saying, yep, you can depend on me. Right. That way every bank and insurer and FinTech doesn't have to individually check, if you see what I mean. So I can see these three kind of ways of handling it. Most of the current usage is certainly in the first category I mentioned.

You know, given emerging explainability, techniques and methods and innovation in this space, what do you think are machine learning or AI use cases in financial services that will be operationalized tomorrow that we cannot operationalize today?

I think it will be a matter of scale rather than completely new things. Right. I mean there are people who are making credit decisions using machine learning, maybe in a challenger mode, maybe in future they'll not use the challenger mode. Right. Or facial recognition becomes more standard, not for emotion recognition, but for authentication. So I think you will see if you take one or two years, I think the difference will not be kind of brand new use cases, it will be much more of, you know, how deeply it is used without a lot of manual intervention that I think is going to be seen. So some of the so called higher risk areas might become acceptable risk, whether it's credit or some aspects of pricing and underwriting support and insurance, etc. I think there are two or three other interesting topics which I do see getting evolving over time. If you want me to do a bit of crystal gazing, I think one is there are some very hard problems to crack that are coming up in financial services. One of them is frankly actually just dealing with data and data rights. When you have so many different parties collaborating, right, different parties in the third party telecom retailer, search engine and E commerce, being able to get some assurance that the data is not being misused, that itself is a big challenge. And I think that is a bigger problem to solve than algorithmic problems. And to that extent I think there will be progress there. And then the other big nut to crack is of course esg and in particular the environmental obligations of banks and insurers. It is a frightfully difficult thing to do right now. And I do think whether it's AI, but more it's like the role of data and big data in the truest sense of the word, to really ascertain that this piece of financing we are doing is green, right, or to ascertain the non greenness of an existing investment, for example, in a way that is realistic and is verifiable, that is probably going to be one of the biggest opportunities. This is not one to two years, but I would say if I were to put money into something, I would put into that companies that are making the use of data and AI quite fundamental to how the financial services obligations around environment are met and around individuals data rights are met. Those would probably be areas. But in the use cases I talked about I would say it's more a question of higher scale rather than brand new use cases.

And from a skills perspective, what do you think will be the most essential skills for financial services professionals and a future where AI and analytics are operationalized and are part of the daily workflow. And what would be your advice for up and coming data scientists fielding choices between for example joining tech or finance here?

So let me do the last latter because it's certainly an area close to my heart. Look, there are many industries where certainly if you're starting with a tech first company, yeah, you can do cool stuff with search engines and E commerce, recommendation engines, etc. But a you're getting to play with a very different kind of data and a lot of data which is more substantive in some ways than your day to day behavior on social media. Because you do lots of things on social media based on emotion and it might be something transient but you know your financial history over 10 years is over 10 years. It tells you a story of what you've been. And so if you say a data scientist really is interested about how to use data for better decisions, the richness of the data in financial services is far more. But more importantly I think, and you can argue this for financial services as well as for some other sectors like healthcare and maybe environment and so on, the value of what you will do with that data science with that machine learning model is arguably much higher. Right? I mean it's fantastic to have a great translation tool. It will solve lots of pain areas for lots of people. And I think in the case of one, I think it was a Belarusian athlete who used Google Translate to find her way out of Tokyo airport. It can even save lives perhaps. But for the most part a lot of that is about making life easier versus cracking fundamental problems like financial inclusion, etc. So you'll also be solving bigger problems in financial services. That's why I would say it's fascinating. You should give it a go. However, now coming to another question on what skill sets? So I'll answer the question in two parts. What would I suggest from a skill sets perspective to data specialists? Learn about your sector, whether it's healthcare or whether it is financial services or transport or whatever it is. Because anyone who believes that data alone will rule without context is missing it. And I would strongly recommend them reading the COVID 19 experience, for example. So being able to be someone who understands the broader context and knows how to work with, let's say, an actuary in an insurance or a marketing person or a credit person in a bank, etc. That's super helpful. So that would be my advice to the data specialist, to the rest of the bank or insurer or fintech people. I only have one advice. Every aspect, one piece of advice. Every aspect of what you do in a bank insurer on fintech is going to be impacted, right? Some of you might be young enough or curious enough or smart enough to rebuild your careers as a data specialist, but most of you will not, and you might not even want to. You might just like being a marketing person or a learning person or whatever you want to do. You absolutely need, you don't need to become a data scientist yourself, but you need to absolutely become sufficiently smart about the topic, sufficiently data and AI literate to ask the right skeptical questions. You can't ask the right questions, then you're dead right. So I would say build up to a stage where you can ask the smart questions. Don't try and I know how to code.

It's good.

If you know how to code, it's great. It's probably more important to understand how that code works and how data is used to train models than to actually be able to code yourself. So get to the place where you can ask the right questions would be my suggestion.

That is awesome. Finally, Shamik, given that we're ending on such an inspirational note, do you have any final call to action before we wrap up today?

Yeah, this is for the financial services sector itself, I think. Look, as we discussed, there are some real interesting opportunities on the horizon for financial services when it comes to using analytics and AI data and machine learning. So more broadly, including on the environmental side that we discussed, there's also a very real risk of a AI winter again coming up. Certainly in financial services, I mean, people have put in billions of dollars, certainly hundreds of millions of dollars in many major banks and insurers. And I think if we don't watch it and if we just keep focusing on how many new marketing slogans can we generate and not actually seeing whether it's making a difference there's a real risk that this will become yet another AI winter. So I would say let's be cognizant of the opportunity and let's focus on making real difference with machine learning and with data and analytics, rather than going for the hype.

That's great. Thank you so much, Shamik, for the insight. I really appreciate it.

Thank you.

That's it for today's episode of Data Framed. Thanks for being with us. I really enjoyed Shamik's insights on the state of AI adoption in the financial services services industry. If you enjoyed this podcast, make sure to leave a review on itunes. Our next episode will be with Brian Campbell, head of Engineering at lucidchart, on managing data science projects effectively. I hope it'll be useful for you.

And we'll catch you next time on Dataframe.