Prioritizing AI/ML Projects in Your Organization

Sam.

Foreign.

Hi everyone. Welcome to Prioritizing AI and ML Projects in your organization. A live interactive event collaboration between 4th Brain and Deep Learning AI. Backed by Andrew Ng's AI Fund, 4th Brain helps you take your ML career to the next level. Our cohort based courses are designed to help you fill your ML skills gaps with confidence alongside other practitioners looking to advance their career in AI. I'm excited to share that 4th Brain is offering a special discount of $500 off either our MLOps or our machine Learning Engineer program for anyone that registered for this event. It's amazing to see viewers from all over the world joining us. Let's keep learning as a community together today and share a little bit about ourselves. To kick things off, I'll start. I'm Greg Lochnane, the Director of Product and curriculum at 4th brain and I live in San Francisco. What about you? What do you do? Where are you calling in from? Let's see some stuff in the chat. Keep the intros coming wherever you are. As an ML practitioner, you know that the state of AI is always evolving. Just like every company out there is trying to build AI first products. Today we're going to learn how two of the world's largest and most successful organizations make decisions about what innovative AI products they should try to build and productionize. You'll hear about how Cisco comes up with new AI engineering initiatives inside their networking and telecommunications business. And you'll hear about how JP Morgan, the parent company of JP Morgan Chase, is moving towards an AI first mindset when it comes to financial services. You're likely to walk away from today's session with some tips and tricks for creating real value through ML and AI projects in your company. If you hear anything that resonates with you or makes you curious, please post your questions in our online discourse community. You know the drill. Just head on over to the Q and A page to to create a new topic. You can find the links along with instructions in the event description box. A quick note on process before we dive in. We'll hear from our first speaker, then from our second speaker, and then the three of us will have a fireside chat to see if we can synthesize any learnings. Then we'll open it up for Q and A. With that, I'd like to give a warm welcome to our first speaker of the day, Kumaran Ponambalam, Principal Engineer at Cisco, a big data enthusiast. You can find 31 of his courses on LinkedIn learning everything from Spark and Kafka to customer and business analytics Kumaran, I'd like to bring you up on stage. We'd love to hear from you about how you prioritize AI and ML projects at Cisco. Take it away.

Thanks Greg. So in the next, for the next 10 minutes I want, I'm going to be focusing on how we going to pick and prioritize MLA projects for the organization. For your organization. Just a brief introduction about myself. I'm an ML AI big data enthusiast, currently working as a principal engineer for AI in the Emerging Technologies and Incubation group at Cisco where we focus on the next generation of cutting edge technologies and solutions. I'm also an author on MLAI and big data on LinkedIn learning. So there is a lot of excitement around AAML in the past few years. There is a lot of hype around the groundbreaking potential of AI across multiple industries and segments. One side we have the exponential growth of core AAML technologies like deep learning is one of example, Deep learning, computer vision, text processing. There's a lot of exponential growth that has happened there. In parallel to that is that there is a rapidly growing ecosystem for serving machine learning, machine learning, tools, technologies, workflows. There's a lot of things happening here. And on the other side, every business wants to add AI to its products and services. So this is kind of a standard charter for a lot of big companies around the world. Also every engineer wants to build ML models and ML products. So there is also an interest on the technology side. But what we also see is that as everyone hurries to get into the AIML action, we are seeing a lot more failures in getting AIML all the way to production than successes. So why is this happening? What are the reasons that we are seeing these failures and how can we fix them? By prioritizing our projects correctly. So why do MLA projects fail? It first starts with organizational commitment from the business itself. First of all, there is a lack of understanding around what AI is, what exactly it can do and what it cannot do. Second, it is usually run as side projects like okay, let's go and give it a little try, let's hire a couple of data scientists and then see what we can do there. But that by itself doesn't lead to a proper product or service. There is limited investment, there is no focused investment. It is again run as, you know, somebody is a daytime engineer in another project and this is more like, you know, a weekend project. There are no well rounded teams. A lot of people think that, you know, you just invest in a couple of data scientists and you should be able to get it done. But then they do lack the support around those data scientists in terms of data engineering, ML ops, software engineering and a lot of those capabilities. There are no specific metrics and success criteria for planning and executing AI ML projects specifically that is those critical to ML and AI. And then the bigger issue is the training data needs not are not well understood at all. While yeah, everybody wants to build models, the biggest prerequisite for this is good training data and the availability of it for data centers. And that's also not well understood in terms of expectations, pretty similar to the organization commitments. What people need to understand that AI projects are not software development lifecycle projects. And you can't use the same exact principle processes and methodologies that you use for software engineering projects on AI. Second is that models won't be right all the time. Even when you build a model which is an accuracy of 95%, there are still cases where it will fail. And that simple thing is usually not understood. Whenever a model fails to predict something correctly, what you see is a JIRA ticket being raised saying that there is a defect that this use case was not addressed. The level understanding is that models are like software programs. That has to be right all the time. Also the other thing that happens is that just because you built an accurate model doesn't mean you have a successful product. There is a whole journey that happens from building a model to making it a product and then scaling it and deploying it and making it usable by the end users. AA projects can fail simply because the training data does not have the right signals to build an accurate model. So a projects can fail. It is not like somebody gives you a set of requirements and you just do encode it and build a product. That's not how AI works. And regular SDLC processes also won't work in AI because it is not a forward process. It is a more iterative experiment oriented process for AI as opposed to just building and delivering code and regular software development. So these kind of MIS expectations are also the reason why people expect and have unreasonable expectations on a projects. And that's one reason they don't deliver to those expectations. From a technical side, number one, the biggest issue is availability of training data. There is a lot of challenges for getting training data acquired for use by data centers because a lot of this data has to be real. It has to be acquired from production databases and production database have all kinds of access limit restrictions for that. There are also security and privacy issues which customers are worried about if their training data is to be used for Machine learning and we need to get explicit consent from the customers for using their data. And that runs into a lot of legal and contractual issues once we get the data. There is also the challenges of labeling, which a lot of people don't account for. A lot of data that you get is not properly labeled or not labeled at all. And you have to go through some manual labeling efforts to get the data labeled correctly. Also, the data that you use does not have proper use case coverage. And this is a challenge for both those who are building general A products across multiple industries. It's hard for you to get training data from each of the types of industries you want to support. You're usually starting up with one industry or a couple of data sets, building a model for that and then trying to use it for other use cases. And that's where a lot of failures also happen. And finally, there is no coherent data governance mechanism for training data. Training data is kind of just sitting there in files somewhere, just used in an ad hoc fashion by data scientists, is not properly cataloged, managed. So those also lead to a lot of issues in terms of proper machine learning processes. From a product point of view. It also is required to have yearly validation whether this solution is going to work or not. When we are planning for AI, there is no equivalent planning of how does this model that we build make it to an end to end solution? What are the supporting infrastructures needed around them in terms of services and UIs and stuff like that that is needed to make it a complete solution? What are the customer expectations? How does the customers expect this model to deliver? Then there is revenue and cost implications. A lot of the AI models fail because they do not create enough value for the amount of cost it takes to deliver them. A lot of them are built based on deep learning models, which definitely a lot of times need expensive resources like GPUs. But the value they deliver to the end customer and the revenue they're making may not match the cost of delivery. So that's also another concern. There are also alternatives that people don't look at there. There's a tendency to go and build everything in house while there are open source models, libraries available that do the same thing. And we also need to look at what the competition is doing in terms of how they are building these services and delivering these services. There's a lot of thing about, okay, it's excitement about trying to use a new without validation as to how that will be delivered as a product or service and generate revenues. So one of the big so organizational commitments and expectations. This need to be addressed at an executive level. And we're not going to get into that in this presentation, but what we're going to look at is about creating an AI ML project. Evaluation criteria how do we evaluate project ideas for aaml? Choose the right projects to work on so that it reduces risk of failure and increases the probability of success. So we're going to look at some the list of evaluation criteria that you should be using when you are evaluating whether to pursue AML project or not. So the first evaluation criteria would be business value in terms of okay, this idea that we are talking about what kind of revenue it will generate or what kind of saving it generates, what kind of business impact it generates. The second is what would be the cost of serving this. It's very hard to kind of predict both of them at the initial stages, but some kind of a rough cut estimate, some kind of risk modeling would help here to get a better picture here. There is also time to market as to how quickly can we build it? Is it going to take many months of resource effort or is it going to be a few months? That also is under the criteria. There is the build versus buy a decision other whether some of those technologies are available. Ethics consideration in terms of data, what we're using, what we are predicting, is it, you know, it is ethic, is it ethical, is it explainable? All those things also need to be looked at. Storage and management cost. Whenever there is data it needs to be properly cataloged and managed and there is again a cost associated with that. Also a third set of criteria is around building blocks. Now rarely do people build an AI solution right from scratch. So today we reuse something off the market and then build on top of it. So for the model, what are the kind of applications we are trying to build? Are there algorithms already available in the market as open source or even if you had to purchase it or do I have to build it all myself? Can I use pre trained models and libraries that are available or they're already available, I can reuse them more. So other products and services available. If you go to Google or aws, there's a lot of AI services available like sentiment analysis and text to speech and speech to text which you can leverage off the board and you don't need to have to go and build them. What kind of tools, techniques, automation capabilities are available to support your models? This is around the whole ML life cycle, ML ops, experiment, tracking, model management, serving capabilities, all of that. Also what is available, what does it take to integrate this model into your entire solution? Is there a path forward? Is there already a plug available so I can just go plug in my model? Or there is a lot of engineering effort needed on the solution side to use the model so that also need to be looked at as building blocks. And then the final set of criteria is around. The kind of people we have again people and resources is something again people just. We just talk about data scientists. Data scientists. But that's in a team of like if you have a team of 10 people working on an AML project, only two or three of them would be data scientists. The rest would be other categories like okay, you need data engineering folks to work on the data and be able to do feature engineering. We need software engineering folks to build APIs and services around this and make them scalable services and then have create Dockers and Kubernetes pipelines. You need MLOps and DevOps people to actually go deploy them, maintain them, manage them, manage the lifecycle of these services. Then we also need hardware especially we know that deep learning models may need GPUs for better performance. Those are expensive resources and whether we would have access to all of them. So this is the kind of evaluation criteria you want to look at. And this is again just a recommended list. You should be able to add more criteria based on your own use case and your own company. Once you have the evaluation criteria, how do you go through the project selection process? How do you go and select create a list of ML projects and choose the best ones to work on. We always start with your selection criteria you want to create. Have that list as a limited list, have it measurable, something that says yes or no. Or you can have a score to it somewhere that you can quantify how well a given project meets this criteria. It is also good to have weights for criteria or not all criteria are equal. You can have different weights for different criteria. That may again depend upon your use case and your company. The second is to collect a list of use cases, possible use cases. This is where I think people tend to go very narrow in terms of just ask the product manager to come up with use cases. But where we should go and look for use cases is go look at your customers, all your employees, look at what's happening in the industry to collect what are all the possible use cases that we can focus on. You can also conduct some tournaments within your company called innovation tournaments, where you can look kind of have people recommend some solutions. And so the bigger the pool of use cases we can collect, the higher the chance of you hitting upon a gold mine. So the more the better. In the case of use cases, then you go through an evaluation and scoring process. Make sure that this process is rational and independent and you are able to model risk because a lot of times you won't even know the answers. How much revenue will this bring in? I don't know. How do you model that risk? And this should be like a multi step funnel, like very similar to how your sales and marketing funnel looks like. It starts with. So you go through a multi step evaluation like a business evaluation, technical evaluation and a resource evaluation. So it kind of narrows down to a small set of projects. Now the one thing I recommend is not to go for. Just find this one project and fully focus on just that one project. It should be more like how you maintain a stock foot portfolio is you create a portfolio with multiple projects. Because a projects can fail. You don't want to just invest everything in one project. So have multiple projects. Choose these projects across multiple horizons. Like you know, near horizon like next six months, medium horizontal like six to 18 months and a long horizon like beyond 18 months. So there may be some projects that are low hanging fruit based on the evaluation you do, which again based on the amount of effort needed to execute and time to market. And some may be really strategic ones but you know, you may not have all the technologies readily available for you to execute on that. So there you have to go back, you know, and wait for a longer time, do some research. So it's good to have a portfolio and make sure you front end, track each of them independently and finally expect failures. Two of three projects will fail. That's one reason why you want a portfolio. How do you ensure success? First of all, create metrics for tracking and these should be model metrics, product metrics and adoption metrics. Model metrics, all the FN scores and those kind of stuff. Product metrics is like latency, accuracy, those kind of targets and adoption is how well the customers are using it and set these expectations correctly with executives in terms of goals, risks, dependencies, timelines. Review the progress and results periodically and have these decision gates where you can make no go decisions. If an ML project is just going around in circles without making any progress on the model, maybe it's time to shell that project and move on to the next project. So we should be open to that. Do not expect every project to go through its full life cycle and do not expect every project to be successful. So we should be able to. That's why the portfolio approach helps here and key takeaways. AAML projects are unique. Organization commitment is key to success. A well defined selection criteria helps invest in the right projects. More ideas equals better projects. And of course make sure you track them and also act on them and make hard decisions when the time comes. That's it from me. Thank you guys.

Thank you so much for that great presentation. Kumaran, I wanted to kind of double click in on one of the things that you mentioned a few times, which was metrics. We have this difference when we're trying to build AI products between metrics that come out of your model and then metrics that actually are occurring at the business level. Your KPIs. Is there anything specific that you tell your team when they set out on a new project in terms of looking at specific metrics and planning to hit this sort of go, no go gate? You know, timelines are there metrics you aim at more generally than others and sort of what do you think about model accuracy as a metric?

So I see whenever a model goes through, it goes through three rounds of evolution before it becomes production. The first round of evolution is building a accurate model. Can the model actually predict what it is supposed to predict? In that stage you're focused a lot on model metrics like accuracy, you know, error rates and all that, those kind of stuff. Focused on how accurate the model is. The second thing happens when you're trying to then take this model and scale it up and trying to use it for, you know, a wider. No, trying to kind of use it for how do I handle thousand concurrent requests, you know, how do I general large volumes. Then you're focused on a different set of metrics which are more like, you know, latency, throughput, CPU usage, memory usage, cost. That's a different set of things you look at in the second stage once you pass that and say okay, we got these reasonable, then you then get it to a poc, to a customer. At that stage you're looking at more customer usage metrics. Maybe get some explicit feedback like thumbs up, thumbs down, something like that. Or measure based on the customer's next action. You know that if it is working or not. Let's say you're recommending a product, your customer, the customer clicks on the link. It means that yeah, it has been a good recommendation. That kind of metrics. There are different sets of metrics that you collect at different stages of your progress and that's what you use to kind of move the product along to the next stage.

Yeah, I'm going to come back to that in our Fireside Chat. Thank you so much again for your great presentation. Qumran. We'll see you back in just a little while. Next up, we're going to bring in Andrea Stefanucci. It's my pleasure to introduce him. Quick note to the audience, you know, David, Luis, Philippe, we are taking questions and if you, if you put them on discourse, we will answer them in the, in the Fireside Chat or in the Q and A afterwards. So if you put it in the YouTube Q&A, if you put it in discourse, we will get to it, we promise. So with that, we've got Andrea Stefanucci in the house. He is head of AI strategy and product management at JP Morgan AI Research. With over 15 years experience in the financial services industry working with some top firms including McKinsey, Bain, Accenture. He currently leads a growing team of over 80 AI researchers. So we're really excited to have Andrea in the house today. He's going to tell us about how JP Morgan deals with prioritizing AI and ML projects.

Andrea thank you Greg for the very kind introduction and thank you Kumaran for the great presentation. So we'll discuss more in the later Q and A regarding the different topics. But what I was hoping to do now, it's for the next 10 minutes, cover the topic how we're thinking about prioritizing AI ML projects at our firm, right? @ JP Morgan. I'll cover both initially like a use case specific view and then more like a long term view how we think about scaling AI and ML across the firm. So I'll skip this slide. Actually Greg did a great job introducing myself, so we'll skip this. What I wanted to going into the specific topic I wanted to cover initially, like the topic around impact of AI and ML projects, it's broader actually than AI and ML projects. So as we think about impact, fundamentally we think about two dimensions. There is a financial impact. There are the usual metrics around revenue generation or revenue protection, cost reduction, but in certain cases also cost avoidance, like can I do more with the same resources? Thanks to the ability of AI to help us scale our processes, our capabilities. Then there is also a component around risk reduction. Can we reduce frauds? Can we reduce the risk of operational errors of losses of various kind? So there is this financial component that's typically very prominent as we think about prioritization of projects. But not equally important if even more there is a component about non financial impact and metrics, they're a bit more difficult to quantify measures and there Are things like client experience, how we can delight clients through innovative AI capabilities, employee experience, how we can help employees move away from tedious tasks more to value added activities and have AI and ML help more with these more repetitive tasks. And then there is a component about reputation, credibility, both with clients, but also with a broader ecosystem. And finally there is also a big one, especially at the beginning in the AI and ML journey for a firm, it's around culture change. So fundamentally at the beginning, in many cases the business may be skeptical on what AI can do, what I cannot do, even not understanding really what are the AI capabilities. So really in the first phase of AI adoption, it's very important to get some AI successes like use cases where we're transforming parts of the business and showcases showcasing what I can do. And the impact may be from a financial point of view, may be limited, but the culture change impact may be very large and can then open up many more possibilities, right? It can get the business to think more in a more innovative way, unlike other AI opportunities that are larger. So then as we think about prioritization, so I covered in the previous slide the impact, the business value component. At the same time as Kumaran mentioned, there is another important driver dimension that's around cost, feasibility, risk. And here there are many components already Kumaran covered around data availability, data quality, AI talent. Do we have the right AI talent to cover specific use cases, engineering effort, infrastructure, cost, also like business adoption readiness, is the specific business unit ready to adopt this innovative capabilities. And then we go more into the risk area of the equation. So security, privacy, regulatory risk, reputational risk. So really like bringing all this together, we then have an assessment of cost, visibility and risk. And then as we look at use cases on these metrics, impact and feasibility, then fundamentally there are four key areas we can think about. There are like some quick wins where the feasibility, the cost is very low, the impact is high. So there are a bit like cases that are no brainer to really prioritize and execute. Then there are some projects that are much harder, they may have a much bigger impact and this they're riskier, but they're also very important in the overall portfolio. Then there are projects where really the costs and the impact are both low. We typically would not prioritize unless we don't have enough use cases on the top part. And then there is the no good zone where really the cost and the resources are necessary to do the use case. It's higher than the impact, so really the net impact will be negative. And echoing one point that Qumran mentioned, really we should think about a portfolio approach. So we should think about use cases across the board where often like the big bets, some of the big bets may fail, but then we have quick wins that can compensate in the short term the impact and really generate success and excitement in the organization. So this is around use case prioritization. Then in reality as we went through this journey in our firm, like we figured out that use cases are very important for AI adoption. But then however like in many cases, like individual use cases, they don't move that much the needle. In many cases it's really important to take a full business area slice of the business and trying to reinvent with AI and ML at the center, at the core. And there it's really where we see a multiplier effect like step change. So really eventually over time we moved away from like individual use case prioritization more to think which part of the business, which slice of the business we want to transform. And this, this theme was also highlighted in a recent Harvard Business Review article on getting it to scale. So it's really the direction of like how to move away from or like how to complement individual use case with like the scalability and, and like a multiplier impact about AI. So as we think about then this multi year journey and this shift from like establishing first an AI capability in AI team, then to get the scale, the overall impact, there are different building blots, building blots that we put in place over time or we thought about. So definitely there is a component about foundational enablers so establishing the right teams, the central AI and ML teams, the decentralized NML teams align with the different business units but at the same time creating cross cutting capabilities that are common across different transformation different use cases and can enable scale and prevent duplication of effort. Then there is also a big component around like setting up the right ecosystem, the right infrastructure, enabling for rapid experimentation and removing barriers to data access. So really like the component around infrastructure, data access and development environment to enable data scientists to be very productive and effective and efficient. And then clearly there is a component around attracting top AI talent. There is clearly a work for talent. So really this component, it's very important in the AI strategy for the firm how to get the right talent to address and to enable us to tackle the various use cases and the various opportunities. Then on the top part, as we think about the overall journey initially as we think about AI adoption in the firm, there is a big component around like education on AI to the business. Really in many cases there are a lot of opportunities out there in the business that AI can nicely target, but the business is not aware of what AI can do or cannot do. So on our side, we spent a lot of time to really educating the business on the possibilities of AI and also showing successful use cases to stimulate the imagination of the business. And then there is at the same time another component around like transparency in like AI adoption. Really like the journey to adopt AI. It's a multi year journey. So we need to know for a specific process or for a specific area, where did we start, where we are in the journey and what is the target once it's fully transformed with AI at the center. And then as we think about more the midterm view, really as I was mentioning, we were thinking about transforming food slices of the bank or of the firm with AI and eventually launch AI first products that would not be possible without AI and commercialized AI capabilities. So it goes from the initial phase around education, awareness, initial successful use cases, then to full transformation and commercialization. And then in the next couple of slides I'm going to cover in more detail a couple of these topics, the cross cutting capabilities and the end to end transformation. And then we'll wrap up. So in terms of capabilities, as we went through different use cases in our firm, we figured out that there are common capabilities that keep coming up across different use cases. There is really a possibility to establish common capabilities that can be reusable and they can make the next use case, the next project, much easier to target and therefore reduce the cost and the feasibility, increase the feasibility for the next use case. And here I listed in this slide some of the teams and some of the capabilities that emerge in our journey. Some of them may be specific more to the finance domain, but some others are more generic. For example, there is a big component in our space around intelligent client interaction. How we can personalize the offering to the clients, how we can really like engage, manage the communication to the client with the client to be smarter and identifying opportunities more and.