EP 161: Product Strategy in the Age of AI

This is the Everyday AI show, the everyday podcast where we simplify AI and bring its power to your fingertips. Listen daily for practical advice to boost your career, business and everyday life.

How can we create better AI that's really centered around users? You know, so often it's evident everywhere you look and turn and listen and watch and feel there's, there's AI literally everywhere. But what does it mean for users? And how can we create better products that have AI in them and that we can explain? And that makes sense not only to the users, but also to everyone that's benefiting from these products and services as well. That's what we're going to be diving into today on folks, Everyday AI. Thank you for joining us. My name is Jordan Wilson. If you're new here, thanks for joining us. But Everyday AI is a daily live stream podcast and the free daily newsletter helping everyday people like you and I not just learn what's going on in the world of generative AI, but how we can actually leverage it all, how we can use it to, to grow our companies, to grow our careers, to get ahead, to outsmart the future together. Right? Like that's what we're all about at Everyday AI. This is technically, hey, technicalities, here we are debuting this show live. It is pre recorded. Not everyone can do the, you know, this 7:30am Central Standard Time. It doesn't always work and there's amazing guests out there who are doing fantastic things in the world of generative AI. So today is no different. So please with that wind up help me welcome to the show and please still get your comments in. We're still going to be, you know, responding to your comments if you do have questions. But with that, please help me welcome to the show. Svetlana Makarova, who is the AI group Product Manager at Mayo Clinic Split. Lana, thanks for joining us.

Thank you so much for having me.

Absolutely. So hey, tell everyone real quick a little bit about what you do as an AI group product manager at Mayo Clinic.

So as an AI group product manager at Mayo Clinic, what my role is is considered the player and the coach. So I do have, I lead the team of product manager owners, you know, delivery leads and development teams. But I also implement AI solutions on my own. So I actually have development teams that I lead. I develop product strategies for that utilize AI and it's not anything specific per se. So I have experience working with deep learning, machine learning, natural language processing, generative AI, you name it. So yeah, I think that's basically what I do and I'M happy to dive into it a little bit more.

Yeah, absolutely. We like, we're like a minute in already and we've already dropped so many acronyms. So let's, let's hit rewind a little bit. So maybe explain if people aren't super familiar even like, what is AI product management? Right. Like, are you helping create products and then integrating AI into them? Is there already an AI kind of algorithm or an AI, you know, deep learning model and you're trying to bring it to market or you can even just speak in general? Because I know we can't always talk about everything people are working on behind the scenes, but what does that even mean, just AI product management?

Yes. So I think it's, it's depending on the use case. So working on existing systems and then making them more intelligent or, you know, have experience working from complete concept and then taking that all the way to market. So really depends again on the use case and how you would approach it. But it always starts with kind of the strategy aspect of it. And this is where I'm most involved, you know, trying to discover kind of what are the needs, what are the problems to solve. Is AI even the best solution for that specific use case? Not always. And so I think, you know, some of the standard product management practices I think still are at play here. The only thing that changes is that you. I have an expanded tool set is what I call it. You know, I have just more tools under my belt that I have experience with implementing now. I kind of, I understand and I have an eye on for eye out for which product could benefit from the efficiencies that AI could bring. Finding particular potential use cases. Right. So understanding kind of from interviews and things like that, where AI could really bring those efficiencies into, in our case, the practice, the clinical practice, the research and then the education.

You know, as someone that both uses AI and helps build it into products, I'm curious because I don't build a lot of AI, you know, little, you know, toying around with simple stuff here and there. But is there too much, right, Is there too much AI in products? It seems like every single product out there, hardware, software, you know, there's, there's generative AI in it for some reason. Like, is there too much AI out there in products right now?

I do think so. It is. You know, a lot of companies are writing the hype quite heavily. I think generative AI, AI in general is, is basically a buzzword anywhere you throw that in. It's, it basically embellishes every product. So. And I think OpenAI has made it much more easier to bring in into digital products. So for, and I want to caveat that for I think small companies or companies that are selling, you know, quite, you know, streamlined products. Right. So things that are like automation tools, you know, being able to provide summaries and things like that. But for enterprises, I think there's still a lot of challenges of bringing AI technologies because of privacy, data security and other ethical considerations for why you'd want to go about a little bit more carefully. So I think B2C products and things that you and I are much more exposed to on this platform and I think elsewhere. Of course, you know, it's a buzzword, but I think enterprises are still encountering issues with scaling efforts. I think costs and things like that to be able to implement at that scale. So. Yeah, but nonetheless, I think it is, it is sprinkled throughout all of the products at this point. For sure. It's a busy place out there.

Yeah. And it seems like maybe if, and I'm sure there's other factors that are tugging at big companies or product managers to maybe implement AI when they maybe don't need it. Maybe it's because they have to raise funds or maybe, you know, users are just, you know, demanding it in small numbers. But I think maybe if, or do you think if product managers thought more about the user centric approach, do you think that that might allow us to more sparingly or more effectively implement AI into products? Because yeah, I feel overwhelmed because I love AI, I love using it, I talk about it every day, but there's so many things out there. I'm just like, we don't need AI in that.

Oh, absolutely. And I think that's where user centric AI kind of comes in. It's, it's basically a user centered approach of developing products. And I think again, it's not unique to AI specifically. It's just a concept for making sure that whatever you're developing, a solution that you're building is centered around user needs and you're not building or you're bringing this technology for the sake of saying, hey, this tool is powered by AI. You're really looking to the user. Is it helping this product? Is it helping to solve that particular need of that user? Whether it's AI, whether it's a rule based engine, it doesn't really matter. But to the user, that workflow should seem seamless. Right. So if you're introducing AI and it's a new place For a person to access, to be able to get benefit of your solution, you're doing it wrong. So I think a key part of user centric AI is being able to bring these solutions that are embedded into the workflow. So folks should not be noticing like, okay, well now you're entering the space of AI to land and you have to, you know, click this button or interact with the solution a different way. How do you truly embed it in a way that is almost invisible to the user. Right. And one example that I can bring is, you know, Google, right? So as Google evolved over the past decades, you know, they've brought more and more AI technologies into their tool set, right? So behind the scenes they continue to evolve, improve their algorithms, but to the user, they're still interacting with it in the same way, right? They're still typing on Google. But the difference is, is that as a result of those technologies, they're getting better searches, they're getting more accurate results. And so I think being user centric, you need to understand the needs of those users and then how do you deliver them in the most efficient way, that's the most fluid as possible. But then you have the other extreme where some of these AI technologies do get cluttered. And I think another example of maybe AI, in my opinion, not done so well as Amazon, right? Because I think when you go on Amazon, their system is so cluttered and I feel like they're probably running some large developmental teams that have certain components broken into separate teams. And so they're kind of doing their own thing with AI and then they're launching and testing. And so every day that you're coming onto the platform, something changes. And so you want to control for that and you want to make sure again, like is the person who came to your platform or to your product getting their task accomplished in a much more efficient way without having to leave their workflow, that platform or whatever have you. So that's user centric AI done, right?

You know, Svetlana, you brought up a great, great point that I hadn't even thought of yet. You know, when we talk about, and I love the, you know, that terminology that you kind of use, you know, that it's invisible to the user. So, you know, as these large companies, the biggest in the world, you know, they've been the one pushing, you know, AI for decades, but generative AI for the last couple of years, your Google, your Amazon, you know, now you're OpenAI, your Microsoft. I'm wondering if, you know, if we talk about user Centric AI and being kind of invisible, quote unquote to the end user, I guess what happens when the end users are now very use to generative AI everywhere? Right. So I'm almost going against what I just said 5 minutes ago, but you brought up a good point. You know, if we become accustomed to having, you know, essentially large language models that we talk to for everything, you know, you know, for our financial institution, for our insurance, for, you know, all over the board, then is it okay, is it then very pro user centric to incorporate, you know, AI into everything? I know I'm, I'm contradicting myself, but now I'm curious.

Yeah, so I think that there's. Generative AI is really great at certain things at this point in time and maybe not so great at other things. Right. So you may be able to get specific insights but, or complete creativity tasks and things like that, but there are certain other things that it's not yet equipped to do really well. So it works really well on unstructured data and to be able to, and I think that's the biggest use case for it to be able to provide kind of inside summarization tasks and things like that. But things like predictive analytics, right. So if you think of a use case such as again going back to Amazon, you know, real time predictive analytics. So if you're kind of searching the website and you know, you're shopping for something, you know how you have suggested items that you should probably look for, they're kind of looking at your engagement and what you're, you have a tendency to shop at and maybe your propensity to buy at that particular point and then they recommend things to you at that point. So things like that, LLMs would not be able to solve. So from a business standpoint, yes, from users, LLMs can serve specific functions. But businesses use AI too, right. And they need to meet their objectives. And so recommendation engines are there for a reason and those use machine learning recommendation systems too to be able to again surface those users at the right time. And so if you kind of are looking and there's a book that I'm forgetting what it's called, but they talk about kind of the evolution of AI even within the scope of Amazon. Right. So how can these predictive analytics inform some of the business decisions over time so that could benefit the users as an example, and then going into invisible question, going back to your invisible question. So I think over time you could predict basically based on your shopping behaviors, what items you would need in the future. So why do you even need to access Amazon to be able to have certain items sent to you. So you might just wake up in the morning and then you get like a box of coffee delivered at your door because probably chances are you're going to leave your house and then you run out of coffee and then you're going to go up there. So I think that's the beauty of data that Amazon is collecting and all of these data systems is to be able to predict future behavior. Again, LLMs will not be able to do that, but you need some infrastructures in place to be able to accommodate that kind of, you know, scale, basically predictive analytics, which is why it's not rolled out or as mass scale at this point. But that's where the trend is going is really towards that invisible how can you leverage more of that AI to predict certain behaviors? But as you mentioned, you know, some of these other tasks such as summarization, being able to retrieve specific information from documents, get quick prompt, prompt answers and things like that, I think that's also a tendency that we're going to see. But again I think there are different use cases but the tendency going to embedded user centric is where the trend is going with AI.

Are you still running in circles trying to figure out how to actually grow your business with AI? Maybe your company has been tinkering with large language models for a year or more, but can't really get traction to find ROI on Genai. Hey, this is Jordan Wilson, host of this very podcast. Companies like Adobe, Microsoft and Nvidia have partnered with us because they trust our expertise in educating the masses around generative AI to get ahead of and some of the most innovative companies in the country hire us to help with their AI strategy and to train hundreds of their employees on how to use Gen AI. So whether you're looking for ChatGPT training for thousands or just need help building your front end AI strategy, you can partner with us too. Just like some of the biggest companies in the world do. Go to your everydayai.com partner to get in contact with our team or you can just click on the partner section of our website will help you stop running in those AI circles and help get your team ahead and build a straight path to ROI on gen AI. Yeah, and you know, kind of like we've already talked about the, the trend with AI is it is making its way into all of these products. So maybe let's even hit rewind on that. Right? Because you know, if, if you're a large company or maybe you're a decision maker, right? A business leader who's making decisions company and you're figuring out, you know, how, maybe how to implement, you know, a solid product strategy into your product or your service offering. What's, what's the best way to go about that process? Right, because we haven't even gone into, you know, explainable AI and all of those things. But how, you know, as a business leader if, if you are in the seat where you have to say, all right, our, our customers, our clients, our needing, you know, some, some sort of generative AI, you know, to help us make sense of this unstructured data and to, you know, have a better, you know, consumer customer experience. How do business leaders go about making the right decision on putting the right type of AI into their product at the right time in the right place? It's not easy.

Yeah, So I think it depends on. So if it's an organization that has not embraced AI, I think it depends. I think it starts with a first use case. Whether you want it customer facing or you want it internally facing, I think it's a business decision that you'll have to make. So where can you provide the biggest ROI behind that investment? Because bottom line, implementing AI is not cheap. So you want to make sure that you kind of de riskify your first implementation of AI or basically that use case as much as you can. So you know, probably makes sense to start with an internally facing product that is focus on streamlining specific tasks. So are there repetitive tasks that you know, the teams consistently do across different verticals that you could provide some efficiency. So being able to provide those efficiencies over not having them, so something is better than not having not having it. So being able to implement a solution that provides 60% or 40% over those efficiencies is still a win. So I think kind of lowering your expectations for what that use, you know, that first use case is and then seeing is there ROI behind that investment? I think starting with that, you know, first use case and understanding. Okay, here are the business objectives. And then being able to measure again that efficiencies and that ROI during that pilot and then seeing does it make for, make sense for us to scale and then try out addition, you know, new use cases. And I think part of that strategic approach that I've spoken about on my page too is I think you really need three pieces to be able to scale AI in the enterprise. You need data. All AI is heavily dependent on data. So you need to democratize access to that data, you need to take a platform approach for developing AI applications. And what that means is instead of building every machine learning generative AI solution like Rag LLM solution in your enterprise, you would find platform use cases, basically reusable use cases that are applicable across different verticals. So you get the solution developed up to a point and then you customize it to a unique use case. So if you have a need for recommendation engine or some predictive analytics, you know, I'm sure that there's multiple use cases of IT across the enterprise. So you do it once, but then you customize it across different verticals. And you know, number three is the infrastructure. So you need flexible infrastructure that allows you to be able to experiment with the technologies by bringing and really testing and validating iterating quickly. And part of that approach is being able to develop in a way that is modular. And what modular means is that the pace at which AI is evolving right now, right, there's llama2 and then there's matpalm2 that just kind of is, is released. There's new models that are basically popping out the modular infrastructure. Basically your development aspect of the solution needs to be able to swap out some of these components to be able to say, okay, well this model no longer works. Instead of me starting from scratch, I need to just take that module out and then put a new one in and then still have the entire solution work from beginning to end. I think those are the really the three core components from being able to identify the first use case and then really scaling it through the enterprise.

Yeah, and I love what you said there. And this is something I talk about all the time because, you know, individuals, companies, you know, everyone saying, where do we start with AI? And it seems like most people, I think, make the mistake of they look at the platform first or they look at what everyone else is doing and they try to follow their lead. But I love what you said. It's, you have to see where you're doing that repeatable, you know, almost, you know, sometimes mundane work across verticals. And so it's a great point that you brought up that I just want to really hammer home to the audience is, you know, focus on where you're spending the most repeatable time doing that, that manual work. That has data too, right? That has data. But I do want to ask you, like, how important is it to be able to explain it, right? To be able to explain what happens inside of the AI black box? You know, both, both before you kind of go through that three step process that you just laid out for us, but also on the back end, right. And to be able to kind of say, hey, here's the impact, how important is that and how do you go about doing it?

Yeah, and I think that's a, it's a great question. I think it's at the core of being able to truly practice user centered AI is being able to explain how the engine or basically that solution really works end to end. So explainable AI basically opens up the black box and shows the users, hey, this is how the engine or whatever AI came up with the recommendation or the way that it did. So you'll notice that Bard and chatgpt started to include references. And so one of the purposes are kind of the needs that it's solving is being able to kind of build trust.

Right?

People are not trusting these systems because they don't know where that data came from. So being able to surface evidence back to the user for this is the data that went into the system and this is how the machine kind of weighted those signals of that data. And then here are the recommendations and this is why this was a better recommendation than the other. And then so again, depending on the type of system that you're implementing, there's different ways of being able to surface that. And then you invite in kind of feedback, right? So again, going back to OpenAI's ChatGPT example, you have the thumbs up, thumbs down. So was the answer valid? Did that build trust or do people find it helpful or useful? Right. So you take that feedback and implement that back into your system and again it fine tunes and again you bring those results back to the user, then you really show them, open all of your cards and say this is what it is. Do you still feel like this isn't an accurate answer? And then you just go back and iterate. But I feel like that's really helped with implementation. I think rolling out of the solution. I think this is more of a user centric, like a UI piece where you have to really bring that evidence back to that user to instill trust in the results that AI is providing.

You know, such a good example too because if you're joining on the podcast, I was snickering, you know, a little bit as she's talking about that, because I remember during the earlier days, I'm like, there is no user centric it like in this AI originally it did take, you know, the big companies, you know, like Google, like OpenAI kind of a long time to start saying, hey, here's sources or even, hey, a simple thumbs up or thumbs down inside of chat GPT or sometimes you get, you know, two options. I guess maybe can you help explain? Because I know it's easier said than done, right? So from a product management perspective, I guess what goes into that decision, right, of, of how you go about, you know, creating a user centric product, what type of feedback you need, how you get that feedback, what you do with it. So you know, without going, you know, crazy like into too crazy of details like how does that process work and why do some, like why does it sometimes take a little bit longer? I feel at least to, you know, really see that user centric piece.

Yes. And I think again as I mentioned, I don't think it's any different being in AI land than any other kind of digital product is really investing that upfront time understanding the users, understanding their workflows. But as I think you've mentioned is also part of that discovery process is understanding the paper trail. Anything that you want to kind of automate with AI needs to have some trackable mechanism or some data behind it for the machine learning to then learn, let's say patterns from or to be able to use that data, to be able to mimic those tasks and really automate. So part of the discovery process is really again trying to automate and understand the intent. What is the user trying to do? It's not making those assumptions but like really putting those users in front of you and asking them, well, what are you trying to do with this? What kind of use cases are you trying to solve? And then you would invite as many of those users as you can and try to see what's the overlap really, what can, what efficiencies can I provide to those users? And in one of the. So for the paper trail. So as an example, we do have close to 100 specialties of Mayo Clinic. So everyone kind of does things slightly differently. Right. But during some of our discovery processes we've identified that there are certain tasks that again, certain groups were doing manually. We sat them down and we invited them for a conversation to understand, okay, well, what can we truly automate? What data overlaps really exists across the specialties. And we were able to leverage that. One of the other things that we do as we've implemented products, even when we go into production, we do weekly workshares. And so I think that's been a key in really practicing and not just talking the talk, but walking the walk. But instead of doing these sprint wide, which is typically like two week cycles we do these weekly where we put whatever we've done in that week in front of our users. So we have the folks who would be our target users of that solution really see the progress that we're making and then they provide us real time feedback. Are we in the right direction, are we completely off? Or you know, do we need to pivot? So before it even actually reaches production or a potential release into kind of the live production environment, we also have a mechanism to be able to again, like pressure test this with users to see if they still feel that whatever we're putting out into the market is valuable, is something that they could see or is it noise? Right. So it allows us to like, again, pressure test this on an ongoing basis. One of the other things that I highly recommend doing also is again, part of that user centric kind of methodology is friending your users. So do you have an easily accessible channel where you could phone a friend basically and say, hey, I need to just check this, validate this concept for me quickly. And I think you just need four to six users to be able to just validate quickly more conceptually whether it's something that's worth even pursuing from a strategy standpoint. So again, find ways to friend users being able to share progress and then be accepting of that feedback. Don't take it critically. I think your users are going to be the ones who are using your products so you don't develop things in the silo. Basically. If you can create checkpoints with your users along the way, I think that's the best way to implement some of these, these technologies.

Yes, please, please phone a friend. You know, get, get real human users involved. You know, I think there's also this, this rush toward, you know, like synthetic data and you know, these, these AI, synthetic user groups, which is like, all right, that's great and all, but yeah, at some point you, you, you have to talk to human users. So I'm glad you brought that up and all right, so we've, we've gone all over the place in a fun way. We've, we, we've explored, you know, creating a better product strategy. You know, everything from talking about, you know, structured and unstructured data and you know, talking about, you know, the right platform approach, so many other things. But split. Lana, what, you know, kind of as we wrap up, what's maybe the one big takeaway that you want to, you know, other people out there, whether they're, you know, decision makers, you know, trying to implement AI into their product, into their organ, into Their organization or maybe you know, people who are in your shoes, you know, those actually managing the products and building AI into it. What's the biggest takeaway or the best piece of advice that you can give everyone?

Yeah, don't write the hype. So I feel like, you know, just because you have a buzzword in on just kind of in the market doesn't mean that you really need it in your business. So I do see it a lot, I hear about it a lot that hey, I need AI in my business. Where does it fit? You think it's, it's a, it's a jargon or people just kind of make this up. But I've heard this myself trying to fit the technology into specific use cases. I need IT in my business. Well, when you ask them, well what do you need it for? What do you think that it could provide from a business standpoint or what value could it bring to your users? Well, I don't know, I just need AI because it's a cool thing. It's the coolest thing on the block. So I feel like you kind of have to pause and figure out what value can I provide and what solution can help solve for that. And again, the answer is not always AI. Sometimes you can find more efficient ways of solving for a particular problem. One thing I was just brainstorming more recently was about does this use case need generative AI or could we build a much more simpler, maybe a machine learning algorithm or some much more streamlined technology than maybe a rule based engine? And if you think about it, even from a compute storage and just efficiencies cost, like the time that it takes for you to complete for that engine or agent to complete that task is the difference of again starting up a motorcycle versus starting up a boat, right? So you may, you may not need to access an entire large language model to be able to complete that task. Sometimes a motorcycle type of engine, just a small thing would do because you know the type of task that your solution needs may not require that much data and does not require that much sophistication. So I think you have to go case by case basis and really evaluate different solution approaches. Don't write the hype. Just because generative AI is the coolest kid on the blog doesn't mean that you need it.

I love that, I love that because people are always, you know, trying to get on the big yacht, but you just might need the motorcycle. That's such a good point. Or hey, or maybe the electric scooter. It could, could even be right. There we go. Hey, so thank you, Svetlana, so much for joining the Everyday AI show. We really appreciate your time and helping us really dive in to everything that that that is going on in AI product strategy. Thank you so much for joining us.

Thank you so much for having me.

All right. And hey, as a reminder, we still have the news. If you're looking for the news, we still have it. Make sure to go to your everydayai.com, sign up for the free daily newsletter, and we'll be back live again, don't you worry. Thank you for joining us, and we hope to see you back for another episode of Everyday AI. Thanks, y'. All.

Thank you. And that's a wrap for today's edition of Everyday AI. Thanks for joining us. If you enjoyed this episode, please subscribe and leave us a rating. It helps keep us going for a little more AI magic. Visit your everydayai.com and sign up to our daily newsletter so you don't get left behind. Go break some barriers and we'll see you next time.