#248 Effective Product Management for AI | Marily Nika, Gen AI Product Lead at Google Assistant

And in traditional product management, it's okay, we're launching, we're getting hundreds or millions of users. Awesome. Figure out. Whereas in AI product management, success may be, hey, we made hypothesis A, we tested it out, we figured out, hey, that's not going to work. And we pivoted. And that hitting a mouse and of being able to get an answer to the hypothesis is.

Hi, Marilee, thank you for joining me on the show.

Hi, Richie, thank you so much.

Before we get into any of the AI stuff, let's talk about product management. So can you tell me at a high level, what does being a product manager involve?

Every single person you ask is going to give you a completely different definition. I have a little definition which is that a product manager helps their team and their company build and ship the right product. Product. And I focus a lot on the word right product, because that's the heart of it. We're figuring out the what we said, the vision, we set the mission for the team and we're trying to find product market fit.

So we'll maybe get into product market fit in a bit more depth later. But for now, can you tell me, is there anything unique to being an AI product manager that's different from regular product management?

Yes, absolutely. Absolutely. And it's interesting because when I first became a pm, I was an AI PM on day one. So I thought that all the challenges I was facing were challenges everyone else was facing. But, you know, I was doing it before. It was cool type of thing. So when you work with AI, there's just so many little things you need to consider. Number one, the probabilistic nature of AI. This means that every single time you use a feature, you're going to get kind of a different behavior. And when you're building a product that you hope to have a specific experience for the users, you can really guess what that's going to look like. And a lot of people are not comfortable with that. Let me give you an example. I was a part of a product review and I was demonstrating this AI product, AI feature I was launching and leadership was trying it out and they said, hey, I don't think this works well because it's not consistent. Every time I'm using it, I'm getting a different answer. And I said, well, that's the probabilistic nature. This is working as intended. That probabilistic nature has a lot of challenges. It gives you as a PM because you have to make people around you be comfortable with this uncertainty, if you will. But it's also kind of the beauty example is if you use Google, Gemini or Dall E to generate an image with the same prompt, you're going to get a different image generated. Or if you're driving a car and you use self driving, there's always like a little probability that comes to it. So nothing's 100% safe, nothing is 100% accurate. So that's challenge number one. Challenge number two, there's this experimentation culture that is tied to AI products because just because of this uncertainty, you need to do way more experiments than in classic product management because you're just making hypothesis at all times. And you say, well, I believe that, that if I deploy this model into this experience, we're going to get outcome X, but it's not always the case. So there's a ton of experiments, there's a ton of pivoting and you know, if you have a team that's very much used to okay, I'm going to do A, B, C and then achieve Y, they don't really get that. They get demotivated, they don't know if they're on the right track, they don't know if you know, they're progressing in their career. So fostering a culture of experimentation in your team, engineers and stakeholders is important. So I had in the past to figure out what success meant for a product. And in traditional product management, it's okay, we're launching, we're getting hundreds or millions of users, awesome. Figure out. Whereas in AI product management, success may be, hey, we made hypothesis A, we tested it out, we figured out, hey, that's not going to work and we pivoted and that hitting a mouse and of being able to get an answer to the hypothesis is success. So it's more of a milestone based progress, if you will. So yeah, I can speak about AI product forever, but I think these are the things that stand out the most.

Okay, so certainly that first point about things being probabilistic is really interesting because if you come from a data science background, a lot of stuff is probabilistic. But if you come from a software engineering background, like almost everything's deterministic. So I can see how there's a bit of a culture clash.

There's.

Okay, so your second point about experiments was really interesting. Can you just go into maybe a bit more depth about what do these experiments look like and who performs them and what are you expecting?

Let's assume we work for a startup and this is totally hypothetical by the way, and let's assume that the Startup has created an app which essentially replaces the native keyboard on your phone. And let's assume that the whole premise is, hey, this keyboard learns from your texting behavior so that it can actually predict what you want to say and it's going to kind of speak for you and on your behalf eventually, because it knows how you speak, it knows the words you use and you don't need to type everything. You will just say, hey, did you mean to say, hey, how are you doing today? With like two question marks? Just pop gets pre filled. So let's assume that's it. Let's assume you've launched and you know you have a certain amount of users, people like it now in the backend and in the startup you likely have a partner which is a research scientist that gets a lot of the data that come in from users and logs and they can use this data in order in the backend to improve their model. So in the backend the scientists will come to you and they will say, hey, we see the usage, we have some metrics to measure success. These metrics can be, you know, correct prediction of what the user meant to say and satisfaction, all these things. But I have a model that I think that if we run and launch this model and replace the model that's currently live, we are going to get improved metrics. So we're going to run a little experiment and we are going to say, all right, for 50% of the population, we are going to keep the model that's currently out that can predict what merely wants to type as they type it. And then for the other 50%, we're going to silent launch the new version of this model and we're going to measure our North Star metric, which can be ratio of accepted suggestions, let's say prompted to the user. If this metric goes up for the version we're testing, which is the improved model, awesome, we'll start rolling out to the rest of the population, if not will roll back out so that we just discard this. So there's a ton of that. And sometimes I launched the new version of the model even if it was performing worse because I knew that over time you'd get more data and it would get improved. But as a user, I've been in the side where I had a product that I loved and it was perfect and then suddenly I upgraded to like the new version and I hated it. But that's what AI and probabilistic nature is. You kind of need to learn that there's a learning curve in the system that needs to learn you and you need to be careful and you need to be comfortable as a user to know that, hey, this thing listens to me and it understands me and how I behave and it can improve a product for me. But yeah, that's a very simple hypothetical kind of experiment and a B testing, but there's a ton of that.

Yeah, it's a very cool. Just such a simple but powerful technique is just rolling out features to some users and then seeing whether it's better than the existing thing or not and then having to roll things back. Although I imagine having to roll back a feature that you just launched, that's got to be kind of disheartening. How do you deal with that sort of thing?

Very good point. Well, I've worked both in startups and in big tech now. In big tech, I mean, it's massive bureaucracy, if you will, in processes and operations. So you need to account for that and say, okay, in that case, if we have to roll back, here's who's going to do what, here's the timeline that's going to happen. I would likely need to write something that we call a postmortem document. I don't know if you're familiar with it, but essentially it says, hey, here's what we tried, here's what went wrong, here's how we'll prevent any experiment from going through that, you know, something going that valley in the future. But you know, in big tech you kind of expect it because it's something that might happen more often. Also in big tech, you're not going to open to 50% of the population. It's going to be more like 1% is trying this. So you know, the damage, if you will, if something doesn't go right is not that big. In the startup though, things are way faster and you don't really care about a polished experience. It's more like, okay, let's learn. If yes, move forward. If no, you're good. So less operations, less bureaucracy. But I have to call out that for AI, Another very important challenge is ethics. Being responsible, responsible AI fairness. So for every experiment, you need to make sure you can log specific data, you can process specific data. You need to make sure to notify the user and say, hey, this data is going to be collected if you want to opt in to improve the product. So there's a lot there to notifying the users. But yeah, big tech way longer, but you expect it startup way faster. But it's part of the hassling, I guess. Nature of the startup and I just love this startup culture. I just miss startups so much.

All right, so you mentioned ethics is an incredibly important sort of skill set to have as as being a product manager, particularly with AI. What are the other sort of important skills that you need in order to be a good AI product manager?

I posted this on LinkedIn the other day. But every time you come up with ideas and brainstorm, even on day one, you want to ask the question, not just, hey, can we do this? But also, should we do this? So are you affecting users in a way that is not ethical? Are you going to create a product that's not going to be inclusive or provide, generate images that are diverse enough or anything like that? So the skill set there needs to be understanding the risks and I can provide a list of risks out there. Number two is understanding the regulations. I know the EU's first AI act was actually went through a few months ago. And just being aware as a product manager as to what you can and shouldn't, can and shouldn't do is just so important. So staying informed is another skill that's just so important. And just having the empathy and solve for your users without affecting them in a negative way is very important.

Okay, yeah. So certainly understanding legislation and empathy seems like an incredibly important skill, I guess, for almost every role. But yes, certainly being able to understand what your users want is going to be incredibly important. How technical does the role get? Like, how much sort of in the way of AI skills or other technical skills do you need in this role?

So I have a PhD in Data Science and by the way, I never knew I was going to be a pm. I didn't know what the PM role was all about, so I kind of figured it out. But for my case, I had to tone down the technical dial because when I became a PM and I was working with data scientists and research scientists, my inclination was to say, hey, don't do it this way, do it this way. This is what I would do. And then I realized, hey, this is not my role anymore, so I need to tone it down and just learn how to partner with people, because that's not what I do anymore. So that was a different and interesting thing. For folks that are not technical, I tell them they need to have technical awareness, meaning people need to understand how AI works. They need to understand what that probabilistic nature looks like. They need to understand the end to end process, from data to training to human in the loop to the outputs. So they need to understand how Things work. And then another thing, which is technical influence, meaning you need to be able to have technical conversations with the scientists and engineers. So I'm not saying you need to say, oh, I'm going to use back propagation or like a specific model, but when a scientist comes to you and says, hey, for me create this music recommendation system, there are two implementations that I can do and you need to be able to understand the trade offs. You need to be able to make a call as to what the best way forward is in terms of these trade offs. And for you to understand these, you just really need to understand how architecture works. Let me give you an example. Let's say, I mean the most basic example which is, and for there's a imagine like a slider, which is a trade off slider, where on one hand privacy is going to be very high for the users, but the ads are going to get, are not going to be personalized at all. So it's kind of like, okay, is that the best experience versus still having private measures there and consents and all these things for users, but them getting more personalized ads. So that's a trade off. So you need to understand how things work, what privacy means, how much we save users data, how to inform them, all these things. And the software engineer scientists will come to you and they will say, well, okay, what do I do? How do I do it? How private is it? And we need to be able to make these calls and decisions and accept the risks very often.

All right, now you mentioned you had a PhD in data science. I'm wondering, where does data fit into this role?

Very good question. So yeah, my PhD was all about predictive analytics and it was actually a very important and interesting multidisciplinary topic. So I looked into BitTorrent downloads and YouTube views for songs specifically. And I was trying to predict how popular a song will go over time. And I used epidemiological models, so I was able to map online data with epidemiology. And I proved what goes going viral means. No one else had proven this word going viral. Because if you read my PhD, I said that, oh wait, content does go viral and it does follow the curve that influenza has. And interestingly enough, it was like 10 years after my PhD when Covid came out and people started reaching out and they were like, can you predict Covid? I'm like, I'm not touching this. I can. So, so where data comes in. When I first joined Google after my PhD, I was focusing on speech recognition and we wanted to launch Voice search for so many different languages. And I had to figure out how do you collect data for things that don't exist. So there's no concept of mining or logging data. And you know, how much data is good enough and what is the relationship between amount of data and quality and you know, what is the mvq? MVQ is the minimum viable quality for you to launch something. And you know, lots of strategy came in there too. For example, if you're the first one in the market to launch a recognizer for Greek, I'm Greek, by the way, do you care about quality as much as you'd cared about it if two or three competitors have already launched it, in which case you would likely want to match or be better on them? So I was exploring all these different ways to collect data. A ton of experiments on quality and a ton of figuring out ML ops, how to clean this data and how to do it in an efficient way that doesn't take seven, six year months long. So data comes in because the better the data are, the better the quality you're going to meet. Ease and I wish I could communicate with myself 11 years ago when I was doing that because now we can synthesize data with Journey and there's no concept anymore of collecting data or acquiring data or cleaning it up and you can just synthesize it. And that's partially, I work on, on this project at Google Engine AI and synthesizing data and it's just so, so fascinating the potential of it.

Okay, so it seems like there's kind of a lot of data involved, particularly around deciding like what the users want. So I guess traditionally you're doing user research, you're doing surveys, interviews, things like that. But you suggested that you can also synthesize data. So is this synthetic data going to replace those user surveys and things like that, or is that a complementary thing?

Yeah, so I was actually referring to the first step of training a machine learning model, which is collecting data in order to feed this into a model and train the model. Now you mentioned a very interesting point which is in the product development lifecycle, hey, we need to use research and figure out, hey, would users like a product that can generate video out of text? So in that case, in big tech, of course we have UXR departments, so user experience research, where, you know, you will ask your questions to say, here's our hypothesis, would users want this? What kind of user segments do we have and what, you know, can you create a focus group? Can you ask them? The thing with AI is There are two things, right? There is AI product management, which is working with scientists to train the models, leverage AI. And then there is AI for product management, which is using all these awesome tools that are out there like Google, Gemini and cloud and of course chatgpt to just enhance you as a pm. Specifically for the research side, I'm using this app, it's called Como AI. And it's amazing because I can say something like, hey, should I build a smart fridge for families? And it will literally give me Quora links and medium links and Reddit threads and YouTube videos from creators where they answer that very question and not only provides the actual link, it also summarizes, you know, the TLDR for me and it's like, well, according to families, here's the things that are more interesting and here's the links, if you want to click through it more. So it's fascinating to see just how much faster the PM's job can take place, how much less costly. And of course, disclaimer, you shouldn't trust all these tools 100%, but they give you a jump and a head start that's so immense and significant. So when someone asks me, is the PM role going to be replaced by AI? I say, of course not. But PMs that do use AI are likely to replace you if you do not use AI.

That's fascinating. I like that you're using AI to do the job, to make more AI products. It's a good sort of recursive thing. Nice. Yeah. So certainly there's sort of regenerative AI tools for doing research into almost anything that seems incredibly powerful. All right, so I'm curious as to who you work with. So I presume there's some sort of engineering teams for actually building things. Can you just talk me through? Who are the different teams that you work with as an AI product manager?

Yes. And I actually created, I get this question a lot. I created a nice little graph on my newsletter that I'm happy to share. But essentially there are multiple stakeholders. The most important ones are research scientists. These are the ones that will tell you, hey, for this experience to come out, we need to train models xyz. Here's kind of the data we're going to need. Here's what the trade offs can be that we need to discuss. Here's the risks, here's what quality could look like. Then at the same time you have a software engineer that is there to put together an experience. You also have ux. What's going to be the designer? Let Me give you an example. Let's say, well, I was using Zoom the other day and there was this new feature that is called, it was like an AI assistant that summarizes the entire conversation and then after the conversation is done, it sends you an email saying, hey, here's what was discussed. So for this to get created, you have the research scientist that needs to get the data, which is going to be voice data, converted into transcript. So they take as an input the transcript so that they can process it and summarize it by using Genai, of course. But at the same time you need to work with a designer that's gonna create the little widget that pops up on the user that says, hey, do you wanna activate the AI assistant? I think it's called Assistant. AI Assistant. Here's how it works. Do you wanna create it for all the speakers or just for you so it will create the actual ui? You also need to work with UX writers that are the ones that are gonna use the right wording. Because if you ask the scientist, hey, hey, what should we call this? Design is likely going to use more technical terms, but we want to make it more user friendly. So the writer is going to tell you what to write. Designer is going to create the actual design. And after you have designs, after you have the models, you are going to have the software engineer to blend all this magic together and integrate it. You also have ops involved, ML ops, because in order to train these models, you need a ton of training data. And figuring out how to collect all this data is a whole other story. But assuming we have data and we can train the models, there's also privacy and legal involved that you as the pm, need to work with to say, all right, here's the data we collect, here's how we process it, here's when we delete it, here's what stays, here's how we notify the user, here's how we get consent in order to put this together. So that's another party. Sometimes you do have another stakeholder which is a third party vendor that you know. If you want to clean up the data before training model, you're going to use them. If it's a third party app or if there's hardware involved, there are OEMs, manufacturers you may need to work with. I think that, oh, and of course the user, right? You're going to ask the user and be like, hey, is this something you want? How should it work? And so on. So there is a ton involved. But between classic product management and AI product management, the bubble that's missing, I guess it's scientists and maybe privacy because data may or may not be involved.

Okay, so I think you just listed pretty much every team within the company there. It sounds like it's very cross functional.

Oh, it's highly cross functional and it's highly. There's a lot of risk and there's a lot of alignment. So often I tell people, if you want to be PMs you should start by being a traditional PM before you can convert because you need to be able to know the craft before you add in the extra layer of AI in there.

Actually brings up another question is how do you get into this field? Like how do you become a project manager and then how do you become an AI project manager?

There are different ways and different things. I see for AI specifically, it is the case that traditionally you'd find more technical people like myself that used to actually do the work and train the models and all these things. And you kind of, if they have an interest in coming up with ideas, solving for the user, getting alignment, figuring out success and presenting, getting alignment. I keep saying this because alignment is a big part of the job. Then you see a lot of people converting from the technical side. I also see standard PMs that learn the AI intricacies and you know, the technical awareness and influence that I discussed before. But what I tell these people to convert is the best way to become an AIPM is doing it at the company you're at right now. And this is literally going to your manager and say, hey, I love AI, I'm interested. Perhaps, you know, I can upskill by getting a training program. And of course I teach AI product management and certified people. So I get all those people that get reimbursed by their employers and I teach it to them and they get the certification. But another example is going to your manager and say, hey, 20% of my time can I work with this scientist and come up with this idea and just do a little pilot, see how things go so that you can have this AI experience and launch an AI product, which as I said, it's different than traditional. Another thing I see is now with AI, all companies want to just try it out, see what it's like, see if it's applied to them. And I tell people, why not go to your leadership and propose a little labs department in your company where you know, you can be the part MPM there, work with a scientist and come up with ideas that you can infuse AI in some team in their organization and, you know, perhaps launch something if you know it's good. So I guess TLR don't wait for permission, start doing it so that you have some AI experience and then be like, okay, I got it. I am an AI bm.

Okay, I like this. Kind of few different routes into this and it seems like a lot of it is just about make sure that you've played around with the AI so you know what you're doing and make sure you've played around with the product that you want to sort of build just so you've got experience with the product and then you can sort of just dive in and figure out the management side of things sort of afterwards. Okay, cool. So right back at the start of this, you mentioned the idea that you have the right product, so you've got to get a product market for fit. Can you just tell me what is a product market fit and how do you go about determining whether you have that fit?

So imagine three bubbles. Bubble number one is user desirability. So building something that's desirable from users that is going to solve a use case that is going to solve a pain point that is going to add value to the user. Number two is something that's viable from a business perspective, meaning you know, it is going to make money or it is going to add value to the business metrics of your company. Number three, something that's viable from a technical perspective. I'll give you an example of a company that I love and they definitely found a product market fit in AI. So I really admire Adobe and of course they have their own way to generate images and I know they only use licensed data to do that, which of course I applaud. So let's imagine, you know, they have this suit of products. I think it's called Adobe Creative Suit. And let's assume they have a user base which is mostly designers. Their business model is the designers are paying per month to use this product. Right. And a product can be, you know, an image generator. So let's imagine there's a parent of a kid that goes to a designer and says, hey, I want to create a custom invitation for my kid's birthday party. And the kid loves turtles. So make a turtle that has these colors that's called Bob. So then the designer can go on the system, they can generate this awesome very custom turtle, add it in the little invite template, and then send it to client with a couple of tweaks. The designer has created this awesome, very custom work super quickly so we can Take even more clients in. The customer is going to be so satisfied. So of course they make money out of it. So this takes all the boxes. User desirability, because the designer is more than happy. Business viability, because the designer, of course, is going to pay per month the subscription to use this tool. And then number three, this is something that's absolutely feasible from a technical perspective because now we can generate all these images. They've infused these generators, I think, in. In their different apps. So I think that's a great example. And Adobe has done an awesome job.

Okay, I like that. So it's really about, you just got to make sure that you're testing things against customers and seeing are they actually satisfied with what you're doing. And then you sort of iterate on that and just make sure that you keep increasing that sort of customer satisfaction and I guess making some money in the process. Okay.

It's important to have a viable business. Right? There's no, like, the is a big problem. Demos are great, but at the end of the day, you need to build. AI is not a problem. Right. So you need to make sure that something comes out of it that's viable, you can make money and it can sustain a business.

Okay, cool. And so you mentioned this idea of like, making sure that customers are happy or users are happy. Can you just go into a bit more depth on, like, what that sort of user research involves? I think this is something especially like people in the data world ought to know about, but in general don't. So, yeah, talk us through it.

Metrics in product management are so important. And I don't know if the folks listening are product managers or not, but there are certain buckets of metrics. And probably the most important bucket is called product health, which means if you have a product, how are people using it? So, number one, are people satisfied? And there are things like NPS scores and CSAT that show us user satisfaction. That's similar to going to an airport. And after the security lane, you see these little buttons like an angry face and happy face, and they're like, how was your experience after this? That's what that looks like. But you also have things like an acquisition funnel, for example, from all these people that become users. How many come back and how many of them are actually engaging with the product? How many are we're sharing a story or liking a story? How many are commenting? So what is the interaction looking like? What is the engagement? And do people come back to it? Right. If people come back, it means that you're doing something right, retention is going up. I think my favorite metric is called stickiness which is, you know, whether your monthly active users, how many are actually coming back on a day to day basis. Then there are other metrics of course, like monetization, how much money am I making from each user? If money is involved, you know, do they get ads, do they actually pay? If you're playing candy crash saga, let's say, and the user gets out of lives, they get a pop up that's like okay, to continue, it's 0.99. Press the button, continue so you can figure out the user lifetime value or just how much money. There are more metrics like referral, how many more users are you getting from someone? And are there any schemes there for referral? Like doordash tells me if I refer someone I get $20, they get $20. But in AI there are more metrics. There's also accuracy, there are model specific metrics that you need to be aware of. Like how many times am I getting something right? What is the probability of a false accept or a false reject? But yeah, I hope this answers the question for what metrics exist for users and how do you measure success of your overall product?

Yeah, I love that. So this is like some very simple stuff like NPS is just, it's a simple like survey. You're taking an average of like whatever the scores from 1 to 10. And then you got more advanced stuff like how do you measure how often people are coming back to your product, how much money you're paying from your product? Tons of stuff that you can track then. And I guess the key is going to be deciding which ones are the most important metrics that you want to follow in order to measure your success. Okay. Actually, so speaking of success, have you got any success stories from products you've been involved in building that you can talk about?

Wonderful question. Yes, yes, yes.

Many.

I joined Google in 2013 and we launched voice recognition and it's just incredible. I think we launched 200 voice recognizers in all these countries and that means literally adding the little microphone next to the search bar and on phone and on desktop and the next day you go there and you just see people using their voice to search and they can get images and they can get text and you just unlock this ability for people to browse the web via voice and you can imagine that accessibility, use cases or people that don't know how to write can now search and get the images and they want. Right. The impact is amazing. But eventually I was owning also Voice Match, which is the ability of the Google Assistant to recognize that it's merely speaking versus someone else. And that's just an incredible product I led for multiple years because there's something about identity that I love. So let's say you have a communal device on your kitchen and you walk in. It's different to just ask it to play music versus asking it to play music and knows that, oh, merrily, we'll want to hear Coldplay, right? So this personalization element and experience, I think it's very, very good. And yeah, I have a ton of examples, but I think these are the ones that stick out the most because, you know, of the impact we've had.

That is very cool. And yeah, those sort of. The voice interaction is just incredible accessibility feature. So I guess for people who can write and have access to something to write on, then, you know, thinks that, well, I don't really need this. But there are so many situations where you can't write things down and you can't type. I'm particularly driving. It seems like an obvious thing. So, yeah, the impact, that's got to be very satisfying, I think, getting this capabilities, lots of people. All right, so the flip side is, has anything gone wrong? Like, what do you wish you knew at the start of your AI management career?

Well, I wish I knew it was AI product management. I wish I knew what things looked like in classic. I think what happened with me is I became very specialized early on. There's this thing that's called the generalist pm and whatever they throw you, you can thrive and you can do very well. Whereas I started with AI. So for seven or eight years I was just doing AI, which is amazing. But after a point, I was looking for the next step. I was trying to figure out what my identity is as a pm. And I realized that there was only a finite set of things I could do. So I wanted to flex the PM muscles, let's say, in something that didn't involve AI. And when I did that, it was kind of tricky for me because I had this experimental culture and people would say, no, no, we care about launching as soon as possible. And, you know, no experiments are involved. It was kind of interesting to get used to that. So I guess I would want to know that there are different types of PMs and that I guess adding in different flavors as you, you know, as you get shaped as in as a professional would be good. But I will say that with the current market, with everything going on in tech, specialized PMs are the ones that, that are more likely to get a job faster. And I mean, don't quote me on this, but I do feel, and I see from all my students in the bootcamp that people that do get the specialization have more chances in landing a job versus the generalist pm. But that's what's happening now. I think in the past it was, you know, people wanted generalist PMs because there, there was just more diversity in jobs, let's say.

Okay. I feel that's one of those things that sort of swings backwards and forwards as the economy shifts, whether people want specialists or generalists. And it goes backwards and forwards. Okay, so can you talk through a bit about what the tech stack involved is? What sort of technologies do you need to know about as an AI product manager?

Well, you need to understand the different ways that the model can be trained. And there is supervised learning, which is imagine, well, I have a four year old and a two year old. So I walk on the street with my kids and we see a squirrel and I say, oh, that's a squirrel. Or, or if there's someone with a dog, I say that's a dog. And squirrel, dog, bird. So essentially you provide a lot of labeled data. So if you see something, you're like, that's the label of this, this is the squirrel, this is cat. And then eventually the kid gets it and they say, oh, that's a cat, that's a dog. I don't tell my kid, hey, if the animal is smaller and there is fluffy tail, it's more likely to be a squirrel. I let the kid figure that part out. So supervised learning is exactly that. You provide a ton of table of data and you tell the system, okay, now you do your thing, learn, figure out the patterns and eventually the system will say, yep, got it. So for every new animal that comes in that doesn't have a label, the system will say there's a probability that the, you know, it's. I'm 65% sure that this is a squirrel and 1% sure that this is a cat. So I'm choosing the squirrel. So that's how it works. There's also unsupervised learning, which is more like, hey, here's a bunch of data. I'm not going to tell you anything about it. You figure out the pattern and you figure out the cluster and it kind of can create cluster use cases for this is more like news outlets where it's like, oh, I think this article is about sports. So I'm going To add the sports label. I think this articles are more about politics or, you know, pop music and all these things. I actually created a graph that I posted on LinkedIn that has the landscape of AI because I was looking for something like that. There's more things like semi supervised learning and of course now with LLMs, I wanted to call it out in a different bucket because there's just a different technology, different way of trainings, models, transformers and all these things. So take a step back. People need to understand this stack, which is different ways of training, different types of requirements, data requirements or metrics or risks or challenges, trade offs for each one of them. They also need to understand OPS very well. And I tell people this, you cannot have AI if you, you don't have an OPS org staffed because you will need to move data around. You will need a human in the loop to say, yep, good or not good. Trust ethics. Like you definitely need the human to just ensure everything is getting there. And the other thing is you need to understand what it takes to productionize AI. The AI can work perfectly in a lab and in an experimental server, but then when you actually take it through all the hoops of, of putting it through a system, launching it in the cloud or web or whatever, or on device, whatever you do, the quality is significantly going to drop. So you need to realize that, okay, this perfect data I had to train, this model works very well. But when it's out in the wild, am I going to get this data? Am I going to get the same quality? So productionization and the challenges that come with it is just a huge and very important part of deploying AI and an experience that people need to be aware of. And every company does it differently, a startup or a big tech. So there's no like cookie cutter. But I will say if you productionize, the quality does change.

So that's actually a pretty broad set of things you need to know about. So there's a lot of different machine learning techniques there. There's the mlops side of things, putting things in production, there's all that sort of data quality and governance side of things, plus all the rest of the actual project management stuff you need to know. So yeah, it seems like it can be incredibly broad but presumably very interesting kind of job. Okay, so talk me through. What are you most excited about in the world of AI product management then?

At the moment you just don't know what's going to launch the next day. And there may be something that's completely kind of destroyed the vision. You had because there may be this whole other way of doing things that is going to make you pivot. And there's something that my friend Hans said, which is, hey, maybe we should consider that we no longer pivot, but we evolve. So I'm just very excited about evolution, about AI and techniques. And the thing is, use cases remain the same, but the technology is the one that changes. So I'm just very excited to see and explore different ways I can deploy technology for a specific use case. And as an aipm, you never get bored just because of this. And you know, this has a negative side because the next day you may get something to launch that's going to kill your entire business. Right. You're not going to be. I was coaching this incredible. She had a translation agency and you know, people will come to her, she will translate, certified. That's it. She's like, hey, with Genai and ChatGPT, no one is using my services anymore. And I said, yeah, but you have a unique opportunity here. Maybe you can create your own chatbot that can translate, but then you can also say, hey, I manually certify and I review it manually so you can be the human in the loop for that thing. So she started doing that and actually her business is thriving and she works way less. So I think it's a win win. So we have to figure out how to evolve in this area of AI.

I really like that, just being able to make use of the AI technology. You're working less and then your business is thriving. That seems like a perfect outcome. But also what you're saying about how you have to pivot a lot. So I suppose, yeah, you don't know the answer to what those experiments are going to say. So you're going to have to continually change what you're doing. All right, so just to wrap up, do you have any final advice for aspiring AI product managers?

You know, things do get pretty technical, but don't let that overwhelm you because you know, it's not, it's not rocket science. Like, you will understand that, you will learn that and you don't need to learn all the math and all this crazy tech that goes behind it. You just need to be aware, you need to have influence and you need to be open minded. So please don't be afraid to embrace it. If you embrace it now, I guarantee you you'll be ahead in the future because it is coming and it's not an option really. And if anyone is interested, I'm running my bootcamp. I have a cohort based course course that is very, very popular. I'm happy to have people and teach them and certify them on how to be a ipms.

Nice. All right. I like that. The solutions. Just go and embrace the change and keep going with that. And yeah, I'm sure many people will be interested in your bootcamp. Wonderful. All right. Thank you so much for your time, Marie. That was great stuff.

Yeah. Thank you for inviting me. Ritvi. Have a great rest of the Sam.