What it takes to be a top PM today | Robby Stein (Google Search)

Hey everyone, it's so great to be here, and I love being here to get to talk about one of my favorite things ever, being a PM, product manager, and builder. And what's pretty incredible is that it's never been so interesting to be building products. And after 20 years almost of building, particularly in consumer tech, by the way, It's very upsetting to be able to say out loud, but after almost 20 years of building, I've been fortunate to build all kinds of different products over my career. I've built startups, and I've also helped lead teams and products through really big changes like Instagram and Google Search, where now billions of users are using products that our teams have created over the years. Also, I've built some things that zero people use today, by the way, absolutely zero. And What I've learned throughout this process is that there is a little bit of a pattern of the kinds of things that we've done to make the products really great. Sometimes it works out, sometimes it doesn't, but there has been a repeatable pattern that I've seen emerge from the things that have done really well and the things that have not. And that's really what I'm here to share today and impart to all of you. There's lots of things to talk about, but I wanted to give you just a little bit of extra background on me. So I spent some years at Instagram where my team has built out Instagram Stories and shipped it originally and built all of the experiences around We have Reels, Feed Ranking, and Direct Messaging. So a whole bunch of products that came out of that era. And then now at Google Search, where I am working through a brand new experience with the team to help think through what is Google Search in the AI era, so that you can truly ask anything on your mind and get the most helpful information imaginable across everything, whether it's videos or audio, text. And you can tap into all of the knowledge of the web and of Google and connect to the information that's out there. But today, I'm not talking about the transformation of product, although that is something I really deeply enjoy. I'm talking about the craft of product. What does it take to build? What does it take to have a product that people love and to shape that and make that? And how do you do that in the age of AI that ultimately, and from my opinion, any one of you can do now With just a handful of agents or kind of AI assistants, which took us years and whole teams of people to do, um, in the process here. Let's, uh, just mention one thing I wanted to say around the, the role of PM before we get started. And that is really the, the fundamental shift that we're all feeling is that we used to be, I think as PMs, a big part of our value was around getting stuff done, was around organizing, creating energy and momentum. and being almost like a project manager and a product manager. And I think in some cases that's still true, but really, if in the world of today, almost anything you can think of can get built, it's more clear to me than ever that the actual true value of PM is actually around judging. It's around taste. It's around doing something extremely well. And I think what's something I've thought about a lot is that people say the PM's main craft is that you do lots of things pretty well. And I think that is somewhat true, but I think above all else, actually, the craft of PM is one simple thing that you should be the absolute best at, and that is decision-making. It's about making decisions, and you should be students of what it takes to make great decisions. And it's a humbling thing, because most decisions, many decisions are wrong. And these are just the things that I've learned over the years that have helped me and the teams make decisions that have led to some of the great experiences that are now out there today. There's kind of a book that's been unwritten in my mind that I've been kind of thinking about, and 3 chapters around the process that, you know, I've used and teams have used to build, and I'm here to share that today, really for the first time in this format. So, chapter 1 is understanding people deeply. It all starts with people, obviously, and you can figure out lots of things that you can build, but it comes back to the very root of what are the fundamental needs of a person. And I love the book Clayton Christensen wrote, Competing Against Luck. I'm sure many of you have read it and have probably read the Jobs to Be Done framework, which I really do love. The thought is that we don't use products, we hire them to do things for us, and we have to know what those things are. And it's best articulated how to extract those kinds of insights through a story. And so actually, a team that we had at Instagram, we did an offsite with the people who created the Jobs to Be Done framework, and we actually went through this process. So you can all think about this with me. What was the last major purchase that you made? So quickly think about it. And for me, it was a bed. So here are a bunch of beds. And then what you're supposed to do, and the book actually talks about how to do this, we've really internalized this with our process, is to put yourself back into that moment. And if you're talking to a person, ask them details about their life. What was it like that day? Where were you? What were you reading? Who were you with? What were you wearing? Were you with your kids? Were you by yourself? Were you in a store? Were you online? And you really deeply put yourself in the position of that user. And for me, looking at the sea of beds with my wife, and basically we were kind of trying different things out. And at the end of the day, 3 or 4 felt great. But my wife weirdly turned to me and said, can you do me a favor and jump on this? Basically, she was like, do a barrel roll. I was like, what? All right. And so I did like a huge flop, I guess. And she was like, we're getting this bed right now. Thank you. We're done. And I was like, what? This is so weird. She's like, no, the main thing that I need is a bed that will not wake me up if you, like, move. And if you think about it, that's, like, a little offensive. But also I was like, okay, it's real. And if I was a PM for beds and I might be, like, classically thinking, quick, what's the roadmap for beds? Right? You might be thinking, okay, how about cooling? Everyone's doing cooling. Like, yeah, cooling, sure. Or like eco or just pure price. Like, those things do matter. But for us, the thing that made the decision on that day is a concept of transference. And if you didn't talk to me and you didn't learn that, the PM, the team, they would never know that. And so that's what it means to discover the job to be done for someone and how products, things, customers hire what your services do, your products do to help them. And very quickly, just talking 2 quick examples in the digital world, one from Google. Early in our AI experiences, we realized that people actually came for inspirational journeys. But 2 years ago, chatbots were Only text. So we were one of the first teams to look at this need and say, how do you get inspired by what to buy or what kind of couch to get through text? How do you describe a couch to someone? You can't. And so we were one of the earliest teams to really invest in state-of-the-art multimodal understanding, retrieval, and knowledge systems. So you could actually ask about a couch, get an incredible visual response, and do multi-turn conversation with it to say, Oh, I actually want emerald green, or I want fuzzy. Like, what does fuzzy mean? Or I like that throw pillow. How does it create attention on the throw pillow and do a turn and then now do a grid of throw pillows? Like, these were really hard things many years ago, and obviously you see them now in many places, but that was a core need and a core insight that we saw. And then last example was very early in our AI mode exploration, which is a way that you can use Google Search now to ask anything and have a conversation with Google, was people came and it had all this incredible information, like, Here, this is a neighborhood and, like, a food question, but it was missing something. It was missing something that people were coming to Google for and were expecting, which was the knowledge and the accuracy and the experience of actually seeing the places you go, seeing them on the map, getting the Google-level authority of the star rating, the closing time, what kind of food it is, how expensive it is. It's completely missing from the product. And when we brought that together so that now you can talk to Google, ask about food, and they get this rich experience that brings all the goodness of seeing the food and clicking on an entity now and actually seeing the description that you can trust, that was the magical product. And it's still one of the most popular things on the product today based on thumbs up ratings and other things that we follow. But again, back to a human insight of the why behind they were using the experience and what caused them to use it in the first place. And so AI can actually change a lot of this, and a lot of the talk today is, how AI can apply to these. I think on this one, I'm still thinking about some of these, uh, myself, but one of the things I'm excited to try more is how you can actually take the transcript data of these user interviews and actually have it specifically source jobs to be done and scale that much more. Another idea I've been thinking about, um, is actually an agent that can interview people with the exact methodology that I just described so that it can really ask a user more at scale to talk about why they're using the product they're using and what those underlying problems were. So that is understanding people. Next up, you have a seed, you have something you're working towards. The next step, diagnosing root causes. So the way I think about it is you have this initial vision of this problem you're solving for people, this initial understanding of what people want, and then you make a thing. And that thing is usually not great. And most of the things that I've ever been a part of building started not great. Everything did, really. Um, and what's incredible about this process is by having a clear and analytical view of the exact problems that are in your product space, ranking them in priority and understanding them deeply, and then actually fixing them and then asking it recursively again and really looping through, you basically are doing your own training process, kind of like an epoch of model training a little bit. And each update you do with this changelist gets you closer to the market fit you want until you have that fit. And then it works. There's 2 stories I wanted to kind of articulate here, both from Instagram. One was when we originally launched Stories, it actually did take off. It was one of the earliest things that worked pretty well out of the box, but we had many internal versions where Stories were fit into the feed that didn't work great. So there was a ton of headache to get to the final version. But one thing that was particularly clear was that not everyone was adopting it or felt comfortable sharing. And really, the whole premise was, hey, this is an opportunity to share your life throughout the day. Why aren't you doing this? And the process that we followed was pretty simple. We asked a very clear root cause of why you were not able to do the thing that was our goal. So it's kind of the inverse. The first one's like, why did you do the thing? Now you have a thing. It's like, well, why aren't you doing the thing? And it turns out that if you ask that question, people answered— think of all the leaf nodes, like a cloud behind me, of all of the answers people had. And then if you were to group them by theme, it turned out that the top one, when you sorted them was something around audience problems, more or less. Like, my ex is on it, my teacher's on it, my aunt is on it, my sister's on it. I'm not gonna like just put a goofy thing on, like people are on it. And so what we realized in seeing that, and then what we did was we then did a very large quantitative survey, thousands of people taking that as an input, and then we were able to quantify the actual weights across them. And what we figured out was that if we couldn't overcome that burden, there's no creative tool, there's no like interesting product idea that would work. And so the team actually spent 2 years cycling through different ideas in the space until we came across close friends, and it ended up working. And there were all kinds of weird permutations in this, like, multiverse scenario where, like, it's called favorites, and it also has on your profile a backdoor that you can, like, see a private version of your profile, and you can also post to your feed under that kind of close friends label. Super confusing. No one understood it. Like, why would I put this, like, very raw photo in the middle of, like, a glitzy feed of all this stuff that's happening? The only thing that worked was within Stories, so we cut out the entire product, relaunched it as a Stories experience, and it worked. Reels was the second example. We had a launch for Reels in Brazil, and we thought at the time that if you're doing goofy things and dances and posting them, why would you want them to live on your profile for everyone to see forever? Which made sense, and the team felt that way. So we launched it, huge failure. So it turns out that if you invest a bunch of time doing an awesome dance, you don't want it to go away. in a day, which was an ephemeral format. It's very obvious in hindsight. But the bigger insight was the people who were posting them, even if they were young, you know, people, they wanted to be entertainers. They wanted to make businesses out of their content, and they wanted it to live on and go viral. And so we did a massive update. We talked about this changelist, where now it's its own format. Instead of being ephemeral, it lives, like, forever, and it can be something your whole school sees, instead of the opposite, like, oh, I don't want my whole school to see it. It's like, no, that's actually what you want. You want everyone to get the joy from this product. And then obviously, that take has taken off and done quite well over the years. So 2 really clear examples of how that's happened. And with AI, this is actually possible to do a lot easier now using internal tools, and I think things that you can all do every day. And we use, you know, internally Google Anti-Gravity, but we have a system that effectively lets users that have opted into this— they want to talk to us more, and they've decided that they're comfortable sharing as they're using Google products. with us. And so what will happen on the side is we will then ask them, hey, what did you think of this? Where did it miss the mark? And they will put comments in. And so imagine this huge library of feedback from people now that's available. Well, now the model can take that and can both take qualitative feedback like above. This is a way— a shopping example for backpacks. It's like, it's awesome that you knew about each backpack, and it's awesome that you knew which was larger and the sizing, because Google has all this canonical information, but you should have asked me about my son. and the height. Like, that's a pretty key thing, or is it going to be too heavy for my kid? And it was so obvious that when you stack rank them, this was a very common kind of feedback. And so this concept of being collaborative and creating a conversation so you can actually help the person make progress to the ultimate objective, because they don't actually know all the things that they need to know to answer it, was absolutely critical to getting the user to vote and say, this was awesome. And so we invested a ton of energy in this, and it's been one of the biggest things we've done. In terms of engagement, in terms of usage, in terms of value and helpfulness that we've seen out of the product. So that's kind of chapter 2, which is around understanding the root causes. Now, if you're lucky, you have a great seed idea, you've done this process of finding the market fit by looping through, seeing your problems, analyzing them rigorously, and fixing them aggressively. Maybe you get to the stage now where the product is ready for primetime. You're going all out, you're going to all your customers, you're sending the blast email, and And really, this is about the fit and finish and final elements of the product, which is around craft. And I think this is often overlooked, but basically, the way I think about craft is the feeling that I think everyone has a general sense, and I believe people can detect whether the creators of a product actually cared about the product that they created, basically. And there's 2 things that I think drive this. One is, does the product work perfectly, and is there any user pain, basically, in that experience? And the second is, how does it make me feel? Do I feel something, joy, excitement using it? And so I'm going to talk about each real quick. On the pain side, it's really not just about obsessing over it and you all being power users of your own product, but actually how to scale that now with AI, which is really incredible. And so I actually built a tool that I'm sharing here. This is a real tool that I built internally using the Anti-Gravity system to basically look at all of the potential questions we might ask. So here's a sample here on the side. Use the product as an agent. So, like, literally, like, go to Google, type in a question, go to the AI, ask a follow-up, and then record that experience, screenshot it, and then evaluate what you see based on a really comprehensive rubric. And what we found out was that the model actually was incredibly capable to look at things like, hey, there's a math question, or this is like a school homework thing. Is it doing a great job rendering math in LaTeX, or is it you know, emitting weird, uh, characters? Or if you ask about bioluminescence, is it adding a visual tray, which is what the response should have if you're asking about something visual, or is it just text? And you can go through, and you not only can you get to the best response, but you can see breakages really easy— easily. And the team has now invested in a bunch of agents that can go figure out when stuff is just broken or off spec, and then increasingly fix it itself. And you have this feedback loop, and And we used to have maybe a QA team. You may need a PM, or you might need people on your team who are obsessively using every single part of the product. I think many of those days are now over, where we can scale this process much more elegantly for everyone. And then the last thing I'm going to say is around delight and really sparking joy. And this is a little experience. This is what the Google search box looks like now. We reinvented the search box after 20 years, announced it at I/O, what an AI-based search box should be. It should grow with you. You should be able to upload things and have context around files, around videos, around photos. What should that be like, and how daunting of an experience is that? And what we did was we came up with a system that said, okay, we wanted Google to still feel Google, but we wanted it to be this upgrade that felt like AI is here, there's AI inside that's helping you. And so there's this concept of this kind of gradient of color that really represents the AI kind of power behind the product. And you can actually see a couple nice details here that the design team brought together. One was actually in the cursor. If you watch it blinking, it's cycling through the different Google colors. And I can't tell you how many times I've seen this pop up on X that, like, someone noticed this and just felt like posting it. It just blows me away every time. Or if you tap in the search box, it's kind of excited. It, like, jumps out kind of and then loads the ring. It's like, ooh, like, we're getting ready to load. Like, we're here. You know, it has this kind of feeling. And it actually has real feeling with haptics. So you actually see— imagine all that coming together as an experience. I mean, it's just one small example of just by using the product and through motion, through color, through feel, through touch, you know, and the intentionality of the experience, you communicate the humanity behind the product and you give it a soul. And I think many people overlook this part of the product creative process. And then really, when you bring this kind of all together, this is kind of the wrap here. I think that this really means that you can have a playbook for what it means to be a PM in this new, you know, AI frontier. And it's— you have to still do all the same things. Like, a lot of this is not new, but it's possible in a totally new light with AI. So you have to understand people deeply, get at their core needs and problems that they're feeling. You have to be rigorous and analytical at sorting out what are the things I need to fix in order to get the product to win and to work for people against that objective. And I need to build it so that it's awesome and it's simple to use and it works great and it makes people feel awesome using it. And that's really the playbook. And I think that now it's exciting that the craft of PM is so critical. I think it's never been a more important time to be a PM and to do product and to be a builder. And I think the skill of making great decisions that can lead to great products has also never been more important. So hopefully all of you can take a little bit of something. It's just one way. There's so many ways to build products. You probably heard 10 other ones today that were just as valid as this one. You should decide on what makes sense for you. But hopefully you take one thing away that you can take away with you and to build as well. And hopefully the next time you're thinking about your products, you're spending a little bit less time thinking about, hey, here's cool features kind of out of the blue, or here's some tech we're gonna throw at it, or here's a new way we're gonna get it in front of people, and you spend more of your time thinking about the actual people and the humans on the other side of the screen that you're building for. Thank you so much. Thank you, Lenny.