Microsoft CPO: If you aren’t prototyping with AI you’re doing it wrong | Aparna Chennapragada

I have a cheesy Chrome extension. Literally, whenever I open a new tab, it just says, how can you use.

AI to do what you're going to do right now? How do you see the future of.

Product development being different? If you're not prototyping and building to see what you want to build, I think you're doing it wrong. It becomes even more important to have the territorial and taste making at the heart of it, because otherwise you just have a Frankenstein product.

There's this acronym that you taught me, NLX. What is that?

Natural language interface. NLX is the new UX. Often I hear product builders say, oh yeah, with AI, the model leads the products. That doesn't mean it's not designed. You and I are having a conversation. It's a podcast. I'll have another conversation at Microsoft, and that's a meeting. Conversations also have grammars, they have structures, they have UI elements, they're invisible. What are the new principles, new constructs.

In natural language as an interface? saw that Curser hit 300 million ARR in two years. Interestingly, you guys were very well positioned to do really well in this AI coding tool space. You guys said, Copilot, the first tool in the world at this stuff. So ahead of everyone, what happened? I would say, today my guest is Aparna Shenapragada. Aparna is Chief Product Officer at Microsoft, where she oversees AI product strategy for their productivity tools and their work on agents. Previously, she was Chief Product Officer at Robinhood, vice president at Google, where she worked on Google Lens, Search, Shopping, Augmented Reality, AI Assistant, and a lot more. She was also a long-time engineering leader at Akamai and on the board of eBay and Capital One. In our conversation, we chat about how working in B2B is like being Jean-Claude Van Damme doing the splits across two moving trucks, how she's operationalizing her team living in the future so that they're building towards where things are going, why people still need to learn to code, why the PM role isn't going anywhere, why NLX is the new UX and so much more. If you enjoy this podcast, don't forget to subscribe and follow it in your favorite podcasting app or YouTube. Also, if you become an annual subscriber of my newsletter, you get a year free of a bunch of products, including Linear, Superhuman, Notion, Perplexity and Granola. Check it out at lenny's newsletter.com and click bundle. With that, I bring you Aparna Shenapragada. This episode is brought to you by Eppo. EPO is a next-generation A/B testing and feature management platform built by alums of Airbnb and Snowflake for modern growth teams. Companies like Twitch, Miro, ClickUp, and DraftKings rely on EPO to power their experiments. Experimentation is increasingly essential for driving growth and for understanding the performance of new features. And EPO helps you increase experimentation velocity while unlocking rigorous, deep analysis in a way that no other commercial tool does. When I was at Airbnb, one of the things that I loved most was our experimentation platform, where I could set up experiments easily, troubleshoot issues, and analyze performance all on my own. Eppo does all that and more, with advanced statistical methods that can help you shave weeks off experiment time, an accessible UI for diving deeper into performance, and out-of-the-box reporting that helps you avoid annoying, prolonged analytic cycles. Eppo also makes it easy for you to share experiment insights with your team, sparking new ideas for the A/B testing flywheel. Eppo powers experimentation across every use case, including product, growth, machine learning, monetization, and email marketing. Check out Eppo at geteppo.com/lenny and 10x your experiment velocity. That's geteppo.com/lenny. This episode is brought to you by Pragmatic Institute, the trusted leader in product expertise. Pragmatic Institute helps product professionals turn ideas into impact through proven courses, workshops, and certifications designed for real-world success. For over 30 years, they've trained more than 250,000 product leaders at companies like Google, Microsoft, and Salesforce, equipping them with practical strategies to build and scale market-winning products. Pragmatic's full-time instructors each bring over 25 years of hands-on leadership experience. teaching strategies proven to deliver real-world results. And it's not just about what you learn, it's also about who you learn it with. Completing a course connects you to an active community of over 40,000 product professionals. You'll engage in meaningful conversations, collaborate with peers and mentors, and gain direct instructor access to refine your strategies and stay ahead of trends. Get 20% off with code Lenny20 at Pragmatic Institute, .com/Lenny. Aparna, thank you so much for being here and welcome to the podcast.

Thank you, Lenny. Thanks for having me.

When I asked a lot of people that work with you what I should ask you about and what I should know about you, something that came up again and again is something that I think most people don't know about you, which is that you're big into stand-up comedy and you take it semi-seriously. Just how serious are you about this? How much of your life is this? And most importantly, how does this help you build better products?

It's hard to say I'm serious about a funny business, but I do watch and do stand-up comedy. I do open mics. I've done a few shows. I have one set brewing that is around AI, unsurprisingly AI and tech and Silicon Valley. It's really interesting for me. This was an accidental discovery. I'd always been an SNL fan and just comedy be fan, but I went to an open mic because, you know, my son sings and he went to the open mic for singing and he's like, Mom, you should go do this. And I was like, oh, let me go give it a try. And I found that I enjoyed it and was good at it. To your question, though, about building better products, I'd say both have PMF. I mean, product market fit, punchline market fit. But actually, there are a couple of things that I, I do find really powerful and useful because you know, in open mics or even when you're testing these things, it's a very tight cycle of iteration and you get live, like open mics are the real live experiments, right? You put something out there, you get very clear micro feedback from users and then you get tough feedback sometimes. And I think as product builders, that's actually one of the great skills to have, which is, yeah, you sometimes launch stuff that, you know, have a fantastic vision, but the first version is not quite there, right? And Reed Hoffman says this, hey, if you don't, launch the first version and are not embarrassed. You're doing it too slow. Just that gap in closing that, it's good resilience.

Yeah, I never saw these corollaries between these two things. I didn't realize you actually did shows and you're working on a set. I wasn't going to ask you for a joke, but if you're working on a whole thing about AI, is there something that you can share from that set?

One joke I'd maybe share is people think about these AI chat products as women because you don't know what's going on. It's a black box. And you don't know what they're thinking. There's like an entire set around that. But obviously on the flip side too that they're probably more like men in the sense that they hallucinate a lot. They kind of are not yet reliable.

I'm afraid to laugh at this a little bit. This is great. Okay.

And even when they don't know the answer, they make up stuff. They're very confident.

This is good. Where are we going to be seeing the show, by the way? Okay. This is great. Okay, let's get serious again. So you worked at most of your career at a lot of consumer internet companies. You worked at Google, Robin Hood, you're on the board of eBay or on the board of Capital One. Now you're at Microsoft. I'm curious just what is most different about working at a company like Microsoft and building product at a company like Microsoft.

I think intellectually I knew that, hey, enterprise, particularly the area that I look at most at Microsoft is focused on enterprise and productivity and transforming companies through AI. And to me, I think two things really strike as very different. One, in fact, I just posted about this the other day saying in consumer, you're kind of like, oh, we have a playbook for make the product work or make the feature work and make it delightful. But I think in the enterprise, you almost have every time you think you have one use case, you have really two, which is how do you make sure that the feature works well and there's governance of the feature, right? If you think about like even something as simple as sharing a link to a document, you want it to be easy, frictionless, but at the same time, you want that to be secure and kind of safe and being able to have auditability and all of those things. And often I find that When you go from consumer to consumer to enterprise, you fall into a trap of either disregarding that and say, oh, we'll just focus on one side of the house or kind of overly crippling the user experience, right? And kind of leaning on the other side. So I think there's an art and science and nuance and playbook there too. So that's one big learning for me. The other learning, and especially in the AI era for me, has been about this I think there's a famous trailer from the 2000s on Van Damme on these two trailers, two buses.

Doing the splits.

Yeah, doing the splits. Exactly. And I feel like a lot of the companies, including the tech companies, but certainly the enterprises that I talk to are in these two modes. But on one hand, this is the most compressed tech cycle that we've ever experienced. It's all in the order of weeks and months versus years and decades. If you think about mobile and cloud and internet. And there's just like so much happening, the intelligence overhang. On the other hand, there's also like humans and habits that productivity habits change, it's hard to change. And change management through the company is also hard, right? You don't want to kind of be rash on that. So it's like, you know, the future is unevenly distributed, but even within the companies.

On the second bucket of this other, the bus that Van Dam's riding on, of governance and adoption and changing behavior and stuff, is there something you've learned about how to get past that, help that along more?

The thing not to do is hold back folks who are early adopters, right? I think that's the other one learning. In fact, I think that's one of the reasons why recently we've been working with folks to say, Can we have both, which is the longer term change management, being able to do it in a trusted way, at the same time, do this program we're calling Frontier Program and roll out cutting edge experimental features. We just built this world's first agent, deep research agent made for work, right, post-trained for work. And of course it has all sorts of edges, rough edges. But if there are early adopters in an enterprise or outside, how can we put that in the hands of those folks? without insisting that all of the company be completely different, developing different muscles.

This program, Frontier, you're talking about, I wanted to spend a little time on it. What is the idea? The idea here is people are working in this futuristic environment. How does that actually work?

Yeah, I think the idea is exactly this, which is I want to institutionalize and operationalize my personal motto of living one year in the future. and say, what does this-- imagine a company or a set up like frontier consulting group or frontier Inc. And if you did lived in that environment where you had all the AI tools and really advanced deep research intelligence on tap, what are the kinds of questions you'd be asking? What's the kind of work you'd be doing? How would you change how you're going about your workday? So that's the premise. and you'd say, hey, how does it change an individual? But also down the lane, we want to think about what does a frontier team look like? We talk a lot about frontier labs and models. I think models layer is amazing and obviously, like, you know, that's what empowers all these product building to happen. But I want to push us to think about what does a frontier product look like? And more importantly, how does a frontier way of working, right? Like, what does a team with three people and tons of like compute and AI tools look like.

So how exactly does this work? There's like a team within Microsoft that's like, your job is to use all of our latest tools and build product using that. Is that how it works?

Yeah, that is the setup. We are just a few weeks into that setup. But meanwhile, what we've done is like we've actually set up in like a company external, like a fake company and said, hey, if you are somebody who wants to come play with some of the cutting edge science projects and deep research agents and agents at work, come party here.

Wow, okay. And it's only a few weeks in, okay, so TBD how it all goes.

Yeah, yeah. And again, like these are micro, that's the meta point here also is that in the traditional way, we've kind of always thought about across the companies, across industries, really thinking about rollouts in these macro ways, right? You build something and you kind of like roll it out, you have a general availability for, and then you take the time. And that's really important too, because again, we're talking about pharma companies, legal companies relying on this. So we do want to have that. But at the same time, given the compressed cycles of AI, how do we start to have people experience what's the one year in the future? Let's follow this thread in a few different directions.

There's how product chain development changes, there's how engineering changes, there's also just agents. I know you're spending a lot of time in agents. It feels like you're not an AI company these days if you're not working on agents. Or building an agent.

Lenny, we're doing this wrong. We didn't use the word agents.

So.

Far into the conversation.

I try hard to push it out as far as I can. It's like every conversation in San Francisco, it's just how long until I start talking about AI. It's like three minutes average, I bet. Oh man. Okay, so with agents, I know that you're leading a lot of this work at Microsoft. A lot of people are wondering what the hell does this mean? What is going to change? Give us just a glimpse into how you see the world being different in a world of agents being around more.

There's a short term and there's a long term, right? There's a lot of hyperventilated, excited talk about the eventual future and all of that. I take a much more practical product building lens on this, right? And I think about these, at the end of the day, they're tools, right? Yes, underneath it, there's stochastic models versus very deterministic programming. You can tell I'm a computer scientist, like the way that that worldview definitely shapes how I think about this. To me, the short term is there's an evolution. We had apps, right? And now I think we are firmly in the assistance era where there's human driving. That's what we think of as co-pilot, right? I think the human driving in the driver's seat, but having a lot of assistance from AI. So I think of this as then you look at the dimension of almost like autonomy and delegation and intelligence as the intelligence, for example, when deep reasoning unlock happened, of course, then you could say you could delegate more to the agent. So I think to me, I think there's one dimension where you say, Hey, agents are somewhat independent software processes, right? That can kind of like run tasks. And you're not just thinking about hand-holding and fine motor stuff. You're saying, hey, here's my goal. Go make this happen. Like, I'll give you an example, right? So we were working on this researcher agent for work. And last night I said, hey, you know, I'm really, I have an important meeting coming up with the leadership team. I really want to present these frameworks here. And this is the roadmap here. go back and look at all the people that are in the meeting, what are their views on this topic? And kind of come up with how I should be thinking about the right persuasion pitch here, right? And what's magical about this is not just that it's saving time. Typically, we think about the SOPAR AI as summarizing a document or saving time, right? This is like fighting synapses that I didn't quite have and are actually giving me new insights. and giving me, dare I say, superpowers. So that's a natural evolution of AI, I would say. So when I think about agents, I think about three things. One is an increasing level of autonomy and independence that you can delegate higher and higher order tasks. Second thing I think of it is complexity. So it's not just a one-shot, hey, create this image or do this thing or summarize the document. Build me this prototype that expresses my idea of an augmented reality app. It's a complex task. And then the third thing I would say is asynchronous. It works when you're not working. I think that's the other big thing about these things that you don't have to sit in front of it.

This answers the question of what is an agent, essentially, these three bullet points. So it's, what are the three again?

When I think about agents, I think about these three things. One, it's autonomy, and it's a spectrum. It's not a 0-1. It's how do I actually delegate things that it can do. Second, I think of as complexity. It's not a one-shot, hey, summarize this document, generate this image, but it's, build me this prototype or help me knock this meeting out of the park. And then the third one I think of is it's a much more natural interaction. That doesn't just mean chat, but it may be actually jumping on a meeting with the agent and being able to talk through all of it or point it to things that I wanted done differently. So I think all three things, the autonomy, the complexity, and the natural interaction, are at least product principles that will shape really good ones, good agents.

That is really helpful. Along this line of agents, there's this acronym that you taught me as we were chatting ahead of this podcast, NLX. What is that and how does that relate to agents? And why are people not thinking about this enough?

Oh, that's one of my Roman empires these days, the natural language interface. NLX is the new UX. So I think here's the deal. To me, I think traditionally we've thought very consciously about GUI because the graphical interfaces are not something natural and so they have had to be explicitly designed. But they're rigid interfaces, right? What we have with conversational interface and natural language is it's a much more elastic, right? That doesn't mean it's not designed. So people had, often I hear product builders say, oh yeah, with AI, like the model leads the product. So it's just you chat with it. You and I are having a conversation. It's a podcast. I'll have another conversation at Microsoft, and that's a meeting. So conversations also have grammars, they have structures, they have UI elements, they're invisible. And so one of the things that I see and I'm really excited about is what are the new principles, new constructs in natural language as an interface? I'll give you a few examples, right? And actually, like a lot of startups as well as big companies are really experimenting with this stuff. One is if you think about it, prompt itself is a new construct. And that's a new way, that's a new UI element, just like a dropdown was or a menu was. But others that are emerging, especially for agents, I think are plans. So when you give a high level goal, what we are seeing is that when the agent comes back with a plan, preferably an editable plan, that's a new construct. The other one that I think about a lot is showing the work, progress, You see this with different products, right? You see with the Copilot, you see with ChatGPT, DeepSea, this idea of thinking aloud and it's kind of showing the work. But how much do you do it? If it's too verbose, it feels like I'm running some crank job and scripts. But if it's too terse, then I don't know if it's going in the right path and I don't have the confidence yet. So there are all these new elements. So if you're a product builder, this is a fun new space to be digging in for product design.

This is really interesting because I think people chat with all these chat bots and it just feels like this is just the way it is, but you actually are designing every element of the interaction, like how much to share, but how much you're thinking. Here's my plan, what do you think? So I think this will surprise a lot of people just realizing there's so much that goes into just designing even these what seemingly are simple conversations. Yeah.

Another good example is follow-ups, right? You could say, look, you have a you asked me a question.