Why Claude can’t be your PM (yet) | Anthropic CPO Panel

Welcome. How's it going?

Good.

Doing great. How are you all? So nice to see you all. Isn't this a great event? I feel like it's just, it's so happy.

Yeah, it's good vibes. Yeah.

So I want to start this conversation off with the question that I think is on everybody's minds, which is a year ago we were told that AI might make PMs obsolete. And now I feel like the vibes are that everyone needs to become a PM. One of the things I love about getting to talk to both of you is just by being at Anthropic and working with the models in the way that you do, you have a little bit of a sense of where the future is going, especially for different job roles. And I'd really love for both of you to talk to us about what does a great PM at Anthropic look like? Is the PM role going away? Is it changing? And if so, how?

I think the job of product is always to be a bridge between the real problems that people have in the world and the technology that you can use to solve it. And like, that's always been true. And I think a thing that is weird right now is that it used to be that the technology would change like every 10 years or 5 years, and now it feels like it changes like every 2 months. And that's really hard. That's just like a, like as a human, I feel like it's really hard to learn how to deal with that sort of change. And so what I always think about is like the technology side is changing, but the human side, like the human side doesn't change that fast. We have the same problems. And so what I always think about is like, how can I be obsessed with like solving the human problems and then stay super fresh-eyed about how the technology works? Like how can I basically throw away most of what I know about what used to work to solve this problem and then try again, like, knowing everything I do about the problem. And so I feel like that part hasn't changed. What I think has changed is it's almost like rewinding the clock. Like, when I— back when the dinosaurs roamed the earth and I started in product, like, we didn't know, we didn't have definitions. Like, we just didn't, we didn't know what the job was. And so you're just like a general problem solver and you'd like walk around and try to solve a problem until you like ran into a wall, and then you would ask somebody or figure it out. And I feel like that's what we can do now, is go back to a bunch of the roles blurring, but like, you just keep going and try to figure out what's going to work. And then when you hit a wall, you have all these different sorts of scaffolding.

Yeah, that makes sense. Mike, anything from you?

I think for me, I had a real, like, sort of realization moment recently. So earlier, beginning of the year, shifted to an IC role, so mostly building. And we were getting close to shipping something, and recently, And one of the PM leads who works with Ami pulled me aside and was like, Mike, I know you love to build. This project's going well. It's looking really exciting. We really need a PM on this project. I was like, I don't know. Do we need a PM? Claude's got it. We've got a lot going on. And she's like, no, we really need a PM. And it's such a reminder. Once they joined, all of the things and all of the glue and all of the connective tissue and all of the things that were about to get dropped if I'd not done that, that would have happened. And it was just such a cool eye-opening before and after, so much so that a week later, and I was like, Kat, you are 100% right, and thank you. And it was just— and I think about all the things that that encompassed. It was, you know, actually bringing along— we're a company that serves everybody from, you know, prosumers all the way to really large enterprises. Like, have you brought all of that contingency along, right? When we need to go and figure out how to enable our customer success team to, like, talk about the changes as questions come up in real time. Has somebody thought about that? Has somebody looped in safeguards? It's obviously very, very important. Are people getting kept on track? All of these things. And nobody has infinite capacity, even with the assistance of these tools. So to be able to have, like, call it what you will, but somebody playing that role of making sure that everything is going to go well, everything is going to get connected. We've kept the end user needs in mind, Tommy's point. And that the builders who are like heads down and are like, you know, in full-on Claude mode can keep doing that without having to pop out and do the very different activity, which is how do you actually make sure all these things go well is, I think, very essential. I think all that's happened now is that because we can move faster, that role is like, you have to be operationally excellent in that role in a way that even beyond what you had to be before, but it definitely has not gone away. In fact, I have felt this renewed, like, yes, this is a, if you're recruiting for a team and you need people to wear these hats, this hat is increasingly important.

Can I ask you a serious question though?

Only serious questions.

I'm so curious now. Yes, this is serious. Why didn't you just have Fable do it? Like, what is— and presumably there's some like Fable 7 you've got access to that we don't have access to yet. So like, what is that person doing that you couldn't just have Fable do?

I mean, it is interesting. Like, we have, as you can imagine, we like run on Claude Tag, and we have a lot of like, you know, Claude-provided connective tissue too. And it does catch things where you say, Hey, there's another thing going on in this other part of the organization that you should be aware of. We found it via the Slack search. So there is that piece as well. I don't think Claude is a convener yet. I gave a talk a couple of weeks ago on what's the role of the human in the post-super-powerful AI age, and there's a bunch of archetypes I was thinking about. But one of it was the convener. Somebody still needs to bring the Claudes and the people together to get the work done. And so I think that is a role that Claude doesn't have the organizational pull or the scheduling ability. I've yet— this is going to be a moment, Dan, when Claude schedules a meeting with me or for me where I'm like, who scheduled this meeting? It's like, oh, Claude did. I thought you two should talk, chat. That has not happened yet, but not because it can't. I guess we just haven't like turned on that setting, you know?

I—

Stay tuned.

I think we're also living in a world where like the tools are really good and you can build a lot of things and you can build them really fast. In some ways, that just like expands the universe of what is possible to build, and you still need to have really good judgment about like what to build based on sometimes pretty limited information, because it feels like everything's moving really fast. So you have to like figure out what to build, and then I think you, in some ways, you have to be even more relentless about making sure it happens, because there's so many forking paths that you could take.

Yeah.

And so in some ways, just like the universe keeps on expanding, and it, It does take somebody to be like, I have an idea of what we need to do, and I have the feedback loop to the people who are going to use it, and I know what the human problems are, and I'm just going to go for it. And I think that sort of relentlessness and judgment in the face of ambiguity is really important.

That is definitely something that I have found every time there's a shift, every time new models come out. It's sort of like you're looking out on the ocean, you're like, what's beyond the horizon? And then you get to fast forward beyond the horizon, and what do you see? Well, there's more stuff. There's a whole new landscape. It's not like a utopia, and it's not horrible. It's like there's just lots of new things to discover and navigate. I think the same is true for just models in general. When we were talking, Ami, and planning for this conversation, one of the things that you said, and you've had a long career in product management, but prior to Anthropic, is that you spent decades learning how to answer specific kinds of questions as a product leader that you now no longer need to answer. And so I'm curious about, okay, what are the skills that you spent such a long time building that you actually don't feel like are that useful anymore? And what are the things that you're leaning on now more? And then how does that filter down into the team and your expectations for people that you hire?

I think it's really true. Like, I spent a really long time trying to understand how to put myself in the mindset of a user and how something would fit into their lives and like what we should build because it was so expensive to build and ship something. So, you know, literally this would happen where people would have reviews and what we'd talk about is like, Ami, where should we put this button? You know, and we would talk about the interaction pattern and where like someone's finger would go on their phone and what situation they would be in, whether they pulled their phone out of their pocket and what they wanted. And I feel like I tried really hard to get good at that, at that mix of like product and design. And I just like, I don't think anyone's ever going to ask me that question again.

That's a fable. Yeah, that's in the past.

Like now it's actually faster. to just build 3 versions and try them out and be like, oh yeah, this works, this one doesn't. So a thing that I prided myself on is like not actually that useful anymore. And I do think that's really hard just in terms of identity. We talk about the innovator's dilemma when it comes to companies where like you're so good at one thing, you keep on trying to do it even when there's this other thing that maybe you should try doing, but you can't because you're already so good at one thing. And I feel like I think that's also true on a personal level where I sometimes feel like, oh, I know what I'm good at. I'm just gonna keep on trying to do that even when there might be a different way to do it that might be more important in the future. And so that's what I just encourage everyone to think about. Like when you think about that product bridge, part of the technology changing every 2 months is you have to throw away what you knew about yourself in the job using that technology. and just try something different. And most of the time, I don't know if I'm going to be any good at it. You know, like, I've never worked in a model company. Like, I just— this is not a place where I had expertise. And I just kind of feel like every couple of months, I have to constantly be like, all right, well, let's try this other thing and see if I can figure it out, you know?

What are the characteristics of someone that is able to do that well, and what have you had to develop in order to do that well?

I mean, I think it comes back to the sorts of things we were talking about. I think, like, raising your ceiling on tolerance for change and being willing to embrace change is super important. Like, so I think adaptability, I mean, I'm sure this is not new, like adaptability is just such a key thing given how rapidly the world is changing. I think that sense of like judgment and relentlessness as we talked about, those are like the things that I keep coming back to. And really what I try to think about for the team is like, I think it's important to talk about those things because otherwise, it can feel really lonely. Like, you're the one who has to constantly adapt, but actually everybody has to constantly adapt, and I think it's hard on all of us. But I also talk about trying to frame the chaos so it doesn't feel as taxing.

Hmm.

You know, like, I think, I think when there's times of uncertainty, sometimes it's tempting to, like, try to map out, here's exactly what's gonna happen. Like, here's exactly the products we're gonna build. Here's exactly how my career is going to go, you know, like, and that's a way to exert control. And I just, I think that locks you out of trying all the new stuff. And so what we have to do is, like, frame the chaos so it feels safe and it feels plausible to engage with it. But that's where, like, a bunch of the magic and productivity is going to be. So just, like, talking about it, making it safe, acknowledging that there's also emotions around all of this, and it's not easy, but it is a way to just participate and understand and work with the future. I think that's something we talk about a lot.

I think there's an organizational leadership question that I think, Ammie, you do really well, which is like you can have the chaos, but you also need very clear leadership and DRI-ship within some areas. So like you can have a bunch of people sort of exploring something very frontier, but having a clear like this is the person who's ultimately going to make the call whether this is, you know, the thing to pursue or not. In labs, we call them leads. So like We call them bets within Labs. So the bet lead is a really important role, and they are the DRI for saying, I think we should double down on this, or actually, we should wind this down. This team needs more people. This team needs less. And that role is even more important because we're moving so quickly, because things are shifting really rapidly, to still have that— we're all lost together, but somebody has got the pen on what's the next most important thing we can do to de-risk, to understand, to learn, to move things forward to.

We've talked a lot already so far about about the psychology and the skills of being a product leader. One other place that's changing a lot is what you actually build and how you think about the features and the functionality, in particular because we're moving from a world where you'd expect only humans to use your software to a world where humans and agents are using software. Sometimes it's delegated, sometimes it's fully collaborative. I know, Mike, you've been at the center of figuring out what does agent-native type software look like? I think you have a lot of thoughts on where that is going. Can you talk to us about how you think about what to build? What goes into a product versus an agent, and what's the overlap?

I love this question. Maybe there's a timeline to look at where V1 or era 1 was, great, we've got this AI thing. Maybe it belongs in a sidebar or a little mini window, and you can ask questions, and it's disconnected. Maybe it helps you with some customer support questions. Then greater integrations where maybe there are whole features that are AI-powered. Then this movement towards being more agent-native. I think Every literally wrote the book on it, which is— the way I— for the whole year where I was working a lot on this idea of agent-native architectures, I would basically feed their write-up on this and then create a skill out of it because I thought it was a really good encapsulation of it. Just the idea of everything a human could do, an agent should be able to do as well. Very few products actually get this right. I think a lot of our products don't quite get this right yet. But when you do that, all of a sudden, a lot of emergent behavior gets unlocked where the agent can piece together things that weren't pieced together before, or it can proactively offer different ways of doing things. I think we're still on the journey of making that good, but I think the next stage that I've been thinking about a lot is when the interfaces actually become something that is malleable by the agent too. And we've talked about malleable software for years. I feel like it's actually now, like, really manifesting and becoming real. And maybe back to the sort of role of product and connecting to this question, one of the things that we have, we have, like, a pretty complex thing that we're trying to ship right now. And it's got at least 4 independent work streams. And the way we've operationalized it is we've had Claude both sort of keep an eye on how everything is going, but also create the UI by which first the TPM and then the entire team understands what is happening in that project. And What's really cool about that is that if you didn't like that particular display, you're not bound to, like, however it was determined to be built, either by some internal acceleration team or some sort of third party. You can change it, and you can change it along with Claude. And it's just been a very cool thing to see internally. Like, one of the biggest shifts in how we've worked in the last year, I think, at Anthropic is so much of the software we interact with is, like, built, maintained, and iterated on by Claude, of course, reacting to all the things that are happening in the organization, but very much, like, changing all the time. And so that leads to all sorts of other interesting questions, like, who can update the data, right? Is it just the humans clicking through? Is it also Claude in the background? How do you have like provenance of data? It opens up some like interesting questions there as well. But I feel like that is the piece, like the, both the trend and the thing I'm most inspired by right now is how do we make that the way almost all of our software operates? And of course you still want building blocks and predictability and design systems and you don't want, if you wanna frame the chaos, you wanna like make it not absolutely insane, but for the given task, for that given project, for even that given user potentially, what is the way in which we pull in the right ways and make that really feel both personal and extremely useful because of that.

How does someone think about maybe you're not in Anthropic and you have a SaaS app that's scaled and you're thinking about, okay, do I put an agent in it? Do I make it open and agent-native so that Claude can be in it? How do you think about those choices and what a company should care about versus leave to a model company like Anthropic to plug in?

I think a lot of it is really building the right primitives into the product at almost like an infrastructural layer. You can tell. I always poke at— whenever I'm using new products, I'll ask it to do something that a human could do, but it might require somebody to think through or architect the thing where everything goes through some shared plumbing that both agents and whatever REST API you've built can go through as well. And you can tell when they've been thought through that way. And I have a lot of empathy because if you've been building an application for 20 years, just bolting that on is difficult. So I think it's a journey that a lot of companies are going on. But once you get it right, I think that question becomes, you can just evolve it over time, right? You can start with, okay, at first I'm just going to have maybe more of the side painting. I don't— I'm not ready to blow up my whole UI. I'm not ready to create fully generative UIs, but maybe you start experimenting with like, all right, there's also maybe a personal landing page that can be more malleable and is using the same primitives. You're not inventing a whole new sort of infrastructure underneath. So I think that you can evolve your product along the way, but you have, like, you can tell when things feel super bolted on and like, oop, like, That's feeling like it's not a native part of the product.

So, Ami, I'm curious. We're hearing— we can go all the way to agent-native malleable software. But Anthropic, you have to serve this whole spectrum of people. There's this challenge of pushing the frontier. And also, some people just want to use Claude to chat. Or you have big enterprise contracts where you can't just rip out their whole interface and then say, hey, it's malleable in one day. Or maybe you can, I don't know, but I assume you can't. So how do you balance, in terms of thinking about product, pushing the frontier and creating a consistent experience for users that they can understand and bringing along the people who are maybe not going to try all the new things on day one?

I think it's a really hard problem. I mean, our approach is to just try to meet everyone where they are. I think we're just so early in understanding what works. It's not like stuff is stable and we can just build the right answer. It's like we're all trying to explore together. And part of the everything changing every couple months is like the models change, but also like the way people use products and the models changes. And we kind of constantly have to update for that. And so a thing that I think about is like when we're just exploring a product like Mike's talking about, it's okay to just try stuff. Like we just have to try a bunch of things because we haven't actually explored the frontier. And then when we figure out what works, then we have to like make it make sense for all the people who want something that feels more stable and complete. I think this is a really tough trade-off because, you know, like raised in like the product like world that I was, I was all about like simplicity and reliability and total predictability because I think that helps people build like a muscle memory and an intuition But I also think the reality is like, we just don't know yet what a lot of things should look like. And so I actually encourage the team to just like take a lot of shots on goal. You know, it's okay to have things that feel somewhat overlapping or duplicative, as long as when we figure them out, when we hit product market fit, we go back and we reincorporate into something that works for everyone. Because I feel like the fail state is you decide from the beginning, like based on a theory, of exactly what needs to happen, and you over-constrain a product. I would much rather have 5 overlapping products that all work in different ways than one overly constrained one that we never even gave a chance to actually meet the market.

How do you— I definitely see this. We do this at Every, but I see this all around the industry, just doing lots of parallel experiments and seeing which one works, and then starting to consolidate as you start to get get signal. How do you do that in a way that doesn't feel confusing or incoherent? Or if I'm working on the 3rd team doing this thing, it can be maybe demotivating. How do you do that well?

I think there's a couple of things. One, I think when we were just having this conversation this morning internally where there were 2 potential product directions, and one has a team with very high conviction going after it, and the other one doesn't really. And it feels very manufactured to be like, we should explore this other idea because we think it's good, but nobody's really calling for it, then that's like that kind of B-team feeling that I think I've never been like successful tasking a team to go like take a product area that they're not excited about. Like the product is going to suck at the end of that process, right? So like you want, you want the like excitement. But if there are 2, 3 different teams that have exciting directions, then yeah, like let that, let that explore. I think, I think we've made a lot of progress on even the last 6 months is like having the underlying infrastructure to support all of that. So we inventoried this, feels like 6 months ago, it was probably only 3, but there was a point where Chat and Cohort had different memory systems and different MCP and different ways of storing files. And it was like, okay, for anybody to experiment on top of this surface, they're going to have a really hard time because if they built a third thing that then doesn't share memory with any of those, for example, it's going to feel disconnected and almost disadvantaged from the beginning. So a lot of the work, we have a foundations team that has been just thinking about, you should be able to get your memory wherever you are within our products. And it sounds like a simple thing, but then it's Very clearly, you pull back the thing like, oh, it's complicated for these 8 reasons, but it's worth solving that and solving it in a different team than your labsy innovation or frontier product teams, because then those experiments can feel complementary even if they are overlapping too.

I also think this sounds so simple, but I also think it's just really important to talk about it and just name like, hey, if there's 3 different bets, sometimes it can feel demoralizing or like one of you is not set up for success. That's okay. That's the world we're living in. And so what we're asking you to do is come along this journey with us. We don't know the answer. We don't think anyone knows the answer. We're all going to discover it, and we're all fundamentally on the same team. And so we will just learn from whatever works. And just saying stuff like that out loud, I think, is really important because everybody is trying to figure everything out in the dark.

What do you look for to know that something's starting to work?

I mean, I think a bunch of that is kind of the normal product-market fit. It's like, do people use it? Do they like it? Do they come back? Do they tell their friends? Do they get value? Like, again, going back to like the goal of product has remained the same. It's like to solve problems. Thinking about does this actually solve a problem in a way that people can use and use easily and feel good about using? It comes back to the basics for that.

One of the interesting things that feels challenging to me about that is sometimes a thing looks like it doesn't work and the new model comes out and then it really starts working and you don't want to like throw it away too early, but you also don't want to be too far ahead. So how do you do that?

One of the things we do in labs sometimes is just park a project. When we built the first computer use product inside Anthropic in 2024, and it was so bad because the models just weren't there yet. Our first thesis was, well, it's not going to be good at a bunch of automation, but it might be good about education. So you'd be like, all right, how do I change this thing in some complex software? How do I do this really complex thing in Photoshop? And it would get there, but it would get there in the very early agentic Claude way, which was like, Did that work? Nope, that didn't work. I got to close this. It was like 20 minutes later, you're like, I did it. You're like, I learned nothing. It was neither good for automation nor education. But we parked it, and then we would basically just try it with every new model. The way we would try it is we actually just had it running in an eval harness. The way we actually learned that 3.7 was a leap in computer use was just seeing it suddenly take over. Then we actually looked at the transcripts. We're like, wow, it's succeeding more often than it's not. This is a real moment. It is really valuable. We see it as a win in labs if all your project did is show that the models aren't ready for it yet, but you're able to now externalize that as an eval or create some connectivity to the research team. That's great because it means that 2, 3, 4, 5, 6 months from now, we'll be able to revisit it. And hopefully, we've learned something along the way as well. But I think you need to build early so that you can even have that intuition.

I definitely agree. I've been calling this It's, it's cert— there's a sort of, you can get capability blind if you try something and then you don't ever try it again, or you don't make an eval and you're like, oh, then a model's just never gonna do this. And then it just turns out a model comes out like in 3 months and it just completely changes. It, it, it improves in ways you never would've thought it would. I'm, I'm actually curious if you're here, how many of you have used Fable? Raise your hand. Oh, a lot of people are using it. Okay, this is great. This is a good— what about Astra?

Fewer.

Fewer. Interesting. Okay, so this is great. You guys are, you guys are on the, you guys are on the edge. This is, this is very helpful because one of the things I find a lot is I talk to someone and they're like, oh, it doesn't do this. And I'm like, have you used the latest model? And they're like, no, I use like Sonnet 4 or 5 or whatever, like whatever. And, uh, that, that really changes things.

I think that's also part of why it's so important to be willing to throw away everything you know.

Yeah.

Even though it's hard, because then otherwise you just like think that the last thing that happened is going to be true forever.

One big question that I see you guys working on from afar, and I see this happening at OpenAI too, and we run into it a little bit, is when you're doing all these parallel experiments and one starts to really work, then you have to push it back into your actual product. There are several different ways to do it. It's like, okay, you can make a tab, but then you have a bunch of different tabs and there's all the things only work in slightly different ways and that's ugly. Or you can have it just be a separate app and you never merge them. There's all these different things that I don't think anyone has really come to, this is the answer, this is how you do it, but it seems like a critical problem is how to take those parallel experiments and turn it into a coherent product. How have you, what have you learned and how do you think about it?

I don't know what you're talking about. Um, you're right, I don't think we've fully figured it out, but I think some of it comes back to what Mike was talking about of making sure that we're building primitives so that our products feel like the way they should feel, you know, when you're using them. Like, what does a user expect? The user expects to walk up to something and like hopefully not have to think about it too much. Like, and right now it's easy to just transfer a bunch of cognitive load to the user and be like, here's some tools, like, why don't you choose? And so one of the things we, that, that Mike's talking about is like, how do we make sure that we build primitives so the right system knows you? So a UI that feels like really accessible all the time.

Yeah.

And I think all of that is a bit iterative as we see how people react.

I think overall, like, if I had to bet, like, there's some lessons you'll learn and some that are real. Like, the thing that we always saw at Instagram, it's like there's a very strong power law in tab usage. It makes sense. Every single product, right? It's like main feed was like 80% of the use, right? And you can make Explore better and maybe you would go from 10% to 15%, but it was like main feed, that is where you land. That is your big thing. So I think that is— will always be the journey that we're all on, which is like, what is the main What is the main tab? But what is that experience? How do you still make that great? And then luckily, it can be an entry point to a lot of other downstream interesting experiences. But yeah, I think half of the job is saying no to another thing in the sidebar or another tab or another thing and figuring out, can this be rolled in once we've proven it out, but not killing it so early? So this is the entire, I think, art of product development, I think, as things are moving this quickly.

All right, we've got time for one last question, which I'd love to hear from both of you, which is, if we are on this stage here next year and Lenny invites us back— I don't know if he'll invite me back. been cursing too much, but I'll invite you two back. What do you expect will be different? And, and how will being a PM have changed?

I hope it just like that, that people feel like they can just do much more. Like they can just go and build in whatever way they want. They can get answers whenever they want. They can start businesses in like, and we just see a lot more like small teams and solo empowerment. And we, we just, we start to get the uplift of like what that means for everyone else. I think it's always like builders magnify impact for a bunch more people who will never use the tools directly, but who can benefit from people who do build things. I hope that's just way easier.

I think the biggest one for me is right now, I often talk about the gap between what the models are capable of and what most folks are using them for, not through any fault of their own, it's on us. We got to build the products to make that possible. That was a hard thing last— it was a hard thing in '24. It was even harder last year. It's getting even harder now. And so the biggest thing I want to do is close— I think it's very democratizing if you can close that gap because it means it's not just your most clod-pilled software engineer that has this fully working multi-eugenic setup. It's also the person that is doing a bunch of research in parallel. It's also the person that's trying to manage their business, but they feel like they've unlocked it and we've brought them along the way in terms of how do you understand the moving parts? check in at the right places, do long-horizon work, but not in a way that feels disempowering. Like, that's the art. Like, if we succeed a year from now, that'll be closer.

Amazing. Thank you so much, Ami and Mike.

Thank you. Nice to see you.