Exclusive: Inside the Best AI Model for Coding and Writing | Scott White (Anthropic)

I think we're still just getting started. I think, you know, this last year, Claude sort of feels like a very capable assistant. Like something that you have to guide it to a specific outcome. You have to know exactly how to prompt it and give it very specific information to get what you want. I think in the next year, we're going to. We're going to flip the script a little bit, and I think it's going to feel a lot more like. Claude is what I like to think of as a capable collaborator. And to me, that means it's moving up almost a career ladder on a few dimensions. Can it actually take things off my plate? Can I feel like I can delegate meaningful tasks and meaningful work to this capable collaborator as opposed to, like, telling it exactly what it needs to do? That's what I think things should start to feel like this year. And it's not just like one feature that will do that. I think it's the confluence of many capabilities, both model and product, all moving in that direction. But I think like means that it's on the dimensions of its knowledge, its ability to communicate, and its ability to do stuff for you, making, like, really meaningful strides this year so that it doesn't feel like an assistant anymore. It feels like something that's helping you solve your biggest problems and most pressing timely needs and meaningfully taking things off your plate. That's how important for us to dog food our own, you know, early generations of these models and figure out what are the things that are blocking us from being successful. One of my old bosses was Andy Ratcliffe. He's like a professor, and he was the CEO of Wealthfront. And he used to say, the only thing that matters is if the dogs are eating the dog food. And, well, like, we're sort of the dog, and it's critical that we want to eat our own dog food. And so that's been an incredibly helpful thing for us to, like, ground our model in the problems that we practically want to solve and also working very closely with our customers on that. So having that feedback loop between go to market product engineering stakeholders with deep research involvement through the entire process helps us pipeline these learnings and sort of like, match it with the vision of where we're trying to go. Just the mechanics of how you train a model end up being the key difference. But, like, fundamentally, the overall process ends up feeling like product development just with the new nuance that we control this sort of model that also integrates into our product. And it's just a new way of.

Working this episode is brought to you by Vanta. Whether you're a startup founder navigating your first audit or a security professional scaling your GRC program, security has never been more critical or more complex. That's where Vanta comes in. Businesses use Vanta to establish trust by automating compliance needs across over 35 frameworks like SoC2 and ISO27. Vanta can help you start or scale your security program by connecting you with auditors to conduct your audit and set up your program fast. Plus, with automation and AI throughout the platform, Vanta gives you time back so you can focus on building your company. Join over 9,000 global companies like Atlassian, Quora and Factory who use Vanta to manage risk and prove security in real time. Get $1,000 off@vanta.com Peter. That's V A N T A dot com Peter for $1,000 off. Now back to the episode.

Yeah, I really believe that like, you know, like rapid feedback loop is key like with your customer, like doctoring product every day yourself. And like I feel like a lot of PMs don't do this man. They just, they don't use their own product, they don't talk to customers. I mean, I guess with Claude it's.

Like a no brainer. Like that's the most important thing.

Yeah, it's the most important thing. And like if you're not using cloud or one of these other products, like you're basically not, like you're not being very effective as a PM at this point. Like you really should use this, this stuff to become way more effective. Right? I mean it's like, yeah, I, I, I use it for everything.

Yeah, I mean I think that there's like using your own product and then there's using, you know, new technology that can help you be better at like the, the core role that you're doing. But like a fundamental thing. I think every product manager should be really great at and, and prioritizes being the expert on their own product. And for us the product is both what we're building in terms of the user interfaces and the APIs, but the product is also the model. So we of course need to be like on the front lines of using our own model and being like what's blocking us from being successful. So I, I think that's really, really critical and that's the beauty of working at a place where you are the customer of your own product. Like you can just have those feedback loops so rapidly.

So like using Claude for your own job, like, like what are Some most common use cases for like the normal Claude versus the reasoning Claude, you know, like, how. How do you think about that?

Yeah, yeah, that's a good question. I think like the normal Claude, without sort of extended thinking, I use a lot of document, sort of document creation, document summarization. A bunch of the things that I do around creating PRDs, creating evals and based on the PRDs and implementing those evals into the PRD, these are the kinds of things that I Claude's really good at. Without extended thinking, though, there are some times where I will use extended thinking. Even with that, there are instances where I have like a project that I've set up with all of the sort of product requirements templates, documents, and like the eval structure that we we do. And sometimes I'll. I'll like dump in a ton of different documents about like a particularly hairy large initiative and I'll use extended thinking to try to like connect the dots a bit between all of these things. And I have noticed that that helps. But a lot of the sort of daily tasks of summarizing meeting notes, identifying action items, creating documents and PRDs, these are things that I can do without extended thinking. And then I think an area where we have noticed a lot of value on extended thinking is like coding use cases. And we built a product that we launched as well this week in a research preview called Claude Code. And that's something that's become like, integral to the way that we're working internally. And a lot of people have noticed the extended thinking being valuable in that, in that use case, in terms of coding, we like, it's. So that's a good example of where us being our own customer has helped us ship something that we think is also exciting to the market. But it has become so indispensable to us and it is changing the way that we work. And it's cool because like yesterday even I was at my desk and I noticed a bug where we had like a really bad contrast on like a toggle in the ui. And there was a designer who's just sitting like three seats away from me. I just walked over to her desk and I was like, this is an issue. Like, can we quickly choose a color that we want to actually like, have the toggle so we fix this contrast issue and we like put out a fix in under a minute in a way that we just like that that wouldn't have happened even six months ago. So, like, these are some of the like, practical ways that things are changing internally. The other thing the last one that I'll mention is like visualizations have been actually surprising. So there are some instances where you're like, we really want to make it good at this thing. And you like point all of your sort of like efforts towards doing that. And then there are other times where you just end up getting surprised after the how you train a model end up being the key difference. But like fundamentally the overall process ends up feeling like product development just with the new nuance that we control this sort of model that also integrates into our product and it's just a new way of working.

This episode is brought to you by Vanta. Whether you're a startup founder navigating your first audit or a security professional scaling your GRC program, security has never been more critical or more complex. That's where Vanta comes in. Businesses use Vanta to establish trust by automating compliance needs across over 35 frameworks like SoC2 and ISO 27001. Vanta can help you start or scale your security program by connecting you with auditors to conduct your audit and set up your program fast. Plus, with automation and AI throughout the platform, Vanta gives you time back so you can focus on building your company. Join over 9,000 global companies like Atlassian, Quora and Factory who use Vanta to manage risk and prove security in real time. Get thousand dollars off@vant.com Peter that's V-A N T A.com Peter for $1,000 off. Now back to the episode.

Yeah, I already believe that like you know, like rapid feedback loop is key like with your customer, like dog friend the product every day yourself. And like I feel like a lot of PMs don't do this man. They just, they don't use their own product, they don't talk to customers. I mean I guess with Claude is.

Like a no brainer. Like that's the most important thing.

Yeah, it's the most important thing. And like if you're not using Cloud or one of these other products, like you're basically not like you're not being very effective as a PM at this point. Like you already should use this stuff to become way more effective. Right? I mean it's like yeah, I use it for everything.

Yeah, I mean I think that there's like using your own product and then there's using you know, new technology that can help you be better at like the core role that you're doing. But like a fundamental thing. I think every product manager should be really great at and, and prioritizes being the expert on their own product. And for us, the product is both what we're building in terms of the user interfaces and the APIs, but the product is also the model. So we of course need to be like on the front lines of using our own model and being like, what's blocking us from being successful. So I, I think that's really, really critical. And that's the beauty of working at a place where you are the customer of your own product. Like, you can just have those feedback loops so rapidly.

So like using Claude for your own job. Like, like what, what are some most common use cases for? Like the normal Claude versus the reasoning Claude. You know, like how, how do you think about that?

Yeah, yeah, that's a good question. I think like the normal Claude without sort of extended thinking. I use a lot of document, sort of document creation, document summarization. A bunch of the things that I do around creating PRDs, creating evals and based on the PRDs and implementing those evals into the PRD, these are the kinds of things that I quad's really good at. Without extended thinking, though, there are some times where I will use extended thinking. Even with that. There are instances where I have like a project that I've set up with all of the sort of product requirements templates, documents, and like the eval structure that we, we do. And sometimes I'll, I'll like dump in a ton of different documents about like a particularly hairy large initiative and I'll use extended thinking to try to like connect the dots a bit between all of these things. And I have noticed that that helps. But a lot of the sort of daily tasks of summarizing meeting notes, identifying action items, creating documents and PRDs, these are things that I can do without extended thinking. And then I think an area where we have noticed a lot of value on extended thinking is, is like coding use cases. And we built a product that we launched as well this week in a research preview called Claude Code. And that's something that's become like integral to the way that we're working internally. And a lot of people have noticed the extended thinking being valuable in that, in that use case, in terms of coding we like. It's. So that's a good example of where us being our own customer has helped us ship something that we think is also exciting to the market. But it has become so indispensable to us and it is changing the way that we work. And it's cool because like yesterday even I was at my desk and I noticed a Bug where we had, like, a really bad contrast on, like, a toggle in the ui. And there was a designer who's just sitting like, three seats away from me. I just walked over to her desk and I was like, this is an issue. Like, can we quickly choose a color that we want to actually, like, have the toggle so we fix this contrast issue and we, like, put out a fix in under a minute in a way that we just like that that wouldn't have happened even six months ago. So, like, these are some of the, like, practical ways that things are changing internally. The other thing, the last one that I'll mention is, like, visualizations have been actually surprising. So there are some instances where you're like, we really want to make it good at this thing. And you, like, point all of your sort of, like, efforts towards doing that. And then there are other times where you just end up getting surprised after the thing has baked. And visualizations is kind of an example of that. It's kind of intuitive because Claude is very good at coding, so it can code good visualizations, but it's kind of surprising to us. So now there are a lot of ways that both from a product perspective, I think we'll capitalize on that and try to ship stuff that leverages that more. But also internally, in my use case, I end up building a lot of visualizations using extended thinking to, to try to make it clear, like, what the user flow that I'm trying to, to work towards is in a product spec. And that's been a new way, like, something I wouldn't have done. You know, two years ago I wouldn't have had the time to do it, but now I do it, and I even see engineers doing it because they're so passionate and there's so much bottoms up energy here. They now have this new tool in their toolkit to communicate their vision to me and to designers. And that helps us, like, closed feedback loops between us internally on what we want to build.

So when you say visualizations, you mean like, like, like flowcharts, like user journey charts, or like actual mocks and stuff?

Or actual mocks, like both. Both, like in terms of, you know, mermaid diagrams to show what a user flow should look like, but also like full react components to be like, this is what I actually think the page should look like. And like clickable prototypes of going through those experiences. I've been very surprised at how good Sonnet 37 is with extended thinking at that.

I, I, I do that a lot too. I'm, I'm not sure how my designer counterpart feels about this. So usually what happens is like, you know, there's a Figma design and then I'm like, hey, you know, I, I, I plug in some AI tool, I'm like, hey, can you make it better? Can you improve this? And I, I show it to her and then she's like, it's like, this is a great job, Peter. I'm, I'm not sure, like, how, how, how does your designer feel about PM's kind of sharing the mocks or prototypes?

Yeah, I mean, we're all super collaborative here. And I think, like I was sort of saying earlier, there's this interesting intersection. As people have more of these tools, more of their sort of workflows overlap with each other and give them new ways to collaborate. Like, we have way more designers putting out pull requests for, like, things that they just want to, you know, improve from a quality of life standpoint or a craft standpoint. In the product experience, we have more engineers and product people like creating visualizations. And so there are just these new touch points. We have more like salespeople basically writing product specs because they are hearing feedback directly from the field. They have access to all of our templates. So there are these touch points that exist that traditionally wouldn't have. And I think, like, we're, we're learning to embrace them as opposed to be being like, defensive. It's like, these are just new ways that we can collaborate. And I think it's actually quite fun. Like, I get inspired when I see engineers create visualizations. It even happened this morning. Like, before coming into this discussion here, an engineer put out like a really compelling output, like mock of something. And I think we might ultimately go and make some product decisions based off of that later today. So, like, these are just new ways of us working together. And they get me super, super excited.

I totally agree. Like, I think, you know, I think Scott Belsky, he used to be at Adobe, said the talent stack is collapsing. It used to be these product teams have all these roles as a pm, you can only write internal documents or something, but now everyone can do a little bit of everything. I mean, that makes the job so much more fun.

Yeah, absolutely. And I think there's a component of, I am not going to be the production engineer who's actually getting everything down the stack and creating the full flow, but I can create a small prototype of something just to get the product ball rolling. I think of like, static friction versus rolling friction. Right. And static friction is so much harder to Break, and then once you, like, break through that, there's so much less friction in, in, like, going farther down. Which is why often when you're like, creating a new company or something, like, even just creating the website, like, even creating, like, the landing page helps you get started. I think these new tools just help us get started and build momentum on things faster between teams. So while the talent stack is collapsing, I think it's not that, like, we are all doing all of each other's jobs and everyone's, like, doing the same thing. It's more. We have more ways to break through the static friction. And that makes me super excited.

Yeah, I totally agree. Keep the ball rolling downhill, basically. Yeah. Makes sense. So let's talk about the model a bit more. There's been a lot of talk about how all this model stuff is becoming commodities, and then a week later someone ships a great model. Right. But, like, I actually think what's really unique about Cloud is, like, its personality and, like, it just feels like more nice and more pleasurable to talk to than some other models. How do you think about this commoditization thing? Or like, you know, how do you think about giving the model personality?

Yeah, yeah, good question. So I think there, there, there are two components of that and they do overlap in some ways. So I'll go through both. The way that I think about, like, the commoditization question, I think the last couple of years have been. There's been a lot of experimentation. Like, businesses are trying to figure out what's the best way to deploy AI in what parts of our stack are we buying? Are we building, like, what. What is sort of our overall strategy? I think that they've sort of oriented themselves now and they're savvier about how to deploy models at different tiers of the product experience and internal stack experience. And I think, like, overall, what I've seen is businesses are taking like a multimodal approach versus choosing just one. Like companies who build internal tools, one thing that we've seen is the general behavior looks like they basically build this tool and they have like a. Their own model selector, which you can access, you know, through things like AWS and Vertex. And then that enables their employees to switch between models as they need, choosing whatever is the best model for that particular use case within the internal tool. But ultimately there are also just use cases that different models are really amazing at. Like, software engineering and coding is one that we continue to be really excited about. And a lot of companies will adopt Claude for work or, you know, Now CLAUDE code or through a, you know, a different solution like Cursor and end up using our model for software development while they might be using other models for other use cases internally. And then there are also situations where there's kind of like a cost component of this where we have a model family, right? Like we have Claude Haiku and we have Sonnet and we have Opus. There are different use cases that they have internally across different models that leverage different performance and cost thresholds. So we, we've just noticed there are many vectors for choice. Like we also build products, not only just make our models available through APIs but where there is already a lot of choice. We also have new product experiences through Claude AI and CLAUDE for work that people really love to use that are of course built on our own models. And so when you're deploying Claude code, you're using our models to write your code. When you're adopting CLAUDE for work for sales use cases or writing marketing use cases like you're using our models there. And that creates a lot more of a customer relationship and stickiness. There's also a lot of companies that are adopting both at the same time in sort of a more holistic partnership with us. And those are really great customer relationships because we get feedback loops on so many different dimensions. So I think that there are just a lot of vectors of choice and there are a lot of opportunities to be partnering on specific like areas or vectors. So I'm not super concerned about that point. I think one of the vectors that you talked about now to parlay it a bit is personality. We've noticed this be it's like hard to have you know, like evals against this. Like something that's like a very, you know, your hill climbing on, on being a good vibes personality. But it's something we take super seriously. Like Amanda Askel here, she is kind of the, the Claude character leader that we have and this sort of guru genius behind Claude's character. And it's something that we take super seriously. And we sort of steer Claude's character and personality to have specific character traits that we want to encourage the model to have. These are things like curiosity and sort of interest in multiple perspectives, striving to tell the truth, open mindedness and sort of self awareness. These are traits that we want to be reflected within Claude and that's something that helps people adopt it and feel like it is a capable collaborator with them. It's sort of and it is mercurial, like you probably couldn't say why you have a friend, like, why is your friend your friend? It's not because they have a really cool job or like they're good at cooking food. It's like you just like them. And it's some people just like Claude. And that is not only a driver of how people adopt Claude AI or Claude for work, because it is that capable, trusted collaborator or advisor, but it also drives like, you know, customers through our API, who, for whom that character is really important to pass through to their end customers. So companies like Intercom have switched over to us, DoorDash, Lyft. Like these are companies where they want to have a really strong customer relationship with their end customers. And that personality also drives that decision making for them as well. So it's, it's something that is both very important to us as a company in terms of us building a model that embeds those traits and characteristics which we think are sort of like adjacent to our constitutional AI perspective, but it's also important for our customers as well. So I think that is a point of differentiation.

Yeah, I totally think it is. I mean, like, I don't really care if the model is very good at math competitions. I care, like, it's nice to me and like, you know, we can work together.

Yeah, absolutely. I feel the same way.

Okay, let's talk a little bit more about how you guys build products internally. So, you know, at Anthropic, I'm sure other AI companies, research is like a big part of the job, right? Like the researchers, like, you know, improve the model. So how do you actually work with them? Like, how do you do, you're like, hey, are you like, hey, when's this new model coming out? Or like, you know, when should I get ready for this? Or like, how do you work with the researchers?

Yeah, yeah, great question. So I think the, the thing that I would say is if we think about traditional product development, it's been like talked about with the three legged stool, you know, product design and engineering, but I really think it's now like a four, four, four legged chair. Research has to be there through the entire product development life cycle, just like you would have design or engineering. Because we do have agency as a company to change the model. That is one thing that is unique and special about us. And so it would be foolish for us to not like be collaborating super part closely with our partners on research to do that and for research to treat us like a customer because we are ultimately an adopter of the model and they can help make the model better for us. So there's this virtuous loop of like, we want to solve problems, the research team will help us do that. But the research team also wants to make the model better. We are their customer. And so I think that there are a few key components or ingredients to making that relationship actually work. And the product development cycle, I think the first. I mentioned this earlier, but a consistent vision for where our product and our company is going. Like, what are the capabilities that we need to make Claude good at to make it the most useful and safest? That that shared vision about where Claude is going as sort of a capable collaborator, a virtual collaborator is important to ground our collective thinking so that we have a lot of shared language and we're prioritizing the same, like big chunks and things. Then the second thing is de risking research as early as possible and building very early prototypes, having like a constant portfolio of bets from a research standpoint that we can get surprised by. And then like work towards productization or productionization of those can help us with a sort of like high, high reward structure. There are things like artifacts and computer use which are an example of that. Like, being open to surprises is very important so that we can capitalize on those. That's kind of like the bimodal thing here. Like we have that shared vision and we work towards very concrete shapes of things that we collectively prioritize. But then we also take a lot of risks and are very open to surprises. It's like balancing those two things as important ingredients for success. And then the last thing I would say is, and this is very similar to traditional product development and we've talked about it a lot, which is just like feedback loops and having these stakeholders be on the team. Like we have researchers who are embedded in, in our product teams so that we can be pipelining all of the learnings that we're getting into that team. Understanding what's early and successful from the research side that we should be capitalizing on, but also finding the gaps that we are seeing in our attempt to build stuff that we then need to pipeline over to research to make sure that we're actually addressing before we end up shipping new models or new products that are interdependent with the model. That's like the really unique thing is we're not just building models, we're building model features. And there's a lot of interaction between the feature and the model to make the model good at powering the capabilities of the feature. And that's like a unique anthropic thing that we're super excited about. And having that connectivity means you really need like single threaded ownership through the process on research to be like a product expert on what you're trying to do to then like translate that into reinforcement learning environments that we need to do to actually address the failure that we're seeing on the product side. Like, I probably don't need to be that much of an expert, but having the, the researcher on my team, I can give them the shape of the problem that we're running up against and then they can translate it. So having that end to end connectivity has been something that I've had to learn and is like a new skill. Like it's a new way of working, but it's been so much fun and it's also been like really transformational for how we build stuff.

Yeah, because I think everything ultimately starts with like what the customer is trying to do, right? Like accomplish.

Yeah, yeah, absolutely. It's like you have problems and then you just have new tools to solve those problems. And then you have this new language of like evals to be a proxy for how good you are at actually solving that problem. And then once you have that common language, you can, you can work towards the same set of outcomes.

So, so let's, let's make this a little bit more practical. Like, I feel like you probably have a, a way of building a new product now that I think 99% of PMs can like learn from, especially if they have access to cloud.

Right.

So like it's not you mentioning how you put everything into a project first or something like, can you walk through like how you actually build a product?

Yeah, absolutely. So I can talk about something like maybe like styles, which is a feature that we launched a few months ago that helps tailor Claude's output to a specific style. Like whether it's concise or whether it's my writing style or whether it's more about explanatory things. So I think the first step is like any traditional product, you might have a product manager getting a lot of feedback on this being the kind of thing that we're excited about building. And then you would build like a product spec based on that. And I, you know, use a project to house all of our product spec templates that we have internally. And then I would like put in also the customer feedback, you know, speak to Claude through dictation on my mobile app of like ranting about what my vision for, for what this thing is, the problems that I'm trying to solve, what's the evidence for these problems existing. What are the fundamental product decisions that we need to make? And then I'll build out like the. The fundamental early spec. But then the new thing that is really important to do after that is build a set of evals that are indicative.