Dario Amodei: Anthropic CEO on Claude, AGI & the Future of AI & Humanity | Lex Fridman Podcast #452

If you extrapolate the curves that we've had so far, right. If you say, well, I don't know, we're starting to get to like, PhD level, and last year we were at undergraduate level and the year before we were at like the level of a high school student. Again, you can, you can quibble with at what tasks and for what. We're still missing modalities, but those are being added. Like computer use was added, like image generation has been added. If you just kind of like eyeball the rate at which these capabilities are increasing, it does make you think that we'll get there by 2026 or 2027. I think there are still worlds where it doesn't happen in a hundred years. The number of those worlds is rapidly decreasing. We are rapidly running out of truly convincing blockers, truly compelling reasons why this will not happen in the next few years. The scale up is very quick. Like, we do this today, we make a model and then we deploy thousands, maybe tens of thousands of instances of it. I think by the time, you know, certainly within two to three years, whether we have these super powerful AIs or not, clusters are going to get to the size where you'll be able to deploy millions of these. I am optimistic about, meaning I worry about economics and the concentration of power. That's actually what I worry about more.

The abuse of power and AI increases the amount of power in the world. And if you concentrate that power and abuse that power, it can do immeasurable damage.

Yes, it's very frightening. It's very. It's very frightening.

The following is a conversation with Dario Amade, CEO of Anthropic, the company that created Claude that is currently and often at the top of most LLM benchmark leaderboards. On top of that, Dario and the Anthropic team have been outspoken advocates for taking the topic of AI safety very seriously. And they have continued to publish a lot of fascinating AI research on this and other topics. I'm also joined afterwards by two other brilliant people from Anthropic. First, Amanda Askel, who is a researcher working on alignment and fine tuning of Claude, including the design of Claude's character and personality. A few folks told me she has probably talked with Claude more than any human at Anthropic. So she was definitely a fascinating person to talk to about prompt engineering and practical advice on how to get the best out of Claude. After that, Chris Ola stopped by for a chat. He's one of the pioneers of the field of mechanistic interpretability, which is an exciting set of efforts that aims to reverse engineer neural networks to figure out what's going on inside. Inferring behaviors from nervous neural activation patterns inside the network. This is a very promising approach for keeping future super intelligent AI systems safe. For example, by detecting from the activations when the model is trying to deceive the human it is talking to. This is the Lex Friedman podcast to support it. Please check out our sponsors in the description. And now, dear friends, here's Dario Amadae.

Let's start with the big idea of scaling laws and the scaling hypothesis.

What is it? What is its history?

And where do we stand today?

So I can only describe it as it, you know, as it relates to kind of my own experience. But I've been in the AI field for about 10 years, and it was something I noticed very early on. So I first joined the AI world when I was working at Baidu with Andrew ng in late 2014, which is almost exactly 10 years ago now. And the first thing we worked on was speech recognition systems. And in those days, I think deep learning was a new thing. It had made lots of progress, but everyone was always saying, we don't have the algorithms we need to succeed. You know, we, we, we, we're, we're not, we're only matching a tiny, tiny fraction. There's so much we need to kind of discover algorithmically. We haven't found the picture of how to match the human brain and when, you know, in some ways it was fortunate. I was kind of, you know, you can have almost beginner's luck, right? I was like a, a newcomer to the field. And, you know, I looked at the neural net that we were using for speech, the recurrent neural networks, and I said, I don't know, what if you make them bigger and give them more layers? And what if you scale up the data along with this, right? I just saw these as like independent dials that you could turn. And I noticed that the model started to do better and better as you gave them more data, as you, as you made the models larger, as you trained them for longer. And I didn't measure things precisely in those days, but along with colleagues, we very much got the informal sense that the more data and the more compute and the more training you put into these models, the better they perform. And so initially my thinking was, hey, maybe that is just true for speech recognition systems, right? Maybe, maybe that's just one particular quirk, one particular area. I think it wasn't until 2017 when I first saw the results from GPT1 that it clicked for me that language is probably the area in which we can do this. We can get trillions of words of language data, we can train on them. And the models we were trained in those days were tiny. You could train them on one to eight GPUs, whereas, you know, now we train jobs on tens of thousands, soon going to hundreds of thousands of GPUs. And so when I, when I saw those two things together, and you know, there were a few people like Ilya Sutskever who you've interviewed, who had somewhat similar reviews. Right. He might have been the first one. Although I think a few people came to similar views around the same time. Right. There was, you know, Rich Sutton's bitter lesson. There was. Goren wrote about the scaling hypothesis. But I think somewhere between 2014 and 2017 was when it really clicked for me, when I really got conviction that, hey, we're going to be able to do these incredibly wide cognitive tasks if we just scale up the models. And at every stage of scaling there are always arguments. And you know, when I first heard them, honestly, I thought, probably I'm the one who's wrong. And you know, all these, all these experts in the field are right. They know the situation better, better than I do. Right. There's, you know, the Chomsky argument about like, you can get syntactics, but you can't get semantics. There is this idea, oh, you can make a sentence make sense, but you can't make a paragraph make sense. The latest one we have today is, you know, we're going to run out of data or the data isn't high quality enough or models can't reason. And, and each time, every time we manage to, we managed to either find a way around or scaling just is the way around. Sometimes it's one, sometimes it's the other. And so I'm now at this point, I still think, you know, it's, it's always quite uncertain. We have nothing but inductive inference to tell us that the next few years are going to be like the Next, the last 10 years. But, but I've seen, I've seen the movie enough times, I've seen the story happen for, for enough times to really believe that probably the scaling is going to continue and that there's some magic to it that we haven't really explained on a theoretical basis yet.

And of course, the scaling here is bigger networks, bigger data, bigger compute.

Yes. All in particular, linear scaling up of bigger networks, bigger training times and more and more data. Uh, so all of these things almost like a chemical reaction. You know, you have three ingredients in the chemical reaction and you need to linearly scale up the three ingredients. If you scale up one, not the others, you run out of the other reagents and, and the reaction stops. But if you scale up everything, ever everything in series, then then the reaction can proceed.

And of course, now that you have this kind of empirical science slash art, you can apply it to other.

More.

Nuanced things like scaling laws applied to interpretability or scaling laws applied to post tr. Just seeing how does this thing scale. But the big scaling law, I guess the underlying scaling hypothesis has to do with big networks. Big data leads to intelligence.

Yeah, we've documented scaling laws in lots of domains other than language. Right. So initially the paper we did that first showed it was in early 2020 where we first showed it for language. There was then some work late in 2020 where we showed the same thing for other modalities like images, video, text to image, image to text, math, they all had the same pattern. And you're right. Now there are other stages like post training or there are new types of reasoning models, and in all of those cases that we've measured, we see similar, similar types of scaling laws.

A bit of a philosophical question, but what's your intuition about why bigger is better in terms of network size and data size? Why does it lead to more intelligent models?

So in my previous career as a, as a biophysicist, so I did physics undergrad and then biophysics in, in, in, in grad school. So I think back to what I know as a physicist, which is actually much less than what some of my colleagues at Anthropic have in terms of, in terms of expertise in physics. Uh, there's this, there's this concept called the one over F noise and one over X distributions where, where often, you know, just, just like if you add up a bunch of natural processes, you get a Gaussian. If you add up a bunch of kind of differently distributed natural processes, if you like, if you like take a, take a probe and hook it up to a resistor, the distribution of the thermal noise in the resistor goes as one over the frequency. It's some kind of natural convergent. And, and I, I, I, I, and, and I think what it amounts to is that if you look at a lot of things that are, that are produced by some natural process that has a lot of different scales, right? Not a Gaussian, which is kind of narrowly distributed. But you know, if I look at kind of like large and small fluctuations that lead to, lead to electrical noise. They have this decaying one over X distribution. And so now I think of like patterns in the physical world, right? If I, if, or, or in language, if I think about the patterns in language, there are some really simple patterns. Some words are much more common than others. Like the, then there's basic noun, verb structure. Then there's the fact that, you know, nouns and verbs have to agree, they have to coordinate. And there's the higher level sentence structure. Then there's the thematic structure of paragraphs. And so the fact that there's this regressing structure. You can imagine that as you make the networks larger, first they capture the really simple correlations, the really simple patterns. And there's this long tail of other patterns. And if that long tail of other patterns is really smooth, like it is with the 1 over f noise in, you know, physical processes like, like, like resistors, then you can imagine as you make the network larger, it's kind of capturing more and more of that distribution. And so that smoothness gets reflected in how well the models are at predicting and how well they perform. Language is an evolved process, right? We've, we've developed language. We have common words and less common words. We have common expressions and less common expressions. We have ideas, cliches that are expressed frequently, and we have novel ideas. And that process has, has developed, has evolved with humans over millions of years. And so the, the, the guess, and this is pure speculation would be, would be that there is, there's some kind of long tail distribution of the distribution of these ideas.

So there's the long tail, but also there's the height of the hierarchy of concepts that you're building up. So the bigger the network, presumably you have a higher capacity to.

Exactly. If you have a small network, you only get the common stuff, right? If I take a tiny neural network, it's very good at understanding that a sentence has to have verb, adjective, noun, right? But it's terrible at deciding what those verb, adjective and noun should be and whether they should make sense. If I make it just a little bigger, it gets good at that. Then suddenly it's good at the sentences, but it's not good at the paragraphs. And so these, these rarer and more complex patterns get picked up as I add, as I add more capacity to the network.

Well, the natural question then is what's the ceiling of this? Yeah, like how complicated and complex is the real world? How much is stuff is there to learn?

I don't think any of us knows the answer to that question. My strong instinct would be that there's no ceiling below the level of humans. Right. We humans are able to understand these various patterns. And so that, that makes me think that if we continue to, you know, scale up these, these, these models to kind of develop new methods for training them and scaling them up, that will at least get to the level that we've gotten to with humans. There's then a question of, you know, how much more is it possible to understand than humans do? How much, how much is it possible to be smarter and more perceptive than humans? I would guess the answer has, has got to be domain dependent. If I look at an area like biology and, you know, I wrote this essay, Machines of Loving Grace, it seems to me that humans are struggling to understand the complexity of biology. Right. If you go to Stanford or to Harvard or to Berkeley, you have whole departments of, you know, folks trying to study, you know, like the immune system or metabolic pathways. And each person understands only a tiny bit part of it specializes. And they're struggling to combine their knowledge with that of, with that of other humans. And so I have an instinct that there's, there's a lot of room at the top for AIs to get smarter. If I think of something like materials in the, in the physical world or, you know, like addressing, you know, conflicts between humans or something like that, I mean, you know, it may be there's only. Some of these problems are not intractable, but much harder. And it may be that there's only, there's only so. Well, you can do at some of these things. Right. Just like with speech recognition, there's only so clear I can hear your speech. So I think in some areas there may be ceilings, you know, that are very close to what humans have done. In other areas, those ceilings may be very far away. And I think we'll only find out when we build these systems. There's, it's very hard to know in advance. We can speculate, but we can't be sure.

And in some domains, the ceiling might.

Have to do with human bureaucracies and things like this as you write about.

Yes.

So humans fundamentally have to be part of the loop. That's the cause of the ceiling, not maybe the limits of the intelligence.

Yeah, I think in many cases, you know, in theory, technology could change very fast. For example, all the things that we might invent or with respect to biology. But remember, there's, there's a, you know, there's a clinical trial system that we have to go through, to actually administer these things to humans. I think that's a mixture of things that are unnecessary and bureaucratic and things that kind of protect the integrity of society. And the whole challenge is that it's hard to tell. It's hard to tell what's going on. It's hard to tell which is which. Right. My, my view is definitely, I think in terms of drug development we, my view is that we're too slow and we're too conservative. But certainly if you get these things wrong, you know, it's possible to, to, to risk people's lives by, by being, by being, by being too reckless. And so at least, at least some of these human institutions are in fact protecting people. So it's, it's all about finding the balance. I strongly suspect that balance is kind of more on the side of pushing to make things happen faster. But there is a balance.

If we do hit a limit, if we do hit a slowdown in the scaling laws, what do you think would be the reason? Is it compute limited, data limited? Is it something else, idea limited?

So a few things. Now we're talking about hitting the limit before we get to the level of, of humans and the skill of humans. So, so I think one that's, you know, one that's popular today and I think, you know, could be a limit that we run into like most of the limits I would bet against it, but it's definitely possible is we simply run out of data. There's only so much data on the Internet and there's issues with the quality of the data. Right. You can get hundreds of trillions of words on the Internet, but a lot of it is, is repetitive or it's search engine, you know, search engine optimization drivel or maybe in the future it'll even be text generated by AIs itself. And so I think there are limits to what, to what can be produced in this way. That said, we, and I would guess other companies are working on ways to make data synthetic where you can, you know, you can use the model to generate more data of the type that you have, that you have already, or even generate data from scratch. If you think about what was done with DeepMind's AlphaGo Zero, they managed to get a bot all the way from, you know, no ability to play, go whatsoever to above human level just by playing against itself. There was no example data from humans required in the AlphaGo Zero version of it. The other direction, of course, is these reasoning models that do chain of thought and stop to think and, and reflect on Their own thinking. In a way, that's another kind of synthetic data coupled with reinforcement learning. So my, my guess is with one of those methods, we'll get around the data limitation, or there may be other sources of data that are, that are available. We could just observe that even if there's no problem with data, as we start to scale models up, they just stopped getting better. It's, it seemed to be a reliable observation that they've gotten better. That could just stop at some point for a reason we don't understand. The answer could be that we need to, you know, we need to invent some new architecture. It's been, there have been problems in the past with say, numerical stability of models where it looked like things were, we're leveling off. But, but actually, you know, when we, when we, when we found the write on block or they didn't end up doing so. So perhaps there's new, some new optimization method or some new technique. We need to unblock things. I've seen no evidence of that so far. But if things were to, to slow down, that perhaps could be one reason.

What about the limits of compute, meaning the expensive nature of building bigger and bigger data centers.

So right now I think, you know, most of the frontier model companies, I would guess, are operating, you know, roug, you know, $1 billion scale, plus or minus a factor of three. Right. Those are the models that exist now or are being trained now. I think next year we're going to go to a few billion, and then 2026, we may go to, you know, above 10, 10, 10 billion, and probably by 2027, their ambitions to build hundred, $100 billion, $100 billion clusters. And I think all of that actually will happen. There's a lot of determination to build the compute, to do it within this country, but, and I would guess that it actually does happen now, if we get to 100 billion, that's still not enough compute, that's still not enough scale. Then either we need even more scale or we need to develop some way of doing it more efficiently of shifting the curve. I think between all of these, one of the reasons I'm bullish about powerful AI happening so fast is just that if you extrapolate the next few points on the curve, we're very quickly getting towards human level ability. Right. Some of the new models that, that we developed, some, some reasoning models that have come from other companies, they're starting to get to what I would call the PhD or professional level. Right. If you look at their, their coding ability, the Latest model we released Sonnet 3.5, the new or updated version. It gets something like 50% on swe bench and Swabench is an example of a bunch of professional real world software engineering tasks. At the beginning of the year, I think the state of the art was 3 or 4%. So in 10 months we've gone from 3% to 50% on this task and I think in another year we'll probably be at 90%. I mean, I don't know, but might even be, might even be less than that. We've seen similar things in graduate level math, physics and biology from models like OpenAI's 01. So if we, if we just continue to extrapolate this right in terms of skill, skill that we have, I think if we extrapolate the straight curve, within a few years we will get to these models being, you know, above the highest professional level in terms of humans. Now will that curve continue? You've pointed to, and I've pointed to a lot of reasons why, you know, possible reasons why that might not happen. But if the, if the extrapolation curve continues, that is the trajectory we're on.

So Anthropic has several competitors. It'd be interesting to get your sort of view of it all. OpenAI Google XAI Meta what does it take to win in the broad sense of win in the space?

Yeah, so I want to separate out a couple things. Right. So you know, anthropics, Anthropic's mission is to kind of try to make this all go well. Right. And you know, we have a theory of change called Race to the Top. Right. Race to the Top is about trying to push the other players to do the right thing by setting an example. It's not about being the good guy, it's about setting things up so that all of us can be the good guy. I'll give a few examples of this. Early in the history of Anthropic, one of our co founders, Chris Ola, who I believe you're interviewing soon, you know, he's the co founder of the field of mechanistic interpretability, which is an attempt to understand what's going on inside AI models. So we had him and one of our early teams focus on this area of interpretability, which we think is good for making models safe and transparent. For three or four years that had no commercial application whatsoever. It still doesn't today. We're doing some early betas with it and probably it will eventually, but you know, this is a very, very long research bed and one in which we've, we've built in public and shared our results publicly. And, and we did this because, you know, we think it's a way to make models safer. An interesting thing is that as we've done this, other companies have started doing it as well. In some cases because they've been inspired by it, in some cases because they're worried that, you know, if, if other companies are doing this, that look more responsible, they want to look more responsible too. No one wants to look like the irresponsible actor. And so they adopt this, they adopt this as well. When folks come to anthropic interpretability is often a draw. And I tell them the other places you didn't go, tell them why you came here, and then you see soon that there's interpretability teams elsewhere as well. And in a way that takes away our competitive advantage because it's like, oh, now others are doing it as well, but it's good, it's good for the broader system. And so we have to invent some new thing that we're doing that others aren't doing as well in the hope that is to basically bid up, bid up the importance of, of, of doing the right thing. And it's not, it's not about us in particular, right? It's not about having one particular good guy. Other companies can do this as well. If they, if they, if they join the race to do this, that's, that's, you know, that's the best news ever, right? It's, it's just, it's about kind of shaping the incentives to point upward instead of shaping the incentives to point, to point downward.

And we should say this example, the field of mechanistic interpretability is just a rigorous, non hand wavy way of doing AI safety or it's tending that way.

Trying to, I mean, I think we're still early in terms of our ability to see things, but I've been surprised at how much we've been able to look inside these systems and understand what we see, right? Unlike with the scaling laws where it feels like there's some law that's driving these models to perform better on the inside. The models aren't, you know, there's no reason why they should be designed for us to understand them, right? They're designed to operate, they're designed to work just like the human brain or human biochemistry. They're not designed for a human to open up the hatch, look inside and understand them. But we have found, and you know, you can talk in much more detail about this To Chris, that when we open them up, when we do look inside them, we find things that are surprisingly interesting.

And as a side effect, you also get to see the beauty of these models. You get to explore the sort of. The beautiful nature of large neural Networks through the MacintoshRP kind of methodology.

I'm amazed at how clean it's been. I'm amazed at things like induction heads. I'm amazed at things like, you know, that. That we can, you know, use sparse autoencoders to find these directions within the networks and that the directions correspond to these very clear concepts. We demonstrated this a bit with the Golden Gate Bridge. Claude. So this was an experiment where we found a direction inside one of the neural networks layers that corresponded to the Golden Gate Bridge. And we just turned that way up. And so we released this model as a demo. It's kind of half a joke for a couple days, but it was. It was illustrative of the method we developed. And you could. You could take the Golden Gate, you could take the model, you could ask it about anything, you know. You know, it would be like, how. You could say, how was your day? And anything you asked, because this feature was activated, would connect to the Golden Gate Bridge. So it would say, you know, I'm feeling relaxed and expansive, much like the arches of the Golden Gate Bridge.

Or, you know, it would masterfully change topic to the Golden Gate Bridge and integrate it. There was also a sadness to it, to the focus it had on the Golden Gate Bridge. I think people quickly fell in love with it, I think. So people already miss it because it was taken down.

I think after a day, somehow these interventions on the model where you kind of adjust its behavior somehow emotionally, made it seem more human than any other version of the model.

Strong personality, strong identity.

It has a strong personality. It has these kind of like, obsessive interests. You know, we can all think of someone who's, like, obsessed with something, so it does make it feel somehow a bit more human.

Let's talk about the present. Let's talk about Claude.

So this year, a lot has happened. In March, Claude 3 opus sonnet, haiku were released. Then Claude 3. 5 sauna in July with an updated version just now released. And then also Claude 35 haiku was released. Okay, can you explain the difference between Opus Sonnet and Haiku and how we.

Should think about the different versions?

Yeah. So let's go back to March when we first released these three models. So, you know, our thinking was, you know, different companies produce kind of large and small models. Better and worse models. We felt that there was demand both for a really powerful model, you know, and you, that might be a little bit slower that you'd have to pay more for. And also for fast, cheap models that are as smart as they can be. For how fast and cheap, right? Whenever you want to do some kind of like, you know, difficult analysis, like if I, you know, I want to write code, for instance, or you know, I want to, I want to brainstorm ideas or I want to do creative writing, I want the really powerful model. But then there's a lot of practical applications in a business sense where it's like I'm interacting with a website. I, you know, like I'm like doing my taxes or I'm, you know, talking to, you know, to like a legal advisor and I want to analyze a contract or you know, we have plenty of companies that are just like, you know, you know, I want to do autocomplete on my, on my IDE or something. And for all of those things, you want to act fast and you want to use the model very broadly. So we wanted to serve that whole spectrum of needs. So we ended up with this, you know, this kind of poetry theme. And so what's a really short poem? It's a haiku. And so Haiku is the small, fast, cheap model that is, you know, was at the time was really surprisingly, surprisingly intelligent for how fast and cheap it was. Sonnet is a, is a medium sized poem, right? A couple paragraphs. And so Sonnet was the middle model. It is smarter, but also a little bit slower, a little bit more expensive. And Opus, like a magnum opus, is a large work. Opus was the largest, smartest model at the time. So that was the original kind of thinking behind it. And our thinking then was, well, each new generation of models should shift that trade off curve. So when we release Sonnet 3.5, it has the same, roughly the same, you know, cost and speed as the Sonnet 3 model. But it increased its intelligence to the point where it was smarter than the original Opus 3 model, especially for code, but also just in general. And so now, you know, we've shown results for Haiku 3.5. And I believe Haiku 3.5, the smallest new model, is about as good as Opus 3, the largest old model. So basically the aim here is to shift the curve and then at some point there's going to be an Opus 3.5. Now every new generation of models has its own thing. They use new data, their personality changes in ways that we kind of, you Know, try to steer but are not fully able to steer. And so there's never quite that exact equivalence where the only thing you're changing is intelligence. We always try and improve other things and some things change without us knowing or measuring. So it's very much an inexact science in many ways. The manner and personality of these models is more an art than it is a science.

So what is sort of the reason.

For.

The span of time between say, Claude Opus 3 oh and 3.5? What takes that time if you can speak to.

Yeah, so there's, there's different, there's different processes. There's pre training which is, you know, just kind of the normal language model training and that takes a very long time that uses, you know, these days, you know, tens, you know, tens of thousands, sometimes many tens of thousands of GPUs or TPUs or Trainium or, you know what, we use different platforms but you know, accelerator chips often, often training for months. There's then a kind of post training phase where we do reinforcement learning from human feedback as well as other kinds of reinforcement learning. That that phase is getting larger and larger now and you know, you know, often that's less of an exact science. It often takes effort to get it right. Models are then tested with some of our early partners to see how good they are and they're then tested both internally and externally for their safety, particularly for catastrophic and autonomy risks. So we do internal testing according to our responsible scaling policy, which I could talk more about that in detail. And then we have an agreement with the US and the UK Safety Institute, as well as other third party testers in specific domains to test the models for what are called CBRN risk. Chemical, biological, radiological and nuclear, which are, you know, we don't think that models pose these risks seriously yet, but, but every new model we want to evaluate to see if we're starting to get close to some of these, these, these more dangerous, these more dangerous capabilities. So those are the phases and then, you know, then, then it just takes some time to get the model working in terms of inference and launching it in the API. So there's just a lot of steps to, to actually, to actually making a model work. And of course, you know, we're always trying to make the processes as streamlined as possible. Right. We want our safety testing to be rigorous, but we want it to be rigorous and to be, you know, to be automatic, to happen as fast as it can without compromising on rigor. Same with our pre training process and our post training process. So, you know, it's just like building anything else. It's just like building airplanes. You want to make them, you know, you want to make them safe, but you want to make the process streamlined. And I think the creative tension between those is, you know, is an important thing in making the models work.

Yeah, rumor on the street, I forget who was saying that Anthropic has really good tooling. So I probably. A lot of the challenge here is on the software engineering side is to build the tooling to have a, like a efficient, low friction interaction with the infrastructure.

You would be surprised how much of the challenges of, you know, building these models comes down to, you know, software engineering, performance engineering. You know, you, you know, from the outside you might think, oh man, we had this Eureka breakthrough, right? You know, this movie with the science. We discovered it, we figured it out. But, but, but I think, I think all things even, even, even, you know, incredible discoveries like they, they, they, they, they almost always come down to the details and often super, super boring details. I can't speak to whether we have better tooling than other companies. I mean, you know, I haven't been at those other companies at least, at least not recently. But it's certainly something we give a lot of attention to.

I don't know if you can say, but from 3, from Cloud 3 to Claude 3 5, is there any extra pre training going on? As they mostly focus on the post training, there's been leaps in performance.

Yeah, I think, I think at any given stage we're focused on improving everything at once. Just, just naturally like there are different teams. Each team makes progress in a particular area in, in, in making a particular, you know, their particular segment of the relay race better. And it's just natural that when we make a new model we put, we put all of these things in at once.

So the data you have, like the preference data you get from RLHF, is that applicable? Is there ways to apply it to newer models as you get trained up?

Yeah, preference data from old models sometimes gets used for new models. Although of course it somewhat better when it's, you know, trained on, it's trained on the new models. Note that we have this, you know, constitutional AI method such that we don't only use preference data, we kind of, there's also a post training process where we train the model against itself and there's, you know, new types of post training the model against itself that are used every day. So it's not just RLHF, it's a bunch of other methods as well. Post training, I think, you know, is becoming more and more sophisticated.

Well, what explains the big leap in performance for the new Sonnet 3.5? I mean, at least in the programming side and maybe this is a good place to talk about benchmarks. What does it mean to get better? Just the number went up. But you know, I, I, I program, but I also love programming and I claw 35 through cursor is what I use to assist me in programming. And there was at least experientially, anecdotally.

It'S gotten smarter at programming.

So what, like what, what does it take to get, to get it smarter?

We observe that as well. By the way, there were a couple very strong engineers here at Anthropic who all previous code models, both produced by us and produced by all the other companies hadn't really been useful to, hadn't really been useful to them. You know, they said, you know, maybe, maybe this is useful to beginner. It's not useful to me. But Sonnet 3.5, the original one, for the first time they said, oh my God, this helped me with something that, you know, that it would have taken me hours to do. This is the first model that has actually saved me time. So again, the waterline is rising. And then I think, you know, the new Sonnet has been, has been even better in terms of what it, what it takes. I mean, I'll just say it's been across the board. It's in the pre training, it's in the post training, it's in various evaluations that we do. We've observed this as well. And if we go into the details of the benchmark, so SWE bench is basically, you know, since, since, you know, since, since you're a programmer, you know, you'll be familiar with like pull requests and you know, just, just pull requests are like the, you know, the, like a sort of a sort of atomic unit of work. You know, you could say I'm, you know, I'm implementing one, I'm implementing one thing. And, and so SWE bench actually gives you kind of a real world situation where the code base is in a current state and I'm trying to implement something that's, you know, that's described in, described in language. We have internal benchmarks where we, where we measure the same thing. And you say just give the model free rein to do anything, run anything, edit anything. How well is it able to complete these tasks? And it's that benchmark that's gone from, it can do it 3% of the time to it can do it about 50% of the time. So I actually do believe that if we get you can.