Daniela Amodei describes Anthropic's Constitutional AI framework, which guides Claude using principles from the UN Human Rights Declaration and 17 global documents instead of relying solely on human feedback. The approach embeds safety into model training from the start and aims to shape deeper model beliefs rather than just surface behaviors.
Chapters
Tell us more about Anthropic. Tell us about what the foundational philosophical ethic is of the company. You've innovated some really interesting things on how to make a responsive, explainable, steerable AI. Maybe we just start with an open book, share with the audience what Anthropic is all about.
Sure. Really, Anthropic is aiming to build transformative general AI systems that are safe, reliable, steerable, ethical. And that's kind of like it's a lofty goal. Right. So much of, I think why we were founded was this desire to build generative AI tools and systems and products that people could use while feeling really, really confident that what we were putting on the market was trustworthy, reliable and safe. And that's taken a number of different forms over the past few years. But I think specifically we have this sort of belief that incorporating these technical safety streams into training of the model, really from day one, is the best way to ensure that when they actually are in a product form and getting into the hands of customers, that they're going to be safe.
I love how your team has thought about using the UN Human Rights Declaration as a starting point for how to really moderate content and moderate the algorithm's development. Can you speak to that a little bit?
Definitely. So this kind of idea that we came up with, our brilliant research team came up with, is this idea of giving Claude, which is our generative AI tool, something we call a constitution. It's called Constitutional AI. And to just sort of go back a little bit to kind of how we got there, the way that you sort of used to. And by used to, I mean like three years ago or like a year and a half ago, you used to train these models to make sure, hey, they're not sort of saying nasty things or they're not replicating bad inherent biases, was you would do this technique called reinforcement learning from human feedback, which we also kind of co worked on when we were at OpenAI. And reinforcement learning from human feedback is just essentially a way of giving the model grades. Right. It's like A, you said something nice, like D minus, you said something mean. And that was reasonably effective in changing some behavior of the models. But what we found was that a lot of sort of subtler things about how models respond, react, things that they're sort of deeply believing are much harder to train out in those sort of individual cases. And so we had this idea of giving the model kind of a broader constitution and in the way that you would in a society to basically say, what are ways of behaving and engaging that are good for humans, for people, and that don't perpetuate some of the problematic things that might be in training data. And so, so much of what we saw with constitutional AI was a. We shouldn't necessarily be the arbiters of what is good or bad. Right. We're a group of, at the time, 100 people, 150 people now, 350 people based in San Francisco. Let's think about what broader documents that already exist in the world that have grappled with some of these challenges and incorporate. I think we have something like 17 different founding documents in the Constitution.
Amazing. I find it super interesting that you've chosen sounds like a number of global documents. And so one of the questions that come up is society is very, very diverse, and so how do we adjust for specific cultures or ethnic orientations of what good and bad is? And how do you think about that as a team? Because it's complicated. Right. How Claude might respond to. Let's just take myself. An Asian American born in Canada might be very different in conversation with someone who was born in Taiwan who has a very different, maybe view of humanity and morality. So how do you think about those. Those challenges?
Yes, this is a great question. And it's something our teams think about a lot. Right. For sort of the work that we do. So I think constitutional AI is sort of a great framework for thinking about what are the inputs that you would want to put into a model to kind of guide its ethics. But those don't necessarily. Those inputs can change. Right. So depending on the culture, the country, the company, even that's using our models, there's probably some degree of latitude over time that we can build in to say, hey, do you want to change sort of some of the founding documents of the model that you use? Use? That being said, I do think there are some guardrails that we feel strongly should be in place. Regardless, Claude can't help people create weapons. It's trained to ensure that you don't do something harmful to people or animals. And I think regardless of where it's deployed, those are kind of company universal things that we believe strongly in. Another thing I'll point to is we recently did a sort of set of research that was kind of building on this constitutional AI idea called collective constitutional AI. And what we did there was instead of using just a set of founding documents, we actually pulled a very large kind of demographically diverse group of people from sort of around, you know, around the country and in other parts of the world to see. Like, how did they react to some of the things that were in our constitution, and what did they kind of collectively come to on some of these sort of ethical questions? And how far was the distance between sort of what our constitution said and what the collective constitution said?
