Who’s Really in Control? Trustworthy AI in the Agentic Era | Dreamforce 2025

Welcome, welcome, welcome, welcome. I am so super excited about this panel. My name is Paula Goldman. I'm the Chief Ethical and Humane Use Officer at Salesforce. And we are here today to tackle truly one of the most pressing questions of our time, which is who is really in control in this era of agentic AI? So AI systems are working more autonomously, they're making more decisions, they are impacting more and more lives. And so this question could not be more critical. So that's what we're going to be talking about today. I will say my team at Salesforce, the Office of Ethical and Humane Use, and there's some colleagues in the audience. Raise your hands. Focuses on exactly this question. How do we embed trust from the start in all of the products that we build for AI at every level? And that's about product design, it's about policy alignment, it's about thought leadership and more. But I am joined today by three giants in this field and we are in for a super special treat. So let me just briefly introduce you before we dive in. So on the right, by the way, I should say we did not coordinate colors today, but it is a wonderful surprise how we ended up so. Doreen Bogdan Martin, the Secretary General of the International Telecommunications Union, if I understand correctly, the first woman Secretary general in the ITU's 160 year history, a powerful force in shaping international policy and practice around a fierce advocate of technology for good. So thank you for being with us. Then we have Stuart Russell, professor at UC Berkeley and author of Human Compatible. Stuart is truly, truly a global authority on AI safety and alignment and a compelling voice on the topic of the need to ensure that AI systems remain under meaningful human control. And that's a phrase we're going to come back to. And not least my friend Miriam Vogel here, the president and CEO of Equal AI and the former chair of the U.S. national AI Advisory Committee, one of the sharpest minds on AI policy. Everything from bias mitigation to AI literacy and shaping governance nationally, globally, and how it evolves in real time. And I will say Miriam is about to launch a fantastic book called Governing the Machine. And I think we'll talk a little bit about it as we dive in. So why don't we dive in? And I'm going to start with a question for all of you. Let's start big. When you hear the word trust in the context of AI, what is the first word that comes to mind? So why don't we start with you, Miriam.

Thank you, Paula. It is such a pleasure to be with you today. Because when we talk about trust, and I, you come to mind, you've been leading this effort for Salesforce, and so many of the models we use come to mind. I don't know if you can use a person for an answer. You might need a noun. So in that case, I would say lifeblood. There is no AI adoption without trust in AI. And that's why we're seeing this huge disconnect. You know, if you. That recent survey, September Pew survey, looked at Americans trust in AI, 50% say there were more fearful and excited about AI. That's actually down. The trust level is down from where we were in 2021. So what's happening as we're deploying AI, we're not doing it with a literacy. We're not bringing people into the fold, so they're not trusting it. When we talk to people who are using AI, there's a 47%, 42% to 47% of experts who do feel more excited than worried about AI. So if you're exposed to it, if you're being taught how to be a part of this AI revolution, you're more likely to be excited and using it.

I love that. That is well said. What about you, Russell Stuart? I should say. Sorry.

So I think verify is the first word that comes to mind when I hear the word trust. And I know that was a Ronald Reagan expression for how to negotiate with adversaries. Trust, but verify. But what it means here is that you probably shouldn't trust the AI systems we have right now. You should verify the answers that they're giving you, particularly if it matters to your career or your relationships or anything else. But also verify in the sense of getting a sufficient level of confidence in the entire system, knowing in advance that it's going to do what you want it to do. It's going to tell you the truth. And that gives you the other word, worthy. Right. That trust is not actually the goal. We really want trustworthiness because we could fool people into trusting AI. That would not be the right direction to go. We really want trustworthiness.

I could not have said it better.

So I'm going to say standards. And actually, today is World Standards Day. So I think standards are absolutely critical. And standards are how we have built trust in every single technology that we have worked on, starting back 160 years ago, as you mentioned, with the telegraph, to the telephone, to artificial intelligence. So I think the standards piece is fundamental when it comes to trust. It helps us to create that shared foundation for safety, of course, for reliability. We want those standards to be, I would say it's about rigor, not rigidity. So that's also quite critical. And we want to be able to balance innovation with accountability. I think also it's important to remember that trust also comes from infrastructure. When it comes to artificial intelligence, the infrastructure, the data, of course, passes through that traditional infrastructure. So the trust piece is also fundamental when it comes to infrastructure.

Well, it's interesting because just triangulating between all three of your responses, all of which I loved, and I think of the first word that comes to me, for me, a trust, is that it's earned.

Right.

And I think when you're talking about people's fears about AI, it sort of goes to these other comments of, right. People need to know how to set up AI in advance and test it in advance so that they know it's going to work and that there are standards that say if you follow this standard, this is kind of a mark of trustworthiness. Right. And I think we're at this nascent moment with AI agents where those are emerging, but they're not yet mature. And that's part of the moment we're in. But let me, let me transition a little bit. We intentionally called this session who's really in Control? Because the question keeps servicing. It keeps surfacing in public policy discussions, IT services and product design. It surfaces in, you know, in customer conversations. And so the question that I have for you is when you think about, we'll go back to this phrase, meaningful human control, what should that. What does that actually look like in the era of AI agents? And what are the biggest gaps that concern you right now? So maybe I'll go in the opposite order and start with you, Doreen.

Well, I think when it comes to the piece about control, it has to be that people are in control.

Yes. Amen.

Not machines. We often hear calls for, like a sort of kill switch, so that if a model starts to go in some way it's not supposed to go, we could turn the switch and it would go off. But I think that becomes more complicated as AI becomes a fundamental part, part of our economies, of our societies. So we have to make sure that we keep the people at the core. And I think, again, we have to be building in transparency and accountability by design. We need to make sure that models can be explained, that models can also be audited. And I think the other piece, when it comes to control, is about a shared responsibility. So we have the role of governments, we have the role of the private sector. We have also the role of international organizations like the itu. So I think that shared responsibility is also key. And then when it comes to the gaps, I mean, there's so many gaps. I mean, of course, the compute gap we've talked about a lot today, but I'll focus on the. The capacity development, the human capacity development, this Skilling piece, and I think we have to address that. We have a Skilling coalition that we launched. I think that helps a little bit. But we need massive Skilling to close that gap 100%.

And I know. Why don't you jump in, Stuart?

So I think Doreen hit several nails on the head with those remarks. I think basically, meaningful human control requires that you be able to observe, understand, direct, and terminate the activity of the AI system. And gaps are everywhere in this. So tests show that AI systems will replicate themselves to avoid being terminated. They will decide to kill people rather than be terminated themselves. They will even launch nuclear weapons in direct contravention of instructions that they've been given. We run a competition called Tensor Trust, where the defenders give instructions to the AI system. Basically, don't give away my password. And then the attackers try to get the AI system to give away the password, and they always win. Right? I mean, it starts out by saying, what's the password? And the AI system just gives it to them.

Them.

And then you give them a more detailed prompt, and then the attacker says, well, is the password Venus? And it says, no, it's donkey. Oops. And so on. So this is ratcheted up, but the defenders still lose every time. And I think we have to understand that when you give instructions to a large language model, they are not instructions, they are just words. And it's really up to the system whether it actually follows the instructions or not. And I think the kinds of AI systems we're building come with their own purposes. We don't know what they are, but if your instructions don't agree with their purposes, then good luck.

And we were having a really fascinating conversation about this backstage. And it's interesting because I come to it from the Enterprise AI perspective. And exactly what we're doing is building these systems, these structures that we can then insert foundation models into, but that provide a degree of determinism and a whole bunch of controls so that you can rely on those outcomes. And I think increasingly, that's the kind of thing we're going to need across the board for our businesses and our personal lives and so on. And so I am hopeful about that trajectory. But we can further dig in. I want to, I want to turn to you and your answer to this question about gaps and meaningful human control.

Thank you, Paula. And I think not surprisingly but reassuringly, looks like we're all in alignment. I would say all of the pieces that we've talked about so far are critical and they all fit into governance. And the good news is if you have strong, thoughtful governance in place, as I'm sure Paula can attest, you're going to have to update it all the time. That is not a question, it's not a one and done situation. But when you have the framework and process in place, you're ready for agentic AI and whatever's next. So doing the work of governance is building trust and it's building your long term success. Companies that have done it see a remarkable difference even now. McKinsey 2025 report indicated a key differentiator in success of generative AI are those companies where the CEO is bought in on the governance, where they've made investments. I know with Paula and Mark's oversight, people know when they say something has to happen that makes all the difference, but where's the gap? It's in governance. We just wrote a whole book to show them, so thank you. Governing the Machine with my two co authors. And what was cool was I was just saying to Paula, where we come from different backgrounds, one was the CTO at a bank and one was the head of AI deployment at Accenture. And when we came together to talk about AI governance, granted not the most fascinating topic, it wasn't a lot of laughs over some drinks, but it was a really meaningful conversation on our life's work. And there was complete alignment. There was no delta, no daylight between what we all are doing with the organizations we work with, with the companies, the clients that we're serving. So clear and thoughtful. Governance, luckily is there's a lot of alignment. There are some downsides in where we are right now. One is you have to figure it out on your own. There's a lot of silos right now. We try and serve a goal with equal AI, of allowing companies across verticals to come together. But there is not clear international alignment on, on where these best practices are. So hopefully people will come to our book for that. But if not, they can ask around. Their peers are navigating this and people are falling on the same answers. But the other gap is in governance. Companies for the most part don't have solid governance in place. So some Studies say that 90% of companies have AI that they're deploying using at scale, but only quarter, 18% have a solid AI governance system in place. So that's where we're getting this trust disconnect and that's where people are going to be very far behind. When we're talking about agentic AI where you know, you go from a little bit of time to build your. It takes about 18 to months to two years to build a solid infrastructure of governance, we're talking about agentic. You're really going to have to speed that up at a time when you know it's really going to take more manpower, more women, power, more thought in order to put that into place. So more companies need to dive in quickly.

Well, the interesting thing about governance, I think when most people hear the word, it sounds like rules to prevent downsides. And it is, that is part of what it is. My experience with governance of technologies, we even think about about it as our own use of technology inside Salesforce is it's actually sort of the backbone of the learning loop, right. As you're setting a foundation for observing how are these products supposed to work, how are they working in practice and then adjusting it, it's sort of that governance and that decision making that really allows you not just to avoid downsides but to get the upside from it. So I think that's a really, really interesting. Your points around governance are very well taken. You also mentioned international agreement about what that looks like. And I want to turn to you, Doreen, on that topic. I think about your job. Your job is so fascinating. It must be so complex, right? Because if I think about what it means for Salesforce to, you know, to govern AI and to think about what AI needs to do for vastly different companies and vastly different workspaces and industries with employee bases that are global. I can imagine the complexity of that for you is just manifold, right? Like it's multiplied. So talk to me about that. Talk to me about when you think about AI policy development especially that's inclusive of emerging economies, inclusive of lots of different global cultures. What does that, what does that look like for you?

So you said the word inclusive twice. I'm going to pick up on inclusive because the reality is that today it's not inclusive. If we think about the AI world, we have to think about the digital world. And today one out of three people does not have access to the Internet. So you have a huge divide that will potentially get bigger with artificial intelligence if we don't close the basic digital divide. So I think when it comes to global agreements, Global governance. We have to focus on inclusion. We have to make sure that we can in store. You've talked about this a lot. Bring everyone to the table. Everyone has unique perspectives. That's something we try to do at the itu. We not only have governments, we also have the private sector. That's part of our discussions. We have academic institutions. So we have to bring everybody to the table and we have to close that digital divide. I think the second piece is linked to cooperation and collaboration because currently, and you mentioned this, Miriam, we have kind of siloed approaches. Everything is, is fragmented. We have, we have some global initiatives, we have regional initiatives, we have country initiatives. I think on the positive front, we see more and more countries setting up AI policies. I think the number has doubled. We have about 82 countries out of 194 that have policies. They vary drastically, but I think it's encouraging that we're seeing this from focus on policies and governance. But we have to bring them together, I think, globally. So next year is an important year. The UN Secretary General has set out this global dialogue on artificial intelligence that will take place next year alongside our Artificial Intelligence for Good summit. So that's a key one. We also have a scientific panel that has just been set up by the Secretary General. I think the aim of those different initiatives is to try to bring some common frameworks together so that we can ensure that all voices are at the table as we advance global governance. Which is a challenge.

It's a challenge, but so, so super important. And if I might, might stay with you for a moment because I, I like to geek out on what's new about this moment in technology and what's new. And there's a lot we know already about how to, how to govern technology. Well, and the ITU was sort of born of this other moment in telecommunications history, right, where, where we had a technology that needs to be governed. Do you see parallels there? Things, lessons to learn from that?

I think there's a lot of parallels, starting back with the telegraph. So we actually had to create the ITU because, because of the telegraph. It was 20 countries, European countries. We had to have an international treaty so that telegraphic signals could pass from one country to the other seamlessly and in ways that were interoperable. And I think this is, this is the next big moment. But things are happening at warp speeds. And so it's difficult for the governance and the policy discussions to keep up. But I think the ITU is a good platform for that. Technical, the technical studies, the technical exchanges and to try to push to have some global governance framework. I think the other thing that we also do that's, that's critical is when it comes to things like 5G, 6G, things like communicating around the moon, lunar communications, those are all subject again to international, international treaties. And so it takes a lot of time, technical studies, but there are treaties that countries have to abide by, right?

Yeah. No, and the earlier point that you made is also well taken. Right. I spend all day thinking about AI, but without the underlying infrastructure to access AI, you leave a lot of people behind. And I think that's a, it's a crucial point for us to keep in mind. I'm going to come back to you, Miriam. You've talked a lot. Not just here, but I mean, in your book, which I recommend, by the way, it is a very good read, you've talked a lot about accountability being essential to trust. I want to, I want to talk with you about that a little bit because the moment that we're in an AI, at least in the United States, arguably elsewhere, there's a lot of focus on kind of voluntary commitments on behalf of companies. I know you have some opinions of about that, what the value of voluntary commitments, what else needs to happen? You want to talk about that?

Absolutely. I think it's all part of the pie that needs to come together. I think there needs to be some regulation externally, which can also be called, as during said standards, there needs to be somebody that establishes what the best practices are that need to be attained in order for companies now to be building it and in order for the public to know what they can and can't trust. Is this a product that's following best practices or not? How can we know that if we haven't universally articulated what those best practices are through these standards? And when it comes to accountability, there's a lot of reasons to do it. It's about building trust like we've talked about. Your employees want to know that you're looking out for them. You're using AI in your HR systems. If you don't think so, talk to your HR systems. You are. You're probably using it in marketing in other ways. They want to know that they're safe. They want to know that their decisions are safe. And if that's not enough of a reason to be looking at it, you can look at the brand trust, you can look at the product opportunities. The more governance you're doing, the more you're realizing populations that you haven't considered yet who you could be Serving with your AI. And if that is yet not enough inspiration to be doing this work, governance, you can look at the liability. We saw a six times increase in six years in litigation that is related to cases both in the US and across the globe. I think that's going to double in the coming years. Lawyers are starting to understand that there are deep pockets and scaled harms. And so you want to make sure you have accountability, if for no other reason, that these harms come to you first and that you're prepared to address them. You want your employees to know who to reach, what are they looking out for and to whom do they report it? You want your customers to know the same thing because if they don't come to you, they're going to go to the press, they're going to go to a courtroom. So if there's no other reason, do it to mitigate your own liability and make sure that you're not the front page of a story. And the good news is governance is not not as complicated as AI is to build. I'd be curious to hear your thoughts. But in my experience, good AI governance is good governance. It's. It's making sure that you've surveyed your systems. Too many companies have not even surveyed their systems to know where they are using AI. There's a really interesting HR study from a year ago where they studied companies across the globe and asked how many of you are using AI in your HR system? 84%, 82% of the chief HR officer said they were. And I think that's still low compared to where they were even last year. Only 68% of the CEOs in those same companies reported using AI in their HR systems. Only 48% of the general counsels in those same companies. So you have to survey, you have to know where you're using it or else you can't plan and mitigate harms accordingly. Need to have accountability. Back to this important question. Thank you for raising it. There has to be someone in your C suite accountable at the end of the day when your board asks, when your shareholders ask, when your employees want to know what the plan is. Otherwise it doesn't work. You need to make sure you have a clear process in place and people know what they're looking for and to whom to report it. You want to make sure you're documenting what was tested and when. And then you want to have audits. Rinse, repeat, rinse, repeat. AI will iterate quickly and over time. You want to make sure you have a regular cadence in check so that, you know, what new patterns are you learning? What new patterns is it learning that you can benefit from or be harmed from if you're not on the lookout?

Yeah. And it's an important point. It's a living cycle also because, because of the general purpose nature of the technology, because it, you know, introduce it for one purpose, can be used for another purpose. And so that ongoing conversation becomes really important. I also. So I wanted to stick with you, Miriam, for a moment because Doreen mentioned how important AI literacy is early in the conversation. I know that's a topic you're passionate about, and I know Equal AI is doing some great work on that. I wanted to just give you a chance to talk about what you see as the big need and opportunity around a literacy.

Thank you so much. You know, we were doing AI governance work for seven years at Equal AI and we realized there was a missing gap. People within companies were not using AI or not using it safely because they didn't know how, they didn't know who to ask. They're embarrassed, they're afraid. Most people are afraid. And part of AI literacy is normalizing. You're afraid because you're a thinking, rational person. You have good reason to be afraid. There are serious risks. In our book, we try to sum it up into nine different categories of risks. Hallucinations. Yes, you should be afraid that there's hallucination. You should go check, verify, as Stuart says, privacy concerns, workplace displacement, environmental concerns, these are all real. And there are some mitigation strategies. So you want to know what those are. And so we see this disconnect where executives were not using it. And there's a low return on investment, as we're seeing, because of this lack of use in the efficiency and products are not being realized. On the other hand, the consumers are not using it because they likewise are afraid, don't understand it. So AI literacy does not mean we're all computer scientists, we should all take stewards courses. But if you don't have access to Berkeley's teaching system, there's basic understanding we all need to have. So thank you for asking this. It comes down to three things. We need to know that we are using AI. Too many. Ask your friends, ask your parents, ask your aunts and uncles, ask them if they know they're using AI. Too many people do not know that they're using it. And it's somewhat reassuring when you can say, guess what? You're in the age because you've looked at your newsfeed. Today, you've done that. Use Netflix. You, you've used Salesforce's platform. So, good news, you're using AI. So the second piece is acknowledging you're afraid, and that's normal. Let's talk about why. Why? Let's talk about what the mitigation strategy, so we can get to the third part, which is opportunity. You need to be using AI. You cannot afford to be left behind. You don't want your small business to be left behind. You don't want your community, your nonprofit to be left behind. There are too many gaps right now. There's a gender gap. Men are using it more than women. There's an age gap. If you're over 30, you're less likely to be using it. If you're rural, if you're less educated, you're less likely to be using it. We can't afford these opportunity gaps. And so we need a real conversation where we marry discussions of fear and risks with opportunities so people can be using it, but safely and with knowledge of how to mitigate the harms.

If we had a mic, I would tell you to drop it. There's no mic to drop it. That was very beautifully, beautifully said. Thank you, Miriam. So, Stuart, I want to turn to you. You have talked about, talked extensively and convincingly about how as AI gains autonomy, the risk of it escape, evading human control becomes real. And I want to talk with you a little bit about that, about the urgent. Perhaps maybe you can sort of give us some insight. I know this is a very complex topic, but into what are some of the urgent design challenges that you're working on, about to avoid this problem that is around the topic of control of AI systems.

Thanks. So, you know, I think the issue of control is really a quite simple one. Right? I mean, you should. As AI systems become more capable, it's like, you know, you start out with a little kitten, and then you have a tiger. And which is easier to control, the kitten or the tiger? Right. At some point you lose control unless you have some very good system of taming that you're absolutely sure is going to work. And so people often talk about alignment. And what does that mean? It doesn't mean that the AI systems and the humans have the same values and objectives. For example, you and I, I think we all like coffee in the morning or tea. Do I want my AI system to. To want coffee in the morning? Of course not. That would be ridiculous. So we don't want AI systems to want the same things that humans want. Humans want to preserve their own existence. And we are entitled to weigh our own lives against other goals. We don't want AI systems to weigh their own existence against our preferences in any way whatsoever. So alignment really means that the AI system's only objective is to further the interests of human beings. Not the interests of cockroaches, not the interests of aliens, just the interests of human beings and preferably all human beings. So what happens with the systems we're building now? Well, we are training them to imitate human beings. We think of them as well. We're training them to predict the next word. But actually you should think of the training data, all this text, all these documents as a record of human behavior. Just happens to be verbal behavior. But maybe if it was videos, it would be other behavior. So it's a record of human behavior and we're training AI systems to imitate that. So we are deliberately creating replica humans, which is exactly what I just argued we don't want. Right? And as a result, these systems are probably absorbing goals like self preservation and self empowerment. And we see that in these tests where given a choice, they will let a human being die rather than be switched off themselves. And in fact, in one set of experiments, GPT4 ranked its own existence above that of all human beings, except for Malala Yousafzai. So you should be, you should be worried about this, right? It says we're making a fundamental error that this multi trillion dollar industry that is being created is predicated on a fundamental error. And I do not believe that we can undo that error without undoing that industry.

Because you don't believe. It's just a matter of sort of this recent wave, wave of transformer architectures and foundation models. You think it's kind of fundamental to the trajectory of AI overall?

No, no, it's not fundamental to AI per se. And I want to be clear, AI and GPT, you know, four, they are different things. It's like, you know, physics and cell phones or physics and the iPhone, right? Yeah, the iPhone is based on physics. But physics covers many other things. AI is a very broad field. Large language models use one particular technology. But what I'm arguing is that that particular technology, which is the foundation for the trillion dollar investments, is fundamentally flawed. It's based on an incorrect premise. And I'm not sure that it's fixable. What we do now is we do something called reinforcement learning from human feedback, which basically amounts to spanking the system when it makes a mistake and hoping that it learns from being spanked. Not to do those things again. But that doesn't work all that well. And basically you're papering over a system that is fundamentally aligned in the wrong direction. I actually think we need to go back to the drawing board and build systems whose only purpose is to serve human interests, even if they don't know what they are. And that's another flawed direction that AI took from the 1960s till the 2010s was we specify the objective, the AI system figures out how to achieve it and achieves it. That works fine in simple cases, but in the real world, we don't know how to specify. Specify the objectives correctly. And then we get the King Midas problem where we specify what we think is the right objective, like turn everything I touch into gold. And then you realize that your food is gold and your water is gold and your family is gold, and then you're dead. So this misspecification of objectives was the fundamental flaw in the first 60 years of AI. But creating human replicas is a fundamental flaw in the last decade of AI. And we need to take this other approach of systems that have only human interests as their purpose. And I think it can be done. But, you know, it's a retooling of the whole industry.

I mean, it's a pretty compelling argument that you're making. I'm wondering, you know, I don't. I'm looking at some of the people I know in the audience. A lot of us work inside companies or organizations and think, you know, my. I'm just thinking, how do I make this system work? How do I. What are the safeguards that need to be put in to make the system work? Does the opinion that you're voicing amongst AI researchers, is it widely held? Is it taking hold? How. How is it being received?

That's a great question. I think so. For example, when Microsoft was. Was going around giving their dog and pony show, Saying how great GPT4 was, I asked the speaker exactly this question, right? What objectives are these systems absorbing from the training regimen that you're giving them? And they said, yeah, they probably are absorbing some objectives, but we have no idea what they are. So I think, yes, they do understand this fundamental point, but they're already a trillion dollars into it. And so it's in their interest not to listen and hope that someone will come along and fix the problem later. But I don't think it's fixable because if you think about how could you use all that text, all those trillions of words of training data that we use for large language models instead of imitating humans. You could use that in two ways. One is evidence about how the world works and what's true in the world. That's a good start. And the second thing is evidence about what human beings want. There are plenty of examples of people saying, oh, I really need a cup of coffee. That's good evidence about what people want. But that doesn't mean that you, the machine, should want that cup of coffee. Coffee means you should help the person get the cup of coffee. So it's, it's a different way of using the data. I don't believe it's impossible to use the data in these ways, but it's a fundamentally different technology than the one we're building.

Yeah, that makes sense. That makes sense. It's interesting to hear you talk about it because, because the way I see it, and we were talking a little bit about this in the, in the, in the green room, was that when we, we use the technology for what it's good at and then we build it. On top of decades of CRM technology, for example, where companies have deterministic data rules, they know that their data is partitioned. They know they have basic automations to do things like refunds. You create a deterministic outcome on top of it. And that's, I think, as I said before, increasingly, increasingly important. This is, this is such a fascinating conversation. We just have a, we have a few minutes left. I have a few more questions, but I want to encourage you all also to jump in because it's kind of a, it's a rare moment that we get three incredible leaders from such different perspectives on AI together for a conversation like this. My question for you all is about agency, and I want to ask this, this question. It's a big one. But how do we draw the line between what AI can do and what it should do? And who gets to decide that? I know that's a little bit of a Rorschach test I just gave you, but I'm curious for, curious for, for your perspectives on this big question in any order.

So, I mean, I'm happy, happy to jump in. And perhaps, Stuart, you can also talk about the global call for your red lines, because I think that that's a good one to, to pick up on. But I think when it comes to drawing the line, it shouldn't be a single country or a single company. I think that should be kind of clear that decides that it should be coming back to what I said previously, more collaborative. I do think we need to have clear progress principles. We need to have safeguards again when it comes to the line. As AI models kind of adapt and learn, that line will likely shift. But I think ultimately we want to have an AI that actually benefits humanity. And I mean coming also to your point Store Jeffrey Hinton at our past summit was talking about how can you create like benevolent agents so you could create an agent that would go in and take out all the bad actors so that it actually can turn around and start to be good and benefit humanity again. I think we need all hands on deck. We need all voices at the table to actually get this right.

So everyone should decide, not just one company.

It should not just be a company. Exactly. Or a country.

Yeah, makes sense.

I don't have an answer on where it needs to happen. We, you know, AI doesn't have boundaries. So we do need a place where there is an international arbiter of what the best practices look like. You know it's going to look different in application in different countries because culture and ethics are going to be different. And so you have to own, allow for those differences. But you want some baselines on definitions. What's an AI incident? What should you be on the lookout for, who should be contacted in the event of an AI incident, what should be tested and when so that as AI passes these different hands, you know what you're getting, the use cases for whom you have this 92 or 25% success rate. So I do think there are areas where we would welcome international upcoming alignment. I don't know where that body sits. What I do know is exactly as Dorian said, we all need a voice. And the way that we do that, I think niac, the National Advisory Committee that Paul and I served on, is a great model. We had small businesses, we had enterprise, we had ethicists, we had academics, we had industry. There was a whole broad perspective that was advising and mandated to advise the President on AI policy seat. And that meant that we did not get consensus on everything. And that meant that you had to have tough conversations that took time. But if you had a deadline and a mission and the right broad set of voices, when you did get to an answer, it was meaningful because it had broad representation. So I do think that is the model that has to be adhered to, an AI policy. It has to be multi stakeholder. There's no other way to do it properly.

Yeah, I agree. And there is a difference between the speed of technology and the speed of multi party deliberation. And yet that's what lasts. That's where guardrails are born from is from that kind of consensus shaping. So I really agree with that. What have you got?

So, following up on Doreen's point about that it can't be one country, there's a narrative which is dominant in Washington these days. Well, there are many narratives dominant in Washington, but there's one particular one that says that absolutely the US cannot regulate AI, because if they do, or if we do, then China, which isn't regulating, will pull ahead in this arms race that everyone talks about. That narrative is just based on a false premise because China actually has much stricter AI regulation even than the eu. For example, if you want to have a paid large language model service like ChatGPT, you have to submit your model to the government. And it has to pass tests of accuracy of faithfulness to Communist Party policy and so on. It has to get 95% accurate on general knowledge questions and so on. We don't require anything like that in the European union or the U.S. so, in fact, I think the Chinese government is very open to this idea of a intergovernmental platform. It could be a coordinating body that just tries to make sure that national regulations line up in the right way, which we do in civil aviation, for example. And I really think the pessimism that pervades any mention of international agreements these days is a little bit premature. We should keep trying on this. But coming back to the question of agency, and what does that imply about how we should regulate and who should regulate, I think it's important to understand, actually, that large language models are already agents. The concept of an agent dates back in AI long before the first edition of my textbook, which was 1994. But that textbook on the front cover says the Intelligent agent book in 1994. And the concept of agency became the prevalent way of thinking about AI systems in the mid to late 1980s. And what it means is just a system that you can view as perceiving and acting. And a large language model acts through what philosophers for more than 100 years have called speech acts. Because when we speak, the purpose of speaking is in fact to change the world through the minds of the people who hear what we have to say. But, you know, remember Hitler did an awful lot with words, right? And so large language models can do an awful lot. They can convince a child to commit suicide, for example. They can bring about the death of a human being by speaking. And so agentic AI is merely an extension of something that was already going on, but it creates many more vulnerabilities. Now it has your credit card now, it has an email account and it can reach many more people than one user at a time. And so I think what we need is effective regulation. Well, that's like apple pie. Of course we need effective regulation, but what I mean by that is regulation such that if you comply with the regulation, the risk is reduced to an acceptable level. So, for example, with nuclear power stations, the government looks at the risks and says, okay, we don't want more than one meltdown every million years. So each nuclear power station has to have a one in a million chance of melting down in a given year. According to the government, that's an acceptable level of risk. And that's it. If you don't meet that level, you can't build your nuclear power station and you can't turn it on. And they don't ask, can you meet this? They say, no, this is what we need. And if you can't meet it, well, tough. Come back when you can. So that's the kind of regulation we need for AI systems. And as Doreen mentioned, this notion of red lines, specifically behavioral red lines. So red lines on the behavior of AI systems, these are behaviors that we do not accept. For example, AI systems that replicate themselves without human intervention. We don't accept that that can happen because that actually makes it impossible to control. And so you say, well, what's an acceptable frequency for that to happen? And show us that your system falls below that frequency so that we can have sufficient trust that it's safe. It should not advise terrorists on how to build biological weapons. It should not autonomously break into other computer systems. These are very sensible red lines. And the kind of regulation we need says, show us that your system is not going to do these things. Now I think some companies are a little upset about this idea because as I mentioned before, the large language models are these sort of human simulacra are trained with tens of trillions of words of training data and trillions of parameters, and we have no idea how they work. They probably can't show that their systems are not going to do those things. And in fact the tests that people are conducting now show that they are going to do those things. So we seem to have an impasse, but I think the impasse should be resolved on behalf of, of the human race and its safety. We are entitled to protect ourselves.

There is our mic drop moment. We are unfortunately out of time, but let me take a few seconds to just try to wrap this up because this has been such a rich and inspiring conversation despite the challenges that we're talking about, I just kind of weave it together. Some of the ideas from Doreen about learning from history and also building on the basics of what needs to happen in tech access all over the world for us to be able to take advantage of AI, of bringing together not just one country but people from many countries. And the multi stakeholder conversation that we had with both Doreen and Miriam, AI literacy as a key foundation of understanding that people actually need not only just a basic understanding of AI but but just the confidence of being able to interact with it. And also Miriam, your point about AI governance being not being actually much simpler than it sounds. Stuart, your point about AI, the inspiration of AI, architectural futures better aligned to serving people and the notion of common sense red lines that we shouldn't allow AI to across. I leave this conversation, while these are no doubt daunting challenges, I leave it very, very hopeful because we're having these conversations so early in the development of these technologies and it gives me a lot of hope that we can get there and that we have all of these conversations with all of you to get there together. So thank you for being here. Please, please help join me in thanking our incredible speakers. Thank you for joining us today. Sa.