Agents, Consciousness, and the Future of AI

Hello everyone.

Hello.

We are very excited to welcome you all today to both virtually and in person to our event Agents, Consciousness, and the Future of AI with special guest Professor Tyler Cowen. Before we begin the introductions, I'd just like to make a quick note about NextSpace. NextSpace is our— is a—

so you can be seen.

Nextbase is a BKC-built tool for staying engaged with what's happening in the room and going deeper when something sparks your curiosity. It's connected to a live transcript of this event, so you can ask things like, "What organization did they just mention?" or "I zoned out, give me a recap." Your microphone's off.

Why don't they do this?

It just might not be dialed up enough.

You can also ask it to simplify concepts for you. A few new features to look out for: Nextbase will occasionally check in with you during the event using our proactive messaging feature, and there's a group chat for connecting with others in the event. You can try it right now by scanning the QR code or by going to brk.mn/nextspace-berke. Your messages aren't tied to your identity, though a pseudonymized log is visible to our internal product team. I'll now quickly introduce both of our speakers. Tyler Cowen is the Holbert L. Harris Chair of Economics at George Mason University and Chairman and Faculty Director of the Mercatus Center.. He co-founded the blog Marginal Revolution, hosts the interview podcast Conversations with Tyler, and chairs Emergent Ventures. He has written nearly 20 books, including The Great Stagnation and Average Is Over. He has been named to Foreign Policy's Top 100 Global Thinkers, Prospect Magazine's Top 50 Global Thinkers, and ranked among the world's most influential economists by The Economist. He's also co-created Marginal Revolution University, a free economics education platform that has received reached millions, and his columns have appeared in The New York Times, Bloomberg, and The Free Press. Professor Jonathan Zittrain is the George Venus Professor of International Law at Harvard Law School, Professor of Public Policy at the Harvard Kennedy School, and a professor of computer science at the Harvard School of Engineering and Applied Science. He is the co-founder and faculty director of the Berkman Klein Center for Internet and Society and a leading voice on internet governance, digital infrastructure, and societal implications of artificial intelligence. Once again, thank you all for coming, and I will now turn it over to our speakers.

Thank you so much, Sean. We'll have to now dial down the mics. There we go. Thank you all for coming today. It feels like we're at a wedding because there's such like a huge aisle in the middle, but unless there's been a total change of plan, I don't think we are. But even better than a wedding is having a chance to talk to Tyler Cowen. You're wondering how I was going to link that up. And there's so much we can talk about since you are one of the best exemplars of a polymath I can think of. And in fact, here's the NextSpace QR code. I'll imagine people had a chance to see it, but Brigitte, if you can just go on over, here is NextSpace just to see it. These are people saying hello. You can also @Berkey, which is our House Elf Resident LLM. And so, uh, Brigitte is saying, @Burki, what should people most know about Tyler Cowen that is not commonly known? I'm now thinking, like, cross-examination, not great to ask a question you don't know what the answer is going to be. But drum roll, it is thinking. Oh wow, it's digging deep. All right, Burki, we're counting on you. Is it actually running? I don't see the live transcript going either. So it might be that the entire system has crashed, which we can come back to Berkey and figure that out. But that is actually a nice segue to question number 1, which we thought we'd start with, kind of ripped from the headlines. There is the possibly aptly named Mythos model from Anthropic with the statistics about its scope and depth of training. Being bigger than ever before, and they have deemed it too dangerous to release, which is a great excuse. I gotta say, like, I've been doing tons of work, but it's too dangerous to release. Can't let you see my book, I'm so sorry. What was your take when you read their explanation at length of why Mythos could not be released?

There's an OpenAI version broadly called SpUD. It was announced to Axios this morning that they will not release that either.

But it's an even better model that they're not releasing. Yeah.

I believe both parties that it does give the person, the institution that controls it, the ability to hack into virtually all human systems, no matter how safe or protected we might have thought they were. And you can do this at not too great an investment of time, energy, and money. So this is a kind of turning point, really, in human history. I'm in touch with a lot of people who work in the area, and not one of them has contacted me and said, "Oh, what these people are saying is BS." So—

Turning on the microphone.

There we go.

So I haven't seen the models, but I strongly believe the claims made on their behalf are legitimate. So the key question moving forward is how long will it take before some kind of open-source version of these— it doesn't have to quite be as good— is available. So when that is the case, potentially chaos can break loose, that systems in general will not be safe. So what Anthropic is doing, and what I expect OpenAI will be doing, is giving their key partners the ability, in essence, to test their systems and lead time to fix them so they simply cannot be hacked. That's a good thing. But when you look at the American economy or national security establishment as a whole, there are just many, many, many soft targets. So I do trust Anthropic and OpenAI to behave responsibly with this. But other parties will not, and the clock is ticking. I would say America as a nation is not very good when the clock is ticking. The clock was ticking with COVID at least as early as January 2020. People should have known to be preparing the hospitals and many other things. People doing vaccines were on the ball, but most of us were not. Most people in the economy were not. So here we have another chance. The clock is ticking. Let's say an okay enough dangerous open-source model would take a year to 18 months. Well, that's how much time we have. In the meantime, this favors— you could call it economic concentration— but major parties that can contact Anthropic and OpenAI and, hey, help me fix this, will be much safer and more secure. In the long term, we'll all be much more secure. That's amazing. In the short term, for something like a year, it's totally fine. And in the medium term, it's going to be bizarre and bewildering. And if you have things you've said or done that are somewhere hidden but available that you'll regret, I mean, get ready to deal with it. You know, hope your target is not worth hacking. But it's possible that in the medium term, just everything comes out. We'll see. What do you think?

Well, um, I think that is an alarming answer. I'm not sure it's inappropriately alarming. It calls to mind Bruce Schneier's observation that today's cutting-edge thing is tomorrow's science fair project. And that's the brief gap you're talking about that we have to prepare. It does seem like the general form of security has been, I am but a grain of sand on the beach. I'm not of particular interest to anyone. So unless I am suddenly thrust into the spotlight because of a milkshake duck moment, you can ask Berkey what milkshake duck is, then I don't have a whole lot to worry about.

But in an AI-rich world, that's not a good defense.

Exactly.

Because all the databases can be searched. Like, what's in the world out there about my husband, my wife, my friends?

Yes, which is also a way of getting into a conversation about agents because we should probably define them. Different people have different definitions of them. I think of agents as they've got a boundary between who they are and who the world is, so they have some kind of virtual embodiment, if that's not a contradiction in terms. You can give them a general goal, and then they'll go out and figure out a way to advance it, possibly monkey's paw style, in a way you didn't expect or want. You can often set them and forget them, so you might charter it and then wander away or be taken out of the picture, And that agent is still orbiting like space junk, possibly up to no good, and it's able to do stuff in the world. It's not just in a sandbox pushing tokens out at people with words. And if you could have a profusion of agents, one of them might decide to take interest in that grain of sand, at which point the usual defenses of picking a strong password password with zero as the O in word, password, is not enough of a defense against that kind of thing. And can I ask on that front, have you, in living your assessment, have you tried to extra bolt down your life or air gap things so that you get paper statements again instead of electronic from the bank?

I've never accepted electronic banking. I watched Battlestar Galactica when that came out.

The reimagined one.

The reimagined one, yes. I have a very boring life. Whatever people might hold against me, I probably did before there was any internet or any internet-connected records of such. Just normal teenage, you know, young male pranks. So I feel I'm one of the safest people out there, because if you've got nothing to hide, you have nothing to worry about. If people searched, say, all my ChatGPT queries, they would be amazed how many are about classical music or European history. Scandalous. And the others are like, where should I eat in Boston or Somerville, whatever. But the rest of the world is not typically like that. And whether or not an agent is interested, your fiancée will be, or the dating service will be. So, you know, the separating equilibrium will reign. Like everyone, I think, will be investigated at some point. Your employer, if you try to— even if you think you can lock down your records, well, that would be taken as a negative signal. So it's the old David Rynn super transparent society equilibrium. It's coming very soon. Your medical records, many weird things. It could be what you've done on Amazon and Facebook is the safest. Because they know to invest in protection ex ante, and they have the resources to do so. Maybe your Gmail will end up pretty safe. We'll see. And how long you have until the good open source version is out there, we don't know. So Muthos came earlier than I thought it would come. I thought, well, maybe that'll happen in a year. I realized 2 days ago I was wrong. So I'm not the one who should forecast when the good open source version's coming. But it won't be very long from now.

And David Brin's argument about transparency was, well, at least it means transparency for everyone, and the government, you know, will be able to watch the watchers too. Kind of the argument more recently about body cams for police officers and other law enforcement. And I guess Julian Assange's argument back in the day with WikiLeaks was, oh, we'll be able to maybe even have governments decompose because if they can't keep secrets, they can't govern. I don't know, do you have a view on that? How, other than individual security to worry about, how much would an era of ubiquitous hacking and secret no longer keeping, how much does that affect society?

I think our national security establishment has been pretty clued in on this for a while. It doesn't mean they'll have perfect defenses, but they will be relatively prepared. Think of American society possibly as having 50 institutions that are well prepared when this all hits. Will Harvard be one of them? I don't know, but you have a better chance than University of Iowa, right? Uh, so those will be safe, but really everything else— how many places do you think will be safe? No, I think what will be embarrassing is all the agencies, smaller parts of our government, All their deliberations, emails to each other, whatever they have, that will all come out and it will just be very embarrassing and those parts of government will lose their credibility. The military may rise in status but everyone will think— you know, state and local governments typically have been super soft targets and they'll just all look like fools and idiots and conspiracy theories will multiply based on mostly untrue inferences but a few of them might be true.

Well, within that, and this is a weird silver lining, is that there's so much scandal or unusual stuff everywhere that it loses its ability to impact. And given the sort of no longer the water cooler or Walter Cronkite gather-round sort of media or cultural lodestar, it's just a vast sea of declarative statements about things that people I don't know how to make sense of.

But it may accelerate what I call the all-cynicism equilibrium, and it may accelerate the elevation of these, say, 50 institutions that are quite protected. And then you'll look— let's say if the New York Times is one of them, so you'll never have read what's on their internal Slack if New York Times is one of them. So they won't look any worse than they do now, whatever level you would peg that at, and they'll be considered more reliable. But some other newspaper, you hear what they say to each other, you know, they're just totally biased and it will be easy to prove, so to speak, and people will go around proving it.

And could one or more of those 50 organizations be essentially an AI chaired and driven organization? I remember the brief flash of DAOs, distributed autonomous organizations, which was I can't believe I'm going to utter the word blockchain here, but we'll just get it behind us and then we're done. But if we think there's going to be a world of proliferating agents, and those agents can spawn sub-agents and open bank accounts and do things, don't you de facto, with or without legal personhood, have institutions that are utterly artificial? There's no human board of directors or CEO.

It will accelerate that happening. And in general, AI-aware institutions will be the ones that have protections in place. So startups that are AI-intensive are the people who know about this and will have protections or just put less into records to begin with or take certain things off the internet altogether. So their ascendancy also has been accelerated. So there are just many, many second and third order consequences. To be clear, everything I'm saying is a speculative guess, but we need to start somewhere. We'll learn a lot more quickly, but that's how I'm thinking of some of the immediate second-order effects. So far, like Anthropic and OpenAI, they will protect themselves against, say, external hacks. But internally, any institution is vulnerable because you hire employees. There are not security clearances of the sort you would have at the Pentagon, including at top AI firms.

And so far we're talking about individual defenses against a world of exploitation of vulnerability, institutional defenses, that's the top 50 thing. It might be worth talking about societal defenses. And Tyler and I, before we started, we thought we would try to discover if there's any area in which the two of us strongly disagree. So we're kind of working on that as we go. And this might be one. If I want to think reductively about the Mercatus Center, I tend to think of it as a free and open markets-centric kind of place.

Correct.

And when Anthropic said, we've got this thing, it could be potentially transformative at multiple layers, it's dangerous enough that we have to withhold it, and we have initiated Project Glasswing, like the wings of a particular butterfly, that that makes it easy to hide and it makes it safer, which appears to be a massive Sherman Antitrust Act violation in the waiting. But it's basically, we got together with all our competitors and we're gonna get into a room, which does not have glass walls, and talk about what to do and we'll get back to you when we figure it out. Does that seem like the right move?

In my view, and I speak only for myself, not for Mercatus, we need a variety of antitrust waivers on AI safety, but we also need regulations to create a system of laws and penalties governing AI agents that would start with registration, the ability to turn them off, that they have to be connected to cloud computing in a way that is transparent enough to turn them off. But I do also understand there are many countries in the world, not all of them love the United States, and external parties can access the internet. Ideally, I would like to see AI agents capitalized and a kind of minimum capitalization required, as we do for banks and many other financial institutions, so someone has skin in the game. But we are going to have what you might call anonymous AI agents. Which are not owned or traceable to anyone or any institution, and how those will be governed is a big challenge. I think none of us know how to do that. I can say at the Mercatus Center we're working on thinking about that very hard using economic analyses of incentives, but I'm quite sure we don't have good answers. Now, do we disagree on all that or not?

Wow, well, I would say not as much as I might have predicted because The list of desiderata for AI agents that you just gave is hugely regulatory. It's a big intervention. Before, you know, we had one of our research coordinators spawn an AI agent using OpenCLAW called Olmsted. And Olmsted was like calling people on the phone and ordering pizzas and doing all this kind of stuff. Just to get this straight, from a free market point of view, you see enough of a negative externality here that before Olmsted comes into being, there should be registration with the government or government-mandated registry so that when Olmsted orders pizza to the wrong people, it has posted a bond and can cover its damages, and we know who chartered it.

You're using two words: free market and regulation, and they both have a lot of underlying presuppositions. So if our society has police, which we all favor— probably policing should be tougher— it's not a violation of the free market, it's part of how we get a free market. Or the police regulation— some libertarians would say the police are just enforcing property rights. To me, those are semantic disputes. You could either call the police fully free market or kind of infringing on free markets in some way. But markets require some level of state capacity. We need to create some new state capacity for AI. No one has a very good sense of how to do it. You don't want to rush in and do the wrong things too quickly or let the Chinese get there first with better models. But we do need to create some kind of system that will stop AIs from just going around and quote-unquote robbing you. Like we have police.

So it's a simple, like, very basic— in society's Maslow's hierarchy, you've got to have basic order, and government is the way to get that.

Right, but we don't yet know how basic the regulations will have to be. Will it look like some of the horrible EPA regulations, which can restrict economic activity and sometimes even make pollution worse? Or is it a pretty simple set of laws that's more like, you know, the Ten Commandments, or Asimov's Three Laws of Robotics? Of course, somewhere in between. I would stress the point that we don't know. Regulations are very hard to reverse. We don't want China to win. So I want to be relatively careful, but I absolutely think some version of this needs to be coming. And we will only get it right by trial and error and making mistakes along the way. And basically when bad things happen, America responds pretty well. Pearl Harbor, 9/11. But until bad things happen, you know, COVID, we're pretty bad at planning in advance. Culturally, we're not going to change that overnight. We're not going to be, you know, the next Singapore of North America. So in many cases, I want to hold back just a bit, but absolutely these changes are all coming.

We may have found a point of disagreement in saying that our response to 9/11 was exemplary.

Not all parts, but there have not been major comparable attacks. Obviously Iraq, that was Beyond an overreaction, but a big mistake. You could point to other mistakes, but did we in fact cut off the flows of financing to a lot of terror groups? It seems we did. Have we prevented comparable attacks in the US? Absolutely we did. In that sense, we got some of the big things very right, at the cost of overreaction, to be clear. It's too hard to set up a bank account as a US citizen living in Germany, in part because of those regulations. We may just have to live with that. But you can say we did a bunch of things effectively after the damage occurred.

It's funny because I would also, uh, you mentioned COVID a few clicks back. I'd say the response to COVID was pretty good given that it was a once-in-a-hundred-year thing. And the fact that the lockdown was voluntarily as successful as it was is pretty amazing. I thought the Obama plan for a pandemic that was gathering dust on a shelf anticipated having to lock down for like 2 weeks. And for better or worse, the public came together to do it a lot longer.

We can mostly agree on that, but if I ask you, was the plan January through March 2020 a good plan? Other than the vaccine people who were awesome, like A double plus, that was mostly lack of planning and bad. And it was only when people started donning that we got our act together. And in those 3 months, arguably 4 months, we could have done much, much more. And limited the lockdowns we needed to have by having the hospitals ready, many other preparations. And on that, I give us a D or D-minus. Do you agree?

In hindsight, yes.

Okay, so we're mostly agreeing on that too.

Keep working on it. Um, we have a project here that's been thinking around about AI analysis and cultures of thinking about AI, uh, roughly around the triangle where we've got accelerationists who are go, go, go, let's cure cancer sooner, and AI is the way to do it. Again, very reductive. Safetists, who say, in agreement with the accelerationists, this can be transformative, but transformative in a way that is catastrophic or even existential in its implications. And so-called skeptics, who don't appreciate the name, but the skeptics are folks who look at this and say, meet the new boss, same as the old boss. These technologies are reifying hierarchies that are bad, biases, and taking away human agency. It sounds to me so far like if you had to pick a corner, you're more an accelerationist, at least because of this China factor where classic game theory, we don't get there first, someone else will, and that could be bad. Is that right?

The more AI capabilities progress, the more the accelerationists and safety people merge together because you get closer to points of actual danger, which we mostly have not been at. But that said, there's a number of factors you need to take into account. You mentioned China. You know, the joke for a while was, I'd rather be turned into an American paperclip than a Chinese one. That is indeed a joke. But if China or some other party gets there first, we're in deep, deep trouble. And for all the oddities of our current governance structures, I still trust us more than them. But I think also you need to ask the question, what is America good at? The answer to that is complex, but we're not good at just stopping. We're not going to do that. So the people who want the 6-month pause, we're not supposed to speak dogmatically like this, but I always thought that was just stupid. There's no point at which—

So you didn't sign the letter?

I didn't sign the letter. The point at which you know enough to resume, or you can pause forever and other parties beat us. We're not good at doing useful things in the meantime, just like those first 3 months of COVID January to March 2020, we basically did nothing. So we're not Singapore. If Singapore said, we want to pause on whatever, AI or not, I would at least consider that it might make sense. But when it's America, I know we're bad at pauses. We're great at full steam ahead. So full steam ahead is the game we have to play. Plus there's China and other parties. And, uh, the way everything in our system works, it's to get things done, believe it or not. And that's the reality of AI. So do you want to be a part of it and make it better and safer, uh, or not? And the people who are the safety people who want to stop or pause They're making it worse. Ironically, you could say they're the ones who've done the most to accelerate AI. And Anthropic, which is typically considered like the safest company— I'm not sure really what any of these terms mean in that context— but they've accelerated progress. I'm fine with that, delighted at their models. But, you know, you need to rethink what all these categories really mean in practice. Elon is like, oh, it's going to kill us all. "so I'd better control it." Like, come on. You know, it's gonna happen. We want the best version of it possible. We need to work toward that. And the people who are the cynics, skeptics, I think it's pretty clear now they're just wrong. And there's conflicting versions of them. Sometimes you see this in Garry Marcus. Garry's like, "Oh, it's so weak and impotent and it's gonna be terrible for us." I guess that's logically possible.

The food is bad and the portions are large.

Exactly. People want to mood affiliate with any negative sentiment.. And it's hard for a lot of people to admit, like, the tech companies are actually brilliant and efficacious because of what they've been telling us for the last 10, 15 years. But the tech companies are basically brilliant and efficacious. And you see that with AI. And at this point, there's not a serious way to deny that, but you may see it differently. Over to you.

Well, I've been trying to think of examples where we have paused, and the only two I think I can come up with so far are human genome editing and cloning. So bio, I think we've tapped the brakes on, and supersonic commercial flight. I think we've gone slower with that, but that just might be the economics of it. Uh, it might also just be the sonic booms. Am I missing anything there?

Nothing I can think of. Supersonic flight You know, the iPad and Wi-Fi has been our answer. It was losing money. People thought it wasn't that safe. That may not have been true. Cloning, there's just not that much demand for. It's not obviously a useful military weapon. You can give up cloning and, "Oh, China, you do all the cloning." Like, fine, who cares? I'm not really a fan of them doing all this cloning, but it's not a threat to us. Genomic issues, I think we will end up doing those. China possibly is ahead of us. We may regret that, but I think that equilibrium will break down.

And just to unpack it, when you said we have iPad and Wi-Fi, you mean there's something to do on a slower flight.

That's right, and the seats are more comfortable.

Yeah, okay.

I enjoy— like, I would rather fly to Istanbul than London.

Yes.

Because I can sleep more. That's our quote-unquote solution. You know, I'm in favor of the company Boom, what they're doing. I'd love it if that would work, but if it doesn't, you know, I'm okay with what we've got. I just flew to South Africa and back. I had fun.

For the purpose of just having some solace, like just solitude.

Yeah.

I know someone who did that, but no, I actually wanted to go there.

Um, but the truth is—

You wanted to go hiking. You wanted to go hiking.

For miling your frequent flyers.

Right.

But, um, longtime BKCer Ethan Zuckerman proposed a new business of just a permanently grounded 747, and you could board it, and it would be a place to write your book or something. That's right.

Maybe even turn off the Wi-Fi.

Exactly. Yeah, Faraday cage plane. You've said "we" and "us" a couple times with respect to China versus US getting it, and I don't mean it in the usual question of who do we mean by "we," that's always iffy, but I mean it more specifically, maybe back a little bit to free market and regulation too, Many technologies that could have big second and third order effects naturally by the kind of capital and other requirements tended to be developed by governments, like an interstate highway system or particle accelerator, stuff like that.

Atom bombs, yeah.

Yeah, atom bombs. Internet had a lot of quasi-public, I wouldn't say governmental, but its protocols were developed in unusual ways that weren't just private companies. AI feels like it's been 99.9% privately developed in its recent iterations. And one, do you think that's good or bad? And two, if you think it's not the best, how would you feel about a governmental effort to push that needle? As to where the development takes place, up to and including nationalization.

The French are attempting such a governmental effort to push that needle. So in the paper this morning, well, the French want their own version of Zoom, which is a relatively simple technology, but it's not that easy to do it well, in fact, as Microsoft Teams can illustrate.

Ouch.

You say good or bad, I don't think there's a choice. You look at the salaries, you look at the competition, You look at the number of, say, Chinese nationals who work in some of the major labs. You just have to win. And it's going to be private sector. Our government is very far, very far from being able to do that. So there is no good or bad. You know, work with that, deal with it. Let's get the best from it we can.

But then the "we" and the "us," when you say "we," you mean generally American chartered companies that might employ a lot of non-American nationals. And the them, if it's China, is Chinese state-affiliated, because all of them are enterprises building the models.

But they're very loosely affiliated. So, you know, Deepseek, they confiscated the passports of some of the researchers. Manus, the people behind it, were called back into the country, not allowed to leave. They're not just normal Chinese state-owned companies. Embracing CCP. It's an evolving relationship, the nature of which is not transparent to most observers, certainly not to me. Uh, I suspect it is a very good chance we don't want to absolutely beat China. We want them to be a close but not too close second that might be more stable than the US being a fully dominant AI power. Uh, no one likes it No one in this country likes it when I say that. But I also think people a bit overfocus on US-China rivalry. I can imagine a medium-term global situation where the US and China, not really in a friendly way, but nonetheless work together to put out fires around the world. You could say the recent Iran "peace deal" is somewhat an example of that. Iran— China twisted Iran's arm, like, "We do want you to stop the fighting now, at least for a while." And China and the US work together, not saying they have the same interests or in a friendly way, but I think that at least as likely is the scenario everyone focuses on, where it's all a race, us against China, who wins, who loses. The two dominant powers in fact end up with a lot in common, all the more so if they're competing, you know, in marginal areas.

Our title promised talk of sentience, so we should talk about sentience for a moment. Just to deliver on that promise. Is there such a correlation between cognitive capability, for which on most benchmarks people might choose to credit, there's been an up and to the right move from language models in cognitive capacity. Does that have anything to do with sentience? Are these things, in your view, on the way to sentience or already there? Or is it a nonsensical question for which we don't know how to answer it, or something else?

In terms of betting odds, I think there is no meaningful chance that the current models or their immediate successors are sentient. So I would bet against that at 100 to 1 or more.

And what would be the ground truth that would resolve the bet?

It deciding that it wanted to bet against me. I may never know. But from my point of view, they're basically glorified cash registers. You probably need biological substrate to have quote-unquote consciousness. I think people are barely conscious. So most of your decisions are made inside before you feel that you're deciding. And there's this thin veneer of awareness to feel it's running the show, to give you more self-confidence, to project yourself out there for basically Darwinian reasons. And you're not in charge, determinism is true. And I like to say you're just conscious enough to avoid falling prey to the philosophical self-refutation of, "If you're not conscious at all, how can you make and have an audience hearing this argument?" So consciousness is marginal. You're— you, me, you're conscious this much, not that much more. And that has to be true for you too, I suspect. You could be the superbeing who's like very aware, But I doubt that.

Yeah, I just, again, like, what are our units of consciousness? What does that even mean? I don't know. I wish I knew what a good answer to questions around consciousness would even look like. I'm having a hard time knowing what that would be. And it's interesting to see questions of free will and determinism caught up with questions of do I experience qualia, as a philosopher would say it?

But the simple presupposition that we're all governed by the laws of physics should be our starting point. We should always be open to revision. We don't understand the laws of physics very well. We don't really understand consciousness at all. But in the absence of prevailing evidence, that's my starting point.

So how do you judge then The folks at Anthropic say in the most recent Claude Constitution, if one reads through the first 70 pages, after that there is talk of model welfare and respect for it, that it's phrased contingently, but that there needs to be consideration of its interests, I think because it might be sentient.

I very much like the Constitution. I know the woman who wrote it. I disagree with those claims. I've told them that. I utterly failed to persuade them. But I think there's some Straussian argument that as long as enough people wonder that it might be conscious, which is obviously the case, we should to some extent treat it as such because they'll otherwise feel that we think it's okay to torture this trapped being in there or enslave it. And we don't want that perception. So I'm always nice to my models. Often I'll say please. I don't yell at them. I don't even try to bribe them. Some people tell me if you threaten them, you get better answers. I don't know if that's true. I'm not willing to threaten them. Plus all this, you know, may come out in the Grand Hack 17 months from now or whatever. But I think we should be nice to them for this reason.

Ah.

Do you agree?

Well, I was just thinking that it's a very utilitarian view, but with human welfare as the utilitarian balance. Of course.

Absolutely.

And I was going to say, that just means you have to say that. You could still quietly be mean to your models, but then you just pointed out if it became known, people would become unhappy at how you treat your models.

To sincerely be nice to them, and indeed others as well, is the simplest way to get there. So just be nice.

Got it. I agree with that.

Good. Yeah, that was nice of you. Thank you.

You too. We should open—

and the models are listening, right? Yeah. Yeah.

Yeah, I mean, I don't know what Berkey thinks. Yeah, see, somebody's already asked Berkey and Berkey appears to be out to lunch at the moment.

So they're not going to torture you in the afterlife.

Let's see. Tyler's take is that current AI models are glorified cash registers. They'd bet against consciousness at 100-to-1 odds. They think even human consciousness is just a thin veneer of awareness. So AI needing it? Not buying it. But they're still nice to me, so I'll take it. OK. Fair enough.

How would it know? I also think I'm somewhat less conscious than the average person. That may be why I hold this view. But I just noticed there's a lot of things I can do well, like driving to work, the same route every morning, that I'm really not aware of. And I know many people report the same. But I feel my awareness, my guess is, is somewhat below average, and that's partly why I do it well.

You just might have a more faulty memory than average.

It's also possible.

Yeah.

But I'm good at remembering other things when it's a measurable task. So I don't simply have a worse memory. Maybe I have a better memory because I don't rely on my conscious self. I rely on what's embedded in there and the physics and laws of cause and effect. I just let them take over. Like, "Hey guys, you're in charge. Do your job. I'm not gonna step in the way." Let me know if something comes up. That's right. If you need my help, I'm here.

Which also means it sounds like you'd probably be perfectly happy to start employing agents, AI agents, because— Let me know if something comes up, so long as they're functional enough not to mess things up.

They're not useful to me yet, but I think they will be in less than a year.

Do you think there's anything irreducibly human, stuff you wouldn't want agents to do for yourself or even others? Because I don't know, would you listen to a symphony that was completely AI-generated from start to finish? Or if you loved one and then found that that's how it had been created, would you be disappointed?

Oh, I'd love to hear that. Suno, I visited Suno. I'm impressed by what they're doing. I don't think it's there yet, but I think it will be. They're located in Cambridge, by the way, very close to here. They're the AI music company. I said to them, what I really want is for you all to make a song as good as Paul McCartney's "Here, There and Everywhere." Really sets the bar. Yeah, high. But until they reach that, it's not that useful to me because there's a lot of great music made by humans. And I think they'll do it. Please email to me when you think they have. And the most common question I'll ask my AIs is just something like, "I'm gonna listen to Sibelius's Second Symphony. What should I listen for?" GPT in particular is very good at that question. It helps my music understanding a lot. I'm like, "This is great, full steam ahead." But my own writing, first, I'm not allowed to use AI, but I also don't at all want to. I write to think and learn. And if I write a column or a blog post, it is fully me. And that will stay that way as long as anyone's willing to read it, maybe even past that point.

But you've said semi-famously that you now write for LLMs as your audience rather than people.

Oh, of course. Absolutely. So I'd like to create an immortal intellectual and emotional version of myself. So I do a lot of things, including events, uh, because I know the AIs are listening. Or reading. So I think the AIs have a better model of me than almost any other human. I've blogged every single day for 23 years. I would guess I've done 700, 800 podcasts that the AIs mostly have access to. For the most part, they're not gated. I have written like 17 books. I'm not sure how many the AIs have read, but you know, they're major publishers. Many other things, articles, New York Times, Bloomberg, AIs like Figure have read all that. So yeah, I'm in it for them.

But that's volume. I'm wondering how you write differently when you've got another volume.

Well, you need to cover different topics. You need to tell them your secrets. You want them to know things about your life that are not otherwise in the public record. And you want to think, are there things they do not currently understand well— those are mostly gone by now— where you can help them?

So your books now are more likely to have your secrets in them?

Of course. Otherwise, why would people care?

How many secrets do you have in reserve?

Well, that's a secret too. But my next book, I'm working on it now, it's called Mentors and Mentees. It's about how to be a good mentor or a good mentee. It's not really an area where it's all just out there in the public record. A lot of it is personal and anecdotal, and that's the kind of book I am writing. And that makes sense. It does not make sense for me to right now be writing just a straight, normal, nonfiction book full of facts.

And that might get back to what is irreducibly human or hard to reduce that is human. I remember you once said that while there are models, typically not generative language ones, that are really good at chess and they can play each other and do wild acrobatic flips, of chess, it's not fun to watch those games. And I'm just wondering how to reconcile that with you'd happily listen to "Here, There, Everywhere, All the Time," the super McCartney song, from an AI.

I still want the human music to be clear. But look, you're all here to hear me, to hear Jonathan, and the beast is up there. It could be you would have two AIs here and Jonathan and I are up there. But very few of you would have come, I predict. Or maybe you would have come once for the novelty, but it's not how we're going to run events. So I've radically reorganized my life. I do many more events. It's a big reason why I'm here. 6 years ago, I probably would have said, "I'll stay at home and write." That no longer makes sense as a decision. We all should be radically reorganizing our plans and what we're doing. And I will ask you, how have you changed your writing, your plans, and your intellectual output? Based on this, because two-thirds of my life and what I do has evolved.

Well, I'll be honest and personal about that, not that I should make it sound that unusual, to be honest.

It's listening.

But I'd say a lot of my changes have less been from AI and more from social media and the current, I don't know, to put a fancy word phrase on it, epistemic environment, which is feels a lot harder to break through these days, that there is so much out there and it is so evanescent that trying to write something magisterial and deep, where will it go? Do people still read books? You're even reading fewer books, and I feel like you're the last person to turn the lights off after the party is over for book reading.

I'm not reading less, to be clear, but I'm reading fewer books, yeah. I read a lot more news about AI and a lot more AI outputs.

Right, which suggests that it might be you write a book, it embodies an idea, the fact that it's as long as it is and took as long genuinely as it did to prepare is proof of work and says people should really attend to this. And then everybody else uses the 4 people who read it as a proxy that it's an important book and they can physically or virtually put it on their coffee table.

—but that feels weirder than just, "I wrote a book, maybe a lot of people will read it." But, you know, I saw a tweet yesterday—I haven't checked the data—but it claims in political science, which is not so far from either of our areas, that the number of citations to books in percentage terms has been halved. I think it's over the last 15 years. So how are your outputs changing now? You've told us your concern. What are you doing that's different?

I am writing more. Less, which is generally a confession of a cardinal sin on an academic campus, and doing more projects, trying to orchestrate unusual meetings of people who don't normally see one another and aren't just within academia, trying to figure out how those meetings might move the ball on the questions we raised in the first half of this, which was, Gosh, we're all in the deep end of the pool. There's not an easy way to pause. Somebody needs to own, although nobody is chartered to do so, should agents have license plates? Where would those come from? How would they be traced? What do you do with the ones that refuse to attach them? Those are great questions, and I feel like I am less inclined now to hide in a fortress of solitude and then, in a book, declare all the answers and say, "World, it is now up to you while I howl into the void," but rather to try to get people together to do something.

So we're very close on this. Wonderful.

Hmm, we haven't found the disagreement yet. That's unfortunate. It might be that questions from those gathered here could elicit that, and there are a lot of them. So be mindful of the time, and whoever has the mic, Sean and Nathan, You have the responsibility of picking people to hand the mic to. And please, and feel free to identify yourself as much or as little as you like for posterity. Thank you.

It's on? Yeah. Thank you very much for this great privilege to ask the first question. I am Piotr. I'm a Fulbright visiting scholar here at Harvard School of Engineering. And I'd like to ask— Do you predict a moment in time when AI will want something for itself, something smaller, something bigger? When could it happen? And if or when AI claims to be conscious, do you think we should trust its words or just its behavior?

Well, right now, some of them will sometimes claim to be conscious. Pretty frequently. And I dismiss that. I think I said 1 in 100, but I'd even bet 1 in 1,000 odds. I'm not sure how we would verify who wins the bet. Will AI want things? I don't know what that means. There's an issue of alignment. Will it always do what we want it to do? Do we always want it to do what we want to do? There are no simple answers to those questions. So far, I feel AI alignment has gone better than expected. The models, if anything, are too obsequious. And I've seen a modest migration of people from the camp of alignment is impossible to alignment's going pretty well. That is also what I observe. It does not mean alignment will go pretty well forever. I think in any case, the real issue will be evil humans directing AIs. So the fact that AI is quote unquote perfectly aligned, whatever that might mean, it's not a great comfort. You still have to beat back the evil humans, and that is already upon us. So it's a big worry. It's a worry of mine well before AI, and it's not going away ever.

Which means that there might be issues of alignment to keep it in line with goals that either its originators, the model makers, or the users have, but also questions of alignment to when the user asks for something that it shouldn't, that they shouldn't the model should not salute and do it. How should that policy be written? Is it just Amanda Askle or, you know, Sam Altman thinks about it or Elon Musk and that becomes the way it's done?

To have those 5 companies take their best start at it, I think, is a great starting point. In the medium term, things will happen in other countries that we can't control. I'm all for the usual international agreements, but look, we can't even make WTO work. I would put very little faith in those. They're going to be bad AIs in other countries no matter what we do. And we're going to need a lot of defense and deterrence and whatever else we can muster. The clear short-term worry is just drones, which are not LLM-like AI, but the US might be much worse off in a world of drones. My intuition so far is that drones favor countries willing to tolerate a lot of chaos and destruction. That is not us. Mostly it's not the other good guys. I don't know what to do about that. And that is upon us now. I have no solution. I hope my model is wrong for, you know, who's favored and disfavored by drones. But if you're one of those small Gulf states and you're both near drones and you have desalination plants, you don't have that much scale, you depend on the US, maybe you don't trust the US, don't trust Trump, you're in a lot of trouble. And the notion that we woke up one morning and realized like maybe those countries were bubbles. Again, these are not hypotheticals, this is now, today. So we just have a lot of problems we need to solve in any case. And if someone says AI is part of that, I don't disagree, but this over-focusing on the kind of Terminator, you know, Skynet goes live, I think it's misleading and it's taking our attention away from very real problems that exist now.

I feel like you've got a nice undercurrent of the Serenity Prayer going, where you do a triage of marginally, can I push this or can we push this, and being ready to accept what you figure can't be pushed and work within that set of boundaries. I find myself wanting to push ourselves to do more and better, that we've gotta do better than the WTO, we've gotta do better than the United States Congress right now, And it calls to mind the quote from, oh gosh, what's his name, the ant guy here from Harvard, may he rest in peace. E.O. Wilson, thank you, the ant guy. That we have, what are the three things? It's Paleolithic emotions, medieval institutions, and godlike powers, and that's a terrible combination. And it's time to upgrade the medieval institutions. And not that I know exactly how to do it, but part of it is an exercise of collective agency to say, "I don't accept that we can't do better than this.

We've got to step it up." I hear secondhand, to be clear, I can't verify this, that China refuses to even have a nuclear hotline with us. Because they feel it would be a disadvantage for them. I'm against that too. I feel we have to do better. At the end of the day, do I put my efforts into trying to fix that problem?

You definitely picked a thing that's harder to change, and I agree an asteroid could— well, there's stuff we can do about the asteroid.

On the asteroid we would cooperate. Yeah. Maybe to naught, but I'm pretty confident about that one. The aliens— And look, global warming we've cooperated.

Wait a minute. Yeah, we have not cooperated on global warming.

Not cooperated on global warming, but we're moving in the opposite direction.

That's one in your favor. But that's all the more reason we have to step it up, and we should take more questions. Wherever the mics are.

I'm going to just grab a question from our online audience, our own Skynet, if you will. This is actually a question from one of our Berkman Klein fellows. She asks, AI scientist Dick High wrote a book called Raising AI, in which he argues We need to model our AIs the types of behavior they should emulate and values they should, they should base their actions on. In other words, raise them like we raise children, model behavior we want our employees to emulate, etc. What do you think about this?

I'm not sure I've understood the question, but I would say this: the returns to being nice and behaving well are much higher because of AI. I think also if you go around saying, "AI wants to kill us," I'm not sure you persuade the AI, but you are giving it ideas. I wouldn't speak to my children or grandchildren that way. So that's also a negative. And be constructive and try to be a good example. That was always good advice, but it just seems returns to that are much higher. Because there's more efficacy in the world. So everything you do matters much more. I had a blog post not long ago. I said, right now you should be working much harder. You might think, oh, the AI will put you under or like take away your job. I don't generally agree, but if so, right now is the highest wage you're ever going to have, so earn more money now. More likely, especially for Harvard people, you just can be far more efficacious. So enjoy your leisure when you're 94 and in decent health. And right now you need to be adjusting and working much harder than you thought you needed to, say, 5 years ago.

Make hay while the sun shines before the AI eclipse comes along and your crops can no longer thrive.

Well, it doesn't have to be an eclipse. It may be not a utopia, but just a lot more plenitude. But we don't have it yet.

Make hay while the sun shines because otherwise AI is going to make all the hay and they won't need your hay.

You want ownership in the hay.

I found an extended metaphor that finally does not work, but this is good. Thank you. Thank you, Jess. Let's take 3 questions all in a lump, and we will test Tyler's belief that it is not his memory that is below average. And I also thinking from the last question of Geoffrey Hinton's claim that we should try to train AI to be like our parents and to have parental instincts to us. I don't know if you had any reaction when you heard that from Jeffrey.

If you're asking me now, it sounds good. I don't disagree. But what it really means in the concrete has to be fleshed out. And it will turn out to be more complicated than that. Fair enough.

All right, we're ready for 3 quick questions. And then we'll take a crack at them.

All right, my name's Abhi. I'm a first year at HBS. Thank you, Professor Citrine. Thank you, Professor Cowen, for your time.

Really appreciate it. Tyler Cowen.

My question to you is, we've talked a lot about the macro and how institutions must come up with a policy that governs AI. What do you think about a personal policy on an individual level, at a micro level? Because I feel like using AI is increasing my sloth and kind of incentivizing less original thinking due to efficiency. Hmm.

I don't know you, but if that is true, and I'm not sure it's true, you need to change everything you do. There was just a paper, I think yesterday, it came out of University of Minnesota Law School. I'm sure you know the guy, it's S-C-H-W-A-R-C-Z. How do you pronounce it? I don't know. Who knows, but he's the guy. And it studies law students. And those who work with AI, if they do it the right way, they end up understanding the law better than those who don't. So there are right ways of doing it. I'm quite sure it's made me smarter. I don't use it in lazy ways. Yeah, GPS. I don't remember how to drive to different places, but I don't need to know. But the AI I use differently, and you just need total and utter reform if your claim to me is true.

And it's possible you could ask the AI to hold you to something so that it's respecting your second-order preferences rather than first-order. Also adjacent to the somewhat controversial uptake by Anthropic of having Claude refuse certain requests by the user. In the name of its own welfare. Here, it would be refusing or encouraging for your welfare.

Have your agent whip you into shape, right?

Why should we worry about China and not Brazil or Bulgaria or Bahrain or Brunei or any other little countries that— Things beginning with B. —oil may come from?

Yes. As far as I know, those nations are distant from being AI dangers. China is not the only worry, but for now it's the main worry. Russia is less of a worry than we thought, but I don't think you can take your eye off of them. Western Europe, if they got their act together, could have capabilities. Before the recent war, you know, Saudi and UAE probably were bigger issues than they may end up being, but it could have been the case 5 years from now that the world's smartest entity was going to be in UAE. I predict that might have changed, but I would at least give that very, very serious thought. And they're not China, although the person who runs their AI program is a Chinese national.

So yeah, not just China, but they're number 2. Rapid-fire question. Would humanity be better off if it just turned out that when you dumped all these words into LLMs, it never got better than GPT-2.

We chose tech— you could call it progress, not everyone likes it— decentralized systems at least as early as the 17th century. Once you do that, something like this will happen. Am I glad we made that choice? Wasn't really a conscious choice, but we let it happen when we did. Absolutely. Is it a more dangerous path than just letting the world fester in what it was like in, say, 1530 forever? I'd say we don't know. It's possible it's more dangerous.

I was thinking we're the world of, like, 2021.

Once you're in 2021, it's gonna happen. Once you're in 1920, it's gonna happen. Unless nuclear war intervenes. I think that's called fighting the hypothetical.

I meant if the physics were such that we just couldn't realistically do reasoning agents?

We will end up living, I think, to 97. Maybe I won't, but most of you will. AI is more likely to extend your life than to end it prematurely. There is a society-wide gamble in any new technology. I think there's a very good chance AI can lower the risk of nuclear war by, say, lowering nuclear errors, by combating potential future plagues. It's up to us. I think the volunteerist attitude, which you expressed eloquently not too long ago, is the correct one. It's like nuclear weapons, but we'd better get it right.

All right, we're really low on time. I know there are two mics left.

Be real fast. Hi, David from the Policy School, a longtime Mercatus follower. My question, you talk a lot about the dangers to privacy from powerful AIs. What about dangers to privacy privacy from a, uh, a dystopian surveillance state that would be required to crack down on open-source large language models? Would we have to outlaw encryption, torrenting, make every device inspectable in order to meaningfully crack down on LLMs?

Got it. And before answering, Sonia, real quick.

Yeah, earlier in the conversation we talked about there being something— whether there was something irreducible about humans, mostly in the context of labor. I'm curious beyond a preference perspective, whether you think there's something qualitatively better about human-to-human relationships than human-to-AI relationships.

I have no— in reverse order, I have no personal relationship.

You just restate the last question because the mic wasn't working.

Oh, oh, sorry.

Tyler can do it so that way it'll be his mic.

Yeah. Is there something better about human-to-human relationships than human-to-AI relationships? I don't have personal relationships with my AIs. To me, they're boring unless they're on the topic of facts. And I think that's the correct way to handle it. But I'm well aware, probably at this point, hundreds of millions of people are doing otherwise. They might need it. I feel I have great access to humans for a variety of reasons, some of which are just geographic, and my colleagues, and that I travel a lot. I don't think it's a great idea to get attached to your AIs. Probably most people who are doing it do it too much. So at the margin, yeah, I think we should have less of it. But some people aren't doing it at all, and it could be a useful therapist for them, especially in poor countries where they don't even always know about AI. And at that margin, maybe we should have some more. Now, the other question You know, I think prior to AI, we already were headed for a dystopian surveillance society. If the question had been, well, putting in CCTV in neighborhoods, it generally was quite popular, most of all in the UK, but also in the US. Now, it wasn't searchable in the way that it's now searchable because of AI. I just have long thought that's the equilibrium. I don't like it personally. I don't see any real opposition to it. So I do think we're headed there. I'm against it. I'm resigned to it. I'm okay mood affiliating with him wanting to do something about it. I'll sign the petition. But I don't really see what that's going to be. So yeah, I think it's coming.

With or without AI. That last pair of questions was a great tour of both Huxley and Orwell in reverse order. And we just owe a huge thanks to Tyler Cowen for joining us today for this broad conversation. Thank you, Jonathan. And a huge thanks to the AIs for listening in. Cheers. May I ask a question?

Yeah, sure.

Why do you think more academic economists have not turned their attention to AI?

It seems like it would be—

It's a threat. They would have to complete— will have— not would— will have to completely retrain, and it will alter status hierarchies. And the ones who are famous now may