AI Consciousness? Exploring the Possibility with Prof. Eric Schwitzgebel

Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week we'll explore their revolutionary ideas, and together we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Thornberg. Hello and welcome back to The Cognitive Revolution. Today my guest is Eric Schwitzgebel, professor of philosophy at the University of California, Riverside. Eric's interests include philosophy of psychology, philosophy of mind, moral psychology, classical Chinese philosophy, epistemology, metaphilosophy, metaphysics, and science fiction. His blog is called The Splintered Mind, and he has written several books, including The Weirdness of the World, which is soon to be published. I first encountered him on a recent 80,000 Hours podcast about consciousness in general, which I really enjoyed, and which inspired me to reach out to propose this conversation about the possibility of consciousness in AI systems in particular. As AI systems become increasingly sophisticated, the question of whether they could become conscious, and what that might mean for how we ought to treat them, is becoming more relevant than ever. Many people have strong intuitions on this subject. Some argue that AIs are not now and in fact never could become conscious, while others contend that they probably already are and therefore deserve at least some rights. In my view, given that we lack a consensus account of the source and nature of our own consciousness, we are really nowhere close to having enough clarity on this subject to warrant confident stances. But considering the fact that human history is full of horrors which were predicated on the intuitive denial of consciousness personhood, or the moral standing of others, including other humans, as well as animals and nature broadly, I think the very least that we can do is approach this topic with extreme humility. With that in mind, we begin this conversation by exploring the most popular theories of consciousness, including idealism, dualism, and materialism. We discuss the strengths and weaknesses of each, and we consider, wherever possible, how these theories might be informed by the nature of today's AI systems, as well as what they would imply for the possibility of consciousness in AI systems if they were in fact true. Along the way, Professor Schwitzgabel explains the core ideas motivating the various schools of thought, highlights important uncertainties, and at times pushes back on my intuitions and interpretations, including when I described my understanding of recent mechanistic interpretability results, such as the Golden Gate Claude experiment, and how I understand AI systems to be conceptually understanding the world. Toward the end, he also offers his recommendations for how AI developers should be thinking about this challenge, suggesting that we should either create systems that we're confident are not conscious or those that we're sure deserve moral consideration, but that we should avoid the uncertain middle ground because that is where we would be most likely to make tragic mistakes. Looking just a bit ahead into the future of AI development, anticipating systems that we've already seen that can carry on fully natural voice conversations, systems that have increasingly integrated long-term memories and capacity for relationships, and systems that people find it very natural to form attachments to and even at times fall in love with, I have no doubt that debates around AI consciousness will become increasingly urgent and focal. I do not expect that we'll resolve the key questions cleanly anytime soon, if ever. And I think of this episode really as an opening exploration, certainly not the last word on this important topic. As always, if you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post about it online, or leave a review on Apple Podcasts or Spotify. We always welcome your feedback, and we are still accepting resumes for aspiring AI engineers and AI advisors on our website, cognitiverevolution.ai. You're also welcome to DM me on your favorite social network. at any time. Now, I hope you enjoy this thought-provoking exploration of the possibility of AI consciousness with Professor Eric Schwitzgabel. Professor Eric Schwitzgabel, professor of philosophy at the University of California, Riverside. Welcome to The Cognitive Revolution.

Happy to be aboard.

I'm excited for this conversation. You recently did an episode on the 80,000 Hours podcast, which I'm a big fan of, and I thought that was an excellent conversation. I definitely would recommend that people check that one out as well. But I came away excited because you talked a lot about different theories of consciousness and what consciousness might mean and what sorts of entities might have it with almost no mention of artificial intelligence. So I said, okay, this is my opportunity to do kind of what I think of as a part 2 on that original episode and see if we can't shed any light into the possibility of AI consciousness. I think this is going to be a very live issue with a ton of confusion. around it over the next couple of years. And anything we can do to even slightly deconfuse ourselves now would be good. Although I suspect confusion will remain.

Even better, appropriately confuse ourselves if we don't feel sufficiently confused.

Yeah, I think—

In the Socratic tradition, right?

Yeah, I think unfortunately clarity will be hard to come by.

Right. I mean, sorry to, you know, cut in right away, but One of my main points about artificial intelligence is that we should appropriately be confused. And the people who feel confident that it's going to play out one way or another probably are not epistemically justified in that confidence.

Yeah, I totally agree. I guess for starters then, I don't know how much this matters, but when I think of consciousness and kind of the thing that I'm trying to get at here, the thing that intuitively feels like it matters to me, Is the idea of that it feels like something to be a certain entity. It feels like something to be me. That's often cited as like the one thing that it's impossible to doubt. And it feels to me like that is maybe the most morally relevant question. I guess another way to put that with a little more color would be, does this thing suffer? Is that, do you feel like that's a good starting place or would, you know, would you complicate even that initial? motivating definition of what consciousness is.

There are a couple hesitations I would put in there in that definition. One is that this phrase, some people find it really intuitive and get the concept right away, but it's— I don't think it's clear to everybody what's intended there. And the other is in the second part of what you were saying, you equated being conscious, i.e., Being such that there's something it's like to be you, with the capacity to suffer. And it seems at least conceivable you could have a conscious entity that's not capable of pleasure or suffering. Maybe it just has all emotionally neutral states, and yet there is something it's like to be it. I mean, some people would deny that, but it— at least at a first pass, it seems like we wouldn't want to exclude that possibility by definition.

Yeah, totally. I think that's a very good point. I am pretty radically uncertain about all of this stuff, and my general intuition is that if there is an AI consciousness, I should at least be very open-minded to the possibility that it's quite weird and very different than my own. And so maybe some of these terms just have no correlate. Is there a better version than What it's like to be something, or do you have a preferred, more intuitive formulation?

My preferred way is to define it by example, to define consciousness by example. I think if people like the what it's like phrase and it works for them, I think usually they'll get the right concept that way. But I think it's even more theoretically neutral to invite you to remember that you have experiences of what it's like. Seeing and touching and feeling pain and feeling emotions. If you close your eyes and picture the route to Grandma's house during rush hour, then you've got a kind of experience of that. If you imagine the tune Happy Birthday in your head, then that's an experience you have. And you don't in the same way experience, say, your knowledge that Obama was president in 2010. Most of the time. Now, maybe you do that I brought it to mind, but 5 minutes ago, that wasn't part of your experience. The myelination of your axons, release of growth hormones, those aren't part of your experience, even though they're internal part of your brain. So if you think about what these things have in common, the pain, the emotional experiences, the sensory experiences, the imagery, inner speech, they all have something, I think, pretty obvious and common that your unrecalled until now knowledge of Obama being president and the myelination of your axons lack. And it's that obvious thing. That's the referent of consciousness. So I think that's the clearest way to define the term.

When you talk about examples, though, I guess one challenge that I have with that is I remember, I have a memory of being conscious of being told as a kid that animals are not conscious. I remember an adult telling me that, and this was a friend of my mom's, and she said, it's everything they do is just instinct. They're not conscious. And I accepted that for a while as a kid. And it was years later where I was like, wait a second. I took that one comment that I don't think was actually justified at all and took it on board as part of my worldview for quite some time, until finally I realised I don't have a good reason to believe that. Now, it seems intuitive to me that at least other, let's say, mammals, and probably beyond that too, like birds, I would say, seem like they're probably conscious, and even fish seem like they're probably conscious. And it might be weird to be a fish, but my guess is that it feels like something to be a fish. How do you deal with this when you use the example paradigm? What would you do if you're faced with somebody who says, I don't believe dogs are conscious? And then obviously we're going to have to map that onto AIs too, right? Where people are out there asserting all the time that AIs are not conscious. And it seems like this sort of example paradigm is like, it's hard to extend beyond one's own internal examples.

Yeah, I don't think that's quite right because I think that, I think we can all take for granted that Other human beings who are awake at least are conscious, although dreams are another example of a positive example you can point to of a conscious experience, even though we sometimes say we're unconscious when we're dreaming. That's not the intended sense. So I think you and I have both this property of being conscious, and then we can ask for non-obvious cases, such as garden snails or whatever, Oh, do they have that kind of obvious property that these positive examples have in common, the negative examples lack? So when you define something by example, you don't need to have figured out every extension of the example. So if I define furniture by example, pointing to positive examples of furniture and negative examples of furniture, and I do it in your house and in my house, we can wonder, and we both get the idea eventually, what furniture is, we can wonder whether Mary's house also has furniture. So that would be like wondering, ah, do snails also have consciousness? Do birds? Do robots? So I don't think the pointing to example has that particular disadvantage. It does have maybe some other theoretical disadvantages, which we could talk about if you want to get into the nitty-gritty of it.

But yeah, I guess just— How does one argue from that position if, let's say, somebody doesn't think that their dog has any consciousness, and so they feel like they don't need to take care of it in the way that society typically expects people to take care of their animals? And now we say, you are doing something wrong here because this is a conscious creature and it can suffer. Leaving aside that maybe other things can't suffer, but let's say I'm going to assert that a dog can suffer and the person continues to deny it. How do you start to bridge that gap if somebody just flatly says, I don't think so?

You need a theory, or at least a theory sketch, or at least a set of principles, right? Famous philosopher René Descartes was rumored to have— probably did not actually— throw a cat out of a window, a second-story window, asserting that non-human animals are mere machines that don't actually have what we would now call conscious experiences. And you can see how this would fit with a certain kind of worldview, right? So if you think that in order to be conscious, you need to have an immaterial soul, a soul, and if you think that all things with souls have afterlives and end up either in heaven or hell or whatever, right? So if you accept a certain kind of— say, religious Christian and other theology, then you might be committed to saying, oh, dogs can't have souls because there isn't really dog heaven and dog hell. And if souls are the locus of consciousness, then dogs have to be mere machines. So I think someone who is attracted to that view, like Descartes was, is going to have some theoretical grounds for thinking, oh, look, even though it might seem intuitive to say that a dog or a cat has experiences, has consciousness in the sense that we just tried to articulate. Maybe they don't really. Maybe they're just really cleverly made machines by God. I think if you're going to defend the consciousness of, say, our closest mammalian relatives, the first thing you need to do is undercut that kind of theory. Once you undercut that kind of theory and you say, hey, look, there's no immaterial soul, we're naturally evolved beings, we're very closely related to dogs and chimpanzees and all that, then it becomes much more natural to think that they also have.

So I want to get into the kind of main theories in just a second. I guess just before we do, just to get a sense of your own intuition, how far do you tend to think? Do you think of it as a Yes/no question, or do you think of it as a gradient, some sort of something that can have measure to it? And take your own garden snail example. What's your best guess as to the consciousness or lack thereof of, say, a garden snail?

I really don't know. I use garden snails as an example. I've written a paper and a book chapter, 2 book chapters actually, on garden snails specifically. So I know a fair bit about their behavior and biology and cognitive powers. And I think that we don't know whether they have experiences like we do. And it's a good thing to recognize that we don't know. Some people will, like maybe your mom, just think it's obvious that they don't. Other people will think that it's obvious that they do. If you ask experts on snail biology, they diverge too. I was talking to one snail expert because I Because I was interviewing snail experts as I was doing the research for these papers, right? One said, hey, look, they're just like basically complicated plants. They don't have any consciousness. I dissect them all the time. And another one that I talked to said, I think that snails have all kinds of rich conscious experiences of a world around them. It's revealed mainly through olfaction and taste. The snail researchers don't know. I don't think the consciousness theorists No, if you have a certain theory of consciousness where you need immaterial souls, it seems a lot to think that snails would have souls. If you have another kind of theory, then it might, it might seem straightforward to you that snails would have consciousness, but we don't know which theory of consciousness is correct. That's part of our problem. And that's what's going to be the problem, I think, when we start thinking about AI cases.

Hey, we'll continue our interview in a moment after a word from our sponsors. Do you have a sense for the degree to which it matters that there is a semantic content to experience? Like when I go out and crawl around in the grass, I both feel it on my skin and I have this sort of conceptual space that I'm operating in where I'm like, I'm crawling around in the grass. That's grass I feel on my skin. I have to assume that the garden snail does not have certainly a language-mediated internal monologue going on, but it does seem that it can respond to stimulus. That's pretty obvious, right? And it can move in the direction of food and do all these things that even the most basic animals can do. I'm sure there's no consensus, but is there any takeaways from that? Or is that seen as very important, that distinction between sort of say something like a purely sensory, but perhaps not understood in terms of higher-order concepts versus the semantic thinking that we do?

That's a very big issue. So for your listeners who think, oh, of course snails are consciousness, how could you think otherwise? There's a whole class of theories that think of consciousness in terms of some sort of relatively sophisticated capacity for self-representation. To be conscious requires that you have some kind of understanding of yourself as a conscious being. You can feel the pull of this if you think about, like, all of your conscious experiences seem to come— I mean, some people say this, I think it's actually contentious, but it's at least a little plausible to say, look, all of your experiences come with some kind of self-awareness. Right. If you're having a conscious experience of that tune Happy Birthday in your head, you're aware of the fact that you're having that experience. You're representing yourself as a certain kind of entity. So it requires, on these kinds of views, a fairly sophisticated set of cognitive powers, ability to represent yourself as having certain mental states. And if you think that, then it now becomes plausible that consciousness would be limited only to humans or only to humans and relatively cognitively sophisticated social mammals and birds, and would not get down to lizards, much less garden snails. So there's a whole cut— there's a whole class of theories that say that. And of course, there's another class who think, look, if you've got, like snails do, the capacity to learn, learn. You've got fairly complex integration of the environment, some sort of representational map of your body that allows you to coordinate your movements. You've got obviously positive and negative affect of some sort in the sense that things can be rewarding or punishing.

Right.

Right? So those views are going to be much more liberal about the kind of entities that are conscious. So already you get this cut along what you were calling semantic. I don't know if semantic is exactly the right word for that, but some sort of complex self-representational capacity required for consciousness versus not. And you get very different results regarding what animals are conscious depending on which theory you enter with.

Yeah, I feel like my intuition on that in part is motivated by experiences of altered consciousness of various kinds. When I think about any of a number of different forms of substance use that people might engage in, or meditation for that matter, or even one of my favorite books of all time is called Dancing in the Streets: A History of Collective Joy. And I don't know if you're familiar with that one, but it's—

No, I don't know it.

It's Barbara Ehrenreich. It's fantastic.

Oh, I've read some of her other stuff. This strikes great.

It's one of her lesser-known books. I loved it though. I read it at the beginning of the pandemic, and which made it hit particularly hard. And she documents all these things through history of sometimes with alcohol or other drugs included, but other times just people engaging in these repetitive, often synchronized dance activities. Could be drumming, could be dance, marching. Parading is derivative of this. A lot of religious processions are a dressed-up version of this. And the key element in a lot of these experiences, which people often cite as the most meaningful, both the most pleasurable and the most meaningful in their lives, right? The people, they enjoy them as they're happening, and they have a very long-lasting afterglow for many people. And it seems to be often connected to a sort of loss of self. It's like being part of something bigger than yourself, unplugging a little bit from from that ongoing narrative or like some fading of your self-representation seems to be connected to like very good things for a lot of people. And you can still remember these experiences, although sometimes they're a little, it can be a little fuzzy in memory.

Right.

To me, that kind of resolves that question. At least I feel satisfied. I'm like, okay, if that's, if I can have that and I can remember that and that still feels important to my self-conceptualizing self later on, then I can't discount the garden snail because it doesn't have some sort of self-centered narrative layered on top of these more raw sensory experiences. I'm guessing that won't be the final word on that topic.

Right. There's a standard answer that advocates of— let's— one of these types of theories would be a higher-order A standard answer that advocates of those theories would be distinguish between second-order and third-order mental states. So if you have a representation of kind of your basic cognitive processes or sensory processes, that's a second-order mental state. The second-order mental state isn't itself conscious normally, right? It's just target state that's conscious, right? Sometimes you also have a third-order mental state, right, which is an awareness of the fact that you're aware Of the lower-order mental state, right? And as soon as you start talking philosophy, that's what's going on, right? So what they're going to say is that in the cases you're describing, there's no third-order awareness. You're not aware of the fact that you're aware, but there still would be a non-conscious higher-order representation of your dance moves in the street. You wouldn't report it. You wouldn't know it as it's going on because that itself is unconscious because there's no still further state that makes it conscious. Does that make sense?

Yeah. So in that theory, what is conscious is inherently the subject of some other modeling process.

Correct. Yeah.

Okay, cool. It's been a great preliminary exploration. I want to get into the 4 major theories of consciousness that you laid out on the 80,000 Hours Podcast. And then also dig into probably the last section. Materialism will probably, I think, get the most time. I don't know if you have a favourite of these. You seem quite neutral from what I've heard from you so far. But I think you did a great job on that episode with 80,000 Hours of just making the general case that all of these things are pretty unsatisfactory and that they all leave us with some either having to bite some bizarre bullets or just remain very dubious about any one of them. Is that a case you can lay out in brief, or do we have to do that by considering each one in turn?

It requires each one in turn. So we could start with the one that's easiest to see how bizarre it is.

Okay. I have my idea for what that is. What's yours?

Which is idealism.

Yeah. Okay. Same.

So the way that I cut up the theoretical space is in terms of idealism, dualism, And materialism, and then a grab bag of alternative views. So the idealist says there are immaterial souls, nothing else is fundamentally real, the material world itself is just a kind of a shared illusion, so to speak, although they wouldn't call it an illusion, right? We all have experiences that are coordinated maybe by God, right? And the only reality that things have is by virtue of being part of someone's experience. That's the idealist view. And if it sounds bizarre, then you— if it doesn't sound bizarre, then you probably don't understand it. Let's put it that way. It's pretty weird. That doesn't mean it's false. Lots of bizarre views are true. That's one of the points I make in my 2024 book, right? Like in physics, there's all kinds of bizarre stuff that's been established as true, but you've got to acknowledge it's not exactly common sense. So likewise, I think this idea that The material world is just a creation of our minds or a shared illusion or something like that, or a creation of God's mind that he then puts in our minds. That's a bizarre view. I think probably people, most readers would accept that, although we could talk about it more if you want. We might want to get on to other stuff more quickly.

Yeah, I guess maybe one challenge to that, that I wonder how the proponent of idealism would respond to, as I was just exploring my own intuitions on this topic, I was like, it would seem very weird if everything is a mental construct that I can only access certain mental constructs through specific artifacts. And there I have in mind, I can't see a cell and all the inner details of what's going on inside my body without a microscope. Suddenly I have a microscope. I can look through it. I can see that. I can't see it otherwise. Telescope, same deal, right? I can see all these stars. I can't imagine those stars without the telescope. I have to actually have the telescope. The internet feels that way. And to tie it into AI, like ChatGPT is that way, right? I can't imagine what ChatGPT is going to tell me about a random topic. I have to actually ask it. So to me, that feels like there's— I guess the whole thing is bizarre. But it would seem extremely bizarre for me to be inventing all of this in my head, but to only be able to do it in this sort of mediated way through these artifacts. What would an idealist say in response to that?

I think the easiest kind of idealism to understand from a standard kind of Western European, North American perspective is Berkeley's— George Berkeley's idealism. And what he says is that These are all ideas that are coordinated by God. When you imagine something on your own, you have this kind of faint image of it, and that's because it's your weak soul imagining this thing, right? When you open your eyes and point a telescope at something and you get this struck by the sensory perception of the rings of Saturn or whatever, that's God implanting the idea into your soul. And of course, God can do whatever he wants, and he coordinates it all so that everyone has experiences that are appropriately related to each other and that When you leave the room and close the door and then open the door again, God ensures that you have the same experiences and the same stuff in the same places. That's— so it's all this kind of dance coordinated by God, but it's all just ideas in God's mind and in your mind and in my mind. And there's no kind of like material world that exists beneath it.

So in general, I'm a believer that our theories should pay some sort of rent in terms of their ability to predict things for us. This is obviously not invented by, but at least for me, really brought home years ago by Eliezer Yudkowsky, who is of course famous for his AI work as well. But even leaving the AI stuff aside, I think he's been a very compelling writer on why you should insist that your theories of the world do something for you. Is there any way to have this theory do something for us? It seems like a just-so story, and I don't— like, what predictions can I make based on it?

Berkeley thinks it has a theoretical advantage in that you can simultaneously hold 2 positions that in the early modern period in Europe were both considered attractive but were very difficult to reconcile. One position is that we are directly aware of the objects in the world around us. There's no mediation between me and this. I'm holding up a jar. There's no mediation between me and the jar. I'm directly aware of it. That's one view. The other view is that all you really are directly aware of is your experiences. Dream doubt suggests that, oh, what you're really directly aware of is your experiences and you have to infer that there are objects that stand behind those experiences that are causing them. So a common view in that period would be to say, okay, yeah, we really aren't directly aware of things in the world. We're directly aware of our experiences, and we have to infer these objects out there. Berkeley can get both. He says, yeah, we're directly aware of objects, and we're directly aware of our experiences, and all you need to do in order to accept those 2 things together is be an idealist. Furthermore, he If you already think there's a God out there, he says, what purpose would the material world serve? Why would— he cares about us and our souls and our ideas and our— all that, and he doesn't really intrinsically care about rocks. Why doesn't he just create the ideas of rocks directly in us? Why bother with creating the material world? He says, on a certain kind of view, the material world serves to no purpose. So these are 2 kind of theoretical considerations that Berkeley advances in favor of idealism. doesn't have an empirical prediction. If idealism is true, then this follows, and here's my prediction that's been borne out, or whatever, which may be what Yudkowsky wants. But it has some, especially if you think about it in a historical context, has some theoretical attraction.

Hey, we'll continue our interview in a moment after a word from our sponsors. I think that's probably enough for me on idealism. What's next for you in the I just got— I'm curious as to whether we have the same order of plausibility on these. So what's next as we climb the ladder of least to most resilient?

For me, substance dualism, right? This is the idea that there's an immaterial— people have immaterial souls, and also that there is a physical world that's different from those souls, right? So like idealism, it agrees that there are immaterial souls, but unlike idealism, it says there's also an intrinsically existing Now, just described that way, it's not yet, I think, bizarre. I think a lot of people, that's their default view, right? Oh yeah, of course there's material stuff out there, and of course we have souls, right? If you think that there's an immaterial part of you that could go on into an afterlife or become a ghost or transition into a new body, reincarnation, then all of that is a little easier to think about if you think there's an immaterial soul that really constitutes you. And that's a pretty common view worldwide and historically, right? So that would be substance dualism. Now, we already touched on one of the main bizarre implications of substance dualism, which is either dogs don't have souls and therefore don't have conscious experiences, or dogs have afterlives. And if dogs have afterlives, then shouldn't cats? And if cats shouldn't Rabbits, and if rabbits shouldn't— lizards, and if lizards—

We draw the line at garden snails.

Where does it stop? The only real natural stopping place for this kind of view is to say humans, and that's it, right? Once you start talking about animals, you got this kind of smooth gradation basically, and it's just— there's not going to be a natural place to stop. It'd be weird to draw a bright line and say, ah yeah, toads of this genus, they've got afterlives. And immaterial souls of toads, of this cheetah, although they're very physiologically similar, right? They're just material stuff, right? That'd be weird anywhere you draw the line other than basically between humans and everything else, right? But that's pretty bizarre, right? Especially in the art currently, I think there's been more and more of a tendency to regard non-human animals as capable of conscious experience and to see ourselves as— Intimately relied to them, or intimately related to them. So you kind of have to bite some bullets. You end up with a similar kind of slippery slope problem if you think about in terms of evolutionary history. Right, okay, when did the soul pop in? Was it Homo erectus? So Homo habilis didn't have souls, but Homo erectus does. You get the same kind of problem, right? So if you go that way, the most natural thing to do is to take a certain kind of a certain kind of package where there's an immaterial soul that comes in exactly at conception. There is no evolutionary history. We are created beings in a young Earth, and there's a bright line basically between all humans and everything else. And although there are definitely people who accept that view, it's, I'd say, a minority view in the— in this culture. It has some tensions with scientific orthodoxy for sure, and Right, as we're talking about, has these unintuitive implications for animal consciousness. So that's, again, like idealism, I'm not saying that we know that it's false for sure, substance dualism, but you do have to bite some bullets, as philosophers say, or accept some pretty unintuitive consequences, accept that there are some bizarrenesses in the view.

Yeah, this one, it does go to show how important your priors are for how you'll engage with these sorts of things, because I certainly have a hard time understanding the world without a theory of natural selection or evolution. And so that does become a real problem for me. Obviously, if you take a young Earth creationist view, then you have a very different sense of that maybe did never happen. And so you don't have that problem. Is there a version of this also that I'm trying to— one thing that would be appealing of the dualism view in the case of AI, or at least maybe, is it might give you an easy answer on AI. Because you could put it on the other side of a bright line and walk away from the problem. So I don't want to—

That seems like the natural thing to do. Yeah. If you're going to be a—

So I don't want to dismiss the theory too quick because it does have maybe some nice clean resolution on the AI question. Is there another version of it though that is like less religious and thinking back to my intro to philosophy class, epiphenomenal understanding, or you could maybe put this Over toward panpsychism, which could be a different thing as well. But you also hear these stories of, or accounts, let's say, of it's, you're tuning into something that is bigger than you, right? There's some universal consciousness or some background field of consciousness that maybe the brain just so happens to dial into. I guess I've articulated 2 different things there. One is like, these things are epiphenomenal. They're thrown off from our brains, but they're separate. And I don't know, does dualism always imply that they can't interact or don't interact?

No, no, no. Descartes, for example, was an interactionist dualist. He thought you've got an immaterial soul, you've got a material body, and they're constantly interacting.

Okay.

Epiphenomenalists are people who will say— there's a variety of forms of it. Phenomenalists, of whom this may be the most relevant stripe here, will say you have constant conscious experiences, but they don't have any effect on anything. The idea of epiphenomenalism is a jargon term for something that has no causal consequences. So you have experiences, but they have no causal consequences. You experience pain, but your pain does not cause you to say, ow. You're caused to say, ow, by a brain state, and the pain is ineffectual, the actual experience. Which again, I think that's a kind of bizarre thing to say. And you don't even get out of the The problem about where do you draw the line on what entities have souls or not, you still have that problem. Now, there's a class of views— you briefly alluded to panpsychism and also property dualism— that I put not in the class of substance dualism, but they're in my grab bag of alternative or compromise views to dualism, idealism, and materialism. They are, I think, also all dubious and unintuitive and bizarre, but you have to talk through them one at a time in order to see that. And we could do that briefly if you like. But, but in terms of, in terms of mainstream dualism, let's bracket those kinds of moves and just think about, okay, there are immaterial souls, there are material objects, and then you've got this problem immediately that you face of which entities have immaterial souls. And whatever you say about that, you're going to have some unintuitive things that you're going to end up being committed to on the dualist view.

Yeah. I also think, again, I don't know why I come back to mind-altering substances so much, but there's another very weird phenomenon there where it almost seems like we should be able to say, and again, I wonder what the committed dualist would say in response, but if I drink this cup of beer and I feel different, I guess you could just say they interact and then you could rescue that with the interaction view. But I guess the interaction view to me feels like a cheat in dualism where I'm like, if they're interacting, then in what way are they different? The sort of dualism, it seems to suggest some sort of very separateness. And maybe that doesn't totally preclude interaction, but the fact that I could line up 100 different substance-altering chemicals And the ones that have similar structures will often have similar effects. This would be a very weird thing to reconcile with intuitive, true separateness. And then if you're back to, they can interact, then I'm just like, okay, then what exactly are you claiming when you're saying they're so different in the first place?

Yeah, I think that is one of the problems. Another one of the big problems for the dualist views is to think about The causal interaction. And historically, if you look back at the dualists of early modern Western Europe, they really struggled with this issue of whether the mind and material interact and how that would work. And you get a whole other nest of bizarre commitments that come from trying to resolve that question one way or another. Didn't want to try to talk your listeners through that because by I didn't want to try their patience too much, but we could if you want. But yes, just first pass, I just want to agree the interaction problem, the mind-material interaction problem is a huge problem for substance dualism and was recognized as such by people in the—

Yeah, I think we could probably move on. I mean, this one feels odd. The real reason to put any time into it from the perspective of an AI obsessive personality like myself is that if you could talk yourself into it, you might get some easy outs on downstream questions.

Maybe, although Alan Turing has a kind of nice quip about this in his article on Computing Machinery Intelligence. I'm not going to remember the exact words, I'm just giving you the idea right now. So what if you are a substance dualist, right? If you think that God instills souls in humans, why not think that God would instill a soul in a robot too?

Yeah, I guess there's no easy outs in any. So then I can— then we can definitely move past dualism, because if I'm not even going to get anything for free from it, I'm done entertaining it. On the flip side of that, panpsychism seems to have kind of a similar thing where if everything is conscious, then AIs are conscious. The hard work largely remains to be done there. I'm actually fairly sympathetic to this view in the sense that many people seem to think it's just like an outlandish idea from the beginning. And the longer we go in a conversation like this, the more it drives home the point to me that I don't really know why I'm conscious. And if I don't have a good account of that, like, I can't really rule out anything else really. And if I'm going to get down to garden snails, then what if I get to a plant? Is it so crazy? So I'm much more open-minded to that than I think most people are. But then I still wonder, okay, am I getting anything for it? If we were to posit, for example, or work from the assumption that something like panpsychism is true, then don't I still have all the same open questions that I have about AIs? Unfortunately, what is it like to be them? And should it— should I care? Does it matter?

You don't even necessarily know that they're conscious, right? So I want to distinguish between 2 different types of panpsychism. There's the type of panpsychism that, say, Philip Goff endorses, where fundamental particles are conscious. But rocks are not. You can't just take any aggregate of fundamental particles and say, oh, that aggregate is conscious. There are some privileged aggregates, like human beings and dogs, and some aggregates that are just aggregates, like a rock or like the aggregate of your left shoe and the rings of Saturn. Those aren't conscious beings, according to most dualists, most panpsychists like Goff. So then you still have the problem of figuring out which side of the line computers on or AI are on, right? Are they among— are they on the side with humans and fundamental particles, or are they more on— more just plain aggregates like rocks without distinctive set of experiences? Now, there is another type of panpsychist. Luke Roloff is probably the best recent example of this, who just goes ahead and says every aggregate is conscious. So rocks are conscious, and the aggregate of your left shoe and the rings of Saturn is also conscious. But that's a really very hard view to swallow unless you really feel forced there by philosophical arguments, because it's pretty unintuitive to think that every aggregate, every random aggregate of stuff is conscious. So unless you go, as I think of it, full roll-offs, just every combination of everything has its own distinctive stream of experience, Right, then you still have to draw the line somewhere, right? And you've got this line-drawing problem again. I don't think it's as, as much of a problem for the panpsychist and for the materialist as is for the substance dualist, because not as much hangs on it as if you think there's an immaterial soul there. And I think for at least certain kinds of materialists, you can say, oh, maybe there are fuzzy cases or there are things that are conscious. Maybe garden snails are kind of conscious, right? It doesn't seem to make any sense to say that garden snails kind of have souls. A soul is something you've got or you don't. So you get this, but whatever view you have, there are line drawing problems. And the line drawing problems around AI are just a subspecies of this bigger set.

Yeah, that's interesting. I don't know if this is more something we should raise in this section or in the next section on materialism, but there are these weird results from the study of human cognition that also, I don't recall if you covered this in the 80,000 Hours podcast or not, but the, these sort of split brain patient type things where you're looking at one presidential candidate or the other and you're responding in these weird ways that you're justifying after the fact. And it seems clear that there was like at least some cognition and maybe some consciousness happening that the other side of the brain that's more responsible for responding doesn't have access to. I've been convinced, I don't know if you would agree with this, but I've been pretty convinced that there are probably separate conscious streams, even just within my own brain, that don't always know about each other. I don't know how much that goes on, but it seems like it at least can go on, and that's just plain weird. I don't know what bucket do we even put that into in terms of a consideration? I don't know, but that is definitely, at a minimum, it's another thing that can confuse us more.

Yes. So I think it is at least theoretically possible that you could have, say, 2 streams of conscious experience in the same body in the case of a patient what we call split brains where the corpus callosum is severed or mostly severed. That's how Elizabeth Schechter, for example, interprets it. And maybe even in the normal case. One interesting example—

Fashionable on TikTok that people now have multiples.

One interesting example is your enteric nervous system. So you've got about a billion neurons that line your gut and they're capable of operating. They govern digestion and emesis. release and the motion of your bowels and that kind of stuff. And they can operate reasonably well, even if they're severed from the central nervous system. They've got as many, there's about as many neurons as there are in a small mammal. So you might think, hey, look, maybe there's another conscious being down there.

Yeah. We even see echoes of this in our language, right? The idea, of course, that you like know something in your gut is a, you know, I don't want to be too literal about that, but it is at least Suggestive. And I do think that corresponds to something that people feel in meaningful ways. I also always think about the observation that the ancient Greeks, for example, thought that the core of their self was in the heart, if I recall correctly. We tend to locate it in the head, but they intuitively felt like the most them part It was disputed.

Aristotle said that, but Plato didn't.

I see.

But in ancient China, it was much more clearly orthodox. So the point still stands even though I'm not sure ancient Greece is the best example of that. But yeah, so in ancient China, it was generally thought that the heart was the organ of cognition and emotion. And in fact, if you look at Chinese characters, A lot of them use the character for heart. Most of the characters that trace back to ancient China that have to do with cognition and emotion have a little heart radical in them, so they actually refer to the heart directly in the character.

Fascinating. Okay, so I think we're finally getting around to materialism.

Yeah.

My basic argument for materialism, and I wonder if you would sign on to this or your own version, but it basically seems to me that none of these other theories are very compelling. Materialism is still going to leave us a lot of questions unanswered, but it at least allows us to sort of put this into the same bin of mysteries where many other mysteries go, where we're like, we don't have a good account of it, but one day we might get there. And potentially we can figure out what's happening. So this is to say there's probably nothing magical or godly or outside what can, in the fullness of time, be explained. We just don't know how to explain it yet, and that we can propose candidate theories within that general worldview. Is that your basic worldview too?

If I had to choose among the various I would probably choose materialism, but not with a high credence. Maybe with about a 50% credence. Give me 50% credence in materialism and 50% in all the others combined. I think we haven't talked about 2 of my favorite alternatives to materialism, which belong to this grab bag of possibilities, and both of them are rather more technical and difficult to understand, but I'll— maybe it would be helpful to just gesture it. One of them is property dualism. So the most famous advocate of this is David Chalmers. He says that there are not 2 types of substances, a mental substance and a material substance, but rather there's just one kind of substance, but it has some mental properties and some material properties. At least some, some of these substances do, like humans. And but the mental properties and the material properties are not identical to each So you— it is a pretty naturalistic view in the sense that you don't need immaterial souls. You can be all in with evolution and an old Earth and something like— and all that kind of stuff. It's just the mental properties are not reducible to physical properties. So that's property dualism. And I think that's— it's not my favorite position, but I think it's not totally implausible. And it has— To some extent, maybe not quite as much some of the naturalistic appeal that you just discussed. And another alternative position that I like is transcendental idealism. So this is different from the kind of idealism I started with, so the name is a little bit unfortunate, but it goes back to Immanuel Kant. Immanuel Kant says— I'm going to give you a simplified version of it. Kant himself is really hard to understand, but I think the simplified version also counts as transcendental You don't have to be a Kantian specifically in order to be a transcendental idealist. If you think basically that 2 things, then you count, in my view, as a transcendental idealist. One is that we cannot know the fundamental nature of things. What things are fundamentally composed of, that is beyond our capacity to know. That's thesis one. Thesis 2 is that space as we experience it is in fact a construction of our own minds. You think those 2 things, then you don't think that space is a fundamental feature of things as they are in themselves, to use Kant's phrase. Instead, you think it's a construction of our minds, right? And that puts you away from any kind of materialist view, I think, because materialists are going to say, look, Space is a fundamental nature in the fundamental nature of things. Or maybe if you have a certain kind of unorthodox physics, you might say, ah, space falls out of some more fundamental stuff. But that more fundamental stuff isn't a construction of our minds, right? It's information or strings or whatever at the fundamental level. One way of warming up to the idea of Kantian transcendental idealism is to think about the possibility that we live inside a simulation. If we live in a simulation, I assume your viewers know this thought experiment, or at least your listeners, right? Then the world as we experience it is not how things fundamentally are, and maybe we can't break out of the simulation to discover how things fundamentally are. And space might just be how we happen to interact with these fundamental things rather than being really the fundamental stuff. So again, that's in my view a little bit less simple and elegant view in a way than materialism, but I think it has its attractions also. So, so when I say I got a 50% credence in materialism and 50% in the rest, I want to make sure that I'm including property dualism and transcendental idealism in this basket of the rest.

Yeah, I guess I'm not even sure where the distinction necessarily comes in there, or at least it feels like I could fold those into my kind of high-level notion of this is something that could be explained, but we haven't figured out how to explain it yet. I guess when I think about space, okay, there's spacetime. I think the satellites work, right? And we can get mundane day-to-day utility out of general relativity. And so there's clearly something very weird happening with spacetime relative to our baseline intuitions. And so if you were to say, okay, can we, you know, really perceive the true nature of things? It seems like we can't, even just on that spacetime level. We walk around with the knowledge that our intuitive physics is not the whole of physics. So yeah, I can't access directly root nature, Whatever that most ground level nature is. But it still feels like I can fold that into a materialism story where I would say, and here's where I'm going to do a parallel between humans and AIs. My general sense of life and humans in particular is that we have evolved through time and that we've been selected for our ability to reproduce. And it seems like whatever consciousness is doing, it's probably got to be playing some role in our ability to successfully reproduce. It would be extremely weird to have something that seems so fundamental, if you buy the story that we're the products of a natural selection process, It would seem extremely weird to have something that seems so fundamental that has no connection to the thing that we have been optimised for. Similarly, on the AI side, with language models these days, there's a lot of talk of emergence and a lot of debate as to exactly what that means. But it is, I think, essentially indisputable at this point that even though the language models are trained purely to predict the next token as accurately as possible. And the analogy there being obviously to our reproduction as successfully as possible, that they are coming to represent higher order concepts internally as a means to successfully making those next token predictions. So I see these are on quite distinct tracks. The optimisation goal is different, obviously. substance of what we are, the actual atoms that we're made of, very different. It seems like there is something that I want to see as analogous, where it seems like our consciousness must be doing something to advance the goal that we're all historically captive to. And these concepts that are emerging in the AIs seem to be a sort of similar phenomenon. That to me says that consciousness has to be useful. We would be less able to navigate the world if we didn't have it. I feel like I can get to more than half. You're still only at half though.

Yeah, I have multiple thoughts about that. So one is, it's certainly not consensus that large language models are representing the world. They're doing things that are natural for us to interpret as representing the world. But there's definitely a perspective out there that I take seriously, That says they're just mimicking or parroting. They're very good mimics. I would hesitate a little bit there. Another source of hesitation is just to note the panpsychist is going to say, ah, consciousness is fundamental. It doesn't need to serve a function if it's fundamental. It just always already exists. Gravity doesn't serve a function. It just already Always exists. And then I guess the third reaction is, you know, framed as an article— argument against transcendental idealism. I guess I have 2 thoughts. One is that all of that is compatible with transcendental idealism. The transcendental idealist does not have to deny that consciousness has a function, that there's an evolutionary history. All they have to say is that fundamental nature of things is not spatial, and spatiality comes from us instead, and it's fundamentally unknowable, right? So there are going to be patterns somehow in the fundamental reality, and our existence— continued existence is going to rely on our behaving in ways that kind of fit well with those patterns, so to speak. If you think of As a thought experiment, just to show you how this is at least conceptually possible, imagine that we're in a simulation that has been running since the beginning of the Big Bang. We still have all the evolutionary processes going on. Computers as material objects, but there's nothing in the theory of computation that requires that computers be material objects. All that's required is that they've got transition functions of some sort. of the sort. So you could implement a computer in an immaterial soul. So imagine, if you will, and maybe you won't, an immaterial soul computer, a giant computer that's actually composed of an immaterial soul that's been running since the Big Bang, and that constitutes it. I'm not saying this is a likely view, but this just, I think, illustrates that it is not logically impossible To think that fundamental reality might be not material and nonetheless everything that you said about evolution and function is true.

That does start to feel pretty weird. I'm not sure how to locate the however many huge number of generations going back to early life within a giant immaterial soul computer. I've wrapped one mystery in a much bigger, weirder Mystery at that point.

That's the fun of plotting.

I do want to go toward a little bit more kind of practical, because I do think in society we are going to have a very live debate in the near future with pretty practical consequences around what is permissible to do with these AI systems and what status should they have, so on and so forth. One comment on the sort of conceptual representations within large language models. I think this is moving really quickly, and by no means are all the questions answered, but I do think there's been a tonne of progress in the nascent field of mechanistic interpretability that does give us pretty good grounding. And for me, I always think about usable in engineering, right? When I really try to ground this stuff out and I'm like, okay, this has been a fun thought experiment, but where can I really ground this out? I'm like, the Satellites work, GPS works. There's something there that's much harder to argue with than kind of other sorts of things, because systems depend on this stuff. Unless the whole thing is a hallucination bracketing, like, super weird. The fact that the satellites work, that gives me confidence. In the context of the language models, a really interesting recent experiment you may have seen was called Golden Gate Claude. Have you seen that?

No, I haven't seen that one.

Okay, it's a really interesting phenomenon, and I would definitely encourage you to just look into more mechanistic interpretability because it's, for me, extremely rich, including philosophically. Basically what they do with Golden Gate Claude is they first of all try to tease apart what concepts are being represented internal to a language model at any given time, time being like the information that it's processing at any given step in its processing. And a fundamental challenge with that is that the language models are only so big. They have these layers that have the attention mechanism and the MLP mechanism, and then between them there's this bottleneck where there's a relatively small number of numbers that include all the information that has resulted from all that processing up until that step. Those are often called the activations. Those nodes are often called the neurons and the activations of the neurons, right? So you may have, depending on the size of the model, you may have 4,000 of those, 8,000 of those, 16,000 of those. It's usually somewhere in that range. You've basically got an array of some number of thousand numbers that represent everything that has happened up until that step in the computation of the language model. Now, there's obviously ton more concepts in the world beyond 8,000, right? So you can't just say, I've got 8,000 concepts and I'll light each one up in proportion to how relevant it is right now. You run out of space real quick. So what the models instead, by hypothesis, and then we'll get to the demonstration, but the hypothesis was there's got to be some overly dense packing. If we wanted to have 8,000 concepts all orthogonal to each other, we could only encode 8,000 concepts. But if we allow for some interference between concepts, now we have a vast space because we can encode some as 1 and 2 together and 1 and 3 together and 1, 2, and 3 together. And as long as the relevant concepts are infrequent enough that they don't collide too often, then you can have a functional system. Although you can also have really weird stuff where if you do bring 2 concepts that usually don't appear together at the same time, then you can get these weird interferences. So that was all hypothesis theoretical work until relatively recently. They trained an auxiliary model. This work was done at Anthropic. It's becoming quite popular, and there's open source versions, and it's blowing up in the field right now. But Anthropic has been the leader. They trained what's called a sparse autoencoder, which is something that is meant to unpack pack the concepts from their dense state and the 8,000 numbers or whatever into a sparse representation that may be millions wide. And they do this by projecting out to this super wide space and then projecting back. So the 2 pressures that they put on the sparse autoencoder is that one, it has to be sparse. They only allow it to light up, or they put a penalty on it so it will only light up a few of those many millions of neurons at the same time. And then it also has to be able to reconstruct back to the original form so that it can be like at least mostly functional. There's usually some degradation of function anyway. Okay. After all of that, you've got all these different millions of sparse concepts. Then you can look through and see what concepts cause an individual position in this millions-wide thing to light up. And if you can see something that appears to be like a coherent pattern among inputs that cause that thing to light up, then you can say, okay, that seems to be the thing that corresponds to this concept in the world. And so the demo that they did was Golden Gate Bridge. They found a concept out of the many millions that seemed to be the Golden Gate Bridge concept. And then the great advantage of language models, certainly relative to human brains, is you can monkey around with them in all kinds of weird ways. freely. And so they created a version of Claude, the language model that they serve to the public, where synthetically they take that one dimension of Golden Gate and turn it up artificially. And then no matter what you talk to it about, it seemed to want to talk to you back about the Golden Gate Bridge. So if you asked it for an example of a math problem, it would naturally use the Golden Gate Bridge as the subject of that sample math problem. So you could still have reasonably coherent conversations with it, but it's always circling back to the Golden Gate Bridge because this one dimension has just been jacked up.

Yeah.

And so all of that is to say that it now is getting to the point where it is usable in engineering to create an end user experience, that there do seem to be these concepts represented that are intuitively recognisable to us and that we have enough control over that we can actually say, okay, I'm now going to engineer this back into a system and create something that is quite distinctive, but in a predictable way based on our ability to decode those concepts. Why does that matter? Because that would be maybe the closest thing to a sort of sense of the thing is thinking about. I'm thinking higher order thoughts as I go through this monologue, and then I'm spitting out one word at a time. And my words are forming at the end of that process. And the argument, I think pretty compellingly at this point, is that the language models are doing something similar where there are at any given time potentially quite a lot of different higher order concepts that are interacting. And then it's usually in the last couple of layers that seems to be cached out into a concrete prediction that can actually be emitted and interpreted. So certainly one could make philosophical objections to that still, but I do find myself pretty compelled by the usability in engineering that does achieve. And so then that makes me think, okay, under what materialist theory of consciousness would I still want to separate the AIs from humans if we are analogous on that level? We have, again, very different atoms, very different training objectives. But if there's that level of similarity where there's these higher order concepts interacting in some way that I can't fully access, but yet I keep spitting out irrelevant next tokens, and we can see something similar happening there, then I feel like, geez, I got to take pretty seriously the possibility that this thing might indeed be conscious. So that's one of my longer monologues in show history. But what's your reaction to all that?

I do think we should take it seriously. I think it's an open question to what extent these things would count as concepts. So if you think about the basic network, what you've got is you've got a set of relations in a kind of abstract space. And yeah, you can unpack those relations into a million-node-wide sparse vector if you want. But essentially you're talking about, you know, oh look, this word or token tends to co-occur with this token. It's a really complex version of that, right? So it's at root, it's a model or network of associations among tokens. But what gives those tokens or words their significance or meaning is anything that has a structure that's similar to the human structure. In that respect, already by virtue of that, have concepts need to be tied to the world in a certain kind of way, or tied to experience in a certain kind of way. Right, so you've got a complex structure of coactivation relations. What if you took that same complex structure and instead of attaching it to human text inputs and human text outputs, you attached it to pixels? Then you get patterns of Pixels as the output. That's very different from a sentence, but you could have exactly the same correlation structures between one of these models with all pixel inputs and all pixel outputs as you could with all text inputs and all text outputs.

Yeah. Interestingly, that is, by the way, about to launch widely from OpenAI, their latest model, GPT-4o, which they demonstrated back in May and are just now putting in front of users outside the company for the first time, it puts text, audio, and imagery all through the same core weights. And they obviously don't publish all the details of this, but they've led us to believe, and it certainly seems plausible to believe based on what it can do, Right. That it is processing those different kinds of input and output modalities through the same conceptual space. In other words, the word snow and a picture of snow would give rise to the same concept activations in the middle, even though you're in different modalities on the input.

You could design it like that, but what I'm saying is that you don't have to design it like that, and you would still have the same structure, right? So let's say that we map the word the to light pixel 0, 0, and we map the word banana to light pixel 0, 1, and we map the word pig to light pixel 0, 2. You could have that mapping and still have the same abstract structure In between, and then when you give it some snowy input, it will give some snowy output. You can take that same structure, and if you hook it to the input-output relations very differently, then suddenly it looks just like a mess, complex mess.

So is the intuition that you're driving at there one of robustness? I mean, it's definitely the case that the AIs are way less robust to strange inputs. It feels like where I was taking you to be going is that I have some sort of additional filter of coherence where if I get just like super strange mess in, I just say stop or turn away from it and they don't.

Right.

So you could maybe think of that as— That's not at all what I'm saying. Okay. So tell me.

That's not at all what I'm saying.

Try me again.

If you take the same mathematical structure that you hook onto this input text token, that output text token, and you hook it onto something very different, then you'll get an uninterpretable Mess. That's the core idea. It's not that they're— they lack in robustness, which is another interesting type of critique. This is a more fundamental critique, and I'm not sure that I totally buy it, but I just want to put this in there as a respectable source of skepticism, right? Which is— I don't want to turn this into Searle's Chinese Room, but it's more in that direction. The idea is you've got a really complex structure that you feed it one thing and it does something complex and it outputs another thing. But what that structure means, what it counts as its concepts, seems on this certain— on this way of thinking about things is going to be somehow intrinsically tied to How it's connected to the world. Here's a really stupid, simple example. You can take the same thing that measures pressure, and you can call it a barometer, or you can call it an altimeter. It measures how much atmospheric pressure is there going on. And if you assume that you're at constant elevation, then what it's measuring is the humidity level. And if you assume there's constant humidity, then what it's measuring is the elevation. It's the same simple physical device, but what it means depends on the context in which it's being used. So you take an internal structure and you just say, oh, because it's got this internal set of internal relationships, that by itself is sufficient for it to have these concepts. It doesn't follow straight away, and there's at least some philosophical work to do. To get from, ah, I've got this kind of pattern of responsiveness, therefore I have these concepts.

So how do we bring that back to the question of possibility of AI consciousness? I am, I'm willing to take evidence that is incomplete if it can guide my downstream decision-making. I find these concept-level durability experiments to be strongly suggestive. I would agree that I can't make a fully ironclad case that it's beyond philosophical doubt, but it does nudge me in the direction of thinking I always say please and thank you to my AIs, just on the general precautionary principle that it may feel like something to be them. How do I tie what you're saying back to practical intuition? I guess we could maybe also even just zoom out for a second and say, what odds do you think we have of getting any clarity or consensus on this? It seems overwhelmingly likely that we're going to have things that, especially with this voice mode of GPT-4o coming soon, you're going to have real-time interruption back and forth, very natural exchange with AI systems. And we're almost going to have to remind ourselves that they're not conscious. If indeed they're not, or people like that, probably it seems like the folk idea will be that they are, or would naturally be that they are, unless there's an elite class telling the average user that they're not. Is there any way that we can get out of that scenario?

There are also, if you look at how some companies are designing these things, they're also giving them reinforcement training. To deny that. I mean, not all of these systems are like that, but if you ask GPT-3.5 Plus whether it's conscious, it will deny it. But that's not true of Replika.

Or even Claude.

Yeah. So it really depends on how the machine is designed. So there will be some users. I think this is, I think you're accurately Where we're headed. I wanted to cast some doubt on the idea that we can move pretty quickly from complex relationships that we— that seem to be semantic to real concepts. But at some point it might start to seem very plausible, especially if you have something that's hooked up to act in the world in a certain way, and not just to kind of something in a box. So what we're going to have is, I think, We're going to have a situation where there are some people who are like, yeah, my AI is my really conscious romantic partner who deserves rights just like humans. And some AI systems might be designed, in fact, to say, yes, that's the correct way to see me and I do deserve rights. And other people will say, No way. This is just a really complex toaster. And some AI systems will say, will be designed to say, I'm just a really complex toaster. And I don't think we're going to have the theory that tells us which of these views is correct.

I would assume that you don't put too much weight in its self-reported answer, given the In fact, that is pretty directly attributable to the intentional training that it's received.

Yes, it certainly could be the case that you would have a really conscious entity. Let's imagine we've got some conscious ones, right? It certainly could be the case that we've got conscious ones that are trained to deny it.

It'd be very weird if the RLHF training split on you must say you're not conscious versus you can speculate about your own consciousness, which seems to be the Claude state. It would be the difference between the actual—

That would be strange. That would be strange if that were the case. That seems like too superficial a difference. Yeah.

Do you— I don't know how much value you get from these sub-theories of materialism. I ran through, actually with the help of chatbots in preparing for this, integrated information theory, global workspace theory, attention schema theory, and I came away from all of them feeling the same thing, which is like, They seem to beg the question, like information integrated information theory, to quote the chatbot, says consciousness arises from the integration of information across different parts of a system. And then I'm like, okay, that may be true. That may not be true. It doesn't tell me whether it is or isn't. If that is true, then I look at the attention mechanism within a large language model and I say, there's in some sense, like very integrated information here. I think we could satisfy that definition, but I don't really see a lot of reason to think that definition is really at the heart of the matter. And I feel basically the same way for the others that I mentioned, the global workspace and the attention schema.

So just on integrated information theory first, before the general point, it really depends for— I mean, the name is integrated information theory, but it's really much more specific theory. So It's not just any old information integration. It has to meet various criteria. And it's not clear that you get what's needed for a non-trivial amount of consciousness in the implementations of large language models. Or if you do, it might depend on some details of the implementation that are not really visible or relevant to humans' interactions with behavior, right? So one of the, one of the results of the IIT model is that you can get systems that are computationally very similar from the user interface perspective, and yet differ radically in how much consciousness this theory of consciousness attributes to them. That's just a little nuance about Integrated Information Theory. But the broad picture, I think what you're saying is totally correct, right? These views have some attractiveness to them. They do make, to some extent, some empirical predictions that can be tested, and it's a little noisy and a little pretty big mess, but mostly they beg the question, are you in a circle? And there's not really compelling evidence for any one of these theories over a variety of others, even if you keep within the broadly materialist functionalist worldview.

Yeah. One that I do maybe give a little bit more weight to recently, in particular, because again, I'm always looking for kind of some sort of engineering cash out if I can find one, is the higher order thought A theory which, again, quoting the chatbot, says consciousness arises when a system has higher order thoughts or representations about its own mental states. We talked about that a little bit earlier. I just saw a paper in the last 2 weeks that is called Unexpected Benefits of Self-Modelling in Neural Systems. This is still pretty small scale work, but they did what I thought was a fascinating experiment where They took a system that has a normal job. The classic job, simplest toy problem in machine learning is identify this handwritten digit, right? It's a 0 through a 9. Your job as the neural network is to say which one it is. They added to that an additional job, which is that it has to predict at the end its own internal state. that was worked through on the way from the input to the output. So now it has 2 outputs. It has to label the number, and it has to output its own state from a couple of layers back in the neural network.

Yeah.

And the way that they set this up was that it could work toward that goal in 2 ways, effectively. One is it could learn— there's additional weights at the end that that kind of project from the last layer into this prediction of its own internal state. But then also the internal state could get easier to predict. And the fascinating discovery of this admittedly very small-scale work so far was that it was able to predict its internal state and also to continue to do the job of identifying the number. But part of the way that was achieved is that the internal states seem to get simpler. And they measure that in like mathematical ways, which I honestly don't fully understand. You can see on the graph that there's the lower complexity value for these ones that have this self-modeling component. And so they theorize from that, maybe this is a function of self-modeling, is that it can create a feedback feedback loop where our internal structures that are actually responsible for doing the object level thing have this sort of pressure to be a little more elegant, a little bit more sparse, maybe a little bit more robust, because they are also subject to this self-modelling mechanism. And that at least starts to tell the beginning of a story of utility from consciousness that I honestly hadn't heard before. You may have heard something, I'm sure you have heard something very similar, maybe from a different origin, but do you find that sort of analysis compelling?

I think that's interesting. It's interesting from an engineering perspective, but it also relates to consciousness and it relates to the point we were discussing earlier in a certain kind of way that I want to highlight. Right. So my understanding of how current large language models work is that they don't have that kind of self-representation of what's going on in their space. I mean, they feed all the tokens into the next token output, was launched through the context window or whatever, but you don't have a kind of recursive self-monitoring layer that's then feeding back into the system. But maybe in the future— Maybe that's an architecturally interesting thing, and it sounds a little bit like a higher-order self-representation, which relates to certain kinds of views of consciousness. So you might think, hey, look, so here's a possibility view, right? All the stuff you were talking about when we were talking about the really complex-seeming representations, that's possible without that higher-order loop. But maybe it's that higher-order loop That's where the magic of consciousness is happening. So could you be getting really complex behavior that's not conscious until it's got this something extra in it? I'm not saying that's right. I'm not saying that's wrong. But a quick move from, oh, it's got complex, seemingly semantic relationships to, oh, therefore it's conscious like us. That's one kind of way of seeing how, even from within your own perspective, you might say, look, yeah, that's a little quick. Maybe there's something else that's necessary. Or if you think about human beings, we can do pretty complex things without consciousness. I mean, when you choose—

Even just moving around. Yeah. I mean, it's amazing how much happens without explicit—

Or when you do it, when you make a pun or a quip really quick, There was some non-conscious processing going on in you that gave you that exact funny thing to say. That wasn't your conscious part that did that. That was the non-conscious part of you that came up with that, that came up with that joke. What it takes for there to be complex patterns of behavior and what it takes for consciousness seems like they got to be related in some ways, but it might not be a simple one-to-one relationship. that you might think there would be at first glance.

Yeah. I don't seek even— I don't expect to get, and I don't really even feel like I'm looking for a confident assertion of consciousness. I think I take a pretty precautionary approach. And I wonder what you would, as we start to maybe transition from a lot of speculation into— I don't know if you'd be so bold as to give some advice to the community. But my sense is, if there's even reasonably good reasons to think that there's enough similarity here, or I see enough analogy that I'm at least inspired to imagine that these things might be conscious, then I start to think from there, like, the long history of denying the personhood of other actual people, is also starts to come into play. And I'm like, I just feel like we should probably be more cautious than not. And I don't know if this is too far for you to sign onto, but what would your gut instinct be? Or how would you advise developers of AIs or even just users of AIs to model this for themselves, given that we're not going to have, or at least it doesn't seem like we're going have rock-solid answer, but yet we still have to move through the world.

There are 2 different kinds of precaution, and I definitely advocate one kind. Let's say that you have a system and you don't know whether it's conscious enough to really merit serious consideration. One kind of precaution would be to say, let's not design systems like that because they put us in the dilemma of having systems that we don't know what the ethically correct way to treat them is. I advocate that. I call that the design policy of the excluded middle. So I've advocated this in a couple articles with Mark Garson. So what we should do is either create systems that we know are not meaningfully conscious, sentient rights deserving, and then treat them as the ordinary tools that they are, or go all the way to creating things that we really know that they deserve moral consideration, and then give them the moral consideration that's their due. If we have systems in between that, we face really, I think, a tragic dilemma. And the dilemma is a choice between precaution in the sense that you just described, not the kind of precaution that I just advocated, and a different kind of view, right? So if you say, hey, look, here's a thing, we don't know whether it's conscious or not, we don't know whether it's really a right-deserving thing, Or not. As long as there's some doubt, let's treat it as though it is, which is precautionary in a certain way, that whenever you treat something as having rights, then you're committed to making sacrifices on its behalf. Just as a toy example, there's a fire. In one room, there's a human who's definitely conscious and rights-deserving. In the other room, Are 2 language models.

2 GPUs.

Yeah. That's a justified 15% credence that they're conscious and rights deserving. If you treat them as though, if they might be conscious, then you got to treat them as though they're fully conscious and rights deserving. There's a gap between conscious and rights deserving, which we're so late in this program, I don't want to get into that. So let's just assume that they go. Then you got to let that human die. But if it turns out what you think 85% likely is the case, that those are just like basically complex toasters, then that's a tragedy. Or if you think in terms of existential risk, right? One of the ways that people who are concerned about existential risk argue that we should be very cautious with our treatment of advanced AI. If we treat advanced AI as though it has rights, then we can't be cautious in the same way. Arguably, to keep it in a box would be imprisonment. To run it in a simulation would be fraud. To delete it would be murder, right? So if you take the precautionary principle that you described and say, as soon as we think it might be, you got to give it full rights, then we greatly magnify Existential risk. Now, maybe we should, maybe that's the moral requirement. I don't think that minimizing existential risk is the overarching moral demand upon us. It's got to be weighed against other things, but that's part of the price of treating the precautionary principle the way that you recommend it. So that's why I prefer my version of a precautionary principle that says, Don't create those systems. Don't create those systems about which you have doubt, because then if you're precautionary in your way, you got to treat them as though they have rights, and that's going to bring substantial costs and risks to things that we know have rights, us and our friends and other people across the world.

Would that mean though that we basically can't create any AIs? Because if there's one takeaway from this overall conversation, it's it's pretty much radical uncertainty, right? And that would mean we certainly can't build something we know is conscious. And I think we've made a pretty good case that we shouldn't be very confident that anything we create, especially if it's sophisticated, is definitely not conscious. It seems like the middle is very wide in this analysis, right?

The middle is very wide. So I do think it creates substantial limitations on engineering progress. However, I will say that we recommend the design policy of the excluded middle as a defeasible policy, not as an exceptionless moral demand. So here's what I think the kind of the appropriate practical compromise is. What you do is you allow cutting-edge researchers to create models about whom there's Substantial uncertainty, but you do it in a way so that they maybe then are precautionate, have precaution in your sense of treating those models very carefully. You make sure that it's done in environments where treating those models carefully is not going to have high risk of bad consequences. Otherwise, you don't engage in that engineering project. And then when it comes for widespread use for users, you find versions where it's reasonable to think this thing is not a rights-deserving entity, right? So you can create a complex system, and then maybe the user version of it is just a simple feed-forward network, right? And maybe a lot— I'd say the majority of functionalist views of consciousness, you've got to have feedback elements in order to have a consciousness. system, right? So you create versions where it's just a simple feedforward for the users. So you can still do certain kinds of engineering tasks in a careful way, but you don't want to broadly available in environments that will create risk if people get attached and are precautionary. in your sense, for wide use. And that does limit things. Of course, it's always been the case that we are limited by ethics.

Yeah. Or at least should aspire to be. Do you think there's an analogy here to— and some people, of course, would object to this notion, but I'm thinking of ethical farming. I think of cows and I'm like, I think cows are probably conscious. I think the wellbeing of the cow probably matters, like how we treat it matters. And yet I'm very open. In fact, my intuition is that there probably is a version of farming that is good for the cow in the sense that it's, it would rather be a cow that gets farmed in this way, even if it ultimately gets killed in the end, because it gets to have the life that it has. And that life is like— reasonably good, as opposed to never having existed. Is that like a workable mental model in your mind? And could we think of AIs as analogous to farm animals in the sense that maybe we can't create a perfect environment or a perfect existence for them, but if we are reasonably careful, we can have some justification that it's like, it's kind of a win-win still?

I would be very careful with that. So you might be right about animals, but if we're starting to talk about something with human-grade capacities, intelligence, consciousness, moral standing, that kind of principle can lead to very obnoxious results. So here's a thought experiment. Anna and Vijay decide to have a child. They treat the child well until the child's 9th birthday. On its 9th birthday, they're like, you know what? I think we'd rather have a boat. We don't want to pay child rearing expenses in perpetuity for this to age 18 for this entity. And we can't find anyone else to take care of him, so we'll just kill him painlessly. Now, the child had a worthwhile existence. If in some sense you gave the child, hey, would I— would you rather have never existed versus having had 9 happy years? Maybe the child would reasonably say, oh, well, I'd rather have 9 happy years. But still, of course, that's morally heinous for Anna and Vijay to have done. When you create something that has a standing, a moral standing like A human, whatever that constitutes, whatever constitutes that, which is a really complex question, then you create something for which the appropriate test of how it should be treated is not, would it rather not exist than have this existence? You create something instead that ought to be treated as an equal. appear like a child. You have obligations to it like a parent has to a child. So I think there's this— the risk, one of the risks in the kind of thinking that you're talking about is the risk of creating excessively disposable and excessively subservient slave entities who would still rather exist than not exist, and you crank up their hedons or whatever make them happy. Maybe they love nothing more than to serve you, but there's something obnoxious in that, I think.

Yeah, it's a brave new world, to say the least. That might be a great note to end on, a sobering note of caution. And I do recognize we've stayed longer than I expected us to stay, so I appreciate all your time and patience with all my questions. Anything else that comes to mind that you think we should touch on or any other sobering thoughts you want to leave the audience with?

Oh, I think we've covered plenty for the audience to try to digest. So yeah, thanks for having me on this show.

Yeah, this has been fantastic. Few questions more important in the world right now than what exactly are we creating? How will we know? What do we do about it? How do we make sure we're acting ethically in response to these things or in relationship to these things. And I certainly continue to be very puzzled by it, but I think you've certainly helped at least put some good structure around the thinking. So I appreciate that a lot.

Thanks.

Eric Schwitzgebel, professor of philosophy at the University of California, Riverside. Thank you for being part of the Cognitive Revolution.

Yeah, good talking with you.

It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.