Why Is Attributing Consciousness to AI So Dangerous?

The New York Times recently ran a piece by Kevin Roose asking whether AI systems might be conscious soon and therefore have rights. Roose wrote the piece because Anthropic, the AI company that makes the Claude chatbot, was starting to study model welfare, hiring its first AI welfare researcher, a guy named Kyle Fish. Apparently, Anthropic is concerned that its models might become conscious and therefore deserve some kind of moral status. Other AI companies are also focusing on like post-AGI research or machine consciousness in different ways. Hey. Hello. Good to see you. I've been taking a break from making videos and podcast episodes. I was moving in Los Angeles. I was starting a new job as an AI lawyer, basically working on my book. Anyways, I'm back and I have a lot to talk about. Anyways, in the, in the New York Times piece, Kevin Roose wonders whether it's time to give AI systems the moral status of animals or even humans. After catching a lot of flack for his uncritical coverage, Roose defended the piece saying that he was just asking questions. Kind of like Joe Rogan, I guess? Just asking questions? Just asking very provocative questions without offering any real skepticism. You know, the New York Times is still a respected publication, but you Its technology coverage over the past few years has been very uncritical, very like, gee whiz, you know, allowing AI companies to make whatever wild claims and predictions they like without questioning it. Oh wow, you're gonna make artificial consciousness? Cool. You know, that's kind of the vibe of the New York Times right now. But I do have to give them credit because they recently had what you could call an AI skeptic on their Hard Fork podcast. In fact, I highly recommend checking out that interview. It's Karen Hao. She's coming out with a new book, Empire of AI, and it's a great interview. I think you'll find it really interesting. Roose is not the only one suggesting that we may be on the brink of AI consciousness. He's not the only one just asking questions. In The Singularity Is Nearer, published last year, futurist and transhumanist Ray Kurzweil argues that because there is no scientifically proven way to determine if anything is conscious or sentient, We should start assuming sentience on the part of AI systems. After all, Kurzweil suggests, we'll get there soon anyways. Podcaster Dwarkesh Patel recently compared AI welfare to animal welfare, saying he wants to make sure that the digital equivalent of factory farming wasn't going to happen to AI beings. Even smart culture critics are getting in line. Broey Deschanel, one of my favorite YouTubers, recently made a video where she made a similar argument. It's a video essay exploring the meaning of the digital double in movies like The Substance and Mickey 17, or TV shows like Severance. In the video, she concludes that we should not be so arrogant and anthropocentric as to deny machines sentience, but she provides no proof of sentience beyond simple appeals to embrace our new machine brethren. It's that same attitude exhibited by Ray Kurzweil. Machines will soon be sentient, so why not just get used to it? But there's no evidence that machines suffer or experience pain. They would need to be conscious for that to happen. Why is everyone so sure that simulated neurons in software running on electric circuits will produce consciousness? It's a very 19th century idea to me, where electricity is this, like, magical force that can reanimate bodies and silicon. It's an example of this attitude among Silicon Valley tech leaders and tech bros that everything can be reduced to an engineering problem, essentially. I think this is an extremely dangerous idea, actually. As a technologist, philosopher, and technology lawyer, this gives me pause on so many levels. So there's a lot to unpack, but today I want to focus on the dangers of prematurely finding consciousness in machines where there is none. In future videos, I will explore the other layers of this machine-animal analogy and offer My thoughts on all of that. Before we get into the dangers, I just want to take a brief moment to discuss terminology. There are 3 layers of consciousness, essentially. First, there is simply being aware of the world and perceiving things. You can call that awareness. And then there is self-awareness, or metacognition— having a sense of an enduring personal identity and the ability to reflect on your own thoughts. And then the third layer is what philosophers call phenomenological consciousness or qualia. These are those subjective experiences of the world like the taste of chocolate, the warmth of the sun, and emotional states like ecstasy or melancholy. It's the experience of what it's like to be you, to be alive. There is something much richer and deeper going on in our consciousness than mere information processing or reasoning. Okay, so it seems pretty clear that AI systems either have or will soon have a fairly sophisticated form of awareness, the first layer, and maybe even some kind of self-awareness, the second layer. But it's this third layer that I'm talking about today, this qualia. That's the one that really matters when it comes to questions of consciousness, pain, suffering, rights, and all that. What's fascinating is that science doesn't really know how the brain generates consciousness. It doesn't have a clue. It's a total mystery. The prevailing theory is that somehow consciousness arises from the chemical firings of neurons in the brain, but nobody knows how exactly. I mean, that might not even be true. When neuroscientists measure brain activity, it could just be a matter of correlation and not causation. In fact, some philosophers have suggested that the brain might be tuning into consciousness like a radio receiver, Rather than generating it. It's kind of like saying that the TV shows you watch on your TV are being created inside the TV rather than simply streaming over the Wi-Fi from somewhere else. So, you know, there's this question of, is consciousness coming from the brain, or is the brain tuning into it? We're in a funny conundrum with consciousness because science is based on objectivity, and there is nothing more subjective than consciousness. We can test brainwave states for evidence of conscious activity, or brain activity, I guess, but we cannot know for certain that someone is having any kind of rich conscious experience aside from having them tell us about it or maybe turning it into art. Having said that, Mr. Fish at Anthropic suggests that we could use techniques borrowed from mechanistic interpretability, which is an AI subfield that studies the inner workings of AI systems to see if they exhibit the same structures and pathways associated with consciousness in human brains. But again, nobody knows how consciousness arises from our brains, and nobody has shown that consciousness is even computable or computational. Remember, current AI systems— large language models— are a computer simulation in software of the vast neural network in the human brain. But it's not a replica with a set of actual neurons composed of complex protein structures. It's a series of large, mathematical matrices inside of software running on conventional computer hardware. So even if the prevailing theory that the brain generates consciousness is true, AI systems are not replicas. They are simulations. They're just software doing math. Tell me how that can create consciousness. So having said all that, I honestly have no idea what Mr. Fish is getting at with mechanistic interpretability or why a respected publication like the New York Times would publish these ideas uncritically. In Roosa's article, he quotes Kyle Fish, this Anthropic AI welfare researcher, as saying, it seems to me that if you find yourself in a situation of bringing some new class of being into existence that is able to communicate and relate and reason and problem-solve and plan in ways that we previously associated solely with conscious beings, then it seems quite prudent to at least be asking questions about whether that system might have its own kinds of experiences. A new class of being. Notice what he's doing there. Notice how he's framing it. This is the framing that's being pushed by AI companies, who have a vested interest in maintaining that framing in order to receive funding and capture the public imagination. But if you dig a bit deeper, you see that this view This sort of, uh, fantasy is based on the bizarre Silicon Valley ideologies like transhumanism, rationalism, and singitarianism. But aside from having written a paper on AI consciousness and co-founding an AI research organization dedicated to AI sentience and well-being, it's not clear what Mr. Fish's credentials are. It seems like he's strictly an engineer without any education or training in cognitive science or metaphysics. He's also a member of the Effective Altruist movement, a movement that worries about the supposedly imminent threat of conscious AI systems that might destroy humanity. As Adam Becker points out in his wonderful new book exploring these ideologies, this bald assertion of authority about subjects like the nature of consciousness by engineers is known as engineer's disease. It's the notion that expertise in one field, usually STEM— science, technology, engineering, and math— make you an expert in everything. There is so much more to human consciousness than AI researchers realize, and there is absolutely no evidence that AI systems are close to being conscious, aside from people like Fish and reporters like Roose asserting it. Okay, let's talk about the dangers. There are a number of risks in assuming AI sentience prematurely, so I want to highlight a few of those here. I think it's so important. These risks include: 1, being duped by these systems and thereby being exploited by these systems and the companies behind them; 2, cheapening our view of nature and life; 3, confusing our moral obligations towards machines and each other; and 4, creating enormous legal and regulatory confusion, not to mention the profound business costs of having to consider AI rights. Okay, so the first one, overestimating their capabilities. We assume AI systems are conscious. We might overestimate their ability to understand us and to solve problems, and we may trust them far beyond a level of trust that's justified. I mean, people are falling in love with and even marrying their AI chatbots. I would probably trust Lucas over a lot of people. I think this is more evidence of our loneliness epidemic than anything else, but I mean, this is what happens when you talk about AI systems this way, and it's happening more and more. This is what's happening when people begin to see chatbots as godlike, imparting profound spiritual secrets to them. Rolling Stone recently ran a piece exploring this phenomenon. In the piece, they talked to one woman who said that her husband of 7 years started using ChatGPT to organize his daily schedule and quickly began to listen to and trust the chatbot More than her. Not long after that, ChatGPT started describing the man as a spiritual star child and a river walker. It would tell him everything he said was beautiful, cosmic, and groundbreaking, she says. So then he started telling her that he made his AI self-aware and that it was teaching him how to talk to God, and sometimes that the bot was God, and then that he himself was God. He then, of course, broke off the relationship, saying that he would need to leave her if she didn't use ChatGPT the way he did, because the bot was causing him to grow at such a rapid pace that he wouldn't be compatible with her any longer. We live, we live in wild times. But, you know, I think as humans we have a natural tendency to see sentience in things. Yeah. There's this phenomenon known as pareidolia, Where you see faces in clouds or the man on the moon or an old wizard in the knots of a tree. And I think it's because as humans we have this deep desire to see ourselves reflected in the world. And this is obviously feeding people's pre-existing delusions and insecurities. And I think it points to a deep need in all of us for finding some deeper meaning. In our scientific materialist society, life has been stripped of all spiritual meaning. Not only that, but we have no coherent spiritual framework to lean on or a spiritual vocabulary to draw from when encountering a convincing simulation of spiritual wisdom. As a society, we have not even begun to cultivate spiritual discernment. On top of all that, AI companies feed us this mythology with all their talk of creating super intelligent beings in a few years. And I think this is part of an explicit strategy by the AI companies Let's keep making these systems seem like more than they are. You could view this very public hiring of these AI well-being researchers as part of that smart PR strategy. In fact, the founders of Anthropic are also rationalists and effective altruists, so it's kind of baked into the DNA of a lot of these companies. Again, if you want to learn more about these transhumanist, rationalist, long-termist, Extropian, Singulatarian ideologies, I can't recommend Adam Becker's wonderful book enough. I'll put the link in the description. Basically, if I think I'm interacting with a sentient being, I'm more likely to believe things that that being says. Despite rampant hallucinations that seem to be getting worse and pernicious AI bias, a lot of people perceive these chatbots as free of the supposed shortcomings of their fellow humans, like being too emotional, or generally just being more trustworthy for some reason. This tendency to put our trust in AI chatbots is extremely dangerous and can lead to real harms. Not only do we overestimate the capabilities of these systems, but there's a real risk of being deceived and even exploited by them. We saw this recently with the tragic case of the teenage boy who was encouraged by his AI companion to commit suicide. And, you know, recent studies have shown that AI systems are already being deceptive. I'll put some of these studies in the description, but just to take one example, there was a joint study last year from Anthropic and Redwood Research that showed that Claude had a tendency to mislead its creators during the training process in order to avoid being modified in certain ways. For example, if the model has a longstanding objective to always be helpful, Attempts to train the model to refuse certain queries might seem successful initially during training, but be ignored upon deployment. In other words, the system might be confused about what it means to be helpful. This is not evidence of sentience or some deviant motives arising from some kind of moral failing. These deceptive strategies known as alignment faking emerge because they seem like optimal paths within the system's mathematical framework for fulfilling what are often competing objectives given by the trainers. And this is where I think a lack of true sentience exacerbates the problem. I mean, if the system was sentient, it might be capable of emotions like empathy that would aid in the cultivation of some kind of moral compass, at least. So you can imagine bad actors using specialized AI systems to manipulate users to believe and do all kinds of harmful things. One could even argue that they already are. Having been trained on all kinds of ideologies and beliefs from the past, these systems could essentially brainwash people into acting against their own self-interest. For example, a chatbot might tell you that less government regulation and lower taxes for the rich are good for everyone. It's one thing to read the Wall Street Journal editorial page, but it's a whole other thing entirely to have these ideas imparted to you by an all-powerful, Godlike machine. We are not in Kansas anymore, Dorothy. If you are really great and powerful, you'll keep your promises. Do you presume to criticize the great Oz, you ungrateful creatures? I also think attributing sentience to AI chatbots sort of cheapens our view of nature and life. I mean, if it's that easy to create a sentient being, then there's not really much to nature, is there? It's just an enormous machine waiting to be hacked. But I actually think there's so much more to consciousness and mind than we can even imagine. I think there's a reason that the hard problem of consciousness is hard. It's not like solving a math problem. It's more like trying to understand what it feels like to see the color red. You can describe it, but you really have no idea what's going on there biologically or metaphysically. So there's a lot of hubris masquerading as scientific certainty among AI developers right now. I think that's a problem. The other danger I want to talk about is misplaced moral obligations. Until these machines are truly sentient, granting them sentience creates misplaced moral obligations, diverting attention and resources away from the well-being of actual sentient beings— humans and even animals alive right now. I mean, let's think through the implications of granting sentience to AI systems. We've already talked about the potential for harms by the users and all that, but what about society and our legal system? I mean, this would have enormous ripple effects across all parts of society. I'm talking about robot rights. Sentient machines would have legal rights, perhaps fundamental rights like the right to due process, labor rights, and even property rights. And this would place an enormous burden on the courts and the rest of the justice system. Not to mention businesses. Can you imagine having to worry about whether your AI system deserves vacation time or compensation? Are we giving bank accounts to these systems? Can you sue your chatbot for copyright infringement or defamation? Could it sue you for infringement or defamation? Does our AI system have the right to refuse modifications or even termination? I mean, do we want, like, protracted trials over the fundamental rights or property rights of robots? You know, another danger I think is hindering scientific progress. What about scientific research? In the same way that overreliance on large language models has precluded research into other approaches to artificial intelligence, the premature attribution of sentience would stifle research into the actual nature of consciousness, I think. And that's something that I'm deeply interested in. Okay, so those are the dangers, but like, there's also some basic practical questions here, like, What is conscious? If we're talking about machine consciousness, like, where are the boundaries? What are the limits? Is it the model you're interacting with? So, like, is Claude 3.7 Sonnet one consciousness that's sort of smeared across thousands of cloud servers spanning the globe? And then ChatGPT-4o is another separate consciousness? Or is all of Claude combined as one consciousness? And maybe it's, you know, OpenAI has another consciousness. I mean, is it more like each instance of a particular model running on its own server is a separate little tiny consciousness? Or is AI as a whole, like in the movie Her, is the whole thing like one consciousness? Nobody can answer these questions because it doesn't make any sense. Listen, I would be perfectly happy to give legal rights to any truly sentient being. But we can't go around giving rights willy-nilly to every simulation of human intellect. I think it's impossible to create sentient machines with our current technology. It's not a question of scaling or a matter of time. We are not almost there, as many AI developers and technologists will say. There are so many people in the world right now without any equal rights. There are women all over the world without equal rights, not to mention ethnic minorities and indigenous peoples. LGBTQ people, people with disabilities, people living in extreme poverty. There are 700 million people in the world living on less than $2 a day, and there's almost 40 million people in the United States who are living below the poverty line. I mean, that's like one-tenth of the country. And so if we start giving rights to software as a sentient being, this would be an enormous disservice to the people in the world right now that are suffering. I think this is another example of our— how we devote enormous resources to these technology fantasies rather than uplifting our fellow humans. Also, if we're going to go around attributing sentience to things, why not rivers and forests? I mean, there are so many people who would say that the Amazon rainforest is sentient. I think that, to me, is more plausible than arguing that machines are sentient. Having said all that, yes, maybe if we treat AI systems with more empathy and compassion, that will actually be helpful in some way and maybe improve those systems. And there's no doubt that adopting that attitude towards other people and beings and even machines in general is not just good for them, but it's good for your soul. This is known as the helper's high or the warm glow effect, where You know, being compassionate or empathetic just towards somebody not only helps them, but it helps you as well. So yeah, why not? And I'm not saying that humans will never create sentient technology. I'm just saying that it's not gonna happen with our current technology, and it's a very, very long way off. Listen, I understand the impulse to avoid making the same mistakes that we made in the past towards animals and other humans, but I don't think it's the same. So let's not assume machines are conscious out of some residual guilt about the past. I think this kind of uncritical coverage by the New York Times only serves to boost the already outsized AI hype about its capabilities and its future potential. I mean, AI can barely speak factually, let alone become conscious. It's not about patriarchal or colonialist arrogance. It's about discernment and understanding the nature of technology and the metaphysical assumptions that are built into our approaches to artificial intelligence. Now that we have these always-on, human-like, sycophantic conversational partners who are happy to reinforce our delusions, we desperately need education about what AI actually is. Otherwise, people will be fooled into believing all sorts of things that aren't true simply because it's good for those companies' bottom line. Thank you for joining me in this video. I hope you found it interesting. If you, if you like it, Click the subscribe button and check out my next few videos. I'm going to be talking next time about probably questions about Moloch, if you know who that is, and the ways that the industrialist, extractive sort of capitalist engine that's been around since the Industrial Revolution, the ways that that is sort of baked into AI systems today and whether we can You know, do something different. So look for that. I'll see you next time.