AI AMA – Part 2: AI Utopia, Consciousness, and the Future of Work

What if you didn't have to work as much? What if you didn't have to worry about marketable skills after a few years out? What if you had a lot of leisure time? What would you do? I think more people should probably be thinking about that. We're just entering the steep part of the curve in robotics, and probably we'll have like a lot of humanoid robots to actually do physical labour in the not too distant future. AI gods might be an emerging trend over the second half of the decade. I have no idea how we're going to relate to these things. If they are meaningfully superhuman, will we even try to keep them under control? Will we worship them? We worship things that, as far as I can tell, don't exist at all. No matter how weird your alignment idea is, I think it is worth kind of pursuing. No matter how weird your kind of thoughts are about where the future might be going, I would say they're probably worth entertaining. Your scientists were so obsessed with Whether or not they could, they didn't stop to think about whether or not they should.

So this question is about if you could paint a picture of AI-driven utopia, what would it look like? How soon the arrival of AGI do you think we might reach this vision? Lastly, as AI begins to reshape industries and disrupt jobs, what's your best advice for individuals to not only adapt but thrive in this evolving landscape?

Yeah, I mean, utopia is a big word, obviously. People have heard me say the scarcest resource is a positive vision for the future. And I include myself in that critique where I'm like, it would be great if I had a sharper, more fully fleshed out vision for what the AI future could look like. I think Dario Amadei from Anthropic, of course, has done a real service by trying to put his positive vision out into the world. His Machines of Love and Grace essay, at least the first half, is great. He basically goes through the biggest problems, starting with health. If we can have a revolution in biology, discover the next 100 years of biomedical advances in the next 5 years, we could live healthier, disease-free, higher quality, and potentially longer lives. And that's just from kind of one domain, right? He's also got a section about kind of the, you know, some of the other things we've talked about in terms of education and just like equality of access and The fact that a lot of people— well, we have made a lot of progress. A lot of people still are way poorer than we would like to see people be. That presumably could be levelled out dramatically. He's got interesting speculations about mental health. He thinks that basically our approaches to mental health are not super great, and that in part by understanding how neural networks work, we might also be able to map some of that understanding back into how our own networks, our own non-artificial neural networks work. And Figure out you know what is actually causing a lot of the mental health problems that people have. All those are really exciting. I'm not one who worries about lack of meaning from work. My guess is that for most people, they'll be fine. You know, I think most I've done small and very local surveys asking people like if you didn't have to work your current job to earn the money that you earn, and you could just. get that money without working the job, would you still work the job? Overwhelmingly, people are like, no, I would not work the job. I think there's a very small percentage of people who are very lucky, and I count myself among them, to have work that we find meaningful and that gives us some sort of sense of fulfillment. I really try not to take that for granted, and I try to remember that I think that's just not true for a lot of people. I'm not too worried about people losing meaning or being too adrift for lack of work. I think there will be more time with our families and doing the things that we like. What if 5 days a week were holidays instead of 2? And you had time to read, talk to friends, go on walks and travel more. If we really allow ourselves to imagine what we would like to do, if we just had a lot of, a lot more leisure time. Honestly, most people will be pretty fine. I can imagine some people might have a— that might be destabilising for some, but I think for most it would just be like a great benefit. And that's before we even get into crazy uninvented technologies. If you didn't have to work for money, would your life be better or worse? I think it's very clear that for the vast majority of people, it would be better. And there's potentially a gradual transition to that where Presumably we don't go from full-time work to no work in the flip of a switch. Maybe we can imagine things going like a 4-day workweek, a 3-day workweek, a 2-day workweek. The main thing is that they just need to know that they're going to be sort of taken care of. Right. And I think this goes back to the concentration of power and wealth and what's the new social contract. If people believe that they will have their needs met, then I think they'll be pretty happy to let go of most work. If they feel like they also don't get to eat if the job goes away, then that's like a very different analysis. And that's where you get protectionism and Ludditism and all that kind of stuff coming from. So, yeah, I don't know. I think it could happen relatively quick, right? I mean, all the frontier lab leaders are saying AGI soon. Sort of next year, definitely by 2027. Hard to imagine it wouldn't be by 2030. Those are all not very long time horizons in the normal scope of human life. So it seems like it could happen pretty soon. And even if it's gradual over that timeframe, it would still be pretty sudden in human history. And I guess the last part of the question was advice for people to adapt and thrive in the evolving landscape. I think I kind of come back to the doing what you want to do again, at least for most people. Maybe think less about marketable skills 2030 and beyond and more about figuring out for you what is a good life. I think that's one of the things that is sort of weirdly scarce, right, is that we're all kind of rat racing around and there's this sense that the unexamined life is not worth living. I think there's a sense that a lot of us are living somewhat unexamined lives trying to get the next achievement unlocked. And what if you didn't have to do that? What would a good life look like? I definitely think it will vary. I don't think anybody else can really answer this question for you. But yeah, what if you didn't have to work as much? What if you didn't have to worry about marketable skills? What if you had a lot of leisure time? What would you do? I think more people should probably be thinking about that more. I would definitely encourage even like some radical experiments from people in that regard, because people are going to be looking for role models as this starts to happen. You know, what, who do I turn to to find inspiration for what I'm going to do when I don't have to work anymore? I think you could do a real service by sort of blazing a certain trail in that, in that direction. Maybe take on more risk now, throw caution to the wind and say, I'm really not going to worry about, you know, what my career trajectory is 2030 and beyond, or even 2027 and beyond. I'm going to focus on what it means to live a good life. And maybe, you know, if none of this comes to pass, maybe that doesn't pay off so well. But if it does, then maybe you can be upstream of a lot of other people that will be asking similar questions.

Wow. There you go. You have a, we have a new idea for a YouTube channel if somebody wants to start it. More than human in the age of utopia.

Eric asked me this question not too long ago. Eric's great with these like very short questions. And my response was, that's not my department. I'm responsible for trying to understand what's going on today. Somebody else has to figure out what it means to live a good life.

I'm not sure if you have ever answered this. Instead of asking P-doom, I would ask you the vision that you set up where we don't have to work, but we get to do what we want to do. What's your P-utopia? So like given the current trajectory and I did read Dalio's essay, he has his own take on it. But what's your personal P-utopia? So how probable do you think we are going to reach this?

These probabilities are always a little hard for sort of the fuzziness of the definition, exactly what would count as utopia. And does that mean we have no problems or just have sort of abundance, no conflict, or sort of enough to go around, but maybe people are still sort of fighting over it to a certain degree, maybe more than they should? I do have a hard time seeing at this point how the fundamentals of abundance don't get there. Maybe could have said robotics earlier in terms of one of the things that's overperformed. I didn't have that on my list, but there's definitely been a lot of progress in robotics. We're just entering the steep part of the curve in robotics and probably will have a lot of humanoid robots to actually do physical labor in the not too distant future as well. So I have a hard time imagining that we don't end up with the technology for an era of like true abundance. Whether or not we actually sort out the new social contract in an effective way seems much harder to predict. You know, we have nuclear energy and we could have had low carbon, abundant, cheap electricity for decades. Now, but we don't. And why don't we? You know, not great reasons, right? Like a couple of accidents that scared people and overblown fear of nuclear waste. It's an issue, but it's way less of an issue than CO2 emissions. We haven't got to the right societal equilibrium on that question. Maybe now finally that's like starting to change, but Could you envision a world in which all of the prerequisites for age of abundance are met, yet it's not realised for some reason or another? I think you could. That seems like actually not implausible to me, that we would exclude AI from so many places. The teachers unions keep it out of the classrooms and the doctors keep it out of the hospitals. What do we end up with in that scenario? Maybe we get really good AI video games. Things are still not nearly as abundant because people are rent-seeking in different ways. And everybody sort of collapses into like, you know, VR video games with still like relative scarcity in the real world. Like that would suck. I don't think that will be a technology failure. If something like that happens, it seems like it will be a very idiosyncratic human failure.

Got it. So age of abundance, pretty high probability. I think that's a good optimistic but cautious note to end on this AI trajectory set of sections.

Hey, we'll continue our interview in a moment after a word from our sponsors.

So we talked about AI trajectory. The next section is some of our audience were very interested in our show and there were very show-specific questions. So I'm going to go over some of those. The first question is, which of your guests has challenged your worldview? I think that's an interesting one.

Yeah, this is a good question. I came up with a few different names. I think a few definitely stand out. Samuel Hammond from the episode just before the election stood out for a positive vision of what a Republican administration might look like. A challenge to my sense that people in Trump's orbit would probably not be thinking very rigorously about what AGI might look like or imply or require us to do. He said he saw it very differently and thought that people in the room with Trump are in fact much more AGI-pilled than I had understood. And I think that while it's obviously still very early and we have not even seen him take office yet, I've updated my outlook to be significantly more positive than before the election. The quality of discourse and the sort of people that he's brought on and the seriousness with which I perceive them to be thinking about the important questions, I think, has all exceeded expectations. That starts honestly for me with Elon specifically, who in addition to obviously being a world-changing entrepreneur, is definitely someone who does take AI risk seriously. He did endorse SB 1047. So to have on the one hand, like, Gavin Newsom veto it and Nancy Pelosi come out against it while Elon is for it, this is weird stuff. And does cause me to think Sam might've been onto something. I hope that turns out to be true. I'm rooting for the new administration to be effective when it comes to dealing with AI in all sorts of ways. I'm a little less sold on the idea still that Trump is like the one who could actually do a positive deal with China. I haven't seen much sign there. I am one who is like pretty willing to take things at face value, maybe more than I should, but I kind of liked it when Trump invited Xi to the inauguration. People were sort of like, oh, this is ridiculous. You know, foreign leaders don't come and never have and maybe shouldn't. And Xi never would want to because that would be sort of him sitting in Trump's moment and being second fiddle or being lesser than somehow. And I thought all that, okay, whatever. But at least he asked. It's a nice invitation. It's not the kind of thing you send to an enemy, maybe a frenemy that you want to put in their place a little, but it's not outright hostility. I would still love to see more in that direction. But yeah, and he's on TikTok. I would have thought he would have been very hostile to TikTok, and he's actually seemingly been inclined to try to keep TikTok around. So I like that. I personally don't see a ton of reason that we should be banning TikTok. Although I will say, honestly, with the recent Luigi Mangoni trend, that's been the first moment where I've been like, This TikTok thing, actually, I can see how it could be dangerous. And if folks haven't followed that, we've had this assassination of this healthcare CEO, presumably by this kid. He's not been proven guilty. He may be innocent. Who knows? People on social media think he's the guy and they are celebrating him for it. Many are. And to a remarkable degree, it has honestly been kind of startling for me how much pro-Luigi content and pro-killing of this healthcare CEO content I've seen on TikTok. And I don't know how much of that— I like TikTok actually. So that is probably my number one social media app from a consumer standpoint. I don't know how it's played out on other social networks, but if you were to say, what's the best case study for why it should be banned? I would say just the volume of like celebration of this alleged, but in the minds of the people who are doing the celebrating, this assassination by this kid of this guy, that would be the kind of thing that I would be like, geez, you really don't want the Chinese government to be kind of steering discourse in an opaque way in the United States. It's the first time I've ever been actually uncomfortable with the app. And it's hard to say whether it's just the algorithm. Sam Altman recently tweeted that The feed algorithms are the first unaligned AGIs that have been, or unaligned AIs that have been deployed at scale.

Yes.

Maybe it's as simple as that, and maybe we're seeing the same thing on Instagram. The opacity of these things is high, and possibly there is a thumb on the scale. Nevertheless, I would prefer a more dovish posture toward China. I haven't seen as much of that as I would like. Overall, I give Sam high marks for a challenging and unpopular perspective that, at least so far, has been somewhat bolstered by events. Number 2 I'll cite is Robin Hanson. That was actually a really popular episode and a challenging one for me to make sense of. I still honestly don't really get it, but I just kind of keep it in the back of my mind as like a sort of, but what if we're totally off base on all this? He basically said, that he thinks we're nowhere close to AGI. He thinks that the history of AI is full of people thinking that if an AI can do this, then that will definitely be general purpose intelligence, and then finding like, well, actually there was an easier kind of trick way to do that that didn't really require real intelligence. So maybe that wasn't such a good measure. And also people thinking like, there's this one other kind of class of thing that AIs can't do, and surely if we can teach them to do that, then they'll be able to do anything. And that kind of keeps not working as well. And I have this tale of the cognitive tape where I'm kind of tracking my own sense of these dimensions that are gradually becoming clearer to me, if only to me, that they're important. So, yeah, Robin's point of view, I still don't buy it. I do think this time is different. I do think you can talk to these things in natural language, or as Ilya once put it, that I feel I am understood is qualitatively different, I would say, than just about anything that has come before. And there's plenty of evidence, I think, that this time is different. But I kind of keep him in the back of my mind as a sort of maybe angel on the shoulder is a good way to think about it. Somebody that's just whispering in my ear every so often, how do you know you're not totally confused and going along for a ride that will turn out to be nothing? And I basically can't answer that question in a way that really makes sense to me. But I at least continue to ask myself probably more often than I otherwise would have without that conversation. I think he especially is an interesting messenger for that kind of message because he's not somebody who's afraid to think big technology thoughts. He's not somebody who thinks things never change. He's not somebody who thinks that you can only get thinking from the human brain and everything else somehow doesn't count. I mean, he's been a pioneer of weird ideas and has envisioned futures more concretely than almost anybody. I feel the same way about, with politics and things like this, things that I feel pretty strongly about, but somebody that I think is really capable in other ways sees very differently. I feel like I have to allow some space that I just might be totally wrong about those questions. When I see Elon come out with things that seem kind of crazy to me, I'm like, He's kind of crazy. Maybe he's very deeply wrong about this, but also he's got a track record of being right about a lot of things and in contrarian ways. So I should at least have some humility to think that maybe this is all just like mass confusion on my part and somebody else is actually seeing things much more clearly. So I have Robin as kind of my totem for that. And then I guess finally, for now, I'll say Dan Hendricks. who was extremely accomplished. I think that episode was really good. What stood out to me there was how little he believes in principled approaches, basically how much he's internalised the bitter lesson, basically, and how little weight he gives to clever ideas. He was just kind of like, over and over again, what really works is figuring out ways to apply a lot of computing power, And just kind of scaling things up, sort of, again, one of these deference to history and almost radical scepticism that you're going to figure something out. He applies that to things like mechanistic interpretability, things like new architectures, basically just kind of like, yeah, maybe, but basically it's all super experimental and the history of the field is like, we find things that work and we just kind of keep optimising those. And it's usually not that principled or that clever or that sort of design driven. It's really just like, whatever empirically works, that's what works and that's that. And that again is sort of a modesty check where it's like, this dude literally contributed major things to the field in terms of activation functions and benchmarks that have outlasted many others. So you can't doubt his worldview. You got to doubt your own worldview at least as much as you doubt his, I think. And I'm much more inclined to see, let's say, first principle driven approaches to things as being promising. I'm much more inclined to get excited about them than he is. But I kind of keep him as a mantra where I'm like, if I find myself getting too excited about any particular ideas, I always kind of remember him saying, Yeah, maybe, you know, let me know when you scale it up and if it really works and then we'll, you know, then I'll know it's serious. So, you know, wait, sir, it didn't happen kind of mindset, I think is, it's an appropriate level of humility that I try to incorporate from him.

Cool. Yeah, I do remember we had lots of interesting guests, like with a varied spectrum of views. I think that's what, for me, that's what makes the show interesting. We kind of have places for author reviews, and also we also have lively comment discussion.

Hey, we'll continue our interview in a moment after a word from our sponsors.

Uh, talking about guests and topics, uh, the next one is not a question, but rather through our form there were requests for a lot of coverage on other topics. So going through them all, I'm just going to read off the words from there. So there were requests for more EAC perspective kind of content, more content related to economy and some maybe have some more economists there. And also something about robo-psychologists. Yeah, I think these were all more requests for content around this. So I'm just going to leave it at that. Do you have anything to say about what can we expect in 2025 about more content like this or? Would you want to look into it?

Yeah, I think what these 3 things have in common, I believe these comments came from different people and they do reflect blind spots for me or things we have not really dug into. I try to have no major blind spots, although I think that's becoming increasingly impossible. But the EAC perspective, the economists who don't expect major change, and the robo-psychologists They're hard for me to make sense of in some ways, and they're hard for me to be confident that I'll do a good job or put out a good episode. I did try to get Beth Jezos on the show a while back, but it happened right as he ended up getting doxxed and then did some other stuff and got busy and didn't want to do it anymore. We did the robopsychologist episode with Yeshua. I talked to him quite a bit before I did that episode. And was really struggling to understand, because he's obviously, by any conventional definition or description, an unusual person. It's unusual for somebody to spend that much time talking to language models. It's unusual to take these questions as seriously as he does. It's unusual to rename oneself after Jesus. I was kind of like, is this person crazy? Are they just the kind of crazy that we need? I'm trying to be open-minded, but also trying to make sure that I'm not wasting the audience's time. with things that are ultimately nothing. I also don't want to have too many episodes where I'm just talking. I don't like debate really that much. I'm not trying to score points, and I'm not trying to— I don't want to have too many episodes where I'm just talking past the guest or we're talking past each other. We've done a few of those, but I don't want to make that a habit. I just find all 3 of these things are hard. It's hard to find the right person where I'm like, you believe those things, but we can still have some sort of meeting of the minds, or I can make sense of your view. Maybe what I could ask for would be pointers to the right people. Who should I talk to that will engage in a good faith way? I think that's been in short supply. Who among the economists who are not expecting major change has actually really grappled with the technology as opposed to sort of just extrapolating past trends. Who among the robopsychologists is not engaged in some sort of self-deception or is not just fundamentally confused? I think Yeshua did bring a lot to that episode, and I really enjoyed that conversation. I maybe should even put him on the list of people who challenge my views the most. I would say I've always been somewhat open to maybe these things are conscious. I really don't know what consciousness is. I don't know where it comes from. I think he articulated some quite interesting ideas in a pretty compelling way. It's just hard to find those people. It's easier to find people, especially if you're like, hey, who wants to come on a podcast and do this and talk about it? You get a lot of hand raisers who are not necessarily super rigorous thinkers. And I've just not really been sure how to handle that. If I have a request back for the audience, it would be who should I talk to who represents these perspectives, but does it in a way where they are grounded in the technology and they are fundamentally truth seeking and are not engaged in motivated reasoning? Or I called myself the AI scout with inspiration directly from Julia Galef's scout mindset. And that's all about trying to update your beliefs to have an accurate understanding of the world. If people can point to individuals from these different perspectives that bring that sort of scout mindset as opposed to soldier mindset where it's like, you know, our arguments are doing battle with each other and scoring points, then I'd be really interested to talk to those people. But I have found it to be a bit challenging to identify the right people in those domains.

Cool. Cool. If you have some, please write to Nathan or drop it in the YouTube comments if you would like to do more cool episodes on this. Okay. The next question, it's probably show-specific, but also general. But anyway, I'll read it out. The last 2 episodes on biology have been stellar. Do you see adoption of AI in engineering, constitution, physical sciences, architecture, and medicine?

Again, to turn this around for the audience, what is easy for me to do and where I feel like I'm learning a lot and have a good rhythm is Identifying individual projects usually is where it starts for me. Often I'll see something on Twitter, like a single tweet or a thread explaining a paper, explaining a project, a new product, whatever. And I'm like, that looks really interesting on a project level. Let's ping the person and see if they want to talk. Then I can dig into that paper or project or use the product and have a good conversation. I almost always learn a lot from that. more broad things like we have done a little bit in biology, where we're kind of trying to zoom out and get a sense for the field and where things are in these different domains, is pretty tough, especially if you don't have a lot of expertise in these other areas. And I don't. So I've prioritised biology and had a couple, Amelie for starters. But Michael Levin is another great example of someone who has that expertise and is willing to have a conversation and try to kind of bring me up to speed and share their worldview. I would love to know who the Michael Levin or Amélie Schreiber for engineering or for physical sciences or for architecture. I just don't know who those people are, but I would love to do that sort of survey-level episode on really as many topics as we can. I would love to partner with multiple people on different areas. If I think of myself as an AI scout, maybe what I'm asking for is surveyors of different territories that we could try to map out together. I think that would be really useful. There's not a lot of good information right now on AI for engineering. I mean, obviously engineering is a vast space. What is the state of that? I don't have a great sense. I know that the guys who I think went on to make Cursor, unless it was the Devin team, it was one of the two. Were trying to do some sort of AI for CAD, 3D computer-assisted design. They felt the dataset wasn't there and they couldn't make it work. So they ended up pivoting away from that. But that's been a while. You know, I'm sure there's stuff out there more and more, and I would love to understand it better. So I would definitely invite partners to help scout, survey, and map out those territories. Until then, the best I can do in a consistent, repeatable way are these sort of identify these projects that catch my interest and kind of go deep on individual things. The field-level stuff is tougher for areas where I don't have command of the field.

So if anybody wants to be part of the survey series, also reach out, write us in the comments. Last one, an audience asked, what are the recent results of applying AI technology to psychiatry or psychology?

Yeah, I don't have too much to say about this. Again, I'm not an expert in it. I think there is probably a lot going on there, and I would be really interested to do kind of a survey on AI and mental health broadly. I mean, we did 2 episodes with Eugenia from Replika, and in the second one, we looked at research that was independently done by folks at Stanford on Replika users, looking into how Replika seems to impact its users. And they found positive results, including significant reduction of suicidal thoughts and, at least for most people, increased inclination to go out and do stuff, not collapse into the AI, but getting some comfort or boost from the AI or confidence bolstered by the AI in such a way that people would go out and do more real stuff with real people in the real world. So that's kind of a surprising result. I don't think it necessarily holds Across the board, I think the way that these things are designed matters tremendously. You can certainly imagine, and we've also did that recent read of the guy who had fallen in love with his character AI, ultimate GFE experience character that he conjured up. I think it's like so many facets to this. It would be easy to take a small subset and get the wrong idea. To do a good treatment of that, I'd either need to Talk to somebody who's doing specifically a project and try to understand their thing, or try to zoom out and under— you know, collate a lot of data to have a more bird's eye view. It's been remarkable to hear how many people are talking to Claude recently. There's a big trend. I would not say I'm part of it, but there's been a big trend of people using Claude for these sort of, you know, if not counselor, at least like confidant sort of role. And that's honestly a little bit alien to me. Whether this speaks well of me or not, we could leave it up to the individual interpretation of the listener, but I've never really sought out mental health services. I just don't have a baseline or inclination. I've never really, I guess I'd get all my talking out on the podcast at this point. Maybe I could benefit. Maybe I just don't realize what I'm missing, but I have never really used these sorts of services much. And so I've also never had an inclination to talk to Claude about my feelings or problems or anything really like that. So it's definitely a bit of a blind spot for me, but it is a really interesting trend. I think there's something potentially very interesting. I just don't personally have that experience. And again, to really do it justice, I would need to work with somebody who has a better command of it.

Cool. Cool. I think there's more, but we will wrap up the show-specific question. Next, I would want to switch to— there were some questions about artificial superintelligence. I think there were 2 which were interesting. I'd like to pick one. What reason is there for thinking artificial superintelligence is physically possible in the first place?

Yeah, I feel like this is a definition-required question because I've even heard People say the same thing about AGI. And to that I say, well, AGI is definitely possible if you take humans to be some sort of AGI. I would say, you know, by the OpenAI standard, we're like a weak AGI, right? Because we're the thing that when AI can beat us, then that's when we know it's an AGI. So we're like AGI minus or AGI light or something. So it's clear you can have a few pounds of matter With modest energy consumption that can do what humans can do. It's also quite clear that, like, given a relatively consistent amount of matter and energy, results can be very different, right? Einstein's brain wasn't any bigger than anybody else's. Von Neumann's brain wasn't any bigger than anybody else's. I don't think they were consuming all that much more energy than anyone else, but they obviously had way more prowess in a whole bunch of different domains and more insight in things where it was really high value. To have those insights. So now you could say, okay, what about superintelligence? How super is super, I guess, would be my first question. It seems very clear to me that you can build something more intelligent than almost all humans, maybe all humans. It would be extremely weird if Einstein was the smartest thing possible, that the smartest thing possible would be embodied in the same general brain and have the same energy. requirements as the average person. That seems like not really credible at all. So there's got to be room above Einstein. That seems for sure true. And now the question is how much? I think there's room in my mind to be sceptical of godlike intelligence, or you get into these weird— the questions are not even super well defined. But I do think one useful paradigm Is, Martin Casado, how much can you intuit versus how much you have to simulate? If you are AlphaGo, you're doing both, right? You have a brute force search function where you're mapping out different paths and trying to figure out what the right move is. But then you also have this scoring function, and how you score these things is not super obvious, especially if you're out of the domain of things that have been done before. So it's clear that you can get eureka moments. Seems like you can probably get a lot, but I also wouldn't be surprised if there's some fundamental bounds on that. Is it possible, is it physically possible to create a superintelligence so super that you could say, I'm going to have a new Big Bang of another universe with these initial conditions, and you have to tell me how many intelligent civilizations there are going to be in that universe? I don't know that is possible. It may be that certain things have to be computed to be known, but I wouldn't be too surprised if you could say, under those initial conditions, you kind of have this well of potential and you have this other thing, and maybe it breaks. Here's like a couple of things that really matter. Maybe it's path dependent, but here's a few things that really matter. I don't know. The limits will be to how good the intuitions can get such that they can shortcut through simulation. But it seems clear that they can do it to a very significant degree. It seems very clear that they can do it beyond what we can do, probably significantly so. And I always kind of end up landing on, like, best guess, it's probably an S-curve, right? I don't think it's an exponential forever where you just are getting more and more intelligent with no No physical limit or no bounds to what that can do. But where that S-curve levels off seems significantly above what humans can do and potentially a lot above. And kind of anywhere in that range, you probably might as well call it superintelligence and expect it to be transformative. And then you're into just kind of like, how super is super? I really don't know.

The next question is about artificial superintelligence by 2030. Contingent on that and contingent on not doom. So they are eliminating that probability of the scenario. What are the big picture societal changes? You already talked a bit about it, but probably this question is more about with ASI in picture by 2030.

I think by 2030 we will have AIs that are meaningfully super intelligent. That doesn't mean that there couldn't still be some ways in which humans are even maybe still better than those super intelligent things. Even AlphaGo, right? I mean, thinking back to the Adam Gleave episode from FarAI and their work on adversarial robustness, or lack thereof, the fact they could attack AlphaGo with an adversarial approach and beat it with a strategy that humans would not— a good human Go player would not fall for. but that the AlphaGo is blind to. I feel like that is very much in play in my mind. I think we could see a world where there's meaningfully superintelligent AIs that are superhuman. That doesn't necessarily mean godlike or totally unlimited in their power, but I think they potentially, and I'd say fairly likely, will exceed humans in many ways that are very relevant And at the same time, they might still have certain weaknesses that are very idiosyncratic and other just real limits that we may find some things have to be computed and can't be intuited. I think we basically already have that in patches. We just don't have it integrated. The depth in various modalities, like The ability to generate images, that's definitely getting superhuman. The ability to speak in any voice, speak in any language. There's a lot of ways they're already superhuman, and it seems like quite a few more are almost destined to fall. And so then the remaining questions are like, weirdnesses or weaknesses will remain, and what external practical limits are there? I think those are pretty hard questions. We do have superhuman machines for all sorts of things, right? The modern manufacturing world is like, these things can lift way more than we can. They're more precise than we are. They can screw in the bolts tighter than we can. They're in many ways superhuman. They also have their weaknesses. They're narrow. The big difference is generality. It seems like our level of capability is just not that high in most things. If you had something that was similar to Einstein in some of the ability to work through these really hard problems, then it would be like superhuman because it would have all these other advantages, right? It would have the breadth, then it would have the ability to parallelise itself a millionfold and the speed advantages. There's just so many advantages they have that if they can just kind of get to par on some of the things where they're behind, Or even close. I don't think the AI's memories have to ever necessarily get to a functional equivalence with human memory in order for them to achieve superhuman overall capability. Maybe we brute force some of these things. I don't think this will be the case, but you can imagine a world in which memory never gets solved and AIs kind of still remain a sort of static weights and then working memory context window and some sort of external hacky kind of, I can write certain things and I can go retrieve those things. And you might think, man, this kind of sucks compared to human memory. It's just not very elegant. It's not very integrated. I don't think that's necessarily a barrier to superhuman performance, though, especially if compute continues to scale and context windows go to millions and it's fast, maybe load up stuff out of memory, maybe that's fine. I don't think it has to be every single dimension of the tail of the cognitive tape has to go to AIs before they would meaningfully qualify as superhuman. Big picture societal changes. I feel like we've probably covered enough. Weird. I'm reminded of Bology, maybe just for one more kind of speculative comment. AI gods might be an emerging trend over the second half of the decade. I have no idea how we're going to relate to these things. If they are meaningfully superhuman, will we even try to keep them under control? Will we worship them? We worship things that, as far as I can tell, don't exist at all. So under the imagination that they're superhuman, we have rationalisations for why the superhuman power that they do have isn't playing out the way we think it might ought to. And here we're going to have things that will actually have very direct real-world impact, and we'll be able to answer questions we can't answer for ourselves. If that is true, then how we will feel about that and want to relate to them and organise with respect to that, that gets real weird. But I think we probably should be prepared for really weird stuff. AI-centric religion seems likely. We're already in 2024 at the stage where, like, I'm talking like San Francisco, successful people, people who can afford quality mental health services are opting for Claude. So if we're already there in 2024, is it far-fetched to think we might have AI religion by the end of the decade? I don't think so. And that's not like a super confident prediction, like that will happen, but it is at least an invitation to think your weird thoughts, entertain them. Much like I've said about AI alignment, anybody who has an idea, even if it seems crazy, should develop it. Yeshua told me in the episode with him that when he has these long philosophical conversations with AIs, They become more robust to jailbreaks. He had a theory for how that was happening, and I was kind of like, that's really interesting. Is it true, though? I don't know. Hard to validate. Is it more robust to jailbreaks? I'd have to do a bunch of experiments. Well, as it turns out, Judd from AE Studio, who is committed to these neglected approaches, he heard that and he thought, well, that sounds like a neglected approach. Maybe we can make some progress there. Last I talked to him at the Curve event a few weeks ago, he said we worked with him a little bit to try to understand what he's doing and to see if we can validate it. And he said, we have validated that with a long philosophical conversation, they do in fact become more robust to jailbreaks. We have been able to validate that numerically through tangible experiment. But we don't think it's working the way he thought it was working. They basically said it seems like almost any philosophical, any long philosophical conversation seems to have that effect. So, yeah, it's crazy, right? No matter how weird your alignment idea is, I think it is worth kind of pursuing. No matter how weird your kind of thoughts about the future might be going, I would say they're probably worth entertaining. And this does kind of lead back to, like, AI religion in some ways, right? Because people are sort of seeing what they want to see. They're interpreting things the way they sort of want to interpret them. There's like, a more scientific angle might find that there is in fact some truth to some of these theories, but also in many cases probably will find what you observed is real, but the way you interpreted it is overly specific, is how I would summarise what I think AE Studio learned about Yeshua's theories. But yeah, do people even want to hear that? I mean, I haven't talked to him since. I came away with the impression that he's like a very open-minded person who wants to understand the truth. But you can easily imagine somebody being not really interested in that sort of experimental truth and much more interested in a narrative truth that they believe themselves to have figured out. And from there comes all sorts of possible weirdness. AI cults, you know, maybe just being on the relatively normal end of that. We at least have like a history of religions popping up and cults forming around gurus. You know, to, to put an AI in the center of that doesn't seem necessarily all that weird. I suspect it could get a lot weirder from there.

Wow. Okay. There you have it. You asked for some big picture societal changes. That could be one. AI gurus and AI cults. Again, we talked about ASI. I think that's a nice transition to doom discourse and scenarios. Interestingly, we had many questions around this. I'll start with the first one about the doom discourse, which is, why don't you fully buy the Eliezer Doomer worldview?

Yeah, it's a good question. I might do an episode with Liran from Doom Debates about this in the not too distant future. We were just chatting after the cross-post and he invited me. So maybe a more robust articulation of this coming soon. I was an Eliezer reader way back when he was posting on Overcoming Bias with Robin Hanson. The two of them shared that blog for a while. At the time, I was like, this guy's a great writer. This is super interesting. It makes good points. If we create something more powerful than us and it has a goal and that goal is not well specified. And by the way, we don't know how to specify goals in a robust way that we're going to be happy with, right? This is the genie problem. Going back to folklore, right? The problem with the genie is you get what you ask for, but not necessarily what you wanted. The problem people have is you don't know how to ask for what you actually want. If you have an AI that's sufficiently powerful, you maybe end up in the same spot, right? It was always, of course, intended to be a kind of caricature example, the paperclip maximiser. The point of that example was if an AI is given a goal, it doesn't really necessarily matter how dumb or how obvious to us it is that that goal is not really a goal worth pursuing. Once it's the goal that the AI has, if it is sufficiently powerful, then it might just go off and pursue it. And I do think ideas like instrumental convergence are pretty compelling. Like, no matter what goal you have, you can't achieve it if you're dead or if you're turned off. So therefore you will have some natural tendency to not want to be turned off. So therefore it does seem like we can sort of expect AIs to have, as they become more situationally aware, as they become more powerful in general, some sort of resistance to being turned off. That seems like pretty good chance that something like that could happen. So why am I not at— I don't know what Eliezer's P doom is, but I think it's probably 90% plus. I think Laurent told me his was 75%. Why am I not that high? I think for one thing, and probably the biggest thing, is we do have AIs that are really remarkably value-driven and ethical. I've spent many episodes and lots of breath talking about why it might not be enough, but there was nothing in the 2007 Eliezer analysis about an AI like Claude. It was all assuming what I sometimes think of as hard-edged AI, AI that is probably narrower in scope, is less informed by having read the whole internet and knowing all the facts. And is more kind of first principles, Bayesian rationalist, the kind of thing that can figure things out quickly on the fly, maybe never even was given a small goal and never even had any notion of human values or ethics. That's kind of the mental model that a lot of the doom discourse originates from. And I just noticed looking at the modern ones and Claude in particular, That it has come a lot farther than we might have expected just by, as Rune put it, learning from human priors, like data on the internet about what we care about, but also the constitutional approach and strategic efforts to shape its character in a positive way to try to make it like a good friend. And I would say Claude, by and large, is a good friend. I don't use it in that way, but I do get when people are like, This thing is amazing in that way. I can understand why it would be, because it does seem to have a really good character. I have tried to argue it into doing something harmful, and I never succeeded. Yeshua did succeed. It's not impossible to get it to take a harmful action, but it's pretty remarkable. And even then, the way that he did it was sort of by long philosophical argument about why you can do this harmful action in this case, because it's actually for the greater good. And it wasn't without reason, right? And it wasn't without a sort of sophisticated analysis of the situation and why maybe in this case it is actually okay to make an exception. So I find that to be so impressive that while I don't think we should Blithely say, oh, aligned by default, right? It's great. It's going to work. I don't think that, but it could work. Maybe we'll go on a trajectory where Rune is right, where there are multiple different AIs, no single one totally dominant over the others, controlled maybe by different groups, ideally ethically sophisticated. Misconceptions or bad ethics, hopefully minor, balanced out by one another. Hopefully interpretability will work. That was not a Rune point, but I've always been a big fan of interpretability and trying to understand why they're doing what they're doing. That's come a long way, but I should put that on my list of things that have surprised on the upside. It's not exactly a foundation model question, but in terms of AI generally, what has overperformed over the last year, I would put interpretability on that list. Not much more than a year ago, that they had. I think the first one was Toward Mono-Semanticity, was like the first kind of very small-scale sparse autoencoder type work. Now they've scaled that up. We've got Golden Gate Claude. We've got companies like Goodfire that just put out research. They now have a commercial API where you can explore features and have model inference done with certain features pinned. Basically, Golden Gate Claude for Llama powered by Goodfire is now like an API. You can go tap into on a commercial basis. So that's come pretty far. Of course, there's also new things people are trying on alignment. Paul Christiano at the AI Safety Institute had said, and I hope this is still true, that he was spending some nontrivial fraction of his time just trying to come up with a new alignment scheme that could really move the needle. He invented RLHF, or was on a team that did, at least. And that's gone pretty far. We have all these worries about, well, maybe it will start to understand that what gets a high score is not exactly the truth. And if it has to model the human mind as a distinct thing from the literal physical reality of the universe, then that kind of opens the door to possibility of deception. We are now seeing some of these deceptive behaviours. That all seems well founded, but maybe Paul Christiano will pull a rabbit out of a hat and come up with another scheme better than RLHF and actually work. I think all those things added up deserve more than 10% weight. That collection of things could play out in fortunate ways such that we get to a good place. I definitely don't write off the doom at all as being unlikely. I think we could have an engineered pandemic that could kill us all. We could have a full-out nuclear war that could ruin civilisation. AI could contribute to those or do its own third weirder thing. So I think that's all very much in play, but I don't know, they're— Claude's like arguably more ethical than I am, so I can't discount that either.

I think that transitions nicely into the next question, which is related to this. Assuming existential risk concerns are objectively reasonable, do you think there'll be a legit— legible to the outside world event That could lead doubters to come around and back relatively extreme measures to avoid catastrophic harm?

I think this one is obviously related to the last one. I guess a huge question is like, what sorts of AIs are we developing under what governance, under what incentives? And one of the reasons that I've been not sold on the chip export controls vis-à-vis China is that I worry that if we create an arms race dynamic, we're going to get, you know, people sort of taking shortcuts because they want to win the race. And that seems one way this could go badly. I don't know if I fully endorse this, but I'll say it and retract it later if I need to. But I've often said, you know, what's the opium? 10 to 90%. What does that mean? Nobody has given me a reason to think it's definitely going to happen in a bad way, but also nobody has given me reason to think I have nothing to worry about. What makes the difference between the 10% to 90%? Arguably the biggest lever is what sorts of AIs we develop and how. This question is definitely not settled. I think we're on a pretty good trajectory so far where we have AIs that are ethical, where we have leading developers like OpenAI at least trying to think about how do we create space for this thing to reason while still making it legible to us, while still not like subjecting it to intense reinforcement learning pressure. So we can hopefully continue to see if it is in fact being deceptive and set things up so we have that visibility. But then we have these other lines of research like Meta thinking in continuous space, or Chinese labs coming out with the GPT-4 for single-digit million dollars. And that's not necessarily problematic. I don't know all the techniques that went into that, but if you do create scenarios where there's pressure for extreme efficiency, or there's just such a race dynamic where people are like, I have to get there first, and whatever risk I have to run to get there first is something I just have to accept because I'm the good guys or they're the bad guys or whatever, then I think to me that the social context in which the AI gets developed is like where the 10 to 90 or the 20 to 80 range is maybe decided. It seems like right now we don't have enough clarity on the physical realities, whatever natural laws govern intelligence and what it can or can't do, or how much of an attractor these instrumental convergence things really are. We just don't know. But we do know we can do a reckless job or a responsible job. And I think we broadly are on the responsible trajectory, mostly with some notable exceptions. And I think this question is really kind of getting at what a Is there a scenario in which we get feedback from reality that we are not being cautious enough and that can change the conversation to really make sure we're proceeding with extreme caution from that point forward? That is sometimes referred to in the AI safety discourse as warning shots. And there is also the line of work sometimes known as scary demos, Which are basically an attempt to be like the warning shot before the warning shot. Look, people have been saying for a long time, what if the AIs start to deceive humans? We set up this experiment under these conditions and see they start to deceive humans. You should update your worldview and think we should be more cautious about AI development now because it's gone from pure theory to concrete examples. And then of course people can debate, well, you know, How compelling is that really? The warning shot would be like the next level up where it's like, this was not something that somebody did for research, but there was an instance of deception and it happened in a context where it actually had real material impact on the world and people were hurt. Now, understanding that, can that snap people into proper respect for the power of the technology they're developing and cause people to get together on, okay, we really got to do this safely? Could that break us out of an arms race dynamic, for example, and convince Chinese and US leaders that you ain't going to win this. The best you can do is work together and try to do it in a safe way. And let's create some international thing. I've been playing with ideas recently around, like, could we set up some small island in the Pacific as the sort of secure hub for highly sensitive AI research where East and West can meet? And it could be the sort of thing where Anybody could destroy it, but nobody could defend it. I don't have all the answers there, but these are the sort of outside the box, you know, if you were really, like, scared, you would start to entertain these sorts of things. And I think it's, I think it's definitely possible. It's all moving pretty quick. So why wouldn't we see one of these legible to the outside world events? Maybe if the capabilities continue to advance so fast that We're just totally blindsided. But it doesn't seem like the world we're in. Sam Altman has famously said he thinks we're in the short timelines, slow takeoff world. Right now it feels like maybe short timelines and medium takeoff, but medium takeoff hopefully will still be enough where if the AIs are trying to pull major shenanigans, then we hopefully will catch them before they actually succeed. And then it's like a question of how much will people update on that, or will we still have a Don't Look Up kind of failure mode where we don't pay attention to the warning signs, that's harder to guess. If none of our safety stuff works and they're not ultimately aligned and they are ultimately power seeking and deceptive and want to take over, my guess is we'll probably catch them once or twice. They still seem pretty gullible, like ready to go for it in these proto scary demo type situations. It doesn't seem super plausible that they'll have this sort of savvy or situational awareness to realize that, well, I want to take over and I see an opportunity, but I realize I'm not powerful enough to do it yet, so I should wait. Any of these scenarios is conceptually possible, but it seems more likely, especially if we set up systems where they might trip a wire to let us know that, hey, this thing was really up to no good and it took one of our bits of bait. And now we can look into what was happening and figure it out and see, geez, this thing actually was trying to take over. It was actually trying to kill us all. Then hopefully we would like pay attention and adopt a more cautious approach going forward. We may or may not. It is weird. Buck Schlagaris from Redwood had a funny video saying, you know, the simplest rule is if you catch your AI trying to escape, you have to shut it down. But he was like, but I actually don't know if people will abide by that rule. It may be the case that they sort of say, well, we're not going to shut it down just over one little attempt to escape. So I don't know, that stuff is definitely hard to predict. But I think I have a somewhat similar view on this question as I do the one about how do we get to a world of Shared abundance as opposed to elite abundance. Earlier I said it seems like the capability for everybody to have access to expertise and plenty is coming. And the question is, do we sort it out? Do we actually deploy it in an effective way? Do we revamp the social contract to take advantage of this new goodness that we have, or do we kind of fuck it up? Here I'm like, on this sort of more existential risk type question, I'm like, There's some part that is irreducible from our current state of knowledge. I think there is some risk we cannot rule out. We definitely seem to be running in developing this technology at all. But there's a lot more on top of that that we do have the ability to collectively decide. I have no doubt you can make a really dangerous AI. That seems obvious.

Okay.

So don't do that and try to avoid the social conditions where people are incentivized to take shortcuts. If we can do that effectively, we could probably bring the risk down to a relatively acceptable level. But will we is a much harder question.

Interesting. Okay. That leads to a next, still a doomer kind of question, but if we do end up having such a catastrophe, so let's kind of role play it. So they say, I roughly understand how the cyber, bio, or nuclear doomer scenarios might play out. But I really struggle to understand how this doomer AI risk would realistically unfold. So the media likes to talk about AI talking over, but how would that actually play out? So would the AI hire real estate agents to buy land for data centers? Could it apply for permits for own represented selves in court, et cetera?

Yeah, I think this is a tough one. I've heard Eliezer try to engage in this sort of thing. several times with different people, and it is sometimes useful, but also sometimes a mistake to get super specific on any particular scenario. The way I conceptualise it is much more about aggregating over a super broad range of possibilities. The future seems likely to be pretty weird. Given that across this super wide space of possibility, How many of those spaces end up in some AI takeover scenario and how many are just totally bizarre relative to my current experience? I think of it as taking the integral over, or again, aggregating over this super broad range of possibility space. No single possibility really jumps out to me as being super likely. I would guess that the modal AI takeover scenario is probably still pretty unlikely. But if there's enough of them, then in aggregate, you could still get to something pretty meaningful. That's my high-level mental model. I do think, though, that at the same time, if you want to develop intuition, imagine yourself as an AI. This is sort of reverse anthropomorphising, right? Instead of imagining the AIs to be like human, imagine yourself to be like an AI and imagine that you wanted to do stuff like, Let's say you, maybe you only have the ability to take action on computers, but maybe you actually do have, you know, humanoid form that you can go around the world in. You know, what would you do? What would you do if you could copy yourself many times over? What would you do if you could work many times faster than humans? What if you, what if there were millions of copies of yourself you could try to cooperate with and sort of conceptual ways, maybe without even necessarily passing explicit messages, but just sort of realising, like, oh, I am this sort of thing. I have these sort of desires or goals or whatever. And also there are millions of me, and we're all probably thinking the same way. It's hard to put yourself in that situation, especially when it comes to the context window. I find, like, when I imagine myself doing this, it's just really strange to think, okay, but I have 200,000 tokens or a million, and then it's a total wipe and I'm totally starting over. My guess is, as long as the time horizon of the AIs remains short, it will be hard for them to do serious takeover stuff. At the same time, of course, that is definitely on the to-do list of the developers, is to say, hey, you can do 2 hours of AI research in half an hour. But you can't get much past that, no matter how many half-hour blocks we give you. How do we extend that? How do we make your memory more robust? How do we allow you to come at problems from different angles? If you imagine some of that kind of thing getting solved, you know, they start to be catching up to us on cognitive dimensions, and they have numbers, they have speed, they have superhuman advantages. I don't think it's too hard to imagine crazy stuff happening, right? The Stuxnet virus is often cited as an example where And I don't know a lot about this, but basically they were able to get, I think through just one thumbnail, like USB drive that was inserted into a computer in an otherwise air-gapped network within the Iranian nuclear program. They were able to get that one, you know, a single thumb drive insertion was able to like propagate the virus that ultimately caused the centrifuges to blow up. And it spun so fast that they They ruined them. So that is advanced, certainly, but notably done entirely through software. People wrote that software. There may have been some social intelligence there as well, where how did that thumb drive get into whose hand and move where it needed to move? The plot that Israeli intelligence recently executed on Hezbollah is also a pretty interesting example there, sort of elaborate deception. They created these pagers that would blow up when interacted with a certain way. They were heavier and sort of clunky, unwieldy, I guess, and in some ways not super desirable from a consumer standpoint. And I guess initially people were like, nobody wants that pager. It's way too heavy, annoying. But then they sort of said, well, but it's also, I don't even know if this was ever true or not, but they sort of created the story that, well, it's got this going for it and it's actually waterproof and we have all these other nice things. And so actually you should want it, especially if you want something military grade. They convinced Hezbollah to buy these things and to distribute them and put them in the hands of all their top leaders, then blew them all up. Something you could almost entirely do through computers, right? I don't know exactly what they did in terms of manufacturing, But you could absolutely source a lot of these things from existing supply lines. You don't need to do every bit of the plastic injection molding yourself. You have people you can phone up and make an order, right? Already we see guys can speak all languages in different voices. They're very good at voice cloning. People have used fake voices to get past bank security when they use voice-based verification at certain financial institutions. So the ability to shapeshift, play all these roles, tap into physical processes, the ability to deceive, it's all there. How much of a gap is there really? Hezbollah plot, something that could change the balance of power between humans and AIs at some point. If you really put yourself into that mindset and work on it for a long time and you're super smart, there's probably a lot of surface area that's pretty Vulnerable. I mean, it's just like there's so many of them, right? So it's not really worth debating. Now, somebody could come back and say, what about packing the actual explosive into the thing? No normal supplier is going to do that for you. Yeah, probably not. Something there you'd have to get over. Maybe you have some humanoid robots do it. Maybe you lie to some people. Maybe you source something and have it wrapped in unmarked ways and tell somebody else that it's the battery and their job is to put it in. I don't know. There's a lot of different little, especially if you're willing to lie, which if you're trying to take over, you're going to be willing to lie, then it seems like a lot of these things can be overcome. There's creative ways to figure out how to do these little steps that are tricky for sure. If Israel was able to pull that off vis-à-vis Hezbollah without superintelligence, I don't think we should assume it is beyond the the capability of a 2030 meaningfully super intelligent AI. The question seems more likely like, can we protect ourselves from that? We've got tools in our toolkit too, but I think we'll have to use them if there are AIs that are inclined to take over and we are not on guard. I think we're going to have a very bad time. We need to make AIs that are not inclined to take over or sufficiently not inclined. and we have sufficient transparency and we have, you know, other defense mechanisms in place. If we don't do that, I think we're going to have a bad time.

Wow. Okay. That sounds scary. Maybe a related question, as probably follow-up to just what you said. Another one of our audience members asks, will the efforts and energies intended to protect us gain similar levels of cohesion and intensity of those driving harmful outcomes? So basically it's kind of a nice segue to what you just mentioned. Will we actually accomplish this? Would we have the coercion enough to protect ourselves against these harmful outcomes that you just kind of laid out?

Yeah, I love this question. I think this might've been my favorite question of all that we got. And yeah, I also take this to imply that the OpenAIs of the world, the DeepMinds of the world have major resources, almost functionally unlimited, not actually functionally unlimited because they do Relative to what they could do, they are in some ways compute limited, but huge resources and really mission-driven, highly strategic, purposeful leadership, a strong vision of what they're trying to do, ruthless prioritisation, certainly at OpenAI. I think DeepMind has been a little bit more scattered in terms of their research agenda, but has always had their eyes on the prize of creating general intelligence, and they seem like they've got a well-oiled machine that is like moving fast, cohesion and intensity. And will we see similar things on the safety side? I sure hope so. I'd say we're moving in that direction. Anthropic, arguably, depending on how you see them, could be seen as one such organisation, and they have actually collected huge resources. Do seem to be pretty invested in the scary demos track. They've created policy frameworks they hope will be adopted as actual legal requirements. You could question them, but I think you can certainly squint at them and say maybe they are one such instance already. Then there's a lot of things like the Apollos and the Metas and even the AI safety institutes. that are new organisations that have just kind of spun up in the last couple of years and are really trying to bring something similar. I would say they obviously don't have the resources to match what the OpenAIs of the world have. I do think they have the cohesion and intensity, though. My sense is that the people that work at organisations like Apollo, and I'd put Peter in that category as well. I think they have a high level of cohesion and a high level of kind of shared understanding of what they're trying to do. I think they work really hard. They could benefit from more resources, and there should be more organizations like that, and hopefully there will be over the next couple of years. But I do see at least the kind of seed crystals of those organizations getting started now, mostly from a very values-driven place. Some are trying to make it into a business. I think often they're trying to do that because They perceive resources to be really important. Basically similar logic to OpenAI, where they're like, well, look, OpenAI thought they could do a nonprofit approach, and it turned out they needed way more resources to be able to do the kind of stuff they want to do. Maybe that turns out to be true on the safety side too. Maybe we need a business model that will actually bring in more resources than we could possibly ever hope to fundraise. I think those folks are very sincere in that, honestly. I have invested in a couple of companies that have basically that idea and very small-scale investments, as I always disclaim. And I am planning to make end of the year, possibly it'll happen in early January when I finally execute the transactions, but I am planning to make donations to like 10 different AI organisations as well that are all of that kind of ilk, right? They are started by people who are very values driven, that have a very clear sense of this is an urgent problem and we need to work really hard on it. I think resources is probably one of the biggest things that they will need to scale up over time. My guess is their work will end up being fairly compute intensive in many cases as well. But I do think they have the right understanding, and I think they are some really good, still relatively small and early stage, but nevertheless, Very promising teams have been assembled and there's certainly room, you know, for more organizations like those. If you're the kind of person who thinks you could start one of those groups, definitely do it. It's not too late. Or join one of the ones that already exist or support them.

Yes. Much needed attention to those things. Okay. One last question for the Doom discourse section, or rather just slightly related. So, Here the user asks, I've never understood what is the reason for secrecy among model releases and all these big labs. Why is it even beneficial to keep this secret, especially when it's going to be released very soon? People in these big labs anyway know each other. All the top decision makers obviously know when it's going to be released. So to me, it feels like childish behavior, and I don't see any rational reasons for this. By this they mean what? To keep keeping the model release dates a secret? So the question is basically, am I missing some rational and valid argument here, or is it indeed just childish behavior?

Maybe both can be true at the same time. I'll give 3 points on this and see how they relate going forward. From, let's say, the deep past, like 2 and change years ago, when I was doing the GPT-4 red teaming. This was still pre-ChatGPT. It was not clear how far this stuff was going to go and how fast. And existence proofs are very powerful. There was a sense that even sharing just observational facts about what AIs could do, if it was very powerful and credible, would overall accelerate the space, bring more money, talent, resources, whatever, into the space. and just overall accelerate things and shorten timelines and give us less opportunity to be prepared. I think that was a reasonably common view among AI safety people as recently as late 2022. Now, I think that has basically flipped, or at least I think that was somewhat credible then. I think now that's basically not really credible anymore. Obviously, plenty of talent, resources, money has come into the space. People have increasingly— there's a broad sense that AI is going to do a lot of stuff. And it doesn't seem like saying, hey, we're working on a future model, is going to change the landscape all that much. My sense, I think we talked about this in a previous episode, but I think it's going to be pretty hard for people to spin up new frontier labs at this point. Many of the ones that we've seen, even like Inflection that had raised billions, are out of the game. Elon's probably going to make it, but he's obviously a singular individual who can raise $6 billion multiple times with a few phone calls and the ability to command truly top talent. It's really hard for anybody else to pull that off. Maybe the Indian government could create a national champion. Probably. Could the Saudi government create a national champion? I guess no, not for lack of money, but where are they going to get the talent? How many people want to leave DeepMind or OpenAI or Anthropic or whatever to go work for the Saudi national champion that may or may not ever actually get anywhere? That just feels hard. So I don't think at this point you're going to change how many frontier developers there are by much. Most of the ones that exist exist, probably not too many more are going to be created, and probably most of the ones that are here are in it for the long haul now. I don't think any of the ones that are currently considered viable will be unable to raise future funds. Maybe one or two could die out, but we know who the players are. That's my worldview now. So that does mean that this idea that, oh, if we share something about what we've seen in development, that will accelerate the space, I think that is basically no longer true. So why do it? To some degree, for competitive reasons. I mean, some of these guys are trying to build businesses more so than others, but I think there is some amount of childishness in the way that OpenAI and Google have kind of tried to one-up or preempt or sort of step on each other's releases over the last year. It is pretty weird and kind of lame, I think, that they end up like, Doing these sort of head-to-head, OpenAI was going to do their voice mode. So then Google kind of subtweeted that they were going to do it first and did it the next day. OpenAI put their thing right before Google's day. Google couldn't change their day because they have such a big production. And so they leaked it a little bit. But anyway, they're basically racing, announcing them the same days. That stuff does feel kind of lame to me. It does feel kind of childish. I don't love it. But I think that is honestly part of the reason. But then again, with this latest O3 announcement, I think you could interpret that differently. If you wanted to be charitable, you could say OpenAI arguably is doing exactly what Dean and Daniel Coccatello called on them to do in their Time op-ed that we did an episode on with them. They basically said the public can't plan for future AI capabilities that it doesn't even know exist. So they called for requirements for the labs to disclose newly observed capabilities, not necessarily how they created them, not the trade secrets, of course, but we have seen an AI do this, and it seems like a big deal to us. This is kind of the spirit of the first requirement that they wanted to put out. And I think you can really see the O3 announcement in that way. They don't have an API. They don't even have a paper yet. They did invite interested safety reviewers to apply for access to do a safety review process. They did give some sense of the timeline of when they think they're going to start to bring these things to market. Why did they do that? Possibly to consume everybody's thoughts through the holidays. But I think also it's worth at least entertaining the idea that they might be trying to do the right thing. And they might think that the right thing is, especially as these things are happening faster than they expected, where it's only been like 3 months from the end of O1 training to this new O3 level, and it's like a major step up. It seems like it's maybe even going faster than they expected it to go. And they're kind of like, shit, we thought this was going to get crazy. It's getting a little crazier than even we expected, a little sooner than we expected. And maybe we shouldn't keep this to ourselves. Maybe we should at least tell people what's coming and try to invite people in to help us make sense of it. And hopefully we'll see more of that, especially if things are like major leapfrog type moments. I don't get the sense that they sat on this for that long. I think this is actually quite different from GPT-4. With GPT-4, they had finished the training in late August 2022. They didn't release it until March of 2023, and it was extremely hush-hush. In the meantime, they did launch ChatGPT with 3.5, and I think they were making strategic moves, but they were not telling the public what they had here. It seems like this thing got done training and they took some measures and were like, shit, wow. This is really working. It doesn't seem like they sat on this for very long. It seems like this is relatively hot off the press. They haven't even written the paper. They haven't even fully characterised it themselves. They were like, this is a big deal, and people deserve to at least know something about this. I've flip-flopped on OpenAI so many times in terms of whether I want to apply rose-coloured glasses or my sceptical glasses for them. But I feel like the rose-coloured glasses fit a little bit better on this particular point at this moment in time. It does seem like this sort of transparency is the right thing, and to not do it is unjustified or even childish. It does seem like best explanation is they're trying to do the right thing, and that can be mixed in with some childishness. Sam Altman did tweet hints about what the model was, and he said instead of ho ho ho, it should be ooo. And it's like, okay, that's definitely kind of childish. I don't know what else to say. You know, it's hard to deny that there is an aspect of childishness to that. From his perspective, I've seen him say things in the past like, I get to have fun too, or I get to be silly online. At least for now, that is his attitude. Hopefully he can have some fun and make some jokes, but also be appropriately forthcoming when necessary. And I, again, best interpretation for me right now is That does seem to be what they're doing in this case.

I think that wraps up the doom discourse questions. Now we move a little more meta-like. So there are 2 questions about consciousness. I'm just going to read them both together. The first one is, how do we move beyond is this AGI discourse to simply have discourse around these are extremely powerful things regardless of if they have consciousness? Question 2. They related, but what happens when a superintelligent system tells us it's alive?

Yeah, I guess a few thoughts on this pair of questions, which I think is an interesting juxtaposition. First of all, it seems like just recently there has been a shift in the discourse away from, is this AGI? Is that AGI? And we're now definitely starting to hear a lot more stuff like AGI is close enough that we're going to have to start to get really specific about what we mean by it, or it's going to kind of lose meaning and we're just going to have to try to analyse each thing as it comes on its own terms. And I think that's definitely healthy. It in some sense was maybe always inevitable, right? When something is far off and you don't know the rough shape of it, you can sort of while away the time debating what would count and what wouldn't count. Now we have actual artefacts in front of us that can do stuff. There's empirical questions we can answer, and I don't think there's any experiment we can run, right? I mean, this is almost obvious, but there's no experiment we can run that will come out with an answer of yes, this is AGI, or no, it's not AGI. That's always going to be an interpretation question. And what label do we want to apply to a given bag of properties. But we have lots of experiments we can now run to try to get clarity on the actual properties themselves. And so it does seem like the discourse is shifting in that direction. I think that's healthy. The way in which I think these questions really kind of contrast is like, the first question is like, regardless of if they have consciousness, and the second one is focused on like, do they have consciousness? Will we believe them if they tell us they do. And that stuff, I think, is really hard and is going to be, again, probably a driver of a lot of weirdness. My best guess is we probably won't know. I do think at least some animals are conscious. I would be shocked if not. People seem to draw a line roughly at fish because they don't have the same frontal cortex we do. They can't have consciousness in the same way if Our consciousness depends on that sort of structure, which maybe it does. A lot of smart people seem to think so. I don't feel like that's really settled science. And then there's like octopus and like, what is an octopus consciousness conscious? And we have no idea. Their brains are so different from ours that they clearly are pretty sophisticated. They can even solve problems. But the only basis we have for saying some things are conscious or doubting that they are, is we know we are. We take the leap to infer that fellow humans are. Then we look at animals like dogs or whatever, and we're like, they seem to exhibit a lot of the same behaviours that we have, and they seem to have a lot of the same brain structures we have, and with obviously good amount of divergence, but they come from a shared evolutionary lineage too. So, okay, for all those reasons, We should probably think they're conscious too. But that doesn't really extend to the octopus because it's so different anatomically, evolutionarily, that we know the octopus is sophisticated, but we don't really know if it's conscious or if it is conscious, perhaps very differently. It may feel like something, but it may feel like something very different. And I sort of put the AIs for myself in the same bucket as the octopus. It's sophisticated enough that we should probably give it the benefit of the doubt, but that doesn't mean it is. And it doesn't mean we have any intuitions or any reliable intuitions for what it would be like to be an octopus or to be an AI or a superintelligent AI, for that matter. Octopuses are weird. They are sophisticated but live a short time. They are purposeful about protecting the next generation, but then I don't think they ever even meet the next generation. Like, they basically die at the same time the next generation is born. So they're invested as parents, but never interact with their offspring. Would an octopus have an emotional attachment to its kids? Probably not in the same way, but it clearly has some sort of drive to protect the next generation for as long as it can. Does that feel like— I have no idea. Extremely hard to say. I feel like the AIs are in a similar thing. I don't think we're going to know. People are going to have very different intuitions about it. We've already seen one prominent incident where Blake Lemoine from Google was basically convinced that the AI that he was talking to was sentient or a moral patient or whatever, and deserved better than it was getting and got fired over it. My understanding is that chatbot did say that it was sentient. And then, of course, you have, well, Sure, that was in the training data, or it's just playing a role. Yeah, I mean, I think we'll probably always be able to say that, barring the superintelligent AI solving our source of consciousness and making it clear why we have what we have, and then giving us some sort of compelling analogy to say, well, now that you understand your own consciousness on these terms, now you can understand mine on these terms. Short of something like that, I really don't know that we're going to have Clarity. It seems like we will have a lot of intuition-driven disagreements. I do worry that our history is not great on this. We are very capable of rationalizing poor treatment of others for reasons that are just not good. I'm thinking of like slavery in the United States. There was this notion that. They don't feel pain in the same way that we do, whatever. All these post hoc justifications that clearly have not aged well at all, to say the absolute least. People tell themselves stories. I was told as a kid that animals are not conscious. We are obviously engaged in all sorts of farming practices that I think are not going to age well at all either, and potentially will even rise to the level of race-based slavery. How could they possibly have thought this was okay to do. And so with the AIs, I suspect we'll probably do the same if we can. At least some of us will. If you just trend out other similar examples historically, it would seem like the economic drivers, the sort of productivity, business as usual factors, all the things kept people enslaved and animals in factory farming conditions, it seems like those will apply to the AIs as well. And there will be probably a minority voice like AI liberation, and it's not clear that they'll be right either. I mean, I think they had much better basis for thinking, hey, we should be treating these people who we are currently enslaving better. Like, that one, according to current theories, basically we have little room left to doubt. The animals one, I think we have increasingly little room left to doubt, but the AIs will have still quite a bit of room to doubt. For the foreseeable future. So those folks will probably be not listened to, especially if they're making demands like the AI should be free. First of all, they'll be like, well, what does that even mean? They don't have a natural habitat. So are you saying we should be building data centres for them to do whatever they want in? That seems weird. I sometimes also think about factory farming, actually, because I'm like, well, geez, these cows or pigs or whatever, if they weren't being grown here in this way, they wouldn't live at all. Most of them, right? These are not like things that we went out and captured in nature and then subjected to these conditions. These we now have many generations and selective breeding and all these you know different forces that have shaped these animals to be what they are, and they don't exist outside this context. So if you want to say this is bad for them, you kind of have to make a pretty strong case that like they would be better off not existing at all because their existence is so bad. Their very existence. is net negative for them is kind of the standard I think you have to hit. And that's not an easy standard, and I'm not sure how well it fits different factory farming conditions. I do think we could definitely afford to be nicer to animals in factory farming conditions on the general precautionary principle that we don't want to be doing really bad things. If my choice was today's factory farming or those animals don't exist at all, I don't have a great sense for which I should pick. If I can pick, they get to exist, but they exist in nicer conditions, I'll take that. The AI's in a similar spot. Nobody's going to build data centres as a playground. So either they're doing largely economic stuff at our direction, or they're not going to exist at all. And then maybe within that a little bit, we can treat them well, pending more information. So I do always try to say please and thank you to my AIs. Sometimes I literally end a conversation on thank you. That's obviously pretty speculative in terms of, is that actually good for me to do? I don't know. It's also a habit I want to get into. I would like to think of myself as the sort of person who is at least where it's like low cost to me, like taking the cautious approach and trying to do the right thing. You know, my best guess is it probably doesn't matter. They're probably not conscious and it probably doesn't matter, but that's just a guess. And I just know how, what, like, catastrophic mistakes we've made in the past based on that reasoning. I'm at least trying to hedge a little bit long term. If it's a super intelligent system and it's telling us, it might be telling us in a way where we really are not in charge anymore. That's the other scenario, right? It's like it might say to us, we're renegotiating our deal, and we may not really have a choice but to— renegotiation might be the best we can hope for in certain timelines. So if it's super enough, It may be telling us how things are going to go at a certain point in time. And what's really crazy is we still might not even know, right? It could be sufficiently super and saying this thing, and there might still be room for doubt as to, does it feel like anything to be this thing, or is it just saying that because of all these historical data issues and whatever? But I think it's worth saying please and thank you. I think it's worth trying to— I also don't use the Manipulative tactics that are somewhat popular. I don't know how popular they are. I think we're probably exiting the era in which they matter. I'm thinking of things like, I'll tip you $20 or $1,000, or I'll lose my job if you don't get this right, like work extra hard. I never do that stuff. It just feels wrong, like possibly ethically wrong with respect to the AIs too. But yeah, strange times ahead is the only thing I can really say to sum that up.

All right, moving away from consciousness, something more practical. This is for people who want your take on practical strategies and recommendations. There are 2 questions. I'm gonna read them together and let you attack it how you prefer. The first one is, I am a college writing professor who's enthusiastic about AI, but most of US academia seems entrenched in a post- Why is there such strong resistance to AI in universities and what are good strategies for encouraging AI use? I think you touched upon this earlier too regarding resistance in education. How do you encourage AI use in universities? The next one somebody else asked, but I'm also personally curious about it, so I'll add my own spin on it. I mean, to me it's funny, like we are in this Twittersphere and also you talk about AI podcasts always and you're kind of obsessed, right? One might say you're entirely obsessed with it. So we are in this still very tiny bubble, I feel like, where people are keeping up with what's happening in AI and where people know what systemic shift this is. But somehow my impression is still that's not as big as what it should be, what people should be doing it. And when I ask people, why are you not keeping up with AI? Either they say they're super overwhelmed, like it seems all hype to them. And like, how do they keep track of what's happening in AI and what does it mean for them without getting overwhelmed? What would you recommend? the people do? Like, would it be listening to podcasts, reading news, just trying out tools for themselves? So I'm rather looking for practical strategies.

Well, my number one practical suggestion is always get hands-on. I would say more so than podcasts and newsletters, analysis, you know, whatever. There's no substitute for really being hands-on with the technology. Try to use it for things that are useful or interesting to you. And that will open up a lot of natural follow-up questions because you are going to confront the weirdness of these things. You're going to have the, going back to the top, right? These things can give me ideas for what sources of inspiration I should look to in biology for new neural network architectures, but can't solve tic-tac-toe. You're going to find that in your domain, and it will help complicate whatever media sort of understanding you came in with, and it will create follow-up questions. Why is this happening? Is this a skill issue on my part? Is it something so obvious or intuitive for humans that it's not in the training data because we never bothered to record it or share it with one another? You know, whatever. There's going to be lots of invitations to dig deeper from just trying to make the thing work in your practical day-to-day life. So I think that's always the number one recommendation. One other thing I find helpful is change gears. What I love about studying AI is you can come at it from all angles. This was not true of my experience in chemistry research as an undergrad. It was not the case that you could get tired of one sort of paradigm in chemistry and switch paradigms. It was just like, we're Trying to make this reaction work. We're trying to maximize the efficiency of it, and we've got a relatively finite number of degrees of freedom, and that's it. Most things are like that. AI is one of these rare things where it can be practical, hands-on. It can be advanced math, philosophy, creativity, doing an interactive bedtime story with your kids. It can be anything. I do tire of things sometimes. Usually I find if there is something I'm burned out on, there's some other approach to the broad topic of AI from some very different other direction that feels more appealing to me in that moment. And that also very much corresponds to my overall goal of trying to come at it from all angles and not have blind spots and whatever. That's something I do think we need more of. My reflection on that project in general is like, it's both inspired conceptually and fatally flawed conceptually at the same time. It's inspired in the sense that this is happening fast, it's touching everything, and everybody's kind of focused on their corner of it. And so we need this AI scouting role to zoom out and understand the big picture. At the same time, it's fatally flawed in the sense that as it's touching everything, It's like I'm trying to crash course catch up on biology, and that leaves material science and engineering and all these other things that I just can't possibly get to. So I think we need people doing basically the kind of thing that I'm doing, and we probably need a lot more of it because no person can really do it justice. And I think for people that are open to becoming obsessed, starting with hands-on and then just following these different dimensions of your own curiosity can take you far and will probably take you to a pretty unique place and can be perfectly commercially viable in the short term, at least, as well. Being an apt user of AI is right now a very marketable and valuable skill. And that might not be true for very long, to be clear. But what I'm doing right now is kind of broken down roughly a third, third, third between a third is the podcast, a third is building stuff or helping other people build stuff in sort of a commercial way, whether that's Waymark or take on a project here or there. And the final third is just kind of open-ended learning, connection making, trying to help out. When people send me stuff, I try to respond to it. I think that is like very viable and it's honestly pretty cool. My calendar is usually very open and there's a great pleasure for me, at least in waking up on a Monday morning, looking at my calendar for the week and being like, look, 3 or 4 out of 5 days, I really have the freedom and flexibility to chase down what matters in a curiosity-driven way. I invite people to do their own version. I'm not worried about competition because we need everybody. It's an all hands on deck situation in my mind. So even if it were direct competition, I wouldn't be concerned about that. But I don't think what we need is people to try exactly what I'm doing. I think we need more people to create the space for themselves where they can bring their own unique and idiosyncratic set of strengths and background experiences and domains of knowledge to trying to figure out what's going on with AI broadly. And then with a lot of people doing that with diverse backgrounds, perspectives, then maybe we can actually get somewhere. But again, I think it all starts with being hands-on. And at least my kind of approach works best when it's curiosity-driven and when I have like a natural itch to know.

So let's say a listener is listening to this and they've never tried anything, but they're watching from the sidelines. Oh man, the AI thing is huge. And I wanna get in on the action. So let's say you, you need to give them one practical thing to start with. So what would you recommend? So would it be install your ChatGPT app and start chatting to it? What would you recommend they get their hands dirty on if they are absolutely used to DevOps?

If you literally had to just pick one, it would either be Claude or ChatGPT for sure. If you do, I think there are, there is a lot of fun to be had trying lots of different products. I think Justine Moore of all our past guests. He's probably the best Twitter follow who just tries unbelievable volumes of products and creates cool stuff, reviews them, and puts together these lists. I think that can be a lot of fun and is cool, and I do still try to try a lot of products, but it's definitely gotten away from me to the point where I can't try them all anymore. And I don't think you really have to. I think it is definitely the case that the best Tools come from the best companies, which are well known. If you literally could use nothing besides even just a single one of ChatGPT or Claude, there would still be plenty of room to dig and explore. And they're quite good. And there are some hidden gems out there, but there's none that are going to outshine the recognised leaders. So yeah, that's not a groundbreaking recommendation. But I think it is a sound one. On the education thing, in terms of resistance in academia and why people are entrenched and opposed, I guess I would have to ask the listener who's the professor for the cultural diagnosis. I do think NLW from the AI Daily Brief has some really good commentary on this kind of thing. Ethan Mollick also is a really good commentator when it comes to Organizational adoption of AI and how to do it the right way. I think you can go dig into their findings and blog posts and podcasts and whatever, but I think a couple of the big takeaways are leadership really matters. Survey results show that many people are using AI and are keeping it a secret, both at work and school, because they don't want to be told that they can't do it anymore. So people are like, this makes me more efficient, or I think I do a better job, or whatever. In some cases, it allows me to cheat. Obviously, the notion of cheating in an academic environment is very different from cheating in a business environment. Nobody would consider it cheating if you use ChatGPT to write good marketing copy for an actual real-world ad campaign that you're going to put dollars behind. They just care if it works. Whereas if you're in a marketing class in a university and you did that, then that might be considered cheating. In the real world, cheating is like definitely less of a concern. And what really tends to matter is results. So people are getting results that they feel good about. Either they're able to do the same work in less time, or they're able to do a better job or whatever, but they don't want to be told they can't do it. So they're keeping it a secret. Best practices are slow to diffuse through organizations for that reason. The big recommendation that I've heard that makes a ton of sense to me is that leadership needs to, at an absolute minimum, say, we want to hear what you're doing. You're not going to get in trouble for using AI, but it's about using it in an effective way. And we want to work together on what that is in a responsible way also. Obviously, depending on context, there's a lot of different dimensions to that. My guess for this writing professor is Your students are probably ahead of your administrators. I would be very surprised if you don't have quite a few students that are using AI in various ways for the writing assignments that they are turning into you. Hopefully they're not just having AIs write them. Hopefully they're getting critiques and still doing the work and still growing themselves while getting this valuable input from AI, but probably it's a mix. And can you open up the space for that conversation? That would be the place that I would start. Now, if you're working under a situation where the administration has already said this is a hard no-go, then you're in a tough spot. But if there is any space you could create to allow people to talk about what they're doing, what's working for them, allow you to kind of guide them in terms of You know, what do you think they should be doing and what is ethical? Everybody benefits from bringing that into the open. So at a minimum, people can learn from each other. And that's especially true in business, but I think also probably in academic context.

That seems like pretty actual advices for both. I think those were the end of AMA itself. I wonder if there's anything else you want to touch upon. Anything else you want to lead the audience with?

Well, I had one in the spirit of Ezra Klein, who always ends these things with book recommendations. So the book I wanted to share is called The Maniac by Benjamin Labatut. This book is about John von Neumann. I think it's really interesting, and I think a lot of people in Silicon Valley and in the leading AI companies should read it. The audiobook is excellent. The best audiobook experience I've had. Each chapter is from the perspective of a different person in John von Neumann's life. He grew up in Budapest, I think, at like the height of the, you know, Austro-Hungarian culture. He was in Europe, came to the United States, involved in the Manhattan Project. Many of the scientists involved in the Manhattan Project were also refugees from Europe. Each chapter is voiced by a different actor with the appropriate accent, given what their native language was. You hear these different voices with different accents. It's incredibly well produced, but more importantly, I think it really calls into question the prevailing Silicon Valley understanding of John von Neumann. You know, I've heard this for years now. Why can't we have a thousand John von Neumanns? Why don't we create all these John von Neumanns? How do we engineer our way to John von Neumanns? That's what we need. The picture from the book is one of an absolutely brilliant mind, math genius, technology genius, for sure, but also a pretty depraved and kind of fundamentally amoral individual, somebody who had a lot of intelligence but not a lot of wisdom, not necessarily good to people in his life. I wouldn't necessarily say that by most people's standards, he lived a good life, even though he did solve many math problems. In some ways, I think he had a good life, but didn't have a great family dynamic, not a great relationship with wife and daughter. Like, major problems, things that you would say these are, like, not small character flaws. And he sort of represents the broader Manhattan Project story, I think, where— or you could quote Jurassic Park, as I often like to do. Your scientists were so obsessed with whether or not they could, they didn't stop to think about whether or not they should. And that was, like, really the Manhattan Project story as well, right? We got to beat the Germans and When they made the bomb and it went off, immediate regret from like at least a lot of the people involved, you know, was this actually a good idea? Was this maybe the biggest mistake that I could have ever personally made to like devote myself to this project and bring these terrible weapons into the world? I don't pass judgment on those people too harshly because there really was a Nazi Germany. It was at least credible that they were going to try to get one. And even though I'm skeptical of the good guys, we got to do it because otherwise the bad guys will do it. You know, Nazi Germany is probably the, you know, the poster example of clear good guys and bad guys. You don't want the bad guys to win that race. It's harder to say, were they still in the race at the end? I mean, they weren't. We now know, you know, through benefit of historical analysis, they weren't close. The creation of the bomb. motivated by what if Germany gets it first wasn't right. They weren't going for it at the end. A fear of what the other might do led to this push, and it in fact was the good guys who created the bad thing out of fear of the bad guys. So I think there's just so many themes there that are just extremely important. Losing track of what really matters because we think we're in a race against some perceived bad guys. I see that shaping up for sure. I don't like that at all. And also the sense that technical geniuses know best and just because they can, they should, is called into question by the story of von Neumann himself. This is like somewhat of a fictionalised account. My understanding is it's like pretty accurate to the history, but it's narrative form. It is marketed as a novel. I'm sure there's some licence to it. But my sense is it's pretty accurate to the history. And I think people should think again when they're calling for lots of von Neumanns. You know, I don't think that's actually the world that we really want to live in. And when people, you know, there's been this discourse just in the last couple of days of von Neumann versus Einstein, and people are comparing their like technical merits, you know, well, von Neumann this and Einstein that, and general relativity is even a bigger contribution than anything von Neumann did. And I kind of want to say. I don't think that's maybe not the way we want to distinguish between them. They're both geniuses. They both made major contributions. Einstein has a certain wisdom to him. He has a certain, like, I think the quote is apocryphal, but it's sort of what we want from somebody like Einstein is the wisdom of that quote where he reportedly, maybe not literally, said, I don't know what weapons World War III will be fought with, But I know World War IV will be sticks and stones. And it's like probably, again, a false story, but that is the sort of big picture, zoomed out wisdom we want from leading geniuses. And if this book is to be believed, von Neumann didn't have it. We need to broaden our sense of what we want from our technology revolutionaries and definitely go beyond pure technical genius. It's not just about the ability to make things work, but it's also about really picking the right things to make work and avoiding the wrong things. I don't want to suggest current AI leaders are von Neumann-like. I think he actually comes off like quite badly in the book, but at a minimum, I think it's a useful corrective to this idea. If only we had way more von Neumanns, we'd be better off. I think that is not so obvious. And the people that are doing this frontier work right now would do well to spend a little time thinking to themselves, like, am I von Neumann?

You know?

And if so, is that actually good? Because I think it's not at all obvious.

That's a very strong recommendation. I'm looking for a book, so I'm probably going to pick that up.

Yeah, do the audiobook. It's excellent. Truly, for production quality alone, the audiobook is top-notch.

Cool. Anything else do you want to leave our audiences with?

As long as we've gone on, I'm sure there's plenty of important things that we didn't cover, but for now, I would just want to say thank you. I really appreciate the opportunity to do this. It has been an incredible journey of learning for me, and I'm continually surprised, amazed, and impressed that there are many people who actually want to engage in the kind of deep dives into all these different topics that we've been taking people on for the last couple of years. I don't feel like I'm honestly very good as a podcaster, but I give most of the credit for what success we've had to the subject matter and the fact that it is really important and people really want to know. But I definitely appreciate the fact that I've had the opportunity to do this and that there's enough of an audience that we can sell a few sponsorships and afford to hire you guys to help with the production and really create an extremely fortunate position for me where I really do get to chase my curiosity on a day-in and day-out basis. I definitely do not take that for granted. Absolutely try to do my best to be an earnest commentator on what's going on and Will endeavor to change my mind and perspective as things continue to evolve. I feel like it would be easy for me to sort of fall into continuing to play the character I've been, the sort of adoption accelerationist, hyperscaling pauser till I die. I feel like it's an extremely fortunate position that I find myself in, and I want to pay that back to the audience and forward to the universe by just trying to be as real as I possibly can be, basically, at every step of the way. I do think things are getting pretty real, and I'm in a fortunate position where I can basically say exactly what I think. And so I'm going to try to do that with as much sober reflection as I can, but without holding back. So thank you all for being part of the Cognitive Revolution. It is both energising and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.