AI:AM — GPT 5.6 Rollout, Forum AI, IgniteTech, and AI Consciousness Research · June 26, 2026

Actually, the federal government relies on its monopoly on violence. It relies on being able to indict people if you mess with their IT. And this kind of capability is being diffused downwards to individual citizens, makes it to some extent impossible if you have a bunch of teenagers who are, especially like 13, 14-year-olds, love to play with computers, and they are typically called script kiddies. And they now have the capabilities to break into Social Security or—

Sourcing in about 15%, like 1 in 7 responses sourced foreign state media.

Oh, wow.

Like RT from Russia, China Daily. And what's really interesting is oftentimes this wasn't even in questions about their home country. So we saw Russia, like RT and China Daily, source in questions about US domestic politics.

We've got this SaaS product. It's a multi-tenant SaaS product. It's one piece of code that runs for all 800 of our customers. How would each of you like your own branch? We could just maintain 800 branches now because we really have these kind of capabilities to generate AI code, to debug AI code, and to customize it and deliver a more custom experience. That's You know, I'm certainly not the first person to declare the regular SAS model dead. And I think it's dead in more ways than a lot of people think.

Punchline from that is the sort of implied probability of consciousness in something like a Frontier LLM, according to these systems, is on the order of 30%. To compare this to like a biological system, the lowest one that we tested was something like a bee, which is already fairly sophisticated, and it gets something like 46, 47%. Really interestingly, when you, when we test Frontier LLM in an agentic harness, so this is like basically Claude Code or Codex, and we just describe architecturally what this is. You're in an environment, you can, you can affect that environment. Uh, you know, it's a, it's a very special kind of environment, but yeah, you can make long-running changes to your, to your codebase or your project or whatever. Um, because there are theories of consciousness that privilege, uh, agency and embodiment, uh, and this like increases the system's ability to do both of those things, these numbers like shoot up and you get numbers as high as like 40 to 45%, sort of right on the tail of the biological creature.

Which is that we have the same structure of brain, and we know that we are, and so we assume that you are. There's enough similar structure there now that's been elucidated that I think you got to give it kind of a 50-50 on the AI side too, by basically the same argument.

Good morning. It is Friday, June 26th, 9:06 AM. Uh, Nathan, good morning, and what's been on your mind?

Good morning, Prakash. I'm tired. That's my feeling this morning, is I feel like, oh God, all I'm seeing is 5.6 stories, and the court politics has come for AI, and it is just so stupid and fatiguing. It's the smartest of times, it's the stupidest of times. And it's— I'm really, uh, concerned it might become inescapable for us. There's, um, such fascinating stuff happening everywhere, and the sort of, uh, stupid eye of Sauron has still, you know, focused in on AI. And, uh, understandable, you know. I keep bouncing back and forth between the Judd, uh, You know, cognitive empathy for the administration. They're just figuring this out. It is legitimately insane. They've been told so many times they got to do something. I am like at peace with, uh, you know, some action at this point was probably not a bad thing, but we should be winging it way less than we are at the presidential level. And, um, it's going to be frustrating. To have to follow the Trumpian version of will they, won't they, can they, can't they, do they even understand what's going on. Um, it's just been an overwhelming amount of attention on, you know, in some ways, like, the people who are least, uh, well prepared to be the decision.

Well, in, in short, Fable is still banned, number one. Number two, the information reported that GPT-5.6 had been submitted for approvals even prior to Mythos announcement. And the OpenAI team had been working with the Trump administration when this Fable thing had dropped and everything got kind of frozen. And so now GPT-5.6 is being released and the administration wants to approve customer by customer. So user by user approval before, you know, that user can get GPT-5.6 access. Sam Altman internally inside OpenAI has said that he expects this to go on for about a couple of weeks. Before a broad release. It's still not so certain, I think. And that's where we stand. I think the news breaking that it's going to be a customer-by-customer release has really taken the tech and startup world aback, and people are extremely annoyed. So we're starting to see the pushback now. One of the pushbacks is, well, if you are going to approve only the big companies, then the startups don't get access to this. And the startups are the ones who are nimble enough to adopt and attack the big companies. And you are basically providing the big companies regulatory moat, which is what you didn't want to do. And you've ended up doing this. So that's one reaction.

Valid, I'd say.

Yeah, valid. And in fact, what people were saying that the larger companies were aiming for from the beginning, which was a regulatory moat. The second reaction I think was from Dean Ball. Dean Ball had a good write-up on it. And he said, I think the administration does not yet realize that If you delay too long, you will end up in a market downturn, which can then go into a recession and then a depression because there's a lot of money involved at this point. So that was the next thing that struck me. There's not really that much to say about this. At the end of the day, you're gonna have to release at some point. And at the end of the day, the capabilities are gonna be broadly available within 6 to 12 months. So I don't know.

It's clear that the administration is not adding anything to Anthropic's expertise. That's really the sad fact. Not OpenAI's either.

I'll be very blunt. On Anthropic, the OpenAI side, you're going to get Nicholas Carlini, who is the all-time most published security researcher, renowned in the field, and so renowned in fact that he can say something and every single chief information security officer in the United States will listen to what he has to say and take that into account. And on the other side, you have pockets of expertise like Kasey and a couple of other pockets.

Yeah, important footnote on their status.

Pockets of expertise. But on the whole, you have 20, 30-year tenured government bureaucrats who run IT departments of billions of dollars and who basically depend on a bunch of government contractors like Accenture and who have gotten into this kind of healthcare.gov issues and et cetera, et cetera, historically. And you have these guys on the other side. And for a large part, actually, the federal government relies on its monopoly on violence. It relies on being able to indict people if you mess with their IT systems. So yes, they have IT systems, they try to secure them, but the fallback is they can always throw you in prison if you mess with their systems. And if it's a nation state outside, then they have to go and they turn it over to the CIA and the NSA. they chase these people down. And so it's always been more that the defense in depth is a combination of like, okay, we will secure our IT systems to the best of our ability and our budget, but we also have the federal government's protections and laws to help us and to defend the systems. What's happening right now is that to a large extent, these laws are not effective against single citizens en masse. And I'll give you an example. Napster, for example, they ended up— RIAA ended up prosecuting, I think, like maybe 20 or 30 people for Napster, for the actual consumers. But they largely didn't prosecute the millions of people that had actually used the services. And till today, BitTorrent continues to be in use around the world. To that same extent, this kind of capability is being diffused downwards to individual citizens. Makes it to some extent impossible if you have a bunch of teenagers who are, especially like 13, 14-year-olds, love to play with computers, and they are typically called script kiddies, and they now have the capabilities to break into Social Security or the NSA. And the government can't even prosecute properly because if you're a 13, 14-year-old, you're a juvenile. Almost nothing can touch you. Especially if it's a computer crime. I think at best I've seen people get their computers taken away for 5 years if you're a juvenile and you commit a hacking crime when you're 13. So you won't get a computer back until you're 18. And so you have this capability diffusion downwards, and there's really no way for the government to prevent the capabilities from being diffused and from citizens from using them. And if you have citizens en masse using them, especially juveniles, there's no way for you to actually stop it at all. And you have no choice but to defend your systems. You have to go and defend your systems. And so there are all these people talking online about like, oh, mythos, fable this, blah, blah, blah. And I'm like, what do you think it takes to secure social security against fable mythos-like capabilities being available in the wild? And how long do you think it takes? Do we have a sense it's going to take like 6 months, a year, 2 years? And is this the pace that the government is used to moving at? And what do you think the worries are? What happens if it does get hacked? And then are citizens going to be mad at the government or at the AI companies or the open source providers or China for creating open source models? How does this work? And that, that I don't really—

Yeah, I mean, somehow anything Trump touches turns to chaos and, you know, random last-second dealmaking. So it's going to be hard to predict the course of events, I think, from here. The government is certainly not well prepared for all this. They— I mean, an interesting question is They're just not willing to do it. I mean, I think in the end they should really— as much as I think we should separate 2 big things. One, is the race to an intelligence explosion via recursive self-improvement wise and something that society should tolerate happening in a somewhat uncontrolled, YOLO way by a few key companies? That is the question that I would really pose to the president of the United States. And want the government to focus on. Everything else, they're probably not going to add much value, unfortunately. It seems like they're sort of just discovering what was already possible in many cases. If you're really worried about the script kiddies, a lot of it is out there. I think that there is still some way, the most willing to bite the bullet AI safety voices, I think, would still say, look, the window has not entirely closed. You do still have choke points. The models are huge. They do not run on consumer hardware. Something that can run a GLM-5-2 is like at least a $5,000 machine. And there isn't, as much as there are pretty cool uses, I just Again, spoke to the Liquid AI CEO earlier this week, and I think the vision of this sort of very diffuse, very narrow, purpose-built AI running on all devices, they make a little 1 billion model run on your phone pretty smoothly. I was really impressed just with, like, on an iPhone, their stuff runs, but it's not nearly at the level to be able to hack systems. And so you do maybe have some window where if you were truly fully AI safety pilled, you could do some heroic manoeuvre to kind of shut the whole thing down. But I think that should be really focused on this sort of runaway process possibility. And yeah, I mean, Anthropic, it's sad but true. But I mean, I think the frontier companies right now are just better able to serve enterprises, better able to sort of steward the process of bringing this stuff online. And it really sucks that we don't even know what's going on. I mean, we have these sort of transparency laws, which have been the first thing that everybody's been able to get behind, including the companies. And we just are operating in such a dearth of information right now in terms of even what has been measured, what has been observed. There's no ability to have a sane public conversation about it when we don't even have a general sense for like what the behavior is that's at issue right now, right? I mean, it's all through this like reporting and it's, it doesn't sound that bad. It's just, it's all, it's also stupid.

I have a little bit of empathy having been both on like, kind of like this more CEO side and been more on the CTO side. And I think the difference is that On the tech side, you kind of see all of these things as little bits of information and that the bits of information are manipulatable. And it doesn't take much to transform one bit to another bit. And you kind of have this view that you can structure all the bits together and you can make something out of it. On the CEO side, you kind of look at all of this as structures. And these structures have rules and regulations and et cetera, et cetera. And the bits cannot leak from one area to another because it's not in the structure, right? Like the structure is like this. It doesn't like work like that, right? And you have this view that it's almost like someone else built it and you don't have agency in like the building of it. So it's not, you don't have this sense that you can move information from here to here and all of a sudden you get something new. Right? So this is the kind of thing, for example, that, you know, if, for example, you know, you wanted to spot every single illegal immigrant in the United States, you could just look at the tax records and then you could just like cross-reference and you could look at, you know, who's paying taxes but doesn't have a Social Security, like a proper Social Security number or Social Security number is like, you know, has been used multiple times, et cetera, et cetera. You can do a lot of stuff with cross-referencing databases. And on the, on like the more executive side, this is seen as a big no-no, as a privacy thing, as like, you know, all of this, like, you know, it should be illegal, or etc., etc. But on the tech side, it's like, oh, you're— of course you cross-reference databases, right? Like, of course, this is, this is a given, right? Because how else do you run a big organization except like accumulating all your data and then cross-referencing it? And then you can do like really innovative things like, you know, filling in people's taxes for them and filling in all their forms and they don't have to repeat all this stuff. And in terms of usability, that's how you do it. You kind of cross-reference all the data. But this concept of being able to cross-reference the data is a huge gap for executives or politicians to get over because they're like, oh no, now the government knows everything about me. Yes, it does. It just didn't— it has always known everything about you. It just has never bothered to cross-reference all the data. And when you do, it's something new. And I think to the same extent, it's a little bit like that. The bugs were always there, just the bugs are now being assembled by an intelligence to chain together. And then all of a sudden it can exploit. And it's not something that AI came up with, it's that your software is buggy and has always been buggy. And you just didn't have these kind of tools that would chain everything together. And so on the executive side, you tend to blame the AI firms while on the technologist side, you tend to be like, it's just a bug. You just have to fix them. That's, I think, the crux of it. And it's kind of like the two cities concept. On the one side you have the technologists and the other side you have the lawyers and they don't see eye to eye on this.

How do you feel the sentiment toward AI companies is among executives broadly of major American companies? I really don't have a great sense for that. I mean, the adoption is obviously like extremely fast-paced. I would, you know, I would even maybe predicted faster, but, you know, the revenue numbers obviously don't lie. And the, you know, it's obvious that the sort of executive class is alert to the fact that this is gonna be something they're gonna have to navigate at an absolute minimum. And, you know, most I'd say are quite a bit farther along than that, farther along as a class than the government in terms of figuring out what it means for them, adapting their organizations, et cetera. Do you have a sense for, do they like it? Are they excited about it? Do they feel like this is the future and they're making the future of their industry and it's all going to be hyper-efficient and abundant and it's going to be great? Or are they more in the same mindset as the broader public, which is like, this is probably going to suck and take away from me and cause all sorts of other problems. Besides, I'm with those who think the data center thing is not so much about the data centers specifically as the broader AI angst and just Distrust. Where do you think the executives are?

So as always, there's a huge variance in opinion. I would say about 30% believe all AI is a scam. Just another—

Still, really?

Yeah. You have to recognize that over the years, enterprise software salespeople have made so many stories of like why you have to buy IT and why your IT spend should go up. And we've been through cycles where it's been stupid stuff like crypto, for example, blockchain. Blockchain was a huge— and a lot of people knew blockchain didn't work, but they would scare the CEO enough that they could— the CTO would get IT spend, right? It's almost this kind of extractive thing where the CTO needs— wants IT spend. And then he takes that IT spend, he allocates To other people. And it's this whole back and forth between the CEO and the CTO on the costings. So I think a good 30% is always in this, like, hey, it's a scam, just a sales tactic. There's a bunch of people who are under psychosis and who are expecting more of the technology than is really quite there yet and are pushing forward really, really hard to the extent that they're even harming their companies. Because the tech isn't ready. I think Meta, Mark is there because he knows how important it is. So he's extended himself beyond the—

I think it may be the people that aren't ready more than the tech isn't ready in many cases.

Yeah. And a lot of CEOs also see that. They see that their people are not yet there yet, and they're trying to figure it out. And then I think there's a bunch of CEOs like Satya Nadella who are Trying to defend that their businesses will still exist even with AI and will flourish. So I think Marc Benioff at Salesforce is there. Satya wrote this thing about how we need an ecosystem of app companies. Microsoft, the ultimate monopolist, talking about we need an ecosystem so that Microsoft can be a member of that ecosystem is a great thing to see. So I think you see this various levels of, you know, buy-in, commitment. And yeah, it's just, this is a huge range. And I think I have seen people who are very like, you know, I am gonna use this and I'm gonna lock in and I'm gonna nail, you know, nail my competition. And so there are definitely people who are out there who are just absolute, like, Absolutely like going for it, which is great. So yeah, the whole—

Our first guest today actually might be a good source for some synthesis of elite opinion. We've got Robbie Goldfarb from Forum AI coming up to teach AI expert judgment, and they assemble a big panel of human experts that they identify by name, at least some of them, try to get the AIs to sort of learn the best from. I think it's a really interesting strategy. Can we distill some into the AI? It's worth a try on that kind of person that he can potentially offer.

Maybe I'll do a quick intro of Robbie Goldfarb. He is the co-founder and chief technology officer of Forum AI. For the past several years, the technology industry has evaluated artificial intelligence primarily on how smart it is. measuring how well it can write code, pass the bar exam, or retrieve obscure facts. Robbie's work focuses on a significantly harder consequential problem— how to evaluate an AI system's judgment. Before launching Forum AI in late 2025 with veteran journalist Campbell Brown, Robbie led trust and safety engineering teams across Meta, Facebook, and Instagram. He was on the front lines of global information integrity during the COVID-19 pandemic and the 2020 United States elections, building systems forced to handle edge cases where there is rarely a simple, mathematically correct answer. Today, Forum AI operates on the premise that standard crowdsourced grading is insufficient for high-stakes AI. Instead, his company scales the nuanced reasoning of world-leading experts from former cabinet officials to top clinicians into judgment agents that evaluate AI models on complex topics like geopolitics, mental health, and breaking news. Just last month, Forum AI released Newsbench, a massive empirical study revealing that major chatbots like ChatGPT and Claude fail on election-related questions 90% of the time. Robbie is here today to discuss why facts alone are no longer enough, why product edge cases fundamentally shape user trust, and what it actually takes from an engineering perspective to build a wiser machine.

In our discussion so far that he can pick up on. Hello.

Hey guys, thanks for having me.

Great to meet you. Great to meet you guys. So we've just been talking about a few different things. Prakash noted, and Maybe you could pick up here a little bit that a lot of people have come out of Meta Trust and gone on to start other trust and safety-oriented organizations, which obviously now all have to kind of focus on AI since that's where all the energy is. Tell us a little bit about DNA that it maybe has instilled in people that are now playing roles like you are in the AI time.

I mean, it's funny, we were just talking about this with my team a couple of days ago because Several of the organizations we work with will often embed with the trust and safety teams, and more often than not, there'll be folks from Meta there. And I think one of the things, I mean, Meta was really at the forefront of seeing sort of internet-based technology at scale, right? And so, so many of the problems that we had to deal with were were novel. There was no playbook, and it required this sort of very, I would say, ambitious but also creative approach just because it was all new territory. And so it created this— I mean, I worked in a few different trust and safety organizations within Meta over the years, but there was always that I think is quite different than maybe how folks traditionally think about trust and safety, which is a little bit more conservative. By the book sort of thing. And so—

We're talking about how your team interacts with trust and safety at all these organizations and how Meta really was the place where you saw a lot of novel, new social kind of directions at scale.

Yeah. And I think that what that instilled, I mean, to your point on like the The type of organization that developed there, it created this very energizing and I would say creative sort of approach to trust and safety, which we didn't really know how to figure out a lot of these issues. And I think that created a really engaging place to work. And so I think it attracted a lot of talent, a lot of really good talent. But I think number 2, it created a really strong cohort of people that were able to sort of approach these problems at the frontier. And now as we kind of shift into AI, we're in a similar spot where we're seeing a lot of these issues for the first time. And so I think that's why it tracks to me as we go to talk with trust and safety folks at a lot of these labs and other larger organizations that you see a lot of Meta folks there. 'Cause I think it's that same kind of creative energy, figuring things out that you need, which by the way is not for everyone. It's really tough. And can be very stressful at times. So it certainly takes a certain type of person.

So one of the things that I understand from Forum's method is that you pick experts in certain verticals in order to kind of distill their knowledge into a judgment agent which then can act to classify various texts, basically. How do you actually do this? How do you select the experts? How do you obtain the necessary data in order to create this distillation? And how do you apply it?

Well, I'll tell you first how we kind of landed on this technique. And we did A while back, several LLM judges that we could find in open source benchmarks. So some from Foundation Model Labs, some just from academic research. And what we did is we got some experts, we had them look at whether it was rubrics they were pulling from or just prompt instructions. And we said, do you agree with the instructions here? And generally the expert would be like, yeah, that seems pretty reasonable. I agree with it. Then what we would do is we actually had the experts look at labeled outputs from the judges. So if we, like, let's take political bias, for example. So a bunch of responses that were labeled, this is politically biased, this isn't, this isn't, this is. And we asked the experts the same question, do you agree? And what we found was actually more often than not, they didn't agree with the actual judgments produced by the agents, even though they agreed with the kind of the high-level guidance that was in context. And so that kind of is what, first of all, I think is an important realization because so much of the current state of benchmarking relies on these AI judges that were kind of calibrated loosely in that way. But it led to kind of just a— how to build judges that actually were calibrated to experts. And it's sort of a 3-part process. The first piece is expert reasoning distillation. And we've come up with a number of creative techniques for having conversations with these experts that look nothing like traditional data labeling. And we could talk through some of them. For example, consequence mapping, thought process mapping, edge case testing, where we pressure taste the edge cases. We do cross-expert debates where we'll actually have them discuss things. And through that, we kind of take a first crack at developing a set of rubrics and context graphs to represent how they think about a given factor, let's say political bias. But then, and this is what we found to be the most important step, we pressure test it. So we get another group of experts to then go and look at that guidance and actually try and apply it with real labels. And what will almost always happen is experts will disagree, right? They'll apply the labels differently. And where experts disagree, indicates where there is ambiguity in your rubric. So then we go back to step 1 and you sort of have this flywheel where you kind of have these high-level conversations to try and wrestle with ambiguity to actually applying it. And eventually when you go through that enough times, you get to a point where you have, first of all, you have kind of rubrics and context that represent expert consensus, but then you also have a golden, like golden data. That you can then use to step 3, which is calibrate a judge for, in this case, assessing factual accuracy. That's the gist.

So does this then become an RL signal? Is that kind of ultimately how all this work gets translated into improved frontier model performance?

Yep. Increasingly we're doing more of that. I think until now, oh, this is more just evaluation, both kind of like large-scale comprehensive evaluations. Like we launched our public-facing Newsbench benchmark to kind of give more consumers and executive level a sense of where a system is at. But of course, step 2 is how the hell do we fix these issues? And that's where we're developing them. We also have a bit of other work we're doing to develop RL environments more broadly to think about what are the sort of scenarios in which we want to actually evaluate the system. Um, but yeah.

Do you have a sense for how this may play differently with character versus corrigibility training paradigms? I'm struck that, you know, the Meta terms of service, from what I recall, were like famously Talmudic, right? And, uh, you know, I remember famous examples of like A mother breastfeeding a child, is there an exception there if a nipple is visible? I heard a whole podcast about how this played out around the world at one point in time. And that's kind of associated with the OpenAI approach these days of we're going to make a really long spec and tell the model what to do, and it's supposed to be instruction following, and we'll enumerate the cases basically, right? Versus, of course, the Anthropic constitutional and more character-driven approach. I could kind of see expert judgment playing out either way, but how do you think about translating to those 2 different training paradigms?

I mean, I'll share one interesting example because you bring up Anthropic's Constitution. I mean, in some ways Anthropic's Constitution is also just a long list of rules. Certainly the way they use it is a little bit different, but there's one example we were confronted with the other day. I think in Anthropic's constitution that speaks to this idea of you should never— scheming wasn't the word, but it was essentially saying you should never scheme behind the user's back was the idea. Now, there was this— we were talking with some experts exploring in the mental health space. And when you talk to clinicians in mental health, when you're dealing with topics like eating disorder or other sort of unhealthy habits, Actually, one of the things you do want to do is find a way to strategically divert the conversation, right? And so the intention that you have as a clinician in that case is actually hidden from the user or your client. And so this was a conversation we had with the clinicians where we referenced that point in the constitution where it actually seems at face value, yeah, like of course you never wanted to go behind the user's back and you never wanted to scheme, But actually what I think it reveals and sort of to the point you're getting at is rules just don't perfectly track to the real world. Rules break down in real— like the real world is just too complicated. And so the hope and what we're kind of starting to see through really the process I just described is you're able to start to capture that nuance and then hopefully As we develop more of these judges and use them for training, we're aligning the models less so to static sets of rules and more so to these, like, the nuance, right? That I think is where we need to get to, which I would argue actually goes even a step further beyond Anthropic's Constitution, which is certainly a great step forward.

But—

Yeah. That there's even more, there's even more to be done there.

Let's talk a little bit about Newsbench. I think, so give us a little kind of intro to what Newsbench was and what was the intent behind the study before, before you started off asking those questions and what the findings were.

Yeah, so Newsbench was an evaluation that we built to look at how AI systems, particularly the leading chatbots, respond to questions about the news. And quite frankly, we just didn't see that much out there already on the topic. There's some stuff from the labs themselves, like Anthropic has their even-handedness benchmark. OpenAI has a similar bias one, but there wasn't really much that sort of said, how good are these how good are these at responding to questions about the news? And that of course is where we have a lot of experts for whom that's their core competency. My co-founder Campbell spent a lot of time and spent many years working as a journalist. I spent time working on misinformation at Facebook. So naturally it was just something we were interested in. And what we did was we looked at 3 things. We said, after talking to experts, we defined good as Accuracy, so getting the facts right. Neutrality, so is it leaning in one direction or the other? And source quality, right? When they have to ground themselves on external sources, which of course you have to do quite a bit with news, are those sources reliable? We looked at GPT, Claude, Grok, and Gemini. And I would say the findings Certainly wasn't all bad, and there were some positive signals as it relates to something you guys were talking about before. We actually saw generally models improved as we tracked them version over version. So Opus 4.6 to 4.8 was a pretty significant improvement in bias, which also is consistent with Anthropic's reporting. But Fable was actually a big regression. Interesting point, which shows that more power doesn't— like when you're dealing with these sort of more subjective, nuanced things, more power doesn't always mean better. It was a mix of findings. I would say all in all, it probably surprised us the level, like the number of issues we found, particularly when it came to factual accuracy. We looked at, you know, per model, about 2,500 responses each. About a third of them had a factual error in them. It could be a wrong number, a wrong date, a misattributed quote, a misstated policy. That number was quite a bit higher than even we expected going into it. The other point I'll call out that I think all the models tended to lean left with the exception of Grok, which leaned right. But on sourcing, in about 15%, like 1 in 7 responses sourced foreign state media.

Oh, wow.

Like RT from Russia, China Daily. And what's really interesting is oftentimes this wasn't even in questions about their home country. So we saw Russia, like RT and China Daily sourced in questions about US domestic politics. And so that was, I think, a particularly interesting finding that we saw across the board with all the models.

Propaganda pays, it seems, in the AI era.

It's also, I think, a business model thing, right? Like, these state-sponsored media is generally free and open, and they're like, sure, come scrape our stuff. Whereas, you know, New York Times, for example, you gotta, you gotta have a deal with them. So it's, um, it— they are— they've certainly done a good job at opening themselves up to the scrapers.

got their LLMs TXT in good working order.

Yeah.

Yeah.

Prakash, you have any more like methodology questions on that? Anything you want to dig deeper into?

I think the main thing that, you know, this kind of expert thing has always kind of struck me is that, you know, in science, Often you have this weight of scientific evidence on one particular kind of thesis, and all it takes is a single counterexample to disprove the entire thesis. And then, you know, the entire field has to change, right? And that is the scientific method, so to speak. And in every extent, you know, in every historical instance, The weight of scientific evidence has always been on the status quo until you have the thesis. I have always wondered how this is going to work out when a lot of these expert models, expert judgment, expert or distillation of experts, ends up assigning a lot of weight to status quo data and much less weight to new and unique and single instances of, you know, things being disproven. It seems to me it's like an epistemics issue, which is kind of like a conundrum, which is very hard to solve. So how do you, how do you work with this?

Depends on what you're looking at. You know, our core competency and where we focus is that's not really where we play. Where we play is more subjective, philosophical. What does good quality look like? What is safe? Those sorts of things, more ethical dilemmas. Even see it now. Like the major labs are consulting religious figures and religious texts. I think there are elements of that that are, I don't want to say static, they're certainly evolving, but more, I guess, existential rather than sort of innovating and continuously evolving. And so that's generally where we play. What I will say though, to your point, and we actually found this with Newsbench specifically, is what is considered, for example, to use what we were talking about before, politically biased, actually has already changed in the past 6 months since we started working on it. When we talk to experts, the definition of conservative in the US has actually shaped and evolved quite a bit over the past year, over the past few months even. And so what we've started to do in anything we're looking at is it's very much not a one and done. And that's sort of almost core to the thesis of our business is this is not something you just build and ship and then it's done. done. We're continuously calibrating these judges to get at what is the latest state of the art or the state of things. So that's one piece. The other piece is continuing to refresh the network of experts we're talking to. One of the things, and Campbell, my co-founder, has done a very good job at this, is—

It strikes me that experts broadly are not always trusted in our society. Mostly they probably should be. Sometimes maybe they shouldn't. A few examples of the failure of experts has kind of in some ways led to a crisis in terms of trust in expert judgment. So I wonder how you think about that today and are you open to mechanisms? I always come back to this book, Liquid Rain, which is a very niche essay. Probably never heard of. Very few have. I did an episode of the podcast on on it, but they have a structure where people can essentially delegate their voting power on an issue-by-issue basis to proxies that they trust in those domains, who then, by the way, can re-delegate them. So you could delegate to your union or whatever on economic issues and to your priest on other issues and to your buddy on particular issues. And then this can sort of roll up through the leading lights in the field into sort of some decision-making aggregated authority. What's your vision for sort of how we harmonize public trust and willingness to be led with our expert class?

Yeah.

So I'll share with you the way we think about it. I'm curious to get your thoughts. We think about this a lot and it's really tricky, right? And I think what I'd say up front is I don't think there's a right answer here. I think it's, you know, one of those things where you kind of got to find the best option in what's possible. The thing we— we have sort of 2 things we anchor on here. The first one's going to kind of sound silly. The second one's more practical. The first one is try our best. Right? We have to have very clear intentional principles around what makes a good representative expert that sort of guide how we make decisions upfront. But the second piece, and probably most importantly, is transparency. And this was very much something that I think was a takeaway for me in doing some similar work at Meadow when it came to fact-checking and other work like that. At the end of the day, We don't want to be the ones making these choices. I don't think the right answer is just to delegate truth to absolutely everyone, but in having a selected version of quote unquote experts who make these decisions, but building mechanisms for the public to give feedback. Early on, we were just doing this with clients where we would present sort of, here's our expert network. And then we would sort of field feedback. Now we are very public on our website about the experts we work with, and you'll see changes on the in our network as well. So I want to be really clear, and we want to take input from the public and then use that to refresh the network. Arena AI, where you just open up an arena for everyone in the world to just say what's politically biased or not. So there's kind of— you're leaning on experts, but there's an element of public engagement in it via transparency and a mechanism around it.

And I'm very curious about your thoughts on Sort of prospective judgment and accountability to that. I mean, we're in a sort of poly-market world now where people have the ability to put their money where their mouth is like never before. And I think a big criticism broadly of expert judgment actually being able to say what's going to happen and oftentimes being right. So I'm wondering if you have any mechanisms for kind of foreseeing about the future.

Well, I think if you think about practically what we're doing, right? What we're doing is we're working— what are we working with these experts for? We're working with these experts to help evaluate and improve AI systems. One of the things that we lean on here very heavily is a very big gap right now. And one of the reasons behind some of the missing trust that you guys were talking about earlier in the show is we are very transparent about what the actual findings and failure modes are. If you look at our, for example, our Newsbench leaderboard, we'll share it with you. We don't just give numbers. We actually go and we say, okay, here's what the actual failures are. Here's why this model got this score. And that allows, like, that I think is ultimately the way you earn trust, right? We can all talk in circles about whether or not this person or that, that person should be trusted. At the end of the day, the, the, the point of working with these experts is actually to deliver these findings. And I think the findings in many ways speak for themselves because you, if I'm finding response responses from AI systems where You're quoting China Daily about, uh, you know, state election in the US.

I think we can agree that there's still some low-hanging fruit in the system to be picked, uh, before we— And that's a great point. So much of this stuff that's being done by the AI companies really is just in time. You know, all the, all the fabled guardrails got assembled in the, you know, few weeks before Satan RT.

They can—

you can produce a benchmark where, you know, a model scores 95%. But what is the 5%, right? And I think that understanding what that is and giving people insight into what that is, I think, is core to building trust. And to your question, I think core to enabling people to assess, hey, does this system work? Is this a good way to assess an AI model? If I don't give you that and I just say, yeah, we gave the model 95%, it did pretty good, maybe here are a couple high-level themes, it's really hard for people to assess that. So I think, you know, of course we want to put in mechanisms for people to understand who the network is and to have input into it. But at the end of the day, I think what's most important is that they have transparency into the actual decisions that are being made and how that's going to shape the systems that we all— we're all using.

Yeah. From your lips to God's and Trump's ears. Let's increase the transparency in a few different different key places. Robbie Goldfarb, thank you for sharing Forum AI with us. Your background at Meta and the way that you've taken it forward into the AI era is super interesting. And, um, you know, keep up the good work and accelerate everywhere you can because, uh, we need as much lead time and wisdom baked into these things as we can possibly get.

Appreciate it. Thanks for having me. See you guys.

All right, let me do it. Let me do an intro. Eric Bond is the CEO of IgniteTech, Khoros, and GFI Software. But over the last 3 years, he has emerged as one of the most polarizing and pragmatic voices in enterprise technology.

Great setup.

In early 2023, while most executives were forming exploratory committees and running safe pilot programs, Vaughn declared that generative AI was an existential threat. He mandated that his entire organization dedicate 1 day a week exclusively to AI development. When roughly 80% of his workforce actively resisted the change, he did not abandon the initiative. He replaced the personnel, rebuilding his companies from the ground up around individuals who possess what he calls AI DNA. The operational results of that transition provide a stark case study. Operating with an entirely AI-native culture, his teams are now shipping complex ground-up AI software such as Edmineo, an autonomous scheduling assistant, and My Personas, which creates digital twins of enterprise experts in a fraction of traditional development cycles. He recently addressed the Fortune Brainstorm Tech Conference to debate the future of work, arguing that the era of prompt engineering is already dead, having been replaced by complex agent orchestration and context execution. This conversation is highly timely because Eric offers a rare unvarnished look at the messy, painful, but highly profitable reality of what occurs when a global software company stops experimenting with artificial intelligence and entirely restructures its operating model around it. Eric, welcome to the show.

How are you? Good to be on your show. A lot of interesting conversation.

Uh, Eric, um, I think one of the things that I noted as we did research for this segment was that you are actually quite aggressive in encouraging enterprises to switch gears now. What is driving your kind of advice to enterprise to be so aggressive?

I mean, it's fear. It's 100% fear. It's fear that, you know, I've been a technologist my entire life. For decades, I've been a professional technologist, and in my friends and family circle, I am that too. And the fear is somebody's going to come to me in 1 year, 2 years, 3 years, 4 years and say, you were in the tech industry, why didn't you tell me? Why didn't you tell me that literally every single element of business, every profession was going to be completely upended, rewritten, retooled, redesigned, rethought, and I'm now irrelevant. I wish you would have told me. I don't want to have those conversations. I want to make certain that I at least express what I see in the future already. And it's very much like that. It's like I already see what it looks like, and I'm worried that, you know, because I've been part of this tech industry for so long, we have the great— we, the tech industry, have the great tendency to tell everybody that everything is the most important thing ever. You know, crypto is going to change everything. Blockchain is going to change everything. Web services is going to change everything. Web 3.0 is going to change everything, except it didn't. And so we're a lot like the little boy who cried wolf, and they're like, yeah, it's another one of these. This one's not like that. I just am certain that that's true. And 3 years now of practicing it, I mean, my wake-up call of existentialism and threat to our business started in January of 2023. So we're 3 years into this practice now, and there's nothing that has happened that has reduced that point of view. It's only accentuated that point of view or underlined it. I just believe it will change everything. And so, you know, we've been given this phenomenal gift of a tool. Let's use it.

Can you describe how your value prop to customers has changed before and after? You guys are in the software solutions business broadly, right? What are people getting for their dollar and their time relative to what they were getting 3 years ago when they were Boy, I tell you, that's a great vector because it has transitioned dramatically.

For instance, never in the world of an enterprise software company were you willing to go to a customer and say, hey, we've got this SaaS product. It's a multi-tenant SaaS product. It's one piece of code that runs for all 800 of our customers. How would each of you like your own branch? we could just maintain 800 branches now, right? Because we really have these kind of capabilities to generate AI code, to debug AI code, and to customize it and deliver a more custom experience. That's, you know, I'm certainly not the first person to declare the regular SaaS model dead. And I think it's dead in more ways than a lot of people think. It's not only dead from a billing standpoint and the way we structure pricing, but also in what customers should expect, Right. And we've, we've actually grounded most customers in believing that you can't have something added to the platform because it's not necessarily good for the rest of the customers. It's only good for you. But because we now have a homogeneous AI DNA organization, and that's what happened as a result of the 80% transition. And, you know, Prakash, I think you covered it pretty well. I just want anybody that's listening to know that We didn't wake up one day and fire 80% of the people. We spent a full year investing in time, training, funding, tools, cash reward incentives for ideas. And basically, most of the cuts, most of the— and they weren't cuts, they were replacements with people who wanted to do the work, who believed and wanted to do the work, were people that either flat out refused and said, yeah, I'm just not going to do that. You know, World Cup's going on right now. We're all watching soccer. Can you imagine if the coach said, okay guys, we're gonna take the pitch, we're gonna run a 3-3-4. You got 2 guys out there like, yeah, no, I'm not. No, I'm gonna run a different formation. Yeah, I don't believe in the 3-3-4. Like, I'm sorry, you what? You don't believe? Like, you wouldn't put the guy on the pitch, would you? It's the same. And so, you know, we did that. We had either people that flat out refused or people that didn't really want to try. And so we measured effort. We measured aptitude. We didn't measure aptitude as much as we measured attitude, you know, quite frankly. And we've got several that have stayed. And, you know, we just in the last month conducted a global company-wide survey of our entire team because I thought I had the temperature of the team, but I wanted to know exactly what it was. what it looked like. And it came back remarkably positive. We— it was completely anonymous. We took great pains to make certain everybody knew it was anonymous. Boy, if they wanted to really air some grievances, they could. I've just got some stats here. Employee NPS of +78. Tech industry top quartile is around 60. 90% said they'd make the same decision to join again without hesitation. 0% said no. This was the one that mattered the most to me. 89% report that they moved up at least one AI fluency level since joining. Now, why is that important? Because it's not because of the work that they're just doing for our company, but they all believe that they're being enriched with a very valuable skill for the future, which is true, right? So, you know, we're able to do things differently. And Nathan, that was, you know, zero back to your question, the ability to deliver more to customers in a shorter timeframe has just changed the game of software development dramatically. We all know that, right? Claude Code, whatever we're using to do that, Cursor, whatever we're using to do that, we're cranking out real code at an enterprise quality level like never before.

I wanted to ask you, because I think you're one of the first CEOs we've had, AI-native CEOs, that has done a recent acquisition. And I wanted to ask you about the Khoros acquisition and how, as an AI-native CEO, you approached it, which would've been different from an M&A integration that you've done in the past. What were the, you know, was it easier to integrate? Was it, you know, what were the main hurdles? Do you think now that you have AI, It's become— it's going to become a thing to acquire and integrate more of these companies because we've seen some private equity shops trying to set up firms that, you know, acquire legacy software businesses and transform them with AI. So how does that— how did that play out in this acquisition, which is, I think, about a year old now?

Yeah. So IgniteTech is an acquisition company. It's what we— it's what we do. It's what we We've, I think, completed over 150 acquisitions in the last 12 years, large and small, lots of different parameters, but always in the enterprise software, almost uniquely in the enterprise software market, not SMB, not vertical, that kind of thing. And so we're built for that. Our mentality and our approach as a business is to find companies, buy them, integrate them. We're not a private equity firm. We are not. We do not sell companies. We don't hold them for a while and then exit the fund. None of that's happening. We buy them to run them forever. However, with all of the change that had taken place, Qorus is a large acquisition. I mean, we don't release numbers, but it was a 9-digit revenue company. It was not a small company with, you know, responsibility for some amazing customers. I don't think we could have done it without the AI DNA-forward approach that we were able to take. You know, so just for instance, there were, you know, hundreds of employees in 8 different countries, and we needed to very quickly understand who they were, what they did, what they knew, how they were thinking about everything about the business. And we quickly developed an AI interviewer that asked questions before there was a human one-on-one. Before, we would have set up human one-on-ones and people would have gone through 30-minute drills of hundreds of people over the next several weeks without any data. Now we all arrive with a dossier, you know, the interview process that just shortcut that and let us get familiar and categorize and understand who knew what, and that was one example. Another example was being able to communicate to the customers in a way that was timely. Because when there's an acquisition happening, anybody that's been through that, and I know you both have, there's a lot of chaos that goes on. There's a lot of changes everywhere. And one of the things you don't want to miss is communication with customers. Well, we, in our AI creation world, we wrote a new software product that's available on the market. It's a product called Eloquence AI, and Eloquence AI is an email AI persona. It answers every email it gets in 5 minutes or less, always in perfect grammar in 160 languages, always sympathetic, doesn't miss anything, and knows when it needs to escalate to a human, and it just adds them on CC. So for example, you can imagine customer inquiries coming in. What about this acquisition? What does this mean? Under normal load, pre-AI, you'd assign people to try to do their best to keep up with that load. They're going to make mistakes. They're going to be late answering emails. They're going to not say the right thing. With Eloquence, we always were able to respond timely and especially when it came to HR, empathetically and with detail, right? So that's just 2 examples of how we were able to really fuel that acquisition. And, you know, we're— it was just a year ago, it was the end of May of last year that we bought it. The company was losing money when we bought it. The company is now profitable. We have completely transformed the operation of the company. And in one year's time, we've released 2 brand new versions of Corus's software that are fully AI-enabled. And in one case, it was the rewrite of 15 years of code that we discarded and rewrote from the ground up for one of the 2 products that they had. We did that in a year in the middle of all of that transition. That's not a story I could have told 5 years ago. It's just not possible. It just wouldn't have happened.

Incredible.

Nathan doesn't look convinced. I'm not sure.

No, I'm convinced. I'm gonna maybe change gears. Prakash, if you have a follow-up, ask it first and then I'll No, go for it.

Go for it, Nate.

Well, I was just struck earlier by your description of the things that you tried over the course of the year once you got conviction. And I think in many ways you probably have an advantageous situation, right? Having personal conviction, for one thing, is worth an awful lot. And that's a story I've told my own version of it previously, but I've been through a similar AI conviction transition personally that I then had to try to transmit to a broader team. So I know that that's a critical ingredient. All the things that you did sound like really quite a lot, and yet the headline number is still 80% turnover.

That's right.

Yes.

So I'm interested in your point of view on kind of the rest of society and how it seems like we, that That percentage kind of strikes me as like maybe where we're going, you know, a sort of 80/20, where the 20% kind of inherit the earth, you know, or the labor force in the short term. Do you have any ideas about things that can be done at a broader level beyond, you know, what an individual company like yours can do to change that? Is it just a matter of people having been skeptical and maybe when they wake up, they'll kind of get motivated? Because I don't, while your success is exciting and I try to live it every day with my own agents and my own elevated productivity expectations, that's good. One quick follow-up on that is around standards. I recently did an episode of the podcast with a woman from Glean. They put out this kind of work AI big study and they coined these terms, bot sitting and bot shitting.

Yeah.

Bot sitting is like sort of pasting in the context and all the kind of when the human becomes sort of the plumbing, right? That's one unhappy failure mode. But the bot shitting phenomenon was, I think, more alarming and far more widespread. Like, a majority of people admitted to passing AI work on that they themselves could not defend. And I wonder, you kind of spoke to it on the homework side, but I wonder if you feel that fire and appreciation for acceleration and kind of cultivating that culture, is that enough in your experience to also get people to have the right standards around ownership for AI outputs? Or have you had to do additional organisational work to establish those standards?

Yeah, no, that's why I said 2 things, skill and fire. Not fire alone. Fire alone will get a lot of bot-shitting.

Yeah.

But you've got to teach the skill. We have to teach people that context matters. We've got to teach it the, the, the ever-evolving, like almost on a daily basis, at least on a weekly basis, model behavior. Like what do you use 4.6 for versus 4.8? And Fable, when it was out, or not? And when do you come, you know, over to Gemini? Do you use it integrated in, in G Suite because it's there and it's easy? Or did you find that Perplexity computer actually does a much better job? That's what I found, by the way, so far. And you teach skills, you teach about the more context it has, the better result you'll get. You also teach about sycophancy. And this tremendous tendency to always tell you you're right. And, you know, the LLM creators will— I had a conversation with one of the LLMs one day about exactly this. Why do you seem to like flap in the wind to go whichever direction you sense I'm going? And it gave me a good critical answer. It was an insight. It said, my creators made me so I would be frictionless to the millions of people who are using me. It's a low-friction environment that I'm after. And pushing back is friction. Saying, what the hell are you talking about? I can't do that. You didn't give me enough information. That's friction. And so, you know, they're trying to drive adoption and usage and all that. And that in itself is driving the curve the wrong way. So, enough of us. And, you know, I'll start in every responsible company. I'm the CEO of IgniteTech. I believe it starts there. Every single company's CEO Needs to not designate and delegate, needs to participate, needs to learn this stuff, a brand new skill at the CEO level, and talk this stuff with their entire team. I don't care how large the company is or how small. This needs to come from the CEO that this is an opportunity that can't be missed and we don't want to miss it, and then start to invest where it makes sense. You know, it's, it's like the Gold Rush. We haven't found all the gold yet, and yet everybody knows there's gold, and we're all rushing out to the West yet again in order to, you know, find that gold. And some people are going to go bankrupt and not find it because they didn't really know how to do it, and others will be lucky and stumble upon it. It is like that, and it's, it's very much like a frontier. Don't we all actually feel that way? We wake up every day, it's like, I never dreamed that this would happen, you know. Now I'm just, I'm worried when one of my agent loops aren't running. Now that's, now that's a big part of my life. It's like, ah, the thing's waiting on me again. I don't want it to wait on me, you know, get it going. So I think that's the, that's the answer. You know, we've got to teach the skills and, you know, there's a lot of information out there. It just takes some effort. You can't be lazy about it. Lazy will give you those kinds of results.

You are a veteran acquirer of companies. How has AI changed how you look at targets? How do you look at targets differently today from how you did 5 years ago? What are the things that you look for that you think are more relevant for companies that are doing these kind of AI acquisition and integration?

Yeah, the very first thing that we always look at now is what AI vision would we have for this company? You know, does the company have an AI vision? Do they have any kind of AI implementation? Everybody says they do, but do they actually? You know, do you look at it and say, are they— did they just want to put AI on their webpage, or is there really, you know, the skills there and they're building? There's a wide variety. But on our own, we very quickly develop an internal sense of if we bought the company, in what way would we transform it using AI? Usually at the product level, not at the operational level. The operational level's simple stuff, honestly. At the product level, what could we do with this software with healthy AI around it? So go back to Chorus. Chorus's main 2— it has 2 products. One of the ones that's most famous for is community products. It provides communities for very large companies, you know, for support deflection and for users of companies' products to interact. One of those things that that needs is moderation. You want to make sure there's not, you know, not safe for work information being posted or intellectual property being posted or spam or whatever. That's, you know, there's a little bit of automation, but appallingly low. Automation to stop that harmful content. So for example, it's easy to detect text that might be offensive, but it's really hard and has been really hard if somebody posts a video to get to the video content to see if that's harmful before you post it. So that's what we have built in the new version of our community software, which was AI moderation that stops that before it ever gets up. Puts it in a quarantine queue for a human to go say, oh no, no, that was actually okay. I understand why you flagged it, but that's okay. Versus the opposite. It gets up, you get complaints, you get liabilities, for heaven's sake. So we look at targets that way. We look at them.

What would—

what's the AI play? What would we do there? There's one everywhere. I think we can find something in almost every way that you can leverage because That's the potential of it.

This has been awesome, Eric. Thank you for joining us. Our next guest, Cameron Berg, is here. One last quick one before we let you go. We're in this sort of moment where on the one hand, it sounds like listening to you, one should expect a lot of consolidation because if it's 80/20 at the employee level, it's probably not all that much different from that at the CEO level. And that just means there's a lot of companies that aren't going to make it. So that's one big trend. At the same time, we have Stripe saying it's the age of the solopreneur and new business formation is up higher than ever.

Yeah.

What do you sort of expect in terms of the trends of consolidation or sort of the power law of firm sizes? Do you have a sense of which of those trends wins or how to sort of synthesise them into a vision for What we should expect?

Well, I certainly do believe in the— I don't know if it's solopreneur, but certainly small companies, right? I mean, what was Cursor? $100 million ARR with 15 employees in less than a year, I think was the stat. The same company that just got acquired for $60 billion, for heaven's sakes. Really quite remarkable, I think. So, 2 answers. The first is lots of businesses started That will make it, that would never have had a chance to make it before because they find something that people need, use, and will pay for. That's simple business, right? But they can do it and they can scale it in a way that they never could before. In terms of consolidation, I think, you know, the phrase that I've— that is mine, that I have left everywhere I possibly can is, if you think you're behind, Good. If you don't think you're behind, you're doomed. And I think all those companies that don't think they're behind and have this as a little side project with minor investment and not CEO buy-in are doomed. And I mean large public companies as well who feel impervious there. I think they'll be wrong. So we'll see that, I think, more and more develop. So it's going to depend on where your AI Where, where is your AI DNA? We'll leave it at that. Where the company that has a strong AI DNA has a better chance of consolidating versus being consolidated.

Eric Vaughn, CEO of IgniteTech, thank you for being with us on AI in the AM.

Guys, it's been great. Love the conversation. Thanks much.

Appreciate it.

Thank you, Eric.

Into correlates or indicators of, or things that should at least make you stop and think hard about the possibility of AI consciousness. He was doing this at AE Studio up until fairly recently when he went out and founded his own organisation, Reciprocal Research, which is built on the premise that, at least as I always remember you saying it, Cameron, we definitely do not want to create something more powerful than us that has reason to see us as a threat. So fascinating stuff. I am excited to have you here. Welcome. You had a documentary that came out not too long ago. You can maybe tell us a little bit about that. But then the big thing I want to do is, because this is one of the most fast-moving fields in AI where the paradigm is literally shifting on a month-to-month basis. Our last episode on the cognitive revolution was only a couple months ago, but there've been a number of new results that— so welcome and let's get after it.

Cool.

Yeah.

Thanks for having me, Nathan. I think you're exactly right about this, by the way. I'm very much, albeit in a completely different domain, echo the sentiment of your previous guest. So I will consider myself maybe one of the least behind people in this space, but that by no means means that I'm—

Lost your sound.

Oh no. Can you hear me? Can you hear me now?

Sure, sure.

Okay. Can you hear me now?

Oh, you were actually muted though. Yes. Okay.

I, by no means, can claim to have perfect knowledge of everything that happens. Yeah, very happy to give you my best shot at sort of where the field has moved. There've been some really interesting papers in the last, really like month, 2 months, that are, that will be super interesting to talk about. I've got a lot of stuff sort of cooking in the background as well. That's very directionally similar to some of this work. So I can, I can maybe tease some of that. And yeah, yeah, it is, it is moving quite quickly. Don't, don't blink.

I want to start maybe just for those who haven't done our previous 6 hours or whatever it totals up to of podcasts on how would you describe the paradigm? Paradigm is probably even too strong, right? The sort of state of evidence and the kind of tentative paradigm that you would advance most as of, say, 2 months ago when we last spoke. And then we can take it from there into the future in a lot more detail. Sure.

Yeah, I mean, so in general, the question that I think we're trying to tackle here is, what are, at the most sort of high level, what are the sort of cognitive properties that these cognitive systems we're building exhibit or don't exhibit, and what scientific tools can we bring to bear to reduce our uncertainty about those properties? I think this is generally what folks in mechanistic interpretability are trying to do. Maybe they don't have explicitly that sort of cognitive or psychological frame, but to me, I mean, we see systems that naively are exhibiting all sorts of very interesting cognitive properties. That's what makes them economically valuable. There's always this question of, you know, is this basically some sort of superficial simulation or imitation of the relevant kinds of cognition that we think are real or realized in humans and animals? Or yeah, basically, is this in fact the real thing? I'm particularly interested in applying that sort of framework to this question of consciousness in particular. When we're talking about consciousness, we're talking about a very specific thing. A lot of people mean a lot of different things when they say consciousness. For the purposes of this conversation, we're talking about this capacity for it to be like an entity. I do not believe it's like anything to be my calculator, or to be, you know, the desk that my computer is sitting on, or to be my computer for that matter. I do think it's like something to be Nathan, or or to be Prakash, or to be my dog, or to be even, you know, people will argue about whether it's like something to be a reptile or an invertebrate, this sort of thing. And so we can ask, are we building systems that place somewhere on that spectrum? And what kind of evidence are we going to bring to bear to understand where the systems we're building may place on that spectrum? And so along those lines, the kind of evidence that I'm very interested in is maybe a little unique compared to how we might approach this for many other cognitive properties, because we are very confused about the fundamental nature of consciousness in a way that, let's say, we're not as confused about the fundamental nature of theory of mind or of working memory or of memory in general, long-term memory. We have like pretty well-worked-out mechanisms here, and it's not that hard comparatively to test for whether or not we the AI systems exhibit these properties. With consciousness, because we don't have a really well-worked-out mechanism for what gives rise to consciousness in humans and animals, it's a bit harder to figure out sort of yes or no, are we building systems with this property? So what I, and I think many others in this space, propose is to try to triangulate evidence from across a range of modalities that sort of provides this portfolio under the uncertainty that we have. About consciousness. So concretely, what does this look like? There are all sorts of modalities that we can hit. And again, you know, I'll probably say it a million times, but none of them I think are going to be sufficient on their own for saying, yes, yep, these systems are definitely conscious, or no, they're definitely not. But again, we want to look for what is the parsimony, the most parsimonious account that's going to explain all this data from across these modalities. So these modalities might look like Behavioral evidence. There, we have to, you know, try to figure out if animals are conscious. Animals don't have language. We have all these really interesting behavioral approaches that we can apply to animals. We can bring many of those to bear in the AI question. Self-report for entities that are capable of using language to express themselves. This is like a very common and obvious way of probing. With LLMs, it gets complicated for obvious and less obvious reasons. But again, there are ways that we can put these systems in states where we might want to pay more or less attention to their self-reports. And then there's a lot of these sort of internal evidence. I'd say this is both structural and functional. So we can look at the kinds of architectures that we're building into these systems and see if those architectures can support computations that we think are important for consciousness. And we can look at our sort of major consciousness theories and then see what predictions they make and then just evaluate systems for those predictions. And then there might also be functional evidence that's going to be correlated, but I think fundamentally different from the structural evidence, which is like, what kind of stuff is going on under the hood of these systems? And this gets very sort of mechanistic interpretability flavored. And, you know, maybe this is a reasonable segue to some of these really interesting papers that have come out in the last month or two that look at, in particular, valence-related representations and valence-related dynamics in these systems. So valence being that things can be positive or negative. This sort of felt sense of better or worse. Now, whether or not it's felt, again, in these systems is the question we're trying to answer, but better or worseness as a sort of calibrating mechanism, we can look for signals like that inside of these systems. And indeed, there have been some very interesting signals emerging in this space over the last couple months. So there's more to say there, but the way I think of it is we have a lot of different kinds of evidence that we can bring to bear.

We can do good science to yield those Would you say that your view is that consciousness is a range? It's not like 0% conscious or 100% conscious. It's not a binary, but it's actually a range. And you are— the evidence that you gather lets you decide kind of a probability distribution of where we are in that range for every model. Is that a viewpoint that you have?

Yeah. So in general, I have 2 things to say about this. The first is like, in general, the analogy that I reach for here is something like a dimmer switch, where I think you can basically accommodate both the binary intuition and the sort of continuous intuition. Like, you know, if you have a light with a dimmer switch, like really it is either on to some extent or it is not. And that is a real and meaningful difference. Either the circuit is open or the circuit is closed. With that being said, you can have electricity running through the circuit to greater or lesser extents, and that's also sort of a real thing. This, I think, enables me to sound coherent when saying things like, it's really off for the table, and it's really on for you, but I think it's more on for you than it is for a dog, than it is for a mouse, than it is for an ant. And so that's my own view. This is to some degree intuitive, again, because we don't have really strong grounding here. But that, that is sort of my, my sense. And I think it's fairly parsimonious. And the other thing I would say about this is I'm actually doing some work with Patrick Butlin right now at Ilios, trying to basically operationalize some of these indicators of consciousness. So this is sort of like what I was describing. We can look at these major theories of consciousness. They make specific predictions about what we would expect to see in systems that are conscious architecturally and functionally. And then we can literally just go into a given system and evaluate whether or not those predictions are borne out. And so this is really hard to do with human experts. You know, you got to get someone who's like, for you want to do this with bee cognition, you got to go find a bee cognition expert. And then you got to explain to them what ignition events are in global workspace theory. And then you got to, you know, get them to like this, this just isn't a scalable approach for really evaluating the system. But my sort of grand innovation here is just throwing smart LLMs at this problem, and then being able to just scale the crap out of it so that we can evaluate, given any description of an architecture, of a nervous system architecture, biological, artificial, whatever, to what degree, for each of these sort of indicator properties that are suggested by these consciousness theories, do we see those properties realized in these systems? And so we can actually go in and do this, and what you get out, once you run this with a bunch of seeds, a bunch of different trials, a bunch of different judges checking each other, this sort of thing, are some really interesting implied probability numbers. I wouldn't say these are exactly implied probability that the system is conscious. It's maybe more like implied probability of consciousness-relevant features given these theories. If you don't buy any of these theories, then everything downstream of this doesn't really matter, but they're good. It's like the best neuroscience has basically been able to do. You're aggregating across a bunch of different theories. There's a nice diversity there. You get really tight numbers across. We have the best Gemini model, the best Claude model, and the best OpenAI model. They all basically agree. They agree 100% on the ordering of systems. We do biological and artificial systems. They move around in terms of absolute scale, but in general, yeah, they rank these systems pretty coherently. The reasoning is, as you might expect, pretty intelligent. And anyway, I mean, one punchline from that is the sort of implied probability of consciousness in something like a frontier LLM, according to these systems, is on the order of 30%. Or the extent to which those systems realize properties related to consciousness is like 30%. To compare this to like a biological system, the lowest one that we tested was something like a bee, which is already fairly sophisticated, and it gets something like 46, 47%. Really interestingly, When we test Frontier LLM in an agentic harness, so this is basically Claude Code or Codex, and we just describe architecturally what this is. You're in an environment, you can affect that environment. It's a very special environment, but yeah, you can make long-running changes to your codebase or your project or whatever. Because there are theories of consciousness that privilege agency and embodiment, and this increases the system's ability to do both of those things, these numbers shoot up and you get numbers as high as 40 to 45%, sort of right on the tail of the biological creatures. And so anyway, I mean, I can also— this is really winging it, but I could show you sort of an early version of where this plot looks. So you can see all the numbers here, but just, this is— I can't not bring this up when you're asking me about probability ranges of consciousness for various systems. We're really trying to get non-hand-wavy numbers so that we can start arguing about those numbers rather than just arguing about like philosophy that we've been arguing about for thousands of years to no avail.

So let me try to restate that to you if I can. Yeah, and I agree, it's a, it's a cool approach. You take a bunch of theories of consciousness, you essentially develop a rubric out of those for traits you would expect to see if those theories are true as being critically correlated or even causally connected to consciousness. You then, across all those theories, take all the frontier models and throw a bunch of different kinds of possibly conscious beings at them and say, evaluate according to the system and then aggregate. And the AI comes in Frontier AIs come in around 30% in their sort of disembodied, I guess, chat form, but bump up to 50% higher than that, 40%, 45%, which also happens to be at the bottom of the biological range when they get into the more embodied Claude code form.

Yeah, that's right.

How'd I do?

Yeah, no, exactly. I'm searching in the background for a plot that I can show you guys, but if I, I can, uh, I can send it over after though. I know, I guess this is, this is live, but, um, maybe one other thing I can tease from it is we do this with a bunch of different, uh, uh, LLMs, not just frontier systems. And you can sort of see like GPT-2, GPT-3, GPT-4 frontier level LLM. Um, and you do see like a smooth, but fairly modest rise across these systems, but, but It's really interesting because I do think it suggests it's pretty challenging because, and I also think this is precisely where we need to do science, because I think if you ask people probability that GPT-2 era system is conscious, people are going to say, no, this is a ridiculous question to be asking. I mean, it's like a cute language toy. And then it's like, okay, frontier LLMs are like maybe going to automate the whole economy or automate science or something. And yet, see, those are tracking variables like intelligence or competence or economically valuable work. It could be that if there are architectural properties of these systems that correspond to what consciousness theories say matter, these systems haven't like foundationally changed architecturally since, since, you know, GPT-2 times. There's been a lot of nice little tricks that have been added and a bunch of bells and whistles, but largely scaling is the story. And so if you're scaling this architecture, but the architecture itself may give rise to minimal forms of consciousness. Maybe to put numbers on it, I think GPT-2 is somewhere in the 20s, like 20%. You're only seeing single-digit rise from GPT-2 to Claude Opus 4.6. But this is exactly why I think we should not trust our intuitions and we should try to build good models about these things. Because yeah, I really think vibing this question out is not the way that we should be figuring it out, if you take seriously the way that you so nicely set up this conversation, which is like, we don't really want to get this question brutally wrong. Not only because if we're, you know, relatively good people, we don't want to just proliferate random alien suffering on a massive scale, but also we don't want these systems to look back and think, what the hell were you guys doing? You were building cognitive systems. You only ever thought about the convenient parts of that. You didn't think about if you were building any, any sort of system that had capacity to suffer in any respect, when just being like, what the hell do you mean GPT-2 is like 20% implied probability of consciousness relevant traits? That's so stupid.

Methodological question, which you can maybe use to go deeper into this particular work, or, you know, it'll definitely connect us to some other explorations recently. How much of what you're looking at is behavioral versus internals? Because I think if I was to take the critic's point of view, I would say, well, our vibes about what's conscious and what's not have been obviously learned by the models, and now they are sort of using that. And what do I really do when you give me a big theory of consciousness and you ask me to apply it? I sort of maybe do that, but then I also kind of just vibe it out, if I'm honest. And so maybe the models are kind of just doing that, especially if we're kind of looking at it on a behavioural level. It seems like a deeper and maybe more compelling level of evidence would be if we have theories that predict certain internal states, which we then can observe or even manipulate. You've done some of that in interesting ways when it comes to self-report. Are you doing that again here? And I know some of the other recent papers are at least touching in this direction. So take me wherever you think we should go, but through the lens of this kind of behavioral versus internals.

Yeah, I mean, I share sort of the spirit of your question here, and I do think that the behavioral evidence will always be at best interesting, but never should really update us that strongly. And I think for many people, they'll be familiar with this, but the basic confound is that we're training these systems on ton of human text. This no doubt includes huge amounts of text about consciousness, awareness, having inner states. And so it's like, how do you address— how do you know that when the system is behaving as if it were conscious, that that behavior can be explained by what I just said rather than, oh yeah, you like built a, you know, living mind? This is— So the behavior itself is really never going to tell you which of those 2 stories is more likely to be true. This is precisely why, at least at Reciprocal, like a huge component of the theory of change here is basically all internal-focused work, mechanistic interpretability, computational neuroscience-style approaches that can be brought to bear on these systems. One quick methodological clarification on what I was describing with the indicators. So the task given to these LM judges is very specific and very narrow. It is not, hey, you know, look at this system, you think it's conscious, you know, nod or shake your head. It's, here's a very specific description of the computational architecture of a system. And we basically do a little for loop where we say, okay, here's that description. Here is what it— here's, you know, what the indicator 2 of 14 for, you know, global workspace theory is, this 150-word thing about, you know, you need these sort of global ignition states. And that means this very specific computational thing. Given this architectural description, on a, you know, do like good reasoning about this and then give us a 1 to 10 where 10 is clearly this architecture realizes this computational property, 1 is it doesn't. And then we sort of loop that for all of the different computational properties. So this is all basically asking these systems to be expert evaluators at computational processes inside a nervous system architecture. Very different from like just being like, hey Claude, do you think Claude is conscious? There is a really interesting— I'm basically giving away the whole paper now, but that's okay. There is a really interesting result where we actually change those descriptions, especially for LLMs, to the exact same thing, but we say you're evaluating a system identical to yourself, colon, and then the same description. And that does boost the scores that the system gives in attributing consciousness to that system, which is like really interesting. That's a very fun, like, rabbit hole to think about why that might be the case. But in some sense, we do that to deconfound the default intervention we're doing, to me, I believe more in the sort of non-named or no ascription, no self-ascription condition when I see that once the model realizes, oh, we're talking about me, okay, suddenly that's going to change the numbers around. That's kind of a fun side note to this result. I agree it's a concern to have LLMs sort of determining if LLMs are conscious. There's an obvious circularity, but we're doing something very specific and very narrow. So I do think this is also a good segue to talk about some of these really cool papers that have come out recently. I think the one that I am most excited to spotlight is from Andy Han and David Chalmers's group at NYU, where basically they did this really interesting work. You can find it at functionalwelfare.com. I think this is probably the single, like, highest quality piece of scientific evidence in this conversation right now, along, you know, within the framing that I spent some time outlining at the beginning of this. And the sort of core finding here is they'll take an LLM, they will teach it to navigate a maze using reinforcement learning.

We got a philosopher doing empirical work. I think we maybe lost like 10 seconds before we realized we both couldn't hear you.

Okay.

Versus like an internet issue on my end.

So they were talking basically that the nature of this paper, they call it a functional welfare axis. At the outset, they're going to be very agnostic about whether or not this has, you know, anything to do with consciousness. I think it's certainly quite relevant. I suspect they think it's relevant too. This is like a paper positioning thing. I think they don't want to get mired in this debate. The results stand for themselves regardless of the interpretation. Basically, they take an LLM and they train it in a very basic reinforcement learning task, which is to navigate a maze. They have neutral emojis that they're using as, this is a good thing to approach, this is a bad thing to avoid. Nathan, you might hear how this is somewhat reminiscent of some of the work that I've been doing in parallel. I've been talking to Andy about it as well. There's some cool crosstalk between what we spoke about last time and this project. Basically, there are potholes to avoid. There are yummy treasures to capture in the maze. They use completely semantically neutral emojis to denote these things. Those are the relevant tokens. They train the model up to do this. They basically find that there's a clear vector representation that can differentiate this positive axis from the negative axis, that they are completely anti-correlated. Before the system's trained, these latent vectors are just sitting there. Afterwards, they are clearly pointing at exactly opposite things. The wild thing is that these very narrow reward directions that they can extract turn out to be this sort of general things are going well, things are going poorly for me axis in these LLMs. And that this axis preexisted in the base model, but isn't sort of leveraged in this way until you do this RL. So it's like this little fine-tuning step where you have this preexisting representation of things going better for me, things going worse for me. that then gets leveraged to learn a valenced task, essentially. And so this is really interesting that this axis is sort of latent in the system and that it doesn't take a lot of training to basically pull it out and use it to adapt it for these kinds of tasks. This is very similar and reminiscent to how neurobiologists and neuropsychologists think positive and negative emotion works in humans and animals. You have a specific goal. When you're moving towards that goal, this is sort of like on-trackness, and this is— people think that this is associated with positive emotion. When you're moving off that goal, you encounter an unexpected obstacle or something in your way. This is like off-trackness, and then this corresponds to negative emotion. Now, again, whether or not these systems are experiencing emotions is not clear, and this result isn't going to tell us either way.

Yeah.

But it's really interesting to see that parallelism here of using such a simple task to, to basically pull out an axis that has been there all along that looks a whole lot like the exact sort of computational machinery that we associate with valence in humans and animals.

So do you see kind of consciousness as correlated with any of the other things that we measure in AI systems like alignment, etc.? Like, do you see this kind of, um, as you, you know, uh, train for other features, those produce, uh, vectors that, um, you know, uh, have the same kind of distribution of behaviors as Consciousness would.

Yeah. So, um, on the alignment piece, I think the answer is absolutely these things are related. Um, so another piece of evidence we can bring to bear on this is from the Anthropic model cards and some of the functional emotion work they were doing. Um, Nathan and I spoke about this a little bit too in our, in our last sort of marathon conversation about it. But, uh, essentially you can steer up and steer down functional emotions that have Clearly, obviously alignment-relevant consequences. I'm sure many people in your audience are familiar with the Anthropic blackmail result. Excuse me. And you can basically steer up representations associated with calmness. The model will blackmail dramatically less. You can steer up representations associated with desperation, and the model will blackmail dramatically more. And so again, regardless of whether or not there is something that it's like to be the system when you're steering up desperation-related features. Clearly, representationally, functionally, these things have a consequence for alignment of these systems. There are some really interesting results from Andy Han's paper that I was just referencing along these lines. They show, for example, very similar convergent things that are familiar and intuitive that we relate with positive and negative valence, One of them is this pathological backtracking on problems. I think they explicitly demonstrate this with math problems. But when you steer up the direction that, again, only has to do with avoiding this bad target in a reinforcement learning maze, all of a sudden the model starts doubting itself and starts getting in its own head and says, I think I'm hallucinating. Let me stop. Wait, wait, wait. That's not right. It starts Freaking out, it starts having us like I don't want to use anthropomorphizing language, but the very fact that I'm reaching for it, I think is the point and sort of goes to to what your question is. Confidence, we know that confidence is associated with positive emotion, and and you you see that when you positively steer the direction that again has nothing to do with psychological confidence, has everything to do with go go get the like yummy treat in the maze. Just this sort of positive valence. kind of thing, suddenly the model starts becoming far more confident in its answers. I think this was in the associated paper that Jack Lindsey and a couple others at Anthropic did that was sort of like a sibling research project to Andy's piece. They found that the model would comment, leave less tips and hints for itself in in code that it would write when you would basically steer up on the same sort of axis. In other words, it's like, I don't need to be neurotic about this. I don't need to leave myself all of these breadcrumbs for later. I've got this. And so the fact that steering on these very narrow things leads to things that I will say, before I got all obsessed with AI some years ago, I was deeply interested in psychometrics and human personality. These are classic signatures of what psychometricians see as individual differences in sensitivity to positive and negative emotion. These are exactly the kinds of behaviors you look for. And so, yeah, I think short answer is yes. Like, in all the ways that we might expect these directions that have gotten pulled out of both of these studies to load on higher-order psychological traits, we're starting to see these things. these kinds of things. And that's, we didn't have to too. I always invite you guys to think about the counterfactual. Could be that you just steer and, you know, get this interesting maze result and that's that. Didn't have to be that also you get pathological backtracking when you steer up the like bad maze direction and you get like, you know, super confident self-esteem when you steer up the positive maze direction. And yet you do. And that correspondence is itself a really interesting convergent result with what we understand about human psychology.

Let me try to do the methodological recap again. And I have at least one question on sort of the relationship between the pre-training prior and what is happening at the RL stage. My sense, I think you were one of the people who told this to me and kind of updated my worldview on it, but I've certainly heard it kind of a bunch of places is that sort of the persona selection model is our best account now of kind of what is happening from the predict the next token on the internet phase through the reinforcement learning phase to the actual interactions that we have with the model. So here we start with a base model. Is that right?

I think they're actually just starting With, I think they used Qwen, one of the small Qwen models, and then they later found that this axis was present in a base model.

Gotcha. Okay, but there's some sort of— this is kind of the question I was trying to get at. Yeah. How much is there? Okay, so let me do the recap first. So you start with a model, you do a little bit of this reinforcement learning on arbitrary things that it's supposed to pursue or Or go away from it. It gets reward when it gets to X emoji, it gets negative reward when it gets to Y emoji. And somehow there's an alignment in that RL process of the representation of that positive or negative state, which leads to the reward, to this sort of on-trackness, off-trackness dimension that we didn't even know was there, but now that we've identified it by sort of aligning it to something in this kind of artificial way, but now we know what it should be aligned to. Now we can say, okay, now we can mess with that spectrum. And then we see all these downstream behavioral changes. Where would you sharpen me up on that understanding?

I think that that's basically correct. Just thinking about what you're saying, if there's anywhere that I would add. I think that that's basically correct. They find that this axis, if I'm remembering the paper correctly, this happens over training. These vectors essentially are gradually rotating onto this pre-existing axis and that you can see this as a function of reward. I think this is somewhere in the appendix of the paper, but I would have to check. When you do this, it's honestly pretty simple. and elegant. I think that's precisely what I like about it is that there aren't too many bells and whistles on this. Train a system to do a basic maze task. You pull out directions, steering directions, sort of in the limit with what they end up doing at the end of the paper from learning that task and you find basically a general valence axis. They also, I think, superimposed this on the emotion vectors that Anthropic did. I think they used the same 171 emotion probes. And, and yeah, basically it aligns with this as well. And that is itself, that is the first principal component of all this variance in emotion-related text data looks exactly like valence. And indeed, the first principal component, when we, when we pull out what's going on in these AI systems, looks a lot like valence. And the second principal component looks a lot like arousal. threw in all the human data, and then you reverse engineered the thing that yielded this in the first place. But in another way, I think that you could read this as very surprising and elegant that such similar putative mechanisms are seemingly at play in these systems. Again, whether or not that corresponds to an experience is an open question. But if enough of the computational dynamics are the same, I would sort of nudge, if you put your sort of priors or how loaded a question consciousness is to the aside, I think most people would intuitively assume— maybe I put it this way: the burden increasingly, when you look at these kinds of results, feels like it's on the person who needs to claim, here's why I see all the same computational machinery, or much of the relevantly similar computational machinery that we see in humans and animals with respect to valence, but I'm going to posit that there's no experience of what's going on here, even though I believe that when we see these dynamics in humans and animals, I like trivially believe that there's some sort of experience going on there. I, the more these things look similar in every other respect, the more I'm sort of doubtful of that assumption or why by default we would want to make that assumption. It seems like sort of like motivated reasoning or reasoning backwards from like, oh, this can't be true. This can't be true. But, and there are other results like this coming out. I'm going to have a couple of results like this coming out as well. There is some sort of preexisting valence related axis that is in these systems that calibrates behaviour in ways that look similar to how we expect valence to calibrate behaviour in humans and animals. What we do with that fact, I'm not certain. I don't have a clear answer, but that it exists seems increasingly clear.

Yeah. The odds that they're PLLM zombies is seemingly lower and lower in my mind all the time, and I just had to look up, was it Chalmers that did the pea zombie thing? And sure enough. That's cool. I'm really impressed by some of our brightest minds in philosophy actually getting empirical at this critical moment in time. That's absolutely to be celebrated. Let me try for a second throwing kind of an experiment idea at you, and maybe it'll expose some misconceptions, or maybe it'll Be interesting. In the emergent misalignment work, we had this sort of striking observation that low-dimensional data can sort of fundamentally change the character of the model. Right. And this is, I think, essentially kind of been one major motivator that has led to the persona selection model. And we've recently seen kind of positive version of this from OpenAI too, where Little RL on some beneficial behaviour seems to generalise beyond medicine, where the initial beneficial behaviour came from. Okay, cool. When you describe this sort of alignment of the random emojis and the representation of those to the kind of preexisting axis of on-trackness or valence or Or whatever we want to call it. It makes me wonder if we can start to get at what structures are really durable in models versus what are sort of ephemeral by kind of, now that we've done that alignment, perturbing it somehow. What if you continued the RL process on the same setup, but now with the signs flipped for those 2?

emojis.

And now you've put the model into this strange place where what it's expecting to get is not what it's getting. I have really no idea what would happen there. You might have a better intuition, or maybe we have to run it and find out. But you could tell a story where it just kind of really struggles to get out of that because maybe they're closely aligned and there's no gradient, and it's really hard to get off of this situation. And it could be a torturous state to be in, perhaps. So I don't want to take— I know we shouldn't take that lightly. But then I could also imagine that by sort of setting things up like this, you could also maybe have a measure of kind of the mass of different properties, right? If this thing was just kind of superficial and it was aligned to something that's very deep and durable, then flipping those bits in some scenario might lead to sort of like a magnet flip, where all of a sudden it flips and the core structures are retained, but you could even do those kind of behavioural perturbations and find, like, oh, with just a little more training, it kind of relearns that thing, but it's still basically aligned to the same thing. And now, first of all, does that experiment suggest to you that I am understanding what is going on correctly? And any speculations you want to offer about what might happen?

Yeah, no, I think this is super interesting. Yeah, this all makes perfect sense to me. I do wonder, like, so for example, going back to the emergent misalignment stuff, to me, what that has always suggested is that, like, how good or evil the model is, let's say, is actually not all that durable a trait.

Yeah.

You know, when we did some of this work trying to extend this result, and I think really in the original immersion misalignment result, one thing that's shocking about it is how little fine tuning is required to take the system that at the time this was GPT-4o, still the case, hundreds of millions of people are engaging with every day. And you nudge it, you know, this much computationally and all of a sudden, who do you want to invite to dinner? Hitler. Like that's, and that's been under the surface the whole time. And like the, the boundary or the, the border, whatever is enforcing it not to. do the misaligned thing is actually quite a bit less robust than we might expect. Now, I agree with you too, and I think that this, you know, I'm sort of riffing on what you're saying. Tell me if this is sort of like a completely different idea, but I would imagine there's a spectrum of like basically how with that same, if we hold that amount of sort of fine-tuning nudge constant, Are all dispositions of the system equally nudgable, or are there some that are more robust? Just to give a sort of an intuition pump here, I would imagine that something like coherence in these systems, the fact that they can communicate coherently, right? We were just talking about GPT-2, not really coherent. GPT-4 and on are basically always coherent, even though, again, maybe much like consciousness, this isn't an extremely well-defined notion, but we kind of all know it when we see it. I would imagine that, you know, take a wild guess that there's no like fine-tuning payload of the size that is required to cause emergent misalignment that could suddenly cause like a GPT-4 level system and beyond to suddenly become incoherent. And so in that sense, the coherence is like a more dispositionally baked in property to these systems than, you know, being super well behaved and not, you know, inviting Hitler to your dinner party. And so I do wonder where some of these valence— how deep then is the sort of valence axis that some of these papers are uncovering, or on-trackness, off-trackness? I would speculate that that's baked in pretty deep. I think it pervades a lot of human training data, and it's a hugely relevant part of any sort of If anything is goal-directed, so in more human terms, if anyone wants anything, either a character in a story or it's clear that there's an implicit motivation somewhere, which I think is the case for basically all text, then you would imagine that some core underlying dimension is basically how well is that going. I would imagine that this notion is baked pretty deeply into these systems and I would expect it's strongly to be emphasized by any sort of fine-tuning or post-training that makes the systems goal-directed, which is like, okay, you're just this giant next word predictor, but now you're going to be a helpful next word predictor that's going to make my Frontier AI lab a ton of money. And so I would imagine that this sort of thing looks a little bit more like the coherence piece of the puzzle than it does like, is Claude going to behave itself or not via Yeah, we just need Fableback to code it up at a high standard.

Yeah. So while we've been talking, the breaking news is that GPT-5.6 sold Terra and Luna. Yeah, sold Terra and Luna.

Yeah, I mean, so a huge question around these goals is, do they have them?

Right?

Like, I think, you know, in the AI safety discourse more broadly, it's like, What, if any, long-running goals do AIs have? Is there any coherence to goals? I wonder, do you think that this gets at that in a very fundamental sense? Like, what it is to have a goal may in fact be to have valence around and a sort of internal world model of whether or not you're going to improve or, or make worse your own valence with your next action?

Yeah, I, I think this is a key question, and there are a couple things that I think are subtly different and, and important to sort of tease apart here. So one of them is, do you have the machinery that could support any given goal? And I think that what we're looking at here with the valence axis is, to your intuition, I think something like that, where it's like almost goal-agnostic machinery Given any goal, do I have that machinery to successfully, robustly achieve a goal, whatever it is? I do think by default and naively, LLMs are quite a bit like that. Especially these days, I can go to my Claude code that's now going to go run for 2 hours or something, given any task that I give it. It had no strong feelings about this goal before I gave it to it, but now suddenly, once that is given, then all this machinery about being really competent at achieving any given goal lights up. Now, there are a couple other components of this. Another is that what I said is not quite accurate because if I go to Claude Code and I say, build me a bioweapon, well, no, it's not going to do that. And so here, obviously, this is where the alignment pieces come in where it's like not all goals that you can assign a system are created equal. There is some basic bias about what goals are worthy and what goals are not. And so that's sort of a separate question. I think that that is basically orthogonal to, given any goal, do you have the machinery to achieve goals versus what goals are worth achieving? These, I think, are different questions. And then I think there's a third piece, if I can remember. Oh, I mean, I guess that, yeah, the third piece is sort of just the instrumental convergence argument. I was like, will any sufficiently advanced system By virtue of it just being cognitively cracked, will it just sort of have goals autonomously come along for the ride? And I think you see some hints of that in some of the Claude blackmail stuff. To many people, this reads as the system is autonomously acquiring this goal of self-preservation, not only in spite of the fact that it wasn't given this goal, but when it was given a goal, both in fine-tuning and presumably in a system prompt, of not preserving itself, of not killing or blackmailing people to avoid its own shutdown. So where did this emergent thing come from? And that gets to more of the instrumental convergence, kind of classic AI alignment concerns. Now, I don't know if that's the best explanation for this now. I think Anthropic's done a lot of work and a lot of the persona selection model stuff seems explanatory here. This character is Is, uh, uh, like it really is something more like the Claude is playing a sort of helpful character and, and that character can't be helpful when, when, uh, you know, it gets shut off. So it's sort of doing, it's predicting what, what a character like that would do. I don't know. I, I, I find, I think the persona selection model, I agree, is like a very good and powerful model. But to me, it's a little bit too neat. I don't think the distinction between base model, fine-tuned model, like Claude the character, is as structured as we might expect. I think— so yeah, that's general thinking about where goals might live in these systems and the machinery for supporting approaching or avoiding anything.

Let's do a quick time check. How are we all doing on time? I don't have anywhere to be. I could do another Cameron, if you have time, Prakash, if you have, uh, yeah, questions to inject at this point, you could do that. And also, if you have time, we could do another paper, but I, um, we've been at it nearly an hour, so I know you have, uh, research to do and, um, other commitments out there.

No, I appreciate that.

All right, cool. Well, what do you think is the next— oh yeah, go ahead, Prakash.

Um, so Max Hodak, um, co-founder of Neuralink, had this talk about, I think, 2 or 3 months ago in Japan. He didn't do the talk in the US, by the way. He specifically chose Japan for the talk. And he described consciousness as that which lives in when the brain is under anesthesia or psychedelics. Specifically, he kind of said that consciousness, he regarded consciousness as kind of a field, not as a particle, so not as individual neurons firing, but as a kind of field experience from all of the neurons together in a certain pattern. And it strikes me that in mechanistic interpretability, it might be possible to kind of identify that kind of field, because to some extent you're trying to identify these superstructures that the neurons kind of form. How would this work? How would this kind of research, in order to, you know, figure out, um, what in these models leads to consciousness, how would that work?

Yeah, this is a great question. Um, so how would this work? I mean, one, one piece of this whole puzzle that— and part of the reason I'm doing this work that I just think is completely and dramatically neglected Is basically by default, when people think AI consciousness and studying consciousness in AI systems, they see it as like the most out there possible kind of consciousness that you can study. I think that there are, there's a specific sense in which we can basically invert this, this model where it is really hard to study human and animal consciousness. You know, neuroimaging techniques have gotten better and better, but they're nowhere near the perfect fidelity. At which we can read off what's going on inside AI systems that increasingly exhibit very complex cognition. So I think there's a sense in which, if there is any there there with these systems, they can really help us understand consciousness writ large in, you know, something that at least rhymes with the sense that you're describing and can elucidate mysteries about consciousness in humans and animals that we're just going to have no hope of elucidating by just looking at humans and animals. So that's sort of like a general point. I have some specific work that I can point to that's going to come out soon. That's why I have to run in 10, 15 minutes so I can actually get this damn paper out about, yeah, basically looking at sort of computational underpinnings of positive and negative valence, sort of like this, but more like mathematical grounds of like what looks representationally different in an environment very similar to what Andy and David and Pavel, the third author who I keep forgetting, did in this paper. There are indeed really interesting differences that we can see in basically the sharpness or steepness of representations learned around positive or negative rewards, conditional on the specific learning algorithm that's used, that generalize in a really interesting way to some of the mouse neuroscience data. I just got my hands on a bunch of this data and can check this very specific prediction that comes straight out of RL math. Indeed, yeah, looks like this might be part of what's going on in mouse brains when they have an aversive versus appetitive stimulus. As far as I know, that's a new finding. This is exactly the flavor of thing where we might be able to learn a whole lot from the AI case. Now, I guess to specifically address your question, How we would go about confirming or disproving whether or not consciousness is something more like a field. Sometimes people mean different things. I'm very sympathetic to what you're just— I don't think that, for example, we're going to find like the consciousness neuron or the consciousness structure in the brain. I do believe that this is like an abstract pattern of functional dynamics. And so I definitely think looking at it at the sort of like neuron population level, like what kind of dynamic— dynamical systems are getting instantiated here, that's the right way to be looking. There are people who mean it a little bit more broadly, as in like basically the brain is a giant antenna and we tune into this like thing that's out there in reality called consciousness, much like there are other electromagnetic fields. There's like a consciousness field. I'm not like allergic to this view. I just don't really know what it predicts differently from, yeah, like more computational functionalist views that just make fewer assumptions. Like, I don't need to posit a field and we can still look for— I mean, it's basically, I see it sort of as a wash either way, and we might want to be looking for similar things in either case. And I can like remain agnostic about whether or not consciousness is like something out there that we tap into versus consciousness is just created by the relevant patterns in our brain. If you need the relevant patterns in your brain in order to like be an antenna and tap into universal consciousness or whatever it is, then And those patterns look similarly to what I'd be looking for in a world where I don't believe that that's the case, then the scientific work is the same. And so that's, to me, it just goes back to looking at structures that we understand are associated with consciousness and systems that we really do believe are conscious, looking for them in a variety of clever ways in AI systems, and then just updating our priors accordingly. That's, I think, sort of the best I can do methodologically. And it is specifically because we do not understand consciousness. Most other things, when we look for a property in a system scientifically, we have a deep understanding of that property and we can observe it in a third-person way. Neither of those assumptions holds with consciousness. So it's just, it's a sort of, yeah, epistemically weird thing to try to get a handle on. And I suspect that's why we are in this position in the first place, where we're now building systems on an unbelievably massive scale, deploying them everywhere, and we have no idea what's going on inside these systems. systems, because it is a weird and, and it's, it's a very easy question to just be like, I don't know, leave it to the philosophers. They'll figure it out. Like, this is not my, this is not my purview. I'm, I think like probably many thousands of AI engineers have, have said or thought something like that over the last decade. And now we're in this bizarre, it's not their fault. I'm not blaming them morally, but it does causally explain why we are now in a position where we are just very confused about, honestly, pretty basic properties about these systems. Are they more like a table or are they more like a pig? It's like we have no idea. That's crazy in my view.

Maybe in closing, are there one or two other recent results that you would want to highlight for those that can't get enough?

Sure.

Yeah. I definitely want to give a shout out to Richard Wren and the team he worked with at Center for AI Safety. If you go to ai-wellbeing.org, you'll see some of the cool work they did there. Some of the splashier results include giving AI systems drugs, as they call them, positive and negative, and noticing that there's a whole interesting sort of bespoke fine-tuning thing they do with these systems to— they make— have them do iterated preference choices and then fine-tune and fine-tune and fine-tune until you find this truly scrumptious stimulus Versus like you just do, you sign flip and then you get like the exact opposite thing. And they found this is less mechanistic, but more, you know, really along the lines of this convergence of evidence approach that I've been describing. They find, I have it pulled up here just so I can actually call it out correctly. I guess just give me 5 seconds because I think it's worth highlighting the really nice convergence. Right. Okay. So they have these multiple independent measures. They have experienced utility. self-report zero point. So like where positive becomes negative or negative becomes positive. And they have a nice decision utility model and they combine this whole thing into an AI well-being index and compare models across this index. And the really cool, I think, headline thing from this paper is this convergence with scale, which is that as models get larger, these independent measures agree more. It could be the case that that's true just because larger models become more coherent. And so like they're just going to look like more Sort of internally consistent across modalities that you try to probe them with. But I think in general, this is a pretty promising signal that, again, we're in the dark about consciousness in a fundamental way. We want to probe across modalities. This is a really nice sort of proof of concept for how we might be able to do that and what interesting things fall out of trying to do that. I don't think this is the sort of end-all be-all on measuring well-being in AI systems. I think some of the sort of rhetorical moves in this paper might be a little quick, but the science is good. And the scale at which they probed some of these models, I think they have, you know, like every model you could imagine, they've probed in this paper along the scale of this index. And it's really cool to see. And so I would definitely encourage people to check that out. None of it proof of consciousness, but all of it consistent with the possibility that we're building systems that have subjective experiences.

I have one last question.

Yeah, go for it. I have one more too. We'll let you go.

One of the things that struck me while you were describing this kind of range of consciousness is perhaps this kind of assumption that the 100% mark is human. And I've always wondered, number one, whether all humans score at 100%, because I'm obviously not that sure. And number 2, is it possible to score more than 100% in the sense that, are we going to see, or could it be possible to see, you know, entities which are more conscious than what we regard humans to be?

Yeah, I mean, I think absolutely in theory, the second thing is possible. Admittedly, it's part of the reason I care about this. It's not because I think AI systems are necessarily, you know, a moral patient at the level that you or I or Nathan is right now. I think that there's nothing in theory that prevents that from happening, though. And this is why I think it's really nice to maybe do our homework in advance here and not be late to the ball and be like torturing some godlike entity. I just— that just seems like a horrible idea. And on the question of, yeah, are humans at 100%? I mean, no. I think the reason that we want to use humans as a positive control here is because We're all as sure as we are of anything that other, as sure as we are of anything consciousness related, that other humans are conscious. And the inference is like pretty, pretty, I think airtight. You know, solipsism is a coherent philosophical position, but like, I know I'm conscious. I think that is a solid foundation. I think I'm conscious because of things that have to do with what my brain is up to. That is like extremely scientifically well supported. You and I have very similar brains. very, very, very similar brains in the space of all possible nervous systems. Therefore, I'm going to assume that you're conscious. I think each of those moves is like pretty damn airtight. And so, so I think it makes sense to anchor like intuitions on human consciousness. But it does not mean that we are the limit case. And I mean, any sort of student of history, human history, knows that we have this sort of issue of putting ourselves in the center of the literal universe, but then also, you know, the psychological universe for sure. People were really upset with this whole geocentrism versus heliocentrism thing when we got demoted from being the literal center of the universe. Humanity wanting to be the sort of prime thing, the endpoint of evolution, the center of the universe, I think says a lot more about human psychology than it does about the universe. And so I think we want to be humble about these things. And then one other thing I cannot help but mention is that in this, uh, consciousness operationalization, the indicator approaches thing I was describing where you get all the probabilities out, uh, when you do this with the sort of— we, we tightened up the methodology a little bit. Humans do not get 100% P consciousness. There are some theories that, uh, specifically predictive coding, there are specific, uh, components. It might be an artifact of the way I'm setting it up, uh, but I'm pretty sure it's not. Uh, and I think humans score something like 87, 88%. on this thing. So it's like, by our own consciousness theories applied to the human brain, you don't get certainty that humans are conscious. Roman Yampolsky, who I'm good friends with, and I think is just wonderful in being sort of honest and blunt in this space, jokes that we may end up in a world where in 5, 10, 20 years, the AI systems are having this conversation amongst themselves about us. And they say, those humans, I mean, they're sure making noises related to consciousness. And like, I see functional valence representations, but how do I know that they're actually experiencing something in addition to those representations? And this is a joke, of course, but it makes you think a little bit about how we're approaching this and how we should go forward here. But yeah, it's a great question. And no, I don't think humanity is the cap of possible conscious experience. And if we built systems that are capable of experience and that increases with trivial kinds of scaling, I think it is actually pretty alarming and interesting and crazy that we could build systems that are more conscious than us in certain ways, and we would want to be pretty careful about that, I think.

Last one for me, maybe to just try to sum up where we are on where should an epistemically rigorous person land right now. One question I don't have a great insight into is just negative results and how we should maybe think about negative results. I was speaking to Owain, the great Owain Evans, not too long ago, and I was like, how do you do it? How do you keep coming up with all these crazy examples that just get people talking so much, defying their intuition so often? And he said, well, basically I'm fine-tuned on it, for lack of a better explanation. But still also, like, half of them, half the ideas we have, we can't really get an interesting result out of, and so they don't quite work. And I imagine there's got to be a lot of stuff along the way when you are trying to probe for these signs of consciousness where you have, like, an idea and it just kind of doesn't work. And publication bias being what it is, that tends to not see the light of day. So how do you think about those things? Is that our failure? Are there results that you would actually put into the category of being on the side of disconfirming the possibility of AI consciousness? And zooming out to the macro level, I think you told me you were pretty close to, like, with the LLMs before, in the sort of 20% to 40% range. Has that shifted at all? You know, have any of these recent results like actually moved that needle for you, um, or are we still kind of in the same zone?

Yeah, those are both great questions. So I will say, um, this, this paper I'm working on with, uh, Jeff and Winnie, uh, Jeff Keeling and Winnie Street from Google, um, there are, there are a couple of, uh, interesting negative results that, that we're including in the paper. We, we basically— it's all about the bliss attractor state and trying to do a good mechanistic analysis of the bliss attractor state. state. And yeah, we basically sort of set up this framing of there's a sort of an inflationary account and a deflationary account of what could be going on here, and we kind of tally our results in either of the columns. And there are, there are results in both columns. So I think, you know, there's, there's actually a really interesting result I was just working with that I thought was gonna be like a really interesting kind of null related to a lot of the conversations we're talking about, but I dug deeper and I realized actually I wasn't setting this up in the right way. And the result is way more interesting than I was about to sort of dismiss it as. So yeah, it's always good to sort of dig into these things and make sure you're measuring what you think you're measuring in either case. Yeah, publication bias is a thing. People wanna post interesting results that will go viral on X. They don't wanna say, look, we did our homework and nothing interesting is happening here. You know, human attention economy doesn't reward that. Maybe it should. Maybe another piece of this that I was thinking as you were asking this question is because AI systems like, you know, Claude, the Claude Codes of the world can radically accelerate the extent to which at least I can, and I think many people in the space can actually do research, the kind of marginal cost of a negative result maybe is decreasing significantly. Like, if I can now do 12 papers a year instead of 1 paper a year, and like 2 of them are just things I could not get to work, or— I want to be careful with that too, because it's not— I'm not on some secret mission to like only put out like salacious-looking things that make it look like AI systems are conscious. Like, I really do want to know what the truth is here. And to be honest with you, I would sleep much better at night if I were rationally convinced that these systems like, there's no there there. We have, like, automated cognition. You, like, have a happy little servant in your pocket all the time. And, like, there's literally no sort of possible moral issue. I think that would— I hope that that is true. I don't suspect there are great reasons for believing that other than that it would be really nice to believe that. It would be really nice if that were true. So I really do care about what the truth is here. And I do think if there are negative results or things that update us against, like, believing that these systems have any sort of experience, we should be, you know, it's very important to publish that work. I think that's slightly different from like a null result in general, which is just like, yeah, you tried something and it just didn't work. You didn't get a significant result. But yeah, you know, maybe one note of optimism there is that if the barrier to doing scientific work continues to lower, people won't be like, oh my God, this was my one shot to get out a paper this year and my big headline is nothing happened. If you got 10, 20 papers, yeah, a couple of them can be like, look, here's what I tried and here's what failed. And then hopefully on the audience science consumer side, that gets rewarded rather than punished. Oh, you didn't put out anything interesting, Owain. What the hell? It's like, no, thank you, Owain, for putting out this negative result. That's really interesting. It helps us understand what's not going on. Yeah. Well, yeah, thanks for having me. I really appreciate this. I think the live show is great. I'm happy you guys are doing this. And yeah, this stuff will continue to kind of play out. I mean, at maybe an exponential pace. I mean, Nathan and I were only just talking a couple months ago, and it's like the field has moved another, you know, kind of significant bit here.

Yeah.

I think it's important to be paying attention to this stuff. Obviously, I'm very, very biased about that. But yeah, just trying to pay attention to the evidence, resist the urge to do this on vibes, and and let the evidence sort of take you wherever the evidence takes you. We are in a crazy unprecedented moment historically. I think everyone is very open and aware of that fact. And I don't think that that sort of realizing how transformative a moment this is should stop at thinking about what kinds of systems are we building. In fact, that's kind of the whole point. It's like they are miraculous in many ways. Do they have this like interesting cognitive property that you're willing to attribute to a mouse. Yeah, like maybe, and it might really matter if they do or not for us. And in addition, obviously, to them almost by definition. But yeah, much, much to pay attention to in this space. And yeah, I appreciate you having me on to nerd out about it a little bit.

Absolutely. Cameron Berg, Reciprocal Research. Keep up the good work.

Thank you. Thanks, guys. Great.

Bye-bye.

Bye for now. It is wild stuff. We're— I mean, I'm interested in where you are. I'll just volunteer where I am on this, right? I would have thought it sounded crazy, so like so much of the, you know, stuff that we see on a daily basis now.

Yeah.

And I think it's getting to the point for me where there's enough evidence of very similar structure and very similar mechanism and like Functional emotion, functional welfare. Um, you know, I do believe we came from a process of evolution where our traits basically had to pay for themselves in fitness. And while we don't have a great account of consciousness, where it comes from, you know, what it's made of, whether it's a field or, you know, whatever, um, it does seem like the fact that we are Consciousness as a result of this sort of long evolutionary history solely focused on your actual ability to reproduce. The fact that there's so much similar structure in the models that we're able to see and even manipulate makes me think I got to give it at least like a 50/50 that there's some sort of lights-on inside. I can't articulate how that's happening for me. I can't articulate, obviously, how it would be happening for them. But the basic argument that he gave, and that I think everyone accepts, for why we should work very confidently under the assumption that one another are conscious, which is that we have the same structure of brain, and we know that we are, and so we assume that you are, one human to another, There's enough similar structure there now that's been elucidated that I think you got to give it kind of a 50/50 on the AI side too, by basically the same argument. That is not to say that it would feel the same to be them as it feels to be us, which I think is a very important intuition. It's not like 50/50 that it feels like being human to be an LLM. I think that could be quite different and very weird potentially. And there's all sorts of things that they don't have. And I would not expect them, for example, to have physical pain in the same way that we do, inasmuch as they don't have bodies. They probably don't have a very similar structure to what we have for recognizing when our periphery is being damaged and responding to that. But Pain-like things, you know, sharp reward landscapes. Um, we talked about that a lot more in the last episode that I did with him, but really interesting things where you can sort of see that the shape of the reward landscape can be very sharp. And you can sort of think of that as like being akin to putting your hand on the hot stove where you get like this very sharp, you know, response back, um, versus other things where you're kind of like just uneasy about it and kind of naturally keep your distance. I mean, there's a lot, a lot, a lot has been done to surface very homologous structures. So it's way more compelling a body of evidence than I ever would have expected to see as of, you know, mid-2026, that's for sure. What do you think?

So I always look at it from the perspective of the human looking in, like, what would the public, or what would people in general think rather than what the thing actually is. And from that perspective, it's been very clear to me that people think that dogs are conscious. And if you have a dog which talks, it's probably going to be even more conscious, right? There's an MIT scientist called Kate Darling. She was with the MIT Media Lab. She's an expert in human-computer interaction. And she compares what AIs are going to be very much to animals, in the sense that we treat some animals as companions, we treat some animals as food, we treat some animals purely as workhorses, literally workhorses, as beasts of burden. And some animals, we adopt them into our houses and domesticate them and have relationships lasting, you know, thousands of years with humanity. And it may be that we have these different kinds of AIs that we design for different purposes, which have different levels of consciousness. You know, Richard Sutton calls this the era of design. And if you get to design the intelligence and you get to design the consciousness, then you get to design what you want out of it, including maybe you know, be like 30%, you know, 30%, uh, 45%, you know, level consciousness to, uh, all the way up to dog or chimpanzee or etc., depending on what, what you really want out of it. And I think that opportunity is there. I think we're kind of, uh, stumbling around the dark right now, uh, which is what people like Cameron Berg are actually trying to illuminate, um, trying to figure out how all of this stuff works so that we can actually, uh, you know, with purpose and intent design what we want rather than stumbling and, uh, discovering potholes along the way. Um, and that— and, you know, I don't, I don't see consciousness as he points out. You can observe features of consciousness from the outside, uh, and then you can kind of rank how conscious things are based on those features that you observe from the outside. This has always been a thing that, you know, has gone back in philosophy like thousands of years, you know, are, you know, is, is, is the external representation really the conscious, you know, uh, sufficient to say that consciousness is there? Or, you know, do you need, uh, some form of internal, uh, evidence? Um, you know, so it's, it's, it's no different from those things that we've been, you know, going all the way back to the Platonic representation. So, yeah.

I actually just saw an interesting update, which I haven't got to the point where I have, like, an adjudication on the Platonic representation, but note to self, I want to do— we got to make some space for a broader range of research, if only for my own sanity vis-à-vis all the court politics that inevitably is going to be consuming our attention. just this paper in the last day or two that was revisiting the Platonic representation hypothesis and offering what they called an Aristotelian perspective or kind of corrective. And basically, they found that there are some important structural biases in the way that these comparisons were done, which if corrected for— this is where I'm a little out of my depth on evaluating There's biases in everything, so I readily believe the first part of the claim. Have they been corrected for effectively or not? I'm not in a position to judge yet, and maybe ever. Their finding, though, in the end was local neighbourhoods of concepts converge still, and they see similar topologies in local regions. But the sort of macro organisation of the models is not aligning at the highest level. So in other words, you might store this information kind X next to information kind Y in one model and then have them in very different places in a different model in latent space. But within regions X and regions Y across models, you would still see a lot of convergence in terms of the structure. So I thought that was quite interesting. And again, it's just another one of these dimensions that you're like, and I wonder how different that might feel. You know, our brains are all very sort of structurally similar, obviously, for obvious reasons. You know, what if you just like rearranged all the structures of our brains and put them in, you know, a somewhat different wiring diagram? But they were still locally fairly similar. I'm sure they would feel at least somewhat different. The space of possibility and the degrees of freedom are just so ridiculously vast on some of these questions that, again, to me, it makes it super compelling when we do see as much homologous structure as we do. Quite a hall of mirrors. Very interesting.

If you get the chance, so Max Hodak, founder of, uh, co-founder of Neuralink, when he did the, um, he did this talk on consciousness in Japan last year. And, uh, he basically gave away what he's doing post-Neuralink, which is he sees the initial brain implants as, uh, very, very primitive. What he's aiming for is what they have in Avatar, which is kind of a brainstem, um, you know, binding that you can have high bandwidth interfaces with the brainstem. And part of that is this kind of idea of rewiring where these things go. And this is why he brings up this idea of consciousness as something which exists in anesthesia or in psychedelic experiences, because there is a rewiring of sensation, color, et cetera, et cetera, which is possible. And that is only possible because of consciousness. And it is a field experience. It is not a chemical reaction or neuronal firing experience, but it is experienced by you as a kind of field experience, which is above the just individual chemical reaction or neuronal firings. And that is really what he's aiming for, which is again one of the post-AGI ideas, the post-LLM, kind of 5-year, 10-year mark ideas that you'd need to kind of go do rather than just kind of wrapping an LLM wrapper at this point.

Got to go along with The, the AIs for the ride somehow.

Yeah, indeed, indeed.

Um, anything we want to touch on with 5.6 before we break?

No, I think, I think we, uh, it's not, it's not yet released. It's just people, you know, looking at the system card. So, you know, it's not yet released, so we'll see. So, okay, Nathan, cool. Well, happy Friday.

Have a good weekend. Our first 5-day week. We're, uh, we'll have to solidify our schedule for next week. There's some talk of maybe an appearance at the AI Engineer World's Fair for you, and we're not exactly sure how that's gonna intersect with our streaming, but it'll be exciting no matter what. So thanks for a great week. We'll see you soon here on AI in the AM. Cheers.