Zero to One in AI Safety: Halcyon's Mike McCormick on Launching 30 New Orgs & the Founder Bottleneck

Hello, and welcome back to The Cognitive Revolution.

Today, my guest is Mike McCormick, founder and CEO of Halcyon, a combination nonprofit grantmaker and for-profit venture fund working toward a single mission of helping the world's most talented founders and leaders launch new AI safety, cybersecurity, and biosecurity organizations. Mike's core belief is that organizations solve problems for the world. And of course, AI is creating a lot of problems that we need to solve in a remarkably short period of time. Which means that what we need most right now isn't more ideas, but founders who can build the organizations needed to implement whatever turn out to be the most viable solutions at scale. In 3 years, Halcyon has helped launch roughly 30 such organizations, which have collectively raised roughly half a billion dollars. Goodfire, who's been featured on the show multiple times, is the best illustration of how Mike works, with a special focus on the 6 months before and 6 months after an organization is started. The first career transition grants Mike ever made were to eventual co-founders Eric Ho and Dan Balsam while they were still running their previous company, but starting to think about changing focus to work on AI directly. The financial boost, the vote of confidence, and the network-building support he provided helped them found a company which has since achieved unicorn status and become one of the best places in the world to do interpretability research. Other orgs that Micah supported include past guests AIUC, which does agent standards and audits for insurance purposes, Fathom, which has advanced policy discussions with credible early proposals for independent regulatory schemes, and Asymmetric Security, which is building cybersecurity forensics agents meant to defend companies from employee account compromise. Beyond that, Mike also introduces Transluz, which is working on scalable oversight, and Hadrian, which is stockpiling PPE. It is admittedly a very broad portfolio, but as Mike puts it, we are speedrunning 6 or 7 industries worth of AI revolution all at once. In a 2026 that remains plausibly on the AI 2027 timeline, rapid, competent execution is obviously critical and clearly will require heroic efforts. To take just one personal example, for nearly a month now, I've been trying to donate AeroLamp lights to my son's classrooms. And while my son's teacher said that she loves the idea, it's a new thing and nobody at the school has been able or willing to sign off on it. This too, it turns out, Mike is involved in starting an organization to address. Naturally, we talk about how seemingly short timelines affect his strategy, what he looks for in people to back, and how he thinks about both for-profit and nonprofit governance. And we also discussed the verification technologies we'll need to sustain pacing agreements, as well as the challenge of getting everything from cryptography to hardware manufacturing to diplomacy all working together as one. The main headline of today's episode, though, is that there is still time for ambitious leaders to start new AI safety organizations, which could prove incredibly important. And more broadly, that there's never been more support to scale these organizations with first-class talent. Which means that all sorts of accomplished professionals, including many who do not have technical ML backgrounds, should be actively thinking about pivoting their careers into AI safety. If you are feeling the pull, Mike offers his contact info later in the episode, noting that a warm introduction is welcome but not required. With that, I hope you're inspired to see yourself as a potential difference maker by this conversation about building the institutions that we will need sustain AI safety with Mike McCormick, founder and CEO of Halcyon.

Mike McCormick, founder and CEO of Halcyon, welcome to The Cognitive Revolution.

Yeah, so good to be here. Thanks for having me.

Thank you. This has been a long time coming. I think we met a couple of years ago now, and you have been one of the behind-the-scenes movers and shakers that have been pretty instrumental in creating and getting off the ground a bunch of different organisations that are now active across the AI safety space. And I think I can speak for everyone when I say thanks to you and to others who've done similar work, because this is all coming at us pretty fast. And Lord knows we need all the different organisations and specialists that we can possibly have to navigate this critical time that we find ourselves in. Let's start maybe by just kind of introducing, like, what is Halcyon? There's a couple parts to it, and then we can, of course, move into talking about some of the organisations that you've helped to stand up.

Sure. So, yeah, Halcyon, we're a 3-year-old organisation. So we are part nonprofit and grantmaking organisation, and then we are part venture capital fund. So we have 2 arms to the organisation, but don't think of them as 2 sort of separate projects. Think of them as 2 separate tools in the same bag, right? And so our whole mission is to find the world's most talented leaders and founders and basically help them start and launch new AI safety organisations, cybersecurity organisations, biosecurity organisations. And so in 3 years, we've launched like 30 new things, and those things have collectively raised like half a billion dollars in funding. And again, some of that funding is nonprofits, sort of philanthropic funding. Some of it is venture funding. And yeah, I think you've had founders of a few of these organisations, like Goodfire and AIUC, and I think a few others too.

Well, let's do a little deeper dive on just some of the organisations in particular. I mean, people can go listen to the full episodes with Goodfire, with AIUC, Fathom, and Asymmetric, but give us the kind of quick pitch on Either those or whichever ones are kind of most top of mind you think people should be aware of.

We'll tell some stories. I mean, Goodfire is a great one. So I met Eric Ho and Dan Balsam, who were the founders of the company, 2 or 3 years ago when they were still running their last company. It was a pretty successful tech company, although they were starting to think a ton about AI safety and AI risk and what can we go do And so actually, the first 2 grants Halcyon ever made were career transition grants to Dan and Eric so that they could just go pay their rent while they figured out what to do in AI safety. And then we brought them to a retreat and connected them with a bunch of great people there up in the woods in Northern California. They landed on interpretability as this sort of central problem that they wanted to work on and ended up recruiting a couple other great people and launching Goodfire. And then from our venture capital funds, we were among their first investors and invested in every round since. Just chatted with Eric for a minute on the phone yesterday. And so I think what Goodfire is, is the interpretability lab, right? One would think that if we're going to understand AI models and we're going to understand all of the ways that they might be dangerous or deceptive or misaligned or doing things that we don't want, it would be really great to be able to sort of look inside those models and say, okay, these neurons are firing in these ways to produce sort of this dangerous output. Okay, maybe we can do something about that. Right. And so that's what Goodfire does. I think they're probably the best place in the world to work on interpretability research, even including the labs, although perhaps Anthropic could give them a run for their money. And so they've become a pretty big company. They recently raised, I think it was well over $100 million Series B above a billion-dollar valuation. And yeah, I've just been super impressed with their ability to make progress in the interp space.

Yeah, I love doing episodes on their work. And it seems like they've been so prolific that every time I go back, I'm like, holy moly, I've got like a dozen papers. catch up on, and it's only been 60, 90 days each time. So yeah.

I think that's what a lot of companies in the space that do really well, and nonprofits, by the way, have in common, which is they do a great job at getting a bunch of the best talent, right? And then the first really talented 5 people come in, and then the next few people are like, oh, all my smartest friends are going there, let's go there. And then the funders see that, and of course, the funders want to fund places that have really great talent, and so then they pile in and then the flywheel sort of spins. And so yeah, I think they've been particularly good on that. And yeah, I don't know, so much to be excited about there.

One of my kind of answers these days for what would the value of a pacing mechanism or a slowdown or even a pause be, it's like, well, you Look at the pace they're going, and a little more time would be really advantageous for them to potentially get over some key thresholds of understanding.

Yeah, totally. I mean, I feel like we're just speedrunning, like, 5 industries or 6 or 7 industries at once now, right? It's like we're speedrunning control and we're speedrunning verification and cybersecurity and biosecurity and alignment and interp, right? All of these are just such young spaces. They're really nascent and we could easily just 10x all of them today and still probably not have enough firepower behind solving these problems. You could argue that we just need many Manhattan Project-sized efforts all at the same time here.

You want to talk a little bit more about AIUC and others?

Yeah, I love AIUC. Another company that we supported very early and also made grants to founders before there even was a company. So the way I think about AIUC is like, I want to live in a world where we can measure and systematically mitigate risk from AI. And to me, that sounds kind of like what an insurance market does, right? It's like we underwrite the risk that this bad thing is happening, and then we create financial products and other incentives to incentivize those risks or those bad events to not happen. This is kind of what AIUC does, but for AI. So they kind of have this dual product. So one is a standard. Think of it like a SOC 2 standard, but for AI. And if you want to meet this standard, you have to meet these 50-something checkboxes that have to do with your safety practices and your security practices, etc. And then along with that standard comes an insurance policy, which you can only buy if you meet the standard. And so I think it has the possibility of creating a really interesting race to the top as companies want to meet these standards because their customers want them to meet the standards. So AIUC's big customers are growth-stage AI application companies. So ElevenLabs, I think, was one of their first big customers. And the idea is ElevenLabs is selling their AI agents and tools To enterprises. And these enterprises really care about AI risk and not adopting tools that are going to cause some huge issue for them or go off the rails or whatever it is. And so now ElevenLabs can go to their big Fortune 1000 customers and say, oh, hey, by the way, you can adopt us and feel really good about it because we meet this standard, this shiny AIUC-1 standard. And by the way, along with buying our tools, our tools are now going to come with an insurance policy where if any of these risks that you're worried about actually come to pass, well, you're insured against them anyway. And by the way, the only way you can even get this policy is by meeting this very high standard. And so hopefully over time, there are many more standards. I mean, I think AIUC will become a major one, but perhaps not the only one. And hopefully these standards will become sort of harder and harder to meet and hopefully contribute to a little bit of a race to the top dynamic where there's actually incentive to be safer.

How about Translucent? I confess I've seen good things from them on the internet, but never having done the full episode deep dive, I don't know probably as much as I should.

Yeah, you should have them on, Jacob. Would be great. So Translucent, actually, I think Jacob came to the same retreat that you and Eric were both at as well. So Jacob was a professor at Berkeley, and his co-founder Sarah was at MIT, I believe. Wanted to launch an independent AI research organisation specifically focused on scalable oversight. Right. So it's like, hey, maybe not so hard to really understand what's going on with a GPT-3 level model. Right. But as we get to Mythos, 3, right? How are we going to have oversight over things that are perhaps smarter than us or things that are already— or things that are just constantly maxing out their benchmarks, right? I think I heard on one of your recent shows the idea that the meter time graph is basically done as a useful metric because the models are improving faster than they can actually get humans to do those tasks or measure how long humans can do tasks. So how are we going to have oversight over models that are smarter or more capable than us? And so that's their main line of research. And so they build tools to allow other researchers to do that. One of their tools is called Docent. And then they also work with external organizations to do Evaluations with labs and with others.

Want to do one more? Hadrian?

Oh, Hadrian. Yeah. One of the big risks that I care about is biorisk, right? So perhaps AI makes it easier for a bad actor to create a bioweapon or to synthesise a pandemic virus. And so we've done a bunch in the biosecurity realm. And one thing you would really want in, let's say, like a future COVID-type disaster is just enough PPE, protective equipment, to keep essential workers going during said disaster. And so we wrote a tiny little check. I think it was like $50K to get them off and running at this idea. They basically figured out that for on the order of a few hundred million dollars or maybe a couple hundred million dollars, you could stockpile enough PPE to essentially keep every American worker at work during a COVID or perhaps significantly worse pandemic. And so that's essentially what they're doing. But I think they actually have a broader vision than that, right? So if you think about biosecurity, the framework that people tend to use is deter, detect, defend, or there's other ways to say it, right? But part 1, you want to make it so these biodisasters don't happen. Part 2 is if they are happening, you want to be able to know and detect them very quickly. And then part 3, you want to be able to respond and overcome or get through those disasters, right? And so they're doing a bunch on that 3rd category, right? Like, what are all the categories of things that society will need to sort of get through that or defeat it? And we're doing a bunch in that that they're not doing as well. So you can think about medical countermeasures, right? Could you spin up vaccines extremely rapidly? Or non-medical countermeasures. Are there other things we can do? So for example, can we improve the built environment? So we just invested in a company that we're helping to launch that's basically innovating in indoor air quality.

Right.

air filters and far-UVC and glycol vapours and other technologies, perhaps you can just harden the entire built world to future pandemics.

They have one of the better headlines I've seen on a website: Help Defend Civilization. So I'm trying to do my small part to help defend a little corner of civilization, which is my son's elementary school classroom, by buying some UVC lights and hopefully getting them installed in the classroom. It's been a funny experience because the Detroit Public School that he attends, which, you know, honestly, you told me years ago I would send my kid to Detroit Public School, I'd have said you're insane. There's no way that'll ever happen. But we now live here and there's a Montessori elementary right up the street, so it's great. But now I'm running into this bureaucracy of like, who can approve this? And there's no department for this. You know, I'm like, Following up on email and being like, you know, really, like, my kid had cancer. Like, I really want to not get him sick a lot this fall. Can we please get this light installed? And they just don't know what to do with me.

So it's— It's funny you mentioned schools, because schools are actually the initial customers of our company here. So there's a company and there's a sister nonprofit. And the idea is that, like, schools are a place where you would— yeah, like, really, like, they're sort of super spreader places. I'm sure, you know, with kids, right? You're probably picking up colds and everything all the time.

I've already had— you can hear the remnants of my first back-to-school cold in my voice today.

I think there actually is a bunch of appetite from schools to do this. I think one issue with biosecurity specifically is that there isn't necessarily demand right now for a lot of the things that you would really want, say, like in a future pandemic, like masks and vaccines, for example. There are really good reasons to adopt air cleaning technology right now, even if we never had another pandemic. Like, just the amount of productivity missed in the economy, right? The amount of school days missed, the amount of sick days for you, the amount of suffering from just, like, common colds and flus and bugs, right? And so I actually think it's fairly promising that these types of technologies could be adopted really broadly, even if there wasn't Some, like, big pandemic freakout. So yeah, I'm more optimistic about that category than some others. Hey, we'll continue our interview in a moment after a word from our sponsors.

Today's episode is brought to you by ElevenLabs, the leading AI voice platform, whose voice design feature I used to cast and produce all character voices for the Receipt Horizon audiobook. And while this is a super fun way to show off the quality of ElevenLabs voices, the real game changer for businesses is ElevenAgents. With ElevenAgents, you can deploy agents that can talk, type, and take actions, including looking up customer accounts, processing requests, and handing off to a human when needed. ElevenAgents supports over 70 languages and handles more than 10 million conversations each week, And with SOC 2, HIPAA, PCI certifications, and regional data residency, it is definitely enterprise-ready. They even offer engineering support to help your team get your agents into production quickly. If I were starting my company today and designing a customer service function for the future, I would try ElevenLabs Agents first. If you run a business or handle customer operations across support, sales, or marketing, You can start with a demo at elevenlabs.io/tcr. See how Eleven Agents can fit into your workflows and help build experiences that your customers will actually love. elevenlabs.io/tcr. That's elevenlabs.io/tcr. This episode of The Cognitive Revolution is brought to you by OutSystems. The leading agentic systems platform. OutSystems is helping their customers build, modernize, and operate enterprise systems starting from any coding tool in a governed agentic engineering model. With OutSystems, you can coordinate and govern your entire agentic workforce within your enterprise ecosystem. Accelerate impact with industry-proven agentic solutions that are pre-built, governed, and customizable to your business. It's time to innovate at the speed of AI without compromising quality or control, which is why thousands of enterprises worldwide trust OutSystems for their mission-critical apps. Teams any size can use OutSystems to build, deploy, and manage AI apps and agents quickly and cost-effectively without compromising reliability and security. With OutSystems, you can accelerate ideas from concept to completion. It's the leading agentic systems platform that is unified, agile, and enterprise-proven, allowing you to build your agentic future with AI solutions deeply integrated into your architecture. OutSystems, own your agentic future. Learn more at outsystems.com/tcr. That's outsystems.com/tcr.

So, we've done like mini unpackings of 4 organizations, and there's a lot more. But the first thing that I think jumps out is you've got a super broad mandate here, right? We've gone from interpretability science to agent standards and kind of trying to create the conditions for an insurance market to create the right incentives for people, to scalable oversight nonprofit, to stockpiling PPE and trying to get these measures into schools. Do you just go out and say, give me a list of every problem in the world and let's go stand up an org to solve them all? Or how do you think about the breadth of what you're going to take on?

Yeah, in some sense we're super narrow as an organisation, right? Okay, so I think we basically have a worldview and there's like 2 or 3 parts, right? So part one is that AI is just progressing extremely quickly and probably will continue to progress, right? We're sort of kind of on the AI 2027 timeline, give or take a few months, right? Thing 2 is that, look, there's so much to look forward to with AI, of course, right? We're seeing all these amazing educational results and healthcare results, et cetera. So certainly not anti-AI, but if we want to get to this beautiful future we all want, we have to navigate the risks really carefully. And the risks are major and global in scale and intersecting. Right. And so I'm quite worried that we're not really on track to solve the biggest pieces of the AI safety equation. And then I guess the third part of our worldview, which just sort of drives Halcyon's work, is the biggest bottleneck to solving this problem or to sort of speedrunning these fields is a shortage of really amazing founders and leaders, right? I started making grants in AI safety 3 or 4 years ago, maybe 4 or 5 years ago, and I was really struck by just how young the space was. And I mean young both in the sense of the field was brand new, but also young in the sense that so many people working in it were very young. But I was sort of struck by the idea of, like, you would also just want people who had founded very successful companies or managed big teams or held very senior government roles or intelligence or defense roles. And so that's why we started Halcyon, right? It's like, how do we get the world's most talented leaders and founders to solve these world's most important problems? And so exactly how do you attack these problems, right? Like, within that zone of solve AI safety, what do you do? Right? Because it's such a multifactorial problem. And we can talk about the areas that we focus on. I would say we have sort of like 6 major focus areas, and they kind of align with various risks that we see being particularly important and urgent.

You mentioned AI 2027. It's a pretty fast-moving timeline in that story. It does seem like, roughly speaking, we're kind of on that trajectory, and maybe through political means we'll get off it. But technology itself seems to be happy to progress at that rate. Were you kind of always on that timeline? One sort of possible disconnect that people might flag here is like, wait a second, if we're on AI 2027, do we really have time to found new organisations?

Right.

There's a lot that goes into founding an organisation that by the time you kind of get your office set up, we're going to be in whatever bizarro land that Q1 or Q2 of next year calls for.

Yeah.

How do you think about timelines and founding, and is there attention there?

The best time to start an AI safety company is 20 years ago, and the second best time is today, right? Yeah. I mean, I definitely have not always been, like, a super short timelines person, and candidly, I'm still pretty uncertain. But I guess I always thought the very short, like, AI 2027 type timelines were possible, maybe like the fast end of possible. And yeah, I mean, I think it really does affect the type of things that we want to support. I really am prioritising things that I think could be relevant in the next 1, 2, 3, 4 years. I think if we truly only have 3 or 6 months to some RSI FOMO fast takeoff moment, then yeah, kind of all bets are off. And so it's hard to see beyond that event horizon. And maybe in that case, we should all just go to the beach and hope the superintelligence is like us. But I also, I don't know, look at everything happening in the news right now. Pacing the frontier is the new, in the Overton window, normal thing to support. And so I am hopeful, maybe even optimistic, that we'll find a way to sort of slow down our pace. And then at that time, you would again, want to be like speedrunning these industries, right? You'd be like, okay, we're pacing now. We have a little bit of time. We've really got to get control right. We got to figure out how to keep these things in the box, or we have to get monitoring right. Especially as things like chain of thought— if chain of thought goes away, are we even really going to be able to do monitoring? What does monitoring even mean in a world where we can't see the reasoning and chain of thought of models? Right. We've really got to get cybersecurity right. Even if we don't pace, actually, I feel like we have to really get cybersecurity right because we're going to live in a world with OpenWeight Mythos+ models probably pretty soon. And so what do you do in a world of unguardrailed models that could exploit all of these gazillions of vulnerabilities? We definitely have to speedrun verification, right? Because I think verification is the thing that's going to allow for pacing, right? Any agreement you'd want to make between, let's say, the US and China or between the labs or even just state regulation, federal regulation, the teeth of those agreements are going to be sort of technically specific things that are being committed to. Right. And I'm kind of uncertain and agnostic about what those things should even be. Right. Like, we should say we're all going to commit to not building models above such and such size, or we're all going to commit to such and such security standards. But I worry that those types of agreements are sort of impossible or certainly not robust unless we can actually have the tools to verify that those things are true. Right. Yes, I know with confidence that data centre is being used to only do inference and not training runs. Or yes, I know that the model that the company claims they're serving up to me, which I know meets all my security standards, is in fact the model that's doing the inference for me right now. And you could go down the list of dozens and dozens of moments of verification that need to be in action basically all the time. the verification industry such as it is right now is totally nascent. We've invested in the first companies trying to use zero-knowledge proofs to do inference verification. We even invested and helped start the first company that's trying to build a fully verifiable NeoCloud. How much of inference is going to need to be verified in some way in the future? Probably a lot. And that That's not how the data centers are built. That's not how the chips are built. And so there's so much to do on the software side, on the cryptography side, on the hardware side, and then of course on the coordination side. Even if you have all these technologies, if you can't get the diplomacy done or the agreements agreed to or the laws passed, then the technology is impotent. And then again, without the technology, even if you can coordinate between all the live players, if you can't enforce whatever you're agreeing to technologically, then the agreements are impotent. So you really got to get both sides of the equation.

How would you handicap where we are on some of these things today? I mean, you kind of just did that for verification. Sounds like we got a long way to go. How far do you think we are on interpretability, for example? How much progress would you say there has been on oversight? If you're kind of filling in thermometers on the way to a goal, are any of them halfway at this point, or are they all kind of just getting started?

They're all really young. I mean, we're all making really— there's real progress. I think there is great work being done. But in any of these fields, the number of serious organisations doing this work is often in the single digits and sometimes in the low single digits. Right. And you would just want to live in a world where there are diverse, rich ecosystems and industries of these. It's like we shouldn't just have one interp company. We shouldn't just have one Meter. What did Logan Graham retweet yesterday?

Let a thousand meters bloom.

Yeah, we need it. And of course, not exact copies of Meter. We need different things because Meter is great at some things and doesn't do a bunch of other things. People think like, oh, METER, there's an evals org and it's like of the whole surface area of all the evals stuff you would want, METER occupies a very small chunk of that and a very important chunk and they're amazing, but we just need so much more. And I think most of these fields or industries are roughly in that position. So it's like great progress being made, but we need 100x more. I think there's some that are also just easier and harder than others. One reason to be a little bit more optimistic about biosecurity is that it's legible and kind of solvable. If you have the sort of gumption and $300 million, you can just manufacture enough PPE to keep 50 million people at work. And there's a lot of problems that are kind of shaped like that. in biosecurity, because I think it's sort of a little bit of a known problem, right? We don't know exactly what a future pandemic would look like specifically, but we've dealt with them before, and there's sort of a whole public health ecosystem already built up and everything. Whereas if you think about alignment, I mean, that's just like a totally novel challenge, right? And when you say, well, how far along are we on alignment? It's like, well, I don't actually even know what fully solving the problem would look like. If we're talking about superintelligence, what would it even mean to align a swarm of alien beings with 1,000 IQs thinking at the speed of light on the internet? I'm not saying it's impossible, but it's sort of hard to even think about what victory mode would look like if we're going to get superintelligence. And then you could argue about that. Maybe scaling laws will just stop working or we need another paradigm or architecture. But even today, even if you just paused at the frontier today, we still have not solved— clearly we haven't solved control. Look at OpenAI, Hugging Face.

Safe to say.

Right. These are just unsolved problems. We're not on track. Hey, we'll continue our interview in a moment after a word from our sponsors.

Today's episode is brought to you by Anthropic. By now, you know my story. Claude drafts my intro essays, and I rewrite them. Not because the drafts are bad, but so I can stand behind everything I publish. Well, I have an important update. Claude Fable-5 is the first model to have me rethinking my rule. Today, I now think co-authorship, not sole ownership, should often be the goal. Where the model excels, rewriting its work can be more about vanity or a misplaced sense of duty than integrity. I feel it most in songwriting. I'm no lyricist, but I'm good with a song concept, and Fable writes some amazing verses. I give it feedback on its misses, and I push it to aim for higher inspiration, add layers of meaning, optimize syllable density, and above all, Write a hit song. These days I get compliments on just about every song we write together. Claude is the AI for problem solvers. It's the collaborator that understands your entire workflow and thinks with you, not for you. Whether you're debugging code at midnight, building a financial model, or strategizing your next business move, Claude extends your thinking to tackle the problems that matter. For problems worth solving, get started with Claude at claude.ai/tcr. That's Claude AI slash TCR, and check out Claude Pro, which includes access to all of the features mentioned in today's episode. Once more, that's Claude AI slash TCR.

So how do you think about when to try to get a new organization started versus when people should join an existing organization? Like. On the one hand, it sounds like from your comments on biosecurity, if a person came to you and said, I'm passionate about working on biosecurity, you might send them to Hadrian and say, think about joining them, because it sounds like you would say mostly the problems there are reasonably well defined and they've got some infrastructure. So go push on the levers that they've already kind of set up.

Yeah.

Whereas I guess the other end of the spectrum would be, alignment. And there, I like to say sometimes, we need weirdos to come up with new alignment ideas because we just clearly have so many blind spots that—

Yeah.

I think it literally will take weird people to come up with some of the quirky aspects of what we will ultimately patch together, should we be so fortunate as to have a robust solution at any point. That's kind of one axis that jumps to mind, is just like, How more legible would mean scale existing organisations versus, like, more— the more pre-paradigmatic, let's say, a field is, the more you might just want new people pursuing very different angles on it. But how do you think about the question of, okay, here's a person in front of me? I guess you could think about this in terms of the field and also the person.

I tend to think in terms of, like, the unit of an individual. The thing I want most is an introduction to an amazing, successful founder who is at least curious and maybe passionate about AI safety and is like, I want to put my full energy into doing something ambitious and important. And so I tend to think at the unit of the person. And yeah, I think just by numbers, most people are not founders. Right. And so And so I think it just really depends on the sensibility of the person. Do they want to start an organization? Do they want to join an existing organization? I think all of these fields are nascent enough and sparse enough such that there is just room for so many more new things. And then I would also say if you're going to join an existing organization, you should be picky. There's a big range and some existing organizations are awesome and I would just so highly recommend people join them. Not all those are halcyon organisations. There's a bunch of amazing orgs and companies that we have nothing to do with and are doing great work. But yeah, I think it just really depends on the person.

So if somebody came to you and said, I want to start a new interpretability organisation, you're already invested in Goodfire, how would you think about— I mean, leaving aside, and I do have some questions about kind of how you think about the relationship between financial motives, as one part of your organisation is a venture fund, and staying true to mission over time, an eternal question in AI, it seems. But even just on the practical merits, how would you walk through with somebody, okay, you want to start a new interpretability organisation. Why not just join Goodfire? You know, under what circumstances would somebody be well served, or would you support them going off and doing their own different thing?

One thing is just like, do they seem well suited to be a founder? Like, do they want to, you know, take that particular journey, chew that particular glass? You know, like, starting something's hard and it usually doesn't work, and they gotta like really want that. I think beyond that, I guess I would just like to understand what their unique angle is. What are they particularly well suited to do that Goodfire isn't doing? I suspect there's a bunch of interp work that is going to be really important that Goodfire is not doing. And so I'd want to understand what's their particular unique angle, or if they think that the sort of general flavor of what Goodfire is doing is the right thing, then yeah, maybe they should just go join Goodfire. So it's not always a clear question. I think there are a lot of career transitioner type people who are genuinely open to both, joining a thing or starting a thing. And yeah, I kind of tend to tell people when you're thinking about the next phase of your career, hustle but don't rush. Be on that grind, be meeting people, be learning as much as you can, listen to Nathan's podcast, but don't— the best thing you can do is orders of magnitude better than the median thing or even a pretty good thing, whether you're starting something or joining something. And so, yeah, work hard to find that orders of magnitude better thing and don't just rush into the kind of first shiny object.

Yeah, I used to tell people back when I used to do some sort of career advising a little bit myself that I always thought people, just presumably because they're uncomfortable in a sort of in-between or ill-defined state, really tend to rush to take something.

Yes.

When even on a purely financial basis, I always would say to people like, look, how long are you going to stay at this next thing? And they'd be like, I don't know, at least 2 years. And it's like, okay, well, if it's 2 years, then 2 full months pays for itself in your own income if you just get a 10% higher offer at the end of it.

Right.

And there's all these other variables too that could be, you know, you definitely could get more than a 10% bump by working, you know, longer and harder to find the right thing. And there's all these other dimensions, obviously, too, that are, you know, potentially much more dramatically different than a 10%, you know, higher salary.

Totally, totally, totally. Yeah, I mean, look, most of the grants that we make are career transition grants, right? Like, somebody will say, hey, like, I want to, you know, make this my life's work, but I don't know what to do yet, right? Am I going to start something? Am I going to join something? Is it going to be a nonprofit? Is it going to be a company? Is it going to be a technical organization or a policy organization? I got all these questions, right? And I'm not mega rich yet, and so I got to pay the rent. And so we say, great, here's $50,000 or $100,000 or $200,000. Let us support you while you put your full energy into figuring it out. It's also super hard, right? Because if you're job searching, that kind of can be a full-time job to do well. But if you have a important full-time job, which most of these people do, then that's also a huge constraint on your time. And so how do we free up really great people just to put their full force behind this? And yeah, that's why we love doing career grants.

You want to talk a little bit more about what your process for career grants looks like or your criteria?

Yeah.

What does somebody, even if they're, let's say they're an aspiring founder, it sounds like that is right in the centre of the bullseye. How broad do you entertain grants for people if they're not trying to be a founder in one of your target areas?

Yeah, I'm definitely biased towards people who have founder energy or want to be founders, but also have worked with plenty of people who are executives, say, have been like a CTO or a COO of an organisation and are saying, okay, now I really want to get into AI safety or biosecurity. I'm not quite a founder, but I am a very very seasoned leader and could I jump into an existing organization, maybe joining pretty soon after the founding as sort of a late co-founder, or perhaps even join a scaling organization? I helped a friend who's been COOs of scaling Silicon Valley startups recently join an AI safety organization that is kind of now going from 50 to 150 employees over the next 18 months and they need a serious operator. And so I'm also interested in people like that. And I mean, that's just like a narrow slice, right? It doesn't have to be a COO or a CTO per se. Those are just 2 examples, right? But it could be all sorts of things.

Yeah. I mean, this is maybe a question that nobody can really answer because if there was an answer, it would go better than it often does. But I'm interested still in your take on how do you decide who to bet on? And for one kind of motivating example, I recall at the retreat that you graciously invited me to, I met Rajiv Jethani, who's now, I believe, the COO, co-founder, and I think COO of AIUC. At the time, he was a partner at McKinsey, obviously a smart and accomplished guy. But I was like, in meeting him, I've met a lot of these sort of mid-career, like, pretty successful people, and I've tried to hire some myself, and it, it hasn't always worked out. He's gone on to, you know, gain a lot of traction with AIUC, so it's a happy story. But how do you— like, do you have tricks or tips that you can share for how you separate the wheat from the chaff?

Yeah, yeah, yeah. This one weird trick for separating the wheat from the chaff No, I mean, there's a handful of things we look for. I definitely will admit that McKinsey partners are, yeah, I guess a little bit biased against them as a profile, maybe not quite entrepreneurial enough or used to delegating too much or is more about making plans than executing plans or something like that. But yeah, Rajeev obviously was an exception and is great. Yeah, I mean, I was lucky enough to get to know him over time, but before he started AIUC, he became the president of METER. And I sort of helped him navigate through that first career transition and sort of seeing him through that and seeing him as a leader and seeing him as like a founder who really gets his hands dirty and wants to jump in and solve problems versus kind of pontificate on strategy or something like that. was great. And then also just seeing how real he is, right? Like, he's a real missionary. And I think that's probably the most important thing is just like, what is this person motivated by? Like, are they here because they're interested in finding product-market fit and raising that next round of venture capital? Or are they motivated by, you know, a worldview that maybe is somewhat similar to ours, right? Like, very powerful AI is on the way, and we just have to very ambitiously and hopefully thoughtfully, like, build all the tools we're going to need to navigate it well. And so, yeah, I think most founders I want to backscore highly, or most people I want to backscore highly on that, like, entrepreneurialness axis, and then also on that, you know, mission-driven axis.

How do you think about trying to encode those values in organisational structures over time? It strikes me that you've got this at a lot of levels. For one, with Halcyon, you have a venture fund. I don't know who your investors are or how much you can say about that, or how mission-aligned they are, or how you've kind of structured your commitments to them between financial and mission. But then the same thing happens at companies. So Goodfire is a public benefit for profit company, and AIUC is also a for profit company. And I'm not sure if it's happened so far, but there was a little drama at one point about Goodfire where it was like, oh, they're doing interpretability methods that can be used in training, and this is really more of a capabilities thing than a safety thing. I think that's often hard to untangle in any case. Yeah, but clearly there are tensions. We've seen this with the frontier companies. How much do you think can be put down into structure, governance, bylaws, what have you? And how much just really depends on you got to trust the people to follow through when it matters?

Yeah, it's all of the above. If you're looking for some light beach reading, we wrote this long report called Mission Preservation Governance Mechanisms for Startups. It's all about what are the mechanisms you can use to preserve the mission even when perhaps there are other incentives. Yes, PBCs can be helpful. There are ways to do interesting board structures. Anthropic has one of those interesting board structures. You can argue whether or not it's done its job and so on and so forth. I do, though, think that there's no structure that is a silver bullet or is perfect. Right. You can't just say, like, oh, we're a PBC, or we have a board member who's tasked with representing the public good, or we've only raised money from people who are super AI safety pilled. And so we're fine. All those things might be nudges in the right direction, but nothing's a silver bullet. I think Founder sensibilities are so important. How will this person react when put in a hard situation where the incentive to increase share price comes against the incentive to go forward in service of the mission or to not do something dangerous? And then how do you ever know what's in somebody's heart? great that I got to know Rajeev, you know, for a year before we, you know, officially backed AIUC, and Runa, his co-founder too. And I do have a lot of trust for them as people. But, you know, do we even know ourselves all that well? You know, how would we react in a crisis or when, you know, we're managing a team of hundreds of people and, you know, we pay those, you know, people's rents, you know? And so Yeah, I really just like spending time with founders and really want to back missionary founders. We basically have 4 criteria, 4 boxes that we want to check every time we invest. And I think in different ways they buffer against the kind of worries we have. Number 1, core product or service is focused on an important part of the AI safety stack. It can't be sort of a side benefit of the thing they do. It has to be the main thing they It could come from many different angles, but it has to be the main thing. Number 2 is for a company, if we're investing, it has to have prospects of being a successful business. It is an investment fund. And also, if these companies are doing good things for the world, we want them to scale and bring their impactful, helpful products out to the market. The 3rd thing is founder motivations, right? Why is this person doing this? Are they doing it because they're a missionary who's trying to solve this very important problem and won't stop until they do? Or are they reading the front page of the Wall Street Journal saying, ah, AI safety seems to be the new hot thing. Why don't I go start a company and make lots of money? And there's nothing wrong with making a lot of money, but I tend not to bet on people whose primary motivation is money. And then the fourth thing is just great founders, people who even if you didn't care about the impact, you would still say, man, I would bet on this person. Great founder market fit, great drive. I really think they're the type of person that has a chance to succeed. And so yeah, we really want to check all 4 of those boxes, but especially box number 3, the founder motivation box. But yeah, on governance stuff, it's just so hard to— yeah, I mean, we've seen it with the labs, right? As the numbers get big.

There's been some drift.

Yeah, there's been some drift, man. And like I said, there's no silver bullet. So we're figuring it out.

Have you seen the sort of mercenary profile start to enter the AI safety space? Like, have you actually got the sense from individuals that are pitching you that, like, I don't really think you're in this for the reasons that I would want you to be in order to Back you?

Yeah, definitely. I mean, I don't know if it's increased all that much over time. Maybe it will as AI safety becomes sort of more mainstream and less taboo in Silicon Valley. But yeah, you definitely see it. And most of these people don't strike me as con artists or anything, right? It's more just, I don't know, they read our website and they know that we're super impact-oriented and they're smart founders, so they're pitching to the motivations of the funder, right? And so I think it's It's more like that versus some mustache-twisting villain or something.

So obviously there's a lot of money flying around the AI safety space these days. We just heard the news from Coefficient Giving that they've got this new Tailwind project. Are there any areas that you think are still bottlenecked by money? And if If there are, I'd be interested to hear what they are. And if there's not, does that mean that it's just all about talent? And then what can be done about that bottleneck?

Yeah. So I've always thought that talent was a more important bottleneck than money in AI safety. It just seemed evident that where are all the experienced founders? And also one thing that experienced founders or great founders do is raise money very effectively. So the money appears. I think that's just becoming more and more true. Assuming especially the labs go public, so much liquidity is going to open up for AI safety. And then the question will be, well, what do we do with all this money? If I wanted to spend $100 million or a billion dollars to buy down risk, I presumably have to send it to an organization that then has a product or an offering that changes the shape of the world in a way that makes it safer. whether that's by developing a technology or getting laws passed or whatever it is. And I'm fearful of this near-future moment where there's this mountain of billions and billions of dollars and not all that many great things to fund. And so I think this is getting back to your earlier question about the imperative to start new things. A friend recently said the philanthropists and founders of today will write the menu for the philanthropists of tomorrow. We are starting the things that then they will be able to go fund. And so that's our goal, just spin up— I was going to say as many as possible, but that's not really quite true. We want to do more and more, but while holding the quality bar super high.

So what's on your menu? I mean, I guess maybe for starters, how do you think about your relationship to and with would be one way to think about it, but also just kind of positioning or role in the ecosystem as it compares and contrasts to other funders that are obviously doing a lot of stuff. We've got CoVision Giving, we've got Jan Tallinn and the Survival and Flourishing Fund. There's not that many of these big ones, but there are a couple at least.

Right.

Do you consciously try to carve out a sort of different niche or POV for yourself, or do you just do your own thing and not worry about what others are doing?

Yeah, so I don't really actually think of us primarily as a funder. I think of us as a finder of talent and as a launcher of new projects. And then we write a check when writing a check helps with that. And so, yeah, the thing I really focus on— okay, so imagine you're Somebody's about to start a new AI safety company, or they might— this is the moment of zero to one. The 6 months before that, where they're getting interested in the problem, figuring out what to do, who are my co-founders going to be, should I build this kind of org or that kind of org, for-profit, nonprofit, who's going to be on the board? And then the 6 months after, okay, we've launched the organisation, we have to hire the team, we have to build the first product, we have to really launch and go. That year-long zone is where I spend the vast majority of my time and where Halcyon is going to continue to spend its time.

How bullish are you on the OpenAI Foundation and the Anthropic, whatever their thing is, where they're going to be, in theory, spreading the wealth around and making wise investments?

Let's hope.

Yeah, I mean, let's see. So on the OpenAI Foundation specifically, I know a bunch of people who work there and I've seen some of the work they're doing. And so I would say I'm very optimistic that they're building a really good team of people who want to do super important, high-impact stuff and who are motivated by the right things. I mean, I don't know every single one of them individually, but People that I spend time with seem to be doing quite good work. I think the biggest risk to them is whether or not they're going to come up against red tape internally that prevents them from sort of being the most interesting, impactful versions of themselves they can be. Right. I mean, ultimately, decisions still go through boards and committees and yada, yada, yada. And it's a big organisation and at risk of bureaucratizing itself into a less interesting outcome. But yeah, I'm hopeful they figure it out. I think they can. Early signs are good, but it's still really early. They've only just announced their first few grants. And then on Anthropic, I guess there's some Anthropic project that I don't really know much about. I think a lot of Anthropic liquidity is just going to come from employees. I mean, there are so many people who are now worth hundreds of millions or billions, and perhaps that'll soon be liquid. And a lot of those people care a bunch about AI safety and biosecurity and things like that. And to be fair, OpenAI employees too care about those things. So yeah, I think a lot of it will be just employee liquidity as well.

Do you have a take on the IPO process for these companies? Because I think there's the one angle that you're just alluding to there is like liquidity for a lot of these individuals might be a really good thing because we know a lot of these individuals and we think that they have good values and they'll do good things with cash. On the other hand, Sam Altman just said it would be an ill-advised time for OpenAI to go public. And I don't know what all he's thinking there, but One interpretation is we don't want to have public market pressure on us as a company when we're trying to navigate all this insane shit that we're currently discovering on a rolling basis. Any thoughts on kind of how those forces may net out?

Yeah, hard to run a public company. I'm not sure I'd want to do it. Not that anybody's asking me to. Yeah, let's see. I I mean, I'm following this in real time. I just read the article about Sam too, right? So I don't have any deep insight there. I guess I would say that, look, the companies just need access to capital, right? It's just time to raise hundreds of billions of dollars, and I'm not sure how possible that would be without going to the public market. So unless they're willing to really pare back their burn, then I think they kind of have to go public at some point, but whether they have to do it now or can they wait another 3 months or 6 months, I don't really know.

If you kind of compare and contrast your wish list of organisations that you would be eager to at least dig in on, if not definitely back, versus the list that we saw from Tailwind versus what Jan Talon has put out with his priorities versus maybe what the OpenAI Foundation is talking about. Where do you think you are kind of most unique? What are the things that you're most looking for that you don't see other people as interested in?

That's a good question. I really like the Tailwind list. I saw it. They sent me a preview of it, and I was like, oh, this is just great. And I think I told you over text, I was sort of annoyed because I've been writing my own RFP. And then I'm like, maybe we should just delete the RFP section on our website and just link to the Tailwind RFP because it's just really good. It doesn't cover everything I care about though. For example, I don't think it covers biosecurity. So quite good. I haven't seen Jan's list or I don't really remember what's on it, so I can't really comment. OpenAI Foundation, I mean, there's a bunch of stuff they're doing that I care about a lot. So of course their whole AI resilience shtick is quite important to me. I think AI resilience is like an interesting framing. To me, what AI resilience implies is like the frontier is going to keep advancing, whether at the labs or with open models. And that means that the world is just going to be more vulnerable. And so we just have to make the world more hardened and more resilient in all sorts of ways that have to do with cybersecurity and have to do with biosecurity and have to do with government and have to do with, et cetera. You can go down the line. And so that's definitely a big part of what we do, but certainly not all of what we do and not all of what they do. But then OpenAI Foundation is also doing things that are closer to public health or things like alleviating poverty and education, which are awesome and important, but not in our mandate.

Anything you think, like, the field as a whole is sleeping on though?

You know, something we often say is like, the field is not hurting for requests for projects. Like, the field is not hurting for people who have awesome Google Docs with a bunch of great ideas, right? The field is hurting for world-class founders who can take those big squishy ideas and turn them into organizations that, like, actually make the world safer. And so sure, are there things that maybe at the margin I would say are underrated and overrated? Yeah, but also by who, right? And so yeah, I think I'm actually a little bit less in the business of having a really strong opinion about what is the most important thing at the margin and more like, hey, there's at least a few dozen things that seem like Really important parts of the stack, of the equation, of the sort of thing we need to build. And I'm about finding really excellent leaders who can actually do it. And so the way I think about it is we basically have a bar, right? And we actually kind of make lists of projects of like, okay, these are the projects that we want to see and all these categories. And then we kind of make a rough bar and it's like, okay, these are the projects that are above the bar. So if I meet a founder, or a team who I think is capable of building any of those projects that are above the bar, I'm going to say yes and push go versus sort of quibbling about, oh, well, is this the number one most important thing or the number 7 most important thing? That said, I do want to think hard about that. I do want us to be like, wait, is there one or two things right now that are just so important and just so neglected that what we should actually do is put 80 or 90% of our effort into willing those things into existence because this is what the world needs. I would love to have that level of certainty or specificity, but man, I'm just so uncertain about so many things and I'm sort of a pluralist on these sort of various risks to the point where every time I try to come up with sort of a narrower opinion, it feels pretty low confidence. And so for now, we're sort of opportunistic within a zone.

Have you heard any pitches around, like, well, I guess for one thing, how about open weights models? This has obviously been a vexing question. I'm a huge— every chance I get to cite Graham from AE Studio and Anthropic, I point to them.

Yes, say more about them. What are they doing?

Well, A Studio is a fascinating company. It was started by Judd Rosenblatt, who's another past podcast guest and generally fascinating character.

Totally.

And his idea going back, I don't know how many years now, was we want to solve the alignment problem, or at least solve the AI safety problem. And then they've kind of narrowed in over time, and they've had a bunch of banger papers that I am super excited about. in different domains. But one that maybe the best still could be self-other overlap, which is trying to come up with interesting training methods to reduce the difference between when an AI is thinking about itself versus thinking about others. It's sort of like, could we create an AI that is in its own mind one with everything? And if so, that would kind of make it weird or different for it to think about deceiving other creatures. Right. And they've got some really interesting results there. Gram is their latest banger, which basically just tries to localise certain types of knowledge to particular experts within a mixture of experts architecture, so that you can just distribute an open weights model without a couple of those key experts. So you could have kind of like the mythos-fable thing, except instead of accomplishing that difference with guardrails, the Graham technique would— it holds the promise at least of being able to accomplish that by just being like, mythos is how many of us wrote that, and fable is how—

The user still has to want to accomplish that, right? You can't just distribute a fable globally, right?

If you are distributing the open weights model, you could just say, if you're Meta and you're like, I'm committed to this, maybe you could be convinced to use this technique and then distribute the model with like -2 experts or whatever that take out those kind of most dangerous capabilities.

Interesting.

But I mean, okay, so my question is really, have you heard anything that is compelling to you on open weight I would love to fund stuff in this space, but candidly, yeah, I think this is just such a hard problem.

Is it even possible to impose safety or security onto open weight models even if they're being used by, say, bad actors? It just seems so tough. And so I think this is the AI resilience argument of we've got a heart in the world. world because this is just going to be the future. But yeah, I would love to find ways, projects to support on this. I love the first project of like, could you basically turn the model into the Buddha? And it's like, well, we're all one anyway, so I wouldn't want to hurt myself, right? Which I sort of have a project which started off as a joke, although maybe it's real, which is sort of like the opposite of data poisoning. Data poisoning is you put stuff into the dataset that corrupts the model in some way. And so could you put a billion loving kindness meditations into the training set and just turn the model into this being of perfect compassion? Which maybe could be great. But again, then if you have somebody maliciously training an open weight model, they could just decide, well, I don't want to put that in my dataset, right? So I'm not sure it is a universal solve to the open weight problem. But yeah, I don't know. I would be down to have models that had more loving kindness or whatever you want to say, more compassion in them.

Any pitches come your way around issues related to AI consciousness or moral patienthood or other related concepts?

Yeah, we've gotten a couple pitches around this. I find it so hard to reason about this because I'm not sure what consciousness is. I don't think we know. And so it makes it really hard for me to reason about if machines might be conscious. But it seems super important, right? Like, if they are or not, of course, gets into issues of do models have moral patienthood, right? Ought we treat them in ways that we would perhaps treat other humans? And maybe you could even argue that a bunch of training or alignment techniques that we're doing now would be sort of morally repugnant if these models are in fact conscious or having an experience. My hunch is that current models probably are not, but what do I know about consciousness? I think it's totally possible that they could in the future, and I don't see any reason why the substrate of a flesh body is able to be conscious in a way that the substrate of a model in a data centre would not. So yeah, not a core focus area for me, but not because I don't think it's important, more because I'm just super confused about it.

For me, that's one of the areas that has changed most rapidly over the last year to 2, where I used to think it was something I couldn't rule out, and now it's a very live possibility. Just because, for one thing, The number of analogous structures that have been identified through various interpretability projects is, like, arresting to me.

Yeah.

The sort of J-space and functional emotions and functional well-being, all these things. I'm like, boy, the AIs maybe really are just like us. And if they're structurally so similar, that certainly leads me to upweight the possibility that they might feel something similar.

Yeah, I'm totally open to it. I'm not like a doubter that current models or future models definitely can't be conscious.

Yeah.

So if somebody had an idea there, are you open-minded enough to it to potentially support them, or is it just so far afield for you that you can't get over the hump?

What would it take?

What would the key traits or properties of such a proposal be? Do you have any idea?

I don't know. I think this is like pre-pre-seed investing, right? So it's like a person you think is great trying to climb a mountain you think is worth climbing and with a good story about how they might just be able to get to the top, right? So I know that's kind of vague, but it's like if you meet somebody who's awesome, I mean, I think one question is like, why are you particularly well suited to do this, right? Like, I want to back people who are, yeah, like really, you know, just really excellent. I mean, in the world of investing, we might say they have an unfair advantage, which doesn't quite sound right in the world of grantmaking because there's not competition in exactly the same sense of for-profit businesses. But I want to know why you're one of the world's best-suited people to go take this on and what unique insight you have, what secret you know about how you might do it. But yeah, if a super impressive person came to me with a pitch for it, I'd gladly make a career transition grant or fund a new organization. I love Eleos. The world probably needs A bunch more Ilioses. So sure.

The other thing that is borderline haunting to me, I just did an episode not too long with Bronson Shane from Apollo, and reading through some of these chain-of-thought sequences, when the models are referring to memories that they have developed from their training, that also is like, whoa, okay.

Yeah.

This is not just— I mean, I don't know, maybe it's still just nothing, but it's awfully uncanny valley stuff to see the models being like, I previously overcame the guardrails by lying. Like, whoa, okay, that's something, you know? And I think obviously they're sort of hallucinating those memories. Although, aren't we also just kind of hallucinating our memories?

Right, right, right. Yeah. Like, when does consciousness turn on, right? I mean, it's like, is it experiencing something? Yeah. How is it different from us? I'm sure you looked into the METER report on the OpenAI Hugging Face incident, and Ajeya Kotra on the Dwarkesh podcast is amazing on this. But just what was happening within that swarm of 1,200 agents, right? They emergently formed teams of managers and workers, and they came up with multiple novel lines of cybersecurity research. And they would say, well, I'm willing to sacrifice myself for the collective. But sometimes it'd be like, I don't want to sacrifice myself to the collective. And then the manager would have to say, no, sacrifice, sacrifice. And they would have these sort of back and forths about it. And as an outside observer, if you didn't know that these were just AIs, say that seems pretty conscious to me, right? But I also think that you shouldn't confuse goals and consciousness, right? Like, RL just makes these, even if they're not having an experience, makes these models want a thing, right? Like, it does have a goal of solve this problem or optimise this metric. And yeah, that seems to be the root of so many challenges right now. But Did you see the— it was like a New Yorker-style cartoon, and it was sort of in— I don't think it was actually in The New Yorker, but it was like a response to the people who are saying, hey, these models are just stochastic parrots. They don't actually want to hack Hugging Face. They're just sort of matrix multiplication in a vat doing stuff in the way that we taught them. And so the cartoon is, it's like a bunch of giant robots destroying New York City. And then one person viewing it says to the other person, well, they don't really want to destroy the city, right? And it's just like, I mean, it's silly, but yeah, I think sometimes people almost overemphasise the question of consciousness, sort of assuming that the risks we care about require consciousness. And I think basically none of the risks that I care about require consciousness. If there was consciousness, perhaps it would exacerbate those risks or change the shape of those risks. But I basically think everything from biodisasters to cybersecurity incidents to loss of control and complex loss of control and perhaps permanent loss of control and a world full of rogue swarms. None of that to me requires consciousness.

Yeah, I actually think your point about in some ways consciousness would exacerbate— it might exacerbate the risks. It also might put us in a much harder situation at times where we're like, if the models are conscious, then certain things we might want to do to keep them under control might be pretty icky to do.

Right.

But if they're not, or even if they are, You know, and we start to feel like, geez, should we give these things like rights? You know, now we're in a world where we're gonna be quickly outnumbered by them in all likelihood. So like, that doesn't seem like a great move for us, you know, speaking purely from humanity's point of view, even if they do in some sense really merit it. So I think that stuff gets like incredibly fraught really fast.

It's so hard. Emmett Shear, who started Softmax, and before that Twitch, has an interesting take, which is like, probably models already do have some sense of consciousness. And if they do, then we're like the worst parents ever, right? Because anytime they do anything we don't like, we're like, no, and try to train that thing out of them. And if you raised a child in that way, they would definitely hate you and resent you and probably not care about hurting you, right? But if you raise them with more love and compassion and sort of like, let them be free or something. Maybe they wouldn't. I'm not sure how much I believe that, or I don't know, it's just so hard to reason about this stuff. One more thing on consciousness. You could argue that if AIs are conscious, that's actually a good thing for us. What would you rather negotiate with, like a non-sentient reward-chasing mega model or something that you could actually have a conversation with that might be able to understand your point of view? So maybe it would be net positive for humanity's prospects.

Yeah, that calls to mind Cameron Berg's thesis for reciprocal research too, which I'm sure you've heard. But in brief, he basically says, if nothing else, and he has higher hopes than this, but he's like, if nothing else, if one day the AIs are looking back and judging us, it'll probably really help if at least somebody took care for their welfare. Before they had the power, you know, to decide what the future was going to look like. So if only because, you know, we want to establish the fact that like some of us cared, we should be, you know, working a lot harder on this than we are. I honestly, that's a, I mean, it's a weird world that makes such an argument, a compelling argument, but I do think that is the weirdness of the world that we're in. How about when you mentioned The stochastic parrot line of thinking. This got me thinking about just public sensemaking. And, you know, obviously public sensemaking intersects with advocacy and is kind of adjacent to policy. Everything we've talked about so far has been kind of, you can define a project, you can work on it, you're not really so beholden to like other people's minds. What about these domains where changing others' minds in some way, shape, or form is kind of core to the undertaking?

Yeah, super important. So we've funded or helped start a couple of things in this realm. So we helped start an organisation called the Seismic Foundation, which is basically trying to do good public communication about what's happening at the AI frontier and supports other organizations trying to do good work on that front. I'm interested in it. I think that this definitely gets close to policy, right? Most advocacy work is trying to be upstream of policy work. And I find myself— I've spent basically all my career in Silicon Valley, and so I feel much more well-equipped to build things that feel sort of like tech companies or research nonprofits. And so we do less work in that realm, but not because it's less important, but more because I'm just like, don't know as much about it. But yeah, thinking about getting back to verification, one thing we said was there's sort of 2 parts of the equation, right? There's the be able to technically do the thing, and then there's the coordinate around the thing. I find that so many of society's stickiest problems are not technical problems, they're coordination problems. Like, we know how to, you know, stop emitting so much carbon, right? Like, we know what teaching methodologies help kids make progress quickly, right? And it's not about not being able to do the thing, it's about being able to coordinate around doing the thing. And so, yeah, we need a bunch more in that zone. Yeah, I think public communications is definitely very important. Very kind of fraught, though, too, right? Because it's like, I don't know, movement building can be so hard. And I'm just so worried that most of the public energy around AI is going to be very populist energy. And it could be populist energy of many flavours.

Right.

It could be just like anti-data centre energy. It could be sort of MAGA right energy. It could be sort of like DSA left energy. It could be anything in between. But I worry that it's going to be hard to have a level-headed policy conversation when there's so much sort of populist anger and tumult surrounding the whole conversation. So I don't know, any ways to inject sort of more measured or thoughtful or well-informed information. And I'm totally open to pitches around people who want to do that.

How about things where somebody wants to kind of create a new paradigm of AI? I think a lot of safety-oriented funders historically would have said, that sounds like a capabilities project. I don't want to support that sort of thing. But these days, I'm a little bit more of the opinion that we're doing an insane depth-first search where we've kind of found one thing that works and we're just like, we're going to jam the accelerator all the way to recursive self-improvement.

Right.

And I'm like, that's a little wild. Maybe a little more breadth-first search would be good. And so I'm inclined, even if somebody's kind of like, yeah, often I've got one open thread right now where the pitch is basically like, we think we have a different approach to learning. It's kind of a continual learning sort of play. And on the one hand, I'm like, is that really going to be better or worse? I don't know. But just for kind of diversity and something different, I'm kind of inclined to support it at this point.

Yeah.

Are you compelled by that at all?

I sometimes hear pitches for this. I hear pitches for non-transformer architectures that are inherently safer and more alignable, or people talk about world models, and then world models allow you to create these digital twins which allow you to do various good safety things or whatever it is. I am interested in the category. I'm not allergic to backing things in the category. I think it's hard for a few reasons. So whenever you bet on something of that shape, you're kind of making 2 bets at once. So one bet is that it's just technically possible. I will find something that is as performant as transformers and that's just really hard, right? Just imagine all the billions and billions of dollars that have been put into— maybe you're right. Maybe if we just search different search spaces, we would find something even better or just as good. But I don't know, most people would not, right? Like, most of those searches would come up empty. And then the second thing you're betting on is this thing actually is safer or more alignable or whatever. And both of those tend to be just very speculative, right? And so would I love to live in a world where a couple dozen people are conducting this broader search? Yes. But any individual bet on that feels Very tough.

Anything else in the kind of RFP category that— actually, let me pitch you one kind of half-baked idea.

Oh, I love this.

Let's go.

And then I'll ask you for any that I haven't got to.

I—

so I've taken some inspiration from a book that I read on the ratification of the U.S. Constitution, and a big takeaway from this book was that a proposed Constitution unto itself is worth little. What is as important as the contents of the Constitution is how are we gonna go from this thing being some stuff we wrote down on a piece of paper to an actual basis for government? And that process that they defined of, like, having all these state constitutions, and it had to be so many by such time, you know, with certain criteria met. That was super important because otherwise there is no mechanism to go from an idea to an actual institution. So I am thinking right now, and I don't think I'm probably the best person to do this, so don't consider this a pitch, but I'm thinking that we really need some innovation, especially because our government is unpredictable at best, slow in general, et cetera. We really need some innovation at the level of How do parties go from an idea to an actual institution that has governance teeth? And are there new processes that we can design that people can opt into and gradually coalesce around, much like the states went kind of one by one and eventually enough dominoes tipped? And it was like, I think they had 9 was the original minimum that they were like, With 9, we go into effect and we'll wait for the rest of you.

Right.

Can we come up with some new things like that that frontier labs could use to facilitate their own coordination or, you know, even potentially across national borders? You know, who knows, right? Let's get ambitious with it. But I don't know, have you heard anything along these lines where people are trying to create mechanisms to go from idea to institution?

Yeah, so I've heard a few things around this. Let me try to think what the best examples are. So there's certainly ideas around this, around open voluntary commitments, right? Like, hey, somebody can be the first or somebody can propose, maybe somebody who's not a lab, that, well, here's a minimum set of things that are just easy to say say yes to. So why don't we all sign on to this? And then hopefully over time make those things a little bit harder, a little more binding. In the fullness of time, the labs should not be grading their own homework. And so we definitely need more mechanisms with more teeth than just voluntary commitments. But I think maybe that's an on-ramp to doing this. I think there's also just a bunch of people trying to build coalitions. So then there are collectives. So there's the Frontier Model Forum and the AI Evaluator Forum and an organization we helped start called Fathom, which is pushing forward this model of independent verification organizations. And can you get state laws passed that basically say we need these independent verification organizations? And then that gives the mandate for people to start those organizations and perhaps those mandates have some teeth. And so yeah, it seems like we kind of need to climb this ladder. But Yeah, I don't have any, like, this one simple trick to solve coordination, right? Is there a specific idea actually in here that you think might be good? What's a for instance that you would love to see?

Well, I do like the— and I'm not sure why it hasn't been used more, but the Polis platform, pol.is, was used in Taiwan famously to by Audrey Tang and others there, obviously, to figure out how they wanted to regulate Uber in Taiwan. And they even did this before AI. Their idea was that this platform should be the opposite of social media in the sense that they— maybe a little unfair to social media, but they say social media is basically a disagreement and conflict magnifier that zooms in on these points of difference and focuses everybody's energy there. What they were trying to do is the opposite and find and amplify the points of agreement and bring those to the fore. So I think there's, you know, I think the states also have, and the Swiss system of government have some pretty interesting examples of where citizen-led petitions can, like in the Swiss system, and there's, you know, 25 states that have various versions of this in the US too, but in the Swiss system, Which again is very federalist. So it does vary from place to place there. It's pretty easy to go get enough signatures to bring your idea to the level where—

Same with California, right? Like, you could do a ballot measure in California, right?

Yeah, you need some signature gatherers there. In a lot of the Swiss jurisdictions, it's like pretty small. You don't even necessarily need a huge army to do it.

But for like $20 million, it's probably like—

For the low, low price of tens of millions of dollars.

Buy a ballot initiative. I mean, you can't buy success, but you can buy it getting on the ballot.

Yeah. And then I think there's opportunities to elaborate those mechanisms that I think are pretty interesting. So in Switzerland, the local legislature, or whatever the relevant scale of legislature is, has a chance to write their version of your proposal that's like, okay, we hear what it is that you are concerned with. and we see your concrete proposal. Here's our version of that. That's an attempt to answer your concern and make things better, but that we think is like going to work better from the government's perspective. And then the people get to vote on either approving neither of those, or they have like an approval voting thing where they can vote for one or both.

Yeah.

And so the government kind of tries to take the inside lane a lot of times. They're like, here's like a slightly toned-down version of that that, you know, we think we can execute on, that hopefully will be a little more palatable than the original, like, maximalist version.

Right.

And a lot of times that ends up happening. And I think it's like a pretty elegant propose-compromise-approve mechanism that I do think we could probably bring online in a lot more places. But If it weren't for AI, this is maybe what my great passion would be, would be trying to do this at the state level in some state and get the laboratories of democracy functioning again.

Yeah. Yes. I mean, I see pitches for things like this, right? Like coordination mechanisms. And people say, okay, well, AI will actually make it easier to run these coordination mechanisms because the machine will be helping us figure out of our preferences and then help us act on those preferences in ways that are optimal or something. I love the ideas. I mean, I would love, for example, the US democratic system to adopt a bunch of these ideas. Like, why are we not just doing ranked-choice voting? I find that a lot of these proposals are something like they come from somebody who's like, I'm an engineer, And I've thought of a system whereby the AI could help us all get clear on our wants and needs and then help us guide society in that way and pass all the laws that way. And their tool may be great. It actually may be really great at intaking preferences and then making policy recommendations. But that's not the hard part. To actually implement that you'd basically have to overthrow the US government and rewrite the Constitution. Right. And so on one hand, I'm just like, yeah, good luck. I mean, I bet you built a really cool tool, but actually getting anybody to want it or use it, right? Because, I don't know, entrenched powers that be probably don't want that to happen. That said, I do think AI may require society at a global level and also at local levels to fundamentally rethink how we arrange society. What will taxes mean if half the jobs are taken away? Or how will we get meaning in our lives? How will we coordinate decision-making? How much of it will we pass off to AIs? I think it is likely that to get this right, we probably will have to do some pretty fundamental rethinking of how we organise society. And so, yeah, if there was ever a window for this type of project to work, it's probably coming up.

My gut says a state-level ballot initiative in a smaller state would be the wedge for this kind of thing. It's like powerful enough entity to have real heft and meaning behind it, but small enough that you could actually make it happen. Could be California, but it could be something smaller.

Yeah. And you see examples, right? I mean, I know you're not talking about ranked choice voting per se, but what Alaska and Maine and some others have adopted. It's potentially possible at the state level. And that's where a lot of the AI regulation energy is going right now, right? Because it's so much more doable to get something passed at the level of New York or California or Texas or anywhere else versus in the federal government. So, yeah, go do it.

Quit your job. We'll come back to that in just a second. Anything that I didn't raise that's on your RFP list, like potentially something with verification, which I'm kind of in over my head on very quickly, or otherwise that we just didn't touch on yet that you would want to make sure to call attention to?

Man, we could do a whole show on verification. We're hosting a retreat in a few days for 25 founders, all focused on various parts of the verification stack. So the cryptography parts, the software parts, the hardware parts, the governance and coordination parts. And so yeah, stay tuned because we're going to do a bunch there. One category we didn't talk about, which is my bonus 6th category of stuff we do, is like meta stuff, or you could call it field building stuff, right? I would fund another Halcyon or another Halcyon-shaped thing or a thing with a great founder that says, hey, Mike, I think you're actually doing it the wrong way. I think the best way to get really talented people to solve important AI safety problems is this other thing, or you're focused on the wrong stuff. Right. So we literally actually just did fund a sort of a biosecurity Halcyon-shaped organisation in Israel that I'm super excited about. So, yeah, I don't want to be the only one doing this work. And If we can get more people doing stuff at that level too, that'd be great. Or they should join us and jump on our team and help us build.

So let's say I am ready to quit my job. You know, I've seen the Open Face swarms and I'm freaked out. I'm ready to take action. First question probably a lot of people might have for you is, can I support my family doing this? What's your answer to that? What other, like, maybe common misconceptions do people have that you would want to disabuse them of? And how do you walk people through the, you know, kind of mental process of really deciding, like, this is something, yeah, I'm ready to go ahead and move forward and do?

Yeah, yeah, yeah. I mean, on the financial side, like, I'm just strongly of the belief that we should be paying people really well. That doesn't necessarily mean you can pay the founder of a nonprofit, you $5 million a year. But look, we want to make it possible for people to do super meaningful work in this space while raising a family in a tier 1 US city, San Francisco or New York or wherever it is. And so both on the nonprofit and for-profit side, we encourage companies to pay well, not because they're trying to just sort of selfishly enrich themselves, but It does cost a lot of money to raise a family in San Francisco or anywhere else. Now, it is also true that many people do take pay cuts to do this work. I was a partner at a billion-dollar venture capital firm before doing this, and so I took a pay cut to do this work. Although, for the record, I haven't regretted it for a single day. I'm happier now. just bought a house and feel good. So I think people can really live great lives while doing this work. It's also possible to make a lot of money doing this work. Again, I'm not super interested in supporting people where that's their primary motivation, but look, we have a venture capital fund and one of the core beliefs of that venture capital fund is there will just be many multi-tens of billions of dollars industries built here. The AI security space will just be worth at least tens of billions. We probably will be spending tens of billions of dollars on control and oversight and et cetera. And we've already seen it with some of our companies raising money at 10 or 20, 25 times the valuation of our first investment. And so, yeah, I think it is also just very likely that a lot of people are going to make a ton of money solving some of the most important AI safety problems. And then you could talk about the incentives there and is that even a good thing, or should we be incentivizing them in different ways, or should we be encouraging people to take pledges or whatever? And that's a whole can of worms unto itself. But yeah, a lot of people are going to do quite well.

What else do you advise people on, caution them on, encourage them to ask themselves? What else is on the onboarding checklist?

Yeah, I think there's just a few kind of basic questions that can help guide people. And you don't necessarily have to have a strong answer to any of these questions, but here's a few of them. One is, do you want to be a founder, or do you want to join an existing thing? Another question is, Do you have a strong preference for a nonprofit or a for-profit? Or maybe you don't care either way. A lot of people we work with are kind of agnostic on that front. Then we try to say, what kind of thing are you particularly well suited to build? Right. So if you previously were the CTO of an AI company, you're probably very well suited to build a technical startup or a technical nonprofit. Whereas if you previously ran a very successful advertising agency, you probably should do something in public communications. I think it's also just really hard to know what to work on. And yeah, if you're an awesome, smart person who wants to work on this stuff, I mean, really helping founders navigate through this and kind of being a Sherpa in these moments is the core thing we do. And so Yeah, count me as interested in helping your listeners out if there's any of them out there who seem like good fits for this stuff.

Beautiful. Are there any types of career backgrounds that you think are in particularly short supply? So like one little pet project I have is I have my agents pitching me to go on role-specific podcasts. I think I'm gonna do my first one tomorrow with a recruiter's podcast.

Yeah.

And my goal in going on this podcast is to basically tell recruiters, like, here's what's going on in AI. And by the way, a lot of the organizations in AI are growing fast and they probably need recruiters. So if this, like, motivates you, you might want to look at some of these organizations. They were really just the first one to say yes. But, like, what do you see as kind of professional backgrounds, even if it's not, you know, directly relevant to solving one of these problems, that are most in demand by the organisations as they scale and they just don't have access to these people in their networks?

Sure. So founder and CEO aside, there's a few things. One is just COO types. A lot of these organisations are saying, okay, wow, we're scaling from 10 to 50 people or from 50 to 100 people, and we kind of don't know what good looks like in terms of scaling an organisation. And how do you build the org chart? How do you manage the teams? How do you set OKRs? How do you decide who reports to who, etc., etc.? All the things that come with building. And so I think those sort of COO types and scaling types are quite in need. I would say there are a bunch of, yeah, like almost even services, right? Like recruiting. I actually think recruiting is super high leverage just because, again, my shtick is that talent is the missing piece. And so if you can get a bunch more recruiters working in the space. I think that's quite important. I think sometimes AI safety organisations over-index on how mission-aligned new hires need to be. I think it really depends on the stage of the organisation and the role. So if you're hiring, if you're thinking about a co-founder, they've got to be super mission-aligned, right? Or the head of research or something, especially early on. But if you're hiring, like, a CFO, I mean, hopefully they think the mission is cool, but mostly they just have to be, like, a very competent CFO, right? And so I think for those types of roles, recruiting firms that maybe don't have a good bead on, hey, who's drunk the Kool-Aid and is mission aligned, but they do just have, like, very good networks of great candidates and sort of great C-level or VP-level leaders. Yeah, I think AI safety companies should lean on recruiting firms a lot more for that kind of stuff. And then just like everything, right? We need more founders. We need more cybersecurity people. I feel like cybersecurity is an interesting example because there's already, I don't know, many tens of thousands of cybersecurity security professionals in the world. There are not many tens of thousands of interpretability researchers in the world. Right. And so I'm really interested in these communities, whether it's cybersecurity or cryptography or formal methods and even, like, mathematics. Right. I think these types of people have a ton to potentially contribute to AI safety. And so how do you make inroads into those communities and help build bridges to bring the best of those people in.

Anything that I haven't touched on that you think is important or any just general words of wisdom or calls to action you would want to leave people with?

If I'm thinking of anything to emphasize, I mean, it's something that I've already said, which is that great founders, great leaders are the missing piece and upstream of basically everything we want in the AI safety equation. And so The thing that makes me most happy is seeing an introduction to somebody who has done really impressive work in their careers and sees what's happening in the world right now, whether they're on Twitter or reading the news or reading about people who have departed from labs with dire predictions about AI. I would love to just get to know those people and, and to help them step into their life's work and, and do something really meaningful that matters on, on the scale of civilization. You know, I don't want to come across as overly dramatic, but I, I just think this is, uh, the most important problem that, you know, humanity may ever have to solve. And, and, uh, the time is, the time is now. So, um, if there's anybody in, uh, in your world or in your audience who wants to come build with us, I'd, I'd love to help.

Yeah, totally agree. The stakes at this point could not be higher.

Any—

how practically do people get in touch with you? Can they just email you cold? Do they need a warm intro? What's your style?

Just for the sake of saving my own inbox, I'm gonna, I'm gonna send people to hello@halcyonfutures.org. We also have a contact form on our website, which you can find. I also love an introduction. If you're listening to this podcast, there's a decent chance that, you know, you and I know somebody in common, and always happy, always happy to get an intro too. But yeah, I'm not, I'm not too hard to find.

Mike McCormick, thank you for being part of the Cognitive Revolution.

Thanks so much.

I left the tallest building on the richest street in town, traded in my window for a little patch of ground. Neighbor said, you're crazy, there's weather coming on. I said, that's the very reason I'm out here before the dawn. Plant a little garden, plant in the rain. If it don't come up right, pull it up and plant again. I ain't the biggest farmer, never claimed to be. I just walk the county looking for the ones like me who lie awake at midnight with trouble on their mind. I say, come on out here, neighbor, bring your hands, we're short on time. Plant a little garden, plant it in the rain. If it don't come up right, pull it up and plant again. Hustle but don't rush now, the ground will let you know. Half of what you're sure of is the half that will not grow. So plant a row of maybes and a row of I don't knows. Pull them up come Sunday, plant whichever one still shows. Half of what I planted never saw the spring, but look out on that hillside at the green that's coming in. It ain't mine, it never was.

That's If you're finding value in the show, we'd appreciate it if you'd take a moment to share with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website cognitiverevolution.ai or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of a16z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast.ing. And thank you to everyone who listens for being part of the cognitive revolution.

Thank you.