Your Biggest Lever: Designing your AI Career for Maximum Impact, with 80,000 Hours founder Ben Todd

Hello, and welcome back to The Cognitive Revolution.

Today, my guest is Ben Todd, co-founder of the nonprofit career strategy organization, 80,000 Hours, and author of the book by the same name, which 10 years after its first edition is being re-released today, May 26th, fully rewritten for the modern AI moment. Looking back on the last decade, I think you'd have a hard time finding a source of information That would have better prepared you for the present-day situation than 80,000 Hours. Their emphasis on pandemic preparedness predated by years and was tragically validated by the COVID experience. They were well ahead of the curve in recognizing the importance of AI and encouraging people to work on AI safety projects. Their podcast and blog have been a consistently excellent source of fresh perspectives on cutting-edge ideas. And their free one-on-one career advisory service, was useful to me personally as I made the jump from entrepreneur to full-time student of AI roughly 4 years ago. With that in mind, today's conversation is an overview of Ben's latest thinking on how you, yes, you, can apply your skills to improve the chances that AI really does end up benefiting all humanity. We begin with Ben's thoughts on AI timelines, a question that he cleverly reframes, encouraging people to ask not when exactly AGI or superintelligence will arrive, But when, under varying assumptions, your own personal impact will peak. Ben argues that under all but the most extreme short timeline views, there is still time to invest in positioning yourself for maximum impact. From there, we turn to the top problems that Ben and the 80,000 Hours team see in the world today. That we might lose control of AI systems. That AI could concentrate power in unprecedented and deeply problematic ways. And that we're still not well prepared for the next pandemic. We get Ben's perspective on arguments for and against AI safety-focused people going to work at frontier AI companies, including his thoughts on the critical discipline of continually questioning one's own motives and the importance of peer effects. We discuss different lines of work, including technical research, policymaking and advising, communications, and organisation building, all of which Ben believes have an important role to play. We assess the funding environment with the encouraging conclusion that there is currently plenty of funding available to support ambitious projects. We analyze whether it makes more sense to join an existing organization that's working to scale or start a new one from scratch. And we get Ben's take on what new ideas, including concerns around AI welfare, gradual disempowerment, and space governance, are potentially undervalued today. To be clear, this conversation is not a substitute for the book, which contains a lot more practical advice than we were able to cover today. But I hope it serves to inspire you to think bigger and more prosocially about your own career, and to alert you to the outsized impact that your personal contribution can still have on the shape of the AI future. With that, I hope you enjoy my conversation with Ben Todd, co-founder and author of 80,000 Hours.

Ben Todd, founder of 80,000 Hours and author of the new book by the same name, 80,000 Hours, coming out in the United States on May 26th.

Welcome to The Cognitive Revolution.

Thanks so much for having me. I'm a longtime listener.

That's very kind. I've certainly followed your work for a long time as well. So premise of today's conversation is AI's gonna be a big deal. The decisions that get made around exactly how to shape it, how to deploy it, how to govern it over the next few years could have a really outsized impact on the future for a long time to come. I want to get into that and get your take on how people who maybe don't work directly in the space yet, or who do a kind of work but are not sure if they're making the biggest impact that they can, how they might think about pivoting their careers to try to have the most positive, try to make the most positive contribution that they can. But maybe before we get into that, I'd love to just set the stage with what's the backstory behind 80,000 Hours? What's the story of the name? What are the principles that you guys use as you think about how to advise people and how to think about their careers? And with that table set, then we'll dive into the AI moment that we're in.

Cool. Yeah, the name 80,000 Hours is taken from the typical length of a career. So 2,000 hours a year for 40 years. And the idea is that's the biggest decision you'll ever make from the perspective of your personal life. But even more so from the perspective of your impact on the world, that's the biggest lever you have to pull. And yeah, maybe almost sounds obvious, but so much discourse about what it means to have an impact is focused on things like buying fair trade or turning off the lights or cycling to work. And it's like all these every little help type decisions. But because your career is, for most adults, it's like about the majority of their waking life, it's about half their waking life. So it's more than everything else you two put together. Even a really small improvement in your career, especially in terms of its impact, is going to dwarf the impact of all these other— small changes you could make. So it's really worth thinking about really carefully. But at the same time, so much careers advice just, it's basically a bunch of stories of successful people, or it's just a bunch of slogans like follow your passion. And it's not really given like the huge stakes of this decision. It's not treated with all the seriousness and research that you should actually bring to bear on such an important question. And the organization comes from, well, we were students at Oxford trying, asking ourselves, well, what should we do with our own lives? And we wanted to find something that would be enjoyable and would pay the bills and would make a contribution to society. And we were like, well, which paths would be best? There's like more than one path we could go down. It was really hard to get any good advice on which to pursue. So, We started actually just doing our own research on this. And that turned into a talk we gave at Oxford. And of that, there was 24 or so people in the audience in the first ever talk. And about 6 of them eventually totally changed their lives to coming and talking about the ideas. And several of them came to us and said, well, you should really start an organization about this. This is like clearly These are clearly powerful ideas no one else is talking about. And yeah, that was the start in 2010, was the first ever talk. And then I started working full-time on it straight after I graduated. And then that's what I did for the first 10 years of my career, was then running an organization trying to figure out which careers are best.

So a very small version of some similar advice that I've given to people over time, When somebody is so perhaps unwise as to ask for my take on their career is simply don't be in a huge rush to jump into the next thing. And a very kind of simple bit of math I always tell people is how long do you think you'll be at your next job? Typically the answer is at least 2 years. And then I say, okay, in that case, if you could spend an extra 2 months finding the next job, and even in only considering your narrow self-interest, and it paid you 10% more, then that would be totally worth it. And yet people are often so anxious and feel like an urgency that they can't be idle or not doing something for even 2 months that they'll, in my opinion, rush to do the next thing and really under-optimize what their next phase of their career could have been. So I think you're taking that same thing and making it a lot more prosocial and zooming out to the scale of really thinking about your entire career. One thing that definitely jumps out about 80,000 Hours, and this has been true of the podcast and many of the different things that you guys have put out over time, is that you've been super early on AI and advising people to think about it even back when there really wasn't all that much AI, at least of the sort that we know now to be concerned with. So how did you do that? What was the sort of habit of mind or principles framework that you were using even 10+ years ago that allowed you to focus in on AI being so important. Obviously that take has aged well. If you have any other takes that maybe haven't aged so well, you could take a little detour and share those as well. But I think that will be helpful for people to understand, given your track record of forecasting what would be a good place to spend your career. But I think that'll help build confidence in the rest of advice that we'll get from you today. Yeah.

And part of the context is one of our most important pieces of advice is to focus on problems that are big, neglected, and potentially solvable. And so then there's this question of what are those problems? And yeah, we had AI as a potential kind of nexus of issues on our radar very early on because partly I think it was just because we happened to cross over in Oxford with Nick Bostrom, who wrote Superintelligence, which was published in 2015. And a lot of those ideas originated earlier. And I think some of that was, at that point, they were almost just reasoning based on this very big picture, kind of Ray Kurzweil style argument that computing power is growing and eventually there will be enough to get to a human, a human level AI. And that could happen at some point in the next couple of decades or Even if it's 50 years, it's a huge deal. But then I think where things really heated up a bit was maybe around 2016. And I wrote, I wrote a small call to arms piece in 2016 where I pointed to a number of things saying maybe now is the time to really focus a lot more on AI. And one big thing I think was there was the, there was AlphaGo beating Lee Sedol, which was big news at the time. But then I think the thing that really illustrated was that actually deep learning is a really successful paradigm, and it seemed like we should just expect more breakthroughs to come if you just drew that trend line forward. We didn't predict that it would be LLMs would be as good as they were, but the very broad trend of more important things would come out of this paradigm was borne out. And then also around then, we had OpenAI being founded. So there was this, there was like a definite move towards people really like betting on this approach. And then, yeah. And then also it was like, you could see this, this, the potential stakes here seemed enormous. And it was still, I think people, it's very easy to forget how weird some of this stuff seemed back in, say, 2014. And there's that famous Andrew Ng quote where he says, worrying about AI is like, worrying about AI taking over is like worrying about overpopulation on Mars. And that was like the attitude of a lot of people in the field at the time. But like, yeah, and that meant that this, these things were really neglected. So we thought like, we don't know exactly when it's going to come, but there's a trend. And if it does come, it's a huge deal and it's still being really neglected. That was the basic case for it at that point.

So fast forward to today. And I wonder how AGI-pilled are you now? I think this is also a really important table-setting part of the conversation because so many AI conversations go sideways based on differing expectations of just what's gonna happen, how fast, how big of a deal it's gonna be. And for the purposes of what to do in the next phase of your career. I think there's like very legitimately some people these days that are like, I may only have one more phase of my career. And so it's kind of time to go all in on something or liquidate whatever various forms of capital I might have accumulated. And obviously others are planning for longer timelines. So I don't know that it's— that you wanna put yourself into a box as having a very particular view on that. But how are you thinking about it and how should that color the rest of the conversation?

Yeah, I would say I'm pretty AGI-pilled on the scale of people. Though I really think like this, it is a spectrum. Like over time, as more and more developments have happened, I've become more and more AGI-pilled. And it like, it takes time to really move your gut on this. But I like to think in terms of, this is a big simplification, but I plan in terms of 3 main scenarios. And the first is that something basically AI 2027 or what people like at Anthropic or OpenAI say they're planning to do and expect to happen, which is that they manage to automate AI R&D in a couple of years from now, like 1 to 4 years from now. And that can cause an algorithmic feedback loop, which then means you get maybe 5 years of AI progress in 6 months or 1 year or 3 months, something like that. And then we have like pretty powerful, pretty general purpose autonomous AI, say sometime around there, maybe even 2028 in the more aggressive versions of that. So that's what kind of seems to me as the like realistic, shorter timeline, faster takeoff type scenario. That one's quite interesting because you have this like powerful AI before most jobs are automated and before you have robotics. So then there's this kind of like Crazy deployment process that happens next. But then I do still think it is quite possible that those timelines are a bit too optimistic and AI that can do AI R&D will take a bit longer. Maybe it's in the early 2030s. Compute scaling will start to slow down probably in the late '20s because just like at some point you use up, we're basically almost there, fab capacity is all used up and then you have to start building new fabs and it takes a bit longer. Compared to just switching over capacity. And then it's still very, it's very possible that an algorithmic feedback loop isn't possible and you can't get like in a big acceleration. And then I think then a lot of people think, oh, intelligence explosion won't happen. That's wrong. It will still happen, but it will just have to be driven by building way more chips. So then maybe it takes 3 to 10 years instead of 6 months. So that's like, you're getting to really powerful stuff by the end of the '30s. So you can think of that as like a medium timeline scenario. And then the third one is just, maybe this seems increasingly unlikely, but just the current paradigm runs out of steam and like compute scaling slows down, it becomes too expensive, the revenue isn't high enough or whatever, because they can't keep improving it. And then you might have a longer plateau until there's like a new paradigm. Or just like you wait for economic growth to build up enough computing power to push on to the next level. So I like to think in terms of like, you want to be trying to plan across these 3 scenarios.

Nice. How, I don't put too much weight on that last one, for what it's worth. It seems to me like the path is pretty clear. And I do have a little bit of a tendency to, that's interesting. I wouldn't even say I've necessarily underestimated the timelines for capability advances in AIs. If anything, I've probably slightly overestimated how long it would take for models to get as good as they have already become.

Yeah.

On the flip side, I've tended to expect impact to come a little sooner than it has come. And that disconnect is definitely an interesting thing to try to constantly recalibrate myself on. But I can't find the wall that would lead us to the plateau scenario.

I could agree, but I do think the compute issue could be that wall. And one part of, one way of seeing this is that we have maybe 4 years of current scaling roughly, or till 2028 and a bit slower till 2032. And if at that point you haven't automated AI R&D, then you might plateau around there and it might just be very hard to clear the final bottlenecks.

Yeah.

That's the kind of way I would make the case. It is also interesting to see people like the AI 2027 forecasters. If you look at their confidence interval, I think it's their 80% confidence interval is something like 2027 to 2050. So even Daniel Cucatello is saying there's a 10% chance that it's beyond 2050. So yeah, I think there's some chance of it, but it was not the main thing I would bet on myself. Yeah, I agree.

Hey, we'll continue our interview in a moment after a word from our sponsors. So anyway, given that context, there's, and people can choose for themselves, right? Exactly where they wanna sit there. You have a 5-part framework for how people should think about their careers. Maybe introduce that, and then we, you can maybe comment a little bit too on how you would reweight those different pillars of building a career, depending on where you sit on that timeline question?

Yeah, in the book, there's many different frameworks, but we have this one which I call the career framework. And it's if you're comparing a list of options and you want to do a side-by-side on them, what things should you look for? And yeah, that in brief is the impact of the option, the career capital you'll get from it, which is overall career development, skills, connections, credentials, character, Your personal fit. So how good you will be at it compared to other people, all the other things that matter for job satisfaction and your personal goals. And then you could have exploration value at the end. So that's what will this, taking this job tell you about which jobs are best for you in general? What will you learn about your career from taking this? Because many jobs are an experiment as well. And then, yeah, with the effect of AI, The key thing is the longer your time horizon, the more important gaining career capital and exploring are, because you have more time to make use of those skills you've gained and that information you've gained about what fits you best. And the more you have a short time horizon, the more you should just focus on doing whatever is best in the next couple of years. And so as AI timelines have come down, Career capital and exploration value are a bit less weighted, but you wouldn't ignore them entirely. And one way to see that is like, suppose you have a very short timeline and you think, okay, and also to be clear here, when I say timeline, what I actually mean is not when we will have AGI. It's not an AGI timeline. What matters is which years will be most impactful. So, If you want to have an impact, which, at which point will you be able to have the biggest impact? And the time between now and then is basically your time horizon for acting. It's like when the best opportunities are. And so if you're very focused on AI, that will probably be the years immediately before and after AGI is developed. Though in a slower takeoff, it could be 10 or 15 years after, it might still be all kinds of crazy stuff happening that you could help with. So probably your timeline shouldn't be 3 years. Like, in expectation, there will be things to do after 3 years that are really impactful as well. So you should probably be planning over— Ajeya Kotra had this recent post about whether you should get married. And she said, the expected length of your marriage now is 10 years, which was like her averaging over different timelines in terms of Like when the world would just become like so crazy that it's not worth planning beyond that. And maybe 10 years, it feels like a little bit long for a career planning timeline, but something like 5 or 10 years, I think everyone should be thinking about some, at least that long a period. And then, yeah, should you ignore career capital and skill development? No, because if you could spend 1 year and make yourself 20% more productive, You would recoup that after roughly 4 or 5 years. So even if you only had a 5-year timeline, time horizon, you should still make those investments that will pay off over that period. And there's often a lot of things people can do to increase their impact by more than 20%. Yeah, retraining in machine learning, lots of people have done that, or entering a policy career. You can often transfer into one in a year and then maybe have much more impact after that. So yeah, I still think it is really worth thinking about what you can do to maximize your impact over that whole time horizon, but it's just a bit shorter than in the conventional world where maybe you have 40 years to play with and you can really invest for 10 or 20 years and then pay it all off in your 50s.

Let's also talk for a minute about what you think people should be trying to move the needle on. AI is one of the premises of this show, and one of the reasons I enjoy making it so much is that it's obviously a general-purpose technology, a horizontal layer, something that kind of intersects with everything. And so increasingly, like, work in AI is almost saying nothing because it's like in any sector, in any business, there's going to be some sort of AI touchpoint soon, if not already. What do you think are the big levers that really matter most? And maybe give a little argument for each of those big focus areas.

Yeah, I'd start with what problems to focus on, and then you could talk about the best interventions or solutions to those problems. And yeah, again, here we would use the framework of which problems are most important, most neglected and most solvable. And with importance, I often, you can partly think, you can think like, how likely is this problem? How big would the scope be if it happened? I also quite like to think about urgency because some problems come before the others. So you need to address them first and then deal with the other ones afterwards, or they might help you deal with other problems. So you want to deal with them earlier. And then, yeah, we have a list on the website that we're always updating depending on how the assessment of these factors is changing. But right now, yeah, we have loss of control of autonomous AI as the top, just because the stakes would be so big if this is loss of control of human-level or beyond AI, which might be irreversible, could eventually result in total human disempowerment. And it is much less neglected than in the past. But we're still probably only talking about 1,000 or 2,000 full-time people working on these risks compared to now, what is exactly where you draw the circle around the AI industry is unclear, but could easily be 100,000 or a million people essentially working on AI capabilities and accelerating it even faster. So that, yeah, that still seems like a big Neglected important risk. But then over time, we broadened out other risks that we're focusing on. And one that is a bit newer to the list is concentration of power. So there's a bunch of ways AI could be very concentrating or very centralizing. One is if there is an intelligence explosion and a feedback loop. In exponential growth, the gap between the first project and the second project stays the same. It's, if it's a 2-month gap now and they both grow exponentially, it's still a 2-month gap in a year's time or 2 years' time. But if you go into hyper-exponential, like super-exponential growth, then the gap actually widens. So you could have one company drawing well ahead of the others, gaining a digital workforce that is equivalent to a whole nation's worth of people today, which would be like more power than any single company's ever had before. A bunch of other ways AI could be centralizing. We saw the Department of War were like pissed off that they— Weren't being allowed to use Claude to spy on every American. Yeah, that's a thing that wasn't technically possible in the past because they just didn't have enough staff to analyze all the data and figure out what everyone was doing. But with enough AI, you could literally trawl through every bit of public data about people and probably build up quite a big picture of, quite a good picture of what they're doing. So it makes surveillance much more effective than it's ever been in history, which Would be great news if you're a wannabe dictator of any kind. And then, yeah, a lot of these elements of this concentration of power issue, I'd say, are even more neglected than alignment. There's very little thought right now given to things like exactly what instructions will more powerful AI obey in different circumstances. Is it possible for the CEO of a single company to just tell the AI what to do? Or should there be more safeguards than that? And then the third, the third is the, maybe I should have started with, is engineered pandemics, which seem like they'll be possible eventually, even without AI, but AI could make possible sooner. And yeah, it's like, definitely seems possible to make a pandemic that's much worse than any naturally occurring ones. Countries like North Korea might do this for Deterrence. If you invade us, we'll release this virus. Mutually assured destruction, but one that is much easier to build than 1,000 nuclear weapons, which is what that kind of thing used to require in the past. And then that could also just get out because of a lab leak. There's lab leaks all the time. So if these things ever get built, they'll probably end up getting out at some point. Yeah, those are the top 3. And then there's also, which we can talk about later, some Even more weird and neglected issues that are interesting for some people to work on, even if they're not like our kind of mainline suggestions?

These AI companies and their marketing campaigns, they're getting weirder and weirder. First it was, the AI might kill us all, we better be really careful about it. Now, I was just at an event in San Francisco this past weekend, and there was definitely a lot of talk from people at the frontier companies about concentration of power. And that's another really weird marketing campaign for them to be running. The, we better be careful about this technology because we might end up having a crazy amount of power that nobody else is going to be happy with. It's a very, just a very strange way to hype this technology.

I mean, I just, this whole like, it's marketing thing just doesn't really ring true to me at all. I just think these people, they just actually believe AI is a huge deal and that there are these risks. And that's like the simplest explanation of what's going on, not that it's a some kind of like plan to make themselves seem really important and therefore raise more funding.

Yeah, it's quite the double bank shot if it is. Let's do maybe just a minute of marketing for the AI companies, or at least I will. Obviously all those top cause areas that you laid out are downside risks, and I think 80,000 Hours, fair to say, has been associated with downside prevention or downside mitigation over time. Is it fair to say that is in part because, speaking for myself, my gut says if we can avoid all those problems, we're going to have it pretty good. And I don't know what life is going to look like. It's definitely kind of a fuzzy picture. I do think one of my own personal cause areas that I recommend all the time is just trying to develop more concrete visions of what a positive future and a rewarding life in the context of AGI abundance could look like. But are you coming at it from the same perspective where your belief is like, there is a lot of upside as long as we make it to the other side of some transition? Or like, how do you think about upside, making sure we realize the upside? Do we need to, or does it just take care of itself?

Yeah, in general, I just want to focus on whatever has the most impact, which could be increasing upside or it could be avoiding downsides. And the question is just, what are the best opportunities right now? And yeah, my, as it, maybe this is going to get too theoretical, but I think it's interesting. So there's like, you can think of roughly 2 ways to improve the future. There's speeding it up. So there's like bringing better future earlier in time. And then there's making sure it happens at all or changing where it ends up in the final reckoning. And yeah, there's this interesting point that if you think the future is going to get way better in the future, and you can speed that up one year, it doesn't only help people now. it helps, it actually means you have one extra year of that great future. And Beth Jezus actually makes this point in the AI doc. He's, well, we need to develop AI ASAP because we're bringing forward the glorious future of everyone being in space and like crazy technology and all these things. And this is the argument that Nick Bostrom makes in Astronomical Waste. And that is the astronomical waste he's talking about. But then there's like a second half of the paper where he points out that If we're going to get to this great future, it up only gets you like a little bit extra of that. But if you can avoid an extinction risk, that means the whole future never happens at all. That's like way higher value if you actually believe in this better future. And so, yeah, like I was joking that Beth Jezus is like someone, he's read the first half of Astronomical Waste, but then kind of got bored and didn't read the second half where he's like, no, this is not actually what to focus on. And Toby Ord has a paper where he works this through in more detail. So that's the very high-level perspective I come at this with, is AI development is already going extremely fast. And by working on just bringing that a little bit earlier, that's not having that big an effect on the world. If there's these risks that might jeopardize the entire future, and that might be a lot better because of this technology, then There's like huge value in reducing those risks. And then at the same time, that's way more neglected because as we just said, there's already like a million or so people just working on accelerating AI capabilities, but the number of people trying to avoid these big downside risks is in the thousands. And so that's, yeah, that's how we see the basic case for focusing on those right now.

Hey, we'll continue our interview in a moment after a word from our sponsors. Gotcha. Okay, cool. Let's get even more into the nitty-gritty of it. I think obviously a lot of this stuff is gonna be highly individualized, right? So I wanna preface by saying nothing can really be one-size-fits-all cuz everybody has their own skillset, background, et cetera, et cetera. But we can, I don't know if you have a better way to frame it, but I think of As we get more and more specific framing the conversation in terms of if this is, here's how to think about it, and individuals will probably only see themselves in certain sections and not all that. There's to some degree only so much that somebody can change themselves and probably shouldn't try to go too far in terms of making themselves into something that they're not, or that would be like too unnatural for them to try to become. But with that caveat, and you can elaborate on that caveat, What sort of, how would you break down kind of the roles that people, and maybe we could do roles and skills. What are the kinds of jobs that exist? Which ones are in most demand? Who are they looking for?

And on the really broad approach, a very simple approach I like is just come up with a shortlist of plausibly impactful things that could tackle these problems and then choose between those based on personal fit. So if you have 5 ideas, try and figure out which one you're best at. And that's often the best approach. If you really simplify it down. And then, yeah, in terms of what roles to focus on, a couple of the most important types. One would just be, there's a lot of technical research and just like engineering that needs to be done. Loads of subprojects in that. One that's on my mind recently is Meta, which does some of the best AI evals so that we can actually know what is happening. How close are we actually to automating AI R&D, which is, that might be the most important thing going on in the world right now. We have very poor measurements of it. They're pretty much the leading group in doing this. They were saying recently on Odd Lots, they have 20 really useful projects they'd love to do, but they only have capacity to do 2 or 3 of them because you need a bunch of engineers to implement all this stuff. Yeah, AI control research, AI interpretability. And one interesting thing that's happened in the last few years is a lot of these, in the past, this was more of a high-level conceptual research kind of bottleneck. But now it's much more about actually engineering and there's just a lot of just like concrete, okay, can we use this AI to monitor this AI? Can we red team in this way? Can we detect deceptive behavior quickly? And just like figuring out all the systems to do that type of thing. I think second would be government and policy. A lot of this stuff will have to be, will have to involve governments in some way. There's generally a There's still a big lack of expertise, people who straddle the AI technical world and the government world. And so just really building up that scene of people and working on some of the priorities there, which we could go into. Third, I might say, is communications. Like, the level of understanding of AI is still really low. Very few people are working on the risks. Just getting the ideas out there, improving understanding of them, mobilizing people to work on these things. There's huge amounts to be done there. And then a 4th big category would be just organization building. We just need people to run the organizations that do all these other things. And that's just management, accounting, legal, HR, recruiting, just loads of getting stuff done type skill sets. There's also lots of other need for just all kinds of specialists in other areas, lawyers, economists, engineers to work on biosecurity stuff. Even historians have useful things they could potentially do. So yeah, but as a very 4 broad categories that apply to a lot of people, that would be, yeah, the ones I just said, technical research, government, comms, and organization building.

Fun fact, I have had one historian on the podcast in the 340-some episodes that we've done. And Mark Humphreys, he was doing some pretty interesting stuff in terms of just using, and of course his pipeline has changed a lot, but using AI models to transcribe and make sense of all these old documents that exist in the Canadian archives that just nobody has really ever looked at. But also because his problem is so different from almost anybody else's problem that's using AI intensively. He had a really interesting and like quite viral post on Gemini 3 just before it came out, showing how, like, zeroing in on a particular ability that it had unlocked to kind of reason about the, what it was seeing in the visual documents in ways that prior models just hadn't been able to do. And I think that is a really interesting example of how something that just seems so far afield, because AI itself is getting so far afield, there is, even in such an unexpected niche, there is the opportunity to make interesting discoveries that really contribute back into the mainline discourse and advance people's understanding. So, I would again just encourage people to think pretty broadly about just how many different opportunities there might be. And I also can say again from this event this past weekend in San Francisco, yes, METER in particular is desperately trying to find people.

All their—

sign of the times, all their— this is maybe a little bit of an exaggeration, but the attitude was, yeah, all of our measurements are pretty much saturated and it's getting tough to take the skill size higher than we already have. So there's a lot of work to be done as the models are racing past our ability to measure them. They are looking to staff up and get them, bring as much talent to bear as they can on keeping a handle on exactly what the current capability frontier really looks like. Many of the things that you outlined there, Notably happen both in the frontier companies that are developing the AIs and in a variety of other organizations that kind of orbit them in some ways, in some ways check them, in some ways maybe even oppose them. This has been debated for a long time, and I think people have very different intuitions, and some of the discourse has gone around in circles at times. How do you think right now about whether or not people should try to go to the frontier companies and contribute there versus trying to make, perhaps holding the person themselves constant and the kind of contribution they're going to make versus doing that from some other outside angle?

Yeah, I think the short answer is it's complicated, but yeah, if you want to do technical alignment and control research, they are some of the best places in the world to do that research. They have some of the strongest research teams and they're also in a great position to actually implement the research, which I think is actually a big part of it because you can come up with some idea, but if someone doesn't actually see through all the details in implementing it in the product, it doesn't actually help that much. But on the other hand, a lot of people have done great Research outside of the labs, I would name Redwood Research as another group here who helped to pioneer the AI control agenda. A bunch of other really useful pieces of work. They did the deceptive alignment research with Anthropic. So it's definitely possible to do work outside of the labs. Yeah, I think another big factor is a big, the big argument against is maybe you're speeding up AI development, which is hastening the end of, it's hastening lessening the risks. And I think that is worth thinking hard about. And each individual has to make a judgment on that themselves and how they want to relate to that. I think also another big factor is how aligned you are with the people who work in the labs. And yeah, in general, I try to avoid adversarial strategies. But if you have the attitude, well, AI is happening, And I prefer to have a more social-minded or safety-conscious company win, and I'm going to try and help them win. I don't think that should be entirely ruled out as a strategy. I think a lot of it just comes down to, I think what drives a lot of the disagreement here is just what P-doom is, basically. People who think, well, we're very likely to have an existential risk and alignment research doesn't really have any hope of working. Basically think the only option is to have an indefinite pause. And by working at the labs, you're not really helping with that or actively making it worse. Whereas people who more have the attitude that, A, this is happening, we can't really stop it, and B, we'll probably get through, but it's more a question of making the chances as high as possible, tend to be much more keen on working at the labs. So I think that's the other— That's another really big source of disagreements over this.

Yeah, there's so many different angles on it. I find them all compelling, but I do also find myself going back and forth on it at different times. I guess maybe just to name a couple of examples that people might want to check out, or a couple of arguments that people might want to check out. One, which I think it came from Redwood Research, is the 10 people on the inside argument. Which is basically that unless there is some international treaty or whatever, which I certainly don't rule out, but we're not that close to it at this moment, then this is going to happen. And if it's going to happen, then a small number of people working within the companies who really care about the right things and can take important actions at critical moments Could be one of the most important places to be, one of the highest leverage places to be, certainly. Then another argument, especially if you want to do alignment or interpretability research, is you can do a lot of that research outside of the companies. But one of the big things that we're worried about is that models are going to change in important ways, potentially In a pretty short period of time, potentially due, and potentially in surprising ways, even to people within the companies, due to emergent, emergent properties that do arise, right? Like we see the general pattern of AI capabilities advances is like the loss function is dropping smoothly, but the specific tasks that the model can do often seems to have these more like discrete jumps from one generation to the next. And we have seen obviously many of those on the capability side, but also some of them on the problematic behavior side where it seems like we've gone from models didn't really do any sort of deception to now they sometimes do some deception. And obviously that's not something that the companies train for. But with a certain scale of RL or what have you, all of a sudden that seems to pop up from one generation to the next. So again, being on the inside of the companies and having access to the frontier models might be pretty important to being able to discover that next bad behavior in time to make a difference about it, or to just understand the internal, if you're thinking interpretability, to understand the internal workings of models well enough to, or soon enough, or at the frontier that matters the most to really make the biggest difference. I'd be interested in your reaction to all of those, if you have any. I guess I would say for me, the alignment thing kind of resonates more than the interp side. I feel like with alignment, You do see these kind of qualitative shifts in behaviour. And I think trying to align whatever is the best open source model at this moment is probably quite a different activity from doing alignment work on Mythos, for example. Whereas our interpretability understanding is still sufficiently nascent that you could probably still make quite a bit of headway on something like a GPT-OSS or any of the Chinese models, there's still plenty there that is not understood. Any other analysis you would give on those fairly well-known but still important arguments?

Yeah, I think you're exactly right. It depends on the specific project. Maybe to throw another argument in there, I do think it's very easy to be biased to want to work at It's pretty cool to be like, oh, the most impactful job for me is to work at famous, rapidly growing, highly paid company that everyone is talking about in the world. Like, that's a very convenient place to be in, right? So I think people should also like question their motives when they've concluded that's the best option. In general, if someone's going to do it, I would want them to have a pretty concrete strategy that they've actually thought about and like why it needs to be executed in this way rather than something that could be executed in a different way, like you're suggesting with the open source models in Interp. Yeah, with 10 people on the inside, I think, yeah, part of the idea is because it's like diminishing, there's diminishing returns, there's low-hanging fruit. So being able to just take the first couple of opportunities might increase safety a lot. Even if we're very far from what would optimally happen. You might also be able to do things like raise the alarm about stuff that's got really out of hand. But it does seem like most people, it's not sustainable to work at a company where they just disagree with the mission of the company, and they tend to only last a couple of years before quitting, as we saw with pretty much everyone's quit OpenAI who was especially safety concerned over the years. So in general, most people should probably work with a company where they're aligned with the mission and they don't have that source of friction. But if you are that very unusual type of person who could work somewhere where you really disagree and you think you might be able to make things a bit better at the margin, I do think it is worth considering because, yeah, it's like basically all the options seem bad in different ways. The idea that no one who's worried about, that worried about safety works at any of the top labs or top companies is also a pretty scary world to me. It seems like ideally, if it was possible, you'd have some people at all of them. But then, yeah, whether there's actually people who can actually carry through with that and make it worthwhile is another question.

Let's click on that in a couple of different ways. I think I really appreciate you saying people should question their motives. I think this is like profound, of profound importance for individuals at AI companies and for AI companies as organizations. There's just a lot of somebody's gonna do this and we're better than the other guys, so we better do it and therefore we need to win. And yikes. This sort of leads us to a competitive situation, which I think is like quite problematic. I always say it's the smartest of times, it's the stupidest of times. And this image that I have of Sam Altman and Dario from the AI event in India where they couldn't bring themselves to join hands as everybody else on the stage was joining hands, it will be a real time capsule and a real shame if we end up messing this up. And there's some, Alien historians or whatever are one day excavating the planet and come across this image and they were like, oh, there's, they were so close. They almost created a world of abundance for themselves. But in a sort of, in a scene fit for a Greek tragedy, here you have the 2 leaders who are basically building the exact same technology at very similar companies with very similar strategies, hating each other as individuals so much that they couldn't manage to figure out any way to work together or couldn't see the virtue in what the other one was doing.

It's a real mess.

That is a real mess. And I think that's something that everybody needs to think about for themselves. Like, how do I play a positive role in making that less of a mess and not contribute to just everybody escalating the same dynamics that currently exist? So with all that said about like counterfactual analysis of this sort of like Somebody's got to do it, so I guess it should be me. And, and how do you advise people to really take an honest look at their own character and assess, am I actually going to be able to go into such an environment and do what I, in the abstract now, think I will do or want to do? That's a hard question. And I think it's hard to get that kind of objectivity On oneself, but it's a pretty important crux for whether you should do it or not, right? Is can you hold up under that pressure and be true to your values when the stakes are high and when it really might cost you something? Yeah. How does one get a good sense for whether or not they're up to that challenge?

No, I mean, I think this is, is a huge topic. One starting point would be like, like you're kind of gesturing at, I think a lot of the reasoning in there is, it's pretty shallow. It's just, well, like, company has the good vibes. So I'm going to really encourage people to try and be more objective than that. Like, think of these companies like political actors. They have, all the leaders have many incentives. They have many different goals. Not all of which are noble. And then try and just like actually get down to the level of specific decisions and track record that they have. And did they actually make the hard decision in that case or not? And try and be grounded in objective decisions as much as possible. Or when they made this commitment, did they follow through on it? What do people say about this person's character who know them well? And like actually try and take a view on it rather than just go with vibes. That's about assessing the company. With assessing yourself, that's even harder because you're going to be super biased there. This is just one small technique, but I do think cultivating some close friends who are willing to call you out on stuff. Is really helpful for this. And most people won't really call you out. They'll just go along and they want to have a friendly relationship. So if there are anyone, any people in your life who will do that, I think that's really valuable. Yeah. I mean, potentially making commitments to them as well about almost like a written down commitment. If this type of thing happens, I will do this. And that can make it, I think it's very easy for kind of your standards to slide over time gradually, right? Ways to just have some kind of pre-commitment in the line where you're going to reassess, I think is really helpful as well. I guess a fourth thing that comes to mind is like, you will become more similar to the people you work with. So again, that just means really taking seriously the context. And like we said with the 10 people on the inside, Most people can't, if they think there's like a company that's being really reckless, most people can't actually hack, like going in there and trying to make it better. That's like actually a very difficult thing to do. So I think be skeptical of your abilities in the prevailing culture, in the people around you. And so just focus on picking the right culture in the first place.

One other argument that people have made a lot who I think are especially skeptical of AI companies, but I guess just maybe just have a hypey doom, as you framed it earlier, is that safety work is ultimately still counterproductive. Sometimes that's framed as saying safety ultimately is just an accelerant of the capability frontier. For example, we had things like RLHF that were supposed to be a safety measure. And in some sense, they do make the model, any given individual model, safer to use. But then they also have had the property of making the AIs much more useful and just accelerating the overall pace of development in the space, bringing lots more resources in as it became clear that, hey, these things can in fact be harnessed and become super useful. And I wonder how you feel about that now. The argument that those people usually make is just try to shut the whole thing down. And I, this is not my point of view, to be clear, because I do really value the upside that we're getting. But there's, there is some truth to that argument. I can't quite, I can't quite dismiss it. Where, how would you— assess that argument and how moved people should be by the sort of, even AI safety, even technical AI safety work is still bad argument?

I mean, technical AI safety definitely does seem to sometimes improve capabilities and therefore accelerate AI overall. The question is just, are you getting enough benefit out of it to make up for that cost? Because in a world where we don't do any safety research, Well, it seems pretty scary, right? My sense is it's best to have a thriving safety ecosystem who are really tracking the biggest risks and trying to do something about them, even if they will also accelerate things a bit even more compared to a counterfactual where basically things are still going very fast, but we don't have a safety research ecosystem. Yeah, it does depend on whether you think, if you just think alignment or AI control is ultimately impossible, then Then yeah, it doesn't make sense. I think a bigger point I might make about this is like, I guess I come at this from a perspective of just, we're very uncertain about all of these different priorities and what's going to happen. And that, at least for me, makes me intuitively want to see a bit of a portfolio of efforts across different scenarios. And so I'm like, I'm pretty into the idea of there being some people who are really trying to build support for some type of international pause. I think like a 1-year pause at the point at which an algorithmic feedback loop becomes possible or some sort of temporary pause seems like it could be extremely helpful. At the very minimum, that's like a thing that I think many people could agree upon. But then I do also want there to be a lot of effort going into the scenarios where no, this is happening, like it's not really pausing. And just betting 100% on the pause seems That's leaving a lot of like low-hanging fruit on the other scenarios on the table.

Yeah.

Portfolio approach, I think, is kind of always an answer to this. It does seem like there's room for—

I mean, it depends on your probabilities on the different scenarios, like what your portfolio would be. But at least for me, they're also pretty spread out.

Yeah. Okay. Let's keep moving through a couple of different other areas. I don't know if you have anything more to say about working at frontier model companies. Is there, if there's not, we can move on, but if you have any other kind of specific advice, I think one of the things people are gonna ask, there was a pretty good post on this just yesterday that we can link to in the show notes. I forget who it's from, but it's, it was a, I think a pre-training lead at Google, and got some plus ones from, uh, others around the space, including Sholto from Anthropic. It was very much leaning into just like grinding hard on technical skill. I don't know if you would second that or highlight anything else that you think is the most important way to position oneself for work at those companies. Obviously it's super competitive. There's not a ton of— They're hiring a lot, but it's still sufficiently competitive that I don't think it's an easy place to find your way into. So any other thoughts on that before we move on to kind of other venues for possible impact?

You mean like advice on how to get jobs at these places? Yeah. Yeah, this is just like very general purpose advice, but there's nothing to really substitute talking to people at the company and just Saying, okay, if I was going to do a 3-month crash course to be in the best position possible to get this job, what should I do? And that answer will depend on different people, but often, I, you know, learning technical skills is really useful, though they're also hiring a lot of other roles. So it would depend on what role you're going for. And yeah, I wouldn't have a ton to add on top of what people in those companies would say.

Gotcha.

Then let's change gears to policy. This is another, I think, very black box sort of thing to many people, right? There's, if you're not inclined to do hardcore technical work or research, and especially if you have a higher p-doom, then you're like, how can I get into the policy arena? I'd be interested to hear your thoughts on, first of all, like what policies do you think are actually robustly good and worth advocating for? And then how should we think about how we can position ourselves to actually move the needle on the likelihood that those things do in fact come to pass?

Yeah, we touched on a strategic pause earlier, but unlike, I agree with the current administration and just the vibe, that's very unlikely to happen in the short term, but we need to think out A couple of years in the future when people will be freaking out even more. We might be on the cusp of an obvious feedback loop. It might be a new administration. That could be a pretty different scenario. And I think at least a strategic pause could be something that a lot of people would sign up to. They'll just see that it's better for everyone. And the common response is like, well, what about China? But like, You would say to the Chinese would obviously have to be part of the deal. And you would say, well, we're going to pause if you pause, that's the deal. And they actually have more incentives to take that deal than us because they're behind. So like pausing is better for them. And they also care about the risks a bunch as well. Like they've had high-level talks about AI risk among the like very top leadership of the party. So I do think like kind of laying the groundwork for that. And one thing in particular is you can't really do that unless you have some way of enforcing it, which means you need at least some minimum level of tracking the compute and knowing like where major AI training is taking place. And if that's not set up in advance, you just can't actually really do it. So I think things around compute tracking. are really useful. This almost sounds too simple, but getting the capacity to have some type of off switch, because right now we couldn't actually, there's not an easy way to actually just turn off a lot of compute quickly if there was an autonomous AI replicating in a bunch of data centers. And that seems like a relatively low bar thing you might want to have prepared. And then there's a bunch of just Maybe more boring sounding things, but I do think more transparency is good, like making sure the government can actually track what the capabilities are over time. And right now it would be possible for a company to start an intelligence explosion and keep it secret potentially for what, like a couple of months at least, which could be important. So measures to like really make sure we would know if that was happening seem really good.

Yeah.

There's a bunch of other ideas. Like, I do think the idea of having red lines and emergency response plans is also, seems like pretty hard to object to. So like dangerous behavior where that would trigger further investigation or certain responses by the companies and having that like widely agreed upon across the industry. Yeah, but this is more on the alignment stuff. Obviously, there's a whole set of other things for pandemic prevention and avoiding concentration of power and a bunch of other risks. But yeah, in general, I just feel like there's a lot that could be done, even if it can't be passed to the next administration. Like, in the current administration, you can be— there's a lot of groundwork that can be laid to make it much easier to implement these things in the future.

How would you handicap where we are in terms of the development of policy proposals versus advocacy or other efforts to bring them into the Overton window and eventually make them happen? I do, I guess one way to say that is, do we have most of the good ideas we need, or do you think there's a lot of like blue sky policy room still out there to explore and develop new ideas? For the ideas that we do have, are they well developed or do they still need a lot of work to really flesh them out? And then, yeah, the third part is obviously these things are not immediately about to happen. So for that one, I guess it's more like, where do you think people should go? What's the best margin to work on to actually move the needle probability of it happening? But yeah, I gave you a lot there, so take your time.

Yeah, I, yeah, I'm going to slightly cop out with like portfolio approach. Again, I think all of these things are pretty needed. Like a lot of policy proposals aren't super fleshed out. There's a lot more that can be done with fleshing them out. I think we also need people developing new broad approaches. Like many of the approaches have only really been developed in the last couple of years. So I'm sure there's a lot more that could be discovered. And then I think there is also a big kind of like political will gap to actually get any of these things implemented. My sense is like the understanding of AI is like very low still among— it's much better than in the past, but the number of people in Washington who really like know what the meter time horizon chart is seems actually still pretty small from what I can gauge. And that seems like a kind of very basic level of understanding of the situation. So I think just like more, yeah, training, like getting people to understand the issues, care more about them. I think also just like more popular support helps as well because politicians won't do things unless there's some type of electoral incentive ultimately, or they will sometimes, but it's much more touch and go. And yeah, there's still very little of that. There's a lot of just general negative sentiment about AI, but it's being directed into banning data centers, which doesn't really seem to help because those companies will just build those data centers somewhere else, probably potentially the UAE, which seems significantly worse though. If we're lucky, they'll build them in Australia or the UK or somewhere like that.

That, uh, classic cradle of American values, the UAE, which has also proven to be quite a dangerous place to have a data center as of late.

Uh, yeah, I don't know how those plans are going now, but—

I think some of them are definitely, uh, have to be at least be rethought on a couple levels, I'm sure. Let's do one more beat on pandemics too.

I think Maybe.

And to the degree you can focus this on the AI angle on pandemics, I think the audience is, if there's one, it's a pretty diverse, actually, I've learned over time. But if there's one thing that unites us, it is a general obsession with AI. So I think obviously some of the things that people can do on pandemics are not going to be AI related, but increasingly it seems like some And I'm interested in your thoughts on pandemic prevention, but especially as it intersects with AI.

Yeah, I think pandemics is a really exciting area because there's just so much to do that would really help reduce the risks. And it, like, with all these alignment issues, it's, everything is much more debated. We don't really know what's helping or hurting much more. But with pandemics, people agree there's a lot of things we can do that would very likely help. And then in terms of using AI to help reduce these risks, I think there could be some things in this area. We could be using AI to do better disease surveillance and to monitor all these gene synthesis companies and check that they're not being used to manufacture viruses, like super deadly viruses. But the main thing that comes to mind to me is just a lot of the things we need to do in pandemics is just It's like these very entrepreneurial engineering build type projects. And I would say if you're an AI, if you're like, you're working on AI now and you have this kind of, I just know how to build stuff with AI. I know how to just make systems with AI, get stuff done. Then you could just apply that to these biorisk projects and take that, just take that skill, that kind of like entrepreneurial doing stuff skillset and apply it to this cause. And like a couple of the projects that need to happen there are like wastewater monitoring so that we can pick up, detect new pandemics really early. The cool thing about this is you can just synthesize all the DNA and then you can look for anything that's growing exponentially. And that's a signature of pandemic. And that lets you pick up pandemics that are even completely unobserved in the past. You can potentially detect them earlier.

Early.

There's, yeah, red teaming all the gene synthesis companies to mean that it's not easy to use them to start developing dangerous segments of virus DNA. That's like preventing it from happening at all. The wastewater stuff is making sure we know quickly when it is actually starting. And then the third is just lots of measures to prevent spreading once we've noticed something. And again, we could just design much better PPE that's cheaper, more effective, more comfortable. And then we could have huge stockpiles of it just ready to go for when the next pandemic happens. And then we could just give them to all the essential workers who during COVID did keep working, but that's because COVID was like not actually very dangerous for young people. But if you had a pandemic that killed 10% of people or even 80% of people who got it, I don't think you'd see a lot of people packing shelves at supermarkets. So we would need PPE to keep society functioning. And that is the kind of thing that costs billions of dollars, but that's not expensive on the scale of the risks here and budgets and so on. And then, yeah, like stuff like air filtration, like in homes, if you're, if you have positive air pressure, you filter the air that comes in, have positive air pressure, then it's like basically preventing air from flowing into your house and it can keep it like pretty clean. Like UV lights can sterilize the air. And we could just have those installed as standard. These would also make the common cold way less common, which would just be like a really nice side effect, even without all the pandemic, uh, prevent, help on pandemics as well. Yeah. And then the final pillar would be really rapid vaccine development. And yeah, like these are all very kind of like the type of person who could run a startup is the type of person who could do one of these projects. And yeah, I think it's maybe been a little bit neglected compared to AI at the minute, like, especially these projects really aimed at cutting off the worst-case scenarios. There's just not that many people working on them. We're talking about a couple of key organizations really focused fully on them.

You mentioned people coming from entrepreneurial roles, and that reminds me of a conversation I recently had where Somebody said they're trying to help staff certain roles in the government and we're looking for people with an entrepreneurial background. And I was kind of like, hmm, that's an interesting juxtaposition. I'm not sure how many entrepreneurs are gonna want to go join the government actually, because it, I think that personality type is often one that started a company in the first place because they didn't wanna work at a big company. And in some ways the government is The even bigger, more bureaucratic, more frustrating environment to work in for people like that. Do you have a taxonomy of backgrounds or personality types that you think map particularly well onto the problems that are most important or the different organization types? I'm interested also who should go into government. Maybe it should be entrepreneurs, but if it's not gonna be entrepreneurs, Who should we be looking for to take on these governmental roles?

Yeah, I've also heard that entrepreneurial thing. I think just basically the entrepreneurial skill set, it's like making difficult things happen and like building coalitions of people and like setting up systems. And these are just super valuable skills everywhere. And yeah, I think the key filter would just be like, could you actually hack this? Or would you find it too frustrating? But if in doubt, I would say to try it. You might, I think people have a lot of misconceptions about, they see careers in a very picture book vision of it. Maybe they picture the DMV or something where they think of like the government, like, but there's so many different organizations and teams, and maybe you find some that are, you are really aligned with, and it's a very different experience working with them. But yeah, also in government, there's a lot of need for more just analysis, technical expert type roles, which again can fit people who are a bit more nerdy or researchery type people who, but want to do something more applied where they can actually see the effects of their actions compared to academia. That can also be a really good profile for that type of person. And then yeah, the kind of just operational getting stuff done skills as well. Really, I think, as far as I can tell, really needed there as well. And also in all the other orgs as well, also need that skillset.

How do you think about joining versus starting? Obviously this is somewhat of a timeline question, but I've heard both points of view recently, both being like, we still have a lot of gaps where we need somebody to step up and start a new organization. And so if that could be you, you should do it. And then on the other hand, I've also heard there's a lot of orgs that are trying to hire and there's not enough talent to go around. I would say my subjective sense right now of the AI nonprofit space is, and obviously there will be exceptions to this, I don't mean to paint with too broad of a brush, and you can also feel free to disagree, but it seems to me like there's a lot of funding available for organizations that can demonstrate that they're executing reasonably well and have any capacity to scale. So what are your thoughts or how would you talk people through the starting versus joining decision point?

Yeah, and there definitely is funding available. It's a, it's a difficult thing to think about in a way because more funding is still helpful because there's always like more you can do. But I think it is true that if you show up with a really good organization, you can raise a lot quite quickly. And in the past, coefficient giving was responsible for a lot of funding, but that's really actually broadened now. It's in the last couple of years, there's a lot of new funders entering the space. And there might be a lot more funders if some of the people who've made money from Anthropic start donating, which they have pledged. The founders of Anthropic have all pledged 80% of their equity, I think it is, to donate eventually.

Equity is doing quite well at the moment. Yeah.

Yeah. That is, that we're probably just talking about, we're probably talking about tens of billions of that alone, though obviously it won't all come immediately because they wouldn't want to sell their whole stake immediately. So there's a lot of funding. Again, I kind of think both of these things are true. There is scope to start new organizations. There's a lot of important gaps. But there are also orgs doing great work and you could just join them and try and multiply them. And I think entrepreneurial people are often a little bit biased against the second option because if you start an organization yourself, it feels much more satisfying because you get to be like, well, there was nothing and now I made this thing. Whereas just making another organization 5% more effective is harder to feel tangibly. But if that other organization is having a huge impact, then it can often be better to just throw yourself behind the thing that is working and double down on it. I would consider both paths. Founding is obviously difficult and most, it's maybe not the thing for most people. So if you are that type of person who do it, then I think that's, it should be taken seriously. Yeah. There's an interesting thing about founding, like how possible is top-down founding where you're just like, Well, this idea I think is really important. I'm just going to go and build that thing. It seems to not work that well in the for-profit sector. And I remember Y Combinator really encouraging us to, you know, like the best companies just develop out of problems that you were solving for yourself or almost like side projects that you didn't think were a company and then you realize they actually were. I think that maybe applies more to for-profits because the space is so much more efficient and competitive. It's harder to spot an idea. Whereas I think in the nonprofit world, it's more possible to just be like, okay, there's this funder who wants to fund this thing. There's this important gap. I'm just going to fill this gap. And there's definitely been cases of people doing that. And if you're really motivated by the impact, you can actually put that into action. There's a couple of different organizations helping people who want to do this kind of thing now. There's one called 6 Impact, which is an accelerator in this space. Maybe also worth mentioning for mid-career people, Successive, which just helps to transition mid-career people into careers working on AI risk in some way. And also there's a lot of fellowships, like Horizon Fellowship transitions people into policy careers in 1 or 2 years. So if you're like a technical person now, but you're interested in policy, you could apply there and they could, they can help you switch really quickly.

Yeah. Cool. Shout out also to Halcyon as another network that I've seen with a very focused and not super scaled, but high success rate approach. They've also been pretty good, especially I think for mid-career people making moves into the AI space. Do you have a request for startups list? What, for the founders among us, are there kind of top priority gaps that you see?

Yeah, there's, so Catalyze Impact, who I just mentioned, they have a meta list of all the other lists. So you could just go and check that out and then browse all of the like 20 or so lists of project ideas they have there. And there's, yeah, there's a lot of ideas at the minute.

Yeah.

I like none in particular in terms of what most stands out.

Yeah.

One, I think you asked about If you have an AI application skill set, what's something valuable you could work on with that? And one broad area of things is if you think we're going to be facing these big risks soon that are going to be caused by much more powerful AI, then one strategy you want to, might want to take is think about how you can essentially use AI to address these problems because then your tools are getting better as The problems are getting more severe. They're growing in tandem. And one category of things there is using AI to improve epistemics, improve our understanding of what's happening, improve decision-making. And there could be a lot of scope to build tools here. Could we have automated fact-checking? Could we have automated looking at people's prediction track records, like different politicians and pundits, and saying, well, you said all these things in the past, but like, this fraction were actually correct. This could improve our discourse. And then another thing would be like, government, like people in government are going to be relying more and more to make, to use AI to make decisions rapidly. They're going to have AI advice, especially if the world starts speeding up the pace of change. It might be hard to stay on top of things just with our normal old human brains. And It would be nice if those people had AI advisors that weren't necessarily just developed by the companies that they're supposed to be monitoring, right? Ideally, there would be more impartial AI advice that we could check is designed from the ground up to do exactly what this government wants it to do. So there's this kind of AI chief of staff or AI decision advisor idea application, which, yeah, is the kind of thing that the market might not build, because there's not going to be that. You might be able to build a profit model around it, but the key application would be more of a nonprofit application to these key government decision makers. So Four Thoughts Research have written about this idea. So if you want to see more of the project ideas in this space, they have a couple of articles about it.

As one kind of calibration point too, for just how helpful AI can be on some of these things. I'll say, we'll put that in the show notes. And then the show notes these days are now fully AI generated, generally not even really reviewed by me, but they do tend to come out with a couple dozen links. So all of the organizations that we've mentioned, you're like the, the list of lists that you mentioned, I think that Claude will be able to find that with some help from the Brave Search API, I should say. And that all that stuff will be in there. And if you go check that out, know that I probably didn't look at it between now and the time that it hits the website, which means that there could be some errors in there. I wouldn't say you should trust it to be fully comprehensive or fully error-free, but that kind of scale of project, I'm confident enough now just letting the AI run with, and it's clear that it's doing a pretty good job and adding value, even to the point where I'm usually a bit precious about publishing AI stuff without a review. But at the helpful links level, it's reliable to the point where I let it fly. Do you think much— we've focused obviously very squarely on these big picture risks from AI and what to do about it. I know that you also have over time put a lot of thought into other ways to help people and the world, and there's been 80,000 Hours career guides for how to help the global poor and improve public health and improve animal welfare and all those sorts of things. If somebody is really interested in those topics, but also is developing an AI skillset, how much room do you think there is to go apply AI for those organizations that are executing already in those different domains? My guess would be that they, probably have not caught up to the AI frontier or anywhere close in a lot of cases, although I could be wrong. What would you say is the state of just mundane opportunity to apply AI for mundane, but ideally well-chosen causes and make a difference that way?

Yeah, I do think the skill set of managing teams of AI agents to do real work is, that's going to be one of the most valuable skill sets of the next few years. And you can apply that skill set to any problem. And I think you're right that most industries are still, and fields are still not really applying AI anywhere near where they could be. And you could work on global poverty, but just be an expert in doing it as effectively as possible using the best tools. And those tools will also, they'll continue to get better. So your leverage will increase as AI improves. Yeah, in general, I think there is a bit of a disconnect here because the people who want to work on these other issues tend to kind of like, there's a reason why they don't want to bet on AI being the key thing. Often they want to do something that's more grounded or evidence-backed or common sense. And then that same thing with like, Also prevent them from wanting to make a big bet on AI being a steal. Because if you did believe that, then it's like, it's harder to see the case for these issues in some ways. If we avoid concentration of power and loss of control of autonomous AI and a massive bioattack and all of these things, then we're probably going to end up in a world that is far wealthier than today because we'll have this massive population of digital workers. who can hopefully give us super good advice about all these problems. And I mean, many economic models show that you would, if you really work that through, you could easily have GDP that's 100, even 1,000 times more than what it is today, eventually, potentially even higher. And yeah, part of the way that happens is just that loads of goods, like services, become basically free because you just get The models become rapidly commoditized and just, it's like very small cost of compute to get basically world-class services. This is like eventually where it goes on any topic. And then you have robots that produce goods far more cheaply than now because a robot eventually costs like $1 an hour, whereas human workers are like $10 or $20 an hour. And that just makes it a lot easier to reduce poverty because The world is much richer, everything is much cheaper, and that will make a huge difference to reducing it by itself. The main kind of lever there would be just making sure that the global poor don't get cut out of the kind of AI windfall, which kind of gets us back to the concentration of power stuff. Because I do think there is a good chance that America really draws ahead of other countries. And it's not particularly guaranteed that they share the benefits with the rest of the world, but making sure that everyone would get some of the spillovers because we would get the cheaper services and things like that. But yeah, sometimes I talk about, this is another policy idea, it's like eventually we want to work towards there being a grand bargain on AI where all the major countries agree not to race and struggle over it in exchange for sharing of the benefits among everyone. And if we could come to that type of agreement, it could reduce the risks a lot and also Be a, not avoid concentration of power and be like a fairer outcome for people. And I think working on something like that would be more what comes to mind for me when I think about what could be a really high leverage neglected thing to help with global poverty these days. But it comes back to a more AI-focused approach rather than just going and using AI to distribute malaria nets another like 10% more efficiently. Though, I mean, that's still a good thing to do, but it's not what comes to mind to me as like the highest leverage thing you could do if you're really taking AGI seriously.

Yeah, got it. Speaking of taking things seriously, we teased a little earlier, uh, a couple other more speculative or esoteric topics that could potentially be really important. One that's on my mind, and I want to hear what's top of mind for you, is possibility of AI consciousness, subjective experience, moral standing. I'm still very uncertain as to whether that is something that I need to worry about, but the evidence does seem to be mounting quickly that I should at least be taking it somewhat seriously. So that's kind of on my radar now. What else is on your radar, even if it hasn't quite cracked the, the inner circle of the most important things yet?

Well, or that, like, the way I like to think about this is, as other, if you're trying to have the biggest impact, you want to work on neglected problems. And so this means that there's always this dynamic where you start working on something and then a bunch of people join you and then it becomes less neglected and then you kind of want to move on to the next even weirder thing. And it kind of means that like people always think you're crazy at every point, but like. You always want to be one step beyond what— not always, but you want to be seriously thinking about being one step beyond what's already accepted, because otherwise it's not going to be as neglected. And so now there's this question of, as alignment and catastrophic biorisks, and even like you're saying at this conference now, even people are taking concentration of power, which is quite a big success because only a year or two ago, no one was really talking about this. And just a couple of groups like Forethought Research, who I mentioned earlier, have got this as part of the discourse in a couple of years, which is impressive. But then, yeah, as those efforts are successful, what's the next thing? And yeah, I do think what to do about digital minds is potentially a really big one. I think eventually it will become a huge topic because we'll be just talking to AIs all the time who seem basically exactly the same as humans. And at that point, many people will I think they have rights as well. So it's gonna become like a huge debate. The question now is, can any groundwork be laid so that when that debate happens, it's in a better position to come to a reasonable answer? And yeah, I don't think we're ever gonna, we probably just need to plan on never solving the problems of philosophy of mind, 'cause they've already not been solved for thousands of years, and it's unlikely that we do it in the next couple of years. But we then need to think, given that we don't know what policies would make sense anyway, like on a balance of the different views, basically the different views about what to do about this. But yeah, a few others. Another one that I was tweeting about recently is space governance. And yeah, these basically all sound— I knew when I was writing these into the book and my editor was like, these ones I think you should cut because this is like getting too out there. And I was like, okay. That probably means like these are the ones I should keep, right? Because if you already knew all this, then it would be too obvious. But yeah, if AI accelerates technological progress, it could become technically possible to settle, spread out into space much sooner than it seems from a kind of common sense point of view. And in particular, we're not talking about humans going out. We're talking about sending AI, like small AI probes out that can then replicate themselves, build solar panels when they arrive somewhere. But like the way this would likely work by default now is a land grab type situation where just whoever launches their stuff the first claims all the stars. And that's where like most of all the energy and matter is like 99.9 with, I think, 29s after it is of all matter and energy is like accessible matter and energy is not on the Earth. So from a kind of really big picture perspective, like ultimately what matters is what happens with space. Yeah, there's basically like no one talking about this or thinking about, is there anything in current laws that could set stronger precedents here about how this would happen and making sure it's not just a land grab and maybe thinking about even just modeling the dynamics of this. It seems like there might be a kind of whoever gets there first can't be dislodged type effect because If you arrive first, then you've already built an industrial base out in space, and then it's hard for people following you to then displace you later. We don't actually know if that's how it would work, but it seems like at least good to understand if there's a massive first mover advantage or not. So this would be a kind of an area where people could do more research. I think it could be quite cool to have an institute for space governance. This is another startup idea. There's a specialist think tank that's just like, we've got all the experts on this topic. We're thinking about it. And then the idea would be like, when people are like, oh, this might actually happen soon, they would turn to that group as the experts. Yeah. I don't know if you've had an episode about gradual disempowerment, but that's another one on this list that is quite likely to me as a kind of, even if we had aligned AI and avoided concentration of power, things could just be in a way that humans get competed and the economy becomes pretty unfriendly towards us. That seems like Quite a likely thing to happen to me, and no one really seems to have a proposal of how to prevent that. Like, I guess basically the plan right now is like, use the aligned AI to advise us on how to avoid that outcome, which might work. But, and this is like very few people thinking about this, it gets into your point earlier about visions for what a good outcome would look like. There's like very little material on that really. People just, they're just like, well, the Culture Series. That's pretty much what we got, which is like leaving a lot to be desired. What do you—

this is a bit of a— I don't want to shortcut your list if there's more you want to add, but one idea that I've been coming back to over and over again is writing utopian fiction to just try to make the future, the goal state, a little more concrete. I think we've seen a little bit of evidence this last couple of weeks that could even have a positive effect as it flows through the training data and gives AIs a conception of themselves based on our imagination of what they maybe should be like in a mature state. Any feedback on that idea writ large? Or if somebody was actually going to go try to seek to steer the future through utopian fiction, any advice you would give them?

I've heard people speculate about ideas like, can you flood the internet with positive training data and therefore influence the values of the AI? I don't I presume there's a lot of reasons why that wouldn't work. With utopian fiction in particular, I do think one thing I would say is the track record is very bad. Like most people who've tried to write a utopia, when you look back on it, it just seems like a dystopia to us today. And like, it's just very hard to not be like very trapped in the values of your times. So I'm quite into the idea of what we should be shooting for is this idea from Will MacAskill, which he calls a viatopia. which is not specifying a particular end state, but it's trying to just get civilization in a better position to navigate to the best end state later. So for example, one thing that would help is not having an irreversible existential risk, because then you've cut down a lot of options. But also, yeah, not, it seems not having an authoritarian government again, would help to preserve debate and mean you have more options to navigate the future. Having more information about what's happening. These are the kind of things that seem robustly good for navigating things that a lot of people could agree on, even if we don't know the particular end state. The analogy Will uses here is, imagine you're lost in the wilderness and you don't know which way to go to escape. But you might still be able to say, well, I need to get water. I need to get to higher ground so I can look around. And these would still be robustly positive goals to work towards. And then like, hopefully from there later, you'd have a chance to figure out which direction you should actually walk in.

Yeah, I like that. Via Utopia. You've been generous with your time. I know it's getting late for you across the pond. Maybe just in closing, any other organizations that you would want to shout out that we haven't mentioned that can magically show up in the show notes? Any other topics or tidbits that we didn't touch on that you would want to make sure you leave people with? And then maybe, you know, just tease a little bit, like, what else is in the book that we didn't talk about? So people are still not left with the impression that this conversation substitutes fully for the book and are still motivated to go pick it up.

Yeah, I mean, I mentioned a lot of organizations except for 80,000 Hours itself. So maybe just briefly mentioning, yeah, like it's kind of amazing in a way that we can actually, there are actually jobs that help with maybe the most important, one of the most important and maybe the most important moments and transitions in history. And like, you can like actually switch into these and have a really big impact and do something incredibly interesting as well with your time as well. And 80,000 Hours, like we have a job board that lists around 1,000 open jobs across these issues, as well as like lists of funding opportunities and fellowships and that can help you transfer into them. And then we have free one-on-one advice. So you can talk to someone and they can introduce you to people in the issues that you're interested in and help you transition into the field. Yeah. If you, so if you're interested in switching, those are probably the two most useful things to know about. The book itself is quite a, it really aims to cover like all the big questions of careers advice, starting from what even like actually makes for a satisfying job in the first place. Like what should you look for in an enjoyable job and a meaningful job? And why like just simple answers like follow your passion don't make sense. And then it goes through, okay, which Skills are most valuable given AI, and it has like a list of concrete skills you can learn. It has advice on the fastest ways to learn those skills. It has like how to choose a problem to work on and which problems are most pressing. And then also just a lot of practical advice, like, okay, if you have a bunch of options, how do you decide between them? What are the most common decision-making mistakes? How do you get a job? Yeah, what's the most efficient way to do job hunting? All of these questions. So I've really tried to make it like the most research-backed ever career guide that covers all of the most important questions, including how to wrestle with AI woven through it. And yeah, I think the thing I would say is just, yeah, when you look back on this time, maybe like one of the, some of the biggest challenges in history we're facing, like, how do you want to look back on it? Do you want to be like, well, I was like— Trying to escape the permanent underclass, so I like made a bunch of money, or I was like accelerating AI, or do you want to think, no, I like did the best I could to help this go well. And yeah, we don't know how it's going to turn out, but I did my part. Yeah. Yeah. You only have one career, so it's really worth thinking about what you can do with it that's the best.

Hopefully one day we'll all have grandkids and when they ask, what did you do in the great AI transition, grandpa? It'll be satisfying to have at least a decent answer. Yeah, recommend checking out the book, 80,000 Hours. It comes out May 26th. You can preorder it before that as well. I am also an alum, so to speak, of the free career advice offering that 80,000 Hours makes available. I did that really just as I was getting serious about AI and got into it on a full-time— now I'm this full-time student occupation. But I got a number of helpful introductions out of that meeting, so I can definitely endorse that service.

Cool.

Become a bigger and bigger deal, and we've got a lot of need for a lot of different profiles, a lot of different kinds of people to help. So with that, Ben Todd from 80,000 Hours. Thank you for being part of the cognitive revolution.

Thank you so much.

If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts which is now part of a16z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast.ing. And thank you to everyone who listens for being part of the cognitive revolution.