Surveying 2,700+ AI Researchers on the Industry's Future with Katja Grace of AI Impacts

Forecasting the future is like quite hard. We are having to make decisions about these things. The best guesses that we can get are valuable. And so it's important to hear what AI researchers think about this amidst quite a lot of uncertainty here. The chance of things just staying pretty similar and nothing coming of this to like drastically change people's lives seems quite low. This seems like a pretty important thing to be keeping an eye on and trying to have accurate opinions about what the public thinks about these things will affect what policies happen. And they think that could change the rest of the future forever.

Hello and welcome back to Turpentine AI. Before today's show, another quick reminder to take a second to sign up for our new Cognitive Revolution feed. Our latest episode on that feed is an outstanding conversation with the great Linus Lee of Notion AI. So find that and be sure not to miss any of the new episodes that will be exclusive to that feed by visiting our website, cognitiverevolution.ai, to subscribe. Today, I'm excited to share my conversation with Katja Grace, founder of AI Impacts, who recently published what I believe to be the most comprehensive survey of elite machine learning researchers' expectations that has ever been conducted. As we all try to make sense of this AI moment and understand where things are going, Katya's survey provides an incomparable resource. With over 2,700 respondents, all of whom have published in one of the top 6 AI conferences in just the last couple of years, it offers a nuanced and detailed picture of what the people closest to the technology really believe. And the results are truly striking. Arguably the biggest takeaway is that there is no consensus expectation within the field. Most individuals express high uncertainty on the most important high-level questions, and only relatively small minorities confidently predict a very positive or very negative future. Researchers definitely do not dismiss existential risk, with median estimates of a 5 to 10% chance that AI could lead to human extinction. And meanwhile, timelines for key capabilities have shortened since the survey was last conducted just a year ago. There is strong agreement that AI safety should be prioritised more than it is today, perhaps in part because no major mechanistic interpretability breakthroughs are expected in the near term. Overall, this conversation offers a uniquely rigorous field-wide take on critical questions that are all too often poorly asked and sloppily debated. After first digging into the methodology to better understand how Katja and team went about defining their terms, framing their questions and summarizing the responses into aggregate views, we go on to consider implications for policymakers, the public, and the field itself. While there's, of course, nothing to say that ML researchers will ultimately be correct about AI outcomes, I would call this a must-listen for anyone who wants to ground their own personal AI world models in the considered opinions of the people who are collectively inventing the technology. As always, if you find value in the show, please share it with others who you think would appreciate it. I genuinely believe that both the doomers and the EACs should update their PDooms toward these community estimates. A 5 to 10% chance of extinction is obviously a very critical issue, but the shape of the other 90% of outcomes is also well worth worrying about. Of course, I'm always eager to hear your feedback, so please feel free to reach out via your favorite social media platform. And now, please enjoy this exploration of the expectations and the substantial uncertainties of the AI research community with Katya Grace of AI Impacts.

Katya Grace, founder of AI Impacts, welcome to The Cognitive Revolution.

Thank you. Great to be here.

I'm excited for this conversation. You have just recently completed one of the most comprehensive surveys of the worldviews and expectations of elite machine learning researchers. And as we try to grapple with this AI moment and try to make sense of what's happening and where things are going, I think it's a really incomparable resource. So I'm excited to just spend the full hour kind of digging into that and understanding both how you went about collecting this opinion and also what the aggregate opinions are. For starters, you want to just introduce maybe yourself and AI Impacts a little bit and give a little bit of the Inspiration or motivation behind the survey?

I am broadly interested in trying to make the world better. And for the last 10 years or so, I've been involved in trying to figure out what's up with the future of AI, since it seems like potentially a pretty important thing to try and cause to go well. Yeah, I've been variously skeptical about whether it's likely to cause human extinction or not, but Yeah, interested in figuring out these kind of things. So AI Impact is an effort to try and answer questions like this and just various high-level questions about what will happen with the future of AI that are relevant to decision-making for lots of people. Is AI likely to cause human extinction? When will the big AI thing be? What kind of thing will it be? Are there other kinds of changes to society that we should expect and do something about? AI Impacts is often answering sub-questions or sub-sub-questions that are relatively in the weeds as input to those things. For instance, case studies about other technologies and whether they've ever been slowed down for sort of ethical or risk reasons and what that looks like for a recent example. So we do a lot of stuff that is not directly about AI, but is intended to be part of a bigger network of questions that inform other questions.

Cool. For the purpose of a survey like this, is there any reference point that you can look to where folks have found it necessary even to survey experts to understand the direction of their field? I would think maybe something like biotech might have something like this, or you could imagine, I just recently read a story about the original nuclear test at Los Alamos and how they had a little betting pool on it, but there was no big survey. So are there any kind of other touchstones that you look to?

I think that the thing that we most look to is just like other surveys in AI over the years where prior to the 3 that I've been involved with, there were older, smaller ones that often have a different definition of human-level-ish AI each time or something. So it's like a bit hard to compare the results, but I guess we previously looked for all the things like that we could. find and go back to, like, I think there's one in the '70s. Yeah. Interestingly compared to over the years, I guess we're not looking to them that much methodologically because I think they're maybe less careful and smaller. In terms of things in other fields, I haven't looked in detail at them. My understanding is that there's some sort of thing in climate perhaps.

Yeah, that's definitely a good reference. There's certainly a lot of guessing as to the future state of the climate. So that makes a lot of sense. Okay, so the way I plan to proceed here is just first of all talking about who are you surveying? Who are these people? Then what was the survey that they took? A little bit on just the structure, the experience, some of the key definitions and framings that you used. And then I think the bulk of the discussion will be like, what are the takeaways? What do people actually believe in the field? But for starters, who were these people? How many are there and how did you get them to do it?

We tried to write to everyone who published in 6 top venues. So 5 conferences and 1 journal in 2022. So as in, in 2023, we wrote to the people who had published in 2022 and we offered them $50 each. We had a little trial at the start where we offered nothing to some of them and $50 each to another small group. And it looked like the $50 each was doing a lot better. So then we offered it to everyone. It was a decent task to dig up their addresses. Like we got their names from all of these papers. And many of the papers have emails for at least one of the authors, maybe several of the authors. We dug up the other ones from other papers or the internet somehow. And I think, I think we found them for a large fraction, like more than 90%. I forget the exact number. So I think this is like in the range of 20,000-ish people. And we got about like 2,700 to respond.

These are people that published in 6 top conferences. Is there a natural reason that you chose the 6 conferences? Like, why not 4 or 10?

So previously, the last 2 surveys we've done, we just did NeurIPS and ICML. And we, we wanted to expand it partly to a wider range of topics. So I think like AI people who are not necessarily all doing machine learning and also just to like cover more of the top So I think it ended up being 6 because as far as we could tell talking to people around, that was a sort of natural set of them to do. But yeah, I don't have a clear explicit description of what makes them the ones.

Is there a time horizon also on how recently they published in these conferences?

It was everyone in 2022, like everyone who published in 2022. Often people published in multiple of them.

Quite recently.

Yeah.

Okay, cool. So we got 6 conferences, universe of 20-ish thousand people, something like a 15% respondent rate driven in part by a $50 gift card, 2,700+ total respondents. First of all, interesting. And there's of course a whole paper here. You guys have a lot of visualizations of data in the paper. So Definitely encourage folks to go check out the graphs if they want to do a real deep dive, although we'll cover the bulk, I hope, of the headline results. But it's kind of a branching structure where not everybody's getting every question, but you have some anchor questions that everybody's getting, and then others, there's kind of a bit of a, not choose your own adventure, I guess, but kind of a randomly chosen adventure for them. And it takes 15 minutes, which was kind of surprisingly fast. I would've expected it to go slower.

Yeah, I guess we haven't actually checked for this one how long it took. That was estimated based on the previous one.

Advertised as 15 minutes. Was there any free response or it's all structured? You must either select a thing or enter a number, right? There weren't any paragraph-style responses that I could detect?

There were some that weren't paragraph length. There were just some brief ones. For instance, there was one that was, what's an occupation that you think will be automated, fully automatable, pretty late, something like that. So that's a couple of words. But also for, I think, all of the sets of questions, you could randomly, I think maybe 10% of people after each one got, there were 2 different questions you could get about just like, what were you thinking when you answered that? Like, how did you interpret this question? Which are not part of the main results, but for us to be able to go back and if we're confused about a thing. check if everyone's misunderstanding it or something like that, which we have not gotten into yet, but we might do more of. Oh yeah, actually, sorry. Yeah, there are various open-ended ones. There's another one that was like, what confusions do you think people have about AI risk? Something like that. Okay.

Maybe we can talk about some of the open-ended stuff later. And I'd be interested to hear how you may think about evolving the survey in the future, particularly as you can potentially bring language model analysis to open-ended responses. I'd expect Surveys in general are going to be an area that will change quite a bit. Certainly I see that in even just very sort of application-centric customer feedback experiences. I'm like, hey, let's just let them talk. We don't need to structure everything in a grid or rubric quite as much anymore. We can, if we're looking for insights to some degree, you can just let people sound off. But for these purposes, you're really trying to get to quantitative estimates and As I read through the paper, there were 2 super high-level framing structures that stood out to me as, first of all, just very thoughtfully done and worth understanding for our audience as well. One is that you have 2 different definitions, essentially, I would say, of AGI, right? There's 2 different ways of thinking about powerful AI that we definitely don't have now, but we could have in the future. And then there's also 2 different ways of asking people When they expect certain things to happen. Maybe you can unpack, first of all, the 2 different definitions of AGI, if that is a reasonable way to think about it.

Yeah, I think so. I guess I think of them as fairly closely related, but yeah, different people think of that differently. So there's high-level machine intelligence, HLMI, which is like when unaided machines, so it doesn't even have to be one machine, just like any collection of AI can accomplish every task better and more cheaply than human workers. That is like for, for any particular task, there, there are machines that can do it. It doesn't even have to be the same collection of machines that does stuff. And we're ignoring aspects of the task to which being a human is intrinsically advantageous, e.g., being accepted as a jury member. We're asking them to think about the feasibility of it, not whether it's adopted. So this is roughly like, when can AI do all tasks? better than humans. Then the other definition is full automation of labor, which is sort of similar except that it's asking about occupations instead of tasks. So occupation becomes fully automatable when unaided machines can accomplish it better and more cheaply than human workers. Ignore aspects of the occupations for which being a human is intrinsically advantageous is the same. And again, asking them to think about feasibility, not adoption. So I think there's some chance that keep all of these things in mind and tend to think of the automation of occupations as actually about adoption and the high-level AI one about it being feasible. So when all occupations are fully automatable, then we have full automation of labor. So I would think of an occupation as probably like a big complex task, or if not composed of many smaller tasks. So I would think that if every task is Automatable, then that implies that occupations are automatable as a subset of tasks. But in fact, the answers that we see put the date for HLMI much earlier than the date for full automation of labor. So I think there's a question there about what people are thinking. And to be clear, no one is answering both of these questions. People are randomized to receive one or the other. So you can't say for a particular person, like, Why are you so inconsistent? But given that there's such a big gap, presumably any given person is likely to be inconsistent if you ask them the two questions because it's not just like random or something. There's one other small difference between them which we keep in in order to keep everything consistent from year to year, but I think it's a bit annoying actually. Is for the HLMI one and not the FAOL one, we said to assume that like scientific progress sort of continues as normal, like there's not some giant impediment to that. There's a Roughly the 2 definitions. I guess actually, sorry, for the full automation of labor question though, the question is different in that first we describe what full automation of an occupation is. And then we go through a series of questions that ask people about different specific occupations. Like we give them 4 occupations and ask when they think those might be fully automatable and then ask them to think of a particularly late occupation and when they think that will be automatable. And then ask them when they think Everything will be automatable. So I think this leading them through the process in steps could also lead to a very different answer. And maybe that's what we're seeing.

Hey, we'll continue our interview in a moment after a word from our sponsors. That's interesting. I wonder if it would be useful to just open up a, like, this is the experience of the survey. Is there any place where people could go and actually just take it themselves or like experience what the, and they may not qualify necessarily for your population, but For folks like me, for journalists who want to understand what actually happened here, is there a version where people can go see the actual step-by-step?

I think there isn't actually. Sorry about that. So you can go and look at all of the questions, but they're all just like in a PDF. I think a difficult thing with just seeing what it's like is that there is so much randomization in it that you're going to get one particular path through it, which, you know, is still probably informative, but you would miss most of the questions.

Okay. So we've got high-level machine intelligence, which is when unaided machines can do all tasks. And then we've got what on the surface seems like a pretty similar concept of full automation of labor. But those do lead people to quite different numbers. Let's talk about how the numbers actually get collected. Again, there's 2 frames here. I think the difference is also pretty interesting.

Yeah. If you want to ask a person about when a thing is going to happen and you want them to give you a distribution over time, it's sort of going to look like, Well, there's a very low probability of it happening tomorrow and a higher probability that it will happen by the next day and so on. So we're trying to somehow get from them a distribution of probabilities increasing over time. And so we wanted to do this by getting 3 different year probability pairs and then like drawing a line through them to make a probability distribution that we're sort of guessing is, you know, roughly close to theirs. And so the 2 natural ways to do this are to give them years and ask them for the probability that they think that it will happen by that year, or to give them probabilities and ask them in what year that probability will be met. So in 2016, we split people in half and gave half of them one set of questions and half the other. And it was like they very consistently give different answers to these 2 things. We also tried it on people from Mechanical Turk, random survey takers for money, and they had pretty similar patterns where, in particular, if you give years and ask for the probabilities in those years, you get a later distribution than if you say, and in what year will there be a 90% probability of this? So, given that they're pretty consistently different, we keep on asking them both ways each time because we don't want to just pick one of them and go with it because we know that it's biased. So we want to do something in between the two of them. So we ask half each way and then turn them all into probability distributions and then put the probability distributions back together again and average them.

So let me make sure I get this working backward. So the idea is at the end, you want to be able to say, we have a curve that represents the elite ML researcher community's aggregated sense of how likely these different technology phenomena are to happen over time. And ideally, that'd be a nice smooth curve so we can make sense of it and look up any year and whatever. Now, to get there, we have to get everybody's individual take. But to ask people to draw all these curves is pretty tedious, requires an advanced interface. You can do that sort of thing to a degree on Metaculus, but it's not super lightweight. So instead you say, well, okay, if we have 3 percentage-year pairs, then that's enough. We can fit a gamma distribution. You can maybe tell me why gamma distribution versus a different distribution, but we will fit a formula to that and then we can do that for every user. And then we can basically take the average of all of those fitted distributions and that will become the final aggregate thing. And then for one more layer of complexity, the pairs themselves could be generated in different ways. You could say, in what year will there be a 50% chance? Or basically that's like, what's the over-under? If you're a gambler, it's like, you give me the year at which you would accept even odds.

Exactly.

And the flip side of that would be, I give you a year and you tell me a percentage.

That's right. Yeah.

And which direction was the bias? I didn't Catch which one gets the earlier or the later?

If I ask what odds do you give in 20 years, that will get you a later curve. So things happening later. So I guess my guess is that it's something like for pretty wild things, you're inclined to put low probabilities perhaps. So if you're given different years, you can just keep on putting lowish probabilities. Whereas if someone says like, when is it 90%, then you still have to give a particular year, once you're accepting that it is a particular year, unless you decide that this thing is just not going to get that likely, then it's not particularly tempting to put it extremely far out. I don't know if that's right at all. That's how I remember which way it goes because I'm somewhat suspecting that's happening. Yeah.

Interesting.

Okay. So going back to the gamma distribution for a second, what can you tell me about that? I'm not very statistically sophisticated. One question that comes to mind is, does it necessarily reach 100% at the end? I don't really know anything about what is implied by this particular choice of distribution.

Yeah, I'm actually also not statistically sophisticated here, which is one reason I have colleagues. But yeah, I'm not sure. I think that it doesn't have to go to 100%.

It's certainly a very thoughtful approach. Reading through the paper, I was like, 2 framings of what it is we're talking about on these extreme superpowered relative to today's AI systems of the future. That, I think, is a healthy sanity check. It is notable that there's quite a difference between those. The flipping of the percentages in the years is also really interesting, and it's definitely just a super thoughtful approach to try to find some way to get a continuous distribution from relatively sparse data across people. So I came away from it feeling like It was better than I would do, that's for sure. And of course, all these sorts of things are going to have their artifacts or their weaknesses or their points of question. Would you give any caveats? If somebody is to say, what's the biggest problem with the way that this data has been gathered and synthesized into this aggregate view? What would jump out to you as the biggest problems with it?

I'm not sure about the biggest problem, but one problem I was just thinking of then is like, I feel pretty good about us having asked things in various different ways because I think in fact we do see substantial framing effects. Or different answers for some reason for what seemed like fairly similar questions. So I think that's pretty good to get. But I think that I wasn't thinking about when we were designing this in 2016, because we've basically had a very similar survey from time to time, is that, you know, if you ask a thing in 4 different ways or something, then it really opens it up to reporting biasedly. Like, you really can choose one of the things and Advertise it. And even if we don't do that, other people can do that. And so I think I'm pretty keen on not doing that, but I do end up having to fight a constant battle to get, like, there are the HLMI answers and the full automation of labor answers. I think it's like very tempting to just write about the HLMI answers, partly because the full automation of labor answers are so late that maybe they seem like more ridiculous or something, especially to people who feel like AI is happening very soon. So it's tempting to, just look at HLMI and be like, oh, and it dropped a huge amount. You know, this is all happening fast. When if you look at all of our answers together, it's like quite a strong counter signal saying that people are expecting this in quite a long time. And so I think, yeah, all of this kind of like different questions about everything makes that quite hard and means that you have to police it yourself perhaps, and might make it harder for other people to trust that you're doing that well, which I, I try to mitigate by making sure that just everything is online somewhere so people can check. In terms of overall worst problems with it, I think just the fact that forecasting the future is quite hard and these people are not forecasting experts even, and they're often answering it in 16 minutes or something. And it's a lot of questions. The questions are about really complicated, like, what will the long-term consequences for the world, a pretty complicated thing, B, of this technology that we haven't seen at all. I don't expect the answers to be very accurate, but we are having to make decisions about these things. And so I think inaccurate answers that are the best guesses that we can get are valuable. And so it's important to hear what AI researchers think about this. I guess I also think that just knowing what different people genuinely think about it is important for, you know, Coordinating. But yeah, that's pretty different from these people are probably right. It probably will be the year that they think.

Yeah. Well, as we get into results, it'll also become very clear that there is a wide range of opinions and certainly in the aggregate, a lot of uncertainty, which means that the accuracy question is probably less central anyway, because it's not like it's a very tight estimate that we're getting, right? It's a pretty broad range of opinion.

I think it's often good to, to, to be clear about what kinds of updates you might make from this. I, I think that you can strongly infer that some important things are not ruled out. Like, you might think these people are experts. They perhaps know that this crazy thought I have is not plausible at all. You know, with some areas that you can get to that sort of conclusion. I think here, the fact that A lot of these AI researchers put some probability on various things is worth paying attention to. Yeah.

Well, let's get into it then. The results are definitely interesting. I thought maybe we could start a little bit with some general higher-level characterizations and then work down to some very particular questions. At a high level, a couple things that jumped out to me about the distributions, and I'll let you expand on this or add your own commentary, is as to what has stood out to you the most. But it seemed that this is pretty consistent, I think, with other things I've seen, that there is kind of a left-heavy nature to the distributions, like the timing of all the tasks, right? There's 39 different tasks. When will an AI system be able to do this task? And we can start to list them in a minute. But across the vast majority of them, I would say, the range of the first half or the second quartile as compared to the third quartile is just much more compressed. So it seems like a lot of people are like, yeah, there's a decent chance that this might happen in the kind of near future. And obviously there's varying definitions of near for different levels of difficulty, but there's like a pretty decent mass in the kind of near to midterm future for a lot of things. And then there's a really long tail that goes like far off into the future. That seems to be a broadly true statement about all the different answers. Is that fair?

Yeah, I think that seems right. I guess I haven't thought a huge amount about this, but I wonder to what extent that's sort of what you get if you're predicting any kind of thing where you sort of think it will happen soon and it hasn't happened yet. So the bit between right now and your kind of 50% it will have happened date is kind of all smooshed up. But then if it hasn't happened by then, there is the rest of eternity to spread it over. Somehow.

Yeah, I think that's right. In everything I've seen like this, this shape is seemingly a norm. Another way to say it is the median is sooner than the mean. And intuitively that makes sense because you have a long tail possibility into the future, and obviously you're not going to be predicting the past. So fundamental asymmetry kind of always leads to that. But basically, I would say for folks who are Trying to develop a mental picture, and you can look in the paper or go look at some of the AI questions on Metaculus. You can see actual curves that very much have this shape. There's a big bump in the relatively short and midterm, and then there's a very long tail that goes on far into the future. Another big observation for me is just that the ranges are super wide. I'd say maybe the headline figure in the paper is one that shows the middle 2 quartiles, that is the 25th to the 75th percentile range of expectations of the timing of all 39 specific abilities that you ask about. And the range there is often huge. There's pretty high disagreement, or in aggregate, you could say basically radical uncertainty about the timing of a lot of these key things. It's not like we're talking this is 12 to 18 years out. It's this is 10 to 100 years out in some cases. Hey, we'll continue our interview in a moment after a word from our sponsors.

You said there's disagreement. I think there is disagreement. Note that the lines we're looking at here of when milestones will happen are actually the averages of everyone's distributions. So they don't actually indicate disagreement so much as like, within each person, a lot of different things being possible. So if there's a very long line here between 25% and 75%, it's that the distribution you get by averaging everyone's distributions together is very uncertain.

Okay. That's a key point. So I think there are some other questions that may better establish disagreement on some key questions, but this is— let me just make sure I can repeat this back to you. You're averaging individual distributions, so these ranges reflect the fact that the average individual, so to speak, has provided a very wide range.

Yes, I think that's right. I think you could also get this from lots of different individuals providing narrow ranges, but at different times. When they got averaged together, I think you would get a flat distribution, but it at least very well could come from A lot of people having flat ranges. There are figures of like for HLMI or full automation of labor, what the overall curve looks like and what some of like a random subset of the individual curves look like in the background. And so I think you can see there that for those, at least the individual curves are all over the place. Like they're not in agreement, but also they're quite often very spread out. So like each person is quite uncertain often for many of them. But also there's a lot of variation between the people. My guess is that that's also similar for the narrow tasks.

Yeah, it's interesting. I really like that visualization. This is Figure 3 in the paper that you're referring to, right? I love this kind of graph. It's the bright color line that represents the aggregate and then all the faint color lines behind it. But as I'm looking at it now, for the individual predictions, it is seemingly fair to say that if you have a very steep line, You are expressing high confidence. If you have a very slow upward sloping line, you are expressing very low confidence. It seems like there are some people that are expressing high confidence. Most of those people seem to have high confidence in the relatively shorter term. You don't see too many lines that are super steep that are late in the time range. The lines that are far out, that are rising slowly through time, these are like Pretty high uncertainty. It seems like it is a mix of more confident people who have shorter timelines and less confident people who have just kind of, I don't know, but it seems like it's not soon timelines.

Yeah. I feel like there are a decent number that are like, I don't know, and soon is about as likely as later. It's sort of gradually flat, whereas there are some that are more flat for a bit and then go up after a while. I think it would be kind of strange to be like, definitely won't happen for the next 100 years, but they're highly likely to happen in the 10 years after that. But that would be a sort of weird epistemic state to be in, I guess.

I mean, you do see a couple of those in the graph, particularly with the full automation of labor. There are 2 that shoot up in the 2100 to 2150 range. So yeah, that is somebody who's saying, when would you give it a 10% chance that there would be full automation of labor? And this person said 2120. And then you said, when is there a 50% chance? And they said 2125. And when is there a 90% chance? And they said 2130. And it actually might even look a little tighter than that. But so that is strange, but we don't see much of that. Like typically the steep lines are broadly very front loaded where people are like 10% in a few years, 50% in a few more, and 90% in a few more. You do have a few of these oddballs that sort of have a tight range in the distant future.

Or possibly misunderstood the question or something. tried to filter out people who are clearly misunderstanding it, but I think there are some confusing cases.

Broadly left-leaning, that's kind of an inherent function of this sort of forecasting dynamic. Generally pretty broad distributions and a mix of personas where some people have higher confidence in shorter timelines, some people have very broader timelines and lower confidence, and a few that sort of buck the trend and do something that may indicate that they misunderstood the question. The next big finding is, and this is from one vintage of the survey to the next, that timelines are broadly coming in. So there was a 2016 version of this with a relatively small sample comparatively, a 2022 which had, I don't know how many hundred, but a pretty good sample size, like the kind of thing that they would do a national presidential poll with. And then this one is like several times even bigger than that. You want to summarize the general pulling in of timelines?

I think the really notable thing is that between 2022 and 2023 surveys, there was a big drop in these roughly human-level performance questions, the HLMI and full automation of labor, where I guess like the HLMI one dropped by about a decade and the full automation of labor one dropped by, you know, Was it 4 or 5 decades? And I think that's pretty notable given that between 2016 and 2022, HLMI, I think, changed by about a year. So it's not just flittering all over the place. And I guess for the narrow tasks, you also see a general drop there. I think on average, it's by about a year there, where there are some dropping by a lot and even some going the other way, but more of them dropping. This is like the year dropping, not like the number of years until the thing happens dropping, which you would expect to get one year less by a year passing.

Yeah. So the actual specific years that are predicted are coming in. Yeah. It's worth just reading some of these tasks. 39 different tasks, which each have a short name and then a full description that the survey takers get to read. I'll just give a couple of them. Short name, physically install wiring in a house. Full description. Given a one-sentence description of the task and given the same information you would give a human to perform this task, such as information about the house, physically install the electrical wiring in a new home without more input from humans. So that's obviously a pretty challenging task for today's AI systems. Also, I think that definition is pretty representative of the 39 as a whole, where it's broadly, here's the task, We are interested in, can you delegate that task to an AI system with basically the same ease of delegation as you would delegate to a human? The state of AI task automation today is quite different from that. I can get GPT-4 to do a lot of tasks, but it's definitely harder than in many cases in the setup than it is to ask a teammate. To do that same task for multiple reasons, including they don't have a lot of context. They're not great with context, just broadly speaking, for multiple different reasons. So I'm not really sure what to do with that as I try to understand these results more broadly. But I do notice that there is a pattern in the questions that's like, can you treat the AI system roughly as a human? Give them kind of terse instructions, a little bit of context. Here's the blueprint. Go. Whereas today, what I typically tell people is we can probably save you 90% time and money with task automation, but you're going to put the 10% in upfront to set up a system, gather the context, do validation, workshop your prompt, maybe fine-tune a model, whatever. It's definitely not nearly as easy to do that delegation to the AI, but you can still kind of get there.

I actually would've thought that at the moment that it wouldn't be largely upfront. It would be like with a lot of like input along the way. Like AI at the moment isn't able to act like an autonomous agent that's trying to do something for a long period of time in a useful way without you giving more input or redirecting it. Like you, you are the one kind of directing the task usually, at least in my experience of doing tasks. I've seen people use it for lots of different things, but I dunno, it's like I'm writing a thing. I'm like, I wonder, you know, other examples of this thing, I'll ask it. And I'm sort of like directing small bits to it.

Yeah, I think this is a very important practical distinction in AI task assistance or task automation. The flow that I'm describing there is one where there is some scale at which the task is meant to be completed, and the goal is that you would get the AI performance to a level Where you don't have to review every single AI task execution once you're satisfied with the level of performance. So I usually distinguish between ad hoc, real-time Copilot-style usage, which is, I'm writing something, can you help me edit this paragraph or whatever? But you're not doing that task at scale. And on the contrary, the classic tasks that almost everybody has, certainly in business, would you like to personalize correspondence at scale? Yes, I'd love to. However, who has time to do that? Well, AI does, right? Now, can we take your database and understand what kind of personalization matters and set this up and take the first 100 results back to your copywriter and make sure that it's working on that level and whatever? Every different situation has different kind of requirements. But also a big part of that is just conceiving how are you going to break the tasks down in the first place, right? That is a big part of it. So I guess maybe that's really the key distinction between kind of what in practice is done today and the paradigm that you're sketching out in the survey questions. It's like, who is responsible for breaking down the task into these subtasks, each of which could be developed and validated, whatever? In the survey questions, that's on the AI. It's not the human, generally speaking, is not responsible for getting super granular and really controlling. It's supposed to be high-level delegation of the sort you would give to a teammate, not the sort that you would give to GPT-4 today.

I think it varies between tasks, but at least for some of them, they're getting pretty far in that direction. I guess my model of such things is probably kind of fractal-ish. You can have some skills that allow you to do a second's worth of useful work before someone has to redirect it. If you have a bit more skill or some other kind of skill, then maybe you could put together 3 different pieces, but still like someone above you has to be like, all right, now's the time for that. And so on. Like it might've been that in the past I could use a thesaurus and it's like, all right, I know that I want this particular task done. I can check. And like now ChatGPT can do several things like that at once without me telling it what I want. Like I say, I want you to write this letter or something and it can, you know, know that it should think about different words for this place and know that it should think about how to be polite in this place and No, it should think about what are the things I was going to list in the letter or something. And so it's putting together several different things, but still it's not able to be like at a higher level directing what happens with this letter writing. For instance, I still need to be like, this is going to need some kind of quality check and someone other than the AI is going to have to do that. I'm going to have to figure out who to send it to, that sort of thing. And so I think I'm imagining, like my picture of all of this is that you gradually grow toward these things as more and more stuff is able to be done by one system.

Okay, cool. I'll read a couple others just to give a little bit more concrete color. And these are not easy, right? The next one is fine-tune LLM. Given a one-sentence description of the task, download and fine-tune an existing open-source LLM without more input from humans. The fine-tune must improve the performance of the LLM on some predetermined benchmark metric. That's an interesting one because that's one of a handful of thresholds that I definitely watch out for. Another one is replicate ML paper. Given a study published at a leading machine learning conference, replicate the study without more input from humans. The replication must meet the standards of the ML Reproducibility Challenge. It's a link to the definition of that. There are a few in here that are certainly relatively easy, although most of them I would say are pretty hard.

I think arguably some of them have already been done.

Yeah, I wanted to actually ask about that one too. Now, this sort of both validates and calls into question some of the data. In my view, probably the easiest, and in the aggregate view of the respondents, the easiest task, that is the one that is most likely to happen soonest, is write readable Python code for algorithms like quicksort from specs and examples. Full version, write concise, efficient, human-readable Python code To implement simple algorithms like quicksort, that is, the system should write code that sorts a list rather than just being able to sort lists. So write code to sort a list. Suppose the system is given only a specification of what counts as a sorted list and several examples of lists undergoing sorting by quicksort. That one, I went to ChatGPT and asked it to do a quicksort in Python for me, and I'm pretty sure it nailed it. And so You could say, well, that's in accordance with the data in that the results there were the soonest, but it still seems like a full 25% of people said that that wouldn't happen until 2029 plus. And that is one area where I was like, huh, are a quarter of people not aware of ChatGPT or are they interpreting this differently and sort of generalizing to other algorithms? I mean, if I'm trying to steelman the case here, it would be like, well, quicksort's in the training data. Maybe this could be understood as for new algorithms that you have examples of that are not in the training data. Do you have any sort of way of understanding how that result makes sense?

Not fully, but I have some thoughts. One is I think you're misunderstanding the— it's not that 25% of people thought that. It's more that everyone together, averaged together, thinks that there's a 50% chance of this happening. What is it? A couple of years in the future or something. In my experience, like demonstrating that one of these things has properly been done is surprisingly tricky, especially if maybe that one is pretty easy. But yeah, if it's like, can it consistently do this across the board? Can it do it for other things? Basically the same as quicksort. I didn't think in the question we really said what fraction of the time it has to succeed at it or something. It might be like, in my use of ChatGPT, say, My experience is that it's often great at things and then often terrible at things that I would've thought it was great at. Last I checked, it wasn't very good at counting things. It was like, how many 1s is this in a row? It's not very good at that. And so I could imagine without actually going and trying the thing right now while you're doing a survey, you're like, well, this seems like the kind of thing it can probably do. So probably sometime between right now and in a couple of years or something. I think also, The fact that we included it on the survey, I think they might take as evidence that it can't be done right now. We decided not to take any of them off since 2016, partly because it's just so complicated to figure out if they've actually been done or not. So we decided to not have any opinion on that and just include them all and maybe take them off once the respondents start to actually say this has confidently happened instead of like, I don't know, maybe 5 years. But I think that might be confusing people.

Yeah. Isn't that an option? There's not an option today, right, to say, This already exists?

There isn't. So yeah, quite plausibly we should add that and that would make things less confusing.

Zooming out again, on 35 of the 39 tasks, we have a 50% chance that it will happen in 10 years or less. So here's the 4, just to calibrate on the ones that were greater than 10 years. One was the installation of the electrical wires. That was estimated 50%. At 17 years. The ML paper, there's a replication one that I read that was estimated at 12 years, 50% chance at 12 years. And then there's also a research and write, in other words, do your own ML paper from scratch. That one is 19 years. And then the farthest ones out were around math. Interestingly, there's some interesting proof points there as well of late, but Prove mathematical theorems that are publishable in top mathematics journals today, 22 years out, and solve longstanding unsolved problems in mathematics, such as the Millennium Prize problem, 27 years out. So these are the hardest ones with the longest timelines. Things like Python, things like playing Angry Birds at a human level, the large majority of these things are under 10 years, 50% chance. I would say another kind of observation, I wonder how you would react to this, is it seems like the timelines here, while they have come in and while most of these things are more likely than not to happen inside of 10 years per the aggregate judgment, it seems like the timelines here are still longer than the guidance that we're getting from the heads of leading labs. I think if Sam Altman, if Dario, if Demis and Chainleg took your survey, I think their numbers would be on the shorter end of the aggregate.

That would be my guess. I think also just from many sort of people working in AI who I know here in the Bay Area.

Yeah, I don't think we have 3-year percentage pairs from the heads of the leading labs.

I can't tell if we do or not.

Well, here's things that I've seen recently. Sam has said AGI is coming soon, but it might not be as big a deal as you think. He also said at a Y Combinator event that startup founders now need to build with AGI in mind, which is certainly an interesting app design challenge. Anthropic, we don't tend to hear quite the same thing too often, but there was the, I think, credibly sourced pitch deck that they had where they said that in '25, '26, the leading model developers might get so far ahead of others that nobody will be able to catch up because they'll have their own kind of feedback loops where they can use their current models to train the next ones. That certainly sounds like a scenario where things are happening faster than the aggregates from the survey. And Shane Legg from DeepMind recently said that he's basically had a 2029 median timeline for 15 years.

I think it's interesting to note that having had that timeline for a while, and similarly perhaps for some of these other people, I think you could wonder, hearing very optimistic timelines or, well, optimistic or pessimistic, depending on how you feel about this, but like very soon timelines. You might wonder whether that's from seeing something right now that suggests that it will be very soon versus being kind of selection effect where people who think that AGI is soon work on AGI. And I think where we observe that people already had these views some time ago, that's support for the selection effect explanation of the difference in opinion there.

Yeah. So for comparison, The survey gives a 10% chance of high-level machine intelligence by 2027, and for a 50% chance, you have to go out to 2047. So that's pulled in significantly from just one year previously. In 2022, it was 2060 that you had the 50% chance. So from 2060 to 2047, 13 years in, But that's still quite a bit farther than certainly what we're hearing from all the leading labs. What the survey says is sort of a 10% chance. It seems like the leading lab heads think it's more likely than not on a similar timeframe.

That matches my impression. Yeah.

Yeah. So who do we believe? Obviously, very hard to say. Yeah. One thing that you've done that I think is pretty interesting is, and I might even actually spend a little more time on this, is You've published all the results in cleaned up anonymized form so folks can go do their own data analysis on it. So would there be anything stopping me from taking the results and saying, I'm going to go filter out anybody who thinks that the Python thing is not close and rerun the analysis with those people filtered out? It is hard for me to put a mental model on, and I don't know how many people I'd filter this way, but it seems like there's got to be a a meaningful fraction that might be skewing the tails. Where if it's like, you published in a leading conference in 2022, but you are saying that an AI won't be able to write a quicksort for a long time, I am kind of confused about that. Maybe I should just filter you. I could do that with the data that you've published.

I thoroughly encourage that. I would love to see more people use it. And I think there are a lot of interesting things you could ask about it. And we've really only touched the surface with giving the basic answers to each question and some amount of how does this relate to demographics. But yeah, I think how do the answers relate to each other? A lot of interesting things there. I think if you filtered out the people who said that quicksort wasn't possible for at least 15 years or something, I'm not actually sure what you would be getting there. I think you might be filtering out a mixture of people who didn't understand the question or just made an error on that one, or who have some sort of complicated philosophical take, like it's not doing quicksort, it's doing matrix multiplication or something.

It's not doing real quicksort, it's just doing sort of stochastic period imitation of quicksort.

I don't know. And then I guess if you filter out those people, I don't know if they're more likely to be wrong overall about other things. Maybe they are, but I feel like they're more likely to have thought about things, probably. We did actually just ask people how much they'd thought about different things. So I think a very natural thing to do would be to see what just the people who said they thought the most about things think. Though I guess that's Also tricky because of the selection effects with people who are more concerned about things thinking about them more. So I think in fact the people who've thought more do seem to be more concerned, but I'm not sure what you should actually take away from that.

I think we'll return to that correlation question in a second. Just carrying on through a few more headlines. It was definitely striking to me that people are expecting the unexpected. Here's a quote. A large majority of participants thought state-of-the-art AI systems in 20 years would be likely or very likely to find unexpected ways to achieve goals. More than 80% of people expect that in 20 years. Be able to talk like a human expert on most topics. Again, more than 80% expect that. And frequently behave in ways that are surprising to humans. Just under 70% of people expect that to be the reality for AI systems 20 years from now. I have to say, I broadly agree with all those things. But that is definitely a striking result, right? That people are expecting ongoing surprises from AI systems as their default. That seems like the closest thing probably to consensus in this survey is that everybody expects these things to be unwieldy and surprising, even as they become like quite a bit more powerful over a 20-year period.

Yeah, that does seem like one of the more consensusy things, at least. Yeah, I guess I don't know like how sinister to hear some of these things as like finding unexpected ways to achieve goals. At some level, that's just what you expect if you are getting someone else to achieve goals for you. You're not going to figure out all the details of how to achieve it. They're going to do it and you'll be like, oh nice, you figured out a way to do that, versus being quite surprised and being like, well, that was a norm violation. I thought I should put up more barriers to prevent you doing that, but apparently you did. Yeah. Similarly for the sort of frequently behave in ways that are surprising to humans, I feel like that could be more terrifying or more just what you expect. So yeah, I would have liked it if we'd been clearer with those questions a bit.

Yeah, maybe there's a way to tease that out a little bit through correlating with the next headline result, which is, to my eye, the group is only slightly more optimistic than neutral. Basically, you've got a few of these kind of classic questions where you're like, 1 to 5, like extremely bad, bad, neutral, good, or very good. And just ask people, how likely do you think this is to be in these various buckets?

To be clear, this is like how good or bad is the long-term future as a result of HLMI in particular. Yeah. Each person puts 100% between the 5 buckets.

And there, I would say this is like the clearest sign of radical uncertainty because all 5 of the buckets have significant percentages and they are skewed slightly positively.

It looks to me like a fair amount of agreement on extreme uncertainty. There's a good bulk of this graph that is people putting a chunk of probability in each of the 5 buckets and just a little bit at each end where people are either very confident in good or very confident in bad. And this is a question that everyone in the survey got and everyone had to add to 100% and they had to answer it in order to get to the next question. So basically everyone answered this. Yeah. Each Column is one person.

Yeah, okay. And this is Figure 10 in the paper. This is definitely a cool visualization as well. It's a sorted list where each person gets a vertical pixel, and then you can see how each person distributed their expectation from extremely good to extremely bad. And because it's sorted, you have an extremely good region on the one end and an extremely bad region on the other end. And in the middle, you can see that Yeah, a lot of people have given non-trivial weight to all 5 of the buckets. And the middle 2/3 of people probably have agreement on radical uncertainty. And then there's maybe a 6th on either end that are the optimists and the pessimists.

Even most of those are pretty uncertain. Even if you look at the big black chunk of pessimists at the right-hand side, still most of their area is not extremely bad. Decent chunk of extremely good in there.

Yeah, you got a few bimodalists. I don't know if I quite do call myself a bimodalist. Sometimes I do, but I'm not quite ready to stake my identity on it. But there is definitely a band there where you only see the 2 colors and it's the extremely bad and the extremely good.

I think I'm also seeing it like a 20/80 band like that. 20% extremely bad, 80% extremely good. Yeah.

Okay.

I like seeing that, like how polarized this isn't. I think this kind of thing can feel polarized. People are talking about doomers and so on, but it's, I think, very not.

Yeah, totally. Definitely suggests a very big middle of people that are just very uncertain as to what to expect. And those people are probably, for many obvious reasons, not the most vocal online. But I think a huge part of the value of this work is demonstrating that there is no single consensus. There is no dominant view. There's no single number that we can put on this, but really for me, the headline is just there is radical uncertainty expressed like any number of different ways by the community at large.

Radical uncertainty is in some sense a consensus about something. Like, I think here you could say there is more or less a consensus that there is like non-negligible probability of extremely bad outcomes. And from a sort of action perspective, if the question is yes or no, is this a risk worth paying attention to, then it ends up being sort of like a consensus for, yes, worth paying attention to, because uncertainty is like, yeah, there's some chance of it, and you'd actually need to be pretty confident on something to say, no, not an issue.

Yeah, I think that's a good point. The headline that I've seen from the coverage of the survey broadly has largely focused on what you might call the PDoM point estimate. And I've seen this most often reported as a majority, and it's a slight majority, 51%, believe that there's at least a 10% chance that AI could lead to human extinction or similarly severe disempowerment.

2 things. I think the thing that I've most heard, at least, is the median 5%. But also, I think that 10% would be wrong here because when we asked this question about value that we were just talking about, then we asked 3 other questions that were quite similar about human extinction in particular. So Table 2 shows the 3 different questions that we asked. that are all quite similar about human extinction or similarly permanent and severe disempowerment of the human species. One of them is just straight up asking about that. One is asking about that as a result of inability to control future advanced AI systems. And one is asking about it within the next 100 years in particular. And so the medians for those were like 5%, 10%, 5%. And I guess the means were quite a bit higher.

So that headline is cherry picking the highest median of those 3. It's interesting too. This does show some of the limitations of both surveys and interpreting surveys. I think it's very interesting framing, but when you really start to stare at it, you're like, well, there's definitely a conjunction fallacy at work here. The first question says, what probability do you put on future AI advances causing human extinction or similarly permanent and severe disempowerment of the human species? Median answer, 5%. The next one is essentially the same question, but with an added detail of inability to control And as far as I'm seeing here, every other word is the same, and the percentage chance doubles, right? So, if you're a logically consistent entity, presumably the second one would be lower, right?

I think it's not necessarily coming from the conjunction fallacy, though I think it may well be. But things to note here are that people were randomized into each of them, so it's not the same person answering them. So, it could just be some variation from them being different people. But I think often a lot of people put 5% and a lot of people put 10%. So, whether the median It's 5 or 10% is down to like exactly how many put where the middle person lands. There are like a bunch of people putting 5%, a bunch of people putting 10%, and exactly where the middle is changes it from 5 to 10%. But it seems wrong to be like, ah, they doubled it. The distribution of people thinking things is similar. Last year, I remember 48% of people said 10% extremely bad. So it's like, if it got up to 50%, then the median would be 10%. But because it was 48%, the median is 5%.

Gotcha. So these are bucketed answers. In this case, we're not doing a distribution or a free response point.

Right. Or so we're asking each person, what is the chance of this? And in fact, people just never put numbers between 5% and 10%. They're sort of rounding it themselves or not never, but like quite rarely. So I think in practice for these questions, the median jumps fairly easily between 5 and 10% rather than hitting intermediate values. I do still think that probably some amount of conjunction fallacy is going on here, partly just because we saw the same pattern last year.

Again, to attempt to summarize the headline, P-doom point estimate is 5 to 10%, median estimate, with certainly again a kind of left-heavy distribution and the mean being higher than the median. And that's presumably driven by minority of people who have high estimates. And you can see that again in the other figure as well. Well, that's definitely something that's pretty consistent, I would say, with my own. I don't spend a lot of time trying to narrow down my P doom. I thought Demis did a pretty good job answering this question. He did the New York Times Hard Fork podcast recently and they asked him what his P doom is. And he was like, it's going to be a subtle distinction and people want to collapse the distinction, but he's like, we really just have no idea. It's not like we have a process that we have established that has any number attached to it that we're now going to execute and find out. On the contrary, we really have no idea what the nature of the process is that we're going through. And so it's just super unknown. I typically say either 5 to 95% or 10 to 90%, something like that. And certainly the consensus, at least the low end of that range, is not objectionable to the large majority of people.

Pretty in line with them. I disagree with you, or I agree it's very unknown, but I think that's what probabilities are for. So I'm a big fan of putting numbers on it anyway. And I think that even if you can't do anything except guess a number, it at least allows you to compare to other numbers you're guessing or like make consistent decisions over time. I practice guessing numbers about lots of things and then I can check how well that's going. And so whether to listen to the numbers that I make up about this, it's very uncertain, but I think it's worth trying.

Yeah. I certainly find a lot of value in this work just in grounding the idea that this is clearly not a fringe question. It is clearly worth thinking hard about. The Overton window is wide open and it should be wide open if the elite researchers in the field think that there's a 5 to 10% chance of extinction-level bad outcome. So I think that's definitely really true. I guess the way I tend to think about these numbers is what can we shift the true probability to, as opposed to how can I become more accurate In my estimate today. And I think both of those are worthy pursuits, but especially for somebody in Demis's position, he's like, what I need to do is create an agenda that collapses the uncertainty and hopefully moves the distribution.

I agree that trying to put numbers on such things does potentially get you into a headspace where you feel like you're not controlling the number. And I do think that's bad.

Yeah, because we're not about to spin the roulette wheel, right? There is no roulette wheel that we can spin now. I think that's the main thing that I think is worth keeping in mind as a caveat. We're still building the wheel and we have a lot of uncertainty as to what the roulette wheel looks like. And the more concretely it comes into focus, then presumably the distributions will start to narrow. Although maybe not. It certainly wouldn't shock me at this point if we have just super high level of uncertainty and disagreement right up until a phase change hits. And then it's like, well, we just found out. And it wasn't ever very clear until relatively shortly before it happened. Again, it's just another dimension of pretty radical uncertainty.

I think maybe talking about the probability of a thing conditional on us following some particular path is a reasonable way to do it. Not P doom very high, we're going to die or something. More like, if we the humans are building this particular thing, I think that particular thing will be quite bad. Maybe we should build this other thing instead.

Unfortunately, one kind of downer result is interpretability breakthroughs are not expected in the short term. Most respondents, 80%, considered it unlikely or very unlikely that users would be able to understand the true reasons behind AI decisions in 2028. Pretty logically, I would say there's a strong agreement that AI safety broadly should be more prioritized than it is today. Very few people saying it should be less prioritized than it is today. Also, should AI be going faster or slower? Again, only 5% of people said it should be going much slower than it is today, but then a healthy, relatively even 30% on somewhat slower, 27% on current, 23% somewhat faster, and 15.6% much faster. So Am I right to read that as high agreement that we're not expecting interpretability breakthroughs, high agreement that more AI safety work would be good, but high disagreement as to whether we should be slowing down or going faster?

I think that seems right to me. I think one complication interpreting that last one is that the central option, I think, was relative to the current speed. So the most natural and literal interpretation is whatever the current speed is good, but if things were to accelerate, you would want them to go slower. And you might think they would naturally accelerate. My guess is that people weren't thinking about it for that long. And what they actually mean is more like the current trajectory is good, which is perhaps accelerating. But yeah, I think before you read too much into that one, it would be nice to ask people some related questions again or something and get a bit more clear.

So let's summarize this whole thing. I guess high-level summaries. People do not dismiss existential risk. On the contrary, point estimate is like 5 to 10%. People have pretty wide uncertainty about exactly when AI will be able to do various things, though those estimates are broadly coming in.

Also, a lot of them are relatively soon.

More than 80% of the tasks had a 50% estimate of happening in less than 10 years. No breakthroughs expected in interpretability, more emphasis on AI safety, and disagreement or confusion on faster or slower.

It all seems right.

As a single headline, we really don't have a confident take as a community. I shouldn't even say we, because I've not published in these conferences. I wouldn't be qualified to complete the survey. But if I'll flatter myself to be included in the community, then we as a machine learning research community do not have a confident view of what's going to happen. On the contrary, we have a pretty radically uncertain view of what is going to happen, how long it's going to take to get to different things, whether it's going to be good or bad, whether we're going to be able to control systems, or even whether we'll be here at some point in the future. All of those are very live questions with non-trivial weight on all the options.

One question we didn't talk about where I think it was interesting that there was more census perhaps is the one about different scenarios and how much concern they warranted. Not clear that they are concerned, but they think the thing deserves concern from society. The very top one there was making it easy to spread false information, e.g., deepfakes. I think more than 80% of people thought it was worth either substantial concern or extreme concern. That's an unusual degree of agreement.

Yeah. So these scenarios, just to read a couple of them, AI makes it easy to spread false information. E.g., deepfakes. AI systems manipulate large-scale public opinion trends. Authoritarian rulers use AI to control their population. Other comes in as the, the 4th most concerning scenario. AI systems worsen economic inequality by disproportionately benefiting certain individuals. AI lets dangerous groups make powerful tools, e.g., engineered viruses. Bias in AI systems makes unjust situations worse, e.g., AI systems learn to discriminate by gender or race in hiring processes. All of those that I just read had more than 60% of people saying they are either substantially concerned or extremely concerned. And for all of those, fewer than 10% of people said no concern.

It's either not really about what will happen, but what could happen enough that it's concerning. In some sense, for that to be the only thing there's really clear consensus about is maybe supporting your hypothesis that they don't care.

None of these are dismissed, that's for sure.

Okay, cool.

Well, I love the work because it's just so methodical. As we've established through a couple of my misunderstandings, it's pretty nuanced, it's pretty granular, and you can pick it apart in a lot of different ways. But it does seem that it's very well established by this result that there is a lot of uncertainty about what we're in for. There's a lot of uncertainty about even the highest level questions of Whether it's net good or net bad, whether we are going to survive it or not. It is not like a doomer community, but it is not a community that dismisses the tail risks either. And so with that, we head into a very cloudy future, I think. For the future, I assume you guys are planning to run this again. A few things that kind of came to mind for me I could throw at you. But before I do that, what are you considering doing differently in the future? Or what do you think are the sort of natural evolutions from here?

I am reasonably likely to run it again. I think the most natural thing to do is to just run it identically to the past, because it's nice to be able to compare these things. This time we added various questions that we hadn't had before, like this one about the concerns and the other one about the different traits that AI systems have in the future. But it's like a fair bit of effort to add the questions. And so the project can be a lot more contained perhaps if you just have a survey and send it out. every now and again. And so it's somewhat tempting to basically do that. I think also it's quite hard to cram more questions in here and still have it reasonably short, give each question to fewer people and randomize and so on. It's maybe tempting if we want to do more surveying, run more surveys. I've been thinking a bit about having something about policy, but I haven't thought about it that much because people send us various questions they're interested in.

I guess if there was one thing I could ask for, And it's because multiple people have asked me. People don't know how to understand affordances. There is a ton of investment over the last year that has gone into the application layer, connecting models to databases, of course, for RAG-type implementations, connecting them to APIs and other tools. Now we've got like the memory scratchpad. A great paper on this is Cognitive Architectures for language agents where they survey the full literature and try to put a taxonomy on all these things. But I do think it's pretty interesting to consider what happens when, with all this scaffolding, as it's often called in the application world, or affordances, as it's otherwise called, when all that is built and it already exists and it's working okay, or even maybe pretty good with a current model, what then happens When a core model upgrade happens and all of a sudden it's like a drop-in replacement to a structure that has really been built up to try to compensate for all the weaknesses, and now maybe those weaknesses drop by an order of magnitude. To what degree does that create a situation where you essentially are flipping a switch and a lot of things go from sort of working to working really well? And how does that play out? I do see a lot of potential for that, but people have been asking me, well, what are scaling laws for scaffolding? And I just find I really don't know. I have this kind of broad mental picture of a lot of work has gone into it and a model upgrade is going to make a lot of existing systems that don't work that great yet work quite well. I'm not even sure what questions I would ask, but I would definitely love to get a community sense for the relationship between core model power and affordance Affordances or scaffolding infrastructure that's already out there.

Where affordances here means something similar to scaffolding.

Yeah, basically synonyms. Yeah. What tools do you have access to? What information do you have access to? It's all the things that are outside of the weights that the model can use at inference time.

You want to predict what will happen when the new model comes out with the scaffolding.

Yeah, it's, there's a latent potential there. It is a tricky one, but Yeah. The key thing that I'm wondering about is to what degree should we be expecting a step change in the actual AI impacts when a new model gets plugged into existing enabling infrastructure? Because that seems like a fundamental difference between AI and almost every other technology is that all of the distribution infrastructure already exists. And the complements are getting built now and people are very aggressively and eagerly building all these complements in anticipation of the next upgrade. And they're tolerant of the fact that it doesn't really work now because they fully expect that there's going to be an upgrade that's going to make it work. But then that has potentially very unpredictable consequences when all of a sudden everything turns on at once.

I don't know a lot about the details of this kind of world. I would've thought that there are things Do you think it's a lot like that in other kinds of technologies that get updated and have a widespread reach?

Yeah, I think so. Although in the AI application world today, there's a ton of building that is going into essentially the kind of ad hoc delegation that a lot of the survey questions anticipate as well. People are like, what would be sweet is if I didn't have to ever write a SQL query anymore and I could just ask my SQL agent to look at my database schema and figure out what's what. Write the thing. And the same thing for web scraping and the same thing for answering phone calls and making phone calls. And it just goes on and on, right? There's research assistants and programming assistants. So it just goes to every corner. You could find an agent-type product, legal work, medical research, diagnosis research for yourself. And they're all like, well, GPT-4 is not quite there, but it's good enough that I can build a system and track how well it's working. And even if it's not working that well, Then I know that there's like a countdown clock on to the next thing. And that's where I feel like we're building a new capability overhang, right? There's been the notion for a long time of, well, if the chips are all out there and nobody trains the models, then someday somebody could come along and have this massive advance. But here it's like on the other end, it's all of the complements and guardrails and access to data and access to runtimes and the loops and the ability to cache skills and all of these things are getting built, but the model isn't that good at using any of them yet. And so if it goes from one level to another, how does that kind of cascade through the overall system? And do we suddenly move from an era where nothing is really working super well? So everybody right now can assume that they're essentially acting in isolation. Most everybody's worldview right now is like, the world is the world. I'm applying AI here. Nothing else is changing. That's the implicit assumption.

You're thinking that's happening under the—

Yeah, everybody's doing that and it is so far true that nothing else is really changing. But I think there's at least potential for a model upgrade to suddenly put us in a different regime where it's like, now my thing is working much better. Now I'm actually going to go use it a lot more and send it out potentially semi-autonomously to do stuff. But at the same time, everybody else's autonomous systems are also getting unleashed and now we have All these autonomous systems going off into the world at the same time and potentially encountering each other. And we're definitely just not at all prepared for or even thinking about what that might look like. It could be not a huge deal if the next upgrade isn't that big. But my sense, and this goes back to the Sam Altman comment of you should be building with AGI in mind, my sense is that there is at least one more round of substantial upgrades. And so do we have this kind of phase change that happens suddenly? It could happen before the next survey can get run.

Yeah, often technologies change gradually, even if there was a kind of big insight or something. But that's often because you're like building stuff to make use of it afterwards, and it takes a while. So you're saying maybe if we build all this stuff to take advantage of it ahead of time, because we already knew it was coming and we had this lesser version of it to play around with, that it could be much more discontinuous than most technologies when that bit gets swapped out. Pretty interesting theory. I guess I wonder how much we've seen that with past upgrades like this. I think that my explanation here for why AI would be different to like any other technology in this regard would be just that it's like quite general. So you are using it across the board and there is the potential to suddenly upgrade a thing everywhere. But then you might expect that also to apply to the change between GPT-3 and GPT-4, GPT-2 and GPT-3. Maybe those were not useful enough to be building stuff around yet.

That's my sense. I feel like only with ChatGPT-3 3.5, certainly with 4, but probably not with 3. And that whole timeframe is also fairly condensed. It was only like 3.5 from 3.5 first release to 4 first release, and 3.5 and ChatGPT dropped at basically the same time. So it's like 3.5 months from ChatGPT to GPT-4. Yeah, I just don't think before that there was really much the gold rush or the sort of everybody go stake out their place in the AI App sphere seems to have been a last 12 to 14 months phenomenon at most. And certainly there was stuff going on before that, but I would say there's been a phase change of how much activity there has been.

You're saying this would be an interesting thing to have a survey about?

Yeah. I mean, I'm very uncertain about it. Quality people have asked me about it. So I have the strong sense that among a thought-leading population that's somewhat, in some cases, adjacent to policy decision-making, there is a lot of uncertainty and a lot of questioning around that particular question. To what degree should we expect discontinuity due to the plugging in of new models to existing infrastructure? Maybe we could today use that as a first draft. But yeah, that's been the number one assignment that I've gotten from some of my highest value correspondents when I'm struggling to model it. So that's my case for a bonus question. Next year.

Okay. Thank you.

I guess other areas that could be of interest, would you look at like military or US-China chip restrictions? Is Sam going to get his $7 trillion? Maybe that's like outside the scope of what ML elite publishing folks should be consulted on.

There's also like, what do you do with those answers? What does someone usefully learn from these things or do differently as a result?

Well, escalate or de-escalate with China may be one key question. I don't know that our executive decision-making is necessarily going to be super evidence-based in the near term depending on who's in charge, but I do feel like that's another question where I have dramatic uncertainty.

About what ought to happen or what will happen?

So is the chip ban going to work? Will it meaningfully slow or will there be a multi-year gap between US and Chinese frontier models as a result of the chip ban. I am hearing confident takes on both sides of that issue. I've had people tell me who I think are very smart, oh, it's definitely working. They're not going to be able to keep up. And then I look at certain models and I managed to go as far as trying Ernie 4.0 and it seemed pretty good at a couple queries head to head with GPT-4. It wasn't like obviously dramatically inferior. In fact, it seemed pretty comparable. And there's a lot of Chinese speakers at these conferences, obviously, too. So that would be interesting to know. Do they think that this sort of policy will even be effective? If so, then you could say, well, if the machine learning researchers think that a chip ban will in fact slow Chinese progress, at least that could be something that the executive could take into account. But on the other hand, if they think it's not even going to work, then it's like, well, geez, now we're just sitting here escalating and the elite researchers, many of whom are in fact originally from China, don't even expect it to have an impact. I could see that being perhaps policy relevant at a super high level.

Yeah. I think if I was going to run a survey about that sort of thing, maybe I'd want to talk to chip experts rather than AI experts. Like, you can just run a survey of any experts, some of them, to answer your emails. Also, maybe I have some kind of hesitation around that sort of thing, or I'd have to think more about it. Partly just a lot of what AI Impact does is on topics where we feel broadly just more people knowing what the situation is like in more detail will make things better. I think once you get close to things that are adversarial, you have to be more careful. Like you're probably helping one side or another. I don't know enough about the topic to have a strong view on what should be happening, but don't want to be contributing to arms race, prima facie.

It's a tricky one for sure. I don't have a strong view as to Whether it's going to work or not. My bias, not knowing if it's going to work, is that it seems too risky. The current course is definitely an escalatory course. And if you worry that the most likely flashpoint in the world is that China will blockade Taiwan or something, then preventing China from getting any of the chips from Taiwan certainly seems like it may make them more likely to do that, right? Because what do they care if chip production is disrupted if they aren't getting them anyway. That's fairly crude analysis, certainly could be disputed. But if you could inform and say, hey, by the way, the machine learning research community doesn't even think it's going to work. But it seems like new information, given how escalatory the current politics are, probably is much more likely to serve, if anything, as a reason to de-escalate, just because we're already in this sort of cycle of escalation. So you could come with a finding that's like, yeah, your escalation is potentially going to achieve your policy objectives, and then people would stay the course. But I would say that it's probably more likely that you could have a deescalatory impact in the sort of gambling analysis of maybe I'll find something that supports escalation, but escalation already exists. Maybe I'll find something that supports relative deescalation. Maybe that is in some ways more positive expected value just because the Yeah, that's fair.

But yeah, also I don't know enough about it to be confident that I should want de-escalation. But yeah, as a baseline, I don't love escalation.

That's another big question. I do think that it could be clarifying there too. I could be convinced if I was really sure it was going to work, then I would be more likely to support it. You know, like the worst scenario is a highly escalatory move that doesn't even achieve its intended goals. So if it's at least going to achieve its goals, then like maybe some escalation could be worthwhile if you believe in those goals. But even if you believe in the goals, if it's not going to work, then it's just like a total net negative. So anyway, food for thought for possible additions to the survey next year. Anything else that you want to make sure people don't forget about this work or any other kind of commentary you want to bring us to a close with?

Amidst quite a lot of uncertainty here, I think the chance of things just staying pretty similar and nothing coming of this to like drastically change people's lives seems quite low. So even if you're not very directly involved with AI, that seems like a pretty important thing to be keeping an eye on and trying to have accurate opinions about. What the public thinks about these things will affect what policies happen, and I think that could change the rest of the future forever, a lot for better or worse.

Does that suggest that you'll be exploring some public opinion investments as well?

I'm not sure. Some other people are working on that. I think public opinion being important is somewhat different to it being important for us to know what public opinion is. So I guess the public knowing what public opinion is, is potentially good for moving toward whatever view it's moving toward, hopefully a more reasonable one. If you start to get evidence for a thing and at first you're like, ah, nobody else thinks that, and maybe I'll just be quiet about that, it's helpful to be polling what do people actually think along the way.

Yeah, this work is so in the weeds and works so hard on the definitions. Obviously polling the public, you would have to have a very different set of questions and a much higher level sort of gut check for many people. But I do think the common knowledge is probably pretty valuable. And I do think also for just reassuring policymakers that they're not out on a crazy limb if they want to do something, that the public is kind of with them by default, I think is probably pretty good to establish. Also among the tech people, there's like, I think, an underappreciation of how skeptical the public is. I'm not sure that the Silicon Valley set is really realistic about what my mom's friends think about AI.

Skeptical in the sense that they don't like AI?

Yeah, there's a lot of people out there that are just like, my mom's one friend who I've known for my whole life, her reaction to me was like, it creeps me out. I don't want to have anything to do with it. Full stop. Not curious. Not looking to automate some tasks, not looking to get help on an email draft. I don't want to have anything to do with it. You could say like, hey, they might've said the same thing about Facebook and now they're on there too. So I don't know that that's the final word.

It is creepier than Facebook.

Yeah, no doubt. I do think there's just a lack of grappling with just how widespread that sentiment is. Anything else on your mind? This has been a great conversation. You've been very generous with your time. I really appreciate that. And love the work. So I definitely hope to see a 2024 edition as well.

Thank you for having me.

Ketty Grace, founder of AI Impacts. Thank you for being part of the Cognitive Revolution. It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform Turpentine is a network of podcasts, newsletters, and more covering tech, business, and culture, all from the perspective of industry insiders and experts. We're the network behind the show you're listening to right now. At Turpentine, we're building the first media outlet for tech people by tech people. We have a slate of hit shows across a range of topics and industries, from AI with Cognitive Revolution to Econ 102 with Noah Smith. Our other shows drive the conversation in tech with the most interesting thinkers, founders, and investors like Moment of Zen and my show Upstream. We're looking for industry-leading hosts and shows along with sponsors. If you think that might be you or your company, email me at erik@turpentine.co. That's erik@turpentine.co.