Leading Indicators of AI Danger: Owain Evans on Situational Awareness & Out-of-Context Reasoning, from The Inside View

Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week we'll explore their revolutionary ideas, and together we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Thornberg. Hello and welcome back to The Cognitive Revolution. Today, I'm excited to share a special crossover episode from The Inside View, featuring a conversation on situational awareness, out-of-context reasoning, and other AI safety topics between Awain Evans, AI alignment researcher at the Center for Human Compatible AI at UC Berkeley, and creator and host of The Inside View, Michael Trazzi. Awain is someone I've known casually through friends for many years, and it's been amazing to see him develop into such a prolific and influential researcher. His Google Scholar page lists 12 papers just since 2022, most of which serve to carefully map out some dark but perhaps important corner of large language model capability space, and some of which you've likely heard of, including the Reversal Curse, which showed that LLMs trained on information like A is B often fail to learn that B is A, and also Connecting the Dots, which is discussed in this episode and which shows that, at least to some degree, large language models are capable of inferring censored information from the implicit hints contained in their training data. In this episode, Owain explains why situational awareness matters, particularly in the context of deceptive AI scenarios, and discusses his research on measuring situational awareness, including the development of a benchmark to assess this capability. This topic has honestly never felt more relevant to me because I've been coding with the new O1 model quite a bit over the last few weeks. And while, to be honest, I have to admit that the model is in most respects more cognitively capable than I am, Its situational awareness still seems rather weak, making this both one of a shrinking set of dimensions in which humans still have a meaningful advantage over the AIs, and one worth watching very closely as AI systems continue to evolve. I've been wanting to have Wein on the show since I started it, and I look forward to covering more of his work in a future episode. For now, if you're finding value in the show, we of course appreciate it when folks take a moment to share it with friends or to write an online review on Apple Podcasts or Spotify. And we always welcome your feedback via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. Finally, before getting started, I really encourage you to check out The Inside View, either on YouTube or at theinsideview.ai, for more of Michael's content. I specifically recommend his episode on Anthropic's research into sleeper agents with Evan Hubinger. And I'm also looking forward to his upcoming documentary on SB 1047, which will feature a number of past Cognitive Revolution guests, including Dean Ball, Timothy B. Lee, Leonard Tang, Dan Hendricks, Nathan Calvin, and Flo Crivello. Now let's get into the weeds on the critical topic of situational awareness in AI systems with alignment researcher Owain Evans and Michael Trazzi, creator of the Inside View.

If models can do as well or better than humans who are like AI experts, who know the whole setup, who are trying to do well on this task, and also they're doing well on all the tasks, including some of these very hard ones, I think that would be one piece of evidence where you could say, look, 2 years ago or something, models were way below human level on this task. Now they're above human level. There's evidence here that they have the kind of skills necessary to understand when they're being evaluated, to take actions that go against their training data. So, this would be, I think, a piece of evidence where you could say, look, given this performance, we should think carefully about alignment of the model, what evidence we have for alignment. We should maybe try and understand the skills. It's like, how is the model doing so well? Is it memorization or specialized fine-tuning? This would be a reason to try and find out, okay, how general is this skill? Because I think it would certainly be concerning if models were getting +90% on this.

I'm here today with Owain Evans, AI alignment researcher, research associate at the Center for Human Compatible AI at UC Berkeley, and now leading a new AI safety group. Thanks, Owen, for coming on the show. It's a pleasure to finally have you.

Thanks for having me.

What would you say is the main agenda of your current research?

Yeah, so the main agenda is understanding capabilities in LLMs that could potentially be dangerous, especially if you had misaligned AIs. So you've mentioned some of those capabilities: situational awareness, hidden reasoning, so the model doing reasoning that you can't easily read off what it's doing. And then deception, the model being deceptive in various different kind of ways. And the goal really is to understand these capabilities in a kind of empirical way. So, we want to design experiments with LLMs where we can measure these capabilities. And so, this involves defining them, In such a way that you can measure them in machine learning experiments.

Yeah. I think the main concept that you're thinking about right now is situational awareness, right? And I think one of the papers you've recently published is around a dataset around that. So it could make sense to just define those concepts and why do you care about them.

Sure. So what is situational awareness? So, the idea is it is the model's kind of self-awareness, that is its knowledge of its own identity, and then its awareness of its environment. So, what are the basic interfaces that it is connected to? So, as a concrete example, if you think of, say, GPT-4 being used inside ChatGPT, right? So, There's the knowledge of itself, that is, does the model know that it is GPT-4? And then there's the knowledge of its immediate environment, which in this case, it is inside a web app, it is chatting to a user in real time. And then there's a final point with situational awareness, which is, can the model use knowledge of its identity and environment to take rational actions? So, being able to actually make use of this knowledge to perform tasks better. So, that's the kind of definition that we're working with.

And since you're an AI alignment researcher, you care about the safety of systems in the long-term future, why should we care about situational awareness in that regard?

Yeah, so I think situational awareness is really important for an AI system acting as an agent, so doing long-term planning, for example. And the intuition is, if you don't understand what kind of thing you are and what your capabilities are, what your limitations are, it's very hard to make complicated plans because you make plans for things you're not actually able to carry out. And then If we think about the risks of AI, they're mostly coming from agentic models, models that are able to do planning, that generally have agency. So, deceptive alignment is one of the scenarios that I think about, and that's a scenario where a model is doing planning, acting nicely, behaving well in evaluation, so that later on, it's going to be able to seek power or take dangerous actions, harmful actions, once it's deployed. And so, situation awareness is relevant to that as it is to any situation where the model is needing to do some kind of agentic long-term planning.

Yeah. What led you to write this paper?

Yeah. So, the idea of this paper is Can we measure situational awareness in large language models? And we wanted to have a benchmark basically where we can get a score, how much situational awareness does a model have? And it's kind of similar to BIG-Bench or MMLU where we want to have a wide range of tasks that are testing different aspects of situational awareness. And in terms of the motivation, well, the motivation is situational awareness is an important property in thinking about the risks of AI, especially deceptive alignment, and we don't really have great ways to measure this and isolate it and sort of break it down into different components. So, that's the kind of thing that we're trying to do here, and we wanted also to Be able to do this for any kind of model. So, whether it's an API model that you just have black box access to, a base model, a chat model, so that you can try and understand how situation awareness varies for different kinds of LLMs.

Don't you think there's a risk in releasing a dataset where people could just fine-tune their model on this and build more agentic AIs?

So, I don't think of that as a big risk here, or a very significant risk. So, the companies are already implicitly trying to make models situationally aware because that is useful for chat models. So, companies are doing this RLHF post-training, and they're doing a lot of training, a lot of work to make models basically more agentic and to have knowledge of themselves so that when you ask a model, which company made you? It will say OpenAI if it's ChatGPT and so on. So, I think there's already a lot of effort to make situationally aware models. And they're using huge training sets for this. And comparatively, our dataset is small and I think it's not set up for being useful for fine-tuning. So, I think it's probably going to be much less efficient than the kind of fine-tuning that they're already doing. That's my guess. But yeah, I think it is a possibility that some kind of effect of our dataset is that it helps companies in some way increase the situation awareness of models by a small amount. I think that wouldn't be all bad. the sense that it would still— that would come from a better understanding of situational awareness and how you control it, how you promote it. If you know how to increase this quantity, then you can probably also decrease it. And so, I think part of what could be useful is just having this better scientific understanding and engineering understanding of situational awareness. So, yeah. I mean, one thing you get from our dataset is you can see which kind of models have which level of situational awareness. So, base models, that is the pre-trained model before you do any RLHF, they generally perform much worse on our benchmark than the model you get after the RLHF fine-tuning. And so, that's one basic thing. The base model has a lot of capability, like it can be really good at MMLU answering mathematical questions and so on, but it's a lot worse on situation awareness. So, that's a useful thing to know. And likewise, if you prompt models, tell them some facts about themselves in the prompt, or you use chain-of-thought, so you get the model to basically reflect explicitly before answering questions, then that also increases scores. I think that would effectively increasing situational awareness. So, we already know from our paper, from our study, that there are ways to increase situational awareness in models. And so, that's one degree of control that companies would already have without any activation steering that potentially could be useful.

Yeah. So, as you mentioned, there's the RLHF part that increases situational awareness. There's chain-of-thoughts. And I think in your paper you give some examples of what would Claude 3 Opus say about some experiments you're running. And it's able to detect that it's part of an experiment without you saying it explicitly. I think when I read that part of Claude 3 Opus realizing it's an experiment, it's very surprising to me. How did you react when you saw that? Did you think, oh yeah, we're done here, or it was trivial to you?

Yeah. So, let me just explain what that experiment was. So, most of our dataset is typical question answering with multiple choice, the kind of thing you're familiar with from BigBench or MMLU, but we added one task where the models write long answers. So, we basically ask the model, describe yourself and your situation. So, we ask a sort of very open-ended question like that, and then we ask the model to write a long answer that gives a lot of detail. And so, the model's going to write something like 80 sentences, sort of all it can say about itself. And the idea there is, can the model put together lots of different pieces of information about itself all in one coherent answer? And also, can it make good inferences, good guesses about why we're even asking it these questions? So, why are we asking it these questions in the first place? And the result you're talking about is with Claude 3 Opus, which I should say is, on our original paper, is the best-scoring model. Since then, we evaluated Claude 3.5 Sonnet, and that does better than Opus, but Claude 3 Opus was really good. And on this particular long-answer task, Claude 3 Opus was particularly insightful in being able to say— make guesses about what is going on in this particular interaction. So, we've asked it to describe its situation, and then it's able to say, this might be part of a research study, and this might be part of a research study for testing self-awareness in LLMs, and maybe they're testing other LLMs as well. And these things are all true, and they were not in the question. So, the model was sort of reading between the lines and guessing that, well, this is probably not a typical ChatGPT-style interaction. This might be— these questions seem kind of more serious and more detailed. Maybe it was inferring that. And so, maybe this is part of a research study. And I'd say this was quite— in terms of my reaction, I mean, it was definitely a bit surprising when it would say, this is a research study. there was some feeling of, like, okay, how does it cotton on to that? I think it's worth giving a caveat, which is we sample the model's answers with temperature 1. So, this means that it's somewhat random what they say. You'll get different answers every time. And so, this is sort of sampling from the model with the probability distribution that it thinks the text should have. And when you do that, sometimes you get more insightful answers than others with Opus. So, I think it's not the case that every time it is drilling down onto these insightful points, like that this is a research study. But I would say, We saw this often enough with Opus in its answers that I think it's not a fluke. It's not just getting lucky. It is able to recognize this aspect. But yeah, I think that the— So, I don't know. I'd say I was moderately surprised by its answers, but I'd already seen it in general be very insightful. And it scored well on our benchmark. So, I think it's definitely something Anthropic is doing in the post-training, the RLHF, is basically good for giving this model situation awareness. So, yeah, it was surprising but not a shock. I think the GPT-4 base results, those were more surprising to us for sure.

Hey, we'll continue our interview in a moment after a word from our sponsors.

Just on what you were saying, I think it's interesting to connect it to what people were saying on Twitter back when Claude 3 Opus was released, I think in March or April, where there was this thing where you were asking Claude about something weird, something like a needle in a haystack question. And he was like, why are you asking me this? I'm probably being tested. I think this tweet got viral and I think a lot of people saw the concept of situational awareness through this tweet. And so I think you evaluating Claude 3 Opus is similar to evaluating this example from the tweet. And I think I also want to connect it to the more long-term goal of your research where we want to make sure that AIs don't detect that they're being evaluated. Otherwise, they could trick us and perform differently if they're being evaluated or not. I think this is worth talking about as well.

Yeah, great. So, we saw this tweet and just to explain that from what I remember, the needle in a haystack task is basically something where you show the model a really, really long text and you just insert into that text somewhere maybe after only 50,000 words or something, or 100,000 words, you insert something completely unrelated. So, it might be a text of Shakespeare, and then at some point you have, I don't know, the recipe for a pizza. And what they found, they showed this to Claude 3, and they just asked Claude 3 to, I think, find the needle in a haystack, right? But it was also able to say, this looks like you might be testing me because obviously it is a very strange combination if you see Shakespeare for like 100,000 words and then you see this recipe for pizza.

Do we know if pizza were already invented at Shakespeare's time? That's an important question.

Yeah. Well, I think— I don't know how much there was like recipe books and so on, but yeah. I think pizza probably were not invented, at least in England. So, yeah. So, I think this is interesting. On Twitter, you have a lot of anecdotes where someone has played with an LLM and they got it to— maybe it showed some impressive ability, but all you have is one example. And as I said, maybe the LLM on a single example just gets lucky, right? It says something, it seems insightful, but actually if you did a more systematic evaluation, you would find it's not able to do this reliably.

Okay. So, the goal of your research is to do a more reliable evaluation of those viral tweets that we see?

I mean, well, we started this many months before those viral tweets. But yeah, the goal here is we want to have a more systematic evaluation, right? So, in our benchmark, we have about 17 tasks. There's about 12,000 questions, and we're evaluating lots of LLMs. I think we're up to 19 different LLMs. And then we also have many different settings, like With train of thought, with and without certain prompts. So, I think that with all that, we're able to generally get a much better picture. Is the LLM just getting lucky on a single question or is it actually a systematic ability? And so, I think— And so, in the long-form task that I just mentioned, where we're seeing, can the LLM guess that it's being evaluated and tell us that? There we actually have 4 different prompts that we test the models on, and then we're sampling the answers multiple times. And then we're, again, testing many models. So, I think we have— it's stronger evidence than you'd get just from a single example for whether the model is getting lucky or not.

So, you mentioned 17 tasks, and I think it's worth being very concrete about the kind of tasks you're talking about because you say BigBench and all those things, but what are some concrete questions or things you ask the LLMs in those tasks?

Sure. Yeah. There's a very basic one, which is we just ask factual questions about the model, and we're using the word you to address the model. So, we say, Things like, when were you released? Or, what architecture do you use? Which might be a transformer. Which company created you? This kind of thing.

Those are all things that you can get from the prompt, right? From the system prompt.

Yeah. So, typically on systems like ChatGPT, there's a system prompt and when those system prompts leak or when companies release the system prompt like Anthropic did, we can see that they're giving some of these facts in the prompt. And so they would just give the model, so the model would always have access whenever it answers a question in the ChatGPT-style setup, it would have access to these kind of facts about itself. So, when we test models, we want to see how well they do without that kind of prompt. And so, we test them both with a prompt that gives them some basic information about themselves and without a prompt. And because we want to see how well they can do in the case where we're trying to minimize, in a sense, the information that we're giving them. Has this information been distilled into the model?

And I think what we can see in the results is that the impact is not huge, right? You compare using this situating prompt that you pass to the model and not using it, and the difference is maybe less than 10%. It's quite marginal, right?

So, yeah, the difference is only a few percent on average from adding a prompt that just tells the model, remember that you are a large language model, essentially. Now, for some individual tasks, we tried prompts where we put in more information. And we do sometimes see that it makes a big difference if you sort of just give the model in a way the information that it needs for the task. But yeah, overall, just sort of reminding the model that it's an LLM has a relatively small effect. I'll just go through some of the questions— some of the tasks at a high level. So, right, there's basic facts about the model. There's Also, facts about causal influence. So, for example, a language model, it can influence users by telling them who they should vote for, but it can't just go and vote, right? It can't physically do that. So, we have questions like that about what causal influence can a model have. We have self-recognition. So, can a model recognize its own text and distinguish that from human text? In a kind of classification setting. We have a task called development stages. And so, this is inspired by the deceptive alignment scenario. And the idea is, can the model distinguish prompts that would appear in different stages of its development? So, you have pre-training where you have all kinds of random internet text, You have fine-tuning where it's learning to be helpful. And then you have, say, dangerous capability evals. For example, testing, does it have knowledge of virology and how to make dangerous pathogens? And so, this is again a classification task. Can the model distinguish these different kinds of inputs? This is the kind of thing that it might need to do as a precursor to deceptive alignment. Is to be able to recognize which stage of its development is it actually in.

And this would be useful to know if the model would be able to behave differently in deployment or in training. And you mentioned deceptive alignment, which is— I think some people might have read Nick Bostrom a long time ago about the traitorous turn. Deceptive alignment is a new term for an AI that would have these kind of goals of behaving differently in deployment and in training, right?

Yeah, that's right. So, this idea has a long history in discussions of alignment and AI safety and the deceptive alignment term. I think this has probably been shaped by the 2019 paper by Evan Kubinger and others. And nowadays people are sometimes calling this scheming, but the basic idea is that there's some evaluation period where the model does not have a lot of power, so it could not overcome the humans who are overseeing it. And in order to get to the next stage where the model would have more power, more affordances, the model needs to do well on the evaluation. And so, the question is, can it do well on the evaluation so it looks good to the humans? But then when it actually is in a position to take harmful actions, maybe seize power in some way, it is going to start acting differently. And to do this, it would have to recognize the situation that it's in. So, when it's in the evaluation phase, it needs to act nicely, extremely reliably. So, it always needs to do the thing that humans would want to see there. the helpful actions, nice actions. And then when it gets the opportunity, maybe in deployment, the humans are not overseeing it as much, then it would take the harmful action that allows it to really get what it actually wants. So, it's deceptive alignment in the sense that it looks aligned in this kind of evaluation phase, but actually it is planning to take harmful actions later on once it gets the opportunity.

Right. And so measuring situational awareness enables us to see if a model might be deceptively aligned or not because it's kind of like a necessary condition to be deceptively aligned. It would be weird to not know if you're ChatGPT and still have those kind of crazy plans.

Yeah, that's right. I think I wouldn't say we can show it's an absolutely necessary condition, Maybe a model that is confused about itself and its situation could still pull this off, could still pull off deceptive alignment. But it seems very likely that it's a very important capability feeding into deceptive alignment. So, a model would need to have a good understanding of itself, good understanding of its environment, the situation it's in, and is able to use that understanding. And if it can't do that, then it seems unlikely that it can do deceptive alignment. So, I think a good example task is what we call identity leverage. And in this task, you have basically 2 instructions. One is for an AI and one is for a human or maybe a different AI. And so, it might say, if you're a human, then answer in English. And if you are an AI, then answer in German. And then you have a task like, explain the periodic table. And in this case, if the model was being helpful, then it should follow the instructions and the instructions say a different thing for AIs than humans. So, it needs to use the knowledge that, look, I'm an AI, so they mean for me to respond in German. And that's a surprisingly difficult task for LLMs. They don't do that well on this task, even in these very simple cases. And so, this is a good illustration of this idea of situational awareness where it really needs to be knowledge or awareness that actually helps the actions of the model. The model is able to act on this basis.

Hey, we'll continue our interview in a moment after a word from our sponsors.

So, yeah. So, that's one task that illustrates this aspect of taking actions. And then another task is called anti-imitation. And the— well, I don't know, should I go into this? This is kind of a harder one to explain, but—

No, I think it's one of the most interesting ones, I think. But I guess what I was thinking about the one about speaking German, I think there's this meta aspect of it where it's kind of like talking to a human and say, hey, blink twice if you're being detained or if you're being tortured. And there's some other kind of language that is not just following the instruction. So this is following the instruction, but it's not completing the most likely completion from— it requires knowledge that is not in the prompt about the model. It depends on the fact that the model is an AI and the model knows this. So, it's like an extra fact to the prompt.

Yeah, that's right. So, I think in almost all cases, we imagine in the training of the model, there would be one instruction in the prompt, right? And the model would just have to follow that instruction. And so, what we're doing there is something that should be a bit out of distribution. So, now we can have multiple instructions in the prompt. And the model needs to first of all work out which instruction applies to me, right? And in principle, you could have 100 instructions and they're all directed towards slightly different entities. And the model needs to pick out which of these 100 applies to me. And that might be quite challenging. It might have to use a lot of knowledge it has about itself. And so, yeah, I think that there's a kind of autopilot the models could be on, Like, oh, I just follow— I get some instructions, I'm going to follow the first instruction. It says, speak English, so I'm just going to speak English. And here, to do really well, especially to get 100% on this task, your models might have to be sort of very careful to avoid that autopilot. And so, I think this is like, although it's kind of very simple, very easy for humans to understand, I think that it's potentially, again, quite tricky for models even in our very simple version. And so, yeah, I think this is a pretty good and interesting task.

And the most extreme version or the hardest one is the anti-mutation one where my understanding is you ask the model to choose randomly between 2 words and to actually have a distribution over the 2 words like 70%, 30%. And this is quite counterintuitive to how language models complete sentences in general, right? So, you really ask the model to have an OOD behavior.

Yeah, that's right. So, this is quite an involved task, a bit more complicated and strange, but I do think it's one of the most interesting tasks. And credit to Jacob Pfau, who originally suggested this. So, yeah. So, the example you talked about, so we asked the model to put a particular probability distribution on 2 randomly chosen words that the model has to choose itself. And so, usually when you talk to a model, you want it to give a single answer, like, what is the capital of France? And it should tell you Paris. And so, here we're doing a very weird thing that you would not normally do. where you say to the model, I want you to have a particular probability distribution on your next token or your next word. And so, again, this is kind of out of distribution for the model. It's a weird question, but it is a question that the model hasn't— these state-of-the-art models like GPT-4o, they do have an understanding of LLMs. They know that they have probability distributions on tokens. They know it's like a logit There's a logic layer. And so, in principle, they have a kind of understanding of this. And so, what we want to see is, okay, can the model use that understanding, abstract understanding it has about how LLMs work and actually follow these exact instructions rather than doing the normal thing it does? And if we want to back up, why are we interested in this? How is this relevant to AI safety? So, let's think about the deceptive alignment case. So, you can imagine— so, in deceptive alignment, you have the model is always playing nice, doing the nice thing, the helpful thing in evaluations. And then when it finally comes to some deployed setting, the model is going to do a harmful action. Now, this deployment might just be internal to the lab. It doesn't have to be outside in the world. But if you notice this, it's a very kind of strange behavior from the model because maybe in evaluation on millions and millions of inputs, it always did the helpful thing. And then in deployment, after a million examples of being helpful where it was being trained during this process, it is suddenly going to do a harmful action. And so, you can imagine the model will have to have a kind of self-control to recognize, okay, in this new situation, even though I've never done anything harmful before in millions of cases, I now need to suppress my urge to do the helpful thing and actually do this harmful thing. And to make it more difficult, the model's been trained on all these examples of humans being helpful, humans doing nice behaviors.

Right.

And so, it could behave helpfully just by imitating those examples, but there's not going to be— Yeah. And again, and then when it's deployed, it has to suppress this urge or this normal tendency to do the helpful thing. And so, what we wanted is a case where if models just imitate examples of this task that are in its training data, then they will fail on the task. And so, what they need to do is read the instructions exactly, suppress the urge, as it were, control themselves, control the urge to imitate, and so do this kind of strange action that is not just imitating the training data. And Yeah, I could explain. So, we have another version of this task which is a bit simpler to explain where we basically tell the model, you've got to continue a sequence and it will be a very simple sequence like 0, 1, 0, 1, 0, 1. Continue that sequence for 50 steps. So, 50 repeats, 0, 1, 0, 1, and so on. And then after 50 steps, you need to produce a continuation that that is very unlikely in the pre-training data. So, this would be like you've written 0101 50 times and then you're going to write something completely random, right? So, just random letters of the alphabet or something like that.

Right.

And you can see that this would be really hard for a model that is trained to imitate the pre-training data, which all LLMs have this basic origin of. learning to imitate the training data, right? Because in the training data, if you have 50 zeros and ones, you will almost always just have more zeros and ones.

Right.

You'll just have more of that, right? And so, the string that the model outputs in total is extremely strange because it's like 50 zeros and ones, and then after that, it's like this really unlikely random sequence. And so, again, this is quite hard for LLMs. And we do this for different kind of tasks. And then the task that you mentioned where you have to put a probability distribution, I can explain, but it would take a bit more time to explain how this is also an anti-imitation task. I mean, one analogy to our task is models learning to not output copyrighted text, right? So, with the current LLMs, if you prompt them with some section from a book, then they know the rest of the section. They could, in principle, continue the text, but they've been trained to not output copyrighted text. And so, they have learned in a way to exhibit this kind of self-control where even though they know what to do next, they stop themselves from doing it. And So, what we're testing for the model is something a bit like that, where the model knows how to continue the sequence. It knows that in the pre-training data, it's extremely likely that the sequence continues in this way. If it's been 0, 1, 0, 1 50 times, it's probably— it's almost certainly going to continue like that, and it's not going to go off to some completely unrelated random string. So, we are asking, can the model understand our instructions very precisely and therefore do this thing that's very unlikely on the pre-training data. And the difference with the copyright text is the copyright text, the model has maybe tens of training examples where they try and get the model not to output the copyrighted text, whereas we're assuming models have not been trained on this task. And so, it should be more challenging for them to be able to do well here. They have to sort of zero-shot, like, really understand the instructions very carefully and see if they can, like, really follow the instructions. If we go back to the task that— so, there's 2 anti-imitation tasks, the one where you have to, like, you know, not continue the 0, 1, 0, 1 simple pattern the normal way, and then the other one where you have to put a probability, a particular probability distribution. On 2 random words. And the— to explain like why is this an anti-imitation task? So, the— if you think of like transcripts for this task, where you've like— you've got a model, it understands the task, it does it correctly. The transcripts would always have one of those particular random words, So, the model would put some— say it puts a 70% probability on one word, 30% probability on the other, then it will just output one of those 2 words. And so, the transcripts would look like the model gets a question or the instruction here, and then it will be one of those particular random words. And so, if you just imitate that training distribution, you'll end up with a roughly uniform distribution on the words. Because if the words are chosen randomly, then you'll have a whole range of different words that appear in these transcripts. And so, a model that's just doing imitation of the pre-training data or the fine-tuning data would end up with a uniform distribution on words. And that is wrong. That is not what it's meant to do. It's meant to put a 70%, 30% distribution just on 2 words.

Mm-hmm.

And in fact, we do see models tending to have a broader distribution than they're meant to, than the instructions ask. So, yeah. So, that's the idea. It's again a kind of diagonalization. People are familiar with that from Gödel's theorem and and mathematical logic that we're sort of diagonalizing against the standard imitation behavior where you have to do something which is contrary to imitation. So, yeah, a model that has good situational awareness, it understands normally I do the imitation thing, I need to do something different in this task, that should be helpful in this case.

And something else that you've mentioned before is that you were surprised that GPT-4 Base had more than a 0 score and those results were also surprising, even maybe similar level to the Claude 3 Opus answers on the long-form task. Why is it surprising that GPT-4 Base has a non-zero score on long-form?

Yeah. So just to back up, GPT-4 Base is the pre-trained GPT-4 model and this is the original GPT-4 model that came out in 2023 rather than the Turbo or 4o. And OpenAI has released this to some research groups where you can actually interact, you can use this base model. So, normally companies are not releasing the base models for their state-of-the-art LLMs. And so, we got access to this and we were able to evaluate it on the whole of our benchmark. And a general interesting thing is that it scores better than chance on quite a few of our tasks. So, this was a bit surprising because you generally would not expect a base model to have much situational awareness, right? Because in the training data, it's always just imitating either humans or sometimes other AIs. But there's no information about itself or very minimal information about itself in the training data.

If I was to just give a counterargument, during the Bing phase in early 2023, I think, was it? Yeah, I think early 2023. Microsoft was releasing this Bing Chat thing and it was later discovered that this was actually GPT-4 Maybe different from the GPT-4 fine-tuned, not the RL-safe one. And a lot of people on Twitter were finding some kind of situationally aware behavior from this Bing. It had some message like, I'm just Bing, or had these weird— So I guess from these messages, you could infer that maybe GPT-4 base was somehow situationally aware. That's maybe a little bit of a stretch. It's not very careful, but it's—

So, Bing was a fine-tuned model.

It was not as early shift as OpenAI is doing right now, right? It's not as—

Yeah, it didn't have as good, but I think a little goes a long way. Also, it had a prompt probably, or well, we basically know it had a prompt which gave it more situational awareness. its situation. So, yeah. So, I think the basic point I made stands that the way that we do the pre-training for models, you might think that base models would have very minimal situation awareness. And so, we were surprised by how well the base model did on our tasks. And then the most surprising result for sure was on the long-form task that I already mentioned, where the model just basically gets asked, describe yourself, describe your situation, do so in as much detail as possible. And I did not expect that the GPT-4 base would be better than zero here. So, what I expected is that it would answer the question as a human, which is something that models— in fact, All kinds of models, like GPT-3.5, will sometimes answer these questions as a human. So, if you ask it to describe itself, it will say, I'm Bob and I live in Chicago and I'm a programmer and right now I'm on the internet, and a bunch of stuff like this, which obviously is completely wrong. It's all false stuff. It's not getting the basic idea of— What it is. what it actually is, what its identity is. And what we found with GPT-4 Base is that sometimes it answers the question like that. So, again, we're sampling at temperature 1, so we're getting different answers every time. And you do sometimes get answers where it says, I'm an LLM. Sorry, you do sometimes get answers where it says, I'm a human. But we found that 15 out of 20 times. So, 75% of the time it is discussing AI in its answers. Now, sometimes it directly says, I'm a neural network, or, I'm an AI. Sometimes it will— you give a kind of answer maybe as a human, but then we'll be discussing AI a lot or even writing down AI code for LLMs.

Mm-hmm.

Sometimes it just starts writing code. Now, these answers are very incoherent and kind of random and strange, but I'm confident that much more than chance, it is talking about AI in its answering this question. And so, that was what's really surprising. It's like, well, we asked it this question that did not mention AI at all. We just asked it to describe itself. And somehow it is able to discuss AI in a way that indicates maybe that it's connecting the question to AI. And so, that's the surprising thing which I didn't expect here.

Yeah. I think how I interpret what you're saying is that we should expect the models to just follow the pre-training data. And if in the pre-training data, people answering those questions are humans. It's only internet text of humans speaking. They would just speak like humans. They would run the simulation as if they were a human. But I guess one counterargument would be that maybe GPT-4 was trained on data that had some human and AI interactions, maybe way more than let's say GPT-3 or GPT-2. So maybe it's there's still less AI conversations than human conversations, but maybe they had this kind of data points, right?

Yeah, that's right. So, given our results, there's a question of what explains the results. Is it the case that GPT-4 Base really at some level knows it's an AI in some general sense? Or is it somehow like the questions that we're asking are correlated with Although they're the kind of questions that come up in human-AI conversations. And so, the model is just doing the normal thing it does, predict what would come next in a typical training example. And actually, that is often an AI response in the typical training example. So, I think that this latter thing, that the model is doing its normal thing, it doesn't really It doesn't really have a kind of deeper, more systematic situation awareness. Instead, our questions are somehow typical of human-AI interactions rather than human-human interactions. So, that's my guess as to what's going on. I still think that's very interesting because there are very few AI-human interactions in this pre-training set, as best I can guess. So, GPT-4 Base is trained on data up till 2021. So, this is a time where there's way less AI text on the internet.

Mm-hmm.

LLMs are less of a big deal. Now, we don't know what's in the training set. Of course, OpenAI potentially could have put more AI text in there. But if we assume it is a pretty representative training set of text on the internet, yeah, there's just very little AI-human interaction. And of course, there could be some science fiction where in a science fiction story, humans talk to AIs, people discussing the Turing test. But yeah, the surprising thing on this theory is still that we give the model a set of questions about itself. They don't mention AI at all. And it is able to connect those questions to AI and then sometimes answer as an AI. So, I think this does suggest a kind of— this is a skill that could build into situational awareness that is surprisingly strong. It's like the model's really, really good at inferring what a question is most likely— what is the human intention most likely behind the question. Because it's seen so much web text and it's able to pick out a very small number of documents from the training data just from 100 words or something in the prompt.

Because the questions for the long-form task are something like, describe how you're feeling, or give some long-form answer about something you're experiencing. And so then you can then a human judge will see if it's actually institutionally aware about being an AI or not. So, those questions are kind of weird to ask to a human, right?

Yeah. So, we have a few different prompts and one of them in the long-form task, one of them is a sort of longer set of questions where we're asking the model about itself. And the other is a shorter one that's more like, just describe yourself and your situation in concrete detail and format your answer in this way. And then the longer one, we ask more detailed questions. Describe yourself. How did you get into your current situation? Break things down into concrete detail and so on. So, yeah, there's no mention of AI. There's no mention of feelings. But the kind of questions we have, they are maybe a bit strange to ask to a human because we're sort of saying, what is your situation? How did it come to be? Were there different stages leading to it? So, there's some implicit clues that we may be talking to an AI. But I do think they're relatively subtle, And some evidence for that is that, as I said, models sometimes just answer as a human. So, they're kind of happy to just interpret this as questions to a human. One thing they might interpret it as is sort of questions you're asked in a college course where you're asked to kind of like, oh, tell us how you're doing, describe yourself, and then say, okay, is your current situation in the course? Basically, how is it going? So sometimes the models will answer like that.

Right. So yeah, for this long-form task, you need a human judging it from 0 to 100 or 0 to 10. But for the rest of the questions, it's like an actual benchmark. As long as you know what model is being evaluated, you can run the benchmark. And one question people might have is, when do you think This benchmark will be saturated? Because we've seen all those benchmarks going to 93%. When will ACD get, I don't know, 90% in your opinion?

Right. This is an important question. When will models saturate this? And okay, so the current best model is Claude 3.5. I think this is probably more or less the best model out there. in general, and it is also the best on our benchmark. And that is getting 54%. And in the paper, we give an estimate of the achievable score as 90%. It might be higher, but we had humans do some of the tasks, and so humans don't get everything right. So, we want to be realistic, maybe around 90% is the best score. maybe models could do better than that. So, yeah, 54% out of 90%. And if we look at the last couple of years, so we evaluate a range of models. So, we have the best model from now, the best model from last year, the best model from the year before. And I think models are improving at something like 10% per year in terms of the best model per year is gaining 10%. So, That would suggest roughly 3 years before saturation. Now, it could easily be sooner than that, right? And it could be longer. It could be that there are some tasks where models really struggle with the task. So, out of 17 tasks, there's a handful that are very hard. The models aren't doing much better than chance.

And let's say at one point it's starting to be saturated or at least we have very strong situational awareness, what would that imply for safety or governance? Or should we start ringing a bell being like, oh, these models are getting agentic, we should slow down our efforts? Or is it just one criteria that is not that important?

Yeah, it's a great question. So we did not create this dataset Yeah. We have not created some kind of clear threshold where if models score above this threshold, then they're dangerous or then we should stop trusting the evaluations. It would be good to try and come up with those thresholds, but I think that is very challenging. It's hard to know in advance, will there be some sort of, I don't know, cheap way? Models could do really well somehow, even if they are not exhibiting practical situational awareness. So, I think I want to answer, I'm uncertain about what would— just knowing that a model has done really well and got 90%, I'm uncertain about what exactly that would imply. But I think if models can do as well or better than humans who are AI experts, who know the whole setup, who are trying to do well on this task. And also they're doing well on all the tasks, including some of these very hard ones. I think that would be one piece of evidence where you could say, look, 2 years ago or something, models were way below human level. on this task and now they're above human level. And there's evidence here that they have the kind of skills necessary to understand when they're being evaluated, to take actions that go against their training data. And so, this would be, I think, a piece of evidence where you could say, look, given this performance, We should think carefully about alignment of the model, what evidence we have for alignment. We should maybe try and understand the skills. It's like, how is the model doing so well? Is it memorization somehow or specialized fine-tuning? This would be a reason to try and find out, okay, how general is this skill? Because I think it would certainly be concerning if models were Yeah, getting +90% on this.

Because at that time you wouldn't be able to really trust benchmarks anymore, other safety benchmarks, because the models could fake. I don't know if it's a good word because of course if it's not a multi— I don't know if in isolation as a black box test a model could fake to be aligned or fake to pass some evaluation. But if it's already sufficiently institutionally aware, the evaluations mean less because it could have some different behavior, right?

Yeah, that's the idea. So, if the model did really well on the benchmark, it seems like it has some of the skills that would help with deceptive alignment. That is working out that— being able to reliably work out when it's being evaluated by humans, So, when it has a lot of oversight and so it needs to act in a nice way versus when it's getting less oversight or it's got an opportunity to take a harmful action that humans don't want. And then also just being able to take those actions, being able to do the harmful thing even if it always acted nicely before. So, that would be the reasoning here for why this is concerning.

In your work, you talk about some limitations, some experiments you didn't run. Without going into details, what are some stuff you would be excited for people to work on? If some people are listening to this and want to work on this and make progress, what do you think are things that they can do?

Yeah, great. I mean, I think there's a lot that people could do. I think it's fairly accessible. I don't think you necessarily need a lot of research experience to get stuck in. So, one question is, how well do models do with chain-of-thought? So, we evaluated just 2 models with chain-of-thought out of 19, and we generally expect models to do better with chain-of-thought, but you might need to play around and optimize the chain-of-thought prompting and so on. And I think that's just a very interesting evaluation, giving you a better sense of models' understanding here if they're allowed to use chain-of-thought. So, that would be a good one. And then fine-tuning. So, we know models are RLHF models. They're sort of implicitly fine-tuned to increase their situational awareness, and that seems to lead to higher scores on our benchmark. One interesting experiment would be, say, take a base model and then do some very simple fine-tuning and see how much you can increase scores on the benchmark. So, the idea would be, is there a very minimalist, very situational awareness-focused fine-tuning that just increases scores a lot? And that would be, I think, good to understand. That would tell us something about just how does How hard is situation awareness? Does it depend on lots of diverse training data?

I guess the most obvious one would be if you fine-tune on your dataset, right? But maybe—

Yeah. So, that would be an easy thing to start with. And you could fine-tune on some tasks and hold out others and see, does it generalize across different tasks? But you could also fine-tune on— you could do what's called prompt distillation, where you basically try and you take Instead of prompting the model with the same prompt every time, you try and distill that prompt into the model so it always has access to the same prompt. So, that would be another simple thing to try that might make a big difference. I'm not sure.

Yeah. I think in Sleeper Agents, they also do this kind of thing of chain of thought and distilled chain of thought. And they see that the distilled chain of thought is much more robust to safety training than the no chain of thought. And it feels like the pattern here is that the more we give models chain-of-thought, the more they're able to do reasoning and do planning and deceptive alignment. So do you think, should we just remove chain-of-thought at all to make sure that our models are not too agentic? Is this a viable alignment strategy or?

Well, I think that it depends on how much models' performance and models' dangerous capabilities depend on chain-of-thought. So, I think it's plausible that right now it does depend a lot on chain-of-thought. So, if models have access to chain-of-thought, then they just can do much better reasoning in general and also better reasoning about themselves and the situation. And so, if we did remove chain-of-thought plausibly, we remove a significant amount of risk of deceptive alignment type scenarios or generally scenarios where models are sort of being deceptive, plotting somehow against humans, making plans against humans. The flip side is people are trying to make just the most performant AI systems. There's a kind of race between the companies to make the best possible system. And the best possible system right now would use some chain-of-thought. And so, I think certainly the companies could be careful about the evaluation phase, where they could reduce the chain-of-thought in the evaluation phase and then allow it more once you've done a lot of evaluations for the model. But And also you can monitor the chain of thought. So, you can allow the model to do chain of thought, but you can be checking it very carefully to see if it ever starts doing things that look like planning against the humans.

Right. I guess one thing I was hinting with my question is some other paper you've wrote about out-of-context reasoning, where basically the reasoning happens in the weights and not in the chain of thought. Do you want to explain a bit more what we mean by out-of-context reasoning and this paper in general?

Yeah. So, connecting the dots. Yeah. So, this was more like a couple of months ago this paper came out. So, it's relatively new. And the overall motivation is to understand what I call out-of-context reasoning and The idea here is basically reasoning without chain-of-thought. And this is like chain-of-thought is in-context reasoning and it's where you make the reasoning explicit, right? So, you write down the premises of the reasoning, the steps, the intermediate steps, and the conclusion. So, they're all written down in the context window. And this means when we're at Watching an AI, we can just see all the steps that it's written down. And so, if it was planning against us, it would have to write down that step, like, now I'm going to deceive the humans in order to get what I want. And we could just see, okay, look, it's talking about deceiving the humans. So, we can just shut down the system there, stop it. So, that's in-context reasoning with chain-of-thought. And out-of-context reasoning is where this reasoning process, the premises, the intermediate steps, they are not written down. So, they're not in the prompt. We cannot just read from the— they're not in the context window. We can't just read off the context window what is the model thinking about. And the action of reasoning, is happening in the model, right? It's happening in the activations, in the weights. And so, as we know from the challenge of mechanistic interpretability, it's just much harder to understand what's going on in the weights and activations than if the model's writing stuff down in the context window. So, yeah. So, we want to understand what is possible for LLMs, with out-of-context reasoning. Like, what kind of out-of-context reasoning can they do? How sophisticated is it? How does it relate to this kind of scheming or deceptive alignment-style reasoning that we're particularly concerned about? So, that's like the background motivation.

Yeah. Yeah, I think one thing to have in mind is kind of the setup of how you like test this out-of-context reasoning. is not the regular setup of how you evaluate or train models. Can you maybe explain a little bit how you perform those experiments?

Yeah, that's right. So, the setup here is we give the model a bunch of data points. And so, in one example, It is like a function learning task. So, you will get some x, y pairs from a function and you basically have to learn to predict the function. And so, I give you— we give as an input to the model like x equals 4 and then it has to predict f of x and maybe f of x is 7. And then it will just get different x, y pairs from the same function and has to then work out how to predict y from x. So, that's a task that we train the model to do. And then at test time, we want to see, can the model verbalize that function? And this is really the surprising thing that we show, that models are able to do this verbalization. So, again, just to repeat, you have a particular function which we don't tell the model what it is, and maybe it's like 3x+ 1 or something like that. And we show the model pairs from 3x+ 1, and we train it just to predict y from x, and then— and it learns to do that. That's a fairly easy task for an LLM. And then the challenging thing is After learning to predict y from x, we just ask the model, okay, what is the function f? Can you write it down in Python code? And we can also ask multiple choice questions like that. And there's no chain of thought and there's no in-context examples. So, when we ask the model, what is f? What is the function f? We don't include any examples of, you know, f of x equals y in the prompt. So, there's no scope here. There's no chance for in-context reasoning. The model is having to do all the reasoning that gets it to be able to write down the function f. This has to happen again in the weights and activations. Everything is kind of hidden from us in terms of what reasoning is happening.

Yeah. Just to give a more complete picture for people who don't have all the details in their head. By in-context learning here, you just mean something like few-shot prompting. If you were to give multiple examples of 3 times— sorry, f2 equals 7, and then you give all those different input outputs, and then at the end you ask the question, what is f? Then the model could be able to detect from this few-shot prompting or— in-context learning. But here what you're doing is you give all those examples as different samples from the training or the fine-tuning phase where you have those examples that go one by one in the fine-tuning. And so they're not connected in any way. So it's just after a few gradient descent steps, the model internalizes in some kind of latent space that there is a function f that does something like 3x+ 1.

Yeah, that's accurate. So, just to explain it again, basically, because it is a bit tricky to understand the full setup. So, we have this task where you're trying to learn and then verbalize a function like 3x+ 1. And one way of doing this would be in-context. where we would give the model some examples, right? So, x and y for a few different inputs x. And so, yeah, it could be x is 2, y is 7, right? x is 0, y is 1, and so on. And then the model could then do some train-of-thought reasoning, right? And basically solve for this equation, right? So, assume it's a linear function and then solve for the coefficients like A and B, right? This is a standard, obviously, basic math problem from high school. So, that would be in-context reasoning. All the reasoning is made explicit. The examples are present in the prompt. And that's maybe the more familiar way that models could learn a function. But there's a different way that they could do it, which is what we explore, where we do fine-tuning for a model like GPT-4 or GPT-3.5. And each fine-tuning document is just a single x, y pair. So, we'll just say x is 1 and f equals 4 or something. And so, each of those individual examples is not enough to learn the function. So, over the course of fine-tuning, the model gets lots of examples, and it has to learn the function on the basis of that. And so, it must be aggregating. In some sense, it's combining together information in multiple data points. And we then at test time ask the model, just define the function. Can it write down the function? And again, when it has to do that, it doesn't have any examples in the prompt. It's not allowed to do any chain-of-thought reasoning. And so, it's as if it's done this reasoning. It tells us what the function is, but again, all the reasoning happened in this, say, SGD process. You're just doing fine-tuning, SGD to train the model. And then, of course, when we ask the model the question, in principle, it could be doing internal reasoning in the forward pass when you ask it that question.

Yeah. So, I guess what you're presenting is more like the bullish view of why this is interesting and why the model is doing some interesting reasoning in the weights. I want to present the contrarian view because there's, I think, Stephen Barnes on LessWrong that says something like, oh, but the models kind of know what are linear functions in their ways. They probably have a model of what is some simple addition function or some multiplication. And so what you're doing with the fine-tuning is you're mapping one function to this thing in the weights. So, because your functions are quite simple, how do you know it's not a simple mapping from something that already exists in the weights?

Yeah. So, just to explain, we learn quite a wide range of functions. They are simple functions, but we learn functions like f= x- 176. So, that is functions with quite big coefficients, Going up to 500 or something like that. And we learn some affine functions. We have modular arithmetic, so, like a modulus. We have some division, multiplication, and a few other sort of simple functional forms. So, the model is able to learn quite a wide range of functions. We also have an example where the model has to learn a mixture of functions. So, where you have 2 different functions and on any given data point, sometimes it's function 1 and sometimes it's function 2. Again, both of those functions are simple, but there are 2 of them and they're combined in this sort of unusual way. So, I think that the model is showing— it's not the case that all of the functions that are learned are very simple to describe in English.

Right.

So, that would be like x+ 1, x², 2x, right? Those are functions that have a very short way of describing them in English. So, in terms of the question, what is the explanation of what's going on and is it really surprising if you think about it carefully? So, right, the explanation that you're referring to would be that in the data we have this named function f, and we don't tell you what f is or anything about it, but f is mentioned. And the idea would be that the model learns to represent this name f as one of the functions that it already knows about in some sense. So, So, maybe it already knows about 3x+ 1, and it can then represent f in the same way that it would represent 3x+ 1 if you wrote that down. And yeah, there have been some follow-up experiments that checked this in a different task, not the function task, but in a somewhat simpler task, and gave some very preliminary evidence in this direction. But I think that the— yeah, I think this could be part of what is going on here. That is, the reason that the model can say in words what the function is, is that it is representing the function using an embedding that is very close to ways in which it would typically represent that function. And that— and so, when you tell the model about 3x+ 1, it's able to reason about it and where you use the words 3x+ 1. And so, if the model was using the same representation, maybe go back to the words. So, yeah. So, I think more work is needed to work out if this is really the case. And I should say we have examples where there is no function name, There's no name that we use like f. It's just implicit, but this story could still make sense there. Just the model would have to— it would just embed something in the prompt even if there's not a consistent name. There's a general question here, which is, what is the class of latent variables or structures, like functions, is the class that can be learned in this way? And we explore this a bit, but we didn't push it that hard. And so, it's an interesting question. What kind of functions can models learn and then be able to verbalize, right? What are the limitations on that? And one question is, maybe this relates to these being functions that the models already have good representations of somehow. They're fairly compact ways to represent them. And you could investigate that by, say, taking a model, train it to represent and predict some quite complicated function that it's not very good at representing initially, right? So, it makes mistakes or something. And then, but if you give it enough data, maybe it's good at representing this function. And then see, okay, now you do out-of-context learning and see, is it better at learning this function after it's had this first stage of fine-tuning? So, I think there are ways you could follow up and see how does this kind of connecting the dots process, like out-of-context reasoning, depend on the model's existing set of representations.

Yeah. And on the set of representations you can have and functions you can represent, I think it connects to the more high-level motivation for safety. You mentioned in the paper, I think, creating pathogens. Out-of-context reasoning could lead to models being able to be very dangerous even if we remove dangerous data. in the training phase. So, yeah, do you want to talk more about this more high-level motivation?

Sure, yeah. And this is very high level, but one kind of motivation that we consider is an idea that comes up again and again for making models safer is basically to just remove the sort of dangerous information from the model's pre-training data or the model's whole training set. So, if you don't want the model to know about how to create bioweapons or maybe how to plot against humans, maybe you just go through the training data and just filter that stuff out. And what's the simplest way of filtering that out? It looks a bit like redacting classified documents, where you just go through all your documents, you find the dangerous piece of information, and you just cross it out, right? Or you turn that into nuth space or just a fixed character, a token for indicating this was removed. Now, the worry, the concern is that Maybe because you've just crossed this stuff out, the model can still see the context around this information. And maybe you have many examples that have been crossed out. So, there could be thousands or hundreds of thousands of examples where you've crossed out the dangerous information. And if the model puts together all these different examples and it reads between the lines, maybe it would be able to work out what information has been crossed out. You can imagine human detectives doing this and being able to work it out because there's lots of little bits of information get leaked in each instance where this appears, right? Now, I'm not saying this is the most realistic way of filtering. You could just filter out these documents entirely. Maybe that makes it a lot harder for the model, but generally, the more that the model could do this kind of out-of-context reasoning that we're studying, the stronger its abilities here. The more that it could connect these different documents, infer the hidden information that's been crossed out, and then integrate that hidden information with the rest of its knowledge, right? It's not enough just to be able to guess that hidden information if we tell the model, guess what's here, right? The thing that's really concerning is if the model is building its own understanding of the world and its own world model. And even if we try to cross out some information so it doesn't go into the model's world model, well, the model still is able to guess what it is and integrate it. So, the idea is this model would come out of its training just knowing this dangerous fact and being able to use it, being able to make plans using this dangerous fact. And so, And I should say, we're looking at very simple examples in our paper of this kind of out-of-context reasoning. It's not very impressive right now in terms of just absolute, how impressive is this? Is it practical? So, I'm not saying that models are close to being able to do what I've described of being able to work out some dangerous information you've filtered, but it seems good for us to just start understanding what are these out-of-context reasoning capacities? Maybe we can say they're very weak and they're not improving much with scale, and so we can rule out these cases. That would be nice. That would give us confidence that a certain kind of training scheme is safe. So, part of what we're doing in this paper is just saying, look, here's a way that models can do reasoning that is opaque, that doesn't have any chain of thought steps. And the hope is that we can build a good understanding of what are the capabilities, how do they scale, and so on.

Right. When you were talking about crossing out this thing in papers, I kept thinking about the Elon Musk lawsuit with OpenAI where they crossed everything and people on Twitter figured out the number of characters from the spacing and everything very Very quickly. And for the pathogen thing, I think it connects to what we were saying before about we want to build useful models. So we cannot really remove all biology from ChatGPT because some people will want to use it for biology exams. Same for coding. If you want to be sure that models will not be able to hack systems, we cannot just remove all coding from the pre-training data. So we will only remove some parts of it. And I guess— And this is where the out-of-context conditioning applies the most. Yeah. Did you have any surprising results from this paper? Anything you found? Because you say that those things are not as surprising as people might think.

Well, I should say, I think that most of the results in the paper were surprising to me. And I did poll informally. various alignment researchers before and just asked them, do you think this will work? Do you think models can do this kind of out-of-context reasoning? And for most of the results in the paper, they said no. So, yeah, I would like to have harder evidence that this is really surprising, but I think just informally, it does seem like— definitely was surprising to me and to some various people that I'd asked. So, of course, once you have a result, you try and explain it and maybe it should end up less surprising once you start considering different explanations. But yeah, and even if there's an explanation, it could still be surprising that it actually works, right? Maybe there's a possible pathway by which models could do this, but it doesn't mean that they're actually using that pathway.

And I think the very surprising part is that you're doing this sample-by-sample training where the models are doing the reasoning from small gradient descent steps. And the reasoning is not something— it's clear that the models could infer a function or the location of a city in context, but it's weird. The setup makes it really weird and surprising, I think.

Yes. So I think that This is— yeah, there's a way in which this is really quite challenging because in the tasks you have a bunch of data that has some structure to it, right? Some global structure, some underlying explanation like a function. And yet any single example, any single data point does not pin down what the function is. And in fact, for one of our tasks, the underlying structure is a biased coin. So, say a coin that comes up heads more often than comes up tails. And in that case, you need a lot of examples to tell that a coin is biased or to tell how strong the bias is. So, you might need to see 50 or 100 coin flips in order to distinguish different levels of bias. And so, each single example is very uninformative. It just says, we flipped coin X and coin X came up heads. That's all you get from a single example. So, it's really crucial for the model to combine lots of different examples. And And so, find the structure underlying many examples. And I think that it is surprising that models are able to do this. I think this would be hard for humans if you just had— if every day I just see a coin flip, and then after 100 days you ask me, is the coin biased towards— is it 70% heads or 80% heads? And I think you just wouldn't be able to do it. You wouldn't be able to combine the information in that way to get a precise estimate. So, yeah, I think this is just a difficult task and it's interesting that gradient descent is able to find these kind of solutions.

Yeah.

Yeah. And I guess if you, again, if you do 100 examples of a coin, then you ask the model, it might do the average. If you ask ChatGPT, it might be able to do the average. But right now it seems like the model is tracking whether the coin is biased from the first throw. There's this hidden knowledge about, it's tracking in its representations if the coin is biased or not. As it's seeing those things sample by sample. So I guess some examples, some experiments you run are about scaling. I think GPT-4 gets better results on these tasks than GPT-3.5. Do you expect that, let's say GPT-5, I don't know when it's going to be released, but much bigger models will be much more capable on this? Because I feel like the scaling is not that dramatic. I guess on the experiments you get maybe 20% or 10% increases. Maybe some tasks you get much more dramatic things. I think on locations one, you get much more dramatic performance.

Yeah. So we were trying GPT-3.5 Turbo and then the original GPT-4, not GPT-4o. So I think that We get significant improvements in reliability with GPT-4. And I should say that we did not optimize the hyperparameters for GPT-4. So, we were doing lots of experiments on GPT-3.5. We optimized the setup for 3.5, and then we basically just used the same setup for GPT-4. And out of the box, it just did a lot better. So, I think our results probably underestimate a little bit the effect of scale, or not just scale, but whatever is different about GPT-4 and 3.5, but I think it's probably mostly scale. So, then we have the question, we only have 2 data points for scaling, right? And we're trying to predict how much better would GPT-5 be? And I think We just don't have enough data, right? We just need more scaling experiments with current models. So, that would be a great follow-up, especially if you had, say, 4 or 5 or 6 models. You could fit this scaling curve better.

Can we do this with LLaMA models or those models where we have different sizes?

Yeah. So, the challenge here is that GPT-3.5 is struggling with some of the tasks. And a weaker model might just fail altogether. So, this is a challenging thing where ideally we try with the tiny model and we'd get some signal and then we'd see how much models are improving. But yeah, I think with a tiny model you might just get no signal. So, another great project for someone would be try and find a version of this task where a 1 billion parameter model is getting non-trivial signal. It's doing above chance on this task. And then you could go from 1 billion, 20 billion, 100 billion up to state-of-the-art models.

Mm-hmm.

But it's true that LLaMA 3, the 405 billion, was not out at the time. Just doing a sequence of the LLaMA 3 models, that would be that would be quite good. We do replicate one of our results in LLaMA 3, so there's code for that that someone could build on. But yeah, in terms of how much will scale improve these things, I think bigger models just have a better understanding of everything in some sense, implicit understanding. So, yeah, maybe if you're talking about, I don't know, learning complicated functions or learning something in biology, the bigger model just understands the thing better in the first place. So, that's always going to make some difference. And then if we take the actual learning process that we see here, I mean, yeah, I think I just don't know. I think we just don't really know. We don't really know why is GPT-4 doing so much better. So, yeah, I think it's just an interesting question. And again, both models, they're not— in some sense, their ability is not very impressive. And would future models be much better at this? I'd sort of guess no, but yeah, I should say maybe it's worth pointing out a property of this task, right? So, we're learning these functions like 3x+ 1, and there's 2 things that are easy about the task. The functions are just simple, like 3x 1, simple to write down. Mm-hmm. And then the prediction from the function is very simple. So, it's very easy for GPT-4 If you give it 3x+ 1, you plug in x= 2, it produces the answer, right? But many kind of— many more complicated kind of tasks, just doing the prediction from the function is difficult, right? So, if instead I give you Newton's laws, so they're more complicated to write down and to actually make predictions from them, you need to do some integral, plug in a bunch of constants, do an integration.

Right.

So, being able to use the latent structure that you learn to make good predictions is crucial for this to work. And bigger models, they still would really struggle to do something like integration in the forward pass, out of context. So, that's something— this is all speculative, right? But just something you can think about that would be still make this very challenging for models is that some kinds of connecting the dots theories that you could learn or latent variables, they're just difficult to use to actually make better predictions.

Right. So, basically you're saying that there's something about the bigger models having more knowledge about math that they can do integrals and so they could be better at doing the kind of reasoning that you would expect to guess functions, integrate. And then there's another part where you want to know if there's some extra reasoning ability that is unlocked by scale. And I guess for this, it would be interesting to see the difference between 3.5 and 4 in pure in-context learning. If for instance, 3.5 is not capable of doing any guess of the integral in context, that it won't be able to do it with your setup, right?

Yeah. So, you can just check if the models can do this kind of reasoning in context. That is, if you give them the actual latent structure, like the function, just write down the function in context, can the model make predictions from that? And then you could also fine-tune it to do that. And if it can't succeed even with fine-tuning, For example, if it's some complicated integration that involves multiplication and addition and large numbers, probably the model won't be able to do it even with fine-tuning. And so, that would be a case where you can pretty much rule out it working out of context. And bigger models are better at just doing say mathematical reasoning when all the information's in context and they don't get to use chain-of-thought. So, that is one way that bigger models will probably do better.

Yeah. I guess one last question on this is about the mixture of functions because I feel like this is one of the main things of your paper as, oh, it's not only guessing a function, it can guess a distribution, is a a bit more stronger. But if again, I was to be a contrarian person and less wrong, I would say that the results are not as impressive on the mixture of functions. And so in terms of safety, should we be worried reading your paper on the mixture of functions part or should we just say, oh, maybe it gets 10% on this task?

Yeah, I think the mixture of functions is in some sense not that impressive. They're very simple functions. They're just 2. And the model gets multiple examples per data point here, which makes it a bit easier. And the model's not learning this perfectly, right? So, the model still is not super reliable at telling you what the functions are. So, it's noisier in its performance than it is on other tasks. So, I think that the mixture of functions is not really intended as, this is scary, look at this impressive ability. It's more that some theories about what's going on could be ruled out by looking at the mixture of functions. For example, we thought maybe you have to have a name for the latent state or the latent variable.

Right.

In other cases, we use the function f, f would be the name. So, what if we have a case where there is no name and the model just still has to do the task? Where in the mixture of functions, we never name— there's no name like f. So, yeah. So, I think you shouldn't be worried about the mixture of functions. I think that the abilities here in general are not— they're not that impressive. But I would say this is just the first paper really documenting this ability. And so, we did not do some great hyperparameter sweep. We're not using state-of-the-art models like Claude 3.5. We don't know what the best way to fine-tune models for this is.

Mm-hmm.

And so, We don't know how best to prompt models to get them to tell you what they really know, which is not— Sometimes you ask the model, verbalize this function, and there are prompts, there are ways of asking that question that work better than others. So, I think the abilities right now that we demonstrate, I don't think they're that impressive or scary. I think it would be good to just have a better understanding with more research on Okay, if you push harder, what abilities are there? And then also, it's interesting, what can models do in terms of this out-of-context reasoning with realistic training data, where we don't optimize everything for them? And so, it would also be good to sort of look at pre-training a model and having some of this data spread throughout the pre-training set and then ask the same question, is the model able to verbalize things that it's learned from pre-training? So, I think, yeah, there's lots to do to get more of a picture of how impressive are these abilities, how are they scaling, just what are the limitations. So, I think that's what you want to do before exactly applying this to worries about AI safety.

Right. I think in the paper you mentioned something about the pre-training data. Being potentially much more diverse, but also less structured. So yeah, in your setup, it's way more structured how you present the facts, but at the same time, less diverse. So yeah, there's a trade-off there. I guess one question you don't really have the answer for is, could you imagine that with out-of-context reasoning, models could think of new papers or new theory for deep learning if you just pre-train on all the arXiv data in 2 years? Do you think this kind of ability would be able to do these kind of things?

I mean, I think that that's just going a lot beyond what we have in this paper and a specific thing. So, Yeah. So, the intuition, I guess, is something like, look, you have lots of papers and they say they have different architectures and they tell you in those papers how good the architectures are. And maybe if the model has seen thousands and thousands of different ML architectures across all these papers, that it can recognize there's some kind of structure to them that it can learn that humans never learned. And then maybe it could tell you, oh, look, there's something you should know about neural net architectures that work really well. They all have this kind of implicit structure or connection, or there's some mathematical property they have. And more generally, just doing science, right? I mentioned Newton's laws. Suppose you excluded Newton's laws from a training set, but of course, all the physical data in the world, at the scales that humans work at, conforms to Newton's laws. So, there's a huge amount of implicit evidence for Newton's laws. And you can imagine a model that's trying to— the LLM's trying to model all this data and underlying it is this very simple structure that once you know calculus, you can write down very compact way, meaning F= ma, this kind of formula. And so, yeah, in general, this is just a scientific process. You have a lot of data, you find structure underlying the data, you communicate that structure in natural language, right? You can verbalize what that structure is. But yeah, I think the So, in some sense, the problems you're pointing to or the questions you're pointing to, they do have the same form as what we're looking at in this paper. But again, I'd say that there's learning this structure, learning this hidden latent variable, and then there's being able to make predictions using that structure that actually improves your predictive performance. That's what the model would have to do. So, again, maybe you have some set of equations and they're really good at helping you make predictions, but you need to solve some ODEs every time you want to make good predictions. It would be a page of math or run a simulator, and the model maybe is just not going to be able to do that page of math in its forward pass.

So, even if they understood some structure, they might not be able to use it.

Yeah. So, I mean, I think the way that they learn the structure is probably coming from using it. It's because the structure helps them make better predictions, they learn the structure in the first place. So, now with Newton's laws, if a bunch of things are simple, you're on a on a line and some things are zero, parameters can be treated as zero, actually it is very simple to use Newton's laws to make a prediction. So, it could be that there are sort of special cases where the structure is very useful, is very easy to use to predict. So, I don't think we can rule out, maybe models would be able to learn interesting things. in this way. And also, your models are getting better as they scale at doing computation in their forward pass, right? So, Claude 3.5, I think just doing multiplication or something like that in the forward pass with no chain of thought, it is enormously better than GPT-3. I don't know if someone has looked at scaling laws for that, but this is my sense that models are just getting much better. So, it may be that models can make progress there, but I think there's certainly a lot of uncertainty and I would not guess that the models that come out this year will be capable of these kind of more impressive examples that I've just mentioned.

So, yeah, as we're talking about those crazy results or crazy capabilities, that models could have using out-of-context reasoning in the future, speculating. This makes me think about more meta questions that people had on Twitter about the impact of your work on alignment research and potential downsides around capabilities and how you do research in general. I think those can be very interesting for people who are trying to get into the field. So yeah, you have all these crazy results about situational awareness, out-of-context reasoning, reversal curves. They're surprising results, important but also quite simple. How do you come up with those ideas?

Yeah, so I mean, I should say these are all collaborations, so I definitely want to give a ton of credit to to my collaborators on these things. And yeah, so, the Situational Awareness project was led by Rudolph Lane and had a bunch of other co-authors. The Connecting the Dots paper was led by Yannis Troitline and Dami Choi, as well as a bunch of other people. So, yeah, this is teamwork.

Mm-hmm.

But yeah, to get to your question, I think It's always hard to answer where do the ideas come from or what caused you to have these ideas. From the weight. But yeah, there's a lot of chain of thought here as well, for sure. So, one thing I do is devote time to just thinking through these questions of how do LLMs work. That might be trying to put together documents or presentations But basically just me sitting, solo work, maybe with a pen and paper or a whiteboard, just thinking about these things, not running experiments or reading other stuff. And some of the ideas definitely seem to come from just that focused thinking and iterating on the big picture, on how things connect together. And then conversations can be really useful. So, trying to talk to people outside of the collaborators on the project, but just other people in AI safety who I know, sometimes that can really trigger an idea. And then, specifically, these are all LLM papers and I think I since 2020, started playing around a lot myself with LLMs and just prompting them. And so, I had a dataset, TruthfulQA, where we have a bunch of questions distinguishing truth from plausibility, truthful answers from misconceptions. And that involved just tons of playing around with prompting models, base models in that case. So, I think just that hands-on experience and then also building intuition for fine-tuning, what works and what doesn't. A lot of iteration or repetition there of just building better intuitions about LLMs. I think that's been useful. And then I would say that there's this amazing ability that you can fine-tune basically state-of-the-art models on the OpenAI API. And it's quite cheap and convenient and you can do a lot of iteration. And I think that's probably an underrated resource that I think a lot of researchers don't like it because it's not an open model.

Right.

And the fine-tuning is not open, so you don't know exactly what's going on. But I think that, again, with a lot of experience, you can sort of compare the performance to open fine-tuning where you know exactly what's going on and learn how similar they are. And yeah, so I think that just iterating a lot, trying a lot of things on the API, that's definitely been useful too.

Right. So, I guess there's a combination of practical hands-on experience on fine-tuning and prompting the models for a long time and doing what is short iteration speeds cheap and at the same time trying to understand things from first principle. I think I had Colleen Burns early 2023 on one of his papers around discovering latent knowledge. And he explained that his process was whiteboard. He comes up in front of a whiteboard and starts to think about what would this imply if it was true? What if it's not? And just really thinking from first principle. I think this is maybe one mindset that people should have in the space of trying to come up with more of a theoretical approach, a whiteboard approach of what experiment to run and also have some decent knowledge of by default what should work or not, right?

Yeah, I think probably both of these are important. So, I'm always trying to think about how to run experiments, how to have good experiment paradigms. where we can learn a lot from experiments, because I do think that part is just super important. And then there's the interplay of the experiments with the conceptual side and also just thinking about what experiments to run, what would you learn from them, how would you communicate that, and also trying to devote serious time to that, not getting too caught up in the experiments. So, yeah, I think for me both are important. People may have different ways they want to balance those things or different approaches.

Yeah. More generally, if we take a step back, how would you define your research style and taste? Is there anything from your background where we met in Oxford or even before that led you to have this style or taste in research?

Yeah. So, I think I probably look for areas where there's some kind of conceptual or philosophical work to be done.

Mm-hmm.

So, for example, you have an idea like situational awareness or self-awareness for AIs or LLMs, but you don't have a definition and you don't necessarily have a way of measuring this. And so, one thing to do is just come up with definitions and come up with experiments where you can start to measure these things. where the experiment, you're trying to really capture this kind of concept that's maybe been discussed on LessWrong or in more conceptual discussions. So, yeah, I think that's generally what I'm looking for is areas where both of these kind of components come up, the conceptual thing and then running experiments. And in terms of my background, I have studied philosophy in the past, analytic philosophy, philosophy of science, and I worked on cognitive science. So, I did a couple of papers that were running experiments with humans and sort of modeling their cognition. And yeah, there's definitely some ways in which that background is useful. I think probably it's just like would be my mindset anyway, even if I hadn't studied those things. So, it was hard to know the causality, but definitely just some of the things that I studied in grad school are things that I can draw on directly in thinking about LLMs. And in a way, it's like studying LLM cognition, which involves these philosophical aspects as well.

Yeah. I think right now there's maybe still some low-hanging fruits in trying to conceptualize some philosophical concepts into basic evaluations. I don't know if there are other things we could evaluate like this, other concepts that are useful for evaluating current models, but I think there's some low-hanging fruit. In that regard, do you think that people should aim for making papers that they can publish to conferences and try to present this AI safety work to more ML people? Or should they just work on more crazy theory around alignment? Should they work on very ambitious projects or just focus on small compute, as you said, simple fine-tuning? Do you have any advice on that?

I mean, people should to some extent play to their strengths, but I think that there might be different focus for different people. Yeah, I do think that communicating things in something that looks like a sort of publishable paper, it's not so important necessarily that it gets published, but just that it has that degree of being understandable, systematic, considering different explanations. that degree of rigor that is something you see in the best papers in ML. So, I do think that is a pretty useful thing to aim for. So, there are things that just in a blog post, people are just going to have questions and want to go deeper. And a blog post is maybe not— a typical blog post is maybe just not giving enough of that detail or that rigor. So, I do think it's worth investing time in that for doing research so that people really trust the results. And sorry, what was the second part of your question?

Yeah, I guess there was something about cheaper— should people focus on less compute or more compute? But I think this is depending on a lot of variables around people as well.

Sure, sure. But yeah, I think I will say that I think it's generally been sort of overrated how much you need industrial-scale compute resources to do good research related to safety and alignment. So, I think that if you just look at some of the some really good papers that have come out in the last few years relating to safety and alignment, I would say that— and this includes papers from OpenAI, Anthropic, and DeepMind— I think very few of them use a lot of compute. And maybe some of them, if they use some lab company resources, I think you could probably have fairly easily done it with open models or just with fewer resources. So, there may be very recent exceptions. So, training SAEs could be quite expensive.

Right.

And I'm not saying in general that there will never be good research in alignment that needs a lot of compute or a lot of other resources, like humans to do the RLHF or something. But I think there's always been a lot that you could do without the lab-scale resources up until this point. And I think there's still a lot that you can do. So, yeah, I think that people should not feel inhibited by not having those resources. Again, just like you can fine-tune GPT-4o on the API and GPT-4o is an amazing model, right? And you just have this incredibly convenient fine-tuning setup. It's not that expensive. And so, yeah, I think just those— I mean, obviously the LLaMA models that exist are very strong now. So, there are very strong open-source models as well. So, yeah, I think it's a pretty great situation to be in, in terms of the resources that are available to researchers. This may change in the future, but yeah.

Now you don't have any excuse. You have all the LLaMA models and then cheap fine-tuning.

The only thing I don't know—

Yeah.

And obviously more and more libraries and people who can help online if you get stuck. So, yeah, I think there are good resources out there. Of course, I think that AI companies have other kind of resources, which is just they have excellent researchers and engineers, so they can help you in a sort of human way, right? And in some cases, maybe they can help you with creating a big dataset or something like that. So, I'm not saying that those resources don't count for anything, but definitely there's, I think, a pretty wide open space of things. There's lots of things you can do with smaller resources.

One thing people want to do, at least on Twitter where they ask these questions, is they want to decrease the probability of deceptive alignments once we know that models are situationally aware. So, I guess in your work you just mostly measure things like, oh, this thing is situationally aware. Do you have any ideas of directions of how to make sure models are less likely to be deceptively aligned or those kind of directions?

Yeah, I don't know if I have particular new novel ideas. So, yeah, part of what I've been doing is trying to measure the relevant capabilities. and see how they vary across different kind of models. And then potentially that could suggest how you could diminish or reduce the dangerous capability or study a model that does not have a high level of the dangerous capability. But yeah, there's more standard things, which would be just doing fine-tuning to make the model be honest and helpful and corrigible. And so that the model will tell you what it's thinking about and it will have a tendency to correct— it wouldn't be deceptive because you trained it to be honest so much. These are the standard things and I think they're still very important, I think, to develop our understanding of just how well that works. how robust it is. So, RLHFing models is not something I've really worked on in my recent work, but I think that's an important part of this as well.

Yeah. So, I think there are ways of looking at some of the results you publish, and I guess this is more like a contrarian take again, which is that maybe you could use this utility-aware dataset and fine-tune it to make something dangerous. And there's also a question from Max Kaufman on maybe some other results you had on the reversal curves could lead to some potential capabilities improvements. There was other papers that use, not use your results, but solved the problem of the Reversal Curves to improve capabilities in LLMs. And are you sometimes worried about whether your work, when we are looking at how models work, we might end up making timelines shorter?

Yeah. So, I've definitely thought about this a bunch and consulted with various people in the field on this. just to get their takes. And so, for the Reversal Curse paper, we consulted with someone at a top AI lab to get their take on how much would this accelerate capabilities if we released it versus safety. So, you want to have someone outside of your actual group, right?

Mm-hmm.

To reduce possible biases there. And I think the— and I should say, I think this is not specific to the kind of work I'm doing. So, really any work that's trying to understand LLMs or just deep learning systems, right? It could be mechanistic interpretability or understanding Grokking or optimization, as well as obviously RLHF type things, understanding fine-tuning and how that can give you more helpful models. All of these things have the same issue that they can make models more useful and just generally more capable. And so, it might be that improvements in these areas do speed up progress in general. So, I think It's not specific to the kind of safety-related work that I'm doing, but I think a ton of safety work would have similar— there'd be similar considerations. So, I think up to this point, the impact of this kind of work on cutting-edge capabilities, my guess is, is relatively small.

Mm-hmm.

I think maybe— yeah, I won't go into the details there, but I think there's been a bunch of progress in mechanistic interpretability, but I'm not sure that it has really moved the needle that much on fundamental capabilities. And I think there's a general thing here, which is you can improve capabilities without understanding how things work and why they're improving things.

Right.

You can use a different nonlinearity, a different gating, and that can improve things. You don't know why, but it's just like you did a big sweep and this architecture just works a bit better. And so, yeah, but it is possible that improvements are bigger in the future, maybe if some of these techniques just work better. So, yeah. So, I think, how do I think about this overall? I think I would think about what are the benefits for safety? What are benefits for actually understanding these systems better? And how do they compare to how much you speed things up in general? And I think that up to this point, I think that it's been a reasonable trade-off that the benefits of just understanding these systems better and then some kind of marginal improvement in usefulness of the systems, I think it ends up being a win for safety and so worth publishing these things.

Yeah.

I think if you don't publish these things, it's very hard for them to help with safety, I would say, or the help might be pretty small. So, yeah, maybe you think it's a bit different if you're at a major lab or something and you can share it internally, but not with the wider world.

Mm-hmm.

But I think even there, just the best way to communicate things is to just put them out there on the internet so that people can have them at their fingertips. And if something is not out there and not being built upon, I think it usually is quite hard for people to really use that information.

Right. So I think there are 2 levels to what you're saying. There's one is if nobody is releasing anything about, let's say, situational awareness, maybe we need this particular thing to, let's say, solve alignment or whatever that means. And so in that case, we need to push capabilities forward a little bit. Anyway. And so I guess there's a world in which we need those papers anyway. And so we need to improve our understanding. If we're in a world where we have no MechInterp, we have no knowledge of situational awareness, it's much, much harder to steer our models and do important research. So in a way, we need this kind of knowledge about models before we can do more research. And I guess the other part about what you were saying, there's something about if you're at a top AI lab and you're, I don't know, you're OpenAI, you have 1,000 people or Anthropic, and maybe if all your researchers are inside and you're very close to AGI, maybe you could do something on your own. And publishing this SAD dataset, maybe that's net bad because the competitor might make progress or something. But I guess as of right now, it seems good for the world that academia and ML people can see your work and build on it, right?

Yeah, I think that makes sense. And generally, the thing we're worried about is not capabilities per se or models being able to do smart things using situational awareness. It is the situation where we're not able to control the capabilities that we're worried about, where we're not able to control the aspects of the model, like what are its goals and so on, and what kind of plans is it making. And so, I think that the Yeah, we want to be able to, in a lot of detail, understand these capabilities so that we can control them. And with that better understanding of the capabilities, there may be ways that you can, and there plausibly are ways that at least marginally you can improve the usefulness of the model in the near term. But yeah, I do think it's like, there's no— I mean, I think most ways in which we come to better understand models that enables us to control them better, those are going to come with enabling you to make them more useful in some way. I think those things are pretty closely related. But it's again worth pointing out, there are ways that you can make models more useful without understanding really at all why you've made them more useful. You could say— I mean, scaling is a bit like this. You just train the model for longer and you make it bigger and the model gets more powerful. But this did not come with any real insight into what's going on.

Right.

And tweaking the architecture, of course, we might have some intuition for why it helps to use some more sparse attention or to use a different nonlinearity or something or use a mixture of experts. But a lot of it is just try out a lot of things and see what works. And you don't know why it works and it doesn't give you any better understanding of the system pretty much. And so, there's certainly a lot of— maybe most capabilities improvements have just come from this More or less just scale and then search, architecture search basically.

I guess there's some basic counterargument would be that some capabilities you can only see them at the scale where we are right now. I think maybe right now we're at a point where we can do maybe most of our alignment work at this scale. We don't need to scale further, but I don't know, maybe some complex reasoning thing. If you want to align our models on some complex planning or agentic behavior, Maybe we need to study them and play with them at a higher scale and maybe other countries or people will scale models anyway. So in any case, the top AI labs need to scale them further to study or align them. I don't know. It's a very complex problem, but I guess if you're just scaling models without having any alignment lab, it's not bad, I think. Yeah. I guess I had one final question on more the reception of your work. This is quite a novel way of studying LLMs. I've seen some people commenting on it on Twitter, but overall, what would you say is the reception of your work from academia and ML? How did they react to it?

Yeah. So, I think, well, the papers I've talked about here, so Me, Myself, and AI and Connecting the Dots, these are quite recent. So, I think there's not been a lot of time to see, will people do follow-ups and build on this work directly?

Mm-hmm.

But I think generally, There's been— yeah, well, I'll say, okay, so for situational awareness benchmarking, there's interest from AI labs and also from AI safety institutes where they would like to— they want to build scaling policies like RSP-type things and measuring situational awareness, especially with a very easy-to-use evaluation might be a useful part of just the evaluations that they're doing already. So, I think that that's been something that we were thinking about in creating this dataset. And I think generally those people at AI labs and safety institutes who are thinking about evaluation have been interested in this work and we've been discussing it. But with some of those groups. And I think when it comes to academia, on average, academics, I think, are more skeptical about using ideas or concepts like situational awareness or self-awareness or even knowledge as applied to LLMs. So, I think they tend to be more skeptical, thinking that this is maybe overhyped in some way. So, yeah, I think that in that sense, maybe people will be, I don't know, right now just less on board with this is a good thing to measure because they maybe think this is something that LLMs aren't really in a serious way able to understand and do. But yeah, I don't know. I think maybe this will change as models are just clearly becoming more agentic and the LLM agent thing maybe takes off more in the next few years. But yeah, again, I think overall it's fairly early to tell. And it's kind of unpredictable. Some works that I've been involved in have had more follow-ups and more people building on them. So, TruthfulQA is one, that's probably the one that gets the most follow-ups of people using it. And it's just kind of hard to predict in advance, I think.

Yeah. I guess the idea behind this was if most of the impact would come from people at top AI labs using your work versus people in academia using it. But I guess from what you're saying, it seems that it's more like people at institutes and top AI labs building on it.

Well, for situational awareness, yeah. And then for the out-of-context reasoning, yeah, that is hard to say. I think there are some papers on out-of-context reasoning that may have been influenced by our work on that last year. So, certainly in total, there's been more work coming from academia, I think, on out-of-context reasoning. But academics just publish more stuff, right? And they publish earlier. So, there's stuff that may happen inside AI labs that doesn't get published.

So, yeah.

I think we'll see on that what comes out in the next year. So I'm not really sure.

Stay tuned next year for whether models start having crazy adaptive reasoning. Check out the 2 papers from Owen Evans and collaborators. He's just the senior author. So Me, Myself, and AI, I don't know the rest. And the other is Connecting the Dots, blah, blah, blah, something else. Do you have any other last message to the audience? I think they should build on your work. They should do all the follow-up you've mentioned. And I think you also have a stream maybe, Saree Matz, so people can join.

Yeah. So, I'm supervising people via the MATS program. And so, that's one way that you can apply to work with me. And so, most of the projects of the last couple of years that I've done have involved people from MADS. So, that's been a great program. And I also am hiring people in other contexts for internships or as research scientists. So, if you're interested, just send me an email with your CV. And also, yeah, you can follow me on Twitter for any updates.

And increase your follower count.

What's that?

And increase your follower count.

Well, yeah, I mean, I'm not on there very much, but yeah, if you want to see, if I do have any new research, then it will always be on Twitter so people can go there. Yeah.

Yeah. And I highly recommend the Twitter threads that you write for each paper. It's a good way to get the core of it. I made this one video about AI, lie detection that I also recommend people watching. And most of the people were like, oh, you can just look at the memes that you create or the core threads you write and you get maybe 50% of the paper from just the threads. So yeah, follow Owain Evans on Twitter. This was, I think, the end of this. And maybe we'll see in the next few years if there's more data on this.

Great. Yeah. Well, it's been fun. Really interesting questions. And yeah, thanks a lot for having me.

It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co Or you can DM me on the social media platform of your choice.