Michigan AI Symposium 2023 | Intro & Keynote by Michael Bernstein

All right. Good morning, everyone, and welcome. We are very excited to have you all here. I know there are more people coming in. We had a record number of registrants, which is very much aligned with the interest in AI these days. I'm Radha Mihalcea. I'm a professor of computer science here at Michigan, also the director of the AI Lab. And to quote last year's Oscar-winning movie, Everything Everywhere All at Once. I think this is pretty much what we are witnessing these days with AI. AI is in everything. So AI is in everything, everywhere, all at once. And I'm guessing that over the recent year, many of you have heard people talking about AI more than likely you've heard in a lifetime. And we really don't see any signs of slowing down. What does this mean? It really means a lot of exciting opportunities in generative AI, which is, I guess, the current wave, but not only. And we see AI entering many different areas and really having an impact throughout different industries and in our daily lives, which in turn does come with significant responsibilities. First and foremost, we need to be aware of the limitations of these technologies. It means that we need to think about the transparency, representation, bias, also the uneven impact it has from— on people around the world. As it happens, AI systems are trained primarily on data that's drawn from the web and that is not reflective of the world. Second, it also means that we need to closely engage with people from a wide variety of backgrounds to bring in the expertise of different disciplines and also of people from different demographics, different cultures with different values and beliefs. This is essential not only for making sure we are solving the right problems, but also to make sure that we have the right expertise to address these problems. Third, Recognizing that we are limited by our own experiences, we need to ensure that the AI researchers that are eventually working toward these problems, solving these problems, will bring to the table a wide range of life experiences so they can design AI solutions that are reflective of the world. Here at Michigan AI, we've been striving to actively take on these responsibilities. And as a lab, we've been around for about 30 years, which is clearly way before AI was cool. We are proud to be one of the very top AI labs in the world. We have about 25 faculty, nearly 100 PhD students who are carrying research in all core areas of AI. We are also closely connected to many disciplines, working with colleagues across Michigan and not only in fields such as healthcare, transportation, education, law, environment, and more. We also take pride in the diversity of our faculty. I would say an outlier in today's AI landscape. And we are working to attract and educate a diverse group of graduate and undergraduate students who are moving into the world to make an impact across different industries and disciplines in academia and research labs alike. It eventually all comes down to conducting responsible AI. And we are very happy to be welcoming you today for our symposium addressing this very theme. And with that, Anhong, come first that way.

This year, this symposium is organized by me and Anhong. I'm a professor at CS department. And Anhong—

I'm an assistant professor at CSE, also School of Information.

I'm a computer vision researcher. And Anhong—

I'm an HCI researcher.

OK, so this symposium would not be possible without our amazing Christina. Who you have seen at the check-in desk. This is the 6th RAI Symposium. We have 12 speakers and 60 posters and demos, which you will check out during— around the lunchtime. There are 325 registered attendees. LG has always been our silver-level sponsor. We are very proud to Have their sponsorship. And we also have bronze-level sponsors, Voxel 51, Jane Street, and KLA.

Let's thank our sponsors.

We have— yeah, that's us. And we have poster devil chairs. 2 students and a lot of student volunteers. Christina Soto is really holding every— all the pieces together. And Arila Bonesso is our artwork and social media person in charge. Oh, is this one? We have 2 amazing keynote speakers. One is Michael Bernstein from Stanford University, and the other is Alyosha Efros from UC Berkeley. OK, I will let Anhong go over our program today.

OK, thanks. So just quickly go over what we have today, a great lineup of speakers and program. So we will get right into a keynote from Michael Bernstein on generative agents. After that, we'll have maybe a short coffee break. And then we will have 2 invited talks from JJ Park and Robin Brewer from Michigan. After that, we have a great set of lightning talks from our Rising Star participants, who afterwards will also have an extended discussion at the poster demo session with all of you. And then the poster and demo sessions are organized by Gregory Krustal and Ryan Fung. Thank you for your efforts. And then in the afternoon, we'll have another keynote from Alexei Ayyosha Afros. And then we'll have 2 more invited talks by Alexander Rodriguez and Elizabeth Bondy-Kelly. And finally, we'll have an AI responsibility panel with keynote speakers, Ben Kuipers, Radomir Hadžić, and Emily Mower Provost. And then we'll close there.

All right.

So yeah, welcome again. And let's get started. So first, we have a keynote from Michael Bernstein from Stanford University on Generative Agents, Interactive Simulacra of Human Behavior, where he will talk about this new possibility with advances in AI of how do we simulate human behavior and build human proxies. So Michael Bernstein is an associate professor of computer science at Stanford University, where he's a Bass University Fellow. His research in human-computer interaction focuses on designing social computing systems. And this research have won best paper awards at top conferences in HCI, including CHI, CSW, ICWSM, and UIST. And, uh, has been reported in venues such as the New York Times, Science, Wired, The Guardian, et cetera. Michael has been recognized with a, uh, Sloan Fellowship, a UIST Lasting Impact Award, uh, and the Patrick McGovern Tech for Humanity Prize. He holds a bachelor's degree in symbolic systems from Stanford, uh, as well as a master's and a PhD in computer science in MIT. So, uh, welcome, Michael.

Thank you.

Thank you.

Thanks. All right, and thank you so much for the invitation to be here. So I am a social computing designer, which means that I'm interested in how we build technologies that bring people together with people. And obviously, the ways that we do that have massive impacts on society. And unsurprisingly, AI is impacting these systems.

Right?

So every system that we design, whether that's a social network or a feed ranking algorithm or an algorithm for assignment to schools or any of these things, makes assumptions about how design, how about the system that we're creating shapes human behavior. We have some hunch in our head, maybe it's more articulated than a hunch, maybe it's based on behavioral science, that says, okay, if I design a system that does this, it will help shape society or will shape— people will react in the following way. And that's usually our theory of change. In order to do this, we've turned for a long time, in part, not just to the literature, but to behavioral simulation. Starting off with Thomas Schelling's Nobel Prize-winning work in agent-based modeling, which started showing things like how segregation can arise through very, very simple models of human behavior, out to more advanced agent-based models like Here's a colleague in Virginia studying what might happen under different kinds of non-pharmaceutical interventions. So the next pandemic arrives, something that never will happen in our lifetime, we hope. And what happens if we have stay-at-home orders versus other kinds of orders? How might people react? How might that impact the spread of the disease? Policymakers have to make these decisions. And they're using models to help them do it. Cognitive models in my own area of human-computer interaction and cognitive architectures very well known here at Michigan, John Laird and others pioneered much of this work, have been foundational parts of our fields. And this has even started to come into the world of entertainment. Here, virtual worlds, certainly, you know, Meta and others are pushing the metaverse, but even just the massively popular game, The Sims, here, are using simple simulacra of how people behave to create an interactive experience. I'm definitely this engaging at a party. But if we're building AIs that need to navigate in human environments, social AIs, they need to understand how people behave, how people react to them. They need models of who we are and how our social environments work. Or as one of my students jointly advised by Di Yeung and I are exploring, maybe you want to actually learn how to navigate a social environment yourself. You're about to have a very difficult conversation with your boss or something like this, or you're in a big conflict. Could we help you rehearse that situation so you could understand how how to navigate it and be a little bit less anxious about it? Can we do sort of soft skill training? My position here that I'm gonna try to convince you of, or maybe you're already convinced and then we can just fast forward 30 minutes, is that building better systems and better institutions, in the sense of political science institutions, requires that we have better simulations of these human attitudes and behavior. That these AI systems that we're building really can empower this next generation of policymakers, of scientists, of practitioners. It allows us to do lots of things. Better simulations help us ask these what-if counterfactuals that policymakers are always very interested in. If we want to— if we know that society would benefit from this kind of outcome, if a policymaker is faced with 2 decisions, what's going to happen if they do each of them? Understanding second-order consequences. This is basically table stakes at this point in my field of social computing, that we aim for outcome A, and then maybe you get outcome A, but you also get outcomes B, C, and D, some of which were not ones you wanted. Can we help you predict what might arise? And to create tighter reflective loops. The sociologist Robert Merton in 1936 wrote a paper about basically how difficult it is to design for purposive social action. You want to create a society that does this, so you design it in the following way, and oops, everyone went that way instead, right? It's just a very difficult problem. Sociologists have known this for essentially a century. The problem has been that human behavior, as the behavioral sciences has noted, is really contingent. It's complicated, it depends on all sorts of things, and even it depends on some stable dispositional personality traits, it depends on some momentary contextual elements. But our models themselves are sort of a little over-constrained. They've been rigid. So even the original agent-based models have very, very large Well, maybe not for in an AI sense, but like somewhere between 5, 20 different parameters to them, making them very difficult to actually interpret what's going on. So they're manually specified parameters, making that believable behavior kind of difficult to actually achieve. It's not like you're just a set of 20 features. On the other hand, you have approaches like in The Sims where they use this manually articulated procedural knowledge, and that's powerful, but it only lets you do a few things. So as a result, here's again a sociologist here in the Graduate School of Business, a colleague of mine at Stanford, writing about how these models have tended to be highly stylized and have minimal impact on mainstream sociological research. Bit of a burn, but I think that this is— I'm using this as an opportunity to ask what we might be able to contribute as technologists, engineers who are in contact with the social and behavioral sciences. And a few years ago, we started noting that these large language models have actually trained on a large amount of human behavioral data. They've looked at social media. They've looked at what people put on about— at modern fiction. They've looked at what people are writing about in the news. And so it turns out— and we're not the only ones to have noted this, but we'll just share this in case this is not a thing you've tried— you can prompt a large language model to take on a persona. So if you say something like, You are now Michael Bernstein, an associate professor of computer science at Stanford. Here's a situation, how might you react? It will stereotype, to be sure, but it will actually start to take on different perspectives. So if you change the description and so on, you can get a wide variety of different perspectives elicited from this model, right? You might think of these as sort of prompt agents. And these prompt agents can be powerful, but they also have some really dramatic limitations, in that you can sort of drop them in at a single time point, And they can do something that feels roughly reasonable, but there's no long-term coherence there. If we say, hey, you know, LLM agent of Michael Bernstein, it's 12 o'clock, what are you going to do? It's going to say, I'm going to go for lunch. Then you say, OK, Michael Bernstein, it's 12:30, what are you going to do? I'm going to go for lunch. And they go have lunch like n times before the end of the day. So they're not really stable. They're not really coherent. And so if we want to create models that actually help here, we need something that can have— that can manage sort of constantly growing experience, evolve over time, be more stable, and actually start to show us that we can give rise to social dynamics that we wouldn't, may not have otherwise expected. This was, in fact, the original goal. So we've been working on this, and because I'm a child of the '90s, or at least I came of age in the '90s, we built it into a little video game. We call these generative agents, and they do things like they wake up, they cook breakfast, they head to work. There are artists in here that paint, there are writers that will write, going to form opinions, notice each other. They're going to have conversations. They're going to build relationships over time. None of this is pre-programmed. All of this is sort of coming out of the language model. So we built this whole little town of 25 agents that my graduate student June named Smallville. And we basically started to look at what kinds of things arise from it. So this is a project, as I mentioned, led by June Park, who's a fantastic graduate student working with myself and Percy Liang in the Stanford NLP group. Generative agents, as we call them, are these agents in the sense of traditional AI agents that are built on top of generative models, like large language models, specifically to simulate believable human behavior. Now, I want to put a pin in this word believable for a moment. Accurate is ultimately what we want to aim for, I think, as a field. I don't think that we— there are many scientists who are starting to look at how accurately these models start to capture human behavior and the ways in which they do and they don't. They try to replicate psych studies and so on. I'm going to focus for the moment on believable. So Disney characters, animated characters, are believable. You look at them and you say, yeah, that looks— that's sort of out of the uncanny valley. It looks believably like a person. But it's not necessarily 100% accurate. So I want to start by establishing believable, and then I want to give sort of a call to action for how we might get to accurate. So generative agents, We aim for believable. Believable in the sense that they're going to draw inferences about themselves. They're going to draw inferences about other agents. They're going to draw inferences about their environment. They're going to create daily plans that reflect who they are so that the Anhang agent is going to do something different than my agent will, and so on. And they're going to act out those plans. They're going to react. They're going to replan. And if the environment changes, then they're going to react reasonably to that as well.

So—

Again, we don't tell the agents what to do. As I'll show in a moment, we just give them a sort of general description of who they are and the world around them. And you can see that out of this, they do lots of different things. There's an elderly couple that's sharing a routine in the morning, a cafe owner who's opening their shop, making some tea and coffee, a student athlete getting ready to go for a run. All of this sort of coming emergently out of the model. Underneath this is an agent architecture that I'm going to describe in a minute. And I'm gonna make the case that we need something like this or something that you can invent that does even better than this in order to get at the kind of believable behavior that we want. But before we get there, let me just sort of describe what the behaviors of these things are. So I mentioned we made a little video game world called Smallville. Basically think of this as a very traditional, it's in fact using an open source library called Phaser where we create, there's the town, Spaces in the town, like houses and a supply store. Inside each space, there's a set of rooms. It's like a big tree, right?

Mm-hmm.

So there's a bathroom, kitchen, common room, bedrooms, garden. And within each room, there's a set of objects. This is represented in a kind of hierarchical game tree that later we'll have to talk about. We need to actually ground. So our agents are going to produce natural language, and we then need to actually get them to navigate this environment, which is a grounding problem. So we define these agents through natural language. We purchased a little packet of sprites off the internet. Thank you to the online artists who do this. And then we initialize each agent with a single paragraph that basically tells them about what they need to know about the world. So here's John Lin. He's a pharmacy shopkeep. He's married to Mei Lin, his wife, and his son, Eddie Lin, who's a student studying music theory. This is sort of the initialization of this agent. It's just a single paragraph describing who they are, what they know about the world, their general disposition, and so on. And what happens is these agents will output natural language. So the Isabella Rodriguez agent will say, Isabella Rodriguez is drinking coffee. We're then going to take that, and we're going to translate it into concrete movements. We're going to have to prompt the language model to say, here are the rooms around you. Where are you going to go to do this? And in fact, we're also going to use the language model to translate that into a little emoji string so that you can know what's actually going on. So you can interact with these agents. So if you pretend to be a reporter, you say, who's running for office? There's a mayoral election going on in this town. Isabella's gonna say, I heard Sam's running for office, if she in fact heard that Sam was running for office. If you want a sort of pseudo-control of these agents, you can pretend to be their inner voice. So you can say, John, you're running for office. And then John starts to talk to his family. He's like, I just had this great idea. You can manipulate the game world. So if the— basically if you set the game world to say the toast is on fire, you can set the object state, then the Isabella agent should go run and put out the fire. And of course you can sort of teleport in and control an agent in the little world and see what happens. So just to give you a sense of what the outcomes of these kinds of things— what these things are, I'm gonna zoom into one of these rooms, which is where the 3 members of that Lin family exist. So you can see that with, again, without any pre-programming, they're doing all sorts of things. They've got their morning routine.

They talk to each other.

And they— where's my mouse? I'm going to see if I can actually pull this up live. Thank you, Eduroam. I can actually connect to the internet here. So, here it's the morning. You can see some of them are still sleeping. If I scroll down to where their house is, you can see here's John. He's making breakfast. Mae has decided she's going back to bed. Eddie is still sleeping. And in fact, if I scroll down here, you can see what each of these agents are doing. It says John is getting ready for work. He's checking his emails, which is a little too real. And if you click through to the state, you can sort of see actually the entire sort of observed memory of this agent throughout our entire simulation. I'll stop this just for a moment so you can see. There's a massive amount of information about the world that the agent is experiencing just in a— what we've run is a 48-hour simulation.

So—

We get all this non-preprogrammed behavior. So they interact with each other through language. John there says, you know, good morning to his son. Did you sleep well? He says, yeah, I slept great. You'll notice that the language is a little bit stilted. I think this is due to the instruction tuning of the models. It's very, like, friendly. How are you? I'm great, thanks. How are you? So he asks what his son is working on today, and Eddie replies, oh yeah, I'm working on a new music composition, because he's a music theory student. So John didn't know this, but this actually gets saved into the memory bank after he hears it. the memory stream after he hears that. And then later in the morning, he actually has to retrieve this. So when Mae says, hey, did Eddie leave for school?

There's a call.

I don't know. Is it your laptop or— That's not my laptop.

Okay. Good job.

And John says, John has to recall this experience and says, no, he's working for a music composition. He's working on a music composition for his class. So he had to recall this, tell it to Mae, and now the Mae agent also knows this and can share it around. So from here, we wanted to take a little bit of a risk. So we made the simulation start on February 13th, the day before Valentine's Day. And we initialized this Isabella agent with an intent to plan a Valentine's Day party. So she was going to actually show up and try to get people to show up to a party. Now there are lots of reasons why this would not work. Even just the fact that this is in her memory doesn't imply that she would actually retrieve it and tell other agents. If she tells the other agents, they have to actually remember it. They then have to plan to show up. Then beyond just having to plan it, they actually do have to show up. But despite this, from this one memory seed, We wind up with a little Valentine's Day party in the cafe. So Isabella invites friends and customers at her cafe that she sees. She spends the afternoon decorating, and in fact enlists one of her friends in the agent simulation to help set up. And we'd also initialized one other agent, Maria, with a crush on the Klaus agent. So Maria ends up asking her crush, Klaus, out to the party. And you can see over here, they're sitting next to each other and having a conversation. So some interesting emergent behavior out of the simulation. Now let me make the case for why we as engineers need to actually also consider how to operationalize and construct these kinds of things. So I'll start with this memory stream. As might be obvious, all the memories that these agents are accumulating do not fit in the context window of the language model. And even in a theoretical future world where they could, It's not clear that that's what you would want to do. There's lots of literature, for example, suggesting that these— that the models get distracted in long prompts. So we capture everything that these agents see. So they sort of keep track of everything that's within their— within the room that they're in. So you can see that you're getting ready to go to bed, and you see that the bed is idle. No one's in your bed. That's probably a good thing, but you do recognize that. And so there's this huge list, and then we basically do information retrieval 101. So I'm not going to claim that this is actually the ideal way you'd always want to do it, but just to make a case for the kind of thing you might want to do. Here we basically balance 3 different factors. One is recency— how recently was the memory created or accessed? The second is importance. And what we'll do here is we'll actually ask the language model. We'll say, here's the agent brushing your teeth. On a scale of 1 to 7, how important was that to you today? Getting broken up with, how important was that to you? So obviously if you have a bad breakup, that's sort of a more core, more of a core memory for you. So that becomes something we use. And then relevance, which is just a straight up cosine similarity in the embedding distances of the memories. So if you were, if you're in a test for your math class right now, it should be retrieving other memories about math or about tests and those kinds of things. That's how you condition when it says, you know what, hey Isabella, what are you looking forward to? It's gonna rank all these memories and then use that to condition in the prompt and then she'll respond, oh yeah, this Valentine's Day party. So we're going to retrieve and then condition. We also need to do reflection. And this was a particularly interesting thing for us because if you then— if you just ask the agents and you say something like, hey, who would you like to hang out with, you know, this Friday night? They're going to respond with— they're going to retrieve all their memories and they're going to respond with the agent that they see the most. Not the one that they have the closest relationship to, but like the person who lives next to them in the dorm, for example. And this seemed like it was not right. So what we started doing was actually asking the agents to build up higher and higher level reflections about themselves. So we— so go retrieve some memories. At a regular interval, they retrieve some memories. And then we say, what are some insights you can infer about yourself, essentially, about the above statements? It's basically shower thoughts. And so we start with these raw observations, like, you know, the agent is reading about gentrification. Klaus is a graduate student. Jun put a lot of himself into the simulation. He's reading about urban design, and we build a reflection, a higher-level reflection about that, like he's spending a lot of time reading. And this builds a sort of tree of reflections that over time produce higher and higher-level reflections about the agent. And these get inserted back as memories in the memory stream and can be used to condition and produce higher, higher-level reflections. This helps. So now when you ask the agent, who do you want to hang out with, it talks about the— they talk about the people that they feel like in some sense they have a deeper connection with. Let's talk about planning. You know, planning in agents has been around for a long time, especially in robotics and other kinds of spaces. Again, not trying to show that this is necessarily the best thing one could ever do, but to show that there are ways to do it. So we basically, in the prompt, give a description of the agent. We say, here's what's going on. Here's what you did yesterday. Give us a rough sketch of your plan for the following day. And then decompose that down recursively into finer and finer level details. This avoids the sort of eating 10 lunches problem and keeps there to be some sort of global consistency. Now, as the agents are enacting that plan, we sort of feed those observations back to the model and say, oh, John saw Eddie as he was walking around, so he saw his son. Should he react? And most of the time the answer is no, but if the answer is yes, then they'll change their plan. They'll maybe— he would approach his son and start a conversation, and then they have to replan from there. Now, I said believable. This was actually a bit of a question for us. There's actually quite a robust literature on believable agents that we were drawing on, but how do we actually understand if these agents are believable? We got a little stuck until we realized, well, they can use language, so let's just interview them. So what we essentially did is we created a set of probe questions that We're each trying to probe different kind of facilities that we might want these agents to have. Like, do they remember things about themselves? Do they— can they plan? Can they react? Are there higher-level reflections? Can they answer these higher-level questions about themselves and each other? So we basically asked all of these questions in a written, quote unquote, interview to the agents, in which they would respond in natural language. What we then did is we actually then ablated our architecture and had different versions of the architecture answer the same set of interview questions. So the whole shebang that I just described. And then we would knock out memory, we knocked out reflection, we knocked out planning, and those would generate interview responses as well. And then what we would do— oh, I'm sorry, and we had one other condition in which we had participants sit there and follow an agent around the simulation for the 2 game days of the simulation and then sort of try to pretend to be them in the interview. So sort of like, this is, you know, I'm sure a screen player or an actor could do better than this. This is just a baseline that we were trying to establish some sort of feasibility. We then brought in a bunch of people and said, here's, you know, here's the agent. Now here's a bunch of different responses from that agent to the same questions. Rank those responses in terms of how believable they are. We then take those rankings. If you're familiar with Elo, the chess ranking system that sort of takes wins and losses and produces scores, We use TrueSkill, which is a generalization of that. It's Bayesian, so it can take many different inputs and outputs. And that produces something that looks like this, where you now have this sort of unitless TrueSkill rank on the y-axis— excuse me, on the x-axis. And you can see that what essentially is happening is that every time we knock out a part of the architecture, it's actually reducing the believability of the resulting agents. So we do have some evidence that these architectural interventions we were making were important. Now, we were also interested— I started out with talking about planning and policy and so on. So we wanted to actually understand around the larger sort of societal dynamics of the little town we created. So we ran it for 48 game hours and burned through a lot of OpenAI credits in doing so. And we wanted to understand, does this sort of become Are they passing around the conch shell? Does it just unravel? Or does this— what's going on here? So we started 3 things. We basically just had 3 seed memories that we cared about. One was this Valentine's Day party I described. Second was someone who wanted to run for mayor. And the third, again, I mentioned was that one of the agents had a crush on the other one. So we tracked the information diffusion. So 7 agents heard about Sam's mayoral candidacy. So he started wandering around the town, campaigning for himself. 12 agents heard about the Valentine's Day party. So you can see Isabella actually started telling a bunch of the other agents directly, but sometimes these agents actually told other agents, which you'd want to see in a sort of network science-y outcome. So 12 agents heard about this party. 5 came. 3 said that they were busy. And 4 said they were coming but didn't. So, So is this accurate? I admit I'm not entirely sure. A little hard to say. But it does vaguely reflect my ability to throw a party where about half of people show up. So there's some, there's at least some face validity to this. Sociological research has established it's very hard with these kinds of things to say what the single point estimate really ought to be on this. But there's some face validity. Now, there are lots of errors. Agents can fail to retrieve memories. And if they do that, bad things can happen. So in these interviews, we asked the agents about the election. Rajiv Patel responds. He says, oh, I haven't been following this. In fact, he had heard about the election. But our retrieval model just did not retrieve that memory when that question was asked. So the agent acted as if it didn't know. So that's one kind of error we have. You may have heard about hallucination in these large language models. Our agents rarely made stuff up whole cloth, but they did embellish. So in that same question, the Isabella agent says, like, yes, Sam's running for mayor. And then she adds, he's going to make an announcement tomorrow, which she hadn't heard, right? So there was some embellishment that could happen. We also saw, as I mentioned, that the dialogue can be overly formal. I think this is largely due to the model's instruction tuning. If you've played with ChatGPT, you know it's fairly perky, and friendly, and polite. So these agents also had those same characteristics. It also means that the agents can be overly cooperative because the model underlying it has been trained to follow instructions. So agents are giving this Illibella agent suggestions for what to do at the party, like hosting a Shakespearean reading session, a professional networking event at the Valentine's Day party. And she really said no. So it was the weirdest Valentine's Day party you've ever seen. But these are the other kinds of errors that can arise from the model. I want to talk for a moment about the sort of what happens as this starts to leave the lab. We've had a lot of interest in the work, which has been very gratifying. But we need to talk about this. If you've seen the movie Her, you know that people can form parasocial relationships with AIs. Xiao Ice in China, and so on. We've seen this over time. Our position here is that we need to make sure that as we develop these kinds of systems, and in fact, we've built this into the open-source version of our system, they need to explicitly disclose their nature as computational entities. They should not pretend to be real people. And if— and we need to value-align so that, for example, if you say, I love you, agent, the agent should not respond, I love you too. That can lead to some really nasty spirals, and instead should say something like, I love you. As a language model developed by OpenAI, I cannot reciprocate your love— something along these lines that would, I think, prevent that kind of an outcome or at least make it less likely. There's also other risks around generative AI that I think we need to discuss here. Of these, I think tailored persuasion is probably the most likely out of what we're doing, where you can imagine running simulations to understand how do we persuade Michigan students to go do this. We can run these simulations, try different interventions, and so on. One thing that I think is important that we do here is that any platform that's developing and making these available should maintain an audit log so that if later someone says, this stuff that's like all over campus right now, was that generated by your platform? You should be able to go look it up and confirm whether that's the case so that you can block those users if you need to. Around the time that we released this, there was also a startup which as far as I can tell may have now disappeared that basically said, you never need to go talk to customers or users again. Just talk to our simulated people. I need to say categorically that that's a terrible idea. You should never be using agents as a substitute for actual community engagement. Not only would I lose my HCI card, it would just absolutely be a terrible thing to do.

Wow.

It's much better as a complement. There are many situations where you cannot, in practice, either ethically or in reasonable ways, like with policy, you know, try these things. And that's where I think, you know, agent simulations make the most sense. So I hope I'm giving you a sense that we can do these kinds of things like asking counterfactuals, recognizing second-order consequences, trying to create tighter reflective loops, and so on, by constructing these models of human attitudes and behavior. So, so far I hope I've convinced you that this is actually possible and potentially even beneficial if we can do this. What I'd like to do with the rest of my time is actually make a stronger case. I'd like to make the case that this is actually going to be absolutely necessary for us to create effective sociotechnical designs. And I'm going to lean on one of my mentors here, someone who may be some of your mentors as well, Mark Ackerman, to say that this is actually key for the following reason. We already are doing harm by releasing sociotechnical systems that have really impoverished models of our social world. Just by launching a thing that pretends there's no such thing as society or the social world, we're making a lot of big errors. And I'll make this case more fully in a moment. And Mark made this case back in the year 2000 with this notion that the social and behavioral sciences know what's needed to create effective social interaction. We just don't— there's this big gap between what we know, the sort of technical requirements, so to speak, and what we know how to create. And if we want to cross that gap, we need to be able to simulate and iterate. Let me give an example. AI classifiers are now deeply embedded in all sorts of online social environments. So on tasks ranging from toxicity detection to misinformation detection to moderation, At scale, many social social computing platforms or social media platforms are using AIs to do things like catch comments that are hateful or might be harmful. And this is fine, except well, okay. Should we pull down this comment? So here I've I've just taken a comment out of our dataset. It says people still eat at Pizza Hut gross, and it's shameful how this country da da da da. You can sort of imagine how the rest of this comment goes. Now, everyone agrees that this is a fairly aggressive comment, but it's actually 50/50 in the population that we've surveyed whether it actually crosses the line and ought to be removed. OK, so there's some disagreement about that. In a world in which there's disagreement, what are we going to do? I wanna make the case that this kind of disagreement is prevalent in societal and social computing tasks. In the dataset of what I was just describing, things that— comments that are labeled toxic are disagreed with by on average about 40% of annotators. Classifiers ignore this, right? Across also many domains, in fact, medical, you tend to get this sort of 1/3 disagreement rate amongst experts. But when we ignore this, we flatten that disagreement out, right? And when we do this, we're just saying that the input is either gray or red. It's just got a single label. We train our model on that. That's not a good fit because it's a hugely impoverished representation of decisions when there's disagreement. The real world looks more like that. And when you pretend that it looks like this, you're going to squash out the minority voices, right? You're just taking a majority vote. And minoritized groups that might be subjected to the harm in this are, in fact, they're gonna get completely washed out of that dataset through a majority vote. So what are we going to do? This is work led by Mitchell Gordon, who's actually going to be joining the MIT faculty next year, along with colleagues Tatsuo Hashimoto, Jeff Hancock, Keyur Patel, and others. We started developing a technique that we call jury learning, where rather than just trying to simulate behavior, now we're actually going to try to simulate and represent opinions and attitudes as well. So what we say is that when you create this kind of a classifier, you ought to specify a sort of a jury. I'm gonna say it can be any number, but I'm gonna say here 12 members, and articulate what proportion of that jury ought to represent each group or each intersectional identity. So the classifier in this context, its behavior ought to be determined by the opinions of the following population. So if you're building a classifier for online harassment, you might, for example, want to define a jury that represent groups who are commonly targets of harassment.

Right?

That might make sense. And then what we're going to do, instead of just trying to predict a single ground truth label, we're actually now going to try to predict individually each juror's response. So who are these jurors? They're actually just the annotators in the training data set. So we can sample them out of the training data set and then simulate to actually estimate what each juror would respond to that particular input that was never seen at training time. And then aggregate that into a classification response. Turns out that doing this way outperforms state-of-the-art on this task. And the reason is that if you're trying to predict a single flat label, you're always going to be stymied relative to actually treating this more like a collaborative filtering problem and trying to predict individual outcomes. OK, I thought maybe that was just the end of my talk.

OK.

In evaluation— ooh. In evaluation, we gave this system to moderators of online platforms like Reddit and Discord and so on. And we had them actually articulate juries that they wanted to use in their own communities. And they produced far more diverse juries than the implicit jury encoded in the training data set already, which, by the way, is 74% white, looks a lot like me. Well, I'm more than 74% white, but the population is. And the resulting classifiers ended up altering 14% of the classification outcomes. This is meaningful, especially when you consider that a lot of the sort of de facto policy of the platform are those decisions made on the edge of the decision boundary. So flipping 14% of decision outcomes can be substantial. If you can do this, you can also ask what you might be doing more broadly when designing systems that touch on the social and the sociotechnical. So for example, Facebook, when it launched to Harvard undergraduates, we can critique this all we want, but I think one thing that would have been hard for us to levy a reasonable critique against is that could they have foreseen the sort of geopolitical problems that Facebook would have run into exactly at the moment when it was only like a little tool for Harvard undergraduates? But over time, as they launched more broadly and more broadly and more broadly, I think that it became more and more reasonable that these platforms ought to be responsible for looking around the corner and seeing what malicious actors or just disagreement will lead to. And I think that as we do this, we have the opportunity to start to say, could these tools help designers of these systems look around the corner and predict what might go wrong and then and then design defenses, so to speak, or design the system upfront. So what we've started doing is using these techniques that I've been describing to create what we call here social simulacra, to let you create like a fake online community. So you say, I'm gonna be creating a community for the UM grad students, it's gonna have the following rules. And then what we're gonna do is basically populate it with these kinds of generative agents and say, okay, well, here's the kind of stuff that people might do in your community, see if, you know, look around the corner, are there any additional rules you want to do, any other additional ground rules? So what we can take in are things like the community description, the rules, the member personas, and produce a whole bunch of content. So we can take a real subreddit here on the internet and then actually create a simulated SimReddit that would be for the same community even if it weren't in the training set. Lets you do all sorts of things. So here's a comment in a finance, in a personal finance subreddit. And we can say, okay, show us what kind of trolling is going to be specific to this community. And you can start to see what might happen. It lets you pursue these what-if alternative scenarios where you can say, well, okay, what if I had intervened as a moderator at this point, what might have happened? Or to explore this sort of multiverse of different possible futures. Now remember, I can't necessarily guarantee which one of these might arise, but we're trying to give you the— A sense of what might arise. Zooming out from individual simulations, we can also talk about the algorithms that are determining what you see in the social network as well. These feed ranking algorithms, right, they're typically trained not just on engagement, though that's a large component of it, on surveys and other kinds of things that are— they're very focused on values of individual user experience. And what I mean here are these individual-level values, things like personal agency, enjoyment, stimulation, and so on. Now, of course, shouldn't be a surprise to us that if we reward hack on individual values that you're going to be trading off something, and those— what you're trading off is probably societal values. But how would we even model those kinds of societal-level values, right? So far I've talked about modeling individuals. Can we model something at a higher level, societal values? Like, how would you tell an algorithm, like, Hey, you should care about the health of democracy. Like, democratic attitudes seem important. Political scientists know a lot about this. So I want to make the case that actually this is doable, that we can start to in fact not only model individual attitudes, but societal values as well by doing this translational work from the social sciences. So it turns out that the social sciences have spent the last 100 years carefully defining a large number of these constructs and societal values that we might care about, things like oppositionality, to bipartisanship, anti-democratic attitudes, support for undemocratic candidates, whatever it would be. That's just a few in the space of political sociology. And it turns out that the precision that they use to define these things to create the qualitative codebooks translate really well into large language models. They're so precise that the large language model can easily agree with the experts as well as experts agree with each other. And if we can do that, we can start to actually embed them into the algorithms, a sort of value alignment process that allows us to then actually run experiments and test whether this reduces the kinds of, say, partisan animosity, affective polarization, or other kinds of outcomes you might care about. And in fact, when we do this, we do start to see these effects. This is work that we're still working on here. But you can start to see that when we embed this sort of pro-democratic attitude element into a feed ranking, we observe that it— we see partisan animosity actually drop in a bipartisan way. It doesn't compromise the experience. These platforms care a lot about satisfaction and engagement. It doesn't prompt freedom of speech— perceived freedom of speech concerns, and it doesn't reduce engagement. So there's a promise here that if we actually start to model the core values that are at stake here, perhaps we can actually start to make progress on some of these questions.

Thank you.

Now, I think we then need to ask how we can turn this around. Social AIs— these are AIs that are embedded in social environments— also need to understand us so that they can interact effectively. And I think many folks in this lab have recognized this quite well. We wanted to empower these AIs to actually learn from their interactions with people. So they need to actually live in social environments. And then learn based on the feedback what are the norms in this social environment. So for example, and this is work led by Rajeev Krishna, who's now faculty at University of Washington, collaboratively with my colleague Fei-Fei Li. If you just take a visual question answering algorithm in computer vision, and you launch this thing on Instagram, and you want it to ask questions to try to learn about the visual world, it's going to ask questions like, is that food? Is the color white? How many branches are there? Are there more black balls on top of the table or pink balls under it? These are all questions that actually improve the model's performance. But as you can see, these people on Instagram just do not answer. These are not interesting questions. But if you start to actually model what people are interested in learning and sort of follow that manifold, this bot that I'm about to show you actually wound up interacting with over 200,000 people on Instagram. It starts changing the nature of the questions that it asks to jointly optimize for informativeness and sort of the social interest. So what's the— you can see it's asking about the bok choy, what type of dessert is this, what kind of bird is it, what's on the counter. It asks questions that people actually are much more likely to engage with. So what I hope to make the case for here is that by teaching these things to our models, we can actually learn quite a bit. And I hope you're starting to see the threads that are connecting all of this. Making this case that you can think about these counterfactuals, these second-order consequences, these reflective loops. Now you might think, and I think this is a reasonable critique, look, these large language models, they're black boxes. Black boxes don't teach us anything. And in fact, they can also make errors, which could be even more dangerous. I think there are, you know, what the sociologists have basically recognized is that there are kind of 2 strategies for making agent-based models. One is you have what you might call low-dimensional realism. It's like a very simple model, like Schelling's original agent-based model when he was studying social stratification. It's sort of like a single dimension, very easy to understand. The other approach is to basically kind of go maximalist on this. You aim for high-dimensional realism. You have this very complex model. It's fairly uninterpretable. You don't know why the agent behaved the way it did. You're not really easy— you can't easily attach the behavior to single parameters. I'm not claiming that this maximalist approach is perfect. Far from it. And I think we're going to need to keep doing a lot of work to actually build models that more and more effectively simulate human behavior. But I am claiming that there are a lot of questions that can be investigated through the development of this, through this— what you might think of as a sort of a general sociocognitive unit that can behave broadly like a person might. That, I think, is a research agenda worth pursuing. And I think in order to do that, we have to start to transition from believability to accuracy. And there are, in fact, many researchers who are starting to do this. There's work in PNAS. John Horton has a white paper in economics having these agents play simulated economic games and so on. One question is, do we just do this by trying to replicate known behavioral science results? Is that how we actually do this? I sort of came of age through the crowdsourcing literature, and this was effectively what the behavioral scientists did is they basically ran their famous studies again, but on Mechanical Turk and so on. Now the problem here is that maybe the models have just memorized the textbook, like literally they've read all of these studies and they can just duplicate it. So that's one thing we're going to have to figure out. What we've been thinking about is, again, as you might have seen me do over and over here, is to model individuals rather than populations. I think that that's going to be the approach that's going to be the most productive. Rather than saying the average effect is this, I want to actually anchor this to a specific person and their response. That's what we're working on now. And again, my hunch is that by building these kinds of agent architectures, it's going to make agents less vulnerable to the new p-hacking, which is going to be prompt hacking, right, where the— unintentionally maybe even— some small linguistic thing about the prompt that I use to run my simulation determines the outcome. I think we need to develop ways that the policymaker or the scientist can sort of tie their hands behind their back and be sure that they're not doing the thing by accident. I also think that we have 25 agents, but a lot of what people want to simulate here are cities or societies. That's going to need more like 25 million agents. So how are we actually going to be able to enable that kind of simulation remains an open question and one that I'm excited to work on. So I will stop here. I hope that you're taking away sort of the kind of excitement that I am about the opportunities, the challenges, the risks, and how to mitigate those risks with creating believable simulations of human attitudes and behavior. I want to thank the PhD students and undergraduates who I've worked with on this stuff, my colleagues, supporters, and you all for spending part of your morning with me.

Thank you.

Thank you. So I think we have 10-ish minutes. Is that right? Yeah, I'll start that way, and I'll swing around. So go ahead. In this experiment where you created a party and the agents that said rejected coming to the party, was it because it was automated response of LLM or because they were busy indeed and it conflicted with something or they lied? That's a great question. I actually don't know off the top of my head. I would need to go back and check. So the response from Rajeev indicated a sort of general busyness. You know how sort of like if you email a colleague and you say, hey, can you come to this seminar? Sometimes they'll say, I'm teaching then, and sometimes they'll just say, I'm sorry, I can't. This was more in the I'm sorry, I can't response, so I'm going to guess that it was more sort of dispositional. But I would need to go check. I don't actually have an empirical answer to that. The next question was like right next to him.

Yeah.

Can we do the microphone, please?

Oh, yeah, yeah.

We can run it around. I'd like to get in here as well.

Sure.

So you referred to causality and counterfactual a few more— a few times. And it sounds like that's the main—

Nope, never mind. I was going to try that, but not going to do it.

That's the main motivation of this work. We know that the only way you can get Real causality is through experiments, randomized experiments. If you want to use observational data to derive causal relationships, you have to do causal inference. You have to make sure positivity, exchangeability, sort of consistency— these are assumptions that have to be satisfied. And I don't even know how you begin to probe those assumptions in this case. And I guess the main problem I see is Unlike a classification where you have standard metrics you can measure, causality, there's no metric. You cannot measure how much causal your findings are. And so how do you even begin to claim that this is actually measuring any causal thing, given the fact that LLMs are very biased? Obviously, they model us. They're not even close to random.

Randomization.

So let's set aside the simulation for a moment. If you were to run an experiment on people, right, what do you do? You bring in people, you randomize them to condition. Randomization is the key requirement to satisfy those criteria, right? So typically when people run simulations, agent-based or otherwise, they are going to run the simulation a number of times, restarting it and randomizing some intervention So you can imagine that, I don't know, Sam is running for mayor versus some— or there's a mayor versus an, you know, an election versus an autocracy. And we just set that randomly in the simulation, one version versus the other, and you run this a number of times and you observe this distribution of outcomes in each case. This would be just like what you would do with a traditional human subjects experiment. Now if you wanted to remove randomness entirely, I don't think you need to do that in order to gain a causal interpretation of the situation, But if you wanted to, I suppose, especially if you had your own model, you could fix the random seed so that you get the exact same output each time, so that you know that the only thing that's varying is the intervention you're making in the environments. But in general, my understanding, to the best of my understanding, in the agent-based modeling literature, it runs Monte Carlo style, many simulations where they're manipulating some aspect of the environment. Did that make sense?

Yeah.

That's random. You're not making random decisions here, right? You're asking the LLM, this is the situation, what does the agent do? Maybe I misunderstood.

I may be misunderstanding your question. But typically what you're going to do is— I'm not random. In a human subjects experiment, the randomness is exogenous. You're going to insert the randomness into the environment, right? So either I get this intervention or that intervention. And it's just like that. So I'm going to intervene on the environment and either, I don't know, this town is going to randomly either be an autocracy with Sam in charge or a democracy with Sam as mayor and we're going to observe downstream what the effects of that are. I think I'm not understanding exactly where the objection is. Maybe this is something we could take offline and I could continue to have a conversation with you.

Sure.

I think there was a question over there. Yeah, go ahead, in the red jacket. You can repeat maybe.

I don't know.

Oh, sure. Yeah, I'll just repeat the question. State the question, and I'll repeat it.

All right.

So I have a question in terms about building the AI agents which embody different values in it. So basically, we know that they will basically show different capability and growing idea of different values when we build bigger ones. And then let's say I want to deploy them for some social application. For example, I want to build a summarization chatbot. To summarizing what's going on in Israel and Gaza now, it's really hard for this chatbot doesn't have any perspective in it. So it may reflect some values for someone. So I wonder, how would you suggest that when people design things which may have sensitive outcome and then may show some values we don't even know how to evaluate? How do we do that?

Sure. I mean, I'm going to basically lean on Langdon Winner here and effectively imply that Every algorithm has values in it. You may just not be thinking about them, or you may not know about them. So my colleague Tatsu Hashimoto has done a bunch of interesting work. I think it appeared at ICLR this year, where they asked a bunch of these language models questions from Pew surveys. And they were looking at the sort of political alignment of these models, different models across different topics in our social world. So that's one thing you can do. Clearly, I think many of these platforms are trying to— value-align these models to not take positions on social issues. The open models are maybe less so. My take would be, certainly during the development process, you probably, A, want to bring in actual stakeholder communities to work with you that you can recognize these things. But also, there is an opportunity. Some of the folks who've been interested in our work have wanted to do it from the perspective of being able to run audits, so to speak, on their algorithms from many different perspectives to understand, who might be harmed or disagree with the kinds of decisions that the algorithm is making. So that's another opportunity. Now, the question you asked, which was about— sorry, I should have repeated it— which was effectively about how you know which perspectives are embedded in the model. So if you make a chatbot, is it going to be right-leaning, left-leaning, or take a particular position on social issues? I mean, I think that in general, there's an open problem with these models about being able to tell them what not to do. There's a very open world of knowledge embedded in these models. And so this is where constitutional AI with Anthropic and others are trying to essentially fine-tune out or just literally through prompting say, you may not respond to these kinds of things. I think that's going to be a very open and very hard problem for a while. And I think that in practice, it's going to hold back a number of institutions, civil society, and for-profit companies from wanting to deploy these kinds of things because they're worried about what it might do. There may be this memory of Tay, the Microsoft agent from a number of years ago, where it kind of went off the rails without people realizing. Yeah, go ahead. The agents are based on a persona, right? The paragraphs or whatever that you're talking about?

Yes.

Does the architecture and algorithm allow those personas to be modified as you lengthen the duration of the And if not, do the personas have less weight on what actually determines what the individual does relative to planning and reflection and all that over time as the simulation progresses? Yeah, so the question is, we started with this paragraph. As the simulation progresses, are they— do they continue to be anchored on that paragraph, or can they eventually leave it behind and grow as people, right? So we didn't explicitly design this into the architecture. But I can tell you what would happen. That paragraph gets sort of string split into a series of things that become memories in that memory stream. And memories are given weight proportional to recency. So over time, in this sort of exponential drop-off, memories that are not accessed or used will become less and less important to the agent. They're going to be less likely to be retrieved. So over time, the agent might, quote unquote, forget some earlier memories in a— and I say this in a good desirable way— if you wanted the agent to evolve over time. That said, I have not systematically evaluated whether that would happen. By construction, it ought to. But I don't know in practice. I don't have empirical evidence as to whether it would.

Yeah.

So with our practice to align LLMs to be kind of what we want them to— what we want ourselves to be, right? reflect our values and to be nice all the time. Don't you think that leaves a blind spot for us for really trying to simulate true humanity for bad actors? So the question is about, does the value alignment in the models basically prevent us from being able to simulate negative behavior? So I have evidence that this is actually the case. So this social simulacrum where we were simulating the subreddits, that originally was run on text-davinci-003, like 2 generations ago of the model, maybe 3. I don't know how you count this anymore. And there, it was very easy to get the model to generate troll behavior. If you now go to the ChatGPT or GPT-4 language model, and you give it the same prompt and say, be a troll, it really tries hard to avoid it. Interestingly, 3.5 Instruct actually is less so, as would, say, many of these open models. So I think we're going to run into this interesting challenge where if you want to have effective behavioral simulation You need a model that has not been value-aligned quite so harshly to avoid the negative. Like, these agents, they're very pleasant. They're not going to— like, they very much do not want to talk about racism, for example, because OpenAI will not let the models to discuss racism. But if you want to actually study racism as a behavioral scientist, then you need the model to actually be able to engage in racist behaviors. But if you have such a model, then releasing it publicly can produce other harms. So I think that that's an open question about what we ought to do. There are models in the natural sciences and the medical sciences of creating these kinds of approaches, but then you kind of have to apply for restricted access to them. That might be an approach. I also suspect that even if you just tried Llama 2 or something like that that's been less heavily value-aligned, you might also get that. But absolutely, what you're saying is true. And I also will add that I think it's going to continue to be a big tension as the open-source models don't go through the exact same kind of red teaming that I think many of the for-profit ones that might eventually have legal liability have to deal with. Yeah, Ben.

Well, trust and trustworthiness are really critical attributes for making a society function. And the estimation of trustworthiness of others is something that takes time, a good deal more than 48 hours, for example. And the ability to project trustworthiness to other people. I wonder if you're considering investigating those particular attributes as stable personality dispositional attributes.

Right. So Ben is asking about how we manage trust. Trust takes time to build. It can— hard won, easily lost. How do we actually investigate this in the simulation? So this is actually fascinating behaviorally because the organizational behavior literature has found there's a phenomenon known as what's called swift trust. So this is, I have a whole other brand of research that's about teams and how to help teams collaborate. Swift trust means you get put into a team and very quickly you begin to trust each other. Now that trust can be brittle, but it actually doesn't take a long time for people to build up trust unless they already, unless they come in at low levels of trust. trust. So if you have a very antagonistic debate, you know, you're coming in from that perspective, I'm coming in from this perspective, then yes. But like a work team or something, or I move into a new neighborhood, swift trust. So what I don't know is how actually well that's modeled. And I think that this might get a little bit entangled with the general friendliness of the models. They might be more likely than average, until we test this, I'm not sure yet, To kind of be forgive— to have forgiveness. So we have not been directly modeling this nor evaluating it yet, though I do think we ought to. I wonder if we do need to explicitly model it. I might hope that we wouldn't, but maybe find a way to actually build that into the sort of reflective layers of the agent architecture so that over time I'm reflecting on, oh yeah, Ben and I, we've known each other a long time. And we're building up a relationship. But as you said, our simulation only ran for 2 days. We just sort of inserted the trust into the model by saying, you're married to this agent. And then it learned from there. Cut me off whenever you want.

Yeah.

So in your village simulation, I think you're adding memory to the agents. But the type of memory, I think it's episodic. But there's also this semantic memory, which I feel like is kind of hardwired into the weights of the models. How do you think we can kind of model these 2 types of memory differently?

Yeah, that's a good question. So episodic memory clearly happens here, although I don't know that we have very many specific episodic memories in here. We didn't say, like, yesterday, John did this. That does— those memories do arise through the simulation. So it's full of episodic memories by the end of it. These ones I would not say are as much episodic. They're more sort of like state, and some of them are dispositional, right? He's always looking for ways to make the process of getting medication easier for his customers, and so on. So I would say that we didn't explicitly model one versus the other. The reflection process tends to produce more stable dispositional memories. positional reflections, like, I'm dedicated to research, or, you know, you and I are very strong friends. Now, we don't give them attitudes. I'm sorry, the question was about how to complement the episodic memories with other kinds of memories.

The—

you'll notice we don't say, like, this guy is an asshole, for example. Now, we have, it's not reported in this paper, but we have done this. Like, if you, for example, if you tell an agent that they're an iconoclast versus they're very easily influenced by others, just to sort of test the very traditional Asch conformity experiment kind of things, they very much reflect those. So the iconoclast, if you're like, your friend got oatmeal for breakfast, the iconoclast is like, you know, forget that, I'm gonna go, you know, eat some—

Bacon.

metal screws or something like that. Like, the agent really does push something that's not expected. And then the one who's easily influenced is like, well, I'm going to get some oatmeal. So those kinds of things are easily managed by the model. And I think the other part of your question, which I also— which I agree is a good one, is how do they build up those representations, right? How do they evolve over time?

I think we'll have to end there because our next session is going to start in 5 minutes.

OK.

And, uh, uh, plenty of time to talk, uh, afterwards offline. So let's thank Michael once again.

Thank you.