Interactive Simulacra of Human Opinions and Behavior

Is it on?

Welcome, everybody. Thrilled to have you here at this week's Cyber Policy Center workshop and lunch. We are honored to have Professor Michael Bernstein of our Computer Science Department here. He will be talking about Interactive Simulacra of Human Opinions and Behavior. And I will trust all of you to Google each one of those words that's in the title if you don't know it already. And as traditional, he'll talk for half hour or so, and then we'll open up for questions. Those of you on Zoom, if you put the questions in the Q&A, then they will make their way to my cell phone where I will ask him about them. Turn it over to you. Thanks again.

Thanks all. So I'm a social computing designer, which means that I'm interested in how we build technologies that bring people together online. That ranges from social media to crowdsourcing to Zoom, even, the things that you love and that you hate. And so my talk, as you'll see, is largely around interventions we can make in this space. What are sort of this combination of design and technical interventions we can make that might move the bar on some of the challenges that we face? And I'm going to focus on AI. AI has had some attention recently. My claim is that in general, we've largely, in these sort of spaces where AI intersects social or societal issues, we've largely designed it as sort of like this hermetically sealed box. It's there on its own. Over time, of course, we've started to realize it's not sitting there on its own. We have to ask questions like, who are we helping with this AI? Who's it going to positively impact? Who's it gonna negatively impact? And we've made a lot of progress in that space. It's not solved, but we've started really thinking about it. At the same time, there are many other questions that I think we've barely cracked open. For example, whose voices is this AI representing in the first place when it's replicating something? Whose training data is it even trained on? Or even further, how is this AI embedded in this sort of complex tapestry of all of us? And how does it interact with us? Now, today I want to unpack this kind of observation in a couple of different projects, asking this question of how AI integrates with our societal and our social contexts and how we can do that maybe a bit better instead of just viewing it as this hermetically sealed box. So I'm going to take this into sort of 2 projects, and I can kind of DJ this to spend more or less time on either of them. On one of them, I'm going to talk about new metaphors for AI, especially focused on social media contexts. I know this being a topic of my own intense personal interest and also maybe some others in the room. I will also talk about ways in which AIs might empower interactive applications that are maybe a little bit more— that help teach us something about social norms and interaction. The first one is called jury learning. The second one is called generative agents, and we'll just dive in. I'll start with jury learning. This is work led by my PhD student, Mitchell Gordon, and it's really taking this question of what we do when the AIs that we're training are operating over spaces where there's intense societal disagreements. I don't think I need to convince anyone in this room or in the Zoom that AI classifiers are all over the place in our social environments. They're doing things like toxicity detection, detecting misinformation, aiding moderation decisions. There are APIs. Here's Google's Perspective API. But, you know, Facebook is using AI to detect COVID misinformation. It's all over the place. And when you use one of these kinds of things, this is an example of the Google Perspective API. You feed it some text input, an entirely accurate sentence up here at the top, and at the bottom, It gives you a probability, essentially, that this text is toxic for its own definition of toxic. And that one might be clear, but there are many cases that might be less clear. For example, here's a comment from the dataset we've been working with. In fact, a dataset that Google itself helped gather. You can imagine how the rest of this goes. It starts out, you know, oh my God, people still eat at Pizza Hut? Gross, too. And then it's shameful how this country blah, blah, blah, blah, blah. You can probably generate the rest of this comment yourself. Now, it's clearly aggressive. Does it cross the line into something that automated AI systems ought to remove is the question that I pose. It turns out that it depends on who you ask. Let me pull up the covers a little bit on this. How do we gather the data that these models are trained on? Well, you go ask a bunch of people, you call them annotators, you pay them probably not enough money, And then you ask multiple of them, and then you're gonna aggregate their opinions into what you call a ground truth label. So if you asked some set of people, should this be, is this toxic, should this be removed? I'm just representing them here by colors. 3 of them sort of responded with the gray color, 2 responded with red. You're then gonna do something like a majority vote, and then you're going to say, well, 3 people said gray, so we're gonna go with gray. We're gonna call that the ground truth, right? This input is going to be gray. And so we do this over and over again, and we have a whole bunch of these aggregate labeled examples, some that are gray, some that are red. We feed that into your favorite neural network, has lots of lines and circles. I'm not going to go into too much of the detail here. And you use that to train the machine learning algorithm, right? So all of these colored examples are used to tell the algorithm what the correct answer ought to be, what its behavior should be. The problem is that for many tasks, even experts disagree about what the right answer ought to be. So if you have a sort of a task that's the kind of thing that was very common in machine learning when it grew up, something like, is this a cat or a dog? Everyone here is probably equipped with roughly the same perceptual equipment and is hopefully going to say that it's a cat. Right. So on the other hand, something like toxicity detection, the comment that I was showing you, you may actually have far more disagreement about what the correct answer ought to be. Does this cross the line or not? The issue is that this mechanism of majority vote that I was describing that we use all over the place in crowdsourcing, data annotation, and machine learning silences minoritized groups. Because when you have this irreconcilable disagreement, who you sample matters, right? Who you ask matters. So here, if we have 3 people saying it's not toxic, 2 people saying it's toxic, well, typically those that are, that are minoritized groups are often subjected to harassment much more commonly and are going to get overridden by this majority vote, right? So you do the majority vote and the majority group simply just says single label, not toxic. This is how AI often works today. Issue is that this disagreement is all over the place. If you go look at datasets, comment labeled by majority vote as toxic in the dataset that we are using, again, a very large-scale used dataset, 40% of the things that are labeled as— excuse me, 40% of the annotators disagree on average with anything that's rated as toxic. In addition, If you go look at Reddit, for example, Reddit moderators disagree with each other about a third of the time on what ought to be removed. And somewhat off the record, in conversation with a large social network that has to do fact-checking, they feed items to these fact-checkers. Is this article— is there false information in this article? They had a bug. In the bug, they fed the same article to the same fact-checker again later. Turns out the fact-checkers disagreed with themselves a third of the time, right? Same information, different answer. So disagreement is all over the place and it matters, right? Again, not just social media, medicine, you might have different specialties examining different points of view. Design, you have different schools of design. Anything that AI is doing where there's expertise involved, there may be differences of opinion. So what we've been doing is creating something that we refer to as jury learning, where we change the metaphor a little bit of how the AI ought to operate. You could think of the metaphor now as saying, okay, here's a group of people. I'm gonna show 12 people here just because it seems to fit the metaphor. And you are going to articulate what proportion of that jury should represent each group or intersectional identity. So you're going to construct a jury by which that group in that proportion should determine the AI's behavior. So in this case of toxicity or online harassment, you might want to, for example, define a jury that represents groups who are commonly targets of harassment. This is, at some level, up to the system designer to define the jury, but as I will show, when we ask people to actually think about this, you tend to get shifts. So this changes the semantics of the AI a little bit. The AI's behavior now starts to read something more like this. For the following jury that has the following composition, this percentage of them agree that the comment is toxic. Right now, this sounds like something where we're going out to an actual jury of people for every new decision. That's not what I'm implying. And I'll explain in a moment how we can do this, take this metaphor and apply it actually at scale for AI. So essentially what we're starting to do here is modeling individuals, people, rather than some sort of like aggregate vote pseudo-human. So you take the input, we have a bunch of people in the dataset, and what we're gonna start doing is using AI to actually predict what each of these jurors individually would say. So we actually, rather than trying to predict the aggregate outcome, we're gonna say, what would each person say here? So each of them is going to produce what their opinion would be, which we will then aggregate again into some sort of vote. So in practice, it might look something like this, where you specify what the jury might look like. Now, there are— I'm showing here this sort of a top-down decision on what the jury's composition is. You can imagine in many scenarios you might actually want to follow a structured process for how you would select this. But in the following case, just showing what you might do, you might give a certain number of positions to various kinds of characteristics. Here there's 4. Then we have a huge number of people in our dataset. Some subset of them match any of these given characteristics. We're then going to sample randomly from those groups. We're gonna call on them to do jury duty, so to speak. And we are then going to, for each of them, these are all people that we don't need to go back to, we just sort of had them in our training dataset, we can predict what each of those individuals that we randomly selected would say for that new input. So in this case, it might vote not toxic. Or if it's some sort of like continuous number, like how toxic on this is on a scale from 1 to 4, we can give a number. So one thing that's hard to do in real life, but is very easy to do when you're modeling and sort of simulating, simulacra, you can sample many different juries, right? We can do this many, many different times and get a sense of what each of these juries might say, would give you visualizations, sort of a more straightforward interpretation of what the AI is telling you, how likely the result is to have done this way. So we take a median of means estimator, which just basically means, we take the median jury in this case. That's a value-laden decision, but it is the one that we go with for our particular implementation. So here, this median says, give it a 0.99, but each of these other juries are different instantiations, different groups you could have sampled. So again, this gives us some notion of sensemaking, some notion of explainability. We can say, well, it's not just like a single— each identity group doesn't just think one thing. There's actually disagreement even within groups about what people think. We can visualize this for the person who's examining these decisions. We can show you which of those— where the people that were sampled for your jury sat on that axis. I'm going to go very, very briefly into what that in practice means. So I'm gonna unpack this circle here where I say AI predicts each juror's vote. I'm not gonna get super technical here, but I want you to understand at a high level what's going on. Right? So we have a dataset that looks like this. We have a bunch of pre-collected data from people. So there's people sort of going on the columns there, and then inputs that they've labeled on the rows. And not everyone has labeled every input because they have finite time. We now have a bunch of missing stuff, and we have a bunch of people, and then potentially a bunch of other things that we want to predict for. So we know characteristics of the people, and then now some new comment comes in. And for the new comments, we want to say, what would the following jury say for it? Now, it turns out that this problem actually looks a lot like the problem that Netflix has. It's called a collaborative filtering problem or a recommender system problem. If you like machine learning, you can stare at this diagram. If you don't, just listen to my words. Essentially, what we can start to do is we can pay attention to patterns between how people vote. So if Nate votes in a particular way, it would suggest something about how I would vote, right? And so what we can start to do is observe these patterns, and thus when we sample you, we can sort of share information across people to make a guess for any individual annotator. In practice, what that means is that we look something— we look at the content of the input, We look at how similar people are to each other. We look at the characteristics, like again, any identity characteristics that are important for this task. Then we can start to actually make predict for this input, what would this person say? Now, can jurors be biased? Yes, absolutely. That's how it works. But if you gave me a choice and you said, look, option 1 is you can go do this fairness debiasing, post-hoc, in your model, or you could try to select and train appropriate voices. My vote is on the latter. That is to say, there's a lot of work in AI fairness. I think that's all important. But I think if we start upstream and look at whose labels are we listening to, you can actually solve a lot of those problems both representationally and in terms of understandability much more effectively. You might also ask, who gets to decide this jury? This is an exercise in power now. Well, one thing I'd point out is that every dataset, every AI you're using now already has a hidden and implicit jury. You can actually back solve for it using our approach. Tatsu Hashimoto had a really interesting idea for how we did this. The dataset we use turns out about 74% white. These juries already exist. We're just arguing to make it transparent and explicit rather than happening implicitly in the background. Now, I want to show the kinds of things that happen when people start using this. We recruited a bunch of moderators from online platforms, Reddit, Discord, to author juries for this similar toxicity task that I was just describing. We measured a bunch of things. One was whether jury learning represented more diverse opinions than the state-of-the-art classifiers. If you look at that representation, again, the baseline here is 74% white. Not too hard to beat that, you might hope, but we'll see if they do. Second, does this actually materially change the decisions that the AIs make? What I'm going to show here is a graph. The red lines for each of these groups are the representations in the baseline dataset. This one right here is where I'm getting the 74% white from. The blue are the proportional representations in the juries that our participants authored. So what you can see ends up happening is that there's increases in representation across nearly every group, except a reduction in the proportion of white individuals, people who look like me, in order to account for it and make space for it. This leads to material changes in the classifier behavior. In fact, about 14% of classifications end up changing compared to the current state-of-the-art classifier as a result of changing police. Suicide, LGBTQ+ issues, mental illness, police. LGBTQ+ issues, mental illness, police. On the other hand, there are things that are not likely to flip either because they're uncontroversially fine. Everyone's like, yep, thank yous, happiness, that's okay. Or that they're uncontroversially nearly everyone wants to get rid of them, racial ethnic slurs and the like. That's jury learning. We do all the discussion at the end?

Yes.

I'm going to change gears substantially here from something that's trying to ask how we might redesign the existing systems to something that's a little bit more speculative and forward-thinking about how future systems might behave. And this is about how, if we get AI systems to pay attention and learn what— how social interaction works, so to speak, we might create new kinds of interactive opportunities. This is through a paper called Generative Agents. This second piece of work was led by my PhD student, June Park. It's currently under review, although for reasons I'll explain in a moment, that review is somewhat now less anonymous than I was initially expecting. I'll just dive in.

Okay.

There are a number of applications that have been envisioned over the last 30, 40 years in my field that draw on some notion of simulating believable human behavior. Now, I'm going to separate here completely accurate from believable. Disney cartoons are believable, but they're not entirely accurate. These range from being able to do things like simulate how a social system might look, how might this social network behave when a whole bunch of people show up or when the trolls arrive, to, in my own field, cognitive models, agent-based models, people trying to create virtual worlds that feel compelling, social robots, interactive training. You might think of Jeremy Bailenson's work on Striver, trying to train people for rare situations. But the challenge has been resilient, that human behavior is really complex. It's really contingent. We can't really model it effectively. I've even tried in the past. We went and got a huge dataset of modern fiction from Wattpad and tried to learn about human behavior from that. It taught us some stuff, but it was resiliently challenging. On the other hand, we've seen a lot of movement recently in spaces like— you may have heard of ChatGPT. These will go under a heading that I'll refer to and others refer to as a large language model or an LLM. These things, what we found, can be prompted to take on a wide variety of backgrounds, experiences, and traits. So if I start you with, you know, this person has the following description, you can create many, many different agents, so to speak, that take on many of these different traits and different personalities, and then ask what would this kind of person might do in this situation. Now, it's obviously going to be generalizing, it may even be stereotyping, but as I hope to demonstrate, there are actually some spaces where this can be a— a really interesting and helpful and maybe fun application to explore. So we can simply prompt ChatGPT to do something like this. But things break down quite quickly. In fact, we initially built this as an application where, as I mentioned, we want to give you like a subreddit or something. And now you have an idea for a community. And now we want to show you what happens when many people show up. That's great if all you want is someone to sort of show up, look around, and then spout whatever's off the top of their head, so to speak. But if you actually want to have any sort of believable behavior, you need to deal with the fact that they're going to learn about each other over time. They're going to grow friends and grudges. There are going to be social dynamics and so on. So we ended up solving many of these issues and then putting a bunch of these agents into a little video game world here that's inspired by The Sims. Actually, I find it's something of a test Which game you think this looks like. It depends on when you grew up. But you can see each of these agents is sort of acting quite independently. It's living its life. It wakes up, it brushes its teeth, it goes to work or school, it interacts with other agents, it builds experiences, and so on. There are 25 of them in this little town that Jun calls Smallville. We call these generative agents. Generative from the the frame of generative AI, meaning AI that can create text, create images, and so on. These are agents that can draw on these generative models to simulate believable human behavior. They draw inferences about themselves. What am I interested in? What do I make sense of what happened yesterday? I draw inferences about other agents, my environment. I create plans that reflect my characteristics and experiences. What am I going to do today? What am I going to do tomorrow? I act out those plans, I react, I replan. Think of this as sort of like a little simulated world that you can go explore and where these characters are, in a sense, interacting with each other and just living their lives, so to speak. Now, as an example, we create a student athlete agent. In the morning, they wake up, they brush their teeth, they go for a run, they cook breakfast, they head to class. and so on. You can imagine if you change the description, that will change the behavior. At a high level, the thing that we do to make this possible is we sort of keep a really large sort of database, so to speak, of all of the agent's experiences. And then over time, we have to pull in the relevant subset, just like you probably do at some level, to say, given the following situation, what are the things that matter to this moment? And then over time, you're gonna reflect and build higher-level inferences about yourself and about other people. And you're gonna plan. So there's sort of this big loop that happens where you're gathering experiences and memories, you're reflecting on it, you're planning, and so on. So as I said, we sort of put this into a small game village. You can see that June made this manually in like a little— it's called Phaser. It's a JavaScript-based video game creation software. It has a dorm and a college and a grocery and houses. And it's basically what June thinks Palo Alto maybe looks like. And each house has bathroom, kitchens, common rooms, bedrooms, gardens, bookshelves, tables, and so on. So it has objects in it. There's sort of the whole space. There's rooms. There's objects. Each agent is defined. You can see the 25 of them here. We bought a little sprite package, and then we created a single paragraph for each agent. This sort of is their initial memory. It sort of says, who am I? Who else do I know? What do I do? What am I interested in? So here's John Lin. He works in the pharmacy. And then he has a family, and it says a little bit about them. The agents interact with each other basically entirely through natural language dialogue. So here's Isabella. They're discussing who to vote for in the upcoming mayoral election. She says, I'm still weighing my options, but I've been discussing the election with Sam Moore. What are you— or so Sam Moore is running. What are your thoughts on him? And then Tom says, I don't know. I don't know if I like Sam Moore. I think he's out of touch with the community, doesn't have our best interests at heart. A truly realistic discussion of mayoral— of local politics.

The—

they can interact with the game world. So they— the agents output text that says something like what they're supposed to be doing. I'm drinking coffee. That's translated into concrete movements in the game world. We can talk about offline exactly how that works, but it produces little— we translate it into emojis of what they're doing. They move around the game world in order to do this. The user can control it. They can do it through dialogue. So I can pretend to be another agent in the space. I'm a reporter who's running for office. And then the agent will respond, I heard Sam's running for office. Interestingly, you can sort of influence the agent behavior by saying that the person who's talking to you is now your inner voice. So you say, your inner voice says, John, you're running for mayor. And then John starts going and telling his friends and acquaintances about them. You can also imagine interacting with this by manipulating the game world. So I sort of interject into the world that the toast is on fire. And then Isabella runs to go put out the fire. I can also directly control an agent. Like, I can sort of teleport in to become an agent and then interact with the world as you— in the ways you might expect. So just to give you a sense of what this is actually like, here's the Lin family. They live down there. So in the morning, they go through this morning routine of the sort that I just described. They'll chat with each other. They'll have conversations. They'll pack and get ready to go.

They'll go—

you know, John will go begin his workday at the pharmacy. So John starts by catching up with his son, who's a college student, in the morning. So it's like, hey, Eddie, did you sleep well? Yeah, good morning. I slept great. That's good. What are you working on today? Eddie mentions he's working on something for his music theory class. This gets inserted into the memory so that later, when Mae, his wife, wakes up and asks about what's going on, that memory gets retrieved and conditioned into the language model. So he starts saying things like, yeah, he's working on a music composition. And Mae now has that in her memory. Oh, that's wonderful, she replies. Tell me about it. Now, you see it's a little bit stilted, the dialogue. I think that ChatGPT is actually, in an effort to try to be aggressive and problematic, it actually ends up being very pleasant and compliant. Information diffuses amongst the agents. Sam says, hey, how's it going? What's going on? Actually, I'm running for mayor. Tom says, oh, okay. He learns this and reacts in a reasonable way. And then later, Tom meets someone else, and you can actually see that Tom remembers that he's learned about this, right? So their memory is retrieving this. And as you can see, the information is diffusing around. Agents will recall experiences. So Sam's taking a walk. He encounters LaToya. LaToya says she's working on a photo project. Then when they see each other the next day, Sam says, oh yeah, how's your project going? He remembers it. So emergent stuff, coordination. We initialized one character with an intent to plan a Valentine's Day party. There are lots of points of failure here, potentially. The agent might not remember to tell anyone, may not think it's important. The people who she tells have to remember to show up, and they have to decide to show up. But despite all of this, from that single seed, there indeed happens to be— occurs the next day a Valentine's Day party where Isabelle invites friends and customers. She spends the afternoon decorating, gets another agent to help her decorate. And then another character actually asks her own crush, Klaus, to go to the party with her. So I'll very briefly sort of describe how this works, because I think it's helpful when we discuss, like, what are the possible— the threats that this might give rise to, or what the applications are. It's helpful to understand what's actually happening here. The challenge is, in general, if I just record a complete record of everything that happens, it can overwhelm these language models and it wouldn't even fit into the input that you can give the language model. What we end up doing is, yes, we keep track of everything that happens. It's like a complete memory, unlike mine, which is just like a sieve. Then we retrieve them and we do nothing super smart here. There are 3 characteristics we care about. Which is basically, did it happen recently? Is it important to this agent? And is it relevant to the current context? So recency, you can just sort of imagine that it falls off over time. Importance, actually, you can just ask the agent. It says, you know, this person broke up with you. How important is that memory to you? And it says, oh, that's pretty important. But it's like, you just made breakfast. How important is that? They say no. Relevance is sort of, you can think of this in technical terms, it's called a cosine similarity, but it's essentially a question of, you know, If you're doing breakfast now, you'll retrieve other memories about breakfast. But if you're, I don't know, going for a jog, breakfast doesn't matter. At the same time, if that's all we do, the agents struggle to generalize or make any kinds of inferences. So we have to teach them how to reflect. So that's what this reflect loop does. Basically, we take a bunch of things that have happened to this agent, and then we again ask the language model, well, what are some higher-level insights that you can infer from this? So he's writing a research paper, he's reading a book on gentrification, he's conversing with Ayesha Khan about exercise. Et cetera, et cetera. So again, this is showing June's bias. There are a bunch of researchers in this, in this world. Klaus is a graduate student. And over time, there's sort of this tree that forms of higher and higher level reflections that the agent makes about itself. At the top, you can see that Klaus Mueller believes that he's highly dedicated to his research. Props to you, Klaus. Whereas down at the bottom, they were the direct observations that led to this. Finally, there's this question of planning. I can react immediately to a thing that just happened, but how do I actually have a path through the world that makes sense? Essentially, we just sort of take these summaries of the agents. We say, here's their current status, and then say, here's what happened. Here's what you did yesterday. Tell me what your plan would be for today. So that will produce a high-level sketch, and then we sort of dive in and say, okay, for this hour, Give us more detail. And then we dive in again and say, okay, now for the next few minutes, what's that going to be? And that's how we produce the sort of moment-by-moment behavior. Now, what might happen as I'm engaging the behavior is like I see the world. So I don't know, maybe I see Nate raise his hand or something like that. Then again, we can just ask the language model and say, well, given this description of John and this thing that just happened to him, would John react or not? So, okay, I see a flowerpot. I'm just going to keep walking. But if someone stops and says, hi, the language model will say, yes, actually, I think John would react, and then you can replan if you need to. Are these agents believable? Is, I think, a reasonable starting point of an evaluation question. We did something weird, which is that since these things can generate language, we interviewed them. We did so in a structured way. We gave a bunch of categories of questions we would ask them. Like, describe your typical weekday schedule. Memory, like, who's running for mayor? You should know if you were told, right? You should be able to answer that question. Can you plan? What are you gonna be doing at 10:00 AM tomorrow? Your breakfast is burning. What are you gonna do? And these higher-level reflections, like, if you're gonna spend time with one person you met recently, who would it be and why? And then we can have the model answer all of these questions. We can make sure it's not hallucinating fake answers. And so it generates these interview responses. From the full architecture, but we can also sort of knock out parts of the architecture. You have no more memory, no more reflection, no more planning, and so on. Then we also brought in people and had them watch what happened and then roleplay as that individual. We then brought in a bunch of other evaluators to come and rank the responses from each of these different conditions. So say, which of these is sort of the most believable character? Again, sort of like Disney style. And given these rankings, these are ranks. How do you turn ranks into sort of what's better or not? You can do it directly statistically, but we also threw it into something that's effectively the same as like the chess ranking system called Elo. You might remember chess, like, you know, I play you and then you win and then Nate beats me because I'm terrible at chess. And then this gets turned into scores. So TrueSkill is just a generalization of this. Xbox actually used this for a number of years. Generalizes across multiple So it's more than 1v1. And what we can see is that each of these pieces of the architecture I described— observation, planning, reflection— are all contributing to the believability of the agent's behavior. In fact, if you sort of go from the baseline here, where we just knock out all of the parts, out to the top, it's something like 8 standard deviations of impact. And we can run the nonparametric stats on this and demonstrate that. It really does matter. Now, there are errors, right? Agents can fail to retrieve memories. I said what we did was pretty simple. Doesn't always work, right? If you ask Rajeev about the local election, he says, I don't know, I haven't been following the election. But he had, in fact, heard about the candidacy. We can go inspect the memory and see that he had been told about this. So this was an error of the sort that can happen if the agent doesn't retrieve the memories. They can hallucinate. This is a term you may have heard of with these generative AI systems, which is to say they can, invent things that didn't actually happen. Generally, they embellished memory. They didn't whole cloth make stuff up. Isabelle is aware of Sans Candace. She's confirmed. But then she starts adding, oh, and he's going to announce his— make an announcement tomorrow. This didn't happen, right? So she didn't make up something about someone else who's running, but sort of embellished what was in the memory stream. We also wanted to understand these sort of emergent social dynamics. This was sort of my— My horse in this race was understanding the way that this could teach us things about emergent social dynamics. So we actually took this 25-character simulation, we ran it for 2 full game days. Does it unravel into civil war after 2 days, or what? What happens? So we took the initial agent descriptions that I described, we included 3 little things. One, Isabella wants to plan this Valentine's Day party. 2, Sam wants to run for mayor. Third, Maria has a crush on Klaus. Those are sort of the only sort of things that we injected beyond basic descriptions. So information starts to diffuse. 7 people hear about Sam's candidacy. 12 agents hear about this Valentine's Day party. You can sort of see the graph of who told whom here. So some of it's indirect. I can't decide if what I'm about to show you is good or bad, but so 12 of them heard about it. 5 came. 3 of them said they were busy. So here's Rajeev, a painter, saying, I'm focusing on my upcoming show, and I don't really have time. 4 of them showed interest and then just didn't come. Is that the right outcome? I'm not actually sure. It does feel fairly representative of my experience trying to plan parties, however. Again, there are errors and things that we need to consider. When we deploy these systems. The dialogue, as I mentioned, feels very formal. This is likely due to what's called the instruction tuning on the models. What that means is that OpenAI gets a bunch of annotations of like, you should respond in the following way, and it learns a certain style. The agents inherit that style. They can also be overly cooperative. Again, I think due to this instruction tuning where ChatGPT is trying to be helpful to you. As an example, agents kept giving Isabella these suggestions about what to do at her party, including hosting a Shakespearean reading session or a professional networking event, and she just didn't say no. So at this Valentine's Day party, there was a Shakespeare reading moment. So they're sort of overly compliant. Now, I think that there's lots to discuss around the ethics and societal impact of this kind of thing. I'll tell you what we've been thinking about. One risk, sort of the movie Her, we can sort of just go down the science fiction movies and pick out the risks. So one is this potential issue of parasocial relationships, right? That people who use these might try to form relationships with these systems. I think the appropriate mitigation here is to do in some sense what we did, to make sure these agents always explicitly disclose their nature as computational entities and make sure that the developers value-align their models to avoid that. So you may have heard of Replika. This is an AI chatbot. People were using it to do all sorts of things. They recently actually removed that ability. There was actually a big uproar over this because a lot of what people were using this female Replika AI chatbot to do, you can imagine, and those people could no longer do that. Another risk would that be— there are other risks associated with generative AI: deepfakes, misinformation generation, tailored persuasion. Imagine if a set of people all come up to you knowing things about your identity and try to persuade you to vote a particular way in the election, for example. One thing that we think is important to do here is, as these kinds of technologies get developed, we need to make sure that the platforms are maintaining an audit log of what the inputs are and what were generated. This allows us, if we have some question about, is this actually a fake, to be able to go back to the platform and actually just look it up and say, this is in fact generated from one of these generated agents. A third risk, we've seen things like syntheticusers.com recently pitch, oh, you don't need to go talk to communities, just let us go interview these GPT chatbots instead. And I just wanna come out and be very clear here that agents are not a substitute for human or community involvement. There are situations where you would want to do this as a complement in a way that, as we did, You might want to say, here is my community now, help me imagine what problems might arise when this community gets large, as opposed to, I'm just not gonna talk to my community. I think that would be a problematic way to use these kinds of things. I mentioned it's less anonymous than I was expecting. The only reason I'm talking about it at some level is that, so we put this thing on archive while it was under review, and then AI Twitter found it, and then, There was a lot of interest. So Jun, the lead grad student there, you can see him on all sorts of places. So there's some folks that have found our paper. I will close here because I want to make sure there's time for discussion. I want to thank the graduate students, colleagues who collaborated with me on this, those who supported the research.

Thanks.

So that was wonderful. I have so many questions, but I also don't want to monopolize. So if people have questions in the room, I want to make sure I get to them. Simi, did you have a question? Why don't we— I want to make sure that we get everyone in, and that goes for folks on Zoom who can put the questions in the Q&A.

All right, this is fantastic, by the way. So my question actually combines the 2 threads of your talk. So You talked about jury learning, but an important part of jury is jury deliberation.

Yes.

As epitomized in like 12 Angry Men, that movie, right?

Yes.

So I'm just curious if you thought about combining these 2 threads of thought and have simulated jury deliberation and then have your juries kind of vote again after they've deliberated. Yeah, so we've definitely— we've made progress on that. It's not to the point where we published anything at this point. I think there are 2 different kinds of deliberation you might consider. One is sort of generative text, like actually simulate the actual debate. The other would be that you also want to get sort of observations and training data on if this person saw this counterargument, would it change their position at all, and so on. That side is where we had put our effort. I think we would need a much larger dataset of, you know, person's initial position, counterarguments, eventual position, and then we could start to do that more effectively. So absolutely, we are interested in that. We've made progress on it, but we're not done. And the graduate student who was working on it found something else really cool to talk about. So it's kind of paused. If you're interested, let's chat.

Yes, in the back there.

Thank you so much.

This was super interesting. I was wondering when you were talking about the generative characters embellishing their memories through hallucination, isn't that— doesn't that make them more believable? Because this is what humans do as well. It's— that's a great question. So what we had them do in the evaluation is we had people come in and they like watched like the blow-by-blow recording of the agent's experiences. So if in the interview the agent said something happened that didn't, this actually decreases their believability in some cases. I agree with you that there are situations where embellishment might increase believability, like if I'm telling a lie, one thing that Jeff Hancock notes is that, like, adding concrete details actually does increase the believability of the lie. People do this. In the case of our evaluation, that's possible, but it's also— Like, in the sort of scientific lab we set up, hopefully that was caught at least sometimes. So I think the hallucination has generally been viewed as a very dangerous component of these generative models. I guess in our context where it's— if it's like in a little video game world, you could argue that it might be less dangerous. Yet at the same time, a video game developer, say, if that's the application you're imagining, They may want to have strong control over the narrative of the game. And you don't want it to be that the agent randomly spouts off like, oh, I heard that, you know, blah, blah, blah, married blah, blah, blah, when like later in the plot that should be impossible. So I think that that remains, from my perspective, probably more risk than benefit, but I'm open to other opinions.

Well, but let me press on that a little bit, which is if you're trying to simulate reality, Yes. Right? And people lie and hallucinate in reality to a point. The question is at what, how much lying and hallucination, right? I mean, if you have one of your people who's on LSD, right, then we say of course they should be hallucinating, right? All right, if you have someone and you have a certain number of liars in society, and so at some point, right, now it depends what your goal is in constructing the society, But I think that's kind of where you were going, right? Which is that, all right, maybe actually it's not an error, it's just sort of a fact of life.

Yeah, so, okay, that's fair. I think there is, you know, we can talk about there's believability and then there's, let's call it representationalness or something like this, where, you know, if you were going to be used, like, we're not currently at the point where, you know, I would say use this as a, as a straight-up thing that would tell you exactly what's going to happen. We know from sociology that it's even underdetermined, right? Like, the outcomes of these complex emergent systems could go many different ways. But there's certainly one thread of interest we've gotten in response to the paper. One thing that's nice about all of the interest is that I know exactly at some level, like, what all of the— what many of the responses are. Yours was not that, which is cool. It's really good to get to get something new. This is why Stanford is amazing. But there is a thread that basically is a set of behavioral scientists who are interested in asking these questions of whether generative agents can serve as a sort of, as a way to run simulations for their studies. And I think if you're interested in a world in which, you know, you're studying that kind of fabrication, then I could see that mattering, especially if— if that's the material to your study. If it's not material, then it could produce sort of unrealistic social behaviors that might undercut what you're trying to study.

But this is where also I think Samid's question comes, where your 2 papers are joined, right? Which is that if, again, as you said, it depends on what purpose this little society is serving, right? But if you're trying to get a representative sim, situation to predict. I mean, take something like if we don't know how people are gonna respond to a particular COVID intervention or something like that.

Yeah.

You try to create a community very similar, let's say, to Palo Alto, and you say, all right, well, how will people respond to these interventions? Now, we're not there yet, right?

Sure.

But that's gotta be the future, right? Which is that we just to get some, back-of-the-envelope idea about how people will react. Now, the next stage, it seems, is it's not just you kind of inputting random stuff, but you input all of your email and everything that you've—

Yes.

So you create Michael Bernstein in that world, you create Nate Persily based on whatever information you have about me, and then we see, well, you can imagine how political consultants would do this, right, in order to predict how people are then gonna behave.

As you might imagine, we've had a few video game companies reach out to us. That has come up, like the sim me sort of, how would I live in this simulated by, I don't know, Connecticut Yankee in King Arthur's Court situation? But yeah, I mean, I totally agree. I think there's this big— there's going to— there is already a debate about the extent to which these generative models replicate human psychology. If you've tracked this, you may have seen there have been papers that try to take economics studies or cognitive psychology studies, and then prompt ChatGPT with the same inputs to see, does it do what people do? The answer is like, yes, sort of. I think what we lack right now is a concrete understanding of what are the situations where if you want a concrete, let's call it numerical or representative answer, what are the situations in which that would work and when are you way off the ranch? What we have seen is— well, I'll just leave it there.

Well, so let me turn to the next question over here, but before I do, just— this ties back into the jury issue because you have a certain theory of what representation matters, right? And so you're focusing a lot on race, which is understandable given the way we think of juries, for example, in particular, and race and gender, but it's not clear to me that you want the same jury for every one of these kinds of decisions, right? Or where you might put the line as to what, say, is toxic. So the jury for disinformation, right, might be different than the jury for toxicity, which would be different from the jury for child exploitation or pornography or something like that, right?

Yes.

And so leaving aside the deliberation possibility, how you might— but what types of people are the right ones to feed into this? And that does bridge to the next paper and the video game, which is, You know, how should we think about reflecting in the virtual world the true complexity of the real world?

Sure.

So the first piece of this, there was a slide that I went whoop and hid that is all about what we call conditional juries, which is that you, yeah, you may, you can very directly imagine saying like, you know, if topic is Me Too, jury should look like this. If topic is, you know, Black Lives Matter, jury might wanna look like that, and so on. It's relatively straightforward to do in that, like, just one of the inputs to our system is just what's the jury specification? And you can make that depend on anything that might matter here. Should we just jump to the other one?

Oh yeah, no, keep going. I was just telling her to bring the mic over here.

But yeah.

Yeah, about the, like, what is the desired representation? I think that's a broader discussion that's happening also in, like, the media, movies, animation, elsewhere. Like, what is— What is a representation that is sort of like the— I think what we found in our evaluation, I suspect that the moderators that we reached out to were probably rather left-leaning, if I had to guess.

Mm-hmm.

Like, we weren't getting moderators from The Donald, 'cause they don't exist anymore, but, you know, you can imagine. And there's sort of this— the theory of change here ends up being something like, if we ask you to imagine what this ought to look like, and get explicit about it, you tend to be a little bit more forward-thinking than if you just sort of do the— if you never stop to think about it.

Yeah, so sorry, I have a question on your first— on the first part of your talk. So to what extent do you think that complete randomization of the panel, of the jury composition, is achievable and even desirable, or conditional randomization depending on the subject. And to what extent do you think that in real life AI applications will try to signal that, you know, we use a sort of randomization pattern in order to really be, you know, trustworthy, you know, to link back to last week's talk, or whether we could even achieve something like a gold standard, you know, which is reasonably Randomized jury composition, something like that. So basically private responses to the problem that you identified.

Right. So this is getting into sort of the realpolitik of jury selection, so to speak. Like, would a big tech company ever— so we can separate that into 2 questions. Would a tech company ever do this internally but not say anything about it? Like, would they have debates internally? Would they use this kind of thing? And then separately, if they did it, would they ever disclose the composition of their jury?

Right.

I think those are 2 separate questions. One thing we found that is interesting on the would they ever use it front is that actually when you do it the way that I described, performance actually goes up. Why would performance go up? Well, it turns out that if you're building a prospective API toxicity classifier and you're asking it to just make these predictions of what the generalized pseudo-human would make, It's a much harder task than if you ask, what would this person say on this thing? What would this person say on this? So if I know who I'm asking, I can actually predict more accurately. So would they ever use it? I suspect that that kind of thing actually could yield benefits to them. Now, would they disclose it? I'm a little more cynical on that. I do feel like there's a sort of normative slant to the jury learning paper that says that we should be. Maybe there's— we've done some work that's more participatory in nature where we found that the model does allow deliberation and contestation in a way that I think is qualitatively quite different than if you're just looking at precision recall curves. If it's a community of people looking at false positives, false negatives, what ought we to do about this? If instead it says, who ought to represent the decision here, the conversation just changes.