Webinar: AI Agent Simulation of Human Behavior with Michael Bernstein

Hi, everyone. Good morning. Good afternoon. Good evening. That way we cover everything. We're happy to have Professor Michael Bernstein here for this webinar. So let me give you a bit more of his background. Michael Bernstein is a professor of computer science at Stanford University, where he is a Bass University Fellow and and an STM Microelectronics Faculty Scholar. His research in human-computer interaction concentrates on the design of social computing systems. This research has won best paper awards at top conferences in human-computer interaction, including CHI, CSCW, ICWSM, and UIST, and has been reported in venues such as the New York Times, Science, Wired, and The Guardian. Michael has been recognized with an Alfred P. Sloan Fellowship, UIST Lasting Impact Award, and the Patrick J. McGovern Tech for Humanity Prize. He holds a bachelor's degree in symbolic systems from Stanford University and a master's degree and a PhD in computer science from MIT. Please stay muted so that we can ensure good audio for Michael. We would appreciate if you could turn your cameras on so we can have an engaging and interactive session. And we encourage you to write your questions in the chat since at the end of the session we will have some Q&A segment. Now, if we're all ready, Michael, we are thrilled to have you here. So the floor is yours.

All right. Thank you so much for coming. I'm seeing, wow, a fairly global presence here in this webinar. I'm actually somewhat astounded by the number of you who are here. Over 500 of you. So thank you for joining me. We're going to talk about AI, and we're going to talk in particular about a frontier of AI agents that I think is particularly poised to have a lot of impact. And maybe you're an individual contributor, maybe you're a manager or a leader, or you're imagining that, or you're a policymaker. There are so many different applications of these kinds of technologies if we can get them to work. Which they're increasingly showing a promise of. So we're going to talk through what, what is this frontier and what's going on. When we make decisions, we make those decisions based on what I would view as often incomplete information about how people are going to react. Maybe you're at an organization, whether it's a for-profit or a nonprofit, that's trying to decide if we do this, maybe our, you know, our, our clients or customers will react this way, or maybe these people who aren't currently our customers will or won't react that way. Or maybe I'm a leader and I want to say, if I change my management style in this way or reorganize in the following way, maybe the organization will do better or worse. We make these decisions based on the best data we can, but that data is often incomplete. And this kind of incomplete information means that we often make bad bets. What I mean by that is that we make the best guess we can. We take all the information we can, but we're ultimately wrong. And this is not necessarily a problem with you. It's not that you're not smart. It's just really hard. We can't know in advance always what's going to happen because we have limited ability to get feedback. We can only launch once. And so we make these big bets and they misfire across these areas, across like consumer packaged goods, CPG, personal finance, product design, policy management. Even me as a professor, I do this all the time. I'm making guesses about what's gonna work with my students and not. And you think, oh, this is like a problem of like, you know, of modernity and so on. But, you know, like, this is not even a new question. This is a thing that goes all the way back to 1906. Here's Robert Merton, a famous, famous sociologist who pointed out even back then, over 100 years ago, how hard it is to design for what a group of people will do. Like, we all sort of want to get out of the city so that we can avoid the crowds, but then everyone wants to get out of the city and go on vacation at the same time. So we all wind up in the same, you know, the same forest, and it's too crowded there too. So we're very challenged in making these kinds of decisions. And I want to ask today, what if you had a what-if machine? What if you had a what-if machine? What would you use that for if you could sort of imagine in advance what might happen? You know, imagine that you could say, if we— if my organization took this path, how might our customers react? Or if you could say, you know, if we were to fail, or if something were to go wrong, what's the pathway through which that would maybe happen? Or you could say, well, What if we could see how people might react to this policy change or this new product or this strategy we're thinking about before we deploy it? If you had this kind of what-if machine, my claim is that we could make better decisions more often, much more often. And so this has led to a kind of a really interesting question for me as a researcher in computer science. Could we create simulated AI agents that replicate human behavior? Could we create AI simulations that behave like people do? Maybe like your organization would or like your customers would and so on. If we could, then maybe some of these things become possible. Now, simulation isn't a new idea either. It goes all the way back to 1978. Here's Thomas Schelling, won a Nobel Prize in Economics for generating this idea of what are called agent-based models. And these things are still in use today. And I'm showing a screenshot here of a recent model that was run on, like, supercomputers trying to understand, oh, if we engage in the following kind of intervention, does this reduce the kind of spread of a pandemic? And so on. There are these very simple algorithmic models of people— well, actually increasingly complex, but still fairly mathematical. But, you know, simulation is not just a thing that we use in policymaking. Here's entertainment. I bet a bunch of you have played The Sims. It is one of the singularly most popular video games of all time. These are also simulations of people. These people, these agents in The Sims, you can change the space around them, you can intervene on them, you can have all sorts of fun in this little dollhouse environment. Increasingly, if we are deploying AIs into worlds where the AIs need to interact with us, some people talk about AI teammates, What does that even mean? Like an AI that's on your team at work would need to start reasoning about how you might respond if it takes a certain action. And more broadly, to create a set of tools that are what I would term look before you launch. And I think we struggle with this. You know, for much of my career, I worked in the design of online platforms, social media, these kinds of things. And many issues would come up because we couldn't effectively foresee what was going to go wrong. And so this creates the opportunity to create tools that say, ah, what kinds of policy changes or rules do I need to set up before I launch the system so that it doesn't all become a dumpster fire? And then I set the rules afterwards. Now, the problem has then— has been that up till today, our models have been pretty rigid. That is to say, they like— you kind of have 2 options. One, you say, you know, a human is 5 parameters. That seems like it's not quite true, like we can't quite be reduced to 5 parameters. They can explain a lot about us, but, you know, I'm at least 7 parameters. It's essentially like this is just much too of a sparse model to encode the richness of human behavior. The other option is to do what The Sims does, where you basically write scripts and say, you know, if I get punched, then I fall down and I get mad. And there again, it's sort of limited to only the kinds of things we can think to create. And it's always going to be incomplete. So as a result, at least within the academic literature, the basic conclusion is— I'm going to quote here— the models have been highly stylized and have had minimal impact. So not a good track record. They're really interesting, but in practice, we haven't seen them taken up in many cases. And I mean, I'm sure some of you in this call have used agent-based models, but the vast majority of people who I talk to have not. But starting a few years ago, my colleagues and I recognized that modern AI models, so-called large language models like ChatGPT, Claude, Quen, LLaMA, all of these, DeepSeek, have trained on a variety of human behaviors, which is to say that they've literally read the book on all the research about how people behave because they trained on research. They've also trained on a bunch of social media data. They've seen the good, the bad, and the ugly of human behavior. As a result, what we recognized was that you can prompt these large language models to take on the perspectives of a bunch of different people with different backgrounds, experiences, and traits. So we could take ChatGPT or another large language model, LLM. We can give it a prompt that roughly equates to, you know, here's— be this person, here's a name, a description, and in this situation, how would they react? And by multiplexing those descriptions, we can create an entire crowd of different people. And if we create a crowd of people, you can sort of put them in many different situations and see what might happen. And in fact, we started doing this, and in some work that, shall we say, went a bit viral, we created almost like a little terrarium of many of these so-called— of these AI agents that are simulated people. We call them generative agents, and we created a little town called Smallville. This town is full of 25 generative agents. These 25 agents, as I would describe them, are essentially all completely autonomous AIs that are each sort of playing the part of a different person. So you can see the entire town is going about their day. You know, the artist in the town gets up and paints. The college students in the town, they sleep in more, and then they go to classes and do their homework and so on. This allows us to create a space where we can sort of visualize what it would mean to have these simulations of human behavior in a way that we can learn about people and interventions. So, in fact, this work got a lot of attention, and more recently, Andres and Horowitz— I think this was just last month— put out some research, you know, their own research, arguing that this is going to become the next sort of essentially generation of market research tools. And if you go check out this— oh, it was in June, excuse me. If you check out this article, then, you know, they cite our research quite centrally. So what I'm about to show you in this webinar is that AI simulation can create both believable and accurate simulations of human behavior. I'm going to give you a how-to guide of the techniques that enable this kind of simulation. I'm going to give you the gotchas, the things that often people get wrong that you should look out for when doing this. I'm going to talk about the current frontiers of this technology, like what's on the edge of just becoming possible. I hope that sounds good. If you're excited, I don't know, give a thumbs up in Slack or excuse me, in Zoom and we'll dive in. The basic way we're going to go after this is we're going to create an agent by first we paid an online artist to get a bunch of pixel art of different characters, which you can see on the left. these visualizations are there mostly to help you sort of see what's going on. You don't actually have to build a little video game environment like I did. I'm just a child, you know, I was born in the '80s, grew up in the '90s with, you know, 1990s Super Nintendo JRPGs. So it's just a cool way to show it. But essentially what we do is we create a little persona for each individual. So here's John Lynn. He's a pharmacy shopkeeper. We tell him that he runs the pharmacy in this town and we describe who he is You know, he's very helpful, and we tell that agent who he— what it knows about others in the simulation. So we tell him he's married to this other agent in the simulation, Mei Lin, and they have a son, Eddie Lin, who's a college student studying music theory. Now, we have to tell the agent about all of the other agents it knows in this environment, because if we don't, the agent's going to wake up in a bed next to Mei Lin at the beginning of the simulation and go, ah, Right? It won't know. It literally will not know that other person. We have to tell that agent everything that it needs to know at the beginning. Now, what I want to try to explain to you is, over the course of this webinar, the important pieces of what it takes to build one of these. I don't expect that you understand this figure yet as I, as I'm first showing it, but I'm going to build up to this figure over the next few minutes. So that you understand what it is that allows these agents to do what they do, and so that you can create them yourselves. Now, basically, what, what we can do is we just describe the agents, and we're going to just let them go from there. They wake up without us telling them exactly what to do. We just describe who they are. They'll wake up, brush teeth, take a shower, cook breakfast, catch up and talk to each other. They'll, they'll pack and get ready. And what I'm gonna do here is to the global alumni team, I can message you, but I can't message the, the people here. I just sent you to the global alumni team a link to a demo. If you can echo that demo link out to the group, they can actually load it up and see the town in sort of effectively real time. So I encourage you to go check it out.

Thank you.

Yes, you sent out the correct thing. So please go check out that, that link. Now, the agents then act in the environment because these are just AI agents, right? Which means they mostly just act in the sense that they, they use natural language. So they say, this agent, Isabella Rodriguez, is drinking coffee, right? We then need to ground those— that text into concrete movements in our little video game environment. So we say, oh, they're drinking coffee. Okay, they're going to walk over to this chair. And then we're going to render a little emoji that summarizes what it is that they're doing. So if you go and clicked on that demo, you can see the agents are often having these little emoji bubbles that show what they're up to. So we ground their, their actions, which start out as language, into this game environment. But you can talk to these agents. Literally, you can sort of intervene and talk to them. So if you pretend to be a newspaper reporter, then you can say, hey, who's running for office? You know, who's running for mayor? If the agent knows, they'll respond, oh yeah, I heard Sam's running for mayor. If you want to control these agents, then you can just pretend to be their inner voice and say, John, you're running for mayor. Then John says, well, I really need to talk to my family about this big decision that I'm considering. You can also intervene in the game world. I can take the toaster, set it on fire, and the agent will recognize what's going on, and run over to put out the fire and then go make something else for breakfast. We have all of these opportunities to think about interventions in the world and how agents might respond to them. Now imagine intervening and changing the layout of the home or the kinds of information they have access to or the kinds of products in their home and so on. Just to give you a sense of what this is actually like, Let me take it from here. These agents talk to each other, right? They're language agents. They're AI agents that can talk. So in the morning, John sees his son Eddie and engages in conversation, says, hey, good morning, son. Did you sleep well? The son says, yeah, I slept great. He says, hey, what are you doing today? And the son says, yeah, I'm working on this music theory composition for my class. Remember, he's a college student. He's a music theory student. So he's working on a composition, a new song. Now, John doesn't know this, but now he does. He has to remember it. Because soon in the simulation, Eddie's going to leave the house and John's wife is going to wake up and say, where's your son? And John has to remember what he learned so that he can tell her, yeah, he left. He's working on this music theory composition. And now she knows. And this is how information can sort of spread through the environment. Now, that's a simple example. But we wanted to examine a much more complex one. So here's what we did. We initialized Isabella, this agent who runs the café in town, with a simple intent to plan a Valentine's Day party. So the simulation starts on February 13th, the day before Valentine's Day, in the morning, and it runs through the end of the 14th. So we just inserted into her memory this little intent that she was interested in planning a Valentine's Day party. And what I want to convey to you is that nothing about this AI agent architecture that we're talking about today is like a party planning module. Nothing that says, oh, this will definitely work. And in fact, there's a lot of reasons why this might fail. This agent has to remember to tell the other agents about the party. Anyone who's told about it needs to remember that invitation. And those who remember it have to actually show up. And yet, from this seed, we start to see evidence of classic information diffusion patterns, which you might think of as like sort of rumour networks, whisper networks, just word of mouth. So Isabella tells people, people tell other people. You can see this graph of who's hearing about this party from whom as the simulation plays out. And in fact, Isabella goes and gets her friend Maria to help decorate the cafe in preparation for the party. Again, we didn't tell her to do any of this. This agent decided to go get help from the other agent. At the end of that day, like the next day, On Valentine's Day, half the town had heard about the Valentine's Day party. So of the 25 agents in the town, 12 of them heard about the Valentine's Day party. Of those 12, 5 came to the party. 3 said they were busy. And 4 said they were interested but ultimately didn't show up. So is this accurate? It's a little bit of an interesting question. It reads to me as broadly plausible, but it's not necessarily easy to tell, is this exactly how it would play if this were real life? It does, to me, seem like pretty compelling evidence of the kinds of believable behaviors you might get. In fact, one more kicker on this. We not only told Isabella she was interested in planning a party, we planted a memory in the Maria agent, that ML that you see on the left in the image, that she had a crush on Klaus, another agent. And it turns out that Maria asked her crush to the party. So we had a little nascent agent love here. So very beautiful. And in fact, researchers who are not us got this whole simulation running again just this, just this year and found that they could intervene and study different alternatives. So now they said, oh, okay, what would happen if suddenly the agents, some of the agents heard on the radio that there was a new communicable disease, like a new epidemic of swine flu. And you can see in that pathway when they ran the simulation, oh, no one showed up to the party except for poor Klaus. Klaus had not heard about the new illness, and so he showed up anyway. In the no-threat condition, the party happens as usual, and when the agents hear a radio about a non-infectious disease like diabetes complications or something like this, the party also happens as normal. You can think about how interventions into the world let you ask these what-if questions that can be quite powerful. How do we do this? I'm coming back to this image and I want to try to explain the major parts of this architecture. You don't have to be technical to understand this. I want to convey the main pieces. The first thing we need to do to create an agent like this is we need to give them the ability to remember. So we give them what's called a memory stream. This memory stream is essentially a blow-by-blow record of everything that the agent observes. So you can see a lot of this is actually not very interesting. They see a bed, they see a desk, they see a closet. Some of it is more interesting, like, you know, I'm stretching, I'm writing in my journal, I'm taking a break, I'm cleaning the kitchen, and so on. But there's a lot of stuff here. You could fit it into the context of a large— of some of the larger window large language models. But these models, the research suggests, can get distracted with very, very long contexts. So we actually do something that's referred to in the literature these days as RAG, or Retrieval Augmented Generation. So with RAG, what we're doing is we're preferentially retrieving memories that were Accessed recently, that are important to the agent, and that are relevant to the current circumstance. So recency means if it's something that you observed recently or that you recalled recently, that's more likely to come up. If it's important, like the fact— I don't know, the agent brushes its teeth, not very important to the agent. The fact that it got asked out to a party, pretty important. And finally, relevance. So if an agent is taking a math test, then other memories about taking tests or about that math class should be, should be top of mind. So it's almost like running a little bit of a Google search over all of these, these memories. And so now what happens is what you can see on the right. If we ask the agent the question, what are you looking forward to right now? We're going to retrieve these recent, important, and relevant memories, like things like that they're planning a party, that they're ordering decorations for the party and researching ideas for the party. Put those into the context window of the large language model along with the description of Isabella herself. And that's what gets Isabella to reply in a reasonable way, saying, I'm looking forward to this Valentine's Day party. Memory, the first thing you need. The second thing that we feel is important is this ability to reflect. If you have a psychology background, then you might be familiar with what I'm about to say, where the memory I just showed is, is what's called a purely episodic memory. It's just like a log of what happens. But if you think about it, we're more than just all captain's log. This is what happened. This is what happened. We needed to provide the agents with the ability to produce higher-level reflections of who they are, what they like, their dispositions, and their interests, and their goals. What we do is we essentially get these agents to have shower thoughts. Figuratively speaking, we— so what we do is at regular intervals, we pull memories from the agent's memory stream, and we ask the agent to reflect on those memories. Here's 2 examples to produce a higher-level reflection or conclusion about them. So given the 2 observations that Klaus Mueller is reading about gentrification, reading about urban design, we produce a reflection that Klaus spends a lot of time reading. And we can do this over and over again by continuing to group memories and reinserting those reflections back into the memory stream to produce higher and higher level reflections about who this agent is and what their goals are. This produces the ability for the agent to essentially produce— to produce behavior that's much more consistent with these goals, rather than just literally being a robot and walking from step to step. The third thing is planning. So planning is important for these agents to remain believable over long periods of time, and planning is something that is not unique to this project. It's been a long-time issue with, with AI. So what we would do is We would start by having these agents plan out their full day. Then we would drill in, and given that day, say hour by hour, what are you going to do? Then we're going to drill in again and go minute by minute and say, okay, in details, what are you doing? And when the agent sees something as they're acting in the environment, like here, the John agent might see Eddie, his son, taking a walk around his workplace. Then we will ask this agent, given its background— you can see the prompt here on the left— if it should react to the observation. If so, what would be an appropriate reaction? And then replan if we need to. So this is how the agents adapt as the simulation runs. Now, this work got a lot of attention. It really caught fire with a number of people, and it was really interesting to see. And I think I just want to start by suggesting, like, there's some evidence that there's, there's something to this idea of the goal of creating these simulations of people and, um, and what, what that might enable. You can see here, um, news reports that are featuring Joon Sung Park, my PhD student who led this work. Um, and, and yet with this vision, I haven't fully answered the question, right? I asked this question before. About whether we can create simulated AI agents that replicate human behavior. Everything I've showed so far shows believability, but Disney characters are believable, cartoons are believable, but they may not be accurate. If you're going to make decisions based on this, then there's this question of how accurate are these agents at replicating behavior? That's an empirical question that we really need to ground out if you're going to think about whether you want to be using this. So, how do we measure this? Because there's papers out there that just try to replicate this study or that study and say this one works or that one doesn't. But, you know, others then try to replicate that and show a different result. How do we act like you can't— like, how do you actually know? So let's talk through how you might know. Well, there's a few different ways you could go about this. One thing you could do is build what you might call demographic agents. Here you take a sample of the population and you produce a demographic agent version of a person. Here's a demographic agent version of me. It says my age, where I live, my job, and these kinds of things. On another hand, you could create what we did in that Smallville simulation, these persona agents that are more narrative descriptions of the person. Again, on the right here is a persona agent version of me. Maybe adds a little bit more flavor, but also misses some information. But they're both interesting. The issue here, as prior research has demonstrated, is that both of these approaches can produce very simplified and stereotyped behaviors. Most kind of— one of the instances that was most striking to me was that if, you know, June Park, again, my PhD student or now alum who started this work, If you just produce, you know, he's from South Korea, and if you ask, if you tell the model that and say, what is he going to have for lunch? The model says rice. Not great, really, probably not accurate either. It's very stereotyped, you know, has bias, all these kinds of things in it. So what are we supposed to do? So what we found is that there are better ways to do this. In particular, we found that rich qualitative information is an interesting direction. So what we did, and this is some more recent research, is we went out and ran 2-hour interviews with a representative sample of 1,000 Americans. Now, obviously, this is focused in the US, but you can do this yourself in your local jurisdiction if you're interested. So we would run these long interviews, and I'll talk about whether 2 hours is necessary later. It turns out, actually, you don't need 2 hours. We took an interview script from a project here at Stanford called the American Voices Project. Led by David Grusky. He's a sociologist, has this very broad-ranging interview script that literally starts with the question, tell me the story of your life. And then it covers everything from their communities and their jobs and their finances and their health to their politics. It's just this broad backbone of who they are. We use this 2-hour interview to construct a digital twin of that same person, a generative agent of that Same person with that interview as the memory of the agent. Okay, so now we have a real person, 1,000 real people, and 1,000 twinned agents for them. That real person then takes a large battery of surveys and experiments so we can gauge what their actual attitudes and behavior are in, in realistic scenarios. So here I'm going to focus on the General Social Survey. This is a— 170-ish question survey that has been fielded in the United States over the last 50 years. But we also look at the Big Five Personality Index, some big— some behavioral economic games, some replications of experiments. This is broad battery. We then have the generative agent of that same person take the same surveys and experiments. And then now we can finally measure this more accurately. We can say, How closely does this person's agent replicate that person's actual behavior and attitudes? I hope that makes sense. So we have real person, agent version of them, and we ask how well the agent version of them replicates their exact answers. So we run these interviews actually with another AI agent. It's a voice-to-voice interview, like a phone interview. It's on the computer, but it's, um, I've changed many of the details here to preserve anonymity, but I just want to give a sense of the flavor of the kind of information that these interviews yield about people. Obviously, this data all remains private for human subjects reasons, but you can see that the kinds of things we're learning are people's backgrounds and their life stories, their ambivalence toward political identification, what their jobs are and were. Their finances and how much extra money they have to spend on various categories and so on. So it's just a very, very rich background. All we do then, and if you wanted to create these kinds of agents yourself, given this kind of interview, is we put the interview up at the top of the prompt to ChatGPT or another model, and we tell the model, based on this interview transcript, I'd like you to predict how this person would respond to the following survey or experiment. Now, there's more detail to it here. You can see the actual prompt we use. But for the most part, it's just the yellow parts here, interview, and then now please predict what this person will say in response. What we find is that these agents do accurately replicate attitudes and behavior. Now, in order to explain this, we need to say one more thing about the method. It turns out that if I give you a survey today and then I give you the same survey 2 weeks later, you are not going to answer exactly the same thing. Some— a lot of stuff's going to be the same, but not exactly the same. So we need to normalize for this. So we do exactly this. We have everyone, all 1,000 people in this experiment, run the same survey twice. And so what we're going to do here is every result I'm going to show you here is going to be sort of a ratio where 1.0 means that agents replicate people's responses as accurately as people replicate themselves 2 weeks later. I hope that makes sense. So like a 0.1 would mean we can replicate people's responses 10% as well as accurately as people replicate themselves 2 weeks later. Okay, so if you were to do this, if you were to guess randomly on this big survey, uh, you can replicate people about a third as well as they replicate themselves 2 weeks later. That's a baseline of sort of how predictable people are. If you use these persona agents or the demographic agents, you can replicate people about 70% as well as they replicate themselves 2 weeks later. And if— excuse me— if you use these full interviews like I was describing, you can replicate people 85% as well as they replicate themselves 2 weeks later on the General Social Survey. I think that's pretty impressive. Now, it's not just the General Social Survey across a series of other tasks, like the Big Five Personality Index, behavioral economic games, you can see that these normalized coefficients are all generally in this similar range. So we think that there's substantial possibility here. Furthermore, actually, these interviews reduce bias. And if you— so if you have this full interview, if all I know about you is that you're like a Republican, a conservative, then it really stereotypes. But if I know a lot more, then the model can get much more subtle. It turns out across these different approaches that you might imagine, politics is the hardest to model. The difference between the best-performing group and the worst-performing group was around 8% accuracy, normalized accuracy. The worst-performing group for us was far-right conservatives. The underlying language models are pretty unhappy responding to questions as a far-right conservative. They've been value-aligned, Maybe to not say some of these things. And the conservatives, the far-right conservatives themselves in the interviews were a little more coy about their true opinions. Gender and race surprised me. They were actually— they had smaller gaps at a baseline than I expected. But again, the interviews cut out a lot of the difference. So again, we are looking at maybe just, you know, a sub— in the case of gender, sub-1% difference in accuracy between the different groups. In fact, we, as I mentioned, we also had 1,000 people do this. We had 1,000 people replicate experiments that were published in top-tier journals, the Proceedings of the National Academy of Sciences. This is top, top-tier science. So we had 1,000 people, our 1,000 people replicate those same studies, and then we had their agents replicate those same 5 studies. Of those 5 studies, our agents replicated 4 of the 5. And you might say, oh, well, that's not great, you missed one of them. But it turns out that we didn't. The 1,000 real people also didn't replicate that 5th study. That 5th study was actually bad science that didn't replicate. That result with real people was not— that was published wasn't a real effect. And the simulations correctly predicted that that was going to happen, that that 5th study wasn't going to happen. And in fact, my colleague here at Stanford, Rob Willer, has generalized this in, in a really compelling way where they found a bunch of non-public study results that were preregistered. So they got all of these studies, 100,000 people, and found that they could predict the effect sizes of these experiments with simulation with a correlation between 0.85 and 0.9 pretty strongly. Now, what this wound up with for us was this agent bank of 1,000 consented participants from the United States. So we have this agent bank that we can now use to sort of test maybe depolarization or climate change kinds of questions and so on. But if you're trying to do this for your organization, you might want to think about who ought to be in that agent bank that you would build for questions that you care about. Should they be representative of your customers? Should they be representative of people who are not currently your customers but could be?

Should—

maybe your company has internal marketing or user research data that you've interviewed people in the past. You just have a dataset sitting there that you could turn into an agent bank. Here's my advice. Here's my how-to. A bad way to do this would be to define agents with a single demographic variable, like, just be a conservative, be a Republican. This produces all sorts of stereotyping and underestimates the variance, the sort of the distribution. Like, it makes it seem too certain when you need it to be more uncertain. It's just, it's not enough. Not terrible would be 5 or 6 demographic variables. We saw that that was able to replicate people about 70% as well as they replicated themselves 2 weeks later. Better was data as rich as you can gather, like these interviews we were finding. We were actually able to maintain very strong accuracy by having a much shorter interview. So we went in later and we just deleted random 80% of the interview. And we were still— we went from like 85% accuracy to like 79, or 0.85 replication ratio to 0.79. It didn't really decrease it that much. Which surprised me, to be honest. So there's something about these interviews is teaching very rich information. But I do think the— and I see a comment in the chat about this exactly on point. You have the remaining parts of the interview have to be relevant. So if you're only interviewing people about their views on fashion, say, and then you're trying to predict their views on climate change, there's just not enough for the model to generalize it. Okay, maybe If we know that they're very high, you know, high income or something like this, you might be able to extrapolate from there. The models can do quite a bit. But, you know, if I'm only interviewing you about sports teams, and then I'm trying to predict your, you know, your retirement planning or something like that, it makes sense that it would be— that it's a little bit too disjointed. You have to make sure that the data you gather is relevant. Now, we cannot proceed without talking about the risks of this. There are— there's literature. Not every study replicates correctly. And you can see some of the research here that has found examples of cases where this hasn't replicated. And I can show you an example of, of one case where this really has happened. So here's a company that I was talking to that tried to implement this. On the left is ground truth data. This is Pew Trust. And if it's a little blurry for you, then what's highlighted in red is this age broken out by age, 18 to 35, 36 to 51, and so on. And they're asking people how familiar they are with their retirement plan fees. I want you to pay attention to the dark blue part of the bar where, you know, say in the 18 to 35 group, 13% of people said they were very familiar with their retirement plan fees. On the right, you can see their simulation results where instead of 13%, that one was like 1.2%. Now, you might say, oh, well, Michael, you know, you still got the biggest one. That middle bar was still guessed to be the most— the biggest one, and the 2nd one was still guessed to be the 2nd biggest one, and so on. And so maybe it doesn't matter for your particular cost, but maybe it does, right? Maybe the difference between 13% and 1.2% is the difference between we ignore this subpopulation and they're a sizable minority that we would care about. So these things can make errors. And the way that I think about this today is think about it as a ladder where you're— as you climb up the ladder, you take on more risk of issues, but it also gets more ambitious. So you might start out at the bottom of the ladder with a question of possibility. What might this— what might happen? What could happen? We're not going to put a probability label on it or say how likely it is to happen. Just, this is an outcome that you might want to think through. Because it could happen. Further up would be sort of qualitatively oriented outcomes, things like attitudes, anything that is sort of chat outcomes. This is, I think, safer because we're pretty good at modeling this already given the interviews. Higher up and where I just showed the error was quantitative outcomes like histograms, bar charts, where, you know, the difference between 5% and 10% might really matter. And at the top would be multi-agent simulation, the sort of town that I was describing, an entire market simulation. In order to trust that, you need to have a lot more assumptions. Let me dig into what you would need to see to trust each of these. Because this is a totally new method and I want to stress that you should not overtrust such a method. With the possibility rung, this one I would say mostly generally works today. What it needs to generate is a plausible chain of events that could lead to a potential outcome. You should be able to just by looking at the chain of events say, oh yeah, that seems plausible that that could happen. Oh yeah, if a troll shows up in my system, they could really mess with it in the following way. We need to introduce the following safeguard. On the qualitative rung, it's mostly about attitudes. And so we need to be able to accurately estimate individual attitudes. I would say that my, my evidence is that this mostly works today if you have rich enough data. Now, it's not the same as running an interview, and I would never say that you should do this instead of actually engaging with the communities. But I think it is a good way to get a rough sense of what, of what might people say in response to, say, a new policy or a new strategy or a new product. The quantitative rung requires actual measurement of quantitative accuracy. Say, can it recreate your market research surveys? In many cases, we're seeing yes, and in many cases, we're seeing that there are errors. So you need to tread carefully here. I think the way I would view this is that you ought to You know, consider you have 100 ideas of what you might do. Use simulation to narrow it down to 5 that seem promising, and then actually A/B test the remaining 5 on real people to see what's working. With multi-agent, I don't think that we're really ready for that in, you know, decision-making quite yet. If you wanted to do this, you'd have to be able to trust every individual agent to be accurate, right? If the individual agents are accurate, then when you put them together, then the emergent outcome should also be accurate. You could also take a complex systems perspective of like, oh, it's more about the phase changes. I think that's much more difficult to reason through. Multi-agent can be right, but it also can be wrong and you don't know how to distinguish those. I would be very careful. How do we mitigate these risks? Well, for one, make sure your agents have in-domain data in their memory. Again, this is don't interview about fashion if you're asking about retirement. Second, the possibility and the qualitative rungs reduce and mitigate many of these risks because they're what I would call rough-edged problems, where if it's 80% correct, it's still helpful. It's still helping you learn. Whereas the, the quantitative rung is like sharp-edged. It's either right or it's wrong. And if it's wrong, you might be making the wrong decision. And finally, validate important questions on a small subsample to make sure that the model isn't too far off. Now, I want to highlight here that there are frontiers of this space. I'll show just a couple. One is the set of tools around look before you launch. So this is actually where we started. As I mentioned, I was doing a bunch of work for many years in online platforms where people wouldn't foresee the ways that their policies would backfire. And what we did is we found that these tools would be— are actually very useful at this possibility rung to help people fix things before they go wrong. And we would get these quotes from people being like, yeah, everything we do is set in reaction to a dumpster fire. That's really a terrible way to run a community. And in response— and instead, when we gave them access to these kinds of tools, they actually iterated in simulation. They're like, oh, I didn't realize that this low-effort post would do this kind of thing. And then the troll would respond in that way, and they actually fixed it before it launched. I now actually do this every year when I teach a course in online platform design. I have students, I open the firehose of trolls at these, at their systems, and their goal is to make them, you know, stay upright. And I think we get better projects for it. A second application of this is training soft skills. So one that we built in collaboration with an expert in the Stanford Business School in conflict and negotiation, yes, these systems around conflict negotiation, everyone thinks that they're great at conflict until things start going wrong. And we felt, ah, maybe these, you know, these generative agents can serve as sort of like sparring partners or training partners so that you can engage in this conflict or this negotiation. Maybe you're negotiating a salary for a new job or whatever in simulation before you go and have the actual conflict. So we created a tool that let people sort of, you know, simulate a conflict. It's almost like a Monty Python where we then ran an experiment on it where we randomized people into either watching a lecture on the best-in-class strategies for handling conflict or watching that lecture and going through a simulated conflict. Both groups critically did equally well on a book test of what they're supposed to do in a conflict, but only the group that had the simulation did better in an actual conflict later. It reduced— having done the simulation reduced their likelihood of using like an antisocial strategy by 2/3. So there's something about this idea that I could try something sort of in simulation that really aids learning. Now, as I mentioned, there's, I think, going to be a number of business applications for this. I brought this article up. You can look at this for sort of a fairly up-to-date assessment of what's going on if you're in market research or something like this. I will, with some embarrassment, point out that if you scroll far enough down in that article, they actually mentioned this startup that the Stanford research that I'm describing here spun out into. If you're— I'm not here to shill any company, but if you're interested in chatting about sort of commercial applications of this, there's a QR code on the screen I'll leave up for just a moment. This— the company that we're spinning out is called Simile. But there's a lot of open research in this space that I've been— that I'm mostly focused on trying to share here. And what I think I want to do is just highlight that there is so much that we can learn from the opportunity to create this what-if machine. that allows us to really go forward and think through what might happen. And so with this, I wanted to make sure that we leave a lot of time for discussion. We have about 10 minutes here in the, in this webinar for discussion and Q&A. I'm gonna flip this over to another QR code for the, for Global Alumni, where if you wanna learn more, I teach an online course that Global Alumni can say more about here. So I'll hand this over to the Global Alumni team. I think there's going to be a Q&A that we'll engage with over the next 10 minutes, and we'll take it from there.

Thank you so much, Professor Bernstein, for this session and for your time. We are going to do now some Q&A questions, so I can give you the first one. Can AI robots do customer service for drive-through fast food restaurants?

Oh, gosh. Can these systems do customer service? You know, that's not really a question of whether we simulate in the sense that, like, are you really— are you trying to simulate the person that's at the window? Can they do customer service? I mean, I think if you've used many different apps recently, these things are being pushed into doing customer service, like ChatGPT customer service agents. I think there are risks. You may have— here, I'll, for a moment, or in a moment, I'll stop the screen share so you can see my face.

The—

you may have heard of this Canadian airline that did a ChatGPT customer service agent that then promised a refund to some of its customers, someone who had experienced, I think, a death in their family. This was— it said yes, but this was actually out of policy for the company. So they then later came back and said, oh, sorry, that was just our chatbot. We can't promise this. And then the Canadian courts came in and said, no, your chatbot promised this, you gotta do it. So I think that we have to think very carefully about customer service applications here, because it is, at the end of the day, the experience of your organization is their experience with that last mile, with that customer service bot. So do I expect this to be used? Yes, I do. I think you ought to be very careful. Yes. I actually think the thing that's probably going to wind up being even much more further discussed is not simulation of other people, but the creation of entirely artificial characters. Like if you've played or heard of Character.AI, this is something that I think we discuss in the online course, the Global Alumni and Stanford course. That, you know, I expect that in the next 5 years, there's going to be a lot of conversation about people who are talking to AI agents, completely artificial characters, not simulations, you know, in preference to talking to other people. I think there's going to be a lot of hand-wringing about this and thinking about what, how, what kinds of risks those pose to people. My colleague Di Yang at Stanford has actually just recently put out a white paper on this exact topic.

Thank you. So next question, there are quite a few, so I don't think we're going to be able to cover everything, but thank you so much for sharing so much of your questions with us. So does AI mimic or interpret human behavior?

Does— can you say that again? Does AI mimic or interpret?

Does AI mimic or interpret human behavior?

I mean, it's sort of doing both. I would claim that— to be clear, these are not like agents with cognition. These are not alive. These are not thinking agents. They are simulations of people. So I would view them as sort of mimics of human behavior. But in order to mimic that behavior, they are— they have to be able to interpret a lot of human behavior, other humans' behaviors, and the interviews and whatever other information we have. So that interpretation is actually a critical step along the path to getting accurate simulation.

Next question. So human behavior is a complex system in itself. How do you even replicate a complex system in its entirety?

How do you replicate a complex system in its entirety? That is a great question. So I think that that is actually a sort of an open research frontier a little bit.

The—

in order to do that, you would need to— I think we've now got the ability to replicate individual agents with a reasonable amount of accuracy. But the environment, like the world, we haven't replicated. You would need to do that, right? Right now these agents are in a little tiny town. In fact, just as a funny little example, in early drafts of this, there were no doors. Like, we just hadn't implemented the doors. And so the agents would like walk in on each other using the restroom because they couldn't see. And there was no— like, you have to get the environment to be accurate if you want accurate behavior. This is sometimes when I get leery of people who say, oh, this replicates or that doesn't replicate. Because you sort of— it's like they take agents, they put them in like a very, you know, straight white box that's completely dissociated from any realistic environment. The agent had no earlier, you know, nothing happened to it before. It just woke up and it's like given a question. What happened to that agent earlier in the day? Did it have a fight with its kid? Is it under financial stress? Like, we need to actually give that much richer picture and the environment itself, which we know accounts for about 4, you know, 0.4, 40% of the variance in in human behavior. I'll make one other point with a quick slide that is not about this, that is something else I teach, which is essentially that even if you could— that human society itself isn't fully predictable. There's one of my favorite experiments of all time here. Basically, they got over 14,000 people to split into a bunch of different worlds where they could either see or not see the view counts on songs like in a Spotify clone, Like, here is like, imagine a YouTube, you know, on Spotify you can see how many other people have listened to a song, or YouTube you can see the view count. Imagine you couldn't see that. And now they have all these new songs from indie bands, and they would randomize people into whether they could see the views or not. And then they would randomize them further into parallel universes. So now there's 16 different versions of Spotify, and they could ask whether the same music becomes popular across all of these parallel worlds. And what they found was that when people could see each other, how many other people had voted, then it basically blew out of the water any consistency in the outcomes. That the best songs rarely did poorly and the worst songs rarely did well, but basically any other result was possible. And so what I'm trying to convey here is that essentially, even if we can simulate the world, there's no guarantee you get a single outcome in the world. Because if you were to rerun our world multiple times, you would get different outcomes. So the best thing we can do with simulation is kind of do what's called Monte Carlo simulation, run it a bunch of times, and to learn from there what are more likely outcomes, what are less likely outcomes. But I can never give you a straight point prediction of this is definitely going to happen.

Sure. Thank you. Next question. So if we were able to apply this to an experience design use case, for example, creating persona-based agents, how might we reach an appropriate level of confidence that we've answered the important What ifs?

Oh, that to me feels like a leadership strategy question. You know, think about, you know, how does any good leader or manager prioritize the risks that they face? And I tend to think of this in terms of agile. I don't know if you've ever read like The Lean Startup or something like this, where you essentially start by asking, what are the riskiest risks? That we faced. The thing where if we're wrong, completely decimates our strategy. And you start by reducing your uncertainty in that risk, and then you reduce the uncertainty in the next riskiest risk. And sometimes new risky risks show up, and you do this iteratively. So that's how I would think of it, is there's— I can't give you a complete set, but you should be able to rank your risks by how critical it is that you be right. And in fact, if I'm guessing, you could go to one of these reasoning models like OpenAI, Like now it's— I would have said O3, but now it's ChatGPT, it's GPT-5 thinking mode. And ask it, you know, what are— you know, here's my idea. What are the big risks that I should be asking a what-if machine? And see what it comes up with. I bet it'll have some decent ideas.

So next one. Does the LLM bias change the bias of the agent over time? For example, I'm wondering if LLM-instilled values change the agent's biases when the simulations Sorry, when the simulation goes longer due to larger chance of hallucination or value bias?

Yeah, so there are 2 questions embedded there. One is, does the LLM bias influence the agent's behavior? And for that, the answer is likely yes. And we saw the evidence of that in how difficult it was to— the far-right conservatives were harder to model. That's, I think, in part because the model is, you know, trying to not be racist and sexist and so on. But your question was about long term. I actually think that the bias that will play out over long term is more likely to be a sort of conflict avoidance. Like these, you know, ChatGPT tries to be helpful and harmless. And so the— I don't know if you saw in the conversations in the demos, the agents were all very pleasant to each other. They didn't get into fistfights. And so I would expect that what's going to happen over time is we're going to run into a barrier at some point where the big models like GPT-5 and so on have been trained to do math tasks and to be helpful smart assistants, but that makes them less and less accurate at replicating people. And what we'll need to do is train separate AI models that are specifically fine-tuned to be better reflections of people. And in fact, my lab at Stanford has been doing research on exactly this. There was a recent paper in Nature actually that someone else did called Centaur that was doing this in the psych space. I think that's going to be the next generation is AI models that are specifically trained to be like people, not just trying to sort of kludge it on top of Claude or GPT-5.

We have time for one last question. Let's do, what's this? What's this what-if machine versus scenario analysis?

I think a what-if machine is my goal. It's what I'm trying to create. Scenario analysis is a more structured strategy. It's a process that you can follow to try and answer questions. In my head, a what-if machine becomes a tool you can do you can use to perform scenario analysis faster and more effectively.

Okay, so we've come to the end of our session, and I don't know about you, but it certainly— time flew by. So thank you all for joining us today for this insightful session from Stanford Online. We hope it sparked valuable ideas on how AI agents are reshaping organizational leadership. And thank you, Professor Bernstein, for being with us today and for your time. If you want to take the next step and keep learning, Professor Michael leads the online course UI/UX for AI Products. Discover real-world use cases, practical strategies, and how to create positive experiences powered by AI. If you have any further questions or would you like to explore how this course aligns with your career goals, please feel free to reach out to schedule a call with our academic team advisors. Once again, we appreciate your time and interest, and we look forward to welcoming you to the Stanford Online Learning experience.

Thank you.

Thanks for everything. Cheers.

Thank you for joining.

Thank you.