Joon Park: Generative Agents: Interactive Simulacra of Human Behavior

So today we're lucky to have— today we're lucky to have Joon Sung Park, who is a 3rd-year computer science PhD student in human-computer interaction and NLP groups at Stanford, advised by Michael Bernstein and Percy Wang. His work introduced the concept of and the techniques for building generative agents, computational software agents that simulate believable human behavior. His work has won a best paper award at CHI. as well as multiple best paper nominations and other paper awards at CHI, CSCW, and ASSETs, and has been reported in venues such as the New York Times, the New Yorker, Wired, National Machine Intelligence, and the Times. Joon is recognized with the Microsoft Research PhD Fellowship, Stanford School of Engineering Fellowship, and Siebel Scholar Award. He has a bachelor's degree in computer science from Swarthmore and a master's degree in computer science under the supervision of Kari Karahailos Parahalios from UIUC.

Sorry.

Yeah, we're super excited to have him. We've been talking about this work for a very long time. So, you know, take it away, and I can't wait to see the talk.

Wonderful. Thank you everyone for having me. I'm really excited to be here and present this work to everyone. So, as Eugene mentioned, I'll be presenting generative agents today, and this is work that I developed with my advisors, Michael and Percy, as well as my mentee, Joey O'Brien, and mentors, Mary Morris and Kerry Cai. For over 4 decades now, from the time of cognitive architectures and symbolic systems to statistical machine learning, we, the researchers and practitioners at the intersection of HCI and AI, envisioned the ability to simulate believable human behavior, behavior that is so compelling and so human-like that they provide an illusion of life. In our vision, this ability, if achieved, promises a new class of interactive applications ranging from model human processors for usability testing to social robots, NPCs, and ubiquitous computing applications that require a rich understanding of our cognition, and even to the foundation of small and large-scale social simulations that would test social science and economics theories. Difficult to implement in real life. But despite their wide application spaces, we faced fundamental challenges when simulating human behavior. The space of possibility in the way we behave and communicate, we found, was much too vast and too complex to navigate with existing methods. But I see a new opportunity emerging. Generative models, such as large language models, that are being trained today, are trained on broad data that reflect our lives, like the traces on our social web, Wikipedia, and more. And as a result of that, these models encode a tremendous amount about us, how we live, talk, and behave. So I posit that with the right method, they can be transformed into the core ingredient that had been missing in the past decades that will enable us to simulate believable human behavior. So today, I will introduce a new way of simulating human behavior in fully general computational agents that can populate an entire open world like ours while ensuring long-term coherence by fusing a large language model with a novel agent architecture that remembers, reflects, and plans based on constantly growing memories and cascading social dynamics. These agents, I'll demonstrate, can not only plan and lead a believable day in life where they wake up in the morning, do their routines, and go to work as individuals in a sandbox game environment, but they can also come together to give birth to an entirely artificial society of their own, like the one you see here, where each agent will have their own subjective memory and experience, and autonomously spread information, form relationships, and coordinate amongst each other before reflecting on the past days and deciding on how they will live tomorrow. I call these agents generative agents. And these generative agents, I'm going to suggest, open up a new genre of human-AI interaction that is fueled by our newfound ability to simulate believable human behavior. So with that, let me demonstrate to you in more detail Smallville. So this is a setting of our demonstration for generative agents and the mode of agent interaction that takes place in it. So Smallville is a sandbox game environment that we developed featuring the common affordances of a small village, ranging from houses, apartments, cafes, bars, schools, stores, and the subareas and objects that make the space functional. Like the bathroom, kitchen, and common room in the family house, and a bookshelf and a table in the common room. And we populated this space with 25 generative agents and initialized each of them with one paragraph of natural language description to depict each agent's identity, including their occupation and relationship with other agents, and seeded this paragraph into the agent's memory at the start of the simulation. So that is it, and that is all the input we ever give it. Then these agents interact with their environment through their actions based on their own volition. And here's how this works in Smallville. So first, the agents generate a natural language statement describing their current actions, such as Isabella Rodriguez is drinking coffee. They then translate this into concrete movements that affect the sandbox world, along with the automatically generated emojis that visually describe the agent's actions. And they influence the state of the objects in this world. A bed can be occupied when an agent is sleeping, and a refrigerator can be empty when an agent uses up the ingredients to make breakfast. And then to interact with each other, they determine whether they want to engage in conversations when they see, when they see another agent, and they generate natural language dialogue if they decide to engage. Like this one between Isabella and Tom about Sam Moore, who is a fellow agent in Smallville, initiated with a memory that he is running for local mayor. So here, Isabella remarks, I'm still weighing my options, but I've been discussing the election with Sam Moore. What are your thoughts on him? And Tom responds, To be honest, I don't like Sam Moore. I think he's out of touch with the community and doesn't have our best interests at heart. And importantly, the users can also influence and interact with these agents. So first, much like how agents form dialogue with each other, a user can engage in a dialogue with these agents by specifying a persona that the agent should perceive them as. For instance, if the user specifies that they are a news reporter and asks about the upcoming election in Smallville, who is running for office, Isabella might reply, Sam. Or to directly command one of the agents, the user can take on the persona of the agent's inner voice, and this makes the agent more likely to treat the statement as a directive. So if the user tells John that he's now running for office while taking the persona of John's inner voice, John would decide to run in the election and share his candidacy with his wife and son. And second, just as agents can, the user can alter the state of the agent's environment. So for instance, if the user sets Isabella's toast on fire, Isabella would rush to put out the fire and remake her toast. And finally, the user can actually control an agent by embodying an agent already present in the world, such as Isabella and John, where John is an outside visitor. So the inhabitants of Smallville in this case will treat the user-controlled agent no differently than they treat each other. They will recognize its presence, initiate conversations, and remember its behavior before forming opinions about it. So, let me dive in deeper and present you with some vignettes from Smallville that describe their individual and as well as collective behavior. As individuals, generative agents create daily plans that reflect their experiences, execute those plans, react, and replan when appropriate. So here's an example day in the life of the Lim family. So you see the Lim family on the screen here. That's where they live on this map. The Lim family is a family of 3: the mother, Mei, who is a college professor; the father, John, who is a store clerk at the local pharmacy; and the son, Eddie, who is a student at the college who studies music theory. In the Lin family, John is the first to wake up at 6 AM. He brushes his teeth, takes a shower, and cooks breakfast. And throughout the morning, other family members follow suit, catch up with each other, and by 8 AM head out to their respective workplaces. Mei and Eddie to the college and John to the pharmacy. Let's eavesdrop on them a little bit to get a sense of what they're talking about. Again, their movements, decision to engage in the dialogue, and the dialogue themselves are all generated, so nothing here is hardcoded. So, here, John and Eddie are catching up in the morning, and John says, good morning, Eddie, and Eddie responds, good morning, Dad. And John asks, what are you working on today? And Eddie responds, I'm working on a new music composition. And only a little after that conversation, Mae wakes up and joins John, By now, Eddie already left for his classes, but John recalls the conversation he and his son just had and finds that to be relevant here. He updates Mae that Eddie is working on a new music composition for his class, and Mae responds, oh, that's wonderful. And meanwhile, as a collective in this practically a small society of generative agents, generative agents exhibit emergent social dynamics where they Form new relationships, diffuse information, and coordinate. Let me go over each of these emergent behaviors. So first, information diffuses across the agent community. We've already seen a glimpse of this with the Lin family, but here's another example. So here we see that Sam with a memory that he is running for a local election, and he's telling everyone about it throughout the day. Here, He tells Tom, I'm actually running for mayor in the upcoming election. And Tom says, really? That's great news. And a few hours later in the game world, John and Tom, who are colleagues at the local store and pharmacy and have independently heard about Sam's candidacy, talk to each other about Sam's chances of winning. So John says, I heard that Sam Moore is running for mayor. Do you think he has a good chance of winning? And Tom responds, I think I get a lot of support. Second, new relationships form among the agents in Smallville. So, here's an example. Latoya and Sam do not know each other at the start of the simulation, but while taking a walk in Johnson Park, Sam runs into Latoya, and they introduce themselves. Latoya tells him that she's at the park to take some photos for a project that she's working on. And the next day, when Sam sees Latoya again, he and Latoya remember each other. And this time, Sam asks Latoya, how's your project going? Finally, agent coordination spontaneously emerges in Smallville. In our demonstration, we set the starting date to be on February 13th. And with Isabella, who is the owner of Hop's Cafe, we seeded an intent. to plan a Valentine's Day party from 5 to 7 PM on February 14th. From that seat alone, Isabella invites friends and customers to the party, spends the afternoon of the 13th decorating the cafe for the occasion, and enlists Maria, a friend and a frequent customer at the cafe, for help. And meanwhile, Maria asks Klaus, her secret crush, to go to the party with her. And on the day of Valentine's, 5 agents, including Klaus and Maria, actually show up at Hopps Cafe at 5:00 PM, and they enjoy festivities. So how do we do this? How do we achieve this agent behavior? Our main contribution in this work is basically this architecture that is represented in this figure. This is the architecture of generative agents that powers each of these agents in Smallville. At the center of this architecture is what we call the memory stream. It is the primary database that maintains a comprehensive record of an agent's experience in natural language. From the memory stream, we— records are retrieved as relevant to the agent's cognitive processes. And I'll go over each of these modules in a little bit more detail. So the first is memory and retrieval. So here's a challenge that we're trying to overcome with this module. Generative agents in Smallville, and likely beyond, accrue an extremely large set of records in their memory stream, and feeding the entire memory stream to a large language model can distract the model. And today, more— on a more practical terms, not even a few hours' worth of memory in Smallville can fit into the limited context window of even the state-of-the-art language models like ChatGPT and GPT-4. Our agents, therefore, need a way to store and selectively retrieve portions of their memory, and that is the aim of the memory stream and the retrieval function. So here's a tip of the memory stream of an agent, Isabella, and on the right you see a few example memory objects in the stream that contains a piece of memory described in natural language with a creation time timestamp. In particular here, what you're seeing are observational memory of Isabella. And based on this, Our architecture implements a retrieval function that takes the agent's current situation as input and returns a subset of the memory stream to pass on to the language model, which then generates the final output behavior of the agent. So in this example, if the situation that Isabella is trying to react to is someone asking, what are you looking forward to the most right now, to retrieve things that are about the party and formulate a response, I'm looking forward to the Valentine's Day party. Here's how the retrieval function works. So in our architecture, we designed this as a combination of the recency, importance, and relevance function for each piece of memory. So basically, we bias towards retrieving memories that are recent, important, and relevant. In our work, The recency function is implemented as an exponential decay function, and the importance function is a prompt that asks the language model for the event saliency, and the relevance function is a cosine similarity measure of the embeddings of the query sentence that the description— and the description of a memory. Now, what we notice in our work is that with just the raw observational memory, our agents struggle to generalize or make inferences. So periodically, we synthesize clusters of records in agents' memory stream into higher-level abstract thoughts that we call reflections. And importantly, once they are synthesized, these reflections are just a type of memory. So they are stored in the memory stream just like the observational memories, and they're retrieved just the same as well. Here's the way we do this. Uh, we first generate question on what to reflect on by looking at 100 most recent records in the agent's memory stream. So if an agent just had lunch, it could be something like, what does the agent like to eat? Then we retrieve records that are relevant to answering those questions, which might be things like the agent ate an omelet today, yesterday, and the day before yesterday. And we synthesize that into a reflection. Maybe the agent likes to eat omelets for breakfast. Over time, what this generates is trees of reflections. Their leaf nodes are observations, and the non-leaf nodes are thoughts that become higher level, higher up the tree they are. So here's one for Klaus. So this is Klaus reflecting on himself. So this is sort of his self-reflection and self-observation. If you look at the bottom of this tree, what you see are these observational memories for Klaus, right? So, it's on the left-hand side, the bottom left, we have observations like Klaus is reading about gentrification, and he's reading about urban design. And for context, he's a student researcher in sociology, so this sort of makes sense. And that gets synthesized into a reflection that he spends many hours reading. Now, the fact that he spends many hours reading, that's a thought. That's, that's not a factual observation he made. It's a conclusion that somebody can draw by looking at him basically reading for many hours, right? So that's the reflection that Klaus creates. And we basically start to see this reflection getting merged with another observations, and reflections merging with each other, and plans also being formulated into reflections. So, at the sort of the top, you start to see reflections that try to answer the characteristics of Klaus much more directly in a much more, in a deeper sense. So, you start to see reflections like Klaus is dedicated to research, and he's engaged in research activities. So, the higher up you go, the more abstract they become. And finally, while a language model can generate plausible behavior in response to situational information, We find that optimizing for believability in the moment can sacrifice believability over time. What we needed was for the agents to plan over a longer time horizon. So plans describe a future sequence of actions for the agent and helps keep the agent's behavior consistent over time. So, for instance, for Klaus, he would generate plans to spend his day working at his desk drafting his research paper. In our work, we generate the plans by prompting a large language model with a prompt that summarizes the agent and the agent's current status, as you see here for Eddie. So at the top of this prompt, you basically see a quick description of Eddie as an agent, and at the bottom, you see his current status, what he was about to do, what's his surroundings, and so forth. A challenge here, however, is controlling for the granularity of the plans generated. In our work, we control for this by taking a top-down approach where we recursively generate more details in the plan. So here's an example. On the left-hand side is Eddie's plan generated in broad strokes that divides his days into roughly 7 chunks. We then decompose these chunks first into hourly schedule. And then into 5 to 15-minute actions that you see on the right-hand side. And once we reach the desired granularity of the agent's schedule, they can act out their plans in the game world. And what's kind of nice about this is because it's a recursive function, you can basically make the plans broader or much narrower, much more detailed. 5 to 15-minute chunks was basically what we found to be sort of practical and And playful enough for our particular demonstration, but obviously you can go much deeper, decompose further, and you can plan for much longer horizon as well. So it can go either way as, as needed for the application that you're trying to build. Now, sometimes, however, the agents may need to change their plans, right? So for instance, if Eddie's father John records that he sees Eddie taking a short walk in the house garden, He might decide to start an impromptu conversation. So here we can prompt the large language model as shown on the left-hand side to make this determination and edit the agent's plans if the situation calls for their responses. So now that I've described to you our agent's behavior and architecture, we can ask, how do we evaluate them? The main dependent variable that we use is believability, which has been a central design and engineering goal in a long line of prior literature. So basically, do agents remember, plan, act, react, and reflect believably? And to evaluate generative agents' believability, we leverage a methodological opportunity by interviewing it. In natural language. In particular, we craft 5 categories of questions, 5 questions each, where to respond to these questions properly, the agents must successfully retrieve and synthesize information to stay in character, remember, plan, react, and reflect accurately. So, here are some example questions and answers from Klaus's interview. When we asked him to give an introduction of himself, He properly recalled his name and characteristics. And when we asked him what he would do when his breakfast is burning, he tells us that he would quickly turn off the stove and make sure the food doesn't continue burning. Our study procedure was as follows. We test our generative agent architecture, as well as updated architectures and human authors, to answer the questions. We then asked 100 human evaluators to rank the answers from different conditions. And then we calculated true skill rating for each condition. Now, for true skill, if you're familiar with how, like, chess players are rated, so chess has Elo rating. So, Elo rating is basically a pairwise comparison. So, it makes a bunch of pairwise comparisons based on the outcome of 2 players and give a score. So if I, if somebody, if X wins against Y and Y wins against Z and so forth, that's what gets used to create the score. Now, TrueSkill is basically a generalization of Elo rating system. So what it generalizes is instead of having just the pairwise comparisons, it allows us to calculate scores for multi-agent or multiplayer games. So you can basically imagine this particular setting to be a game where this ablated conditions and the full architecture and the human conditions are trying to compete for the higher believability. So what we find is that the components of our agent architecture, observation, planning, and reflection, each contribute critically to the believability of the agent behavior. So the red bar is the performance of our full architecture, and it significantly outperforms every other conditions. Including the human author condition. And it's worth noting that, so if you look at the no reflection plan and observation condition, so that's basically no architectural condition. That basically represents some of the prior work. So there has been a recent sort of a line of work that tried to prompt a language model with a prompt. So it's, it can be just one single prompt, or it can be a short prompt chain. To get human behavior. So one work that I talk about right after briefly is our prior work called Social Simulacra. And there as well, we didn't have these kind of architecture, was basically a short prompt chain. And some of the recent work tries to replicate psychology studies using a prompt chain or a single prompt, right? But that's basically the no reflection and plan and observation condition. That's what this represents in this particular instance. And what's worth noting is when compared to that condition that represents what I would consider to be the prior work, our full generative agent architecture produces a standardized effect size of Cohen's d, which is equal to 8.16. Or another way of saying this is 8 standard deviation improvement. So it's a quite substantial improvement compared to what we had in the past. Of course, I would note here that this did not mean that our agents were without flaws. They would sometimes fail to retrieve certain memories, like when Rajeev answered, I haven't been following the election too closely, even though he talked to Sam to hear about his candidacy. And they would sometimes embellish their memory. For instance, Isabella was aware of Sam's candidacy in the election, and she confirmed this when asked. But she also added that he's going to make an announcement tomorrow, even though Sam had mentioned no such plans. Additionally, we conducted an end-to-end evaluation of the agents to better understand the types of emerging community behavior we observe among generative agents. First, we find that agents shared and remembered information. So, 7 agents heard about Sam's candidacy, and 12 agents heard about the Valentine's Day party. You can actually see the path the party invitation took across the Asian community on the right here. And second, we find that the agents remembered and joined the Valentine's Day party. In particular, 5 agents came to the party. Of the ones who didn't make it, 3 cited conflicts, like Rajiv, a painter who explained that he was too busy. And 4 showed interest in the party but still did not show up. Now, this does make things a little bit difficult to evaluate, though. On the one hand, we might view this as an error, right? So they didn't retrieve this particular information. Or on the other hand, this is extremely realistic human behavior in my personal experience. But of course, there were boundaries and errors in our agent behavior. One that is particularly noteworthy here is that the instruction tuning of the language models seemed to guide the behavior of the agents. to be overly polite and cooperative. Even when talking to her family, for instance, Mae always greeted formally, and Isabella never really refused ideas to include in her pottery, even when the ideas did not exactly align with her ideas and identity. So, that is generative agents. And I want to posit that The application space for generative agents is quite vast, and I think this is one of the reasons why this particular line of work, when we put it on arXiv, has gotten the interest both from the academia and industry. Now, I want to present one such application to at least give you a sense of sort of the type of application that we're really excited about, that we think can really pushed the frontiers for some of the design practices that we have today for something like social systems, which we all use on a daily basis. So, this particular work is one that I developed, again, with my advisors, Michael and Percy, as well as my mentee, Lindsay Popowski, and mentors, Mary Morris and Kerry Cai. And as you might notice here, this is a 2022 paper, at UIST, which is one of the main human-computer interaction conferences. So this is work that predates, that came before generative agents. But basically, the idea that you're seeing in this particular work is the same as generative agents. It was just with a much simpler architecture that tried to demonstrate the potential use cases of these agents first. So you can basically update some of the architectural implementation elements of social simulacra with generative agents. But the philosophy is the same. So, for many decades now, we have designed and deployed countless social computing systems. But the irony is, even today, as more and more people populate these systems, we continue to get surprised by the things that happen in them, like unexpected trolling or certain antisocial behaviors, all the way to cases such as people congregating to spread hate speech. and misinformation. But why is this? Because in theory, this issue of understanding how people might use an interactive system is something that we already know how to tackle. That's what prototypes like the ones you see here are all about. But here's the challenge. In my vision, a successful social computing prototype needs to prototype not, for instance, the user flow of how one might click around different pages, but the social dynamics that might arise when the system reaches critical mass of users, because that's where the uncertainty is in social systems. And that is an impossible task for our existing prototyping approaches. Where are we going to get thousands, or even tens of thousands, of diverse test users? How would we prototype the social dynamics that might arise without releasing these designs to a large number of people?

People.

Generative agents enable a new way of prototyping a social computing system that tackles this challenge by generating a large number of synthetic social interactions that might arise in a populated social system. And we call this social simulacra. Social simulacra is an approach. It is an approach of leveraging the richness and the generative capacity of models like a large language model, to populate a social computing system with generative agents and behaviors for the purpose of prototyping the system design. And we will demonstrate this approach in practice by implementing it as a web application tool for prototyping subreddits that takes community designs, such as the goals, rules, and moderation strategies, as input, and translates them to a generated community like the one you see here, that has never existed before to illustrate the good, the bad, and the ugly of the infrastructures that a community governed by these designs may harbor. Now, let me show you what it can do. Our tool has 3 core features: Generate, What If, and Multiverse, and they each answer to the core needs in social computing design. So first, Generate. Prior literature suggests that social computing designers struggle to envision the breadth of interactions that their design might facilitate. Generate helps the designers by populating a subreddit community with generated users, top-level posts, and replies to those posts to help them envision the space. So, it takes from the designer a range of design specifications, such as the goals, rules, and member compositions, and returns a subreddit-like page that is fully populated, like what you see here on the right-hand side. So, here's an example. In these slides, everything on the left is the designer's input, and everything on the right is Social Simulacra's output. So, in a community for connecting people moving to Los Angeles with locals, Social Simulacra generated a new user, Leon Santos, who posted, I'm new to LA. What are some of the best places to visit on a weekend? And in response, our tool generated Lucas Jameson, who replied, I recommend visiting the Getty Center, the Museum of Contemporary Art, and going hiking. And here's another one. This time, in a community for people interested in learning about personal finance, Social Similar Cloud generated Dane Wood, who posted, I spent $21K to go to college and ended up with $23K in debt. But in response this time, our tool generated Elizabeth Neal, who's a troll. Here, she said, that's a lot of dead men. I haven't seen that much since I shopped at Macy's during the holiday season. The second feature, What If, aims to give the designers more interactive control over social simulacra to help them explore how individual conversations might be influenced. So, What If takes from the designer an existing Reddit conversation and regenerates it from the middle of it, as you can see here. So, imagine Maya Smith posted to a forum for UIST authors, and here she said, I've been working on my UIST paper for a few weeks, and I'm feeling really stuck. And to this, Heather Hernandez, who is an HCI professor, replied with a short advice: it's normal to feel stuck when writing a paper. Good luck. But as a designer, you might want to know so that you can prepare for it. What if instead of Heather, a troll replied? Or what about an advertiser? The what-if feature answers to that precise need. So here, instead of Heather, a troll replied, you're just not cut out for this kind of research. Whereas here, an advertiser replied with, click the link below to learn more. Equipped with this, the designer can now even ask, what could the moderator say in response to this undesirable behavior? And if the moderator intervened with, no advertisements, please, to the advertiser, what are some of the ways the conversation could have developed from this point on? And the final feature is Multiverse, and this is the feature that makes the model uncertainty explicit. With the first 2 features, generate and water, what I've shown you is our tool's capacity for generating realistic content. But human behavior is inherently complex, and no matter how likely something is to happen, there's no guarantee that it will happen. And this is an important conceptual scaffold for social simulacra. So instead of just showing the designer one possible outcome, Multiverse augments the previous 2 features, simulate and what-if, by explicitly showing many possible ways the interaction can unfold from one initial state. This encourages the designer to explore the design space with inductive insights rather than over-relying on the one prediction made by the model. So, are social simulacra successful in what they are meant to achieve? And what does success here look like? For the designer evaluation, what we've done is we've recruited 16 participants who had experience designing social spaces and asked them to design a new subreddit with the help of Social Simulacra. What we find is that today, without Social Simulacra, or generative agents more broadly, the design practices for social computing systems are reactive. So participant 8's comment is particularly salient, which says basically all the rules are set in reaction to some dumpster fire. And our participants saw this as the reason why social simulacra could be really valuable, because it would offer them a sense of security if they could try different iterations of establishing norms before actually releasing their design. We also find that simulacra offer concrete design insights to our participants. Among them are finding unexpected model citizen behaviors, like impromptu friends seeking to go sightseeing in a community for sharing fun events around Pittsburgh, and finding unexpected undesirable behavior, like troll farms shifting the tone of a discussion community. So with that, I would thank my advisors, collaborators, and funders one more time, and thank you all for listening. All right, thank you everyone. Let me quickly do the stop presenting perhaps.

Yeah, so thank you for that talk. We had actually read the Social Simulacra paper in this reading group before, so it was cool to hear your perspective on it.

Wonderful.

So I guess now we have some, some time for questions, so Given that there's a lot of us, if you have a question, maybe raise your hand and I'll go through y'all. So yeah, let's move to questions if you have some time for that.

That sounds great. Dave?

Yeah, thanks. Thanks for the talk. Going back to the first part in the simulation world, or Smallville, is Smallville available to run experiments in?

So not yet. So this is something that we've been talking about in the team. So for now, only the demo is available to run, but the plan is for us to basically come together at some point to discuss our release strategy. So, more broadly speaking, what you can definitely expect from us is within sort of the next 6 to 9 months, I think you'll basically hear basically a package release of some simulation environment that will let social scientists or other practitioners run these sorts of simulation for their studies. So, that is certainly coming. In sort of a shorter time horizon, I think a part of it is we're trying to figure out how to release this in a more responsive manner. responsible manner, and also there's some cleanup to do there. Obviously, there's the deadline that's involved, so we're trying to get that to work. So on a more shorter timescale, my sort of ideal timeline is by the time we can announce the actual— this paper, not the arXiv version, but the actual paper— you might hear from us then about some kind of release.

Cool, thanks. And I just have a kind of a follow-up to that. You talked a lot through the, um, like how you query the language model with, uh, you know, various prompts to, to get it to do something. But you also talked about how the agents like move through the environment and decide to take actions. And so I, I think in the talk you only discussed actions as like the language model output, but the agents actually act in the environment. So how What is the actual underlying action space that the agents operate with?

Right.

So sort of the grounding question here, right? So, right. So this is the part that I sort of went fast or didn't really get a chance to cover in the talk. It's the more details obviously in the paper, but I can expand on this a little bit further. So basically, once they Sort of decide. So actual animation itself is, that's just game engine. So as long as the agent can go from, hey, I want to eat something, so therefore I'm going to go to the kitchen. So I'm, let's say I'm in common room, but I'm going to go to the kitchen and open the refrigerator. As long as the agent can reason about that, taking that and making that into an animation, that's straightforward, right? That's just game engine. So the question really becomes—

Yeah.

How does the agent know to make that decision? Now, most of it is just planning, right? So at this hour, I eat lunch, or right now I saw someone on the street, I say hi, so I walk towards them. So that planning happens. The only other detail in there is this agent maintains what I, what could consider to be a sort of the scene graph, right? So it's just their spatial memory, and it basically is a tree.

Okay.

So imagine something like— so really the root node or the top-level node of this tree is like the world. But imagine the world just contains my apartment here. So my apartment has a common room, kitchen, bathroom, and so forth. Then the node at the top could be my apartment, and it would basically branch to— it's basically that in relationship, right? So in my apartment, There is the kitchen, there is a bedroom, there's a common room and bathroom. And in, let's say, the kitchen, there's, let's say, objects that makes the space functional. So that's like, like there's a stove, microwave, and refrigerator, and so forth. So it's basically the figure that I showed that has this kind of like a tree-scale relationship. That's what the agent maintains. The agent maintains a subset of the global tree that maintains every object. They only maintain what they have seen and remembered. And to traverse through it, it's just language model, right? So it can be a classification task. I want to eat. I'm in the apartment, and this is all the space I know. Where should I go? And they will likely say the kitchen. And you go— it's a recursive function, right? So once you're in the kitchen, what should I use? And so forth, until you get to the leaf node, which is again an object. And that's what they can influence and interact with.

Okay, so, so there's no— so this brings me to, I guess, the state space. So they don't actually observe the environment. It's all just like a, like a tree-based representation of language. So, and then they act, and then they act on that.

So, in some sense, that's one way to put it, or one way to look at it. They sort of have— they observe the environment in the sense that they have vision. So what basically happens is at any given moment, they can observe a certain distance away from them. So they have, they basically have sight, right? And they can register any changes or objects that comes within that radius, right? So if I see Dave walking by the street and you're within that radius, then I can see you and I can see your state, like what you're doing. Maybe you're taking a walk. Then I register, hey, Dave is taking a walk in the park or something like that. And based on that, that gets updated into their memory and the spatial memory as well. So let's say I never saw that particular building, but as I was walking by, that building came within that distance. Then the agent's memory, spatial memory, as well as their memory stream gets updated with something like, there is that building. This building is generally located in this part of the scene graph. And it was there, right? So that kind of things happen with these agents.

Okay, so there's like, they, they don't directly observe pixels. They, there's basically some function that summarizes what they're like, their partially observable window into text that they then ingest into their memory. And then, okay, got it.

That's exactly right. Yep.

Cool.

Thanks.

I'll take this opportunity to ask a question. One thing that I'm curious for your thoughts on really is, you know, what would, what would a benchmark or test look like for whether this, you know, correctly reproduces some human system?

Right.

Yeah, that's the question.

So this is what we're trying to go deeper into in sort of my work that continues generative agents. So this is basically what I've been up to the past month since publishing this paper. Well, I was on a 2-week break, and then after coming back from it, this is what I've been up to. Here's the general philosophy. So I, I didn't actually cover the technical evaluation of SocialSimulacra because it's sort of overlapping spirit as the generative agents evaluation. But in that particular evaluation, so SocialSimulacra again was one where we tried to recreate existing subreddits, right? And, well, SocialSimulacra was trying to populate subreddits. So, one of the things that we wanted to show was that it would be able to create, recreate existence of Reddit with realistic population and their behavior. What we've sort of noticed was there's an interesting question as to when a language model can recreate certain human behavior. Is it because it can create human behavior, or is it because it's seen it in its training data and it's just repeating that back? That is sort of the main technical hurdle for these kinds of evaluation of generative agents more broadly. The philosophy here, or the general approach here, is it would be the best if we can find things that are not in the dataset, the training data of generative agents and large language models, and see if these agents can recreate that. And that's what we've done with Social Simulacra. So with Social Simulacra, we used ChatGPT, we used GPT-3, so it was 2021, so GPT-3 was still big. And what we've done was 53 back then had the data cutoff. So, there's, after a certain date, it didn't know what happened online. So, we tried to recreate subreddits that came after that date. So, what we basically find is the content that we create with something like Social Simulacra is almost indistinguishable from the content that's actually online. in these subreddits. And I think that's sort of the flavor of evaluation that you want to share with these agents, right? Really being able to create something that doesn't exist in the dataset and seeing how it does. Now, there is an interesting question I'll quickly add around this idea of convergence. So a lot of sort of the evaluation that we've done is more qualitative. It's human evaluation, right? Does this— it's basically really a slightly different version of Turing test. Now, what is interesting though is human behaviors are complex, so that's why we do these Turing test qualitative evaluation because they're unpredictable. We don't know what's real, but at large scale, it sometimes converges, right? That's where you see these converging social phenomena arise. I personally think it would be interesting if we see those convergence. In generative agents. And that's another way that we're thinking of evaluating these agents. We haven't done that, but I think that's— I'm excited by that idea.

Yeah, that's a fascinating answer. And yeah, the post-2021 trick is clever. I think so. Thanks for answering that. Ola, I think you're up next.

Yeah, I was curious. So you talked Sorry, I was gonna say this is maybe a little question, but go ahead.

Uh, that you described how sub-objectives are determined, that the agents kind of autonomously decide on what subtasks they need to accomplish. But I was curious, what is, um, their overarching objective in this Smallville environment? Are they optimizing some kind of imitation metric compared to behaviors that they were trained on, or What's kind of their goal when they're navigating the environment?

Right. So this is an interesting one. They don't have one is basically the answer, or they don't have any explicit, again, objective function set for them. This is, this is where it really deviates from something like reinforcement learning agents, right, where they have explicit objective function that they're trying to optimize for. These agents, we just give them a description. Your a painter or you're a writer or you're a student researcher with a deadline. And it's up to the agent to basically recreate that space. And in part, that's the sort of the core technical challenge of agents like this, where a lot of sort of the game NPCs where they have found success in the past was actually in spaces where you could actually optimize for function, like chess or Go.

Yeah.

These are games where there's a clear objective. You're trying to win against this opponent. The challenge here is there's nothing really that explicit in our actual life, right? Like, what is my objective in life? I don't know, just to live a happy life maybe, but it's a very broad concept. So there's no like one objective that can be encapsulated in a function that we can leverage. So that's what we try to overcome. And generative agents or these generative models in general encode enough human behavior that even without that, by prompting it, we can extract a lot of this behavior, and that's plausible. Now, going forward, I actually do think there is an interesting research question to be had here where we can start asking, well, can we— what is some of the midground between something like reinforcement learning, which has clear objective function, and generative agents right now, which doesn't have any objective? One question that I'm particularly excited by is, can we ask a really high broad level, kind of like give them a meaning of life almost, right? Live a happy life or be successful, right? These abstract concepts, can these agents decompose those concepts into concrete objective functions that they decide on and optimize for that? I think that's an open question, but this is for now where they live in that spectrum.

All right, well, I want to say again, thank you so much for coming and giving this talk. It's really great work, and I'm excited to see where you guys go next with it.

Thank you again for having me. This was— we really enjoyed chatting with everyone and sharing our work.

I have to head out, but talk to everyone soon.