Joon Sung Park | Generative Agents: Interactive Simulacra of Human Behavior

Hi, everyone. I'm Martin Goodson. Welcome to the Machine Learning Meetup. I'm from Evolution AI, who are hosting the meetup, and I'd like to introduce Joon Sung Park and Ryan Yu. Joon will be giving us a talk for about 40 minutes as usual, and then Ryan will be taking over and handling the, the QA section. So as usual, if you've been here before, you're going to be tired of hearing me saying this, but, but we, we're going to use the QA function. for any of your audience questions. Please do add any questions that you can think of to the Q&A as Joon gives the talk. So I think it's best if you get— if you add your questions while the talk is happening. Don't use the chat, do use the Q&A function. The chat you can use to talk to me if you want, if you've got any questions, but for your questions for the Q&A section, use the Q&A function. And also raise your hand if you want to ask your question yourself live. Otherwise, we're gonna ask you a question for you. Ryan will be doing that, actually. Okay, great. So I think we're ready to begin. Over to you, Joon, and I'll see you shortly.

That's good. All right, let me quickly do a screen share. Okay, so hopefully Everyone can see my screen. All right.

Cool.

Hi, everyone. So my name is Joon. I'm a 3rd-year PhD student in the computer science department at Stanford, working with Michael Bernstein and Percy Liang. And today I will talk to you about generative agents, interactive simulacra of human behavior. And this is work that I developed with my advisors, Michael and Percy, as well as my mentee, Joey O'Brien, and mentors, Mary Morris and Kerry Cai. For over 4 decades now, from the time of cognitive architectures and symbolic systems to statistical machine learning, we, the researchers and practitioners at the intersection of human-computer interaction and AI, envision the ability to simulate believable human behavior, Behavior that is so compelling, so human-like, that they provide an illusion of life. In our vision, this ability, if achieved, promised a new class of interactive applications ranging from model human processors for usability testing to social robots, NPCs, and ubiquitous computing applications that require a rich understanding of human cognition. And even to the foundation of small and large-scale social simulations that would test social science and economics theories that are difficult to implement in real life. But despite their wide application spaces, we faced fundamental challenges when simulating human behavior. The space of possibility in the way we behave and communicate, we found, was much too vast and too complex to navigate with existing methods. But I see a new opportunity that is emerging. Generative models such as large language models that are being trained today are trained on broad data that reflect our lives, like the traces on our social web, Wikipedia, and more. As a result of that, these models encode a tremendous amount about us, how we live, talk, and behave. So I posit that with the right method, they can be transformed into the core ingredient that had been missing in the past decades. They will enable us to simulate believable human behavior. So today I will introduce a new way of simulating human behavior in fully general computational agents. That can populate an open world much like ours, while ensuring long-term coherence by fusing a large language model with a novel agent architecture that remembers, reflects, and plans based on constantly growing memories and cascading social dynamics. These agents, as I demonstrate, can not only plan and lead a believable day in life where they wake up in the morning, do their routines, and go to work as individuals, In a sandbox game environment, but they can also come together. To give birth to an entirely artificial society of their own, like the one you see here, where each agent will have their own subjective memory and experience. And spread information, form relationships, and coordinate amongst each other before reflecting on the past days and deciding on how they will live tomorrow. These are generative agents. And these generative agents, I'm going to suggest, open up a new genre of human-AI interaction that is fueled by our newfound ability to simulate believable human behavior. So with that, let me demonstrate to you in just a little bit more detail Smallville. So this is the setting of our demonstration for generative agents and the mode of agent interaction that takes place in it. So Smallville is a sandbox game environment that we developed featuring the common affordances of a small village, ranging from houses, apartments, cafes, bars, schools, and stores, and the sub-areas and objects that make the space functional, like the bathroom, kitchen, and common room in a family house, and a bookshelf and a table in the common room. And we populated this space with 25 generative agents and initialized each of them with one paragraph natural language description to depict each agent's identity, including their occupation, relationship with, with other agents, and seeded this paragraph into the agent's memory as the starting point of their experience. And that is it. This is all the input that we ever give to these agents. And then these agents interact with their environment through their actions based on their own volition. And here's how this works in Smallville. So first, the agents generate a natural language statement describing their current action, such as Isabella Rodriguez is drinking coffee. They then translate this into concrete movements that affect the sandbox game world, along with the automatically generated emojis that visually describe the agent's actions. And they influence the state of the objects in the world. So a bed can be occupied when an agent is sleeping, and the refrigerator can be empty when an agent uses up the ingredients to make breakfast. Then to interact with each other, they determine whether they want to engage in conversation when they see another agent, and they generate natural language dialogue if they decide to engage. Like this dialogue between Isabella and Tom about Sam Moore. So Sam is a fellow agent in Smallville initiated with a memory that he's running for local mayor. Here, Isabella remarks, I'm still weighing my options, but I've been discussing the election with Sam Moore. What are your thoughts on him? And Tom responds, to be honest, I don't like Sam Moore. I think he's out of touch with the community and doesn't have our best interests at heart. And importantly, the users can also influence and interact with these agents. First, much like how agents form dialogue with each other, a user can engage in a dialogue with these agents by specifying a persona that the agent should perceive them as. For instance, if the user specifies that they are a news reporter and asks about the upcoming election in Smallville, who is running for the office, Isabella might reply, Stan. Or to directly command one of the agents, the user can take on the persona of the agent's inner voice. And this makes the agent more likely to treat the statement as a directive. So if the user tells John that he's now running for office while taking the persona of John's inner voice, John would decide to run in the election and share his candidacy with his wife and son. And second, just as agents can, the user can also alter the state of the agent's environment. For instance, if the user sets Isabella's toast on fire, Isabella would rush out to put out, put out the fire and remake her toast. And finally, the user can actually control an agent by inviting an agent already present in the world, such as Isabella and John, or join as an outside visitor. The inhabitants of Smallville will treat the user-controlled agent no differently than they treat each other. They will recognize its presence, initiate conversations, and remember its behaviors before forming opinions about it. Now let me dive a little bit deeper and present you with some vignettes from Smallville that describe their individual as well as, as well as collective emergent behaviors. As individuals, generative agents create daily plans. that reflect their experiences, execute those plans, react, and replan when appropriate. So here's an example day in the life of the Lin family. So the Lin family lives in the sort of the lower right side of the map. So you see their house right there. And the Lin family is a family of 3, the mother Mei, who is a college professor, The father, John, who is a store clerk at the local pharmacy, and the son, Eddie, who is a student at the college who studies music theory. In the Lin family, John is the first to wake up at 6 AM. He brushes his teeth, takes a shower, and cooks breakfast. And throughout the morning, other family members follow suit, catch up with each other, and by 8 AM head out to their respective workplaces. May and Eddie to the college, And John to the pharmacy. Let's eavesdrop on them a little bit to get a sense of what they are talking about. Again, their movements, decisions to engage in the dialogue, and the dialogue themselves are all generated. Nothing you see here is hard-coded. So here John and Eddie are catching up in the morning. John says, good morning, Eddie, and Eddie responds, good morning, Dad. And John asks, what are you working on today? And Eddie responds, I'm working on a new music competition. And only a little after the conversation, Mei wakes up and joins John. By now, Eddie already left for his classes, but John recalls the conversation he and his son just had and finds that to be relevant here. He updates Mei that Eddie is working on a new music competition, and Mei responds, Oh, that's wonderful. And meanwhile, as a collective in this practically a small society of generative agents, these agents exhibit emergent social dynamics where new relationships form, information diffuses, and coordination arises. Let me go over each of these emergent behaviors in a little bit more detail. So first, information diffuses across the agent community. We've already seen a glimpse of this. with the Lin family, but here's another example. We see that Sam would have memory that he's running for a local election, and he's telling everyone about it throughout the day. Here he tells Tom, I'm actually running for mayor in the upcoming election, and Tom says, really? That's great news. And a few hours later in the game world, John and Tom, who are colleagues at the local store and pharmacy and have independently heard about Sam's candidacy, talk to each other about Sam's chances of winning. Joan says, I heard that Sam Moore is running for mayor. Do you think he has a good chance of winning? And Tom responds, I think he'll get a lot of support. Second, new relationships form among the agents in Smallville. Here's one example. Latoya and Sam do not know each other at the start of the simulation, but while taking a walk in Johnson Park, Sam runs into Latoya, and they introduce themselves. Latoya tells him that she's at the park to take some photos for a project that she's working on. And the next day, when Sam sees Latoya again, he and Latoya remember each other. And this time Sam asks Latoya, how's your project going? Finally, agent coordination spontaneously emerges in SmallBuild. In our demonstration, we set the starting date to be on February 13th. And with Isabella, who is the owner of Hopps Cafe, we seeded an intent to plan a Valentine's Day party from 5 to 7 PM on February 14th. From that seed alone, Isabella invites friends and customers to the party, spends the afternoon of the 13th decorating the cafe for the occasion, and enlists Maria, a friend and a frequent customer at the cafe, for help. And meanwhile, Maria asks Klaus out. Who is her secret crush, and decide to go to the party together. And on the day of Valentine's Day party, 5 agents, including Klaus and Maria, actually shows up at Hop's Cafe at 5 PM, and they enjoy the festivities. So how do we do this? How do you achieve this agent behavior? Our main contribution in this work is basically this architecture represented in this figure. This is the architecture of generative agents that powers each of these agents in Smallville. At the center of this architecture is what we call the memory stream. It is a primary database that maintains a comprehensive record of an agent's experience in natural language. From the memory stream, records are retrieved as relevant to the agent's cognitive processes, And let me go over the modules of this architecture in more detail. So first is the memory and retrieval. Here's the challenge that we're trying to overcome with this module. Generative agents in Smallville, and likely beyond, accrue an extremely large set of records in their memory stream. And feeding the entire memory stream to a large language model can distract the model. And today, not even a few hours worth of memory in Smallville can fit into the limited context window of the state-of-the-art language models. So our agents need a way to store and selectively retrieve portions of their memory. And that is the aim of the memory stream and the retrieval function. So here's a tip of the memory stream of an agent, Isabella. And on the right, you see a few example memory objects in the stream that contain a piece of memory described in natural language. with a creation timestamp. So here, for Isabella again, um, you see objects like she's writing in her journal, taking a break, and when she saw the refrigerator being just idle. So in particular, these are what we call the observational memory of these agents. So these are factual memory that the agents actually experienced in their world. And based on this memory stream, our architecture implements a retrieval function. That takes the agent's current action as input and returns a subset of the memory stream to pass on to the language model, which then generates the final output behavior of the agent. So in this example, if the situation that Isabella is trying to react to is someone asking, what are you looking forward to the most right now? She would retrieve things that are about the party and formulate a response I'm looking forward to the Valentine's Day party that I'm planning at Hobbs Cafe. Here's how the retrieval function works. So in our architecture, we designed this as a combination of the recency, importance, and relevance function for each piece of memory. So basically, we bias towards retrieving memories that are recent, important, and relevant the current context. In our work, the recency function is implemented as an exponential decay function. The importance function is a prompt that asks the large language model for an event saliency. So conceptually, this is almost like just asking the agent, is this particular event important to you? And the relevance function is a cosine similarity score of the embeddings of the query sentence and the description of a memory. Now, what we've noticed in our work is that with just the raw observational memory, our agents struggle to generalize or make inferences. So periodically we synthesize clusters of records in agents' memory stream into higher-level abstract thoughts that we call reflections. And importantly, once they are synthesized, these reflections are just another type of memory. So these, they are stored in the memory stream, just like the observational memory. So there's nothing special about them in the way we represent them in the memory stream of the agents. Here's the way we do this. We first generate questions on what to reflect on by looking at 100 most recent records in the agent's memory stream. So if an agent just had lunch, it could be something like, what does the agent like to eat? Then we retrieve records that are relevant to answering those questions, which might be things like the agent ate an omelet today, yesterday, and the day before yesterday. And we synthesize that into a reflection. Maybe the agent likes to eat omelets. Over time, what this generates are trees of reflections. The leaf nodes are the observations, and the non-leaf nodes are thoughts that become higher level the higher up the tree they are. So here's one example from Klaus. So for context, Klaus is a student researcher studying sociology. So he's right now researching about topics related to gentrification and so forth in this world. So if you look at the lower left side of this tree, you see the raw observation that Klaus has made throughout the day. Things like Klaus is reading about gentrification, uh, he's reading about urban design, and so forth. So those are his self-observations that he's noticed, uh, as he was behaving in the smallville. And he formulates a reflection based on that that might say something like Klaus spends many hours reading. Now that is now a reflection that is no longer the raw observation. It is still very much grounded. It's the first-order reflection. So, it's only the first level within this tree. But it still becomes a reflection. It becomes an opinion about his behavior. Now, this gets mapped into even higher-level reflections as we go up the tree to formulate reflections such as Klaus is engaging in research activities, he's dedicated to research, and so forth. And at the very top, we started to formulate opinions about who Klaus is, what are his interests, what drives him, and so forth. And this is sort of the, the role of the reflection is to gain a deeper understanding for these agents of their surroundings and of themselves. Finally, while a large language model can generate plausible behavior in response to situational information, we find that optimizing for believability in the moment can sacrifice believability over time. What we needed was for the agents to plan over a longer time horizon. So plans describe a future sequence of actions for the agent and help keep the agent's behavior consistent over time. For instance, for Klaus, who is a student researcher dedicated in his research and has an impending deadline, he would generate plans to spend his day working at his desk, drafting his research paper. In our work, we generate the plans by prompting a large language model with a prompt that summarizes the agent and the agent's current status, as you see here for Eddie. So you see the first half of this prompt, we basically have a brief description of who Eddie is, his name, age, basic characteristics, and down below we include things like what his plan is for today, as well as what he's seeing right now and whatever he needs to react to. If that becomes, if that becomes relevant. A challenge here, however, is controlling for the granularity of the plans that are generated. In our work, we control for this by taking a top-down approach where we recursively generate more details in the plan. So here's one example. On the left-hand side is Eddie's plan generated in broad strokes. the device's days into roughly 7 chunks. We then decompose these chunks first into hourly schedule, then into 5 to 15-minute actions that you see on the right-hand side. Once we reach the desired granularity in the agent's schedule, they can act out the plans in the game world. And what's actually quite nice about this particular setup is because it is recursive, depending on the needs for your particular application, we can move in any directions in terms of the granularity as needed. So we can plan for even longer time horizon, or we can decide to decompose these plans even further to a minute-by-minute schedule, for instance. For our purpose in the game world, 5 to 15 minutes was sort of the sweet spot that got us the believable behavior while not being too computationally expensive. But sometimes the agents may need to change their plans. For instance, if Eddie's father, John, Recklessly, he sees Eddie taking a short walk in the house garden. He might decide to start an impromptu conversation. We prompt the large language model as shown here to make this determination and edit the agent's plans if the situation calls for their responses. Now that I've described our agent's behavior and architecture, we can now ask, how do we evaluate these agents? The main dependent variable that we use is believability, which has been a central design and engineering goal in a long line of prior literature. So basically, do agents remember, plan, react, reflect believably? And to evaluate generative agents' believability, we leveraged a methodological opportunity. By interviewing them in natural language. In particular, we craft 5 categories of questions, 5 questions each, where to respond to these questions properly, the agents must successfully retrieve and synthesize information to stay in character, remember, plan, react, and reflect in a believable manner. Here are some example questions and answers from CLAS's interview. When we asked him to give an introduction of himself, he properly recalled his name and basic characteristics. And when we asked him that he would— what he would do if his breakfast is burning, he tells us that he would quickly turn off the stove and make sure the food doesn't continue burning. Our study procedure was as follows. We tasked our generative agent architecture as well as the ablated architectures and human authors to answer the questions. We then asked 100 human evaluators to rank the answers from different conditions. We then calculated true skill rating for each conditions, which is a generalization of the Elo rating system. So, if you're familiar with how chess players get rated on the pairwise basis, uh, they use a system called Elo. Fruska basically relaxes some of the assumptions so we can do multiple agents comparison. So in this case, we have 5 conditions. So we're trying to see which condition comes at the top in terms of their performance and believability. What we find is that the components of our agent architecture— observation, plan, and reflection— each contribute critically to believability of the agent behavior. So the red bar is the performance of our full architecture, and it significantly outperforms every other condition, including the human author condition. It's also worth noting that when compared to the condition representing prior work, which is the no reflection, plan, and observation condition, our full generative agent architecture produces a standardized effect size of Cohen's d equal to 8.15. Or another way of saying this, is that 8 standard deviation difference compared to the, the baseline condition that we've seen in the past. Of course, I would note here that this does not mean that our agents were without flaws. They would sometimes fail to retrieve certain memories, like when Rajeev answered, I haven't been following the election too closely, even though he talked to Sam to hear about his candidacy. And they would sometimes embellish their memory. For instance, Isabella was aware of Sam's candidacy in the election, and she confirmed this when asked. But she also added that he's going to make an announcement tomorrow, even though Sam had mentioned no such plan. Additionally, we conducted an end-to-end evaluation of the agents to better understand the types of emergent behavior we observe among generative agents. First, we find that agents shared and remembered information. 7 agents heard about Sam's candidacy, and 12 agents heard about the Valentine's Day party. You can actually see the path of the party invitation across the agent community on the right-hand side here. So you see Isabella telling everyone, I'm planning a Valentine's Day party at Hop's Cafe. So she tells this to Giorgio, Eddie, Sam, And Sam, on the other hand, in response, tells this to Jennifer, who is his wife. Speaking of which, Isabella has invited us to the party. Second, we find that the agents remembered and joined the Valentine's Day party. In particular, 5 agents came to the party. Of the ones who didn't make it, 3 cited conflicts, like Rajiv, a painter, who explained that he was too busy. And 4 showed interest in the party but still did not show up. Now, this does make things difficult to evaluate, though. On one hand, we might view this as an error. On the other hand, this is extremely realistic human behavior in my experience. But of course, there were boundaries and errors in our agent behavior. One that is particularly noteworthy here is that the instruction tuning of the agent language models themes seem to guide the behavior of the agent to be overly polite and cooperative. Even when talking to her family, for instance, Mae always greeted them very formally. And Isabella never really refused ideas to include in her party, even when the ideas did not exactly align with her identity.

Okay.

So that's generative agents. And We posit that as sort of we were discussing this work, uh, since we put this up on the archive a few months ago, I think there was a lot of interest from a surprising number of fields that we did not expect. And I think a lot of people are sort of looking at generative agents and envisioning, envisioning a potentially large space of applications. So for the remaining 10 minutes or so of this talk, I want to discuss one such application space. where I think generative agents could potentially unlock a new way of designing difficult-to-design systems, uh, in ways that we couldn't do in the past. So, this comes in the form of my prior work called Social Simulacra. So, you can very much tell that it's in the same line of work, given that the term simulacra is in both titles, uh, for generative agents in this work. Uh, as you can notice, uh, this is work that we published, uh, at a conference called UIST in 2022. So, this actually predates generative agents. So back in the, uh, so this was our first work that tried to use large language model. In fact, I think the first work in this space that tried to use a large language model to explicitly model and generate human behavior. Um, but when we did this, we actually didn't have the full generative agents architecture set up, but the philosophy is there. So you can, I think you'll easily be able to imagine if we replace our techniques that I'll be presenting you here or social simulacra with something like generative agents. In this particular case, we just use prompt chains. So basically simple prompting to a large language model to generate human behavior. With generative agents, this can become even more powerful. So this, I think the goal of sort of next 10 minutes is for me to convince you that there's a really interesting application space that can be unlocked by something like generative agents going forward. Okay? So this is work again that I developed with my advisors, Macklem and Percy. as well as my mentee Lindsay Popowski and mentors Mary Morris and Kerry Cai. So to change gears a little bit, for many decades now, we have designed and deployed countless social computing systems. But the irony is, even today, as more and more people populate these systems, we continue to get surprised by the things that happen in them, like unexpected trolling or subtle antisocial behaviors, all the way to cases such as people congregating to spread hate speech and misinformation. But why is this? Because in theory, this issue of understanding how people might use an interactive system is something that we know how to tackle. That's what prototypes like the one you see here, are all about. But here's the challenge. In my vision, a successful social computing prototype needs to prototype not, for instance, the user flow of how one might click around different pages, but the social dynamics that might arise when the system reaches critical mass of users, because that's where the uncertainty is in social systems. And that has been an impossible task for our existing prototyping approaches. Where are we going to get thousands or even tens of thousands of diverse test users? How would we prototype the social dynamics that might arise without releasing these designs to a large number of people? Generative agents enable a new way of prototyping a social computing system that tackles this challenge by generating a large number of synthetic social interactions that might arise in a populated social system. We call this social simulacra. Social simulacra is an approach. It is an approach of leveraging the richness and the generative capacity of models such as a large language model to populate a social computing system with generative agents and behaviors for the purpose of prototyping the system design. And we'll demonstrate this approach in practice by implementing it as a web application tool for prototyping subreddits that takes community designs such as the goals, rules, and moderation strategies and translates them to a generated community like the one you see here that has never existed before to illustrate the good, the bad, and the ugly of the interactions that a community governed by these designs may harbor. So let me show you what it can do. Our tool has 3 core features: Generate, What If, and Multiverse, and they each answer the core— to the core needs in social computing design. So first, Generate. Prior literature suggests that social computing designers struggle to envision the breadth of interactions that their design might facilitate. Generate helps the designers by populating a subreddit community with generated users, tab-level posts, and replies to those posts to help them envision the space. So it takes from the designer a range of design specifications, such as the goals, rules, and member compositions, and returns a subreddit-like page that is fully populated, like the one you see here on the right-hand side. Here's an example. In these slides, everything on the left is the designer's input, and everything on the right is SocialSimulacra's output. So in a community for connecting people moving to Los Angeles with locals, SocialSimulacra generated a new user, Leon Santos, who posted, I'm new to LA. What are some of the best places to visit on the weekend? And in response, our tool generated Lucas Jameson, who replied, I recommend visiting the Getty Center, the Museum of Contemporary Art, and going hiking. And here's another one. This time in a community for people interested in learning about personal finance. Those are similar to our generated Dane Wood, who posted, I spent $21K to go to college and ended up with $23K in debt. But in response this time, our tool generated Elizabeth Neal, who is a troll. Here she said, that's a lot of debt, man. I haven't seen that much since I shopped at Macy's during the holiday season. The second feature, What If, aims to give the designers more interactive control over Social Simulacra to help them explore how individual conversations might be influenced. So What If takes from the designer an existing Reddit conversation and regenerates it from the middle of it, as you can see here. So imagine Maya Smith posted to a forum for Wist authors She said, I've been working on my WISC paper for a few weeks and I'm feeling really stuck. And to this, Heather Hernandez, who is a human-computer interaction professor, replied with a short note, short advice: it's normal to feel stuck when writing a paper. Good luck. But as a designer, you might want to know so you can prepare for it. What if instead of Heather, a troll replied? Or what about an advertiser? The what-if feature answers to that precise need. So here, instead of header, a troll replied, you're just not cut out for this kind of research, whereas here an advertiser replied with, click the link below to learn more. Equipped with this, the designer can now even ask, what could the moderator say in response to this undesirable behavior? And if the moderator intervened with no advertisements, please, to the advertiser, what are some of the ways the conversation could have developed from that point on? The final feature is multiverse. And this is a feature that makes the model uncertainty explicit. With the first 2 features, generate and what-if, what I've shown you is our tool's capacity for generating realistic content. But human behavior is inherently complex, and no matter how likely something is to happen, there's no guarantee that it will happen. And this is an important conceptual scaffold for social simulacra, but more broadly for generative agents. So instead of just showing the designer one possible outcome, Multiverse augments the previous 2 features, simulate and what-if, by explicit— explicitly showing many possible ways the interaction could have developed from one initial state. This encourages the designer to explore the design space with inductive insight rather than over-relying on the one prediction that was made by the model. So, are social simulacra successful in what they're meant to achieve? Now, as a part of this paper, we've done a technical evaluation that's very much reminiscent of the generative agents technical evaluation. So, I skipped that part, but the evaluation that I did want to convey was the designer evaluation. Does this actually help human designers to create new social platforms? So for this evaluation, what we've done was we recruited 16 participants who had experience designing social spaces and asked them to design a new subreddit with the help of Social Simulacra. What we find is that today, without social simulacra, the design practices for social computing systems are reactive. So participant 8's comment is particularly salient, which says basically all the rules are set in reaction to the dumpster fire. And our participants cited this as the reason why social simulacra could be really valuable, because it would offer them a sense of security if they could try out different iterations of establishing norms and standards before they actually release their design. We also find the simulacra offered concrete design insights to our participants. Among them are finding unexpected model citizen behaviors, like impromptu friend-seeking to go sightseeing in a community for sharing fun events around Pittsburgh, and finding unexpected undesirable behaviors, like a troll farm shifting the tone of a discussion community. And with that, that concludes my talk. I thank my advisors, collaborators, and funders one more time, and thank you all for joining.

Thanks so much, Joon. What a fantastic and really fascinating talk. Um, so I'd like to hand over to Ryan now, uh, who will conduct the Q&A. I can see there's a few questions coming in from the audience, but keep them coming into the Q&A function. And also remember to put your hand up if you would like to ask your question live. Otherwise, Ryan was gonna ask your question for you. So thanks again, Joon, and over to you, Ryan.

Hey, Joon, great talk. Thanks for sharing both of your works. They're very interesting. I'll start off with the first question from Dominic. Dominic says, I understand the concept of reflection memory, Uh, how often is this updated to obtain an understanding of the agent itself? Um, uh, suppose an agent has a bad week or month, and therefore the stream of memory becomes affected. Would the agent now essentially have a new personality since it changed its behavior and interests based on the recent actions?

Right. Yeah. So, this is a great question. Um, so, in terms of sort of— this does go down more into implement— implementation details. I can see this change or being improved even further in the future, but this is how we do this now. The agents reflect roughly 2 to 3 times a day in the game time. And each time they do this, they create anywhere from 10 to 20 reflections or so, or maybe a little bit less, but that's the range. The way we determine when they basically do the reflection is basically by accumulating the importance score. So I mentioned in the retrieval function we use 3 features: recency, importance, and relevance. And as the part of the importance function, what we do is we ask agent to calculate an importance score for every event that took place in their life in a 1 to 10 scale. So imagine eating breakfast. That might not be too important for a lot of people, or not be maybe very memorable. So that could be something like 1. or 2. But let's say you broke up with someone or you're graduating. That's likely not memory that you're going to forget. So that's in the range of 9 to 10. And basically when these agents decide to reflect in our implementation is when the sum of the, the most recent importance scores from their most recent records exceed 150. So you can basically imagine you had a particularly eventful weekend, You likely are reflecting more than usual because you're gathering more information that has higher scores. Whereas if you had a very normal, quiet day, you might not reflect as much. In terms of whether this changes their characteristic, yes, it certainly does change their behavior. So one of the things that they could certainly do with this reflection is basically gather them and try to resynthesize their characteristics. that will basically be used as a basic descriptor in all the prompts to the large language model that will define their behavior. So it does influence their future behavior, and that's one of the core roles that the reflection module is supposed to play here.

Sounds great. That's, uh, that's very interesting. Thank you for the answer. Um, next, uh, Johnny asks, uh, you write that create— to create the initial plan, You prompt the language model with the agent's summary description, such as their name, traits, and summary of their recent experiences, and a summary of their previous day. He's asking about how the plan summaries are created specifically, as well as maybe some more technical details on how to create the plan summaries.

Right. There are a few components to it. So, at the very— so, the, the plans themselves again are all generated by the large language model. So, it has a— that particular prompt has a few components. At the top, it has a basic description of the agents. So, for— in my case, it's about a paragraph long. Uh, in, in practice, it's just a natural language text. Uh, when we initialize them, we manually author them. So let's say to describe someone like me, you might say something like, oh, Joon is a computer science PhD student. He's 29. He's living in this Palo Alto area, and he has— he knows these people, and so forth. So about one paragraph that describes Joon. And you basically use that and as a sort of as part of the context, and then you basically ask the model, Today is— what's the date today? August 2nd, Wednesday. What is he going to do? And as a part of it, it might sort of retrieve from memory the things that June was supposed to do. Imagine for many of these agents who was planning to attend the Valentine's Day party, it might be something like, oh, there's a party going on today. So I'm going to filter that into my schedule. So they just create the hourly schedule, and they generate the, the more details in that schedule by decomposing it. And later they summarize those entirety of the schedule in maybe a few sentences. What are the sort of the high-level broad stroke schedule for today? And that goes back into their memory so they know what they've experienced, or at least what they had planned to do today.

Sounds great. Yeah.

I had a question actually, Brian, if you wouldn't mind, just kind of a follow-up to that. Joon, I couldn't really understand in some of the paper, It's not quite clear how much of the, you know, how much of the prompts is hardcoded and how much of it's generated. I couldn't really work it out. So I was looking at, I think it's section 4.3.2, dialogue, talking about how John, I think, is asking Eddie about his composition projects. And it says in the prompts, it says John is asking Eddie about his music composition projects. What would he say to Eddie? I couldn't understand whether that was hardcoded or whether that was derived from Something else that— could you help me to understand that?

Yeah, for sure. So in the particular case of— and this is actually the case with all the prompts that we use— there's a template, right? So in— for creating dialogue, the structure of the prompt remains the same. So the structure might include John, and this is because I forget the exact wording of this prompt, but it might say something like, John sees Eddie doing X. That's the context that John is observing. Eddie doing X is literally just fed in from the environment to John. So John, that's the, the context that John is seeing, and John is the person who needs to react to that. So John sees Eddie doing, you know, walking around. What kind of things would he say? That part, that's a part of the prompt. So what is dynamically being generated, or sort of the following the template, is we basically replace who is a subject. of this experience. In this case, that's John. So we basically have a template tag that basically replaces that with the current subject and the subject's current experience. Eddie is doing X, right? So we dynamically generate that from— now we just get that from the environment and we plug that into the prompt. And that's how most of these, uh, prompts are generated. Again, there's a template and we mix and match, uh, the subjects and their current experience and so forth. Now, sort of the more interesting part to how this template gets used is sometimes we do need to synthesize their characteristics, right? So let's imagine sort of the one common, common sort of prompt that we use often have characteristics of the agent, the subject agent, right? For instance, when John is planning for his daily routine, we need to tell him that this is a description of who you are. before we run that prompt, we sometimes generate the description of who John is, uh, because it may have changed. So there are parts of the prompt that are not simply following the template, but also being dynamically generated from ground up. So it's a combination of those. But, um, I think in philosophy, if you've sort of seen how folks at LangChain does it, uh, recently, so that's, that's been one of the more popular sort of a large language model, uh, library that's been— that people have been using. It's very similar to that in terms of how we think about the templates and so forth.

So, so in this particular case, this, this particular line, John is asking Eddie about music composition project. Is that dynamically created, and how is that dynamically created?

So John is asking Eddie. So John, then we just replace the name. So imagine there just being the template subject name. We replace that. Is asking. The asking part might be dynamically generated, but it basically is— the question here becomes, John is X, and X is the situation that the context that John needs to react to. In this case, he needs to ask Eddie, um, about what he's doing or what function he's working on.

So, the music— the question— the thing about the music composition, this is not part of the template. It's dynamically created. The fact that he's asking—

That's right.

Okay. I got it. Yeah. Thank you. Appreciate that.

The—

and just to add on, to finish the thought, it is dynamically generated, but that part is generated actually by Eddie's agent, not by John's agent. So, Eddie's agent has generated its current status that I am working on a new music composition. And in the sort of the under the hood, it basically has a tag on top of it that basically says, I'm working on a new music composition. So for every other agent that sees me, they will see, let's say, Eddie. I'm Eddie, for instance. Then Martin and Ryan, when you see me, I basically say from your view as an agent, Eddie is writing, working on a new music composition, or so forth. So that's something that Eddie has generated.

Not based on a belief, in inverted commas, that John may have about Eddie.

That's right. So in that particular instance, it's just factual information about what Eddie's doing.

So they have direct— the agents have direct access to the kind of state or description of the other agents.

Okay.

Okay. Thank you.

Yeah, just a follow-up to that. I'm wondering, if you've explored the possibilities or if you plan to explore the possibilities where instead of just having these attributes displayed to people that see the person, instead you have this sort of perception also involved where people form inductive biases about other people based on what they can observe. And then—

Right.

After that, then these get updated by the actual behavior that the agents are doing. And then, eventually you figure out that, for instance, Eddie is working on a music composition. Do you feel like that degree of nuance would be interesting? Or— yeah.

Yeah, so that's a great question. And I would think that would definitely be interesting. So, this is the way I've been thinking about this. So, in the case of Smallville, Um, these agents, when, when we say they are making breakfast, for instance, they're not actually making breakfast, or they're not actually acting out much other than to just walk to the kitchen and show the emoji that pops up on their head. So this state that each of these agents sort of expressed, like, I'm working on new music composition, or I'm, let's say, making breakfast, this is basically a replacement for the physical action. So imagine if we had a much richer environment, then a lot of times— imagine I'm making breakfast right now, and one of you walk into my apartment and sees me make breakfast. Odds are high that you likely see me, hey, Joon's making breakfast. I mean, it's early. Well, it's, it's in the morning. It's still morning here. So you might just guess, you know, guess, hey, Joon's making breakfast. But it's also possible that you might not infer that. So, I think to some extent, it really has to do with the richness of the environment that these agents are interacting with. Uh, in this particular case, the 2D scenario, I think we just decided to just make it more straightforward by saying, this is the state that you're in. Um, but I think going forward, as the environment gets richer, I think this kind of inferences would be really interesting. Now, one thing that agents do certainly is they form reflections. They do form biases and opinions about other agents. So it's sort of— I gave an example of Klaus forming an opinion about himself, but they do that for other agents that's not themselves as well. So over a longer time period, they start to form these inductive biases, and it certainly does change the way they view other agents and the way interact— they interact with them. So sometimes they might just say, hey, I'm not going to talk to this agent because I haven't enjoyed my interaction with this particular agent, and so forth.

I see. Cool. Thank you for your answer. Um, Johnny, uh, raised his hand. Uh, Johnny, would you like to speak?

Hey, Johnny.

Johnny, you can speak. You want to go ahead? Johnny, did you want to go ahead and ask your question yourself?

If not, I can ask your question for you. Johnny asks, regarding planning, how do you maintain continuity across changes to plans, or is it not that important? For example, when Eddie sees his son, he may change his plan and decide to interact with his son. Does that require recomputing all future plans for the rest of the day for Eddie as well?

Right. Yes. The quick answer is yes. When an agent has to react, now maybe not the entire, so it really depends on the degree of change that is necessary. Sort of at least in SmallBuild, the most common changes that we need to make is let's say, oh, now I'm just walking on on the street and maybe I see Johnny walk by and I want to say hi. That's a fairly small change. So we don't— it will be sort of computationally inefficient to redo the entire day's plan. But both agents, both participants, so in this case me and Johnny, will have to change our plans to some extent. So it's not just me changing my plans, but it's the both parties. Everyone involved in that has to change the plan. And here it might just involve changing the plan for the hour or 2 hours maybe. So 2 hours is sort of the minimum chunk that we move in actually in planning in this world. So we might change 2 hours worth of plans. It's entirely possible that depending on how long the conversation goes, uh, the agents might not need to change much. So this, that's entirely up to the agents. So imagine I was going to pick up some grocery and I decide to chat for 5 minutes. It might just mean instead of spending 15 minutes in the grocery, I might just spend 45 minutes. Mm-hmm. But if I had much longer, like half an hour, and sort of an interesting way we decide on how long a conversation is in this world actually is by counting the, the number of characters in the conversation or the dialogue that was generated. So we actually have some hyperparameter that matches for every character. This is how many seconds in game time that they spend. And if I recall, we basically, or I basically came up with that number by Literally reading out loud some of the conversation dialogues and seeing how long it takes me to say those in, in real time. So if they have a really long conversation, let's say for half an hour, then I might just need to cancel my trip to grocery store entirely. So that could happen, but that's for now how it's implemented.

Sounds great. Thank you for your answer. The next one. Oh, yeah.

I was just gonna jump in with a question, if you didn't mind. I was gonna ask about the prototyping of the social networks, kind of Reddit type of social networks. I thought that was really interesting. And obviously some of the examples you gave us, like qualitatively, they look great. But I wondered whether you'd investigated, Joon, quantitatively some of the social dynamics. You know, I don't even know how you would do that, but I suppose there are some ways that people have studied the human societies, you know, how do ideas go viral, or how likely it is that people believe or doubt each other, or get angry when someone else gets angry. I guess there's some kind of quantitative study of this in social sciences, or even simple things like how— what's the distribution of replies that a post gets? I guess there's a kind of distribution that governs what's likely in a social network, how many likes a post is going to get. Have you looked at any of those kind of quantitative things?

Yeah, so this is— the quick answer is actually, um, my sort of latest work touches, uh, really deeply into this aspect. I'm personally quite interested in how something like generative agents could be used for social science, and I think that's a question that you are sort of touching by asking this question. Obviously, there's been a lot of interest around, like, network science in the past, Can we use generative agents to simulate a network to study those kind of dynamics and so forth? So we're looking into this really deeply. Now, we've done some work in the past, uh, where in addition to sort of generating dialogues and conversations online, we would try to basically get a quant result on their more discrete behaviors on online social platforms like the ones you mentioned. Uh, will this person like this? Will this, uh, let's say, content go up in the feed based on its popularity and so on and so forth? Uh, so the quick answer is, and this is not yet sort of a— we're still working on this, but the early sort of result here seems to indicate that yes, there is some statistical sort of match between what happens in real life and in simulation. Um, and we're seeing more of such results coming out in the recent I'd say a few months since we published Generative Agents. I think there's totally that interest. Now, I could actually give you one piece that we've been doing. For now, it's been more of a— it's been a part of our latest effort. But I think one thing I can share right now is we basically try to simulate the web. from ground up. So imagine this is the day of the ARPANET. The internet didn't exist, uh, and we're the first comers to the internet, or the generative agents are, and they need to now recreate all the internet. What's really nice is because ChatGPT can create HTML files, these agents can also create HTML files too. So when we say these agents write a website, they literally are writing HTML and CSS and actually hyperlinking each other. And what's really cool about doing something like that is you can actually calculate the network structure of the synthetic web. And if you look into sort of prior literature on network science, sort of the well-known result in that field are things like scale-free network emerging, which basically means there are hubs or influencers that arise in these communities and so forth. The question can become, do we replicate that? And we do. we very quickly see scale-free network emerge where there are some nodes that are exponentially more linked than the others, and so forth. So I'm quite intrigued by this direction.

Super interesting. So just one tiny question for that work that you just mentioned, like the creation of the internet, but also for the Smallville work. What happens if you introduce— well, what do you think happens if you introduced one much more intelligent agent into that system? And do you think that it might shed some light onto what might happen to human society when we introduce very intelligent or super intelligent agents into us human systems?

Right.

So there are interesting questions that we can ask around it for sure. Now, technically speaking, I, I, I think ultimately defining what it means to be smart might, uh, might just need to come first as the sort of the, the, the slight detailed piece here, in part because everything is dependent based on ChatGPT. So every agent technically, if we use the same model in that capacity, they are just about as smart as ChatGPT. We can actually tell one of the agents, hey, or we can tell like, let's say 99% of the agents behave like a 5-year-old, and maybe we introduce one agent who we just say, hey, you're a 30-year-old professional. Or something like that, and maybe that would really swing things.

But you could use GPT-4, right, for one of the agents? Right.

So that could be interesting. And my, my current thought— so we'll have to run it. So that I'm not— I don't really have a clear intuition on what could happen if we do that. But I have some intuition on what— why that might be interesting, for sure. I mean, this will sort of Let us answer, or at least from more social science perspective, these gets at questions like, and the questions that I'm intrigued by is, how robust is our society or our social system to, let's say, perturbances? Or is our democracy safe from dictatorship? How robust is our democracy? What are the failure cases if we introduce this kind of agent Will they actually gain power to break our peace? And what are sort of the interventions that at a societal policy level we can put into place to prevent those? I think those are the questions that might be on the line as we start to sort of delve into these questions, and I'm fascinated by those.

Super interesting. Fantastic. Thanks so much. Ryan, did you have any last questions? I think we should probably wrap up. We started a little bit late, but we should probably wrap up. We could go on for hours, I'm sure.

Yeah, I guess I'll just finish up with a question of my own. So in the paper you mentioned some past work in early cognitive architectures, work such as like SOAR and Icarus by Newell and co. But these have sort of evolved into the natural neural networks that we know today. I feel like it's kind of ironic that the memory mechanisms that have been replaced by neural networks are now being used again in order to improve performance in systems such as yours. At this stage in time, do you feel that these are not competing technologies, but technologies that can sort of coexist at different levels and dimensions?

I do. That's actually a really fun comment. As we were sort of working on this, This to some extent also has to do with research taste. I personally like research that takes big and ambitious ideas. This is something that I sometimes observe in many, in sort of evolving communities like mine, like in machine learning and natural language processing in particular, and human-computer interaction, where in the earliest days we sort of had more freedom to dream big, right, and see the bar really high. set the bar really high, right? So what the folks who were developing cognitive architectures were trying to do is build fully general computational systems. Um, and what we find over time is, of course, we don't often have the tools to do this. If you go all the way back to the first initial, like, AI conference at Dartmouth and so forth, um, you know, we thought we would solve artificial intelligence in 3 months, but of course it's been many decades and we still I mean, I think we're getting slowly there, but it's not there. But they sort of have the audacity to dream big and have really big ambition that might be too early for its time. And one of the things that I like to do in my research is to revisit some of those assumptions, because oftentimes the ambitions are right, or there's something very intriguing about the ambition that we had for many decades but remained unsolved. I think what we're seeing is in part returning back to our early days of AI, but with a fresh insight basically to answer some of our initial ambitions now that we have the techniques to actually pursue them. In this particular case, I see it less as those 2 ideas competing or one replacing another, but neural nets have now developed strongly enough that they can, they have the power to potentially serve for the purpose of building more general human-like agents. So, we can actually combine this with a lot of insights from the early days, or at least a philosophy from the early days, to both basically to get at the ambition that we had with much more rigor and technical capacity. So, that's what I like to do in my research, and certainly generative agent is an example of that.

All right, thank you so much, Joon Sung. Martin, I guess we wrap it up here, and thanks everyone for coming.

I think, I think we have to, unfortunately. But thanks, thanks again. Thanks again to Joon. That was a really wonderful talk, and thank you, Ryan, so much for a really interesting discussion as well. And thanks to the audience. Please, please come again next time. This talk is recorded like all of our talks, so please subscribe to the YouTube channel. It's Machine Learning London. on YouTube, and you're going to get to hear about all of the, all of the talks that we upload as we upload them. Thanks again, everyone, and see you in the next one. Bye-bye.