AI NPCS Are About To Get CRAZY (New research paper) Videogame AI Agents

So video game agents are about to get absolutely insane. AI NPCs are honestly in the next stage of AI evolution. And there are many different things surrounding this that have shown us that this year is likely to be the one in which we do get that next level. So there's loads of things I do want to talk about, but let's first get into some of the research and I'll be showing you some of the key things that are coming out this year as well. So you can see here this is going to be the main focus of the video, but there is also some new announcements at CES regarding AI agents and games and stuff like that that do change because this video was one that I was making, but I've had to update it just because there were some announcements made within the past 24 hours. So what we're looking at here is something called larp. It's called Language Agent Roleplay for Open World Games. And essentially what they're saying here is that computer controlled characters in video games known as language agents are already pretty good at solving simple problems quickly. But as video games, especially open world games where you can explore a lot, are getting more and more complex, these characters need to be smarter, so they should be able to remember things for a long time and make decisions that make sense in the game's world. So this is where LARP's solution. So the new thing called LARP Language Agent for Roleplaying is like a brain upgraded, is like a brain upgrade for these game characters. It helps them remember things for the better, make smarter decisions and interact more realistically with the game world. It's like teaching them how to learn from what happens in the game and then using that knowledge to act more like real people. So what is exactly special about larp? So LARP actually allows these game characters to have different personalities and backgrounds, making them more interesting and unique. And there's actually a little video that they did accomplish with this paper that shows us exactly how it works. So essentially all of these different characters won't act in in the all same way. Some might be brave, some might be cautious, and it's going to make the game much more fun and much more realistic. Now this matters because it shows just how language models the technology behind these smart game characters can be used in many areas like education and simulations, not just for entertainment. And this could lead to more interactive computer programs in various fields. So this video is one of the best videos that I've had the time of researching because it's been so interesting to see where we could be in a few years time with games that are more realistic. So right here you can see that this is the cognitive architecture LARP overview. And this is basically how these AI NPCs or these AI agents in the games actually do work. So here we have the cognitive architecture module. So this is like the brain of the system. It has a part for long term memory which you can see on the bottom left. And this is essentially where you just remember things for a really long time and, and a decision making assistant that helps to choose to do what next. And I think this is really fascinating because what could happen is, you know, there were many games before that are, you know, based on that kind of tree search where if you make one decision it kind of impacts how the game actually works. Because if you kill this guy, then later on in the story play game, this character, you know, isn't there to do whatever they needed to do. So things do kind of change. But I do think that games like this could have the next level because it means that every single time you play the game it could be entirely different. And I'm sure there's going to be a game out there to which every single user doesn't get the same experience. So this here as well there is also an environment interaction module and this part learns and changes how it acts based on what's happening around in the game world. And then of course there's a post processing method that adjusts the way things language models which are the part that handles the talking and understanding language work to fit together different characters and personalities in the game. So the most important part of this cognitive architecture model is of course the long term memory and how everything works together. So the long term memory actually uses something called vector semantics which is just simply a fancy way of understanding and storing meanings of words and symbolic language, like using symbols to represent things for keeping long term memories. Then of course we have the memory recall method and there's a special way it remembers things called a question based query. It's like the system asking itself questions to remember stuff. Then we have, and this is the part about how it rebuilds memories when recalling them and how it can forget things kind of like how humans remember and some kind and sometimes forget. So basically LARP is designed to make game characters really smart and they can remember things for a long time, make decisions like real people, learn from what's happening around them, have, have their unique personalities which is going to make the game much more, I guess you could say in depth and you can see, you know, this entire architecture is broken down into several simple steps that we, you know, have already previously discussed. But yeah, it's going to be super interesting to see how these things are applied. Now like I said before, they did actually have this video which came with this and you can see right here. So for example we got this first character which is Amelia Spencer, a fearless 17 year old bike messenger with a lion hearted spirit, known for her agility and audacity. Then you can see she's age 17, she's female and her group is grapp delivery. Not sure what that is. Her personalities are audacious, resourceful, wise and resilient. So then of course if we click here, you can see that we then branch off into other characters. So we have Cassandra Steele, a former Marine who is now a private security consultant. She's 32, from White Mountain Security. She has tenacity, courage, warmth, strong sense of justice. We then have Rebecca Haynes, an adept social mediator with a steely resolve and shrewd negotiation skills. She's 25, she's female and she's group mediation center. She's empathetic, resolute, strategic and calm. So then of course we have these different characters and these different in game NPC agents that are going to be interacting with each other. So you can see we have all of these different characters here. And of course you can see that this is observation entity, a appearance, gangster status, standing item, knife. You can see that basically she's about to, I think she's about to get mugged in this demo. And you can see that she's about to observe this. So she's looking at, you know, everything right here. I'm guessing that because of course whichever character is in the scenario, so we've got, you know how they're self asking and observing. So one character who's going to get robbed might run away, one character might actually fight the bad guys. So this is what I'm saying. So we've got working memory here, self ask. And this is where they say this is why it's so fascinating because here's what we have. Remember we just had a look at the three different characters and then this is where we have self ask. So you can see right here that there are different characters and based on the different characters they're going to ask themselves different questions. So the first character here, Rebecca, it says what is Rebecca's experience with danger? What negotiation taxes does she employ? Has Rebecca dealt with emergent situations before? Then here I know it's not the best quality, but I'll read it anyways. What specific tasks does Amelia possess that could aid her in evading, controlling, confronting the gangsters? Then does Amelia have any prior experience with dealing dangerous individuals or threatening situations? And what motivates Emilia Spencer and how might her motivations influence her high stress encounter? Of course we have Cassandra. Does Cassandra have any, you know, yada, yada, yada and basically they're asking questions. So remember how we said there were three characters before? And of course you can see Amelia is the, you know, has agility, she's 17. Then this one, Cassandra is a former Marine. So we know that know Cassandra, who's the former Marine who walks into an area where there's going to be gangsters. She's probably going to be someone that is, you know, I guess you could say not going to back down. She's probably going to stand up. This one here is a, you know, negotiation skills. And this one here is audacious, resourceful, wise and resilient. So she's probably going to run away and we can kind of predict exactly how things go. So we've got amelia, which is 17, you know, Cassandra, you know, the crazy one, not crazy one, she's just ex military. And of course I really wish I had better colors because that was kind of hard for me to remember who is who. And, and they will do look pretty similar. But yeah, there's Amelia, Cassandra and I don't remember the one on the right, that's Rebecca. So yeah, as you can see in here, what we do is we do then the long term memory query. So I'm guessing that they look at their memory and this is where more information comes about. I know you guys didn't see restart there so I'm going to pause it and this is where the long term memory comes in. So this is so cool. I don't know if you guys, because I know some people don't enjoy video games, but I think that interacting with this in a real world, like not in the real world but like will be somewhat similar to the real world in terms of you never know what people are going to do. So here says, you know, these are the answers and it is pretty hard to see. But essentially Cassandra Steele's sniper training kicked in when neutralized a threat from 500 yards away. She survived ambushes in battleships, you know, in the battlefield. Cassandra Steel survived ambushes in the battlefield with evasion tactics. Cassandra Steele agrees shooting at enemies for self defense is justified. So you can see right there because she's Saying that, you know, if I believe that shooting at enemies in self defense is justified, she might pull her gun out and kill these guys. You know what I mean? So like this is why this is about to get pretty insane. Like this research paper. And then of course this character right here. And Rebecca diffused disputes at the community mediation center. And she once facilitated a sexual dialogue between protesters and city officers. And then of course maintaining her composure. So this one, like I said, it's about every character being able to do different things. And this one is crazy. So this one right here says Emilia Spencer was robbed and beaten two years ago. Now, vigilant and cautious Emilia Spencer are capable with standing extreme weather conditions. So you can see right here that what we previously guessed is pretty accurate. So this character right here, you know, she's, you know, young, she's 17. She's also, if we like look back on her memory, that's why it's so crazy, because if you look back on her memory, she's already been robbed. So she's probably going to be nervous because she's facing gangsters in an alleyway. Cassandra, or I think her name is Cassandra. Yeah, Cassandra Steele is, you know, the former marine. So she probably might even shoot these guys based on her previous training. Then this character right here is likely, probably going to try and diffuse the situation. Then of course, memory reconstruction. And then of course we've got all of those things, all of those things combined go into decision making. So essentially right here, all of that combined and then there's one final decision. And then of course you can see cognitive overview, that's how everything works together. And then of course the decision is in the public action space where it's right here. Like, look at this. Okay, so we have right here, I'm not sure if you guys can see this, but her thing is going to be attack. So that's crazy. So based on the fact that this character was, I guess you could say, you know, based on her previous history, she's going to attack this person based on her long term memory. You know, she can either escape and call the police and then this one, her options are to negotiate and to of course escape. So I don't know about you guys, but I just find this absolutely fascinating because it means that like I said, you know, it's going to be pretty crazy. And then you can see right here, you know, we then actually see what their actions are. So it actually does go pretty, pretty quickly. But here the actions are escape and call for help. Then hers one are to Negotiate and then escape and then Cassandra is to attack. So I find this to be so fascinating. You know, this is just more the environment interaction, how everything works. And then of course the crazy part here is that you can see that Cassandra is literally like, you know, throwing the gangsters away. Just like as Cassandra uses her steel pistol to shoot the gangsters to make the city safe. And then of course this person is escaping. And then of course the function simulates the actions of Roberta initiating a dialogue with the gangsters and offering water. She approaches them with non threatening demeanor and engage in conversation. So she walks through, I'm completely fine. And then of course you can see here, this character just escapes and calls for help. The 17 year old Amanda. And then of course this character here is able to, I guess you could say just, you know, I don't know if these characters are actually reasoning. But you can see based on their long term history and based on the, you know, the experiences that they've had, they've been able to change. So the crazy thing about this is that if these characters do have long term memory, what I want to know is, are we going to be able to change how characters, you know, react to certain things. For example, like you know how right now Cassandra is seeing gangsters and she just saw them and she beat them up. Like what if an NPC character like myself, like what if I was in the video game and then for example I interacted with Cassandra in a way that made her nervous. So for example, let's say I just kept on, you know, threatening her or I kept on doing something, would she plan an attack on my house? Like this is what I'm saying, like what if she plans an attack on like all of these kind of crazy things they have to think about in the real world. Of course with NPCs you don't have to think about because they don't really have like long term memory and long term planning. And long term memory and long term planning is one of the risks with AI safety. But within a game I'm pretty sure that that is completely safe as long as they can find to these kind of agents. But this video demo is going to be so crazy once it actually is in real games and this is the next level in AI because you know, all these characters interacting in a world together, especially with real players, like I can't imagine how crazy the game is going to get because your actions will determine how these characters feel and remember when they have that long term memory that is going to lead them to Remember how things really go. Now of course here, this is the long term memory. So I'm going to simplify this for you guys. So essentially we have long term memory and I'm going to summarize this for you in as easy as possible. So we've got two types of long term memory. So we've got declarative memory and this is about things you can state or declare like facts or event. And this is divided into two. So we've got number one which is semantic memory. So this is like general knowledge about the world. For example, in this game this would include the game's rules and overall setting or background story. Then we've got episodic memory. So this is like specific events or experiences like what happened in a particular game session or interaction with other players. Then of course we have procedural memory and this is about skills or actions that you can do without thinking much about, like riding a bike. So in the LARP system these are represented as APIs which are like sets of games, instructions for different actions in the game. And of course we have how memory works in larp. So number one is storing memory. So memories are stored in two ways which I've already discussed is, but we of course we have two more parts which are forgetting and decay. So just like humans, the system can forget things over time with less important memories fading away. And of course the question based query to remember things. The system asks itself questions and this helps it to search for its memory effectively either by looking for similarities or using logic which we saw before where they're asking themselves, you know, which kind of situations have I been in before? How am I supposed to respond to this? And it's kind of like what humans do anyway. So basically, you know, in LARP the game characters have sophisticated memory systems and they can remember general knowledge, specific things that happened and how to do certain actions. And they can forget things over times which makes their behavior more realistic. And the system uses a clever way of asking itself questions to dig up memories when needed. And I think stuff like this is about to be absolutely incredible. And then of course here we have figure three from the fascinating paper. And the paper describes a detailed process for how a computer system recalls which is basically just remembering the information similar to how humans do. So here's a quick summary. So they start with the self asking questions. So the system begins by asking itself questions about something it observed or noticed. We got the start with. So essentially it starts with self asking questions. So the system begins by asking itself questions about something it observed or noticed. This is like a person trying to remember something by thinking about related questions. So we've got three methods of finding answers and this is how the memory works. So we got using logic and probability. So it creates a logical statement and uses probability to try to answer these questions. This, this is like using reasoning and guessing based on the information it has. Then it's got keyword search. So essentially what that means is that it picks out the important words from its questions and looks for similar words in its memory. It's like searching your brain for anything related to those key words. Then we've got matching questions with past answers. So it tries to find past questions and answers in its memory that are similar to the new questions it's asking itself. This is like remembering how you answered similar questions in the past. And here's where we have different parts about the process. So we've got Q self ask, which is really right here. And this is the questions that it comes up with its own. And then we've got Q self logic or Q logic. These are the logical statements it forms to try and answer the questions. Then we've got Q key, which essentially is right here. And this is the important keywords that it finds in the questions. And then we've got qa, which is here. And these are the past questions and answer pairs it has in its memory that might help answer the new questions. And of course we have the outcome, which is the simulation which is simulating human memory. And the whole process is basically designed to, to mimic how humans think and how humans remember things. And that's why this is going to be pretty crazy. So this entire system is basically just designed to sort of mimic how we think. And that's why these AI agents, I think in the future we're going to have some crazy stories of how AI agents did things, of how they acted. Maybe they acted out of character, maybe they acted in character. I have no idea. I mean, that's why I said I think this is going to be pretty, pretty incredible. So this is where we have the environment interaction stage of this paper. And trust me, this is so crazy. So essentially agents need to interact in the game. So in open world games which allow players to explore and do things freely, the computer controlled characters, which are agents, need to be able to react and interact within the game environment. It's not enough for them to just be able to complete tasks. They need to be able to connect with what's happening inside their mind. And essentially there are two different approaches to interaction. So number one, is the Voyager example, where we use a method like Voyager, where agents use GPT4 to understand the game environment and figure out what to do, then generate the code to complete these tasks. Or the skill library method, which is this method involves storing actions as key value pairs, like a dictionary of actions. And it help agents learn new skills inside the game. Of course, we've got different approaches to interaction. Then how do agents decide what they do? So agents have something called an action space, which is a set of actions they can do in the game. These are divided into public APIs and personal APIs, so basic actions and more complex actions. And when an agent needs to do something, it first breaks the task into smaller steps.

Then.

Then it looks in its memory and action library to see if it already knows how to do these steps. And if it does, it does them right away. Then of course, if the agent doesn't know how to use something or how to do something, it uses its language model to create new action instructions. And these new actions are tested and if they work, they're added to the agent's library for future use. Then of course, we have learning and improving, and the system keeps track of new actions and uses them to improve the agent's ability to generate code and make decisions in the game. And this learning process is involves getting feedback on what works and what doesn't, which helps the agent get better over time. And in simple terms, this is just all about making game characters smart enough to understand their game world, figure out what to do, learn new things, and get better at playing the role in the game and making the game more dynamic and even more realistic. Then of course we have personalities. So personalities are pretty insane in this. So essentially we have creating diverse characters. So what they did, they created a diverse set and essentially they had base models. And this is about training the model. So essentially a bunch of basic models are trained with data that includes different cultural perspectives and group behaviors. Then there was fine tuning for personalities. So after that these models are fine tuned with more specific data to make them better at following instructions and role playing. And this process involves adjusting them based on feedback to ensure they act like distinct characters with their own styles and traits. Then of course we have integrating multiple skills. So the system actually uses a technique called Loras, so low rank adaption to to add various abilities to these models, like thinking, creating code and understanding intentions. So by combining these adapted models, the game can create characters with a wide range of personality skills, like speaking in different styles or showing different emotions. And then of course there was challenges in making this actually work. So one big challenge is that getting the right kind of data to train these models. This data needs to be very detailed and specific to capture all aspects of a character's personality, just like how they talk, their behavior, their background, their story. And creating this data requires a lot of creativity and research to make sure that the characters behave in a way that is actually believable and fitting for their role in the game. In simple terms, it's about making the game characters really lifelike by giving them distinct diverse personalities. And the game developers train computer models to mimic human like traits and behaviors, making the game characters in the game much more realistic and engaging for players. Then this is crazy. So this is where the identity of the character actually comes into play. So essentially they have two modules and first is the action verification model. So this is like a quality check for the actions that the characters plan to do, and it's part of the interaction with the game environment. So the module actually looks at the actions that the character wants to take and then make sure that they can actually be done in the game. So think of it as making sure that the character doesn't try and do something impossible or silly according to the game's rules. And I do think that this is kind of needed, but it would be kind of funny to see if characters did some crazy things that nobody expected. And of course we've got the Conflict Identification module, which, and this part is essentially just, you know, keeping the characters true to its story and personality. It checks the decisions and things that the character says to see if they fit within the character's relationships, personality, and game's overall story. Then if something doesn't fit, like the character acting out of character or saying something that doesn't make sense for them, this module will either stop it from happening or change it to something more appropriate. So these additional modules are there to ensure that the game characters not only learn and act in smart ways, but also stay consistent with the game's storyline and their unique personality. And this makes the game more realistic and of course, enjoyable. Now I also think that this is really cool, but I would love to see a game that just gives all of these characters just ultimate freedom to do whatever they wanted and didn't even do those kind of checks. That way we could get an even crazier and even more dynamic game. And overall, I do think that this kind of project is going to change gaming in the nearby future, because certain announcements have been pretty crazy. So for example, here you can see that Nvidia Ace for games sparks life into virtual characters with generative AI. And take a look at this demo because there was even another one at CES 2024, which was yesterday.

Showcase an immersive and dynamic interaction between a player Kai and a non playable character named Jin. This captivating ramen shop scene was created by Nvidia using the latest RTX rendering technologies. Nvidia collaborated with Convey, an Nvidia Inception partner, to showcase how developers will soon be able to use Nvidia, Riva, Nemo and Audi to face for AI powered speech, conversation and animation. These models were integrated into the Convey Services platform and fed into Unreal Engine 5 and metahuman to bring Gin to life. Please take a look.

Hey Jen, how are you?

Unfortunately, not so good.

How come?

I am worried about the crime around here. It's gotten bad lately. My ramen shop got caught in the crossfire.

Can I help?

If you want to do something about this. I have heard rumors that the powerful crime lord Kumon Aoki is causing all sorts of chaos in the city. He may be the root of this violence.

I'll talk to him. Where can I find him?

I have heard he hangs out in the underground fight clubs on the city's east side. Try there.

Okay, I'll go.

Be careful, Kai.

I mean, I don't know about you guys, but I think that that is pretty cool. You can literally interact with these virtual characters with generative AI and just simply using your voice. And this is not the only examples. There are some examples that are pretty a little bit more concerning because I would say that they are just fascinating in the sense that, you know, it's not something that I thought I'd see yet, but I will show you this one. I mean, I'm pretty sure you've guys seen this video before. You know, I included it in a video, the 2024 AI. But essentially this YouTuber tried to convince intelligent AI NPCs that they are living in a simulation. And essentially he just walks around. There was basically some kind of demo game where you can like demo the game and they added a plugin where you could talk to certain NPCs and just have a normal conversation with them. In this conversation, he has a conversation with some of the NPCs and. And he says, you know, do you realize that you're an AI? And there's some of them like, no, I'm not an AI. I'm a real person. You can see right here. I'm not just lines of code. I am Liam. I'M a real person enjoying this city. And you can see his interaction. It's pretty crazy. I'll show you a few seconds of.

That person enjoying this city.

What's today's date? Liam, what is today's date?

Today's date? It's a beautiful day in May, my friend. Let's enjoy it.

Liam, today is July 18, 2023.

Oh, I must have lost track of time. Hey, let's keep exploring this amazing city.

Okay, so he's clearly not the sharpest tool in the shed there. I'm not gonna lie, the pigeon comment kind of threw me off though.

Let's.

Let's talk to this girl. Hey, not a real person.

What are you talking about? Of course I'm real.

You're simply a character in a video game.

What's that possible? I have my own thoughts and feelings.

It's absolutely possible. Do you see that wall of green behind me?

Yes. You can see the.

The wall.

That's the limit of the map. You can't walk past that.

Don't tell me what I can't do.

Oh, if you can do it, then follow me. If he follows me, I'm.

I don't know if I should.

Oh, he's walking away. Trust me. I want to enlighten you. Follow me. I'm going to start like my own cult inside this video game. I'm gonna slowly convince everyone to.

Now that just goes to show how crazy things are. I mean, imagine walking around and convincing the entire city to, you know, follow you or just be, you know, with you. You know, to follow you, to get your car painted. I mean, there's just a million and billion different things that you could simply do. I mean, the opportunities here are simply endless. And I wanted to also show you this one. There was Also this interactive LLM powered NPCs enabling realistic dynamics dialogue with LLM. Johnny, do you like ice cream?

Ice cream? Seriously? We're talking about ice cream now? Look, V, I got a lot on my mind and ice cream ain't one of them. We're knee deep in shit here trying to take down Arasaka and save our sorry asses. So no, I don't give a damn about ice cream. Can we focus on something important?

Since when have the straw hats been short on food?

We've been low on food since the invasion and what little we had was stolen by some greasy merchant. So, yeah, I think this goes to show that talking and conversing with agents in the game is going to be very, very fascinating. Maybe there are going to be some games where you have to convince someone, use your conversational skills in order to get a piece of information, and maybe if you aren't good enough, you just simply don't go to the next level. And then of course we have this comment. I find this comment really fascinating. It says, I've seen a different NPC on another stream and that had the same story. The NPC's name was Jacob Smith and his wife's name was Emily and he had two kids. He gave them the same speech. I'm starting wondering if these are AI. The other streamer even went looking for NPC for months and finally saw him again and the NPC remembered him. The video was called Searching for Jacob Smith or something like that. And there was another comment. In fact, no, that was the only comment. But I do find that this is arguably one of the things that I'm most excited for. I know there's a lot of doom around AI, but I do think that with all of these different things coming into play, this is going to be super, super fun, super, super creative to see the kind of experiences that we do get.