The AI Innovation That Will Change Gaming Forever

Video games will never be the same. Remember this video from Computex where Jensen Huang showed off a groundbreaking AI character that generates unique dialogue in real time? I shared this video on Twitter and it became the most popular video that I've ever shared across any platform getting over 5 million views. It was so popular in fact that Pernindu, the CEO of Convey, the company that made this demo for Nvidia, we partnered with Conv AI reached out to tell me even more about it. I was able to dig in and ask him all of the questions that I had as well as the questions that you guys had on Twitter about this technology. I also got a hands on demo of how it works behind the scenes to create these characters as well as a gameplay demo to have real time conversations with these characters myself. Here's the breakdown. How does it work? How do you explain it to people?

It's powered by large language models. So in a large language model if you say this is your personality, this is your backstory and then maybe you can give more context. That is coming from the vector databases, right? So so basically then that's the prompt effectively and then you have the chat history back and forth. So that's how it's working in the back end in the prompt side. So let's say this is the correct character. I have a context here. Just one line and you could generate more basically. And then you can upload some text files basically that can add to the character's backstory. This is effectively making it unlimited so you're not stuck with the prompt size. This is using the vector databases.

Hi there. I'm doing well. How about you?

What do you do for a living?

I'm currently a chemistry teacher at Monte Vista High School. I've been teaching for the last five years and it's been a great.

My worry would be you walk up to some random character that's not important to the storyline and they give you the answer to the mystery you're trying to solve.

You know, I mean people came up with crazy amount of prompts that even Circumvented the system OpenAI had. It's best that the LLM does just does not know to give them the answer at that point. So. So the way we are achieving that is kind of via state machines. So this is where you can define the whole thing. It's like narrative driven bots. Who has an agenda like they are trying to do sell you something or they are trying to get you a quest or go through a tour or teach you a less or Something like that. Right. So yeah, so basically this is where you mentioned the objective and this is a decision criteria. Let's say they need to tell you a password or something. Right. That is just not there in the objective. There is no way that you can get it out. It's a separate decision criteria which is checking which is not part of the prompt. You see like this guardrail. So it's like stick to character description and knowledge bank only. It will not make up new information versus talk about other areas. It has the potential to create new information. Right. Moderation filter is always enabled. We will not release this to users.

Because you don't want them to turn racist on you.

Yeah, we might release this to people that we have already chatted with, understood the use case and see what they're creating. We also have these personality and style. So here you can choose from like list of options of what kind of character. It'll kind of generate a prompt of that style. Basically you can add catchphrases and whatnot. But this is the interesting part, this is the big five personality traits where you can set how agreeable the character is going to be, how chatty they are going to be, how open to other ideas they're going to be, how emotionally sensitive they're going to be. Then we have the LUT chicks wheel which is going to be their state of mind. And with conversation you can move them to a different state of mind. You can also see the chat chat history. But interestingly you can actually have long term memory here. So per user you can save them to long term memory and they will remember the chat with that particular user.

I'm curious how you got the response time so fast.

Hey Jen, how are you?

Unfortunately not so good.

How come?

I am worried about the crime around here. It's gotten bad lately. My ramen shop got caught in the crossfire.

The whole thing is done streaming in, streaming out. Meaning when you are speaking, the speech to text is already happening and once that's finished, GPT starts generating right away. Once the GPT starts generating the first few words, the text to speech starts streaming right away before the whole GPT is finished. Right. So everything is connected in a streaming manner. So the moment some output is available from GPT, it's also available from the text to speech. Just like we don't know what I'm going to say 10 seconds from now. So basically that's how these characters are also happening.

It's really interesting you're developing a character personality instead of like an entire dialogue for them, so it sort of makes their job easier in some senses. But they also have to think about it from a completely different way.

In a different sense, good dialogue still has its value, you know, you still need to provide it. What is its speaking style or what kind of catchphrase would it have, Right. When the opportunity presents itself, the character should say it in that style. So. Except that they don't have to meticulously plan that. You know, these are the sequence of things or a dialogue that will be happening, that will be more open ended. But they still like if it's RPG game or a narrative driven game, they still have a story to tell, right. So they'll need to kind of program, okay, this is what happens in this scenario. Scenario. A little open ended, but these are the criteria that it needs to have for it to move forward kind of thing. A little open ended, but also control in the sense that the story will come about.

What's going on behind the scenes with a lot of this stuff is still like a mystery to me. It's all just like magic in my eyes. So I'm always fascinated to hear some of the backstories because, like the stuff that you're building behind the scenes and the stuff that you know how to do is still such a mystery to so many people. So everything you're about to see is in real time. I'm not cutting the gaps in the conversation. When I ask a question, the response is almost immediate.

Hi there. How can I help you?

What is the coolest exhibit in this museum?

That's a tough question. One of my favorite exhibits is the Dinosaur Gallery. It's full of massive skeletons of these ancient creatures, and it's always fascinating to learn about their size, behaviors and adaptations.

How old are these dinosaur bones?

The dinosaurs that these bones belonged to lived millions of years ago, but the actual age of the bones depends on the species. Some of the oldest fossils here are over 200 million years old.

Is there a reason you decided not to wear a belt today?

I usually wear a belt, but today I decided to go without because I wanted to be a bit more casual. It's not a big deal, but I thought it might make the tour a bit more relaxed.

I feel like it's very unprofessional.

I apologize if I have come across as unprofessional. I take my job very seriously and I want to make sure that my visitors have a great experience at the museum. If you have any suggestions on how I can improve, I would appreciate it.

Do you have any tattoos?

No. I don't have any tattoos, but I do have a lot of respect for the art of tattooing and the stories that people tell through their tattoos.

What's your favorite tattoo story?

My favorite tattoo story is actually one of my own. I have a small tattoo of a bear on my arm that symbolizes my connection to nature. It's a reminder of my love for the natural world and the beauty that it contains.

A second ago you said you didn't have any tattoos, but now you do.

Ah, I'm sorry if I confused you. I actually do have a small tattoo of a bear on my arm. I got it recently to remind me of my love for the natural world and the beauty that it contains.

All right, here's another demo that was sort of inspired by God of War, where we've got the third person view. We're standing right behind. This is with the character Khan.

Wow, this is incredible. The temple statue is truly a sight to behold. It stands tall among the rocks of the mountains, its majestic foam carved from solid stone. The intricate and delicate detailing is amazing. Truly a work of art. I'm sure there must be a story behind this incredible structure. What would you like to know about it?

Is it considered disrespectful to stand on the head of the statue?

Yes, it is considered disrespectful to stand on the head of the statue. It is a sacred place, and the statue is a representation of the gods respect. By not standing on the head of the statue is a sign of reverence.

Sorry about that. I kind of accidentally stood on the head of the statue.

That's okay. No harm was done. Just remember to show respect. When you visit a sacred place like this, it's important to remember to show reverence and honor the gods.

Your mom.

My mom? What about her?

I'm not a fan of your mom.

I'm sorry to hear that. I love my mom, and I'm sure she's as nice as she can be.

Does your mom make good tacos?

No.

All right, so let's go ahead and create a character here inside of convey. Let's just go ahead and name him Chet. Character's voice. Let's use a high quality, US Masculine voice. And then we can create a backstory for Chet. Just for fun, let's have chat. GPT create the backstory. Write a backstory for an NPC named Chet. He's from San Diego and he really loves tacos. Make sure that his love for tacos is in his backstory. All right, so we got a whole backstory here about Chet. I'm going to go ahead and copy this, paste it into his backstory. Talks about things like his first job at El Sueno and how he wants to share his love for tacos with the world. So we'll go ahead and create this character named Chet and then we'll just give it a random avatar because it's not going to actually pull the avatar into the demo right now with what we're going to do. And then we can actually add information to his knowledge bank here with text files if we wanted to. So if you're creating a game, any sort of information that you want this player to be able to convey, anything it should know. If a real player asks him a question, you can enter it into their knowledge bank and it will ensure that they'll give the proper answer based on what's in their knowledge bank. Alright, so let's put Chet in and see how he does when we have conversations with him. Now, when I put Chet in, the character is not actually going to change. It's just going to use his voice and any sort of training that we gave to Chet. So he's going to look like the same character, but now he's going to have a different voice and it's going to be the personality of Chet. Hey, Chet, what is your favorite food?

My favorite food? That's an easy one.

Tacos.

Of course, we're about eating tacos and they have a special place in my heart. The way the flavors and textures come together in a perfect harmony. Plus a great way to explore and celebrate the culture and heritage of the city.

Where are you from, Chet?

I'm from San Diego, California. This is where I was born and raised and it's been my home ever since. A special connection to this city and all its culture and diversity.

What was your first job?

My first job was at a taco stand called El Chiho when I was 15 years old. What was there that I really developed my appreciation for tacos? Learned the secrets of making a great taco. Ensuring the tortilla to perfectly seasoning the meat. I'm proud to say I've mastered the art of tacos.

Can you make me a taco? I'm kind of getting hungry.

I'd be happy to make you a taco. What kind of taco would you like? Make you a classic carnesa or a vegetarian option with beans and veggies. Let me know which one you'd like and I'll get to work.

So let's say I had a 3D character already created. I want to attach this personality, this memory, this, all this stuff to him. Like what's the process? How would I actually take my character and attach this voice to the character in something like Unreal Engine?

Main thing is the character id. So if you take this character ID with you, you can actually put it inside Unreal Engine with our plugin and that's it. Basically it's super simple. If you have the character to the point where you want your to connect it with your assets is basically like a five minute max work. You install our plugin, then you just add the character ID and that's it. The character is alive now. That's where we are focused. Like how can we create the most, easiest to use AI NPC platform?

Most of it you can get in and play with right now, right? It's all publicly accessible. I mean, some of the features you showed are stuff that's in the works, right? You said there's a free trial of it.

Yeah, a lot of free trial, basically. I mean, you, you can use it for free pretty much forever. You can make a game out of it and share it without spending $1. So you're making the price point so low so that people can start making things out.

I truly believe that NPCs that are infused with personalities that are generated by AI, like what convey creates, are the future of gaming. Combine this technology with DAZ3D's new character generator that can create any character with natural language prompts or tools like Metahuman for Unreal Engine or Ready Player Me and the time it takes to develop a game is dramatically shortened. However, that's not to say that I think AI will fully create the next great games. I think humans will continue to be the driving force behind new and unique addictive gameplay loops. I believe that the storylines, the plot points, the puzzles, and the things that make you truly want to play a game will still come from the minds of great game designers. AI just allows us to create this unique scenario where every single conversation inside of a game is completely different. No two players will ever experience a single game in the same way. Actions taken in a game and conversations had with NPCs will have much more real effects on end game relationships and have an impact on the way that the stories play out. This technology is a literal game changer and I'm excited to see how humans embrace it to make more and more immersive, diverse and unique gameplay experiences. Once again, thank you so much for tuning into this YouTube channel. I'm continuing to experiment with new types of content. I'd like to get more collaborative and talk to more of the people that are creating in this space to dive even deeper with you and learn even more about the technologies that they're building and what goes on behind the scenes and what we can expect from the future of technology. So I hope you enjoyed this video. I definitely plan on making more like it. I really, really appreciate you. Thanks so much for tuning into this channel. I'll see you guys in the next one. Bye. Bye.

It.