AI Dev 25 | Justin Uberti: Introduction to the OpenAI Realtime API

All right, who's ready to run some code? I want a little more enthusiasm. We're going to have APIs, we're going to have live demos, we're going to have running code. All right, you feel it? All right, all right, that's what I want to hear. Okay, so before I jump in, just show of hands, who knows how to clone, start a next JS app? A few. Okay. Security, everybody else. No, we'll have some other things like in a visual sort of the OpenAI real time playground that even if you're not next developer, you can do a lot of the same sort of stuff too. But without further ado, let us go and get going here. So this is about the OpenAI Realtime API. And the Realtime API is basically a way of encapsulating how to build interactive voice agents. We're going to have a multi step exercise where we slowly build up a voice agent from just basic capabilities where we then add some prompting, we add tools, we add guardrails, and in the end we'll have some really cool examples of what you can do with all this. And I'll get into some of the interesting features of the API. So I've been at OpenAI about three months, which in OpenAI time is about like six years. And I lead the team that develops our real time infrastructure and this helps power OpenAI's multimodal offerings like the real time API 1-800-ChatGPT. Basically things where the models are like listening, seeing, speaking and prior to joining OpenAI, I co founded a startup that was building multimodal voice LLMs. And prior to that I worked for about 15 years at Google where I led the creation of the WebRTC technology that's used within the Realtime API as well as Google Meet Microsoft Teams and many other interactive applications. So one sort of personal note here. When I was asked to do this talk, I thought it would be a really interesting opportunity. Not just go and promote the real time API, but also kind of like, you know, engage with the deep learning AI community. Because three years ago I kind of got bit by the AI bug and decided that AI was going to be the future. And I was like, okay, I don't really know anything about deep learning. I've been at Google, but I've been working in a different department for a long time and I've been missing out on all this cool deep learning stuff. And so I was like, oh, where should I go? Well, deep learning AI. See Andrew, with all his lectures and the problem sets and Everything. And I spent pretty much the summer of 2022 working through all the things there on DeepLearning AI. And then when I actually finished that, I was like, oh, I think I should know enough to go and form a startup. So here I am now on stage, three years later. So. So anyway, thank you, Andrew, and I expect to see some of you up on stage in three. So here's our agenda for today. We'll talk a little bit about the API, what it can do, what even is real time. Talk a little bit about the application architecture that's used for these types of voice agents. Spend some time going through the core concepts of the API. This will be our live build session where we'll check out the repo and incrementally build and add capabilities and then we'll have some interesting, never done before, live demos at the end. So let me just give a demonstration of what it is we're actually talking about here. Here's the actual repo and I'll get to that in a second. But this is the Real Time, the Real Time console. You can see it up here. Platform, OpenAI. Com Playground, Realtime, or just search on Google or on OpenAI using real time Playground. And you can see a few things here. You've got some system instructions that describe the behavior of the agent, you've got the voice. Bunch of other things that I won't get to right now, but what we're gonna do is basically just like, you know, show here's how you actually interact with the Real Time API in this playground.

Hello, how can I assist you today?

Hey, Chat, I'm here at a on stage at a Deep Learning conference. Can you greet the audience for me?

Ladies and gentlemen, esteemed speakers and fellow enthusiasts in the world of deep learning, welcome. It's an honor to have you all here today as we dive into the fascinating world of artificial intelligence and and explore the cutting edge advancements in deep learning. Let's make this an engaging and insightful event for everyone. Thank you.

All right, Chat, thanks a lot for that. We'll talk some more later. Bye for now.

Looking forward to it. Have a great time at the conference.

All right, so you can basically just see that you have essentially a voice agent in a box here that you can describe its behavior just like you can prompt a normal LLM. We'll get a little more to that later. You can pick from a whole variety of voices here. Alloy is the default, but there's some other neat ones.

Hey, I'm Ash, always around. Hello. I'M Ballard.

Yes.

Curious and up for a bit of adventure. Fancy joining me?

This one is like the most manic. If you want that coral checking in.

Here to listen and keep things on an even keel.

Your mileage may vary. So basically you can get a lot of sort of reps in of just, okay, here's how this API works. Here's the way that it might feel if you're putting this in front of a customer, just by sort of filling out the prompt and playing with the actual voice. And I'll get into a little more detail of like, okay, how do you go from this to an actual voice agent? But first, let's talk a little bit about, you know, why even have a whole separate API, you know, real time API to begin with. Why don't we just do this with our existing OpenAI APIs? And so the main thing is that the real time APIs focused on low latency. Our goal is to be able to return a payload of voice in like under 500 milliseconds, ideally like, you know, closer to 300 milliseconds to have that really human, like voice responsiveness. And so like when you talk to a human. There's been studies across many cultures, many languages, and they find that humans typically, you know, the time between human turns in a human voice conversation is actually like, you know, around 200 milliseconds. It's very, very fast, you know, a very large range. But the expectation is that the turn taking and handoff is instant. And what's more, studies have shown that the longer the delay increases between you finishing saying something and the other person responding, the more you're inclined to think that the other person is hiding something. That literally the fact that you say, so what do you think of my performance? And they're like, It was good, it was nice. I mean, there's a natural sort of human behavior to think of, how do I deliver this message carefully. And humans are very aware of this. So bottom line, we want to make sure that when you generate a voice output, it's generated very, very quickly. So there's a very natural communication. There's no sort of like lags and people are wondering, why did that take so long? So anyway, real time API, low latency focus around these really human like interactions. Second, it's focused around voice. We think there's a lot of scenarios in the future where voice is really going to be critical. And ultimately voice is the original user interface. That's how people got things done for a very, very long time before the advent of print and now web and Internet and all that sort of stuff. But this is how we interact the way we choose to interact with human intelligences. And so it also makes sense that this is the way that we're gonna wanna interact with, like, artificial intelligences. I know some of you are out there building like robots and these sort of things. And like, we know, like, you're not gonna go to the turn the robot around and type on the keyboard on the back of the robot or something like that. No, you're just gonna talk to it the same way you would interact with like another person. And finally, there are a lot of other devices that are out there. Like I said, not computers, not mobile. Everything, just like everything got Internet, everything is gonna get intelligence. Smart appliances, smart toys, robots, and voice is just gonna be increasingly just the way we want to interact with these things. So anyway, this is why we have the realtime API to solve all these use cases. So Real Time API is not the first attempt to do basically a voice interface. Many existing voice systems out there are built using a model that we sort of call the component or cascaded model. And what you have essentially in these things is that you have a conjoined set of models. First you have a speech transcription service, or ASR, that takes in speech, turns it into text. Then that text is passed to an existing LLM like GPT4O or something like that. And, you know, the output of the LLM is then streamed out as text into the final stage, which is a text to speech component, which it takes in text and generates speech. Now, this process is a little bit clunky, and it's also very, very slow for a number of reasons. A, each one of these stages has to do with processing and then sort of feed it to the next thing. B, each one of these stages actually has a language model in it. The transcriber has a language model so it knows how to fit phonemes to the actual words. The text to speech thing has a language model to understand how to generate the appropriate emotion in the output speech based on looking at the words in the input text. So there's a lot of redundant processing going on here for this cascaded model. And I mean, there's even worse things that the text to speech actually has to do. On lookahead, you can't just stream the tokens into the text to speech. It has to look ahead to get enough text to know what kind of emotion to imbue in the output speech. So the bottom line, it's clunky it's slow. The Realtime API is different. The Realtime API is a speech to speech model based on the GPT4 Omni technology. And so speech is passed in, that speech is converted directly into the LLM's high dimensional space, not into text. And then the LLM works on the speech the same way it would work on like a text embedding that was passed in like a text LLM. The output then of the speech to speech model is not text, but rather a transcript. Plus these we call VQ codes that then are turned into output audio. And so like the ability to have this single four, this forward pass of the model, taking in speech, emitting speech allows it to be much, much faster because you're doing a single processing loop. And so you get the. It's still very fast. Like I said, we're trying to aim for around 400 milliseconds in response time. And it's still very, very smart. You can always make a model fast just by taking out layers and taking out parameters. But you know, this has like the pretty much the full complement of GPT4O's intelligence while being very, very quick. In addition to that, it's also very steerable because it's a deep model. It can be prompted the same way that existing LLMs can, which means you can tell it to, you know, have an accent, act excited, act sad. The voice can be very steerable and it can follow instructions really well, which is quite useful when you have voice agents. Also, it's multilingual by default. A lot of times when you have a separate speech recognizer, the speech recognizer needs to be told what language it's going to expect. GPT4 is very deep model, so it can process all sorts of languages without needing to be explicitly queued. And then finally, as I mentioned, the actual understanding of the voice is always kept in the high dimensional space of the model. It's never converted into text. And so this allows paralinguistic cues to be preserved. Paralinguistic cues are like the tone, the accent, the inflection, these sort of things. The model now can have access to all this data that it would not have if it was just smashed into text before it was fed to the LLM. Now, there always are trade offs in this case, because the model is fast, we've optimized for a low latency and GPT 4.0 does not have the ability to reason the way that say, oh, 1 or 03 does. But we'll talk about how we can have our Cake and eat it too, in a later part of this talk. So there are a few different ways that, that your application can interact with the realtime API, the architecture that you might want to choose. The initial thing that we released and then the first release of the Realtime API was a WebSocket interface. And this was basically where the application was able to be in complete control of what was happening with the session. That your server would establish a WebSocket connection to the OpenAI API and and would sort of pump in audio as it received it from the client. That would then lead to events coming back from the server saying, hey, you know, okay, I'm generating a response now. Here's like, you know, the stuff that's happening in the response, here's the audio coming back, and then the application will be responsible for taking that audio out of the WebSocket stream and sending it to the client. Tool calls, all those sort of things handled by the server in this architecture as well. The issue really, though, is that it puts a lot of work onto the developer. Anyone who's tried the realtime API previously probably was aware of this, that any of this buffering of audio, converting to the right sample rate, converting into some form of applying echo cancellation, that's all the responsibility of the developer, and we want to make that easier. So we also have some interactions with various partners where you can rely on the API partners to do the media exchange. And then your application just handles the events that come from the API. And so this reduces the amount of work that you have to do in your application. But then you have another dependency to manage. And so our newest approach that we just released in December, and this is the one I'm gonna be diving into more today, is where you use WebRTC to connect directly to the OpenAI API. And so there's an initial step where your server helps broker a secure connection to the API, but then it gets out of the way and the client can exchange the media directly. And then all the sort of WebRTC goodies that are used in like meet and teams and everything can be employed automatically for echo cancellation, noise reduction, that sort of thing, while interacting directly with the API server. Now, there's still one drawback of this approach is that event handling is all sort of done on the client here because the events are streamed from the API server to your client. And so that's okay for some tool use, but for some scenarios like rag, then it's an extra hop required to kind of go and relay that to Your application server. And so the direction we're heading towards is where we have this direct connection for media, for audio, moving from the server to the our API server to, to the client directly. But then there's also a channel that can be used for tool use where the API server can connect directly to your application server if you need to go run some tools. So anyway, that's the overall perspective. The nicest thing of just having the WebRTC connection is that almost all the media handling is done for you, which means that you can write Voice Agent in only a few lines of code, which is what we're about to do. All right, so we'll talk a little bit about how the API actually works. One thing I want to show, let's see if I can do this here. Yes, So there's a bunch of messages that are exchanged from the API. Go ahead and start this back up again.

Hello, how can I assist you today?

So there are these messages that are being exchanged back and forth from the client and server when you're interacting with the real time API. And there's a bi directional channel over which these things are exchanged. When you're using WebSockets, you have a bi directional WebSocket. When you're using WebRTC, you have a bi directional data channel. And what ends up happening is that the audio is sent. There's a process that occurs in the server that says, oh, the user is done speaking. I now need to generate a response. And the response is then created using this response create and response created indication that tells you, hey, a response is now streaming back to the client. There's a bunch of updates saying, okay, various deltas of like, okay, here's a next token that's been decoded, these sorts of things. And then finally at the end you get a response done indicating that the thing is now finished. And so generally that's the flow that will be occurring as you interact with your voice agent. The user will speak. You'll be informed that the user has finished speaking. A response is being generated, the response streams back, you get notified of the various parts of the response coming back. And then finally it tells you the response is done. And now the turn is back to the user to speak again. There'll be more things that will be brought in when we get to things like tool calling. But this is the basic setup right now. Get back to the presentation. So these are just a double click on some of these particular, you know, updates. We have, you know, the session update which can give it new Instructions saying, okay, you, previously you were acting like an assistant, now I want you to act like a pirate. You get back now a response from the service saying the session's been updated. The client can also trigger a response to come back. You know, if maybe a tool call finishes or maybe just some time has gone by and now the application thinks that it's time for the server to respond, it can go trigger response even if there's not any voice detected from the user. Each event in the realtime API is about a particular object and there's a handful of objects that are used. The first one, the sort of meta object is the session, and that sort of controls all the parameters of what's being done and what's the interaction that's occurring right now. That will be the voice, that will be like the instructions, that'll be the set of tools which model we want to use. These sorts of things are all captured in the session. Then there's the input audio buffer, which is tracking the user's speech as it comes in and detecting when has the user started speaking, when has the user stopped speaking? When the user stops speaking, the automatic behavior is to trigger a response. And then there's these sets of responses. As I mentioned before, when it comes the agent's time to speak, these responses start being emitted. We give back the audio, we give back the transcript, so you can see what's happening in the response. And so these are the things that you'll get events on as you interact with the API. So let's see, we talked about the playground, we talked about the events. This is just an example of the specifics of some of the events that'll be happening here. And if you looked at the new responses API that we just launched, you'll see that there's a lot of very, very common concepts here that you trigger a response to be generated, and then there's these very structured events that come back of the response. The conversation item tool calls these sorts of things. And overall, hopefully you'll find that the response API Responsys API and the Real Time API, you know, a lot of the same knowledge can be exchanged, can be used to think about when you're working with both of these things. The Responsys API is unidirectional. It streams like server to client, whereas the Real Time API streams bidirectionally. You can stream events from client to server and from server to client. So there's some other resources that are great for understanding all the events here. There's this GitHub repo, the OpenAI Real Time Console, and we'll be actually working on a project that's very, very similar to the Real Time console. That's what we're gonna go do next. And yes, this brings us to the part where we're actually gonna start getting our laptops out and get ready to do some coding. So let's go ahead and get started. If you go to this URL, it's going to take you to a GitHub project. You can clone it to your machine. It's a next app. So you just do NPM I to get installed, then NPM run dev to get up and running and also just spend a moment now just sort of walking through the repo because there's a whole bunch of different branches that sort of correspond to the different steps that we're going to do in this build. There's gonna be three steps in the build. Each step's gonna have like a start and a finish. And if you choose to skip a step or whatever, you can just cut ahead to the next step just by going to the right branch. So with that, let me just bring up the repo here. This is where the tinyurl RTAPI or slash RTAPI goes to. And you get the most things right. First you need to get an API key. Hopefully you all have a OpenAI API key, but if not, you can create, get your M set up, do your NPM install, get your NPM run dev and if you're successful then you should end up with something that looks pretty similar to this once you actually can load the website after you get the application running. So why don't we take like five minutes right now, go ahead and get set up with that and look to see some thumbs up of people who actually got this code up and running. And I actually, when you. I'll just demonstrate how using it here, once you actually get it going, you can start to and talk to it. Just like the real time playground here. Hey Chat, what's happening?

Hey there. Not much, just here to help out about you. Anything interesting going on?

Well, right now we're working on a real time API application to basically make something just like you here at a dev conference.

That sounds like an exciting project. Developing a real time API application could really enhance interactivity and user experience if you.

Yeah, that's for sure. So anyway, I'll let you all sort of experiment with that yourself, but go ahead and get this checked out and then we'll get into the more interesting parts. Can it Separate voices, different voices apart from one another. Diarization. So you want to be able to actually identify the individual voices. I hear some success over here. So if there are people like speaking, it can definitely hear the different voices. But as far as understanding that that's person one speaking, that's person two speaking, that's something that hasn't yet been trained. Oh, okay. For. For cloning or for just running it? Yes. Question how much has been trained on actually. Speech response? So the question was how much of the training data for the model was basically around like speaking rather than textual? Because sometimes responses feel like they're like a voice version of a text response. I mean, it was trained on speech data, but I think that without getting into the exact details, you know, it's based on an LLM that is at its core, like also been trained on text data. And so like, you know, part of this is where prompting comes in to try to steer it in a way that maybe sounds a little bit more fluid than text. And there's also an evolving area of research. Are these best practices around how you think prompts from an audio point of view, are they shared? We're going to talk about that in a couple minutes. Yes, I think that's actually one of the most interesting things. You've been building on that quite a bit. The best practices, I think are the most needed, I think. Okay, the question was just about the best practices for voice prompting, and we're going to cover that in just a.

Of weeks.

Couple. Couple minutes. Yes. Do you guys have any plans to let people, like, do something like the voice cloning or like, you know, adapt to your own voices? So like, I'm coming from Japan and the, the point of like, you know, having a natural voice in your specific language is super big. So. Yes. So the question was about, you know, are we going to support voice cloning? I can't really speak on future product releases. And like, I can say that, you know, this is something that is simultaneously both highly in demand but also quite perilous. And so I think there's a lot of things being thought about right now. Yes. Can you describe how the paralingistic training works? Is it done in phases? Are the examples labeled for paralinguistic characteristics? How does that work? I mean, you can imagine the question was like, how is paralinguistic training done? And I can't speak to the exact details, but just you can think about how image training is being done with clip in terms of there being identification of what's in the image and Then the model steered to produce a similar sort of result. And you can imagine the same sort of thing of a labeled set of emotions, you know, in audio data. And then the model being steered to produce that same sort of result when it just receives just the audio without the label. In the back. You mentioned the response or like response interactions was how the invented user to stop talking. Have you explored letting the model interrupt the user?

I think that being helpful where you.

Create an agent mediating a meeting and someone is talking. Yes, I mean, I think the question was about, you know, can the model interrupt the user? And the short answer right now is no, although the developer can control this and the developer can instruct the model to interrupt at any point in time. But I think that, you know, the typical interaction pattern, as I mentioned, is that the user talks, the user stops talking and then the assistant responds. This doesn't, as you pointed out, in a meeting scenario, this is not what you want. You know, there can be seven people talking, maybe even having a side chat. And the model shouldn't always be jumping in and saying, hey, how can I help? I mean, so this is something that I think requires the model to have some a presence of when should I jump in? When do I have something new to add to the conversation. And that's actually something that, you know, we think is going to be pretty important for these scenarios in the future. So nothing to report on that right now, but it's an area of active act exploration in the back. Get my slides back up first. Yeah, let me put the URL back up here. Yes. So you know, I work closely with the folks at unlifekit. They sort of want to be at the plumbing for a lot of these AI agents. And you know, they have some frameworks that they use that, you know, support various APIs according to the real time APIs. So they then provide sort of the transport from the client to like the actual API server and then the interaction with the actual API is done over the web socket at that point in time. The question, Sorry, I keep forgetting to repeat the questions. The question was about interacting with Live Kit. How does Live Kit use the Real Time API? And that's the kind of thing where they provide the actual media transport to the client. Yes. Question about Linux Real Time module. Did you do any benchmarks? Because after 20 years of development it's finally solid, stable and I was just curious how it compares to what you're using. I'm not sure what you're using. So the question was about the Linux Real Time Module, like for audio or anything. Okay, well so most of what's happening in the realtime API is happening on the server side and then there's component on the client side to actually take the audio that's received and play that out in real time. And I know that like audio on Linux has been a challenge for many, many years, but like, if that's improved, I imagine that, you know, it should lead to a very low latency experience as well. Any implication to your presentation in Monster Scale Summit for databases, if possibly you could use real time operating system? Oh well, yes, I think that I mentioned these like smart devices. The question was about real time API with a real time operating system. This will definitely be a thing. A lot of the smart devices, things that are being built are being built on rtos. And you know, without getting into any further details, you know, people are doing this in like, you know, various small SOCs, very small form factor devices that are not running like a heavy os. Last question before I get back into it in the back. Yeah, my question is about reasoning. Could you explain a little bit more why reasoning isn't possible in the context of the real time API? And does that mean that there are limitations to the level of intelligence we can really achieve with voice agents? So the question was why can't the real time API do reasoning? And I guess that may be the most important question is like so or the most important factor is that so when you use a reasoning model, it generates reasoning tokens. Those tokens take time to generate, you know, those let the model sort of reflect on, you know, its internal states, hidden states, and then like after that reflection has occurred, it then emits like the actual user visible answer. The challenge is that that process, if you use like 01 or 03, can take many, many seconds. And if we want to have a real time interaction and like you asked me a question and I'm like, The answer is 42. That's a weird interaction. So I think the question becomes, well, absolutely, we need to find a way to bridge real time models and reasoning models. And you know, humans can do it. So I think that, you know, one should expect that we can find a way to do this in the realtime API as well. I'm going to get moving on. We'll have time for more questions, we'll have more breaks, but I want to make sure that, you know, we had an hour and a half, we've already shooed through half of it or half an hour of it. So I want to make sure there's time for the code here. All right, we went through this, this, this. Yeah, this is what I was about to talk about. Okay. So I'm just going to sort of briefly talk through this because this is not critical to understand for the actual coding exercise, but it is important to think about the application architecture and what API calls are made and how the actual pieces interact. And so in the demo we have, we don't want to have the client sending us API key, or we don't have the client have the API key. You know, as you know, API keys are secret. If somebody gets hold of your API key they can use to rack up very, very big charges, this is a bad thing. So we want to have a way to let the client connect directly to OpenAI without having to have an API key. And the way we do this is with something called ephemeral tokens. So we have a setup like this. And what's gonna happen as this first step is that the browser, the actual local application, the client is gonna contact our application server, our next server, to get an ephemeral token. It's gonna make this request to, you know, the token endpoint there, you know, in our next server. Then the server, when it gets this request, is gonna go contact the OpenAI API with the actual API key that's sitting safely on the application server. And so it's gonna go do pass that as the authorization. And in addition to starting the session, it's also gonna pass some additional startup configuration. This will be the model it wants to use. This will be the voice it wants to use. Can also include tools. And so this is all in Server js. And this is also where we're gonna look at first, because the first thing we're doing is some prompting. And, and the prompt is all specified in the server JS file that's up at the root. And so lastly, after we contact the OpenAI API with our API key, it gives us back this temporary token that's valid for one minute. We send that token back to the client in the browser. And now the browser has this API key that it can be used to connect directly to the OpenAI API server and start a session. And because we're using WebRTC, we can do some very simple WebRTC.ish stuff. This was 12 lines of code that I wrote for the 12 days of ship miss at OpenAI. This is all done for you already in the. In the sample application, but that basically uses that token to create an audio connection. All right, so one of the things I Mentioned it was that there's several branches within the repo. And so this is going to be. You can switch your repo to the tutorial one start branch, which basically rewinds the actual application to be just the basics of connecting to the OpenAI service. And so we're gonna show how to prompt it and kind of give it a multi layer Persona. And hopefully the model will do its best to follow that. So here are some ideas like, of, like the things you might want to carry in a voice prompt. There's a lot more. And actually, if you can look at. Oh, actually, let me show you. We'll actually just show some examples of prompting here. So we first did this prompt here that had a very. Where did I close this thing? A very, very simple prompt. You are a helpful assistant. There's. There's a lot of room to. To work with with that. And a lot of times we will actually want to have something that, that's a lot more specific. But we can get some pretty good behaviors. Like, here's a one that we could go try out.

All right, so here's the thing. You know, I'm kind of new here in the Big Apple, and it's a real jungle, if you get what I'm saying. So I'm trying to find my way around, and it's like, you know, every corner I turn, it's like, bam, there's something new. You know, it's a little overwhelming, but I guess that's what makes this city unique, you know? Oh, and the people, don't even get me started.

They're like, you get the idea. But I mean, what we basically asked it for is like, speak with a New York accent and. And then like, add in like, filler words and ums and ers. And just with that, we got a much different sort of voicing than we did with the. Hey, you are, you know, a helpful assistant. And so let me switch back to the preso here. Actually, I'm gonna do one more demo just for fun. Here's one that, you know, might be another sort of character that you could imagine here.

I don't know who you are. I don't know what you want. If you're looking for ransom, I can tell you I don't have money. But what I do have are a very particular set of skills. Skills I have acquired over a very long career. Skills that make me a nightmare for people like you. If you let my daughter go now, that'll be the end of it. I will not look for you. I will not pursue you. But if you don't, I will look for you, I will find you, and I will deal with you.

Deal with you. That's why we don't get in trouble. But, hey, Chat, that was a little bit heavy. Could you say something? If you're trying to inspire a bunch of coders.

Listen up. You all have a unique set of skills, skills that can change the world. You have the power to create, to innovate, to solve problems that others can't even begin to understand. Don't let anyone tell you what you can't do. You have the capability to achieve greatness. Stay focused, stay determined. And remember, I believe in you.

It was even funnier in rehearsal, but you know what a stochastic system will do. But anyway, you get the idea. Even just basically saying the name of a particular character gives the model a lot of stuff that it can draw on. Because again, it's a deep model, it has a lot of world knowledge, it can make a lot of inferences. Now, what's my next demo gonna be? Okay, so let me go back to the presentation here. And the key thing is, like, every, every, every mod, every AI needs to have an identity, like, who are you? If the user asks, you know, what's your name or whatever, or even like, you know, you know, tries to ask sort of questions like that that, you know, any human will be able to answer. It needs to have some notion of identity. It needs to know what the task is. You know, it might just be there to chit chat, but a lot of times people want to deploy voice agents for a specific task, and that may turn into tool calls. And if you can tell the model what it's there to do, it can reasonably. It can apply statistical, you know, sort of values to figure out what it should say. The tone is like the big thing of saying, okay, what sort of tone are you going for? Do you want it to be funny? Do you want it to be straight up? Do you want it to be professional? Do you want it to be a little nervous? These are things where you can really, like, drive the model a lot just by telling it, you know, the right tone and then even things like, you know, filler words. What vocabulary should it use? Should it try to sound like, you know, somebody using, like, modern slang should try to sound like a pirate? The model has an enormous amount of world knowledge, and it can draw on all of that to guide the output voicing. One of the things that you can do within the actual. The tool here. If I switch back to this is there's this little, like these little star icons. You never know what you're going to get when you click on this. But in this case you can basically give it just a textual description of what you're looking for. And, and they will turn that into something similar to what I showed of like, okay, here's the identity, here's the tone. So let me try this. You are a presenter at a tech dev conference. It seems to be going okay, but not all your jokes are hitting. Try to keep the audience engaged. All right, so if I do this, let's see what it comes up with. So again, it generates some things. Usually it generates a little bit more. I need to give it a little more on tone. Let me just try to speak with energy and gravitas and throw in a bunch of 20 twenties slime for the Gen Z folks.

Hello, tech innovators. Welcome to the most lit session of the conference. We're diving into the future of tech development and trust me, it's gonna be a wild ride. We're talking cutting edge, next level AI powered, blockchain infused all around bangin tech that's gonna say, that's fire. So keep those ears perked and those minds open. Let's get it popping.

Let's get it popping indeed. All right, so you get the idea that you can see this a lot. Again, you type in a few sentences, you get something, you put in, you know, like all the headings I showed before. You'll get a lot more and the model is very smart and you'll follow instructions very well. But let's talk more about the instruction following because I think that's part of the interesting thing of, okay, what's it like to program these voice agents that ultimately we want them to do something usually productive. We want to actually have an application. And so here we have just a very simple sort of thing for like, okay, what might a virtual hotel desk clerk being like? And you can give a very specific set of steps saying, first do this, then do this. If the user says this, then branch to this like, almost like, you know, describing a little flowchart. And you can also say, you know, definitely do this. Always use the user's name, their full name, or you can say, never, you know, just use their first name. One of the things I did in my previous company is we had a, you know, a Santa AI that was very popular from middle of December to December 26th and kids love calling it. Parents would put their kids so they could actually talk to Santa. It was actually quite magical. But one of the things that we made sure to include was never promise. Any particular example?