The Voice of AGI: Talks [OpenAI, Known AI, AGI Inc.]

It's like a 5-minute Uber ride from my place to here. Yeah, I'm in Ashbury, which is paid in Ashbury. Do you know where that is? That's like in the middle, right next to the park, right next to the Golden Gate Park. Yeah, you should come by sometime. I think so, the Golden Gate Park is like next to the— it's where people are taking pictures. Yeah, Golden Gate Park is like this huge park that is like almost like central right here.

Welcome to the voice of AGI. We live in an insane time when it is possible to have a conversation, a real-time conversation with super intelligent systems, where you in your hands have the capacity to build agents that human beings can fall in love with, to create the, uh, intellectual experience of virtual tutors that can watch exactly what it is that you're doing and give real-time interactive feedback, to craft personalities, to train models that sound just like you and have conversations with yourself. Uh, it's truly an insane time to be, um, Yeah, I am building voice agents, so welcome to the voice AGI. How do you feel? Yeah, good. So we meet in a space with a rich history. AGI House has been the source of umpteen fantastic collaborators and conversations. If you haven't been here, welcome. There's a glorious verdant view of the city of San Francisco, and the spirit of Silicon Valley is in the room with us. So do co-create with one another in the spirit of the Gold Rush, of the founder worship, of whatever conception of Silicon Valley moves you. I have a lot of phenomenal speakers today, and they're going to bring us everything from some of the secrets behind building these state-of-the-art real-time voice assistants to clarity on how to create state-of-the-art, deeply emotional experiences with AI systems. So first and foremost, I would like to thank some of our sponsors: OpenAI, Zappi, Zoom, AGIA, Bolt, Hyperbola. And with us from OpenAI is Bill, who will give us some phenomenal insights into OpenAI's real-time text-to-speech Okay, let's hear it!

Hey guys, I'm Bill, I'm from OpenAI. I work on the startup team. The way you can sort of think who I am is I'm sort of like a core deployed engineer for all of you guys, all of the startups out there. Um, yeah, and I'm here to talk to you briefly about, uh, GPT GoTime, which is our most advanced GPT frontier model, uh, in our API platform. And I'll also tell you briefly of how you can build cool stuff with it. It might be a little bit too basic for some of you guys perfect for some of you guys who haven't really built this before. Hopefully it'll be useful for all of you. So GPT Realtime, what exactly is it? So speech-to-speech pattern is sort of an emerging scalable architecture pattern. You might be— if you've been building voice assistants in the past, you might be familiar with the chained architecture, which is the user speaking, and then you go to a transcription model, which is speech-to-text, and then goes to an LLM like GPT-4.1 that goes from text to text and then calls whatever function it needs to call, and then it goes to the API, which goes from text to speech once again, and then the user ultimately ends up with voice. And the paradigm that we're sort of shifting into is more of a speech-to-speech architecture that's combining audio and the language model into a single speech-to-speech model. What that means is rather than text tokens. So when you're passing in to a non-multimodal language model, the text is just text tokens. When you're using the speech-to-speech or the real-time architecture, the model itself is actually taking in your voice as audio tokens, and it directly generates audio tokens in turn and then synthesizes that into speech. So what you get from that is better porosity, better expressions of tones, of pronunciation, of voice. And the model itself will also be able to pick up minute details, nuances in your speech, like whether or not the user is sad, whether or not the user is happy. Et cetera. And I'm happy to also remind you guys that GPT Realtime has just recently gone from the preview stage into the GA stage. And with that are improvements like natural speech and reasoning. So it's like just much better at giving you guys like more reasoning capabilities as well as better pronunciation, better— just sounds better, better voice, right? And then also it extends in API capabilities as well. So we added support for MCP server support, image inputs as well. So image inputs is really cool because you can now pass image alongside with your voice into the same API for it to give back to you with voice. And then also upgraded audio quality, async function calls, and a couple of new voices as well. I'd just like to briefly talk to you a little bit of how to use the Realtime API. Just a little bit of quick primer. It might be a little bit basic for some of you guys, but I thought it'd be worthwhile for me to walk through it. So prompt structure is just like text-to-text models as well. The Realtime API also takes in text-based developer prompts. And because you're prompting In audio model, the way you prompt it is a little bit different. So you have different dimensions that you also want to prompt. So for example, the tone of the voice, personality, and those things are things in addition to tools, instructions, conversation flow, as well as safety escalation and things that you would probably include in your text-to-text models prompt as well. Just to zoom in a little bit about defining the tone role and the language as well. So this is kind of similar to text-based models. You know, when you say something like, you are an expert coding model, like, or if you're an expert customer support model here, for example, you should also prompt it in such a way that's role-based prompting. So for example, if you're building a language teacher, you should also tell the model something like, hey, you are a French French-speaking language tutor and you need to pronounce your speech in slow, well-articulated language, that's something that you can also prompt the model to be able to do. And also the roles and objective, I think those are all very similar to how you would prompt the old text-to-text models as well. You can also teach it because now you're also this is a speech-to-speech model, you should also teach it how to pronounce certain words. So the way I sort of think about the mental model here is you're teaching a recent high school graduate different ways to pronounce different terms. So for example, SQL, it might be obvious to us that SQL is pronounced as SQL, but it's not exactly sometimes clear to the model how those certain acronyms are pronounced. Tool safety and escalation. This is all like quite similar to how you prompt text-to-text models as well. But things like, so we have inherent set of guardrails built into the model. So like if you ask the model, how do I build a nuke, it's going to refuse that. But like if you're building a customer support bot with access to all the customer information. You also don't want it to, you know, like, like, dump the customer's information at times. And then also you don't want it to go off the rails for things they cannot be able to handle. So escape paths, like being able to escalate to a human, is important as well. And now that's the real-time API. Now we're talking about agents. I just would love to talk to you a little bit about how you might be able to leverage this to build, uh, agents, uh, forced agents. So the way that OpenAI, how we think about agents, is actually very simple. I think if you ask different folks how you build agents, it's like you get 10 different answers if you ask 10 different people. But for us, agents is simple. In abstraction, it's just a language model, a model that's equipped with a set of tools prompt, and so it's able to execute autonomously a task, perform a task end-to-end. And so for us, the way we abstracted it using the Agents SDK is via a couple of primitives called handoffs. I'll put guardrails as well as pool plumbing, right? And then so we now, we now have speech-to-speech support as well. So there's native interruptions handling truncation of RTSP TCP/IP socket support, also Twirling integration as well, and it works in the browser as well as server. Just to talk a little bit about handoffs, so if you're building a multi-agent systems, like different agents can focus on different tasks. So this helps with limiting the tool choices the agents has to decide between. So if you have, uh, 200 tools, probably don't want to have one agent to be able to decide which tool to call, right? And using tools under the hood to perform from the handoff, the Agents SDK. So let me just quickly show you how this might work. I'm just going to show you something that's the end state of something that you can build with Agents SDK and the Realtime API. So let me turn on volume here. Let me see if it's running.

Hopefully this is running.

Hi there. What kind of work What workspace would you like to set up today? Okay, yeah, so this, like, just let me briefly explain what this is. This is kind of a Notion-like notebook, but I have multiple agents that can help me build this notebook. And let's say I want to remodel, like, this AGI house, like, in a glorious Italian fashion, right? So right now, you can see we're in the workspace manager agent, and I have push-to-talk sort of enabled because I'm talking. And so it's like, it's always hard to, like, demo, voice that won't like while you're talking at the same time. So I'm just going to cheat a little bit by using the push-to-talk. Set me up a workspace with a couple of tabs around design and another tab around material. And as you can see, it's function calling. It's doing a lot of function calls and it's setting up— We've got a design and a material space.

Tab ready to go. Now tell me, what about the Italian style really inspires you for this project? Are you leaning more towards a classic rustic—

Okay, well, let me just voice over this because you can read the transcription, right? So what happened here is it has the workspace manager agent has independently called a bunch of tools to set up those workspaces and now has actually handed off the control to a designer agent. So now the design agent is endowed with a set of prompts that, well, actually are very simple prompts. It can basically search the web to, like, you know, search up different designs and actually populate those tabs in the workspace for me. And then also I have other agents as well, like there's an estimator agent, there's a cargo agent, and those all, like, so the estimate agent Estimator agent would be able to basically take into account all the designs the design agent has put together and be able to like pump out an estimation from that. And you can also see like there is like the prompt like that itself is endowed with as well as its own set of tools like a calculator tool. So like for sake of time, I'm not going to like, you know, do things end to end, but just to call out how we can access this. So the implementation of this demo is actually public. You can go into OpenAI's Build Hours repository if you search up just on Google. Number 14, voice agents, is the one you want to be looking at. So it's all open source. So jumping back into this, just to finish it off with a couple more pieces of advice as you build voice agents, is first of all, start with a clear goal, right? So what do you want to accomplish with it? Building a single agent that serves as one use case and handles the human handoff. Starting off with a clear goal, that's obviously, I think, a no-brainer. You don't want to slap on top of a fully scoped project all, like, to begin with. It's a little bit hard to handle. That helps with reducing complexity and build incrementally. Also, using prompting to bake in logic, very, very easy. Well, like, this is just like, you have, you have to prompt it basically. You want to be careful in prompting. Best practices like XML, JSON prompting all make sense as well. So actually there's a lot of like, I think, debate recently about evals. Like, when should you build evals? Is it like worth it? Is it like, you know, or is it just like a waste of time? The way I think about evals It doesn't have to be a comprehensive entire codebase almost. It also doesn't even need to be captured in code. It doesn't even need to be programmatic. So the way I would view it is human review. That's also part of DevOps. But my advice to you is before you start building something, really capture what sort of behavior you want out of the system early. Even just writing it on a piece of paper and at the end of the day when you finish building it, are you actually ticking the boxes? That part for me is a bare minimum and I hope you guys do that. Guardrails to keep agent on task. So AI systems, it's not 100% deterministic and 98% it might do what you want it to do. To guard against the 2% of times when things might not happen. Working, uh, as you exactly intended. For example, if the customer is injecting malicious requests, or if the model is outputting information you don't want it to be shown to the customer. For example, what if it leaks customer information, other customers' information? Uh, guardrails is a way to, for you to, uh, check as well as eliminate, um, that, those possibilities, those possibilities. That's pretty much it. And like, that's— I think you can go very, very deep, like, you know, like there are cost optimizations, like how caching works. But I think as a primer, like, those are definitely the essential stuff that I hope all of you guys, you know, keep in mind as you set to build your voice agents. Yeah, that's it. And I'll be here for a bit if you guys have questions or issues, and then I'll be back again to see, uh, the cool things that you guys have built, uh, at the end of the day today.

Thank you, guys. Any questions for Bill and for the OpenAI speech-to-speech? Oh, they sent the credits out yesterday.

Oh, so if you didn't get an email with the credits, we didn't get—

uh, correct. But, um, I can ask if they can try again.

I can help with that.

Oh, I had a— is this recorded?

Okay, for the credits part, like, I'll be here for the next half an hour if you have issues with it, line up and I'll try to get it to you. Yeah.

So the GitHub repository, is it like 9 months old? Is it?

How you figure out? Well, the— okay, no, the repository should not be 9 months old. Like, because like the 14th thing, like the, like the different 14 thing is 2 months old. 14th thing is 2 months old. So like, it's like a lot of things called build hours. It's like we have like a lot of live streams where we walk through different parts. So the voice agents thing that you should be looking at is the 14th one, not the 9th house.

On the, on skipping, um, uh, the middle step into text and then back to voice, um, going direct voice to token to voice token, it definitely helps with latency, but how does that eat up our credits?

How does that eat up your credits? So voice, the real-time is definitely more expensive than the text-to-text. So I expect it to be a little bit more expensive. So be mindful of that as you build.

Is that like 2x or 10x?

So we have the There is a— what's it called— if you go on to the model cards website, you can see the exact amount of things it would cost. But so the output tokens for real-time believe for the GA model is 32 per million tokens. And there's also like the input tokens, output tokens are charged differently, and also the what's it called, the text tokens are also char tokens. So the real-time voice model also takes text inputs and also does transcriptions as well. So I would just be mindful of that as you build. I would say the other thing to pay attention to is the context window is 32,000 and you can have sessions that's up to an hour. And so what happens if there's this ground-up truncation, what would happen if you just leave that parameter unfilled would be auto. So what happens is, as when you hit the 20-minute mark, your context window would be full, and so it would start truncating things from the beginning every turn. And so what happened there is like the cache would bust like very frequently, right? So what you want is you want to also optimize for the cache. The inputs tokens, when they're cached, is 10 times less expensive. This is pretty deep in the weeds. Like, I'm happy to like, you know, like to talk to you more about it.

But I feel like, yeah, what is the greatest application you feel has been built with the OpenAI Realtime Speech-to-Speech API? What company, what demo inspires you?

Yeah, so let's see, what can I say here? I think there are use cases where like medical scribes, like they use like sort of— well, like, while Scribe mostly uses, like, I think our transcription model actually doesn't really use real-time. So there's actually a couple of language learning companies that uses us to learn language, right? So the folks that would be sort of thinking about are like Duolingo and Speak. And yeah, but I think there's like also other— there are like a bunch of like cool use cases I think language learning, I think it's one of the ones that's very close and dear to my heart because English is not my first language.

Yeah, that makes sense. All right.

If you have a bunch of agents and only a few of them are voice, do you use the voice on those only and then you use just regular GPT-4 on the others and it will hand off normally? Or do you use GPT real-time on all of them? Yeah, you can use different mixture of modalities. You can have real-time, that's sort of the customer-facing one, you can have the other ones processing in the background. Like the guardrail model doesn't have to be a real-time model. It can just be a very like a 4:1 nano.

All right. Thank you, Sibeltra. Yeah, if you're using real-time API this year and last year, there's a lot of immediate magical stuff that came out. The first thing I did is I conducted an interview for a book that I wanted to write and it just continually asked me great questions based on what I had already said about what book I should— all my ideas, all of the details of the stories. So off of that, I think it's fantastic technology. Anybody who's used the ChatGPT voice tool knows it's powerful stuff.

Yeah, if you use ChatGPT and you use the advanced voice mode, so it's not the transcription thing, like the button right next to it, that is powered by the Realtime API. So you can build the same experience with the Realtime API.

Fantastic.

Thank you.

Thank you. Okay. Who here has been on a date in the last year? Has been on a date? A couple people. Okay, a couple lucky people. So our next speaker has created an AI voice agent for onboarding your dating profile. Her name is Celeste Amadon. She's a Stanford dropout— much higher status, to be clear, than any of these graduates— and is the CEO of Known AI, which will be the future of your dating experience.

Thank you for the great introduction, and I will be honest that I came here a bit unprepared today. I don't even have slides, but I'm excited to tell you about what we've been doing at Known in these last 3 months. So I started this company from the thesis that the future of consumer tech is personalization. And a year ago, I was working in early-stage VC, and I was probably pitched 30 personal AI assistants. And the pitches were all the same. People would talk about creating per-user data notes and how that would create defensibility over time and let them compete with AI labs' consumer products. They would say that their personal— that their bread and butter was personalization. And that led me to the question of, okay, how are you getting per-user data to do this personalization? And that's where the pitches would fall apart. People would either talk about scraping data from all of the other consumer apps from X, from Safari, from Spotify, and aggregating it and trying to create a digital twin of a person and using that to fuel how the AI would interact with the user. Or they would kind of delusionally say, well, I'm just going to ask them to tell me everything about themselves, and obviously they'll— they build the everything app this year. I would look at them like they were a bit crazy. And that left me with a question. How do you really learn enough about someone to actually do personalization? And I don't mean know their to-do list or the reels they like at 2 AM. I mean know them like a friend or family member would. And that's what I'm convinced is the key to creating a generational AI company in the face of AI labs is finding that entry point into people's personalities and preferences. But getting people to reveal enough about themselves is the hard part. So where is the pain the highest, the data the most personal for young people? People: their love life. Everyone here has been single at some point, and if you're not, you may be again. And I don't have to explain to you all that love and sex top the hierarchy of needs, and that's why what we've built compels people to share with us so deeply, because the connection is very obvious to a user. The more I tell my matchmaker, the more honest I am, the more likely I am to end up on a date that I like. And that matters a lot. So I'm happy that Bappi is here. I owe a lot to Bappi. This summer we had two goals. Let's build the best mobile product possible and get it out on the market as quickly as possible. And let's also try and de-risk our product. So we scrapped together a web app with a Bappi voice agent in 4 days. We feature-froze it since then, but 3,000 people have onboarded. In the last 2 months or so to knowns beta. We've sent 1,000 people out on dates. We've been testing monetization and seeing that people will actually pay for the outcomes of dating in a much more substantial way than they'll pay for the optionality that Tinder provides. So Tinder's average revenue per user in a year is $30. Our users have been paying us $30 per date. But I think the most interesting thing we've seen with voice is that Out of those 3,000 people, the average person at onboarding talks to us for 25 minutes, and they tell us everything from, you know, family dynamics growing up to what they do for work and how their boss just doesn't understand how capable they are and what they wish they could be doing instead, to their crazy ex-girlfriend and what she said to him that broke his heart. Or 98% of our users explicitly stating a sexual kink that they are looking for. These are mainstream between people. So some learnings that I've seen from working with voice agents and trying to figure out how to optimize them to get to know consumers. The first thing is that better voice does not actually equal more data. We tried improving our voice. We thought we should try and make it super natural and really human sounding, and then people stopped being as honest with us. So my takeaway from that is that the fact that our voice is markedly not human and actually identifiably a robot is what lets people open up to us and tell us things that they wouldn't say to a normal person. Another thing we saw is we thought, well, maybe men want to talk to a man. No, they don't. But both genders, men and women, have been more open with female sounding voice agents. Another thing that we've seen is that people will tell you information when you prod it out of them. So I told you earlier, 98% of users tell us a niche sexual preference that they have. That seems pretty out there, right? It wasn't always that way. When we first started doing this beta, it was like 25%. But what we found is even the small change of saying, hey, we know that everyone has specific interests whether it's an archetype or a kink. Don't worry, we've heard it all. And then we give some examples like— I won't say it here, but even just giving people examples of the type of information we were looking for radically changed the way that they were interacting with this technology. But you might ask, why does it even matter that we have all this data? What are we going to do with it? Well, I think it's really important because dating online started with websites like Match.com and Harmony and OkCupid. And those were great computer scientists who set out to figure out what makes people compatible. And then the Tinders and the Hinges and the Bumbles came in and they took out all of the vulnerability but also all the data and made it a sexy product, which is why they've been able to scale. But without the data to know who their users are or the results of who's going on dates and how do they go, they actually can't create compelling outcomes. And as I said earlier, compelling outcomes are what you can actually monetize. So in the last 2 months, we've set 1,000 people on dates. We now have 20 couples that are officially dating. I've never started dating someone in 2 months, so I think they're moving a little bit fast. I'm really hopeful to see what will happen when we launch our mobile app, which should be in the next month or so. Security is what I'm the most worried about, so we're going to test that really intensely beforehand. But I think that voice as a modal is such a more enjoyable user experience, and it compels people to be much more open than onboarding where you have to write down what you're trying to say about yourself. And I think it's going to change the future of dating forever because it's the first time that we can collect the data that Match.com and eHarmony and OkCupid had back in the day at scale from mainstream people because it's taken the friction out of the experience. Thank you.

Any questions for the CEO of Noname?

What was the website?

Oh yeah, if you want to go try the voice agent, it's nonedating.com. I might need to leave because I'm going to cringe. We feature-froze it 2 months ago. Yes, in the back.

Yeah, uh, what happens after that? So you started into like, okay, I realized that the best way to get their information information is too dangerous. Are you gonna basically scale Tinder and keep it like that and cross it out of that, or like going back towards like, I kind of know you better than anyone else because you've told me who you have sex with, whatever?

What's going on, right?

Yeah, so we send 1,000 people on dates that are curated matches. We actually pitch our users to each other, so in the way that a friend would introduce you to a friend, you say, hey, this is Celeste, she's from Seattle, she grew up in a big family, she went to Stanford. I think you might like her because of X, Y, Z. On our mobile app that's about to launch, we do— it's all run through mobile chat. The users are given opportunities to be introduced to one another. So if Alex was going to be introduced to Emma, Emma could be introduced to Alex. Our users can ask us questions after seeing the dynamic introduction we give and say, okay, well, you said that Emma's Catholic. Does she go to church on Sunday or is she just raised Catholic? And we can say, I don't know, or we can say, no, no, she goes to church on Sunday, don't worry. And when both users opt in, then they actually book the venue. And that means that we have a closed data loop of who's meeting, where are they going. And in order to get from after the date to seeing a new match, you have to tell us how it went.

Sorry for the follow-up on that, but does the agent have to match leave like knowledge of what the other person is doing.

No, that's been a huge part of what we've been doing for the last 3 weeks on my team is figuring out, okay, we have this onboarding where people tell us really intimate stuff. Obviously a data leak would be really bad, or even just someone taking the time to jailbreak our LLM would be really bad. How do we store the right data in different— basically broken up so that there's just things that the LLM doesn't know. So if you were to say, okay, what is their cake? It's like, well, I don't know, I can't tell you that.

I have no clue.

To my, to my knowledge, they have none.

Yeah. Our final question. Oh yeah, do you personalize the pitches from one user to another user, or is it always the same? Like, if I hear a pitch from someone on the platform, would that be a different pitch if I were a different user?

Yeah, I think a big differentiator, what we can do with tech today versus, you know, in 2003 when they were doing OkCupid is we can take all of the language information that we get in onboarding and we can do language-based introductions that are different per user. So the way that we would introduce Jessica to Adam, Scott, and Matt is completely different based on who those three people are because it's, it's part here's who they are and where they're from and part why they're right for you and what you might want to talk to them about and what you might enjoy about them. Which also means that we're getting people out on dates that they wouldn't have selected off of Tinder anyway. And I think that dating is a numbers game. So if you really wanted to find a partner in SF, you probably need to go on 10 dates if they were set up by your friends who know you. But if you sourced them from Tinder or Hinge, you would probably need to go on 25 because they're not going to be the right dates. And I know one thing about the men in San Francisco, it's that not the biggest womanizer in this city is going on 25 dates. So when known comes out, everyone should give it a try.

Very cool. Very nice. That was Anon.ai. Hopefully the voice agents that you create are as impactful in people's lives as the future of their natural language. So up next, from AGI Inc., we have DivGARC. They are offering you a $1,000 prize. The prize is to whoever can create a voice agent that takes actions in the real world. And we have a benchmark called RealBench on which you can prove that your voice agent is the best action taker that has worked. So I'm going to give Tim a chance to set up audiovisual. One round of applause for Tim.

When did you fill it out? Can you, can you send me an email with your org ID? This seems to be— I don't think I got any.

Uh, cool.

Well, yeah, I'm excited to be here. Um, so we at the event have been operating as a stealth research startup working on like reliable agents that can control your computer, do actions for you. We're actually now number 1 on a lot of public benchmarks, and we are announcing a lot of like really cool things on agents that can run on a device, including smartphones, bunch of cool things that's happening. So look out for an announcement from us. One thing that we have been public so far is some of the evaluations and leaderboards we have built around competitive agents. So this is one effort we've been doing around like how can you have open source sandbox environments for RL training. And we have cloned like the top 20 websites on the internet like DoorDash, Amazon, Airbnb, Gmail, and so on. And you can actually go and train your agents on these environments. And these environments also come with like their own evaluation function. So we have like different tasks where you can like evaluate How will your agent can go and say like, book an Airbnb in Tahoe or order a burger on DoorDash? And then we— and what we're trying to do for this competition is we have generated a bunch of waveforms. So we have a bunch of voice waveforms in different languages. And the challenge is like, we want you to go and build a voice-controlled computer assistant that can interact with browsers or even your desktop. And for the top— winner will be getting a $1,000 prize. And so how this will be working is we have a demo SDK.

Let me show this.

So we have a demo SDK which allows you to interact with the environment where you can use any of the— you can use your own local words like even Swappy, any other library, or Clear10Audio. Once you build a voice agent, you can evaluate on our system. We have given them a folder of tasks where like you have to like do a bunch of things on like DoorDash or Amazon or LinkedIn and so on. And these are all our clones, so we can give you a score of like if your agent completed the task or not. And so we have an example code here where you can actually We download this, set up our SDK, which allows you to interact with our platform and like, uh, do computer control. And, uh, you can use, uh, your whatever voice solution directly go build a desktop assistant using this. Um, so it's pretty simple. We have an open arena where you can also see the results. So whenever someone makes a submission, the results will be visible. You can actually click the code the participate button, you can create an account and log in, and then you are welcome to go and make a public submission here. And then based on that, we'll be like judging later on. I can show you like a one demo on how this will work. And so I have my teammate Diego on the screen who will be better giving this. So let's see how it works.

Okay, perfect.

Here we go.

Um, so, um, we, um, have our code right here, and in this code we have all of our, um, we have a bunch of tasks for your agents to do, um, that are just audio waveforms. Um, so that might be, you know, as Steve was saying, booking Airbnb, um, buying Pizza on DoorDash, right? And to make this extra tricky, we've actually gone and made these tasks in audio multilingual, actually. So you might open one task and it could be in English, and another one could be in French or Japanese. So we added this to be— so anyway, we have all the audio tasks right here, and what you'll do is you will basically use our SDK for building agents, which hooks— which you can build your agent in, and it hooks into our evals. And your agent will go and get a path to a waveform, and its job is to try and accomplish this task on the web, and it will get evaluated based on that. So this, uh, the file is included in the repository run_agent.py already has a basic kind of minimal agent which just operates on the accessibility tree of the browser and uses GT4O. And when you go ahead and you run this file, you'll see it's launching with the Omnisone 1 file. And right here, let's see, I don't think you'll be able to hear it coming from my speakers, but it's—

Search for laptop using the search bar and display the first product in the results.

It looks like it's saying to search for a laptop. This one's in English. And to display the first product in the results. So that's what it's doing. That's what the agent did. And as you can see, it ran and the eval is going to check what the agent did and evaluated it as accessible. So yeah, that's kind of how the SDK works. I'll be dropping in soon, so if anybody needs any help setting this up, I'm also happy to help with this. But yeah, good luck.

Does anyone have any questions?

Is there any setup?

Yes, so we have like a demo. You can hook that with your accessibility testing. So we'll put that in like the hackathon channel. And you can go to like the subset here. You also text spazzy for like interact with the environment. We also have the SDK and like the instructions here. These are all realistic replicas of actual websites, so you can actually go and then like also interact with them. So like this is our DoorDash clone where we have like— we've pretty much like cloned like DoorDash to make it like as realistic as possible. And then you can go and interact with it do things.

You can actually just order.

Also, it can actually like order a delivery engineer, and then we can see if the agent completed the task or not. We'll be around here if you have any more questions. How does it work?

So what is the real challenge here?

I mean, what is the technical challenge for us? So basically, we have a list of waveforms. So these are all the different audio files, and the goal is you want to build an agent that can process all of those different audio files and is able to successfully complete the task in the audio file.

In the mock websites or real websites?

Outside, we can verify it on our mock websites. You can actually, if you build an agent, you can run it anywhere you want. You can run it on GitHub Substream. The nice thing is with the mock website, we have a evaluation worksheets built in so we can evaluate and give you a score. And so if you are the number one on our leaderboard, then you win a prize.

Nice, nice.

Cool. Any last questions for Dip, CEO of AJA? Okay, congratulations. Okay, um, this is the point where we turn the mic to you. Um, so this is the project proposals spreadsheet. Can you raise your hand if you have access to the Voice of AGI spreadsheet? You should have gotten a particle text blast so everyone had access. Okay. What we're going to do is in silence for the next 3 minutes— so it's currently 12:59, let's say we start at 1 and go to 1:03— everyone's going to take the nascent ideas that you have for great voice agent projects and add them here in the Portuguese perspective project's location. Um, so yeah, no need to add location, but we really do want a description. Um, and if you are planning on building on something, please do build on it. Um, the idea though is, uh, please open this spreadsheet, uh, describe a name of your project, your description, short description of your project, and that way folks can know what you're working on and collaborate with you. Are there any questions on that, or does everyone both understand that and have access to the spreadsheet. Any questions? Okay, so it's currently 1 o'clock. Please, in total silence, add your project and a rich description of your idea. If you don't have an idea yet, that's fine. You don't have to participate, but this is a non-committal description of a potential project rather than a hard commitment to something that you have to do.

I have one question.

Yes.

There is Zoom as one of the sponsors.

Yes.

I'm curious, how do you get access to their real-time feature?

Because when I go to their website, they say reach out to sales team and like we'll be back to you.

So I'm curious, how do you actually start working with Zoom features?

Yeah, so Otis from Zoom had a last-minute commitment, so he had to fly out last night. I'll introduce introduce anyone who would like to use the Zoom API to him over email after this. So come see me and I'll introduce you to him. He actually offered a $100 prize for the best use of that API, but because he couldn't be there, it's only gonna be $100. So I'll send an email intro. And then the actually easy thing is if you send me an email, that way I have an email address I can send it forward to.

Your name is also really hard, so I'm sorry, buddy.

Okay, please silence for 2 more minutes and add your presentation idea. Hey guys, silence while everyone is adding it. If you've already added it, please be silent so the people who you're talking to can add theirs. If you want credits, feel free to do it afterwards. Thanks a lot. Hey guys, napkin. Please silence while people are adding their projects to the project.

Appreciate it. What's up? Thanks. ¿Cuál? Personal. Personal.

60 más segundos. If you've already added yours, please raise your hand. Okay, if you're not— if your hand is not raised, please do access the spreadsheet and add yours. You have about 30 more seconds. Okay, time. Um, so totally voluntary, but if you'd like to come up and pitch or describe your project, I do form a line here next to the bookshelves. Um, does anyone else want to give a quick 30— sorry, 60-second lightning presentation? Presentation to kick us off. Hello, my name is David.

I'm developing Bradley AI.

It's a voice agent for helping home health staffing, specifically agencies and physical therapists.

Thanks.

Cool.

Okay, if you want to work on physical therapy, of course, do join.

Okay, what's up? My name My name is Danny, and basically what I'm trying to do is similar to the girl that was here building a matchmaking thing for love, but just for friends. So it's like a friend that calls you to like, you know, introduces you to other friends.

Yeah, if you're interested in joining, I'll create a line here. I am working on health space, Will Schultz. It's an AI therapy agent, so focused on mental health and healthy living. Great. Clear up your mic. Hi, I'm Roman.

I'm going to work on That One Asshole Friend.

It's a video server where you and your friends are dropped in with a voice agent. And it's going to be humorously and aggressively keep the conversation going.

So it's going to use the OpenAI real-time agents and how I like it.

Very cool.

Hello, I'm Micah.

I'm working on a hackathon host agent. So you're traveling to hackathon, you're in the car, you're in the train, and you want to play the context window, talk to the agent, and you'll learn more. And maybe we'll run a party line with other hackers and the agent. Uh, Hackathon Host Agent, I'm Micah, look forward to talking to you. Oh yeah. Uh, I'm Nikita and we want to make Craig, which will scout Craigslist for the best deals and will use Baby to automatically call and, uh, call the number and barter for the best price.

Nice. Automated negotiations. Hey, I'm Ray, and we're going to be developing like a virtual health coach.

You're going to talk to it, tell it what you're eating, and then it'll kind of give you an estimate of the calories, give you info about the macronutrients, and give you some tips on what you can eat going forward.

Yeah, we're hoping it takes a lot of the friction out of like entering stuff into an app, but also, you know, add like a human side to it.

Hey everybody, I'm Max.

Well, no, reasoning as in like, it's not like, so what I've been building is a suite of tools I call the Instalments for both to-dos, notes, and then calendar. So the idea is to have them AI first, and obviously with voice GPT build time, what I'd like is to have a flow of this to use real time to interface all of this and create them. Right now it's actually quite complete in in the sense that we can create figures that otherwise will be sent to GitHub. That's the same thing for notes, but I would like to have the real-time interface for that.

Any other presentations? My idea is to have something to skip YouTube videos over the mobile. Okay. Voice-powered videos. Okay. So the next thing I'd like to do is announce a few of our other prizes. So the aforementioned Zoom prize is $100. But there's also a prize for the best use of Bolt. Who here knows what Bolt is? Raise your hand if you've heard of Bolt before. OK, so basically about 60% of you. So Bolt is an app creator. And if you create a voice agent-powered web app using Bolt, and it's the best one as determined by the judges, there's a $1,000 prize for the best use of Bolt. It's akin to a lot of these other tool builders, and a lot of folk have created fantastic websites with it in the past. So do go ahead and take full advantage of Bolt. VAPI will also be giving to the first, second, and third place winners a prize, a general prize, and VAPI credits and VAPI swag for the best voice AI project. So I think that's it for our prizes. Some final logistics. So lunch has arrived, and we'll be setting it up in a moment. So if you feed yourself, enjoy it. We'll send out an announcement when dinner shows up later today. There's an issue, by the way, with the bathroom on this floor. So go to the bathroom which is 2 floors down instead of using this one. It seems like it's not flushing as of an hour ago. So heads up on the bathroom. There is a Slack channel. I think it's been sent out in the final logistics email, but make sure you have access to that Slack. We'll do announcements, we'll be sending out the AGI link, as well as the OpenAI link, and generally comms will run through that Slack channel. There's also, yeah, I guess, a space for you to put your presentation. So at the end of the day, the same spreadsheet where you put your proposal, you can add your demo, and we'll run demos out of that at the end of the day. Our judges will include our opening act judge, Bill Chen. Thank you for coming. As well as a number of other esteemed folk who will determine whether you are the winners of the prizes. To be clear, anyone who ships something of value has done something great today. So good luck. Some logistics on the house. So you're on the third story of a four-story mansion. There are two floors below this. Please do use those floors and I know people have already scouted out locations and started hacking downstairs. Feel free to take a full tour of the floors below if you want to look in the backyard. It's really nice today, feel free. But those spaces are available too. There's tons of water, sparkling water, standard water in this corner if you need it. Feel free to take it. And yeah, I guess last but not least, I feel like there's this huge void, overhang in the quality of voice agents. We have incredible futuristic technology, and yet the economy seems mostly untouched. And so in my opinion, this is a fantastic chance to build a hugely impactful tool, something that no one's heard of or seen before. I've had insane experiences with these voice mode tools, and I think that it's just rearing to blow up. So good luck in creating something phenomenal.

And with that, let the voice of AGI be Yeah, like I think like it's like trade-off between latency and like, you know, intelligence depth, building a way to smaller the model, there's definitely a path to do that. So yeah, the 4.1 nano is like how small a model we offer. And then the 4.1 is kind of a long-term support model, is that one of those? Well, no, I think both of those will be long-term support, like, but 4.1 is like the biggest of them, it's like the unreleasing one. So like 5 is unreleasing. The other question I have for you is I actually have production traffic and I'm looking for communities where I can talk to people who actually have live customers. A lot of times the communities are people who are first-time builders, no tech, Spanish, cool, but I'm kind of past that and I'm wondering, do you know where I could find people that I can talk to? And then you're like, so, well, we have the online community, like OpenAI Community Forum. And so those people there, they're all builders. Some of them are people primarily in graphics. So that would be where I would start. Definitely, we also have builder events. So if you're a founder, I'll be back. I'll have to go into place, but I'll be back. And then go How do we get credits? Oh yeah, so credits. Oh, that's one thing, one thing or the other. Yes, because I do have to leave in a bit.

Yeah, so I think that you're also asking for credit. Yes, open air credit.

So we sometimes submitted the form, I didn't get anything.

Yeah, yeah, it's the same, right?

Is it too late for the credits?

Yeah, I think all of us get credits because—

It's not too late.

Got it, okay. Yeah, yeah, yeah.

I already submitted the application. Can you guys—

where did you guys get the link? It's in the article, in the first comment. It's in the actual article?