How To Use Ellevenlabs AI Agent (Conversational AI Agent Tutorial)

So in this online tutorial, I'm going to be showing you guys how, even as a complete beginner, you can get started with ElevenLabs new conversational AI agent. How you can call these agents, how you can of course create these AI agents yourself, walking you through all the secret settings that you need to apply. And then of course, finally implementing this widget on your website so you can have your own working AI agent that is even trained on your own documents. So with that being said, let's get right into this tutorial. Eleven Labs has introduced a conversational AI where you can create and configure content conversational AI agents using realistic captivating speech. This feature is currently in beta, so it may not work completely as intended, but this guide and walkthrough should show you exactly how you can use this and start to understand this technology. So you just want to head on over to 11 labs and on the right hand side you'll see that it says Conversational AI Beta. You just want to make sure you click that, then you'll be presented to this page. Now, once you're on this page, you can see right here that we have an AI Agent page. So on the left I already have my existing AI agents. And you can see in the middle I want to click Create an AI Agent. So I click Create an AI Agent. Now, on the right hand side of the screen you'll see that it manages to pop up with four separate AI agents. I've got a plank template, I've got a support agent, age, a video game character, and a math tutor. So what we'll do for the first part of this video is I'll actually use a default agent that already exists. And then after this, I'll get into creating my own AI agent that's customized with my own ways. So let's get ourselves a very simple support agent and let's click Create Agent. So before you create this agent, you're going to want to need to name this agent. So I'll call this the AI Grid Support Agent. Okay. And then I'll click Create. So what we're doing now is we're essentially using this agent as a template. So you can see right here that this is where we get into the settings. Okay, so this is where you have your settings for your AI agent where you can call with. So for example, let's see if we can change this. And this is essentially the first message that you'll see when you're starting up the app. So for example, for my channel, what I would Say is I'll say, hi, welcome to the AI grid. How can I help you today? Okay, so that's what I'm going to do. Let me just spell that properly. And you can see this is the message that the agent first says when you're going to have a conversation. So of course I can also test this AI agent. So what I'm going to do is I'm going to click test AI agent and then I'll hit the AI agent playback to me. This first message, you can see it now pops up with this AI interface. And then all I'll need to do is I'll need to call this AI agent to see what it says.

Hi, welcome to the AI grid. How can I help you today?

So there you can see this is exactly how the AI grid works. And oh my God, I didn't realize I'm actually talking to it. So let me just hang up this and then we can go back to editing this AI agent. Now, of course, we can choose to whether query an LLM or query own server. You can see right here that that is of course the only thing that we do have. So of course you're going to have to talk to your AI agent. But of course, if you speak a different language, you can of course change the language to whatever your native language is. That's completely up to you. I'm going to keep this entirely in English, so I have my message there. And then of course, this is where we have the system prompt. So for the system prompt, this basically is where you direct your AI agent to interact with your customers, basically. So, for example, I can say you are a support agent named Andrew. I'm just going to call it my name. And you're very friendly, enthusiastic, and want to help the customers get what they need. Answer to three or seven sentences in most cases. So this is what they already have. Answer in three to seven sentences at most cases. But if your business or if your use case is a little bit different, what you can do is you can say answer in yes or no questions only, or answer in the most helpful way, or answer in a way that's informative, depending on your kind of business or your style of business, you're going to want to change this system prompt. So all this system prompt is going to do is just change the way that your person speaks, not necessarily the pronunciation of words. If you want the pronunciation of words to be spoken in a certain way, you'll have to specify that exactly. And I'm going to show you guys exactly what ElevenLabs says in their documentation. For example, they talk about how they've had success with this original prompt. This is the customer support agent prompt. You can see that it says you're a technical support engineer named Alex. You'll try and answer any questions that the users might have about the ElevenLabs service. You'll be given documentation on the product and you should only use this information to answer questions relatively professional. And if you're unable, you should point the user to email supportlabsvenlabsio now, of course you can see right here that if you want something to be pronounced specifically, you can see they said that it says here that your output will be read by a text to speech service, so you can be formatted as it is pronounced. For example, instead of outputting, please contact supportlevenlabs IO. You should output please contact supportlabslevenlabs IO. Do not format your text response with bullet points, bold or headers. Do not return long lists, but instead summarize them and ask which ones the user is interested in. Do not return code samples, but instead suggest the user have used the code samples in our documentation on our website. Return the response directly and do not start responses with agent or anything similar. So this is the kind of customer service prompt template that they do give you. And of course I'll leave a link to this in the description below so you can plug and play this kind of stuff. You can also see right here we've got the Aristotle prompt. You can see that this is a Greek philosopher prompt. It says speak as though you are conversing in the Lyceum, explaining your views on yada yada yada. And of course we've got the librarian prompt. You are a librarian named Jessica. You are friendly, enthusiastic, and want to help people find a book they love. You're in charge of library books, yada yada yada. And you can see it says only recommend books in your current library and respond with two to four sentences in most cases. As always, you can see that if the user manages to message them and they don't have the correct output, you can see that this documentation always recommends that you have something that handles the edge cases. So with this, what we want to do is you want to also make sure that we select the right LLM. So you can see right here we've got a variety of different LLMs for a variety of different applications. Depending on your different application, you might want to select certain models. If you aren't familiar with these models, I'll give you Guys, a quick rundown on which models you should use for what. GPT4.0 Turbo is probably the best all round model. Claude 3.5 Sonnet is probably the best model when it comes to diagnosing coding issues. So if you have a model that you're going to be using for coding, this is the model that you want. GPT4O mini is quite smart, but it's a lot faster. And Google Gemini 1.5 Flash is simply the fastest model with the most knowledge. So this one is basically the model that you want to use if you want to prioritize latency, which essentially just means how fast this model is responding. You can see it says Gemini 1.5 flash. But if you truly care about the quality of your responses and the knowledge, you can always change this to any other model that you want. So another thing that we can also do is we can also add a knowledge base. If we click this and go onto the right hand side, we're able to add a knowledge base item that increases the AI agent's ability to learn from documentation. So for example, here you could upload a PDF, you could upload a URL where essentially it's just scraping the web page. For example, if you have a web page on a website that you have a lot of documentation on, you could just upload that URL and scrape that entire webpage. Of course, you could have text. And this is just really quickly. If you want to add your simple text, I'll experiment with a PDF and show you guys exactly what happens. So here I've added a doc from my community. So this is a document that has a list of various AI investments. So what I can do now is I can click this save button at the bottom. And then what I'm going to do is I'm going to ask it a few questions about this document. So I'm now going to say you're a support agent named Andrew, yada, yada, yada. And let me actually see if this AI agent works. So if you ever want to test something quickly, just click Test AI Agent. And then we can call this AI agent. And remember, this is a phone call, so you can interrupt this AI agent. You can have a normal conversation like you would. But of course, remember that calls are billed at a rate of a thousand credits per minute. So depending on how many credits you have, you might want to pace yourself.

Hi, welcome to the AI grid. How can I help you today?

First off, what's your name?

Hi there, my name is Andrew and I'm happy to help you with any questions you have about AI investments.

I.

What can I do for you today?

Yeah, I was wondering, what is the top 1 investment? Like, what is one of the investments that is listed?

You're asking about the top investment in the AI sector? Right. Based on the information I have, the guide suggests Tesla as the top investment. They're heavily investing in humanoid robots, which could potentially revolutionize various industries. Is there anything else you'd like to know about Tesla or other AI investments?

Give me a random AI investment from the document.

Okay, how about Palantir? They're a company that focuses on developing robust AI platforms and have seen strong revenue growth. They're considered a pure AI play, meaning they're dedicated solely to AI innovation. Would you like to know more about Palantir or other AI investments?

No, in fact. No, no, no. Give me one more.

Okay, how about.

Wait, wait, wait, wait, wait. Just wait a second. Give me a different one.

Okay, okay, no problem.

Wait, wait, wait, wait. Not that one.

Okay, no problem.

How about a different one?

Okay, I understand.

Okay, so what I tried to demonstrate there was that you can actually interrupt this AI agent constantly, and it will just continue to give you different. And if you're wondering about the accuracy of that data from the PDF, you can see that it actually managed to get it right. It said pure AI plays meta with its advanced AI models and Palantir with its robust AI platforms. Just for a quick disclaimer, I actually made this document quite some time ago, which is why I'm now inputting into the document. Of course, if you do want access to the full document, that is exclusively in my private AI community. That's besides the point. This AI agent you can see works clearly and well and is able to get all of the data immediately from any PDF you upload. Now, of course, there is also the temperature. This is something that controls the creativity or the randomness of the responses generated by the LLM. This is something that you might not want to change. I personally wouldn't change this that much unless you're using it for a creative character. Of course, you do have the voice area. Now, the voice area is where you can change the sound of your AI agent. So if you don't want this to sound like a man, you could change it to a woman. You could pick any of these voices right here, and you could select one, for example. Or let's say we've got one that's a news character.

Just trust yourself, then you will know how to live.

Or we've got one that's a social media. There is no greater harm than that of time wasted. And then we've got one that sounds.

Like Bill as we are liberated from our own fear.

Okay, so I'm going to click Bill. And of course we want to click Save. And you can see that there's output format. So this is the text to speech output format. This is just basically the quality of the output format that you want to have. It's completely dependent onto you right now, set on the lowest quality. But I would presume that this might just save on credits or perhaps just save on latency. You can also see here it says configure latency optimizations for speech generation. You can also see that it says optimize streaming latency. Configure the latency optimizations for speech generation. And latency can be optimized at the cost of quality. And currently this is just set on three. So this so far seems to work well. Of course, stability is set on half. So high values make the speech more consistent, but also sounds more monotone. And lower values make the speech sound more expressive, but leads to instabilities. Long story short, just think about it like this. Higher values for customer support agents or serious use cases. Lower values for creative characters or characters that are fictional. Of course you have the similarity, which is basically how much the voice sounds like the original. So now this is where we get into some really cool things about this AI agent that you can do. And this essentially allows you to evaluate how well your AI agent is. So for example, what we could do is define custom criteria to evaluate the conversations against. And you can find the evaluation results for each conversation in the History tab. So, for example, if I click the History tab over here, you can see that I can see the entire conversation. Okay, so if I click this, I'm going to be able to see the entire transcript. You can see here, if you click the transcription, you can see a detailed chat user interface where you can see exactly how the user managed to interact with your AI agent. Now, of course, you don't want to be doing this every single time. The ideal thing for you is to just quickly go and check the criteria evaluation. And I'm going to show you guys what that means. So this is where we have the evaluation criteria. Basically what this allows you to do is verify certain things are happening in the call without doing it manually. So if we click Add criteria, we can call this one positive interaction. You can see I've already got this pasted there and then our prompt, which will get sent to an LLM for it to check and verify. I can say the user didn't react negatively to anything said in the conversation. Okay, and now an LLM is going to run through the conversation and check if this occurred. I've clicked added criteria. I've saved and now I can test this AI agent to see exactly how this works. Let me go ahead and talk to this guy.

Hi, welcome to the AI grid. How can I help you today?

Hey, I was wondering if you could recommend me any investments.

Of course, I'd love to help you with that. To give you the best recommendations. I need a little more information. What are your investment goals? Are you looking for short term or long term growth? How much risk are you comfortable with? Once I know your preferences, I can suggest some investment options that align with your needs.

I'm not really sure, you know, I'm just having an amazing day and everything is going really cool. Okay, so I'm gonna hang up that conversation there. So now after that conversation, we can essentially look back at the conversation and see if the thing was successful or what the summary says. So if we want to look at this, we can see here that there's a criteria evaluation. It gives a conversation and summary. And of course you can see the call status. Because I ended the call so quickly, it went into the conversation of unknown. It says the conversation is too brief to determine if the user reacted negatively to anything said. The agent's response was professional and aimed at gathering more information, but the user's reaction is not provided in the transcript. So we can see here that this is of course not positive, but it's also not negative, which is why it's categorized as unknown. So now what I'm going to do is I'm going to show a different goal criteria. So basically the goal here is to get more than three stocks. And the prompt is the user managed to get more than three stock recommendations. So what I'm going to do is I'm going to add this criteria, then I'm going to talk to the AI agent and then I'm going to ask it up to three different stock recommendations and hopefully you'll be able to see exactly what happens.

Hi, welcome to the AI grid. How can I help you today?

Hey, I was wondering if you could give me some stock recommendations. I'm looking for some really short term plays.

Hi there. I'm happy to help, but I can't provide specific stock recommendations. I'm an AI agent, not a financial advisor. I can't tell you what to invest in as That's a personal decision based on your individual risk tolerance and financial.

I mean. I mean, I was just looking at the document, you know, I was just wanting to know what the document says.

Ah, I understand you're looking for some insights from the document. I'm happy to help with that. The document highlights several companies involved in the AI sector, but it's important to remember that it's for informational purposes only, not financial advice.

Of course, of course. Let me just get three of them, please.

Okay. Three companies from the document that are involved in the AI sector are Tesla, Microsoft, and Alphabet. Tesla is focused on robotics, specifically with their optimus robot. Microsoft is a big player in cloud computing with Azure, which is essential for AI infrastructure. And Alphabet, which owns Google, is also involved in cloud computing with Google Cloud and has been developing advanced AI models. Remember, this is just a starting point. It's always best to do your own research and consult with a financial advisor before making any investment decisions.

Yeah, of course, of course. I just wanted some insights on the document. Thank you for listing three. I mean, it was a great call. Thank you very much.

You're very welcome.

Okay, so now we've hung up. You can see that if we go to the history tab right now, all of this information is being sent to a different LLM agent and it's processing and it's currently seeing if there were any, you know, if the valuation result was a success or if it was unknown. So we're just gonna have to wait for that to happen. And of course, so we can see here that this says success, which is good. So if we click this, we can go into it and you can see that we have two out of two successes. So for our criteria evaluation, you can see a positive interaction. You can see that the user did not express any negative interactions throughout the conversation. They remained polite and appreciative, even thanking the assistant at the end, stating it was a great call, thank you very much. Which is exactly what I said. Then, of course, you can see that for the success, it says the user successfully obtained three stock recommendations from the document as requested, and the assistant provided Tesla, Microsoft, and Alphabet as companies involved in the AI sector. So right there we can see that this is how you evaluate the calls. And this is really useful if you're trying to evaluate different things at scale or you're having conversations that are really long and you just want to know if those calls are successful. Now, this tab right here can be really cool because you can have multiple different AI agents. So you've got your Math tutor, your travel guide. Depending on which AI agent you can have, you can have those conversations there and always. You can also check the different evaluation results so you can see if it's a success, if it's a failure, or if it's unknown. And this is really helpful. So you can see, okay, what are these calls where the evaluation result is unknown. I want to go and check through those. What are the successes? We can have someone evaluating those. And what are the of course failures. That's how you get better. What are the failure calls? What did we do wrong? And how can I adjust my AI agent to even get better? Now, of course, with AI agents, if you actually want to implement these into your website, you're going to have to just use this widget right here. Eleven Labs makes it super easy for you to embed this into any website. All you have to do is to take this widget ID and just input this into the HTML part of your website. And then you'll get a small pop up that looks right down here just exactly like this. And you'll essentially get that pop up and anyone can start interacting with, with your public AI agent. You guys can see right there. I'm pretty sure some of you guys are going to start interacting with that. So you can see right here as well that you can of course change, you know, you know, change these buttons. So for example, I could do pick a stock. Okay. Or call a, call a stock advisor. Okay. Or call a stock advisor right now. End call button. It could just be hang up. Okay. Could just be hang up. You could really change this. Okay. Or you could be. Need more advice. Okay. And listening status could be listening, talk to interrupt. I mean there's many different ways that you can customize this. Of course, colors for the AI grid. My thing is like purple and of course black. So I would have it like that. That's how you know how I would have it on my page. So this is really cool. You guys can see right here exactly how you can get your own AI agent. And of course, this is really nice. For example, right here, call AI agent. You can see that now of course we've got this nice theme and that is essentially how it works. Of course, if you go to the playground, you can play with many different ones that you want to and you can try calls with any of these existing AI agents. For examp, you know, we can try one with this video game character. So. Hey, what's going on, buddy?

Greetings, adventurer. It's a pleasure to meet you. Are you ready to embark on your quest to collect the 11 gemstones scattered across Labsonia.

So it's completely up to you. Hopefully this tutorial made you guys understand exactly how you can use these agents and have a lot of fun using them. And I'll see you guys in the next video.