Having an Actual Conversation with Data Using an ElevenLabs Voice Agent and n8n

Hey there, I'm Eric. What information are you looking for today?

Hey Eric, I missed the last meeting. Could you fill me in on the current challenges that we're facing with our AI chatbot project?

Okay, let me check the project database. The main challenges right now are integration issues with third-party APIs causing delays and the chatbot having trouble handling complex customer queries that need multiple steps of reasoning.

Okay, got it. And what team members are working on this project and what do the milestones look like here?

Okay, let me see. The team members are Alice, Bob, and Charlie. The current milestones are basic chatbot framework implementation by January 15th, integration with the website and app by February 5th, and internal testing and debugging by February 20th.

All right, awesome. And we were targeting for the product to be wrapped up by March, right?

Let me quickly check that for you. Yes, the target launch date is March 1st.

Okay, cool. That's all I've got for you.

Okay, great. Is there anything else I can help you with today?

Nope, that's it. Thanks, Eric.

You're welcome. Let me know if you have any other questions.

Okay, so there was a quick demo of the project manager conversational voice agent that we're going to be breaking down today. As you can see, we have the workflow on N8N, and then we have the ElevenLabs voice agent, and it has access to a vector store that has all our information on that Google Doc that you saw right here. About our projects. So let's get into this build. Okay, so AI voice agents, they seem a little complicated, but they're really not. So I'm going to try to break it down in a way that's all going to make sense. And you're probably already 75% of the way there. So what we're looking at here is just a RAG AI agent within N8N. You've probably built one of these before. We have our chat model, we have the vector database, and that's really it. So today we're going to be breaking down 3 main things. We have our vector database. We have the N8N agent, and then we have the ElevenLabs voice agent. So we're going to start off here in N8N. We're going to break down how we're talking to it through the webhooks, how it responds, what's going on here. And then we'll get into ElevenLabs and look at how we actually prompt the voice agent to extract the information it's looking for and then send that POST request to us in N8N so that we can hit this vector store and respond back. First things first, getting our information into the vector database in Pinecone. So we have our Google Doc right here. As you can see, it has 3 projects. We push that into Pinecone using this workflow here where we download it from Google Drive. We extracted the text and then we pushed it into Pinecone in a namespace called projects. So as you can see in Pinecone, we have a namespace called projects. It created 4 vectors, as you can see right here, 4 items. And then in our voice, or sorry, in our RAG agent, we need to make sure that we're sending this agent to that namespace again called projects. So it's looking through for the right stuff. So I know I went over that pretty quick, but you can download this workflow as well as this workflow from my free school community. The link for that will be down in the description. You will just come in here, click on YouTube Resources, you'll click on the post associated with this video, and then you have all the files right here to download. You'll come into N8N and you can download them, or sorry, you can import them right here from file, and then you'll have everything up and running, good to go. Then if you wanna take your skills with N8N a little bit further, get some more hands-on help, please check out the paid community. We've got you know, great community. They're asking questions, sharing builds, that kind of stuff. You get your questions answered a lot faster in here, um, as well as about 5 live calls per week, a great classroom section that's constantly growing as the space evolves, as we're learning more. And yeah, I would just love to see you guys in here. You can check that out in the description. Anyways, back to the agent. So, um, it's a super simple build. As you can see within my AI agent here, we set it up as a tools agent, and I didn't even give it a system prompt because it only has one tool. And it's pretty easy because the query comes in and then it has to search through here. We define that this is a projects database. It says it retrieves information about our projects. So not much area for the AI agent to get confused here. If you hooked up more tools, of course, you'd want to prompt it a little bit. But it's a very simple RAG agent, right? And if you've built one out in N8N before, you've probably connected it through the N8N chat trigger, or you connected it to Telegram or email or Slack. It's pretty much the same thing here, except for now we're talking to it through voice. Using ElevenLabs. So you could even grab an AI agent that you already have built, and all you have to do is switch out how are we talking to the agent and how is it responding back. So it's much more simple than you may think. In this case, we're sending information to the agent using ElevenLabs, and then the agent communicates back and sends it back to ElevenLabs. So that's why we're using webhooks here. So we'll click into the webhook. As you can see, it's a trigger, so it's actively listening for the agent to conversate with a human and then send over that question. So we've got, um, This is a POST method, method, because like I said, ElevenLabs is sending information to us, to n8n. We've got a URL here, and right now it's a test URL. When the workflow in n8n is active, you— this will switch to a production URL, so you'll have to switch out that URL in ElevenLabs. But, um, for now we're doing test, and we'll have to listen manually for each event. The path, all this does is change the actual URL, so you can play around with that if you just want to clean it up. And then as far as authentication, we don't need any, but for the response, make sure you use, um, respond to webhook node rather than immediately or any of these other options. So it's like I said, the same way that you would text it with Telegram. It would get your text, it would search through the vector database, it would think about how to respond, it would create a response, and then it would Telegram you back. It's the exact same process except for we're getting a query from ElevenLabs and then the agent is responding back to ElevenLabs. So that's how this is going to work. Okay, so let's click into an execution and actually see this happen. So this is the first question that we just asked in the demo earlier in this video. Let's look at the webhook trigger. We've got all this stuff coming through, but really all we're looking for is the body, which is the parameter we sent over from ElevenLabs called question. The question here is, what are the current challenges for the AI chatbot project? So this question gets sent over to the agent, as you can see right here, json.body.question, and we just dragged that in from down here where we can see everything that came back from the webhook. So now the agent uses this question to hit the tools it needs to, to create this output over here, which it eventually sends back to ElevenLabs. But let's click into the logs and take a look at what actually is going on. So, um, what we see is that it's getting the question right here, which is, what are the current challenges? It's going to send that to the vector database tool, um, and then we got a response. But before we get the response, the vector database tool has to go search through the vector database. So Here's the question. Here is what's in the vector database. It has to search through the embeddings, as you can see here with the different parameters. And then it's going to send this back to the OpenAI chat model to look through and actually create a response based on the question that the human asked in the first place. So now we got this output, and finally it comes back to the agent where this is the final output that we got. And then that gets sent right over to the webhook. So hopefully that all makes sense. Yeah, it's really not too complicated when you really break down and when you understand the fact that you could probably take this RAG system that you already had built out, this RAG AI agent system, and all you're doing is changing the input and changing the output. And then, you know, exactly, you're probably changing what's going on right here, what the agent's actually looking at as far as the input, but that's really as simple as it needs to be. Okay, so now let's look at what's actually going on within ElevenLabs. So I'm in ElevenLabs, we're at the conversational agents section, and this is the agent that we were just looking at, Eric. He was a project manager and I just It's Eric because that's the name of the voice in here, as you can see.

Riches, complain is poverty.

There you go, perfect. But there's just a couple things that we want to set up here. So within the agent, the first thing to do is set up the first message. If you leave it blank, then when you click call, you'll just start talking rather than the agent talking to you. But we just said, hey there, I'm Eric, what information are you looking for today? Then we set up the system prompt. So this, you know, there's a lot of testing that goes where you set up a prompt, you call it, you talk to it, you see what it's like. And then you come back and change some things. But essentially, here's the prompt we have. You are a project manager responsible for providing accurate and helpful information about current projects. Your role is to actively listen to the user's question, understand their intent, and use the N8N tool, which I'll show you guys how to set up the N8N tool in ElevenLabs in a sec. But you use the N8N tool to query the project database. After you receive the response from N8N, then you provide the user with a clear and concise answer based on that data. And then I came in here at the end and said, maintain a friendly tone, use filler words like, um, and okay when you're actively listening and waiting. Don't repeat the question back to the user, just give a concise answer. So right here we're using the model LLM Gemini 1.5 Flash. It's the fastest, as you can see. You have other options to choose from though. And then for temperature, I think default it's set at like, you know, 0.5 right here, but I set it at 0.8. Basically just the randomness and creativity of the responses that you're going to get from whatever LLM that you choose. And I think that it's fun to have it a little higher, especially with something like RAG right now. But, you know, if you really needed like specific information on policies or, you know, stuff like that where it has to be precise, then you would probably want this to be lower. Anyways, token usage, you could hook up a knowledge base, but it would be a little bit more static. So we want to use a vector database that it's going to be called through NN because then we can update the vector database with some sort of automated pipeline rather than just having like, um, certain files in here, right? But anyways, this is where the magic really happens, the tools section. So as you can see, we only have one tool and it's called N8N. So what we did in here was we named it N8N because just easier for prompting is we understand that it's gonna be sending information to N8N and then waiting for a response from N8N. The description here is use this tool to access the project database. And then this is where we set up our webhook sort of request. So we're We're using a POST because we want to send information to N8N. And then we have the URL, which is just the one that we grabbed from this webhook right here. Click to copy, and then we pasted that right in there. And then you can leave all of this stuff as it is besides the body parameters we actually do need to set up. So this is because, like I said, we're sending information over, which is going to be the question. So here we need to add a description, which is this information is going to be passed to the LLM, and it's going to describe how to extract data from the conversation that you're having with this voice agent. So right now all we had to say was ask the user what question or questions they have. But if you needed to extract more, you know, more detailed information or more information, like if you're appointment setting or you're gathering information like, you know, what's your name, your email, your phone number, all this kind of stuff, then you'd have to set up different parameters for each thing. So you'd have name, you'd have phone number, you'd have, you know, how many guests, what time do you want to come, all that kind of stuff. If, you know, based on the use case. But right here, all we needed was the question, and the description is question from the user. So that's why, um, in n8n, when we're actually getting the response back from the webhook, it came through in the body parameter called question. So right here we have question. Like I said, if you had a different use case, you would— here you'd have name, email, date, whatever it is. But this is question because we called this question right here. So that's all we had to do. We save that, and now we have pretty much this stuff set up, and that is why it knows here to call the tool N8N using that webhook. Okay, so now that we got everything set up within ElevenLabs, we'll go ahead and test out the AI agent. And just a reminder, everything in N8N right now is still in a test environment. It's not an active workflow, so we're using the test URL in our webhook. We would switch this to production and then it would be actively listening all the time. But for now, we have to manually hit test workflow so that it's listening for ElevenLabs POST request. So what I'm going to do is talk to the agent. We'll ask a question. It's not going to be able to access this vector database because the webhook isn't active, and then we'll ask it to try again and it should be able to get the information the second time. So let's do that.

Hey there, I'm Eric. What information are you looking for today?

Hey Eric, could you give me an update on our marketing dashboard project?

Okay, let me check the database. It looks like the project is not in our database. Can you double-check the project name?

Yeah, no, I'm pretty sure it's called Marketing Dashboard. Can you try again?

Okay, let me try again. The Marketing Dashboard project is currently on track.

Okay, cool. Thanks for the update.

You're welcome. Is there anything else I can help you with today?

Nope, that's it. Thanks, Eric.

No problem. Have a great day.

Okay, so that's pretty much going to be it for this one. I just wanted to come in here and make a quick one about how you connect ElevenLabs to NN super easily. A lot more simple than you may think. At the end of the day, all we're doing is changing the input and the output. So if you had something like a personal assistant agent that had the tools connected to it and a pretty robust prompt, as long as the agent's getting a query from something that it can take this query, understand what it needs to do, how to take action with its different tools, and then respond back. You could pretty much integrate this sort of voice agent into any other N8N agent workflows that you already have built out. Just a little bit of prompting will have to go into the ElevenLabs side of things, like we talked about having to set up when to use different tools or when to send information to N8N, what parameters you want to send over within your tools if you need to get more information than just like a simple query or a simple question. But from there, Not too difficult. So I hope that this one helps you guys. Voice agents are gonna be a pretty cool space. I mean, they already are, but just heading into this new year, lots, lots more use cases as this technology gets smarter and faster every day. As always, if this one helped, please leave a like, comment what other use cases you wanna see with some voice agents, and I really appreciate you guys making it all the way to the end. I will see you guys in the next one. Thanks.