I've Built 100 Agents, here's why yours Sucks. [Real Walkthrough]

Don't settle for mediocre AI agent slob that is just permeating the roofing market and every other industry. To be quite honest, something that I'm really quite proud of and I want to share with people as a bit of a cautionary tale, a lesson and an opportunity for you and your business looking at AI is to understand some basics of how it is built. And I give you a little bit of insight into real world stuff that we do. So my name is Ivan Sand. I've built and worked with a team who has built hundreds and maybe even thousands at this point. Workflow automations across various tools from roofing, CRM, Zapier, make N8N. As proof in the pudding, here's our make grid in just one account that shows the million operations that are running in the background at any given point in time. This is a really cool way to display, to show things. We have functions working constantly for various accounts, you know that handle at this point over a billion dollars in roofing revenue has been served by many of these operations that are handling and being run at any given point in time. And so this is just make. Excuse me. And so what I want to share with you today is a little bit about building a voice agent. This is a prototype for Agent three that we are building in collaboration with, in collaboration with of our clients. And the goal of this agent is to handle follow up calls. And we are building it in a way that it is a little bit smarter. So a tool that you might be familiar with or heard of is vapi. There's a lot of different voice agent tools out there that you can use. Retail Synthflow, Charlie Crew, Air, so many of them like anything. And we've said this a thousand times, the tool does not matter as much as the process in which the tool is used. And the process does not matter as much as adoption. And adoption in the case of AI really focuses around training, retraining, fine tuning and evaluations. So when you hear about a voice assistant, you might hear about something like a single prompt voice assistant. So this here is a secret shop agent that we built and it is trained on just um, this is just a follow up call to a secret shop that we did in Louisville, Kentucky. So a lot of agents out there, you get like 800 characters to put in. This is considered small by our kind of standards, but this is how just one prompt goes. But there's so many variables that go into it, such as the temperature files that it has for its backup, how you configure the voice. There's so many different configurations that go into making this work. And this is what I mean by like generic, uninspired AI slob where people just because it's so easy to make something, it's so simple, there are all kinds of settings that go into something like this. And then the next level now is actually looking at the ability to link assistants together. They do this in squads and they do this in workflows. So this follow up agent, the concept of it is there is a global prompt which is very generic at this point. This is just the starting point. I'm actually in the process of improving this in a major rendition today. So this is not live yet, we're in process of developing it, but I have a big project that I'm doing to kind of upgrade this. And so essentially this is written in code. So you can see here like this is written in code. And we use a tool called claude. If you're not familiar with it, it's kind of like ChatGPT. If you're, you know, getting started with AI, we use this to improve the code. And we actually built a agent who is trained on this particular product that helps us prompt AI better. So framework for prompting AI to improve this. We actually use a framework agent to help us prompt and make it better. So basically we built a specialized employee who is trained in improving this particular task. It's not generic. So that way we have a way to say, okay, when we are improving this code, we need a framework to improve upon the code. And so this agent is actually a bunch of agents that work together. So there's the agent that starts out by phoning and saying, hey, I'm just following up on your quote. And in there you have a first message, any prompts, you have the global prompt and then you have the individual prompt. And there's some functionality around making this a global node or not a global node. So this is just the beginning of the call. So this is not global, it never goes back to this. But it's designed to extract certain variables. And the whole idea is to collect the objection. But the magic of a workflow is that you can say, okay, and like, notice here where it says the latency is 860 milliseconds. But then down here the latency is 1.1 milliseconds and 12 cents a minute. So that's because this particular agent is designed to kind of work together to say, hey, if the objection is a price concern, follow this pathway. And so although it's going to have the same voice and it's going to be aware of what the previous thing said. Imagine it like there's six or seven people in the room that have the exact same voice, but they're fine tuned to handle a specific objection. So in this one, the goal of this particular sub agent, if you will, is to say your job is to know how to overcome a pricing objection and you are going to abstract this as a budget constraint. A comparing competitor's pricing constraint. Is this a value question, like questioning value related to price or is it a matter of payment options? And then eventually have we satisfied the price objection and did we collect the customer budget? So this is the ability to do that. Now the reason you see these different latencies is because at each level we can adjust the model provider as well as the model that we use, right? And so we're building now around GPT5 where it makes sense and then you can set up individual constraints around this particular portion of the workflow by itself. So essentially it can then pass these objections back and forth. Now the piece that we're adding now is first of all, I guess one thing we are improving, how it handles each of these. I know about sales, but I don't. And I also know about permanent holiday lighting sales, which is who this is for, but I don't know everything about it. Right. So then whenever you're building there's a heavy research component and so I want to take you a bit into like how we do research. You'll see above here I have Perplexity Chat. Gemini are all open as AI tools. In addition to how you saw that we had Claude over here. But if I go back to Perplexity, the prompt that I put up here saying find me everything you can how to properly follow up on a provided proposal for permanent holiday lighting systems. I'm looking for the best closers Objection overcomes and value propositions. I want to develop a compliment, a comprehensive training apparatus for an AI agent that would take all the best practices of holiday lighting sales and bring them bring those value propositions in from the best salespeople in the world. So yes, what I get is some form of a output, right? And so there's a bunch of great stuff in here, but this is only generic market wide. But this I. And again I have to say that like you have to be able to like have some knowledge and have some qualification to ultimately look at this and review it a good way. A good way to kind of explain this is I was using AI one time to build a video And I could also build the soundtrack in the background. And I remember I was struggling so hard to make this soundtrack music because I couldn't explain it. There's a certain amount of domain expertise that when you get to look at AI as like, borrowing an extra 70 IQ points, well, it helps if you both speak the same language. And so I kind of gave up on that project for a while. And then I went and was watching a movie with my wife and my son Atlas, and it was a kids movie. And it was, ironically, it was the one where it's a variation of this show called Inside Out. But they talk about Dreamland, which is where, like, the producers of the Brain, like, producing the dreams for Riley, the main character. And I remember there's like the dream director and she. It's like she's drafting this dream for Riley and then she's talking about how the music crescendos. And then they show the dream that they made. And in the subtitles along the bottom, the captions, they said music crescendos. And it clicked for me. I struggled to make a soundtrack or like a background for this, for this video that I was generating with AI because I don't understand producer language. Like, I don't understand the music crescendoing and. And I don't understand the different zoom angles. Like, you know, in the movies and the scary movies, where it's like they look at the person in the mirror and it's like, it feels like the camera's going back and zooming in at the same time. There's a name for that. And so if you're going to build something with AI, you really want to have some domain expertise, because you can get hallucinations, you can get generalizations, and there's actually a strategy to improve upon this. My friend Nate actually commented on this the other day on social media. He said that you can't just, like, pump something into AI and then, like, walk away and assume it's the truth, which is accurate. So I wanted to kind of demonstrate here how you prevent that kind of thing. And so I like to get two really smart models to compete against one another. So this is the new ChatGPT5. And what I did was, the very first thing I did was I said, okay, I want to. So I said, help me build a comprehensive training document on permanent holiday lighting for this city. Right? So the framework here is goal context, action output. And I use a voice tool where I can just double tap like this and type things in. Makes it so I can Kind of vibe a little more. It's called Whisper Flow. But anyways, I. So I said I provided with a PDF document based on a web search that is a good starting point as well as a comprehensive document from the customer. Mountainside Lights, because when I took on this project, I should note we also had the customer provide us with his perspective on this, so the context for his specific company and what makes him special. So he provided me with this document which adds to the context. So I said I've provided you with a PDF which was the output from Perplexity's Deep Research. And then I also said I have this context from the client. Right, from the client. Drake. The voice thing put in a little weird, but I said, and I had to say this is my belief on sales, which is that objections kind of surface themselves as circumstances, others in self. You can see I kind of tripped up my word here. But circumstances, others themselves, circumstances usually break out into time, money and fit others is mostly an abdication of their decision making to others, which can sometimes be a smokescreen. So they're just saying I gotta talk to my wife. Or sometimes it's a matter of them having to process what the other person might say and how it can reduce the risk of making the decision without them there. So it's like, well, what do you think your wife would say, right? And then you can usually say, hey, let's both attack the objection without attacking, you know, that it's non confrontational. Sometimes it's more. Look, you're looking for support, not permission, right? So we have to say, are you really just looking for support in this? Are you looking for permission in this? And that usually prompts the person to not be like now they. It's weird for someone to say, no, I need permission from that person, right? And then finally we can say, hey, is the person aware that like you're shopping for this? Like is there certain things that they gave you to like meet in terms of conditions that would make them comfortable with you just making this decision? Is this something you can do for them, right? And then finally the self. So circumstances, others and self. The self, right, is people don't know how to make a decision because something afraid that they happened in the past happening again, or something going on currently that's conflicting with what they want to do or something they're afraid of happening in the future that is unknown and may not actually exist. So in circumstances, others and then self again, the voice text isn't always perfect. But that's the great thing about AI is it can generally get what you're trying to say. This agent will be essentially working its way from the top to the bottom circumstances to self right? The important piece of context is notes agent would be aware of the fact that the prospect has requested, received and open their proposal. There may be one or several options there and the customer knows the price of these three options. And I said hey, typically the options would look like just doing the front of the house with the lights or the front and back or just doing the front and the corner or wrapping around the entire house. So it could be 1, 2, 3 or 4 options. Use usually a mix of 3 or 4. Your job is to research everything related to holiday lighting sales and the value propositions when we're dealing with permanent holiday lights, the kind that install into the soffit area of the home or run a low voltage system by Mountain side Lights in Calgary, Alberta. Action I want you to build a comprehensive training document. Understand the context of the PDF that I gave you which is some of the initial research as well as the customer specific notes. Apply a higher weight to the customer specific notes and making sure we don't break with anything he has said we don't want to have anything that's conflicting or any hallucinations. So if something is not true, don't include it. And if something highly conflicts with what was provided by Mountainside Lights Question and answers document make sure not to conflict or contradict it. Output I want you to put this a list of the possible objections. So again you have to know a little bit about the subject matter to prompt it properly and you have to ask it to to apply a weight to one thing or another and to be implicit in the fact that it's not going to have hallucinations. So then it asks usually asks some questions in ChatGPT so then you answer those questions and it says great, I'll build this document. Now I did the exact same thing. I copy and pasted the exact same prompt into Gemini 2.5 Pro at which and it doesn't ask follow up questions but then it produced its output so that which was this right? So it produced this document here. So I read this through and I found some conflicting things from Chat GPT's response. Now I know there's those conflicting things but I want to work fast. So another good strategy is to say okay, let's take these two documents right that it produced and I said okay, I took the output of one and the other gave them to each other and I said please review the following Document and identify only the inconsistencies that could result in an embarrassing statement made by the agent. This is how reducing risk, not optimizing for versions and variants of our process that could potentially result in conflicting advice or objection overturns. And it provided me with the conflicts. So then I can now work faster through removing what is what is potentially a conflict. Now this is what eventually is going to be spread throughout all of these agents, which we're going to update by editing the code directly, which you can see here. Right. So finally then this will end up in a myriad of different places because there are different tools for automating different things. This is for what you see Zep upload memory. So this is a thing that basically what it does is it says, okay, when we receive a code, a completed conversation, we have to set the transcript. And then it's like either this user is found or not found. So we have a unique user every time in this thing called Zepp, which I'm going to show you momentarily. And it's then going to add it to its memory. So what does that mean? Well, there's different types of knowledge bases, right? So I'll actually load up a tool to show you what that looks like. So if I go here and go rag pipelines, I'll go to Mountainside Lights. So this is a pipeline that brings in data. And so if I was to edit this, you can see. That it ingests two things, the website and a drive where I'm. Where I uploaded all their Sumo code documents, like everything that we know about the company. And then there's a strategy for how we extract and chunk the paragraphs of text and data. And then we vectorize it using an embedding model. And you can look up vector database if you want to learn more about that. And then we build that into the vector database. So that is the company memory, right? So that is the brain about the company, but then there's also the brain about the customer. Right? The brain about the customer is what we call a knowledge graph. And so essentially every time a conversation is had or a text message goes back and forth, you add these nodes or threads where it says, okay, well we have an assistant, right, which is the AI, and we have the user, which is the person. And so it essentially establishes a link between people, places and things. So it understands that the user is someone named Josue. And this is for an air conditioning thing. And so it's like the assistant offered help with. You got to kind of move this again. You don't need to use this. This is just how the brain essentially is designed. But it says, okay, well the confirms the fan is moving, but it's not cooling. Offered help with troubleshooting. It asked for a technician focus. It scheduled service with a technician. So the assistant scheduled a service with Jose because he has an AC issue, right? With his ac. This is their phone number, right? We looking at this, we could say, okay, the user, right, has air conditioning airflow issue, it's not going very fast. So we've identified the issue, right? We've has a problem with their vents because the, the assistant asked about it, right? And then we can see that the assistant tried to call right Joshua. And this is likely because it got the name wrong at first. So then that's unnecessary piece. You can see it's not building around that. And this is Joshua because again, the name is Ho Suway. And so initially it got the names wrong, but it has figured out that this is not the right name. And it's now assigned that this is the person. And so now we can see just all the information. So this is the conversational brain that remembers everything, all the context about the person. So think of it as like a CRM in the brain. So every piece of communication, the texting that we send back and forth, the phoning that we do back and forth, builds up this knowledge graph, this brain about the user. And that's what this has to process. Every time a conversation is done is sending the output of a conversation to here. And so this is the, this is actually potentially available for people who are in like the roofing space, the holiday lighting space, access to people like us who can build it. Something like this just five years ago would have been a million dollar thing. Would like just the, the 16,000 lines of code that goes into just one piece of this. If you Google right how much there actually just searched this the other day for somebody like somebody saying, well, how much would this cost to do? I said, I said, even let's take it into it vibe coding for someone on Upwork, right? Just doing that, it would cost $10,000, 200 hours. And again you're dealing with somebody on upwork who you don't know if they're any good. And you still have to know enough about what to ask for to get them to do it. Because I was showing this to somebody who thought that our price was too high. And I was like, you don't understand what you're talking about. And so I kind of was like, I wrote a big long email showing the complexity. Because even when you build something, if I go into our voice agents, project into our dev folder and go into Agent one, this. Oh, sorry, my bad. Whoops. If I go back here, I keep going, clicking too fast over here. This is just the stuff we've made that didn't work. Right. This is the stuff that we made that and we saved like what happened and like what happened with these things, like this. But if I go like, like this and show what this can look like, this here represents 16,000 lines of code and it didn't work. This was version one. We're on version seven of Agent one and this is just an example of what didn't work. Whoops. Because in here you've got just inside a code block, you've got to write a bunch of code to process the next step. And again, this is the one that didn't work and we improved it from there. So I hope this has been educational and informative and helps people.