This Causes Most AI Agents to Fail and Its Completely Avoidable | Kevin Denman

The opportunity is really big, but it's important to not be naive and think that you can kind of vibe code your way into having a custom agent. You still need an AI engineer. A lot of people that think you can just magically, you know, one shot, prompt all this stuff, and it works.

Hey, I'm here with Kevin Denman, founder and CEO of Agent Graph, which is a decentralized infrastructure initiative that helps support blockchain integrated AI agents. Kevin really has specialized in the emerging agentic web for a long time. He's been a consultant, he's been in vc, he's supported a bunch of projects like Morpheus, and he had his own consultancy previously, so he's actually an expert in AI. He's coming to us from California. Sometimes Telluride, sometimes California. Just depends on where he's spending his time. So he's either the mountain mountain or the beach bum. Thanks for joining today.

Thank you. Thank you. Funny enough, I bought the domain Name Telluride Ventures AI At 2am the other night, just in case.

How do you like to buy your domain names? I've been buying them through cloudflare, mostly because they're a little bit cheaper usually.

I typically buy my domain names in a, in a tizzy in the middle of the night on my phone using GoDaddy. It's pretty much how it goes.

Well, I just, I saved you $3. If you end up changing switching providers because they're the, they end up being the cheapest.

Okay, nice.

Yeah, I definitely resonate with my son, who's eight months now. Almost every time I was feeding him a bottle, I'd buy a domain name.

Yeah, yeah, yeah, yeah. It's, it's, it's fun. And then when you're kind of, when you're operating on the edge of things, you realize that you're early and domain names are just a fun way to grab a little real estate.

Yeah, there, there is entertainment. So anyways, would love to get to know what you're working on with Agent Graph and maybe give a little bit backstory on how you came about it. What, like, what led you to think that Agent Graph is something that's really needed and since you just recently launched it and you already have revenue, so it seems like the market's already validating that what you're building is needed.

Yeah. Yeah. So I think the, where, where we are in the market right now is we're kind of, we're at a point where the infrastructure feels like it's, it's hit a level of maturity that allows everyone to start focusing a lot more on the use case layer. And, and when I think about mature infrastructure, it's like these like GPU clusters, the API standards around accessing inference, having a nice diversity of models from closed source models and open source models on hugging face. And then also, you know, you have to have to mention the kind of killer models or the killer agents which are agentic coding agents that make just software development a lot more accessible to everyone. But then also more experienced engineers can really, you know, multiply what they're capable of with agentic coding. So you kind of combine all those things together and what I, what I kind of came to the conclusion of is that the, the market is really ready to start building high quality use cases. But then when you look at the use, how we're addressing the use cases, there are a lot of vertical agent companies that are building kind of the one stop shop agent for legal hr, whatever that may be. But what, and then they're also, but then on the custom agent development side there's kind of, it's kind of lagging that. Right. So you have a lot of the VC capital flowing into like owning a slice of the verticalized kind of SaaS like agents. But then if you look at agentic coding, the cost of writing software is going down. And so I think there's like a massive opportunity to help businesses go, you know, and start building custom agents that are more bespoke and more aligned to what they're trying to achieve. And whether that's adding an AI layer on top of their existing product stack or applying generative AI like within their business like operations, there are just a lot of kind of opportunities to take a more customer approach. And, and then I think agentic coding is kind of a key unlock that makes a lot of custom development that five years ago you wouldn't really want to touch. Yeah, a lot more approachable if you can build AI kind of first tool chain in order to kind of address that. And so that's, so Agent Graph is really focusing on getting, getting those businesses kind of onboarded to the platform. And then when it comes to custom agent development, there's a key reality that I think a lot of people aren't looking at and that is that you still need an AI engineer. And I think there are a lot of people that think you can just magically one shot, prompt all this stuff and it works. But what I've found is you do really want to work with an expert who understands the agent architecture, understands the model inventory Understands the cost of running inference like up and down the stack, understands the non determinism and how you have benchmarks and observability and tracking. And so there's just like a lot there. And so the opportunity is really big. But it's important to not be naive and think that you can kind of vibe code your way into having a custom agent. Like there's a lot of expertise there. And so that's where the, that's where that like AI engine talent marketplace really comes into play. For Agent Graph where you as a business. The Agent Graph AI layer will help you translate what you want and the objectives of your custom agent into a really high quality agent design specification. But then that agent design specification goes onto the marketplace where our network of AI engineers will bid on doing the work to implement it. And so you get the kind of output of that effort, but then you also get to work with an incredibly talented AI engineer who's like top 5% in the world, has been studying the stack for years and has experience getting things into production. And that's something that I believe we are going to need more and more of as opposed to some of the other kind of more fantastical thinking around everything just kind of being kind of a one, one shot deployment. I think there's a lot more nuance that matters a ton when you think about architecture decisions and getting something to production.

Yep. I mean the, I really, no one fully appreciates the amount of nuance when you're building a specific product and the amount of effort, edge cases or the amount of little things you need to really think through to go from, you know, what you might have spent an hour vibe coding to something that is like actually useful and maybe durable. Right. Like you can rely on it day in and day out to actually work. So what do you, what do you see? The smart CEOs, COOs, the smart CTOs, what do they, how are they getting started with? Are they going and finding their agent, their AI engineer first before they start to do any build or are they more like I got this thing going on lovable and it has a little bit of leg. So now I want to, you know, productionize it. Now I need my like, what's, what's the flow that's working that you're seeing?

Yeah. So it really depends on where the founder is and also kind of what type of implementation it is. If you are really early stage and you're 0 to 1, you have an opportunity to start with like a blank canvas and so then it's Researching all the different frameworks and pieces of the AI stack and trying to be as informed as possible in your. What your starting point is there are a lot of standards that are kind of really early stage right now, and then there are others that are apparently a little bit more baked. So like model context protocol, there's a lot of adoption across the industry and so that one may be more baked. So if you're a zero to one profile, you know that like focusing on frameworks and getting, getting something out as like an MVP is kind of a good place to start. But then you have other, other kind of implementation profiles that are more adding an AI layer on top of an existing product. So imagine, you know, you have like, you have a, you have a Web3 wallet and you know that you want to add an agent into that wallet experience, but you already, you already have your stack and you already kind of have your code base. So in that scenario, you know, it's important to kind of think about how you layer on the AI into your existing product. How do you, and then how do you make it, you know, work within that. And so you can kind of embed an AI engineer kind of into your product team to help think through that. And then in those cases, it's starting with like a really clear use case, that's it focuses on the foundational components and maybe the integration components, maybe the inter, interplay between your application state and the AI layer. But then you're not like biting off more than you can chew and kind of creating, you know, a bunch of spaghetti inside of your product like you want to, you want, you want to start simple and then kind of grow from there. And it will become a more and more kind of critical part of your application as user expectations are changing more and more towards natural language. And so those are the, those are kind of the two flavors that I think I'm seeing more of. I'm also working with early stage companies and founders as a, as a strategy for Agent Graph. There's a lot of spend going into generative AI in Fortune 500 companies as well. And some of those companies are just spending, you know, half a million dollars on a proof of concept that helps them understand the implications. And that's, that's a potential pathway as well. But I like working with kind of the early stage, you know, entrepreneurs who are focused more on creating a differentiated product as opposed to, you know, like a, let's do a proof of concept. We don't care if it gets to production, but we would really need to Understand this tech, they're both really interesting, but like those are kind of. So it really depends on where you're at. Right. As a business, the one thing that's a thread across all of this is user expectations are changing rapidly since ChatGPT was released. And so we're all forced to react to these changing user expectations. And the kind of consumer application of ChatGPT just has a way at like accelerating. Everyone's thinking around that.

Yeah, I think you don't even notice it half the time. You get used to using it and you start to understand again the nuance of using each, each of the models and you can get what you want faster, which is pretty crazy.

Yeah, yeah. And then you start using all the other applications that are in your life and you're like, why can't I just ask a natural language question here? Like why doesn't everything else work like this? It's so much easier. And, and the reason why everything else doesn't work like this is because, yeah, there's a lot more custom agent development that will go into that.

So where are you seeing companies or people you're working with like fail? What are the common traps that you like, you know, keep seeing over and over again that people do?

Yeah, the, the, the biggest failure mode is around the non determinism of agents. So that's something folks will use ChatGPT and it'll be really, really good at like sourcing knowledge in some of the areas that they ask for and it's gets it like 80% right and they feel good about that. But then once you try to do that within your business or within your customer base that you know like intimately, you, failures of non determinism really pop up and you see, oh, like I would not have answered that question that way. Like, and then all, and then you can't, you know, you, so that's like the big challenge. And so this, there's, there's just expectation management when, when you're starting to build something like this. Like a lot of folks have inflated expectations and so Agent Graph tries to be like a trusted partner just in terms of educating those inflated expectations. And then, and there's also just some really practical implementation details that matter. So the ability to evaluate and have observability into the agent that you're building is kind of like a must have table stakes thing. Even if you're building something really simple, you need to know kind of how, what are the inputs and outputs and then you want to be able to evaluate it. There's A technique called like LLM as judge that allows you to use an LLM to evaluate the outputs of your agent and kind of score it. And so there are ways to scale evaluation. But yeah, like one of the general theme of kind of pitfalls is like under appreciating the aspects of non determinism. When you're implementing something like this, even in like a zero to one context, you want to be able to continuously improve it. And you can't do that if you don't have an eval framework and the telemetry implemented in a way where you can kind of make it incrementally better.

Yeah, I mean, yeah, you, that feedback loop I've noticed is, is even tough. Right. You mentioned how they answered it this way. Oh maybe I. That was off brand or we would have said a different way to retrain. It is really actually tough because you're kind of stuck with what you got. You got, you know, right now chat GPT5, you got the model like you can't really go back and edit it. You can update your, you know, brand voicemark down, but maybe that's about it.

Yeah, yeah. And there are layers to uh, you know, it is if you look at the AI stack, everything from your user prompt all the way down to inference, we're learning a lot more about these different parts of the stack and how they translate to quality outcomes at the agent layer. And just in the past few months the prompt engineering as a discipline has started to shift more into context engineering. Just having, just interacting with the LLM alone is no longer an option. Like you need to have memory and memory has a direct correlation to quality. If you're remembering the past, you know, 20 messages in a conversation thread with one of your users and then you're merging that with kind of the other aspects of the stack, you know, that's how you're able to get better outcomes. And so it's just, yeah, it's just having, having an understanding of what drives quality and then making sure that you're incorporating that into your, your kind of first designs I think is the best value that folks can get right now. And Agent Graph is trying to help close that knowledge gap. Essentially the, when you, when you go into Agent Graph and you start talking to the scoping assistant, it has all that knowledge baked in so that when you're trying to, you can focus on just what you want as a business objective and an output. But then the AI layer of Agent Graph is helping kind of close that knowledge gap so that your design specification before it goes into the marketplace has some of those important considerations. And having worked in data analytics, technical scoping while I was at Deloitte in the visualization studio, and like, I've seen this go wrong in so many different ways. And it really, it really, a lot of times comes back to the first kind of couple weeks of scoping a very complex technology and just kind of missing a lot of key components, there being communication challenges. And so that's like, that's the first feature set that Agent Graph is really, really dialed into.

Yeah, you hinted at it a little bit with one of the agents. How is your company and how are you personally using AI right now? Like, what are you.

Yeah, I, I mean, so Agent Graphics, it is, it's a, it connects the buyers of custom agents to the builders of custom agents. And so in, in that sense, it's a, it's a, it's a marketplace of talent focused on this niche of custom agent development. And when you think about marketplaces, makes a marketplace, what makes Marketplace A better than Marketplace B in some ways is efficiency. Like how can your, how can your market be more efficient? Right. How can you understand what the buy side of the market wants? How can you then, in the most efficient way possible, match it to the delivery side of the marketplace? And, and so that, that's just at a very high level. You know, we're applying AI end to end in order to make the Agent Graph marketplace from like a, from like a. What is the marketplace for standpoint, just much more efficient and higher quality by using AI. So scoping, matching, enabling builders with their proposals. All of those things are more scalable with generative AI if they're tuned correctly.

And do you think you'll be able to build this company with, you know, potentially maybe fewer marketers or fewer salespeople or fewer customer service people? Because you can AI ify those types of departments or those types of roles or parts of those roles.

Yeah, I do. I think I'm already sort of an example of that myself because I, I'm doing fundraising, I'm doing product, I'm doing, I'm doing more things than I, than I would have done if I started a company five years ago. So you can already kind of see that as a founder, like what, you know, you're capable of just like net more. And so what that tells me is that a smaller team will be capable of, of net more. And I think. But then just being realistic about, about what's a good candidate for that versus not, I think we, we're going to need great people and I think everything at the end of the day is going to be very, very human driven. Know, I think the agent graph, like market hypothesis is very human centric. There are buyers who own businesses that need to implement new technology and then there are AI engineers that are really, really good at that. And then the marketplace facilitates that. But yeah, everyone will be able to do more with the right tools. And so yeah, I don't really know what that means exactly for headcount or.

Yeah.

Do you think. I do think we're in a unique time where, where business models that had multiples of 4 or 5 can now have multiples of 10.

Yeah, no, I think it's, it's 10 x 100 x more with the same team is really what we keep seeing. And I saw a Wall Street Journal headline, I think yesterday, the day before that was, you know, one of the biggest CPGs lays off I think 16,000 because they've found ways to implement AI that can do that workload. And you're just like, wow, you know, like what? I'm curious to see where the world ends up heading, you know, as things grow. Are you seeing any sort of like specific thing be requested to be custom built over and over again or some sort of AI, like what, what do you see being built on your platform right now?

Yeah, the use cases are pretty broad. Like I kind of, I previously mentioned the kind of the different implementation types, right? Like zero to one AI layer integration into your existing product. Just a proof of concept that's kind of outside of your stack. But you want to invest in that just to kind of understand and learn and see something live. On the use case layer, there's just within, within agent graph specifically, you know, there's agents for clinical Trials, agents for Web3 wallets, agents for large brands that are trying to like release new products around their brands agentically. And so they already have a captive audience, they already have something that's really good and then they, you know, they know that they have that audience and they're not trying to like hire a bunch of people. But maybe they, you know, maybe they release like an agentic newsletter where they already have signups for things like that, where you can, you can grow the revenue of your like media company by implementing agents. So there's a lot of variety already. And so it's, it's, it's definitely encouraging just to kind of see how once somebody learns about, you know, applied AI, they start to look within their particular company and see a lot of Interesting opportunities to implement custom agents. You know, they all, they all ultimately are going to use their own, you know, private data and they're going to combine, you know, like an AI architecture together to kind of enable that. And that's, that's the big opportunity for everyone is just kind of combining all this.

And I think there's so much to be said about, like, the domain expertise. If you can marry someone who's really good at that thing, like, take a newsletter, right? There's reasons why some newsletters are amazing. And the copy and the images and the. Like, you, you look forward to reading that every day versus the ones that just kind of like, you don't even notice. And it's hard to train the AI on that nuance. Right? Like, what is the exact process that that writer or that team does every day to make sure that they deliver that great newsletter? I mean, so I think what you're doing is really interesting where you can combine that. Hey, I'm the best in the world at writing newsletters. Help build an AI that can take my methods and, you know, thought process and what's working right now and help me automate as much as that. So then I can go focus on fundraising or whatever it might be. You know, the other things that I'm also need to get done.

Yeah, and I. The Agent Graph application, it obviously has like an admin area. And that admin area is where I log in and I can, you know, do all kinds of things that like a product admin needs to do. And what I am starting to see, which is kind of interesting, is I'm building agents for myself to help me do my stuff. And so, like, I built, I built something very basic that I can link in a bunch of GitHub pull requests, and then it'll take those pull requests out and then it'll write the HTML for release notes update, and then I'll send that release notes update to all the closed beta participants. And I was able to kind of build that pretty quickly. And now I have a great way to communicate with our earliest users. And I didn't have to like, go buy a SaaS product to like, do that. Similarly. And if you log into Agent Graph, you'll see a little beta thing that pops out and you can click on it, you can copy, paste screenshots in and send feedback to the product team through there. That's another SaaS product that, like, I was, I was looking at paying like 20 bucks a month to have that little widget come out. And then I was like, wait a second, I can just build this like this afternoon. So these are, I think there are a lot. So back to the newsletter. If you Google like newsletter agent, like a thousand newsletter agents will show up. There are a lot of people trying to build the best newsletter agent in the world. And I, you know, that's great but I think what we're all going to find 10 years from now is you just build your own newsletter agent. Like you don't like subscribe to one and that you know, back to like decentralization. Like my take, like my contribution to decentralization is that I'm creating a platform where people can build custom agents with whatever stack they want.

So fast forward like 12 months. Where do you think the world, the tech world is with OEMs? Do you have any predictions for what's coming, coming down the pipeline?

12 months. Where do I think the tech world is?

6 12. You can pick your timeframe. You know, I feel like a year from now is maybe too, maybe too long because things have been going so fast.

Yeah.

You know, the world is also slow to adopt. If you think about it, we. I've been using ChatGPT in a similar style for almost four years now.

Right.

Like it's, it, the models have gotten better but I'm still open up the desktop app, still chatting with it, you know, pretty consistent.

Yeah. I think, I think that in 12 months agentic coding will still be the major narrative around AI. And then I think in the way that the gaming industry is kind of like a, kind of a, a leading indicator of a lot of core technology. I think agentic coding is a leading indicator of a lot of core technology. And you see that in the way that tool use was incorporated into IDEs and then, and then skills and things of that nature. And so I like to use that as like a leading indicator and I think it's going to lead to the continued, the continued software upgrade essentially for everyone. There's a very long tail of businesses that have really, really shitty software and they're all about to get a big upgrade because of attempted coding. And so I think that's like, that'll continue to be a big narrative. I think like multi agent systems that are memory capable are going to continue to grow. I think every business SMBs all the way up to like Fortune 500s are going to have some sort of like AI agent building center of excellence. Like in 2010 there was a big transition in corporate America where everyone started building internal data analytics and data science teams. Those teams are still kind of there today. But generative AI is, you know it's, it's a little different than your traditional, you know, big data business intelligence platforming. And so I think you're just going to see it as like a core capability increasingly, you know, I think we're, I think generative AI spend is on track to hit like 100 million next year. And so I think just folks will just continue kind of pouring capital into the space and so the market will just keep growing. But yeah, I think we're, I do feel like, I feel like the, a lot of the core innovations, like the step function innovations happened in the past three years and I feel like now we just need to build a bunch of agents on top of that.

Can't wait to come build some agents. Thank you so much for coming on and sharing your knowledge. Kev. This is great. Thank you.

Yeah, thanks for having me. James.