AI agents can fail in cascading ways when one mistake triggers errors down the chain, especially when an agent has too much autonomy. The panel explains why 80% accuracy isn't enough for medical or scheduling assistants, and recommends guardrails, better retrieval systems, and safety layers to prevent disasters like agents dropping database tables.
Chapters
What about hallucination? Right now we all face this issue, right?
If you give a lot of data and you don't have a lot of retrieval, doesn't work really well, then you get into a hallucination problem. And with hallucination it becomes a trickling effect. If you have the agent, the main orchestrators hallucinating, then all of the sub agents end up hallucinating and that becomes a contextual problem. Just one of the bigger problems that we see is how you manage hallucination. And there's different techniques to help you do that. But I think the first step is getting retrieval working well and reducing hallucinations with that helps with that experience, basically. And then there's like other things that people do to make sure that you're optimizing the type of prompt that you have that you're sending to each agent. Because depending on the type of prompt that you have, it could get your agent to do work really well or not do work well.
They can try to send so many requests to your application that your model fails to respond to other users that need it. So it's called model denial of service. If you give your agent too much autonomy, it's called excessive agency. If you give your agents excessive agency, what ends up happening is the agent can drop your database table makes a mistake, and if you're giving the agent's right access to your database, it can end up dropping your database table because some user ask about cleaning their data, for instance, he may think that they, okay, I may need to truncate their data. Let me just truncate the entire users table, right? So it can make these kind of mistakes because the LLMs have these kind of vulnerabilities. Like you do need to make sure that you have guardrails in place, so guardrails to check for hallucinations.
When someone tries to build an agent, I usually ask, is 80% performance enough or not for your agents? And in a lot of use cases, 80% is enough. For example, if you're just solving customer success tickets, all right, if you're already solving 80%, that's fine. That's already a big gain for your company. So you can stop here. And on the other hand, there are some use cases where you need 99.9% performance, and especially when it's customer facing, when it's related to regulatory industries or things like this. For example, take a personal assistant agent that is booking meetings for you with your, I don't know, your prospects or whatever people you have to meet. 80% is clearly not enough. If it sends the wrong time slots one time out of five, that's not okay. Your, the people you will meet will be very angry about that. So there are some situations where 80% is enough and other situations where 99.9 is required. And so we are very focused on ourselves to bring agents to 99.9 for those high stakes agents. And so in a way how you can do that is clearly removing some probabilistic behaviors at some points. And why this 80%? Where does it come from? This 80% is the compounding of errors. And because when you are building agents, usually you take a process that has multiple steps. And I don't know, if you take an LLM for example, with a market standard 5% hallucination rate and you have 10 steps in a row at the end, the success rate will be just 60%. So close to one out of two, you will have some hallucinations, anything like this. So that's working for 80% but that's not working for other 99.9% agents needed. So in those cases you need something else that the LLM. And so that's why we've been working on you. You cited this hypergraph to bring a sum higher safety and reliability for the agents in production.
I would say the most effective way is for you to limit the AI to find all the information within the team knowledge. And for every piece of information it retrieves, there is a source that users can click on and check the source. So either it's a chat ID or it's a memo. Everything has to come with source.
So you need to understand from the context of the conversation which one is it.
Right.
And sometimes the speech is just. It's an ongoing conversation and it's not very clear. The problem with medicine is that there's no room for error. Hallucinations cost a lot. You cannot make a mistake. Right. So. Oh, but it does well enough. But whatever chat, GPT or Claude does well enough, enough is not good enough.
Zero tolerance, right?
Zero tolerance, zero error tolerance, Yep.
Over time people realize that they need to have some additional capability on top of it to process much more information in real time fashion. If you are only chatting with LLM, if LLM knows it, it's very powerful. Great. But if you chat about a topic with LLM, most of the time it will hallucinate.
Right.
It doesn't know this information, but it will pretend it knows this information and like we call it hallucination from this. So hallucination has been really at least news for the lm like it hasn't been solved even now today for me it's because LM only know a subset of information and whenever you go out of the boundary it it was started to rambling and later everyone is trying to build this AI gallery of saying. Don't say things you don't know but they will refuse to answer your questions.
