A New Agentic AI Metric: Containment Rate

Live from Schenectady, New York, it's the Metrics Brothers, Growth and CAC.

And I'm Growth, better known as Ray Reich, founder and CEO of Benchmarkit.

And I'm CAC, better known as Dave Kellogg, EIR at Balderson Capital, independent consultant, and the author of Kellogg.

And together we are the Metrics Brothers.

And we go together like communism and containment.

Yeah, I know I said please no more food metaphors, but political ones?

Well, Ray, I'm talking a little bit more about history than politics here. I mean containment in the sense of geopolitical foreign strategy, specifically trying to contain communism during the Cold War. You know, the tin soldiers and Nixon coming sort of era stuff.

Well, that's topical because I know some people in New York City right now that may still want to contain that concept.

Well, well, I know you live in New York, Ray, but, but today's episode, we're going to talk about containment, not in that sense. That was just the intro line. We're going to talk about it in the agentic AI sense, specifically when it comes to agents such as chatbots and use cases like customer support. We're going to talk about containment rate and what that means.

This is really good because in my continued evaluation analysis of ROI and metrics for AI investments, containment rates is one of my new favorites, though it comes from an old category.

Okay, well, before we dive in, Ray, let's hear a word from our sponsor.

The Metrics Brothers is proudly sponsored by Benchmarkit, the B2B AI and software industry's largest, broadest, and most contextualized benchmarking index. Dave Kellogg and Ray Groth Wright have put the Metrics Brothers podcast on hiatus for remainder of 2026. All episodes we have ever recorded are still available on your favorite podcasting app. You can reach Dave at Kellogg and Ray on LinkedIn at Ray Reich. Thank you so much for listening to the Metrics Brothers podcast.

Okay, Dave. Well, I thought since we're talking about a new agentic AI metric, containment rate, I'd start with some definitions. And that is the difference between an AI agent and agentic AI. And from my perspective, an AI agent is a specific instantiation of a digital worker built on agentic AI principles. It has a defined role, like maybe there's a support agent or a sales development agent. And it uses the broader holistic agentic AI stack. And then we have the concept of what is agentic AI. And to me, that refers to a broader kind of systems design philosophy and thinking where AI systems exhibit agency, meaning they can perceive, decide, act in pursuit of goals. They operate with a degree of autonomy, not just reactive outputs. And they can integrate multiple capabilities, reasoning, planning, feedback, tools, and the pursuit of continuous improving. So that's my definition of the two, Dave.

Yeah, I think it's good to start with definitions. And simply put, I think agentic AI is kind of the capability, meaning systems that can act with agency. And AI agents are really the product or implementation of agentic design philosophy, if you will. The question I have, though, it's not really a question, but one of the things that you hear on Twitter and such, or X, is somebody saying, hey, I replaced my marketing team with 9 agents. And I just find it begs a question of, in the agentic future, are people going to sell kind of virtual humans, like a marketing agent named Mark, right? Or as Qualified does, an SDR named Piper, right? Or are they going to look more at the workflows? Are they going to decompose the work done by humans into different workflows and then kind of agentize them? For example, a data enrichment agent, a copywriting agent, a campaign execution agent, an SEO bidding agent, and a web chat agent, right? All presumably orchestrated by some agentic platform. So, I think what's going on is when people think about the agentic future, I personally believe we're going to lean more towards the workflows than the virtual humans. But I also think that's a source of confusion of kind of what granularity are these agents.

Yeah.

Well, for me, I know I think more of it as an automated workflow. That's the agent versus I have a support agent that does the job of 9 people.

Or a virtual podcast partner named Ray. Okay. So let's go on. So now that we've hopefully clarified agents versus agentic AI, let's talk about containment rate.

Okay.

So, so quick definition of containment rate. It's the, the The percentage of requests or tasks that an AI agent completes end-to-end without having to hand it over or a human takes over. Now, this is a little bit broader than the term deflection rate that's been used in chatbots, which is, hey, how many inbound calls do I deflect from having to talk to a human? So this is truly a containment rate. And containment rate, it really started back in the days of interactive voice response, IVR. When they were saying, okay, how many customer calls can we handle without requiring a live agent? Then the chatbots took it over. And then even virtual agents from Google redefined it as, you know, how many steps in the menu were completed during a successful conversation. And now we're talking about containment in the agentic AI context. Did it stay in the bot and did we solve the issue?

Yeah. And I'm not sure the second one is always addressed. We'll come back to that in a minute. But did it stay in the bot is definitely counted. I will tell you my point of view on this and on deflection in particular, because as you know, I ran the Service Cloud at Salesforce. And deflection, I've never liked it because basically, as the customer, on the customer point of view on this, I don't like being deflected, right? Deflection is, can we prevent a case from getting generated? Can we prevent a ticket from getting generated that a human agent has to go handle. And it did result in a lot of people just giving up, right? And they didn't get what they wanted. So, I'm not sure customers like being deflected. Sometimes they do, right? You can flip that coin over and call it self-service. Oh, great. I went to log a case. It asked me about my problem. It dropped me in a knowledge base. It popped up a paragraph on how to solve my problem. I was deflected in that case, but happily so, right? Because my problem was solved. But a lot of times, at least me these days, it's so impossible to get through to a company, I end up giving up, right? And that would count as a successful deflection, right? Like, oh, he never made it through the IVR, or he never made it through the knowledge base, right? So, I think hopefully, we'll see if it can— like, I would argue, Ray, that customers don't like being deflected. We'll see if customers do like being contained.

Yeah.

No, they do like to be satisfied though, Dave.

True. So true.

Well, that brings up, you know, just real quick, since we are the Metrics Brothers, let's talk about the containment rate calculation. And at a very high level, you take those completed tasks by an AI agent with no human intervention, and you divide it by total eligible tasks that it could have handled. Let me give you an example. We have an inbound customer support call. We had 100,000 of those last month, and 70,000 of those were actually closed or solved without human intervention. So you take that 70,000, divide it by 100,000, and you'd have a 70% containment rate. And by the way, I did use the 70% because I looked at some current benchmarks for customer support calls being solved by AI agents, and it's 65 to 75% is the kind of benchmark. today. So that's containment rate calculation.

Okay. So, and let's just be clear. I don't think containment rate necessarily applies. Actually, I'm sure it doesn't apply to all types of agents. I think it applies to agents where we're trying to complete a task that was formerly done by a human and that can be escalated to a human if the bot fails. Basically, completing tasks like, for example, customer service support inquiries. SDRs taking inquiries and turning them into MQLs. IT help desk, hey, my computer won't start. HR help desk, hey, my 401 deduction is wrong. Or back office agents such as, hey, I'm trying to match purchase orders to invoices. Those might have a containment rate, right? Because in all those examples, I think the implicit thing behind containment rate is that some of these aren't contained, right? And they are escalated. And one of the things One of my quibbles with this metric, and I raise all this because one of my quibbles is, you know, as a customer, sometimes getting to the human quickly is good. I don't view it as a failure. If the agent asks me a question and escalates me, it will drop the containment rate, I believe. So this could be a flaw in the metric, but it might actually maximize my satisfaction. Thoughts?

Ooh, that's next level. It depends on how I'm being measured, Dave. If I'm being measured on containment rate, I'm happy with my current definition, but I do think you're right. It's really something that should be built into the holistic process of how do I ensure I'm meeting my customers' needs and expectations.

Yeah. And people tried this with deflection rates. When they threw a knowledge base article at you, they'd give you a thumb up or thumb down on it to say, hey, did this resolve your issue? But I'm not sure, right? That was their attempt at trying to figure out if the deflection resulted in success, right? So clearly, if you said no, you could assume that it didn't help. But I think with containment, again, the definition— look, Ray, the metric itself is quite simple, right? The tasks completed by the bot divided by tasks attempted by the bot, by the agent. I always think chatbot, sorry. But tasks completed by the agent versus tasks, total tasks handed out, the difference being those escalated to a human, right? Presumably, that's what happened. The agent goes, I can't help you anymore. I need to put you through to a human. And the question is, and I don't think we're going to fix the metric, Ray. I just think it's important in interpretation that we can't be black and white about if the bot resolved it, the customer was happy. And if it didn't, they weren't. I can think of both counter cases. One where the bot escalated me quickly to a human that solved my problem. I'm happy as a clam because I don't have to get lost in some— what are they called? Oh, the voice dialing thing.

IVR.

I'm not dead in some IVR system.

Oh, in the doom loop?

Yeah, yeah, the doom loop. Yeah, which I have broken many telephones there. Or in the counter case where I am contained but not happy, right? So, there's 2 cases. One where I'm escalated and it was a success. Oh, sorry. And 2 where I'm not escalated but it was a failure, right? So, to me, it's just important in interpreting this metric that it takes a little bit more than meets the eye.

You know, Dave, I was thinking about this saying, you may be able to contain Dave Kellogg, but you can't control him.

There you go. There you go.

Well, you know, what I tried to do actually was, because we're hitting on such an important point and definitions become so critical, you know, how do you define this completed, or is it resolved in the containment rate calculation? So I'm gonna— even though resolve can mean different things, to me, Resolved, when we're defining this containment rate, should mean that the user was explicitly satisfied. Maybe even we asked them, was your issue successfully resolved in your opinion? And getting yes could say, okay, it was really resolved. Or it could be something a little less stringent, which is, hey, there was no escalation requested. The session ended without transferring to a live agent. But then maybe you need to back that up with there was a recontact window. So this user did not call back in or reengage within a defined period. Maybe that's 24 hours or 72 hours. So I think it's so critical that we define what completed or resolved means in context of that AI agent's use case.

Totally agree, Ray. And that certainly good definitions lead to better interpretations. The other one, I just have to share this story because I do my own tech support. So, I'm always calling lines and going to self-service portals. But I just canceled my American Express yesterday because they raised the annual fee a lot. And I don't want to clip coupons, which is what you have to do. They take, you know, $900 of your dollars and then you have to go build a spreadsheet to make sure you get most of it back if you're lucky. So, I didn't want to do that. So, I just wanted to downgrade to a green card.

And this is—

and the answer was you can't. Uh, that I can't downgrade it. You have to cancel this one and start from new to, to apply for a no-fee or low-fee green card. Um, and the issue was when they did the post-CSAT survey on me, they're like, well, basically, how did the agent do? And the agent was fine. I'm canceling my account because your policies are idiotic, right? But the agent was fine. It's another problem with rating either human agents or agentic agents. Like, hey, you did your job. The problem wasn't your agent. The problem is your— your policy. And they're very bad at capturing that. I guess the only way they do it now is they probably have to log transcripts and do text analysis, right? But it's crazy.

Yeah.

Well, this episode is not about American Express, but I have that same card. And every time I walk into one of their airport lounges, in my mind, I'm like, okay, there's $50 off my $900.

There you go. There's an article in the New York Times about this, where there are people who are literally building spreadsheets to track this stuff. And it's like, hey, my, my, my, no, I'm not going to do that. But anyway, I digress. But these are important points in all these metrics, which is we got to be careful. What does completed mean? What does resolved mean? And are we rating the agent or the interaction or actually the company's policy? Like, they're different.

Yeah.

Well, you know me, the soft benefits that, like, you know, increase efficiency or effectiveness of AI, I'm not a big fan of. I want to get to real ROI. So I kind of took this to the next step. So I took this scenario the containment rate of a customer support call being 70%, and that was 70,000 out of 100,000 being resolved. But then I said, okay, what's the average time if a human did that? What's the fully loaded cost of the human if they were doing that? And then I extrapolated it into a cost savings. So my first step to getting to ROI was at least translating the containment rate based upon the number of total cases contained. into a cost savings. So, I don't think a CFO cares about my containment rate if they don't know how much money it saved me. What say you?

Yeah, look, I think the correct measure on these things, ultimately, you got to translate it to dollars. Like, a containment rate is not a self-evident benefit, right? Just so, so, so you need to convert it to calls avoided times the cost per call, or, you inquiry ticket, whatever you want to call it. So, you could translate that relatively handily to dollars. I think the thing you need to be a little careful of is, hey, if we had just used a self-service deflection portal, how many of those would we have gotten at what cost, right? Because in a typical 3-tier model, you would have— at least a 2-tier model— you would have a deflection portal, then a human agent. So, you're doing some deflection up front. Now, you're effectively substituting that portal with an agent.

Right.

So, but if you want to know the real business advantage, it's not just pretending that it was going to be all humans or agent in front of humans. It's actually the comparator is deflection portal in front of humans. So, if I was CFO, that's what I would be asking about. But yes, I like the idea. You got to translate it to dollars. And just saying we saved this many tickets, it's kind of like fumbling the ball on the 2-yard line because it's not very hard to translate tickets to dollars. But if you don't do it, people aren't going to— Whereas it's not going to resonate with your audience.

Yeah. But Dave, you're being nice here right now because when I first kind of shared this with you on the cost savings, you're like, Ray, that's not an ROI. That's just the cost savings part. How much did you have to spend on that agentic AI or AI agent to get there? So if you're going to do ROI—

Yeah, I was nice there. Sorry. I should quibble. Ray said ROI, and ROI means return on investment. We all know that. But technically, ROI is kind of a get-give ratio, like what did you get versus what you gave, and you measure it in percent. So, if you're going to say ROI is we didn't do this many calls, thanks for playing. If you want to say ROI is we saved this many dollars, most people will let you get away with it. But technically speaking, ROI, like one of the things we both know, Ray, from college physics is like, look at the units. Look at the units, right? And ROI is measured in percent. So, if you're answering an ROI and Anything other than percent, it's probably literally not an ROI. Again, most people will let you get away with it. I would call what you just said is a financial benefit. Dollar cost savings, incremental revenue, incremental cost, incremental margin, right? Those are all financial benefits. Technically, ROI is measured in percent.

Yeah, no, it was a good call out. And then, you know, I am the benchmark brother here. So I thought, okay, I'm going to just at least share some of the current benchmarks for containment rate for some different Use cases like customer support in 2025. And by the way, I had to basically go and look at multiple benchmarks, some from G2, some from Gartner CX Benchmarks, some from vendors like Ada Ultimate and Intercom. So this is really kind of a mashup of all of them. But customer support was in that 60 to 85% range. For an internal case, IT help desk, it was a broad range of benchmarks, 45 to 70%. And then HR self-service, it was 40% to 65%. So very broad ranges. And honestly, Dave, not any really great benchmarks out there by use case or by industry.

Okay. Well, thanks for doing the legwork to try and find some benchmarks. From my brief experience in customer service, those numbers look approximately correct. I think it'll be even trickier because I do believe in the future you're going to have both a deflection portal And an agent working side by side, right, as kind of the frontline defense. Because something like, you know, a password reset, the agent can do it, but you might be able to do it yourself easily in the, you know, just through the knowledge base search portal. So, I think it's going to be those 2 working together as kind of the offensive line for the quarterbacks who are the actual human customer service agents behind. And these deflection rates strike me as reasonable. Or containment rates, sorry, there I go again, running between the 40s and the 80s.

Well, I think it's getting close to time to wrap up this episode. So maybe in summary, I'll kind of give you my perspective on this containment rate. So number one, it is critical to define what resolved or completed is before you calculate your containment rate. I think containment rate is a pretty good way to measure the autonomy of an AI agent. Is it really converting those AI investments into tangible return on investment, i.e., are we closing these cases? Is there a cost savings? And is there a return compared to the investment? I think it's very important to establish the baseline for the time and cost for any process that you think about automating with AI agents. And bottom line, I think containment rate is one good measurement to translate agentic AI promise to a real operating and performance metric.

So let me wrap up with just saying I played the part of Negative Ned during most of the episode talking about all the problems with agents and measuring containment. Let me just remind us that there are some soft benefits of using this style of agent. I don't know what— I don't know how to name the style, but it's a customer service agent. It's an SDR, it's an IT help desk, HR help desk, right? It's this style of agent where you're kind of directly replacing a human who would have taken a call or a case otherwise. And we talked about the containment rate and some of the perils of measuring it. But what we didn't talk about is these things often get you answers, like they can increase customer satisfaction quite a bit, right? Because they make fewer mistakes. They don't sleep. They're available 24/7. They speak multiple languages. Right? So, you're not having to— you're eliminating a massive problem in human call centers is mapping callers to people who speak their language, right, which is a pain in the neck. And there's a lot of benefits to using these styles of agents. So overall, I'm a huge believer in them. I just encourage people to first measure containment rate for sure. Second, to answer my own question, I do believe people would rather be contained than deflected. So, I do think it's a really good idea to use these systems. I don't think that every escalation is a failure. In fact, I think a prompt escalation is a success. I don't know if there's a metric for that. But look, one of the hardest things for a person to do is be able to get to a human quickly these days. And if you can configure your agents to do that, I think your customers will love you. And overall, big believers in this. I'm just trying to encourage a kind of a, don't be myopic in measuring containment rate. Take a balanced view because the real point is not to Resolve the case. It's to make the customer happy. And that's what we're trying to do.

Boy, we're ending a containment metric with the ultimate goal, customer satisfaction. Couldn't say it better, Dave. Thank you so much for today.

Yeah, thanks, Ray. Take care.

The Metrics Brothers AI, SaaS, and More is a production of The Metrics Brothers Growth and CAC and a member of the Benchmarkit Podcast Network. By accessing this podcast, you acknowledge that The Metrics Brothers make no warranty, guarantee, or representation as to the accuracy or sufficiency of the information provided or the humor content of the jokes told. Ray, information, opinions, and recommendations presented are, according to our spouses, probably wrong and provided for information purposes only. This podcast should not be considered professional, or for that matter, unprofessional advice. We disclaim any and all liability for any direct, indirect, indirect, misdirect, incidental, special, ordinary, consequential, inconsequential, or other damages arising out of any use of, or God help you relies upon the information presented here. Ray Grothreich is based in New York City and available on X @RayReich.

Dave Kellogg is based in Silicon Valley and available on X @Kellogg.

Schedectady, which is French for unspellable, is not our actual production location. Thanks for listening.