How to Measure and Monitor Agent Health | Dreamforce 2025

Okay, quick intro. My name is Mo. I'm a director of products on Agent Health Monitoring and this is Ravi. Over the last year we've been working together on this product and we think that we are actually solving a real problem for many of our customers. I'm sure that you're gonna find it useful today. So we're gonna go over a few different key things today. We will give an overview of Agent Observability and what it is. We will talk about Agent Health Monitoring, the product that we've been excited to share with you guys. We'll go over how we can investigate alerts and issues when things kind of go off the rail. And we also have a product demo to show you today. Okay, so what are some of the questions that comes to mind for any AI leader that's deploying multiple agents in their production? After you deploy these products, these agents into production, you're probably wondering if they're doing what they're supposed to do. How do you make sure that your customer's experience, who are interacting with these agents is actually what you intended to be? And you make sure that when these agents go off the rail, you are aware of what's actually happening to them. How do we make sure that we can minimize this kind of customers, this broken customer experience? At the end of the day, you wanna make sure that your end users are not having a bad experience, and if they do, you can be aware of it and quickly take action. So those are some of the things that we try to address with this product. So let's do that. How do we monitor these agents then? So the answer to that is a product called Agent Force Observability, or observe. Essentially, in this product, we are allowing our customers to experience a complete, unified mission control experience of everything that their agents are doing, from monitoring them, optimizing them and analyzing them. This product allows you to go really deep using an underlying backbone called Session Trace Data Model, which essentially has all the different logs and session traces of all the interactions that customers have with your agents so that you can make sure that you get a good understanding of what's happening at any moment with your agents and your end users, be it internal employees or external customers. So Agent Force Observability is essentially part of a broader product called AgentForce Studio, which is going to GA later this year. You can imagine you've probably seen this slide on many other sessions this week, but essentially when you start building an agent, you start from an idea in the agent builder, you interact with it, you build some Kind of an agent idea from then you test it inside testing center with all kinds of different business scenarios thrown at it. Once it passes all your test cases, you deploy it. And from that point forward you want to monitor it. And that's where agent force observability comes into picture. Observability, as I said, has three different parts. We have agent analytics. This is where you look at any historical, essentially data around interactions with your users. It has all kinds of different metrics to give you an understanding of what has been happening over time. We have agent optimization. This is where you optimize your agents. You go deep into every session, you can get a deeper understanding of every topic, every interaction. And then we have agent health monitoring. This is focused on near real time observability of your agents in production. That's what we wanna focus on today. So agent health monitoring. There are a few different things that comes to mind and why we built this product, one of which is silent failures. One of the first things that we heard from a lot of customers was that when we have agents deployed into production, we don't know if they go off trail. We don't know if my agents are erroring out. My customers are having a bad experience. A lot of times customers were notified about these issues after their customers complain about it. And so that's what we wanna solve with this. We wanna enable you guys with proactive monitoring so that as soon as things go off track, you get an alert, you can take action. Also, it's important to know that every business is different. So we are launching with a subset of these metrics that we think are important metrics. We call them the health metrics. But over time we wanna expand this to customizable metrics and tags. Any of our customers, any of our businesses that are using this product will have their own set of metrics that you want to be kind of tracking. And so this allows you to do that as well. So there are three or four different capabilities that we are launching with later this year. We have the monitoring health metrics. Those are the three metrics that I want to cover today. There is agent error rates, escalation rates, and what was the other one? Latency. And there's a lot of research that actually supports that. These three metrics are key to some success of your agents. Even a small improvement in latency can improve your customer's experience. So this is critical for us to make sure that you can monitor these in near real time. We do an aggregated five minutes and we alert all of you all of the customers on a minute by minute basis, you can investigate any spikes that happens through a protocol Scale center, which Ravi is gonna cover in a minute. And also at lunch we are supporting data exports for for more experts out here. If you have an existing monitoring solution like Datadog or Splunk, using our export feature, you can take this data, get it into your own monitoring solutions and basically keep your existing processes in place. Let's take a quick look at the user journey, what it looks like and what we imagine this be used. So the customer experience starts from Agent Analytics. This is where I say, as I said, this is the historical kind of log of everything that has been happening with your agents with lots of different metrics to help you get a better understanding. Then you can hand pick a certain set of metrics that you think are important to your business and pull them into the monitoring dashboard to get near real time alerting and experience around that. From there you can create alerts, you can get notified about it and once you're notified you can take action. And that's what we're gonna cover in the demo later. Okay Ravi, do you want to talk about Scale Center?

Sure. Thanks Mo for the great overview. Before I start, can I get a quick show of hands? How many of you have heard of scalability products? Scale Center, Apex Guru, Scale Test. Awesome. We have a few hands. So Scale center is a generally available product. It provides self server near real time access to your org performance and scale metrics, helping you build and deploy scalable highly performant applications on the Salesforce platform. So on the question you may ask is how is Scale center relevant for agent force health monitoring? And the answer is a lot. So we have certain core capabilities in Scale center and we are trying to bear and customize some of these experiences and features specifically to help you optimize your agents. So Scale center we have org observability so you can get scale and performance metrics for the key org health integrators. So when you get an alert from AgentForce Health Monitoring, you can run deep dive investigations. So we have multiple deep dive investigation addressing several areas like ROLOCs or concurrent apex or Flows or integrations. And when an agent basically executes some of these actions, some of the most common use cases are flows and Apex. So we have a lot of rich insights to deliver to help you optimize and deploy these agents. And one of the key features things you have to also consider is once you build and deploy these agents in production, they not only have to work for Your day to day use case, but but also for your business critical events. As an example, let's say you are a retail customer. So Black Friday Thanksgiving sale could be a business peak event where you expect a lot of end user interaction, lot of engagements with the agents, lot of chatter. So the agent could be very busy at that time. How do you make sure that the agent is performing up to your expectations for that business peak event? So things like that Scale center can come in very handy because if you from the get go, if you build and deploy these agents, you root cause and address the scale bottlenecks. So you will have pretty high confidence that on that particular business day your agent will run up to your expectations. We also have a lot of scale insights specifically around how to optimize your Apex, your flows, your database. So these are foundational capabilities for your org. So the idea is if you have a solid platform, a solid reliable, healthy implementation, then the agents that you build for your use case should also function much better. So Scale center is generally available for all editions now. So all full copy production and full copy sandbox orgs will have access to a Scale center. So all AgentForce customers can have access to Scale Center. We also are announcing the pilot is already currently available so we're planning to ga it in the next release next slide. So I want to spend a little bit an overview of. We're starting off with Flows and Apex investigations because those are the primary use cases for agentic actions. So as an example, let's say you get a spike on errors or latencies or escalation rate from Agent Force Health Monitoring. You can run deep dive investigations for those specific size for those spikes within Scale center to root cause those issues. So if you're running an Apex investigation which we will filter it down by the specific agent types and also the agents so you can go really deep into a specific use case and it will tell you if you're invoking Apex, what are your slowest Apex classes, what are the key Apex errors and then how and recommendations to resolve those issues. And then specifically it will even go to the line of code. So if you're running an Apex class and it has an anti pattern which Scale center already has identified and detected through Apex Guru, we'll surface those recommendations to you as well. So you can optimize those agentic actions which are calling these Apex classes, calling this Apex code so you can optimize these. So when you're running these agentic actions at scale for that peak window, it should run really great and we are announcing the beta for these investigations within Scale center in the upcoming release and then we'll ga it in the following release. Over to you Mo.

Awesome. All right, let's take a quick look at a product demo today. As I said, the product is currently in pilot and we are planning on launching what you're seeing here in a couple of months as a beta release. So if you guys are interested, definitely get in touch with us, we can get you on the beta as well. So let's take a quick look here. So there are two user flows that I'd like to cover in this demo. One is the creation and what basically the dashboard looks like the creation of the alerts and then ultimately what happens after an alert is triggered. So what we are looking at is Agent Studio, Agent 4 Studio and on the left hand side you can see there are multiple different tabs. That's the observed product that I just mentioned earlier. And underneath it there's the analytics, Health monitoring and optimization. So now here we are focused on the health monitoring tab. You can see there are the three metrics that we are launching with. Escalation rate, average latency and agent error rate. Again these three metrics are what we are launching with. But over time you want to make sure that this is basically expandable to any metric that you care about for your business out of the box. All of these are going to be five minute aggregated metrics, meaning that they're looking back five minutes and every dot on the chart is basically a five minute aggregated data points and you are getting alerts for these every minute. So essentially as soon as things go off rail, you'll be notified about it. So here let's take a quick look at how you create an alert. You can pick the metric that you want, there's a threshold you can set. In this case we are setting a 10% threshold for agent error rates. You can decide whether you wanna be notified for a specific agent or a category of agents as well. So all of your service agents or all of your sales agents, et cetera, as well as is supported. So again we're gonna select customer service in this case and that's it. Very quick and easy way to create an alert. You can manage all of your existing alerts up here as well. And if you wanna edit them, delete them. That's pretty easy. So what happens after you get an alert out of the box? We are supporting email. This is actually pretty powerful because if you have for example existing pagerduty set up, you can hook it up to this email. And that already works very easily. But even if you don't have that, essentially you monitor this email address. You get an email that says one of your agents is basically behaving in a certain way based on the alert that we just set. Clicking on that will take you to the dashboard. And then on the top right, we have notifications where it basically summarizes any existing alerts that we have. And clicking on those will take you to the investigation page. So this is where you're getting really all of the power from session trace to show you all kinds of different data that helps you understand what went, you know, what's happening behind the scenes. So in this example, we are seeing distribution across different channels, different agents, agent types. We can look at even a specific step and see what are the errors that are happening at a specific step in a time window that this alert was fired. You can even go deeper and look into scale center through this button on the top right. And then that allows you to see essentially any sort of like, behind the scenes apex and flows logs to understand what are the bottlenecks, what's causing these errors or latencies, and essentially optimize your agents from there. So I think this is gonna be a very powerful product for anyone who is building agents and is responsible to monitor these agents in production. That is the end of the demo, I think. We don't have any other slides, but yeah. Thank you so much for joining the session today. We will be sticking around here. If you have any questions, you can come up and we will help you answer them. Thank you.