Deep Dive into Agent Performance and Analytics | Dreamforce 2025

You.

I'd like to start off by introducing myself. My name is Ishmael Chawla and I'm a product marketing manager here at Salesforce focused on our AgentForce product suite. I have been at Salesforce for about four years and I have spent my time working with customers like you as a solutions engineer and now I've transitioned over to a new role. So I'm so excited to bring all the agentforce innovations that we're working on and, and I'm also joined by Itay. So if you'd like to introduce yourself. Itay.

Thank you. Yes, my name is Itay. Hi, nice to meet you all. I'm a product manager in Salesforce AI Platform focusing on observability and optimization and I'm super excited to be here today and to share with you what we've been working on the past few months.

Awesome. So let's set the stage for today. We're actually going to start off by addressing the visibility gap that we have currently with agentic AI. And then we'll go into introducing agent force observability and deep dive into the product. And finally we'll also show you what the real world impact has been with some of the pilot customers that we've been working with and also give you a sneak peek into what's on the road ahead. So with this wave of agentic AI, everyone defaults to thinking about the front facing outcome, which is the deployed AI agent. But when you think about it, deploying agents is really just the beginning. The real challenge comes after the fact. Just like our human teams, our digital workforce also needs to be supervised and monitored and continuously improved. And that's not an easy task. Monitoring and supervising agents is actually a team sport because of the sheer magnitude of impact it can have on your customers. So let's think about it. When once the agent goes live, we celebrate. But then we have to know, how is my agent responding in the real world? Where are my conversations breaking down, what are my customers really asking? And most importantly, how do I optimize the performance of my agents to continuously improve over time? And so let me give you a quick example. I was traveling to Germany last month and had to get a car for my road trip. Now my 2 year old nephew decided to join us at the last moment and so I contacted the car company to add a car seat to my reservation. And so when I started to have my conversation, I was actually chatting with an AI agent on the other side. Now my conversation started on such a high note, the agent quickly jumped into action and was able to validate my reservation. And then it proceeded to add the car seat to my reservation. And I was so incredibly happy with my experience. What would have taken me about 10 minutes with a human agent has now only taken me about 30 seconds from start to finish. Because I've had such a smooth experience so far, I decide to continue my conversation with the agent and talk to it about what I'm going to need to be crossing borders into Austria. Now this is where the agent starts to give me some not so accurate responses. And so I ask it a more pinpointed question. Thank you. Thinking that that might help, but it doesn't help either. And this is when the conversation starts to go really south. It gives me a completely unrelated answer about tires. And so I'm thinking, what? So now I'm in a little bit of a fix because I can't really determine whether my experience was good or whether it was bad because I asked the agent to perform a couple of complex tasks and it was able to do some of them, but the rest it failed. And so this leaves me in a little bit of a complicated situation where I can't really come to a determination whether my agent performed well and was in scope. Now, we've worked with a bunch of customers over the past couple years as we've developed the solution and they have resoundingly told us the same thing, that the worst thing that can happen is not that my agent is not performing well or how it's supposed to, it's the fact that I have no idea why. And that is simply because we can't improve what we can't see. So the question becomes, how do we measure, manage and optimize our agents? Now, with the agent refinement cycle and with the launch of AgentForce360, we decided to address all critical phases of this cycle and within that cycle is a very critical step, which is to observe. And that's because agent builders and business leaders need visibility into real world performance for these agents to make any significant changes to their AI deployments. And they need the questions, the breakdown points and everything that the sessions have to offer and have the tools to act on that insight. And that is exactly where agent force observability comes in. Think of it as a single mission control for all your AI agents for unified visibility, monitoring and optimization. It gives me extremely granular visibility into every interaction so you can see exactly how your agents are responding. Now think of it as getting visibility into these multiple layers of agentic AI, because that Helps get all the information I need to be able to make the agents perform even better. Now, AgentForce observability is delivered as part of AgentForce Studio and it's foundationally built on top of our session tracing data model which captures every logged event within every session down to a single conversation turn. Now that's powerful and what we're trying to achieve here is simple. It's higher accuracy, faster resolution and better customer experiences. And once you have this detailed visibility, that's when you'll build confidence in your AI deployments to scale them across more of your business. Now, I know I said a lot here, there's obviously a lot to digest, so I'm going to pass it over to Itay who's going to give us a deep dive into observability.

Thank you very much, Ishma her so within Agent Force Studio you now have access to Agent Force Observability. This is the home for improving and monitoring your AI agents. It has three main features. Agent analytics, which provides visibility into usage and adoption metrics. Here you can measure escalation rates, deflection rates and other key business KPIs. Agent optimization helps you drill down into an individual agent to better understand performance and and discover opportunities for improvement. Agent Health Monitoring provides near real time visibility into agent uptime, escalation, frequency response, latency. All of these features are being powered by the session tracing data model which is the foundation for the Agent Force data and being stored in your data. 360. So what is the session tracing data model? Let's talk a bit about that. Every single interaction between a customer and an agent is being logged and stored in the session tracing data model. A session is representing the complete customer journey and can contain a single or multiple interactions. Now an interaction is not merely an exchange of messages between the agent and the user. It consists of the logic behind that as well. So when a customer asks something, the agent starts having steps. A step might be invoking a topic, triggering an action, or executing prompts or flows, or whatever agentforce is capable of until it's crafting the right response and by that, completing the interaction. When we look at this, this looks pretty complicated, I know, but this is the granularity needed to power up the observability experience. Now you can learn a lot from reading through these logs, but this is not scalable because what if you have thousands of conversations? What if your agent is interacting with thousands of customers every day? So let's imagine this. If we have thousands of conversations every day, how do we begin to understand performance at this scale, how do we listen to what customers have to say and what their pain points are? How do we move from reading through thousands of session logs into having a clear, actionable insights view? When we started building observability, these were the questions that we had in mind. How are we able to offer you this flashlight so you can focus on what really matter? So let's go back to Ishmael's example with the car rental agent. The session had many requests and responses with three main intents throughout the session. Adding a car seat, border crossing info, and weather forecast. Now we know for a fact by looking at our pilot customers data that customers tend to have more than one intent in a session. They just ask multiple questions that aren't necessarily related to one another. So when we try to think of how are we evaluating the session, how do we bring value to our customers? We can't really rely on the session as a whole. Because if we look at this example, and this is a common example, we can't really say if the session was good or not. The first part was good. The second and last part left the customer frustrated. So we understand that we can't rely on a session as an evaluation. So what do we do? We break down the session into moments. A clear, granular, easy to track unit that has a single intent. It might have multiple interactions in between, but it has a single intent. A summary of the request and a summary of the response. And then we use LLM to generate a quality score and a reasoning so you don't need to read through the entire transcript. You can simply look at this and immediately understand how the agent is performing. Now that's powerful and that's the granularity that we need to power up observability. And what's even more powerful is because we understand that customers tend to ask the same thing in multiple different ways. So when we look at all of these thousands of conversations, we can group those similar intents together. And when we do so, we can apply this quality score logic and associate it with the intent cluster. What this brings is that flashlight that helps you understand what is working well, what's not working so well, and maybe even find something that your customers are asking that you didn't even know. So let's hear a bit about what our pilot customers had to say about observability.

Thanks, Hitai. Now we want to talk to you about some of the pilot customers that we've worked with who have been able to leverage observability for various different efforts in their organizations. One of them was Smartsheet. They've been an early adopter of agentic AI and were also one of our pilot customers. They have been laser focused on optimizing their agentic conversations for higher customer retention. And before observability they were going and manually extracting all the agentic information and all the conversations that customers were having with AI agents to be able to understand what what is going wrong in those conversations. But once they had observability, they were able to eliminate all that manual extraction efforts to then utilize some of the capabilities that observability offered to them. For example, grouping customer intentions and also identifying customer gaps to then optimize their agentic performance. And then we also worked with Fisher and Paykel who've been very focused on decreasing the service costs that they have by expanding the customer, by expanding the AI agentic adoption of these agents across their organization. And they were able to do that by eliminating the failures that the agents were having based on the observability capabilities and also the KPIs that observability was able to offer them. And then finally we have indeed who has been focused on reducing the job search time for candidates by 50%. And that is only possible if they use agentic AI agents for using some of their human agents to solve those problems that the candidates have. And so that is only possible by building foundational trust in these AI agents so they can expand that across their organization. So overall what these customers had used before observability was, was some of the monitoring tools which are very reactive, they're not proactive. And then they also use dashboards which are very surface level visibility. And then we have testing tools that is testing the same agent a number of times without really telling us how we can optimize these agents. But then we have agent force observability which closes the loop across all these disparate solutions to then give us the three pillars that we've been able to establish with our customers, which is driving continuous agent improvement, measuring behavior and impact, and building trust through real visibility. Now we've talked about what agent force observability has had to offer as of now, but we're working on so many other things. And so I'd like to give it back to ITAY to walk us through some of the roadmap features.

Yeah, we have so much more coming up your way. One of the most exciting features that we're working on is the observability agent. An intelligent on demand analysis assistant it helps you query your agent performance, getting immediate responses to your most pressing business questions. Think of it as a proactive assistant for your observability data, one that bridges the gap between action and end site. Another exciting capability we're working on is Edge Enforce Evals. Agentforce Evals allows a flexible framework to help you define exactly which insights you want to extract from your AI agents. If it's Sentiment Score, or if it's a competitor mentioned, or any other custom KPIs you're thinking of, you'll be able to write a prompt, logic, or an expression, and run the metrics that matter most to your business. Now, this is very powerful. We're currently working on this. We're very excited to hand it over to you to have it in your hands, and we have much more coming up. So we'd like to thank you so much for coming and listening, Watching.