Beyond the Hype: How We Measure AI at Salesforce

Hi, I'm Dave Levinson. Again, as I mentioned, I run a data and analytics team at Salesforce. We do customer zero measurement. So that means how Salesforce uses our own products internally. And so about over a little over a year ago, our team was asked to measure agents at Salesforce. And so that's kind of how this came about. This is Blaine Myers. He's our principal engineer, and he'll be taking you through most of the work today. All right, let's start with the challenge that we had in front of us. So Salesforce, very large organization. We are now well over 300 agents internally. So the scale grew very quickly. There was a gold rush era of building agents for everything. And so we have that agents across 20 different internal orgs. So an incredibly fast-growing portfolio, but I want you to know we started just like all of you on that journey. First few agents, how do we even measure how they're being used? Who's using them? What are they being used for? And we really wanted to build trusted agentic experiences. So how do you know what an employee is saying to the agent? What is the agent saying back? And so what you'll see today is the tools we built in Tableau to do that, to understand these agentic conversations. We had different users. We have executive leaders that want to know at the highest levels, What is the business value? You'll hear about that today. We had product managers that want to add new features. What do we need to build? I'll give you an example. We built a sales agent. Immediately, sellers started asking questions that we didn't anticipate. How do I get a deal approved? You'll see that today. And then agent builders, the technical engineers that want to diagnose problems with the issue or with the agents.

Good morning, Data Fam.

All right. Next up, so this one's really important to me, and I want to just call out Pay special attention today to the visual styling, the colors. I always say if you have a favorite artist, you know their work without having to see their name. If you have a favorite author, you know their writing style. And so we try to build that here. And so we make a lot of intentional decisions. Curation is really important. We think about the museum curator who cares about the light, the colors, where the art is placed, any of that. We try to really build a connected experience. You'll feel that across our portfolio. We want you to know it's Customer Zero measurement without having to see our name on it. And then building that trusted brand. And so I hope you see that cohesive story across the work today. So with that, I'll pass it to Blaine.

All right, thanks, DL. You guys able to hear me? Yeah, okay. So hey everyone, with the challenge kind of framed up for you all, we wanted to just spend some time to talk about the measurement system. And when we talk about measuring agents at Salesforce, it really revolves around these 4 core pillars. The first being engagement, all about how employees are interacting with agents, how deeply do they interact with agents? Do they stay surface level or do they actually get into in-depth content? The adoption, the retention. From there, we move on to efficacy, and that's really all about understanding agent response quality and task set—

Set.

Success, so you'll see a little bit of that in our work we'll show in just a second. Performance, we want our agents to be fast and responsive. And then of course satisfaction, trying to click into the user experience and understand their sentiment. And we feel like if we can nail these four pillars, it starts to lead into a question of business value, which is really the hot topic at Salesforce, right? How do we understand the value that agents are returning back to us? We think about that in terms of time savings and operational impact, and we'll show you how we measure that in our dashboards. So real quick, last slide before we jump into the demo, we wanted to just kind of frame up what powers the system, some of the tools and technology behind the scenes. It really starts with a data foundation. All of the observability data from AgentForce, raw agent logs, conversation transcripts are processed, scrubbed for any sensitive information, and then unified in our data foundation. From there, we have the measurement intelligence layer where our small but mighty analytics team is leveraging techniques like LLM as a judge to help us scale across the large portfolio. I'll call out where you see the LLM as a judge and AI-augmented evaluation that we use to, to measure agents. And then of course, the experience layer. We're at TC. This is my favorite part. You'll see how we're using the embedding API to actually embed Tableau visuals within our own custom web app. We actually also are using Tableau MCP to perform weekly narratives for our users and some custom visualizations. So with that, we're gonna dive into a demo. All right, so the first place we'll start here with our demo is what we call the agent rubric. And this is really the starting point for any one of our personas. This page brings together metrics across those core pillars. So if I scroll down the page, you'll see engagement, efficacy, performance, and satisfaction, all those pillars I just mentioned. And how this works, it's really agent-centric. So you choose your agent, it updates weekly, and it's one place to see all your metrics and get a sense for how your agent's performing. I mentioned the Tableau MCP. So one way we're putting that into action behind the scenes of all these visuals are published certified data sources on our Tableau Cloud Server. What we've done here at the top with the AI curated performance narrative is we have an agent that's specifically trained in our measurement framework, which goes out and queries every week all of the certified data sources, runs analysis, and returns concise performance narratives for every agent in the portfolio. So you can see a one-sentence structure on what's going well in the current week for this agent, and then another sentence around what may need attention, what metrics are trending in the wrong direction.

Mm-hmm.

And then on hover, I get more detail around where the agent's pulling those metrics from. I can go down the page and sort of just validate that output. And I think this has been a huge value add for those users that maybe don't have the time to spend digging through the charts, doing their own analysis, dissecting the trends, and coming up with their own narrative. We're now just kind of curating that right at the top of the page. Beyond that, I want to spend some time around how this page is structured, because like DL mentioned, we made a lot of intentional decisions around our design. So if we look within the engagement pillar here, you'll see that we're positioning this and really bringing one metric at a time into focus. So under engagement, I can cycle through my MAU, my weekly acquisition rate, engagement depth rate, and all the other metrics that we have in this pillar. And then from there, we kind of tell this complete story around the given metric. So we start with where that metric's at currently for the selected agent. And another important thing that we added was a detailed definition right visible on the page. We're dealing with a lot of novel metrics to the business, you know, metrics they've never seen before. So it was really important to have a clear definition right there for the user. As we move right, we have that extended, extended time series to help us understand if this week-over-week trend is something that's persistent or something new. And then lastly, I wanna just spend some time talking about this scatterplot, 'cause it's really my favorite part of this visual. Um, when you understand how a metric's performing for a single agent, the kind of next logical question is, well, how does that compare to other agents in our portfolio, agents that we use at Salesforce? So what we've done here is we take the selected metric on the x-axis and then the week-over-week change on the y-axis, and we plot all agents in our portfolio. And our selected agent is bolded here, but then we can kind of quickly compare against other agents. So I love that comparative view that it provides. And then another important thing that the scatter plot does is you'll see this shaded area kind of calling out what direction is improving for the given metric. And so for all of our engagement metrics, these are metrics we want to grow, grow, grow, build as high as possible. So improvement looks like an upward trend. So we have a shaded green area above. But down— when we move down to the efficacy section, we look at something like negative response rate. That's actually something we want to whittle down as low as possible for our agents to be healthy. And so this scatterplot just sort of reinforces that directionality, highlighting that a reduction in this metric is what improvement looks like. So just kind of help, you know, ground our users in these metrics that they're seeing for, you know, the first time. Okay, from here, DL mentioned the connected experience that we had to build. This is a multi-layered, you know, dashboard system. We start with the rubric, which is really that summary layer, but then we want to dive deeper into any one of these areas. And so what I can do here in the engagement section is view more detail. You'll see a slide over to the engagement section of this app. And open up our key metrics dashboard. And I know this dashboard holds a special place in DL and mine's heart because it was actually the first one we built on this journey. You heard DL talk about the gold rush era. Everyone had their own use case. Everyone wanted to build their own agent. So many questions from the business around who was building agents, how are we using agents, and which agents are driving adoption. And so we designed this dashboard really to answer those, three core questions. A couple things that I want to call out on this page. Given that it was so early on in our journey, there was, again, a lot of questions from the business. So we wanted to build something that was really flexible. A couple ways we did that, I just expanded the filter set here, which tied all of our usage data to our employee demographics data. And this was really interesting to look at specific industries, job families, OUs, to see how they were specifically adopting agents, how they were using agents. [FOREIGN LANGUAGE] So those filters together with the ability to explore any one metric here really made for a super flexible, dynamic dashboard experience. So you see here, I, I explore users and that opens up a dedicated pane where we can kind of dissect this metric from a few different angles. I can see my week over week retention of users, and then I can see which agents are driving the highest user rate. I can even drill that down by topic and so on. And so this together with the filters really made for a flexible tool for the business. Okay, so that's our key metrics view, a deep dive into the engagement pillar. The next thing we'll look at as we navigate back to the rubric is we'll double-click into the efficacy section, which again is focused on response quality, quality of the agent, successfulness of the agent. So I'll scroll down. In our efficacy section, some of the metrics that we feature are negative response rate, And I'll show you in just a second how we calculate that and, and what it looks like. Reprompt rate. I'm sure a lot of you have used AI. If you've ever prompted the, an agent, you know, with a question, you get a response back that wasn't exactly what you're looking for. Maybe you'll rephrase that prompt slightly. And we actually built a model that signals in within a session when there's back-to-back prompts that meet a certain similarity score.

Mm-hmm.

And we actually take that as a response quality signal from the agent because if you're reprompting an agent, it's likely that the first response you got was, was not what you were looking for. So reprompt rate is a really important metric here. And then beyond that, we have a set of scoring metrics which help us further understand the character of the agent. Things like conciseness score. When the agent responds, is it concise or is it really verbose? How decisive is the agent? How helpful is it? Does it help teach the user or is it just giving facts and, you know, details back without really understanding the concepts? So these are the core metrics behind the efficacy section. I'm going to go ahead and drill into the efficacy section now. We're going to load up the response quality dashboard. And this is a key dashboard. This is really core to what D.L. talked about, building the trusted agentic experience. And we do that by exposing some of the actual agent responses here. So, so to our agent builders, our product managers, even our leadership, show specific examples where the agent has fallen short in its response quality. A couple decisions that we made on this dashboard. One, it's built on our 3-tier classification system, and this was intentionally simple for our business users to grasp easily. We use LLM as a judge and prompt-based evaluations to scale across thousands and thousands of agent responses and make a judgment on if a response was negative, if it was missing info and potentially the agent has a knowledge gap, or if it was a positive response. And as you look at this subfield here, you can kind of get a better understanding for what a negative response looks like. Maybe the agent hit access limitations and couldn't, answer the question, maybe it gave incorrect information. Those are some examples. But then further from there, we really wanted to hone in on the negative here and potentially expose opportunities to improve the agent quality. And so everything you see here is just measuring the negative response rate across, across all agent responses. We have our big metric here that aggregates up over the entire timeframe for the agent. And then our time series view, you'll notice that it buckets into 2-week increments. And that was an intentional decision around the design of this specific chart. We knew that negative response rate was a highly volatile metric, something that could fluctuate drastically for a given agent day to day or even week to week. So by bucketing this metric into 2-week increments, showing the range of negative responses over that 2-week period and then also the median, it kind of gives a more stable, reliable trend for our users to interpret. And how we coach users to kind of understand this chart is you're looking for an agent to not only decrease the percent of negative responses, but then also have a really tight range here within the 2-week increment. And that indicates that your agent is not only positively responding and successful, but it also is performing consistently. So that's the top portion, kind of our simplified 3-tier classification. And then I mentioned those characteristic fields, things like answer relevancy, decisiveness, and helpfulness. For our agent builders and PMs to really get root cause analysis, it's helpful for them to interpret these metrics at the topic level. And so what we see here is for answer relevancy, every response is scored. And then we have the topics within this agent here, and I can see which topics are scored the lowest for answer relevancy. And when I click into that, I get direct examples here of agent responses that were lowly scored for answer relevancy. And so again, to build that trust and reliability, give our agent builders some direction, we actually expose some of those underlying agent responses. All right, so that's our response quality dashboard. The last view we wanna share with you today is around business value. So when you think about business value of agents, over 300 agents, many different types, some are helping us sell, some are helping resolve cases autonomously. When you think about business value of those, that's more straightforward, you know, case resolution rate or ACV uplift, for example. But what about those agents that internal employees use to search articles, book meetings, take notes, build goals, how do you measure the value of those agents? Well, our methodology is really the story here. We look at every single agent and we inspect those actions that it's capable of taking. Each action for an agent is scored for complexity. And so think of a basic action like maybe booking a meeting with a colleague.

Mm-hmm.

Pretty low complexity, maybe saves you 30 seconds every time you call it and an agent performs that for you. So each time that action is fired, we assume 30 seconds of time savings. But then what about an agent that takes a multi-step workflow and automates that for you, like creating a group management process or taking notes and then sharing in a Slack channel? Well, that's a higher complexity action that would come with a higher complexity score and then a higher assumed time savings. So by looking at complexity, complexity of the actions within an agent and then the usage of all of the different actions an agent's capable of, we start to paint a picture of time savings and business value. Our agents up in the top right are saving us the most time at the company. Agents in the bottom right have high usage, but their capabilities are more basic. And so it's not, you know, assumed to save us much time. And then we don't have any in this example, but if an agent fell up here, it would be a highly complex agent that's maybe used by a small group of people. Mm-hmm. People. So all shapes and forms measured here. And I think this is a really exciting view for our users internally. They kind of treat it like a leaderboard. They come back from time to time to see where their agent's moving and how it's improving. All right, so that's business value. I'm going to go ahead and throw it back to DL now, and he's going to talk a little bit about the future of this work.

Yes, we really showed you our journey. And again, starting just with simple adoption, who's using the agent, what are they using the agents for, than building the trust in the agents. But I wanted to spend some time today of like where our team's focused next. And it's twofold. First is this kind of chemistry view. So you'll see the different measures. I'll give you an example. Let's say an agent builder wants to dial up the conciseness. They want much more concise answers. That might have an impact on the helpfulness that users are feeling about the agent. And so we're really starting to look at what we call the chemistry of agents, so to speak, and how metrics are moving with each other or against each other. So this is an area we're looking at. And then really towards diagnostics. So everyone's busy, everyone's got a business like us that, you know, they don't always have time to go in and dig in and dig through dashboards to find. So you saw we're trying to bring those insights onto the homepage of the rubric. We're also bringing, these are topics, I believe they're now called sub-actions where—

Sub-actions.

An agent builder can come right in and see immediately, okay, there's something going on with this agent or these topics that I need to look into further. So, you know, I liken it to going to the doctor and you get a diagnosis and you get a treatment plan. We're really evolving our team to move our work towards actionable insights right from the start. I just look at it, I know immediately what I need to do to improve my agent. And so that's kind of where our team is moving this towards.

Okay.

And then, yep, thanks.

Oh yeah, sorry. So yeah, just wrapping it up real quick here. Key takeaways, you saw what the challenge of measuring AI at scale at Salesforce looks like. We told you a little bit about our measurement system and those core pillars, gave you an idea what powers the system from the data foundation to the experience layer, and then we showed you what it looks like in practice. So please give us feedback. We hope you enjoyed this session and Yeah, if you want to have a conversation, come up and say hi.

Thanks for coming.