What Agentblazers Learned about Agent Orchestration & Control

Joining us now is Imran Rahman, Salesforce Technical Architect at LIV Golf, to show us how he built an agent that is fundamentally changing how they support their customers. Imran, thank you so much for being here.

Thank you, Leah, for having me.

Can you tell us a little bit about LIV Golf and what you're doing over there with agents?

Yeah, absolutely. So, so at LIV Golf, we're a global golf league. We have some of the best players in the sport. Bringing them together to play tournaments all across the world. We're currently in our fourth season at the minute, so it's a really exciting time for us. Now, in golf, we have a caddie, and a caddie essentially supports our players across the course. And so we want to try and replicate that for fans, hence Fan Caddie. So Fan Caddie helps—

Clever.

Thank you. So Fan Caddie helps our fans gain information about our latest events or getting information about tickets to get to those events. It can help our fans with leaderboard scores and stats, and it can even show them videos about who scored the last hole-in-one, for instance, right? So it's a really capable agent. And I'm going to quickly go over to our architecture and show you how we've actually achieved that.

I'd love to see that.

So you can see here in the architecture, we've actually got 14 different subagents to help support some of these topics. So all of these different sub-agents, they have a specialty which they can cover and answer questions for fans too. Now today I've actually got my phone with me and I want to show you Fan Kedi. So give me a second. Just going to pop it open. So this is Fan Kedi, right?

Mm-hmm.

And so this is actually the latest version of Fan Kedi using AgentScript that we just launched last week.

Mm-hmm.

And now One of the things I've been working on is how we can personalize messages based on the location of the user. Right. And so I'll give an example. So a question that is commonly asked is, where can I purchase merchandise, you know, before the next event? So our next event right now is Korea. So I've got the question popped into my phone now. So where can I get merchandise from for Korea?

Right.

And now What's going to happen is if we switch back to the architecture diagram, you're going to see that it's going to switch to our merchandise sub-agent, right? And so this is going to redirect the user to our online store to help them make a purchase before the next event. And that's exactly what's happened in the app right now, right? So it's showing us a CTA to visit our website to purchase some of the latest threads before they go into the next event. But wouldn't it be cool if we can actually personalize this message based on where they are? So for instance, if they're actually physically at the career event, wouldn't it be better if we could actually show them a message to direct them to a tent nearby, right? So rather than going online, they can buy one physically right there and then.

Give them more precise location.

Exactly, exactly, right? So I want to show you how we've gone about doing that, right? So I'm going to switch back to AgentForce Builder and I'm going to show you how we've achieved that. Now before I ask the same question again, what I'm gonna do is, I just wanna show you some variables that we've set to achieve that kind of personalization. So I'm gonna expand the variables tab here, and then I wanna show you a few things that we've set up. So, we have a variable called in venue mode, and we've set this to true. And there's another tag here that we've also set, which is the in venue tournament ID. So this is just an internal identifier to tell us that the person is at the Korean event. And now what's going to happen is I'm just going to collapse this and I have the question pre-prepared. So what's going to happen now is, so I told you previously with the app, the agent used Golf Merchandise to answer that question. But now, so we've got a response now. And so he already knows, you can see from the response. So since you're already at the Live Golf event in Korea, why don't you go and visit one of the stands or tents nearby? And it tells you exactly where it is, right? So this has been really, this has been really great for our fans because it elevates the experience not only online but also in person.

Right. Yeah, it gives it that sort of personalization for you to go wherever you are. It directs you where I need to get that merch. And you know how fans are, they love their merch.

They sure do. Yeah.

Yeah. And you're using those variables then to like personalize that location, right? Which reminds me of a use case that we have that's kind of a different scenario but kind of similar. And indeed we have two different business segments. We have small business and enterprise.

That's right.

Before, we built a solution to be able to differentiate them through reasoning.

Yeah.

But it wasn't deterministic. So we had a lot of scenarios where somebody wanted to talk to a human, but they were sent to the wrong business segment.

Okay, that would be frustrating.

That was very frustrating, especially for our stakeholders.

Right.

So we had to go back and revert to a single group across the board. But with AgentScript, now we're able to provide that determinism using variables similar to you. And I can show you real quick how that looks like. So this is the Canvas view, but let me show you the script view. So what we're doing now is we're pulling the segment data into the agents deterministically.

Right.

Grabbing that information and then passing it to the escalation process. So that way it's a variable that's independent from reasoning. So that way it doesn't matter how the conversation goes, we're always gonna escalate to the right business segment.

That's great. So you guys are both using variables in different ways, but using deterministic logic. What I noticed from both your architecture design and as we stepped through your code, you all have a lot of different subagents. How are you handling orchestration amongst those subagents?

Yeah, I could definitely have a go at answering that, Leah, and I'd love to hear all about what you guys are doing now indeed. So if I switch back to the AgentForce builder, so as you mentioned, yes, so we've got 14 different subagents and now AgentForce is pretty good at trying to figure out which agents to use, but using variables we can actually then really enforce which agents to use to get a better or more accurate answer. So I'm going to go into our agent rule, and I'm gonna show you how we've achieved that. So, in the, we've got a section here called before reasoning.

Mm-hmm.

And I mentioned earlier about when we set particular variables, we can then tell it which sub-agent to use. And this is actually how we've defined that, right? So, in the before reasoning section, if I expand this, we've got an in venue mode tournament ID. And when this has been set, we activate our in venue mode. Okay. Right? And so, if I scroll down, I wanna show you exactly how that works. So, if I scroll down into our sub-agent, in-venue concierge. You can see here as soon as we've hit the in-venue mode, we transition the agent directly to the in-venue concierge, right? And so that gives us that accurate answer for customers actually physically in the events. And we're reducing some of that latency required by the agent to figure out which one to use because we've told it deterministically. Now one thing I just wanted to quickly point out also while we're here is, so as part of that before reasoning is we've also preloaded some information. So this is something that's new with agent script that's really kind of improved some of the latency times that we've seen with our customers. By referencing data prior to actually the agent doing any kind of reasoning, we're able to kind of really reduce the time it takes to respond to the answers that our fans are asking.

Well, I mean, that helps once again with latency, right? And accuracy, two huge ingredients for creating a good customer experience.

Exactly.

A great customer experience. Another thing about customer experience Agents are great, right? And at some point they want to— customers want to talk to a human. We talked about orchestration from agent to agent, but how are you using orchestration for an agent to seamlessly bring in a human into that conversation?

Yeah, I can take a stab at that one. Yeah. At Indeed, our service channel is also a revenue channel.

Oh, okay.

So when our employers are going through that hiring journey, right, they somehow get stuck somewhere. And then they reach out for help for us to put him back in that hiring journey. And then that's critical for them to stay with us as a customer, right? So making sure that we're very clear on when do we get those signals saying, hey, let's escalate you to a human when it's failing, it's extremely important for us. And I can show you how that looks like real quick. So I'm going to trigger here the chat experience, and here's where we're rendering the service agent, and then we'll see it get populated with you know, the greeting and all the information that shows like, hey, I'm an AI agent, I'm here to help you. Let's say the user come in and says something like, I need to verify my account, which like I said before, is very critical for them to post a job. Like they just lost somebody in their company, they need to hire somebody ASAP. So they want to check the status of the verification for their account. And then the first answer they're going to get from their agent is that there's additional steps they have to take for the account to get verified. And then at that point, That's where we capture signals sentiment and deterministically.

Right. Definitely sentiment.

Sentiment, exactly. Because in this case, they're going to be frustrated.

That's right.

The AI agent could help them, but they want— they show signals that they want to talk to a human. We don't want to get in the way. We want to escalate.

You don't want to force them into an experience they don't want.

Exactly right. So here's an example of it escalating to a human. So we now are able to deterministically capture that intent. Then we can actually show you here as well how it looks from the human agent side. Because there's two personas here. There's the employer agent and also the human agent as well. So now when you answer the chat, you can see here they have the full context of the conversation, but they also have an AI summary. So that way the human agent can either read the summary, they'll see if that's enough or not. And if they feel they need more context, they can then scroll up.

That's great because they don't have to play catch-up. Everything that they need is right there at their fingertips. That's great.

Exactly right. And this is very important because we want to make sure that average handle time Like it's important for the human agent to be able to save time, which we can do that because the AI did some discovery work, but we also wanna track the end-to-end employer to human experience to make sure that they're not spending more, too much time on it 'cause that affects CSAT and other metrics like that.

Well, yeah, speaking of metrics, I mean, like you just talked about, there's two perspectives you're looking at from the customer reducing their time and then also from the rep reducing their time and spending it. And metrics is important in this conversation on this agentic journey. I know for our customer, Grupo Falabella, We have Stella, you know, 60% of their WhatsApp inquiries are answered autonomously. And then we have, you know, JPW, which is a manufacturing industry. They've gone 40% faster in resolving cases while increasing their workload. So they're seeing really big differences in their work processes. I'd love to hear from you. What are some of the KPIs that you all are looking at, or the metrics to measure success with the use of agents?

Yeah, I can definitely answer that for you, Leo. So at Live Golf, the two most important metrics have been latency and accuracy for us, right? And so with latency, as I mentioned before, because now we've been, we're able to preload some of that information, we're able to use some of the variables to kind of reinforce which agents to use, we've really reduced that latency time. So if you need a figure, I think that in our most recent tests, we've seen a reduction of almost 60% in terms of latency.

Wow.

Now, if I switch to accuracy, so as you can imagine, when our fans are asking about scores, you know, it's important that we get that right. Yeah. And so with our accuracy scores, we've had our QA team vigorously test every single version of our agents. And so what they've said to us is that the latest version of FanCaddy using agent script has come back with a 94% accuracy rating, which is—

94?

That's big.

94%. What about you? What's Indeed? How are they doing?

Yeah, we have two types of metrics that we measure for the service agent. One of them, we have an outcome type of metric to make sure that we're seeing value out of it.

Right.

And then we have a quality metric from the employer's perspective. So we use CSAT for the employer, the employer side, and then we use a resolved, or we call it a containment rate as well for the outcome one. And our containment rate, when we first went live, we were at 4%. And now we're at 25% average.

Oh, that's huge.

Which is just over 6x increase. And then on CSAT, we also started low at 1.8, and then we were able to improve it, and now we're averaging at 4, which is over 2x increase for CSAT as well. That's a great metric.

Yeah.

And you guys are just at the beginning of your journeys, and you're seeing these returns. I can't wait to hear, you know, when you're further down along—

Yeah.

—what the metrics will be. Well, thank you so much for sharing these very useful insights that I'm sure many people are going to be very appreciative of and motivate others to jump on this Agentic Transformation. So, thank you both.