How to Drive Real ROI With Agentic AI

Are you looking to understand how to successfully deploy and maximize ROI from Magentic AI? Hi, I'm Nicole snailing product marketing here at Salesforce and today I'm talking to Rebecca Wedeman from Valoir, a leading analyst from a firm that specializes in research and technology analysis. Welcome, Rebecca.

Thanks, Nicole. It's great to be here.

Well, we've all heard about the transformative potential of AI agents to revolutionize business. But the journey to realizing that potential, it can be costly, complex and slow. And today we're going to cut through the noise and focus on how businesses can move beyond the pilot phase and accelerate their return on investment with agentic AI. Rebecca and her team have conducted extensive research on agenti AI deployments and we're very fortunate to have her here today with us and she'll share her valuable insights. Can you tell us a little bit about your firm, what you do and your role?

Sure. So Valwire is an independent analyst firm focused on the value of technology. So a lot of what we do is helping customers build the business case for their technology investments. We do a lot of independent research like ROI case studies, broader research reports and surveys to help customers understand how to measure the value of technology. And as you can imagine, in an emerging area like AI, there's a lot of thought leadership around helping people understand how to measure things that they've never experienced before.

Absolutely. It's new territory we're in, so let's jump in. There was a recent publication from Velwar all about the key strategies for how to accelerate your agentic AI's time to value. And that report was based on in depth interviews with Salesforce customers who've also experimented with building their own AI solutions. Can you tell us a little bit more about that research?

Sure. So we reached out to our network of Salesforce customers as well as some that Salesforce introduced us to that were early adopters of AI and agentforce to interview them about their experiences. They represented organizations of a variety of sizes across multiple geographies and industries. We were really talking to them about what they had tried to do with a do it yourself or DIY approach and then what they were able to accomplish initially with AgentForce.

That's very interesting. So can you share with us a little bit about what were the common challenges that you heard from organizations that were starting their journey with agentic AI?

Absolutely. What we saw was a lot of initial excitement around generative AI. I think we were all excited in the beginning. Right. And the promise of these agents. Organizations started to have a Real fear of missing out, or fomo. However, as they got into actually trying to build out these agents from scratch with the DIY approach, it presented some real hurdles. Many projects struggled to move beyond the pilot stage. Even for companies that had a lot of resource and AI expertise. This really led to a shift in mindset from FOMO to what we termed fear of messing up, or fomu. Right. As organizations tried to achieve acceptable levels of performance and accuracy, creating AI that could handle complex tasks reliably without human intervention proved to be a major challenge.

That makes sense. It's all new territory.

Sure.

And so before we hear about that FOMO fear of messing up, could you tell us, you know, tell our audience a little bit about that? What were you calling a DIY do it yourself approach? What did that mean when you were doing this research?

Sure. So the DIY approach was really about an organization building all the necessary components to build an agent from the ground up. So selecting, tuning an LLM, sometimes even trying to build an LLM themselves, integrating the various data sources, building prompts, building those necessary security features and guardrails, developing the user interface or integrating with some existing interface and creating those workflows that enable the agent to perform its tasks. Obviously, not all companies were building from scratch on all of these components, but they were often leveraging and cobbling together kind of different open source tools, other commercially available cloud services in house, and sometimes external development expertise.

I mean, that's a lot, and that is significant work. So why do you think companies are attempting this DIY approach?

Well, I think it really goes Back to the FOMO. Right. Everyone on boards in the CEO's office were asking, how are we going to differentiate? How are we going to create competitive advantage? It captured everyone's imagination. And folks were getting a lot of pressure to outpace competitors and to really deliver on that AI promise. People started doing DIY because they were getting that pressure and there just weren't commercially available platforms yet. But they were really feeling the heat to show that they weren't falling behind. We were all getting told that everybody else was doing it. Right. Some organizations also believed that they could achieve a competitive advantage by building their own proprietary AI agents. Others wanted to build their own models and platforms in house because they were concerned about data privacy and leakage. Finally, to be honest, there was a lot of hubris. People thought that because they could code or understood some basic algorithms, suddenly they were prompt engineers and AI experts. As someone said to me the other day, with generative AI in particular, adoption pretty quickly Outpaced understanding.

So what happened? How did these DIY pilots, How did they turn out?

So a few things happened. First, many organizations that were attempting DIY deployments quickly found significant hurdles that led them to abandon their in house efforts. One of the primary challenges was achieving an acceptable level of performance and accuracy. As our report and our research found, many of those DIY projects never moved beyond the pilot phase because they just couldn't reach the necessary accuracy. Even after significant tuning, they found that on average, DIY efforts only achieved around a 52% accuracy, which obviously was unacceptable for most of them. This was often due to a lack of the sort of robust reasoning layer for complex tasks, which led to a lot of hallucinations. Even with a lot of tuning these complex scenarios to eliminate inaccuracies proved to be a really big task. The kind of scenario modeling that could take years. For organizations with large product catalogs, for example, the other significant barrier really was around data integration and management. That big challenge was building out and sort of maintaining the AI guide rails and security. The development cycles associated with user interfaces and workflows was also another big challenge area. They found that even if they could get something up and running, which took a lot of work, they then had to think about the burden of ongoing maintenance. Right. Because you can't just build an agent, you have to keep managing and tuning all of those different pieces. Particularly as the LLM vendors kept coming out with new models and new versions. Many of the companies we talked with said they realized through their DIY efforts that there was just no way to make sure that their data was really secure. At the same time as these DIY folks were realizing that they had maybe bitten off more than they could chew, the landscape really started to evolve with the emergence of platforms specifically for AgentIC AI development. Right. In September, Salesforce announced Agentforce and other vendors as well followed suit. So our research was really looking at these customers, focusing on understanding the value of this platform based approach, specifically looking at the experience of Salesforce customers using agentforce.

And what did you learn?

So we found a real acceleration in time to value for customers that were building Agent Force agents. Organizations using that platform that was already built, tuned and optimized for agent development like Agent Force could deliver autonomous AI agents an average of 16 times faster than the DIY approaches.

Wow. Yeah.

We also found a real big increase in accuracy with an average increase of 75% compared to DIY efforts.

Incredible. So let's dig into that a little bit more and understand those numbers better. What key factors contributed to the significant acceleration and accuracy when customers were using AgentForce.

Well, you know Nicole, we analysts like to put things into buckets, right? So we looked at a couple of key areas where AgentForce provides a distinct advantage over a do it yourself strategy. We found really six areas that we looked at and we talked to customers for each area. The kind of time and effort that was involved with model setup, data and application integration, prompt engineering, AI guardrails for security, building out that user interface and tuning. So let's start with model setup. With diy, organizations often spent months evaluating, prototyping and tuning large language models or building their own retrieval. Augmented generation RAG databases took a lot of expertise. Because AgentForce comes with a pre tuned LLM and integrated RAG capabilities, they were able to largely eliminate this setup time. One customer for example, told us that they had developed a beta of agentforce within weeks, where building it in their own LLM would have taken months. Another big area that we looked at, as I mentioned, was data integration, connecting and preparing data for AI. Because AgentForce leverages Salesforce's data cloud, which connects, unifies and harmonizes that customer data, customers that already had their CRM data in Salesforce were able to see quick benefits. The pre built RAG capabilities in data cloud which helped them bring in all that unstructured data also really accelerated that data integration time. One customer I spoke to for example, that had built their rag in data cloud in just a few hours with the help of a Salesforce solution engineer where they had spent a ton of time trying to pull all of their product catalogs into their DIY approach and never really been successful with it. Another key area that we looked at, as I said, was prompt engineering. Because AgentForce has Prompt Builder, it allows organizations to easily build and reuse prompt templates, reduces the need for that deep prompt engineering expertise and also those pre built prompts with out of the box skills. And agentforce further accelerates that process of actually building out those workflows. One customer shared how a manual process that took them a year to work on in their DIY project was operationalized in AgentForce in just a month. But probably one of the biggest areas that we found that was a challenge for customers in the DIY area was the AI guardrails and security features. So the trust layer in Salesforce provides these essential features with zero data retention, toxicity detection, the masking of data going to the LLM, all of those capabilities that enable customers to take advantage of AI and generative AI while feeling comfortable in being sure that they're Going to have data security. What's interesting is we found that a lot of folks had spent a ton of time trying to figure out how to build their own trust layer within their DIY projects. Some estimated that it would have been like 20 or 30 times the efforts that it took them to configure trust layer in Salesforce and in agentforce.

Wow. So no one got there?

No. And some of them had been trying for more than a year.

What was that big factor that helped them move beyond the pilot?

Yeah, I mean it was really. They got to a point where they recognized that they couldn't get greater accuracy without putting more data into their DIY models, but they couldn't comfortably put that data in there because they weren't sure it could be secure. Another area, obviously the user application development, that interface layer and what we found with Agent Builder and the sort of drag and drop interface, the conversational instruction dialogues, automated a lot of that UI and workflow development process, so they didn't have to think about how they were actually going to present it to the end users. Customers estimated that would take them at least six times longer to develop that customer facing UI and associated workflows without agent force. And I mean we're talking like weeks for folks to get prototypes up and working where they might have spent a ton of time and development effort if they were trying to do it in DIY fashion.

I mean, each of these categories has so much significant work. I can really see how it starts to stack up.

Yeah, and it's not just the time or the work, it's really the expertise. Right. There's a big learning curve for a lot of this stuff. Like I said before, just because you know how to code or you know a little bit about AI doesn't make you a prompt engineer, doesn't enable you to build out kind of the trust and security layers and all these kind of advanced data features. Nor frankly, as one company, can you probably cost effectively do it, even if you do have the resources. So what we find was the data security and guardrails, but also the tuning. We talked about only 50 ish percent accuracy with these DIY projects. That was a real challenge. And what we found with AgentForce, with the Atlas reasoning engine and the testing center, along with the ability to tune agents with natural language instructions, allowed business users, not prompt engineers or AI experts to be able to rapidly and iteratively improve accuracy. As they saw how AgentForce actually worked, customers said they had an increase in accuracy with minor tuning. They were able to get more Accuracy out of the box initially and then really being able to achieve levels they thought it would have taken years for them to do with a DIY approach. One customer saw their accuracy go from like 60% to 85% pretty much out of the gate.

Thank you for going through all of that. That was a lot to unpack in each of those categories.

Well, thanks for letting me geek out on you a little bit, but I think it's important for people to understand like all the different layers that are there.

Absolute. It's important to do that. Right. Because not necessarily obvious how much complexity is at each one of those layers.

Yeah.

You spoke to a lot of customers while you were doing this research and you learned a lot about the use cases. And we don't have time for a full testimonial today, but can you talk to us a little bit about what were the common outcomes that you saw across the use cases?

So just to frame it out a little bit, Nicole, people often ask us about how do I figure out which agent force use case is likely to deliver the most value? And we sum it up like this. You have to think about the four people, process, price and potential. Those agent force projects that impact a lot of people, have a lot of process steps, are expensive to do today and have the potential to really transform operations are those that deliver the most value.

So what would fall into each of those categories?

Sure. So self service that provides a personalized or individualized response based on customer history or particular product skew is 1. I know we talked about one customer manufacturer that had used RAG to bring their product catalogs into AgentForce and they had a ton of SKUs for them. Being able to reason through what product a customer had actually purchased was the difference between an agentic AI approach that gave them a real answer versus their DIY approach that munged together data from multiple product catalogs and gave them a somewhat nonsensical answer. That was the approach that really solved the customer issue and avoided that call, email or other interaction with the customer service rep. Obviously it's early days for AI agents in production for many organizations, but what we consistently heard from customers was significant reduction in the time and resources required to build and deploy effective AI agents and be able to answer these customer queries with self service in a personalized way based on customer history and product and other company data. They highlighted the ease of use, the pre built components and the robust security features as key enablers for them. And this really drove faster deployment, higher accuracy, really translating into quicker realization of benefits And a faster path to return on investment. Many felt that they were achieving results that simply wouldn't have been attainable with their DIY efforts because of lack of resources, expertise, or concerns about data security.

That makes sense. That's a lot. Data is so important. One more question before we dive into a demo. Based on everything you've shared today about valoir's research into AgentForce and the challenges organizations are facing with DIY, agentic AI pilots and deployments, what are the top three recommendations you would give to organizations that are looking to get started with agentic AI?

First, just say no to diy.

Okay.

You know you can make a sweater with a sheep and a pair of knitting needles. It doesn't mean you should. Right?

I definitely shouldn't.

Right. As we talked about, there's a lot of layers to these DIY products. You face numerous hurdles. You got models set up, you got data integration, you got prompt engineering. And it's not just about the initial setup. It's also about the cost of maintaining and governing these projects over time. And that's where you need the platform capabilities, the ability for business users to make changes, the testing tools, the monitoring tools. By leveraging a platform with the pre built models, all the integration organizations can bypass a lot of the technical work, a lot of the AI work, and see a much quicker path to return on investment. I often tell people that this is like the early days of cloud when people still thought they should build and maintain everything themselves again. Sweater, sheep, knitting needles. Why do it? The second point is data is really the foundation for successful AI. This means being able to bring in both structured and unstructured data and having your data architecture, architecture and hygiene in order. DIY approaches really struggle with connecting data and most importantly with ensuring security and privacy. And what we found was that Salesforce's investments in data cloud help customers integrate both Salesforce and non Salesforce data, which everyone has to ground their agents, which was really important both for accuracy and for personalization. A third key point I would say is recognized at time. The same is a journey, not a destination.

Right.

You have the guardrails and the tuning in agentforce to help customers get basic AI agents up and running quickly, but that will help them see what more they can do. Early adopters found their initial efforts showed them how they could get even more value from AgentForce by bringing in more data, refining flows or making instructions more explicit. The good news is that unlike DIY projects that require multiple AI experts and prompt engineers, AgentForce business users to build and adapt their agents over time and as their experience grows, then they can drive ongoing increasing ROI from agent force.

Makes sense. So no knitting my own sweater. Get your data in order and then iterate.

And I would say one final thing Nicole, is remember this is not about a technology project, right? This is about delivering value and delivering outcomes. So if you think about what I want to potentially do with AgentForce, thinking about those four Ps, people, process, price and potential can help you to identify out of the gate what your best use cases are going to be for getting the Most value from AgentForce.

I know this has been incredibly helpful for our audience today. And now for anyone in our audience who would like to see this in action, let's see a demo of how AgentForce works to simplify AI agent development, reliability and performance and get you moving from pilot to deployment faster.

The promise of AI agents is they'll help us work faster by enabling us to express a job to be done and doing that job autonomously on our behalf. But companies are struggling to deliver on this promise and finding that it takes much more than a large language model to make AI agents reliable for business. And here's why. A model will never truly know your business, because your business data is always changing and a model's response will only be as current as the last time it was trained. Which means you'll need to build a lot of data retrieval systems around the model to give your agents understanding about the jobs you're asking them to do. And models, well, they have no concept of security. Once your data goes into the model, your security permissions are gone, everyone can see everything. Which means you'll need to build security enforcement systems around the model to protect your data. And finally, a model, it can't actually take action. It can only recommend action that you'll need to execute. Which means you'll need to build action orchestration systems around the model to turn recommendations into actions in your business. And this is just the beginning of what companies need to build around a model to make AI agents reliable. But what if there was a way to skip all this and just start building an agent? Well, that is exactly what AgentForce enables your company to do, because these missing systems are built in and built on the Salesforce platform and integrated with your customer360 which means the only question is what agent should you build first? And agentforce helps with that too by providing a library of pre built agents like an AI service agent that answers customer questions and deflects incoming cases or an AI SDR agent that proactively works leads and books meetings for your sales reps or an AI sales coach agent that attends customer calls and provides real time tips and objection handling. Each of these agents was built on a platform, and it's a platform you can use to build your own custom agents, which you can start doing by simply describing the agent you want to build in natural language. AgentForce will then use Salesforce's understanding of your business to auto generate this agent. Then you'll customize your agents by describing the topics they can engage with and the instructions they'll use when engaging with age those topics using natural language descriptions and these form the guardrails that direct your agent's behavior. And you can test that behavior right here inside Agent Builder where you can inspect exactly how your agents do a job as they reason, step by step, Learning and adapting as they work. But how does this work? How does AgentForce reason and learn? Well, as I mentioned earlier, at the heart of an agent agent's capabilities is a large language model. But the model is an incomplete solution for turning a job into an outcome in your business. Which is why AgentForce starts with data retrieval, which helps your agents get as much understanding as possible about the jobs they're asked to do. All of that context is put into a grounded prompt that is sent to the model. But we're not asking the model to do the job. Instead, AgentForce is asking the model for a a plan to do the job. And then Agent Force orchestrates the execution of that plan to produce the desired outcome. But this is still an incomplete picture of how this really works. And here's why. When starting a job, it is rare that we have perfect understanding. In fact, it's only by working a job that we get a full understanding of the job. This is true for us, and it's true for your AI agents as well. Which means to be reliable, your agents need to be capable of learning and adapting as they work. And that's why we created the Atlas reasoning engine, which coordinates all these systems into a reasoning loop that allows your agents to work a job step by step, observing, recommending, executing, learning and adapting as they navigate to the desired outcome. This is what makes Agent Force more reliable. But none of this is possible without access to the right data. And AgentForce has built in access to your Salesforce CRM data as well as the 0 copy external data you've connected with Data Cloud. These are integrations you don't need to build, but AgentForce also has built in access to vector databases to power semantic search and retrieval augmented generation, which you can use by making those semantic search results a part of the of the prompts used to ground the model, which gives your agents a way to retrieve data that provides a much deeper understanding of the job to be done and data that is protected by a trust layer that enforces your company's security policies and data that sometimes allows your agent to recognize that it doesn't know the answer and instead of hallucinating, should get a human involved. Which Agent Force also supports with seamless handoffs to your employees. Because AgentForce is pre integrated with your customer 360. Which means you don't need to build a lot of interface integration to create a collaborative experience between your agents and your employees. This is how AgentForce helps you deliver on the promise of AI agents by providing all the missing ingredients that make AI agents real and reliable for business. To get started. Visit agentforce.com thanks for watching.

As you can see, Agent Force provides a powerful and intuitive platform for building and deploying AI agents. We believe it truly represents a paradigm shift in how organizations can leverage the power of AI. So we're now going to answer a few rapid fire questions. Shall we do it?

I'm ready.

All right. One of the biggest challenges we hear from customers is a fear that they don't have the right data. You talked about data?

Yeah.

So what do you think is different about Salesforce's approach to data versus a DIY approach to data?

Well, I think there are two things. First, for Salesforce customers that already have their data and CRM, a lot of the heavy lifting has been done for them. The second thing is because I'm able to build Agent Force agents so quickly and test them so quickly, I can quickly see as a business user, if I do have a hole in my data or if I have a data effort that I need to make right. If AgentForce isn't giving me the right answer, I can see exactly where I have a gap. So you're sort of reverse engineering the data hygiene thing, which means that business users can start with simple tasks, understand where they may have deficits in their data or inconsistencies in their data, and then be able to either add instructions or refine that area of data to make agents work better, identifying that gap.

Right. Okay. Second rapid fire question. In your research, you talk about the challenges that organizations are facing in reaching that acceptable level of accuracy with a duo it yourself approach. So what specific aspects of the data management, integration or other elements become the most critical when you want to ensure high accuracy and how can an organization prepare for that type of complexity?

Sure. So again, it comes back to how do I get not just an accurate answer, but an answer that is right for that customer. That means being able to have the customer data, but also product and company data all in the same model. Grounding those agents so they can pull data that's important for that specific customer. The Atlas reasoning engine. I'm glad you mentioned that because that's a really important part too, in terms of being able to think through which of these different customer examples apply. So I get the right answer for the right customer.

Excellent. Final question. What tips would you share with our viewers around the implementation process and specifically how it relates to usability and adoption? Any tips on that?

You know, we see folks experimenting with employee facing agents first, sometimes before they're ready to put that out there to the customer. But what we see is test early and often and you want to think about testing not just for accuracy, but really thinking about what is the outcome and what is the customer experience. So for example, if you ask for a return and we have a no returns policy, a good customer service agent might recognize that you're a loyal customer and be able to offer you some return. Anyway, so how do I think about as I get better with things like agent force tuning? Not just for did I give you the correct answer, but did I give you the best answer for that customer?

Love it. Really knowing your customers and reaching them in a whole new, more personalized way. Right?

Love it.

All right, well, we've now reached the end of our time and Rebecca, do you have any final thoughts? Anything you'd like to share?

Well, Nicole, the key takeaway really here is that while the promise of agentic AI is immense, the DIY path can be full of challenges and delays. The difference with a platform approach like AgentForce is a significantly faster, more accurate, more secure way to realize the benefits of autonomous AI agents.

Thank you so much, Rebecca, for being here today and for sharing all of your research with us. It's been very helpful and very informative.

Thank you. My pleasure.

For those of you who would like to learn more about AgentForce, please visit agentforce.com and you can also find more information about Valoir on their website@valoir.com thank you again to everyone who joined us today. We hope you found this valuable and we look forward to helping you accelerate your AI journey.