Getting Started With AI Agents: 3 Key Tips and Strategies for Success

Hello.

I'm sitting down today with Rebecca Wedeman, CEO and Principal Analyst of Valoir, to share her insights on Enterprise AgentIC AI. We're exploring different approaches for adopting AgentIC AI, including the challenges of trying to do it yourself, all based on her extensive research and in depth customer interviews. Before we dive further, could you first help our audience understand what a DIY approach to agentic AI means?

So the DIY approach was really about an organization building all the necessary components to build an agent from the ground up. So selecting, tuning an LLM, sometimes even trying to build an LLM themselves, integrating the various data sources, building prompts, building those necessary security features and guardrails, developing the user interface or integrating with some existing interface and creating those workflows that enable the agent to perform its tasks. Obviously not all companies were building from scratch on all of these components, but they were often leveraging and cobbling together kind of different open source tools, other commercially available cloud services in house, and sometimes external development expertise.

So why do you think companies are attempting this DIY approach?

What we saw was a lot of initial excitement around generative AI. I think we were all excited in the beginning, right? And the promise of these agents. Organizations started to have a real fear of missing out or fomo. However, as they got into actually trying to build out these agents from scratch with the DIY approach, it presented some real hurdles. Many projects struggled to move beyond the pilot stage. Even for companies that had a lot of resource and AI expertise. This really led to a shift in mindset from FOMO to what we termed fear of messing up or fomu. Right. As organizations tried to achieve acceptable levels of performance and accuracy, creating AI that could handle complex tasks reliably without human intervention proved to be a major challenge.

Could you share some of the key trends and challenges you observed in organizations early attempts with agentic AI?

First, many organizations that were attempting DIY deployments quickly found significant hurdles that led them to abandon their in house efforts. One of the primary challenges was achieving an acceptable level of performance and accuracy. As our report and our research found, many of those DIY projects never moved beyond the pilot phase because they just couldn't reach the necessary accuracy. Even after significant tuning. They found that on average, DIY efforts only achieved around a 52% accuracy, which obviously was unacceptable for most of them. This was often due to a lack of the sort of robust reasoning layer for complex tasks, which led to a lot of hallucinations. Even with a lot of tuning. These complex scenarios to eliminate inaccuracies proved to be a really big task, the kind of scenario modeling that could take years for organizations with large product catalogs, for example, the other significant barrier really was around data integration and management. That big challenge was building out and sort of maintaining the AI guide, rails and security. The development cycles associated with user interfaces and workflows was also another big challenge area. They found that even if they could get something up and running, which took a lot of work, they then had to think about the burden of ongoing maintenance. Right, because you can't just build an agent, you have to keep managing and tuning all of those different pieces. Particularly as the LLM vendors kept coming out with new models and new versions. Many of the companies we talked with said they realized through their DIY efforts that there was just no way to make sure that their data was really secure. At the same time as these DIY folks were realizing that they had maybe bitten off more than they could chew, the landscape really started to evolve with the emergence of platforms specifically for AgentIC AI development. Right. In September, Salesforce announced Agentforce, and other vendors as well followed suit. So our research was really looking at these customers, focusing on understanding the value of this platform based approach, specifically looking at the experience of Salesforce customers using agentforce.

This is such a rapidly evolving space and there's clearly so much to unpack here. Thank you for your insight, Rebecca. Tune in to our next video interview and learn how AgentForce can help you avoid these DIY pitfalls and get AI done right.