How SaaS Companies are Successfully Productizing AI with Paragon at SaaStr AI Day

I'll just come back at the end to help moderate Q and A by now, much like everyone else, you got a lot of slides, so I'll let you get at it in a second. So, yeah, thanks for doing this. Are you guys ready?

Let's do it.

Good. Okay, sweet. Let's do it. Okay, I'm going to admit, folks. Hello everyone. Welcome back to Saster AI Day. We're so excited for you to be here at our next session. For those of you that are going about all day, this is like your halfway point. So you've now made halfway through AI Day. We're still bringing you really great speakers and content. So excited to have our next two speakers join us. So Brandon Fu, who's the CEO of Paragon, and Ethan Lee, who's the director of product to Paragon, are going to be talking about how to successfully productize AI. I know they're going to be talking about RAG a little bit, which we were just talking about a little bit in our last session. So it's a good jumping off point for those that are watching all day. Excited to have you guys. As always, if you have any questions for our presenters today, drop it into the chat as we go along. They'll try and answer them. If not, we'll share their socials at the end. And with that, Brandon, I'm going to let you take it away and get into it.

All right, thanks, Amelia, and thank you everyone for joining us today. Today we'll be talking about how leading SaaS companies are successfully productizing AI. And as Amelia mentioned, I'm Brandon, co founder and CEO of Paragon, joined here by Ethan, director of products at Paragon, and to share a quick introduction and backgrounds on us on Paragon. Paragon is the integration infrastructure for B2B SaaS companies. So Paragon ultimately helps software companies integrate their products with the third party SaaS application ecosystem. We started Paragon about five years ago and have since raised just over 20 million in venture funding and today work with nearly 200 of the leading SaaS and AI companies in the world to help power their integration and product integration strategy. So we've really been a part of how many of the leading SaaS companies have utilized integrations to productize their AI strategies. And we'll be talking more about that today, starting with some context and background on why LLM wrappers failed as an initial approach to bringing AI to market and how that led to a new strategy of rag retrieval, augmented generation and AI agent tool calling for productizing AI. We'll then talk about some of the challenges of scaling rag data ingestion and agent tools, and we'll finally share some examples of case studies of leading enterprise AI companies that have successfully implemented these strategies to productize gen AI. So without further ado, I'll hand it off to Ethan to get us started.

Awesome.

Thanks, Brandon. Yeah, just to start and contextualize some of the problems that we'll be looking at today, I want to talk a little bit about why LLM wrappers failed. And really what we mean by an LLM wrapper is sort of an LLM with a little system prompt that's specific to the type of task that you want the LLM to do sprinkled on top of an OpenAI or anthropic API. And essentially what LLM wrappers are are essentially a reaction to how popular popular chatgpt was and how people realized that when you're Talking to this ChatGPT bot or LLM and there's all this power that's coming out of it, that seems to be real knowledge, people assume that it could do many more things than was initially anticipated. So there's a couple problems that people quickly realized that LLMs could not do on their own, and furthermore LLM wrappers could not do on their own. And the first problem is really that it lacks external context. So if you've worked with ChatGPT for some time, you probably realize that you have some conversations where it says, oh, I don't have knowledge up until this date, because there is a knowledge cutoff date for those foundation models. And really why that happens is that there's a training phase for those LLMs where it's trained on publicly available information from the Internet and it really doesn't have any knowledge upfront about you or what your business is, what your product is. So what you can imagine is there are some examples here on the right where humorously an Amazon support agent that is an LLM wrapper, you can ask it for generating a React app and it starts spitting out some code in an Amazon support chat. It's just one funny example of an LLM wrapper not exactly having the right context for answering the user's question, or rather getting off task about what it's supposed to do. The second problem is that you can try to inject some data from your application that you have about the user to give the your LLM wrapper some more context, but generally that isn't a complete view of the world that your user is operating in, so it doesn't have all the business or product context that you might generally expect when you're talking to an LLM app that is inside of your product. So that's the first problem, is that these wrappers just simply don't have enough context to answer all of the questions that you might want them to answer. Jumping to the second problem, the second problem with this is that wrappers can't really automate work on their own. So what LLMs are really good at is generating text for you and answering questions effectively. But without any ability to actually perform work in the places that you do work, wrappers can't actually set up meetings on your behalf or create records in Salesforce or different systems of records on your behalf. So text generation is not enough, in other words. And I think people quickly found this out with LLM wrappers, and that's why people are already envisioning what the next step is for AI in terms of what we are calling agents. So next, moving forward, we're going to talk a little bit about where the ecosystem is moving more broadly from these simple LLM wrappers that just have this context specific prompt to really agents that can do work with you and do work for you. In order to talk about that, we're going to focus on two approaches that people are using to build out agents. The first is rag, which is Retrieval Augmented generation. We'll talk about why that solves the first problem of feeding in all the context that your LLM app needs. Then the second is agentic tool calling, which solves that second problem of LLM wrappers themselves can't do work on your behalf. Both of these characteristics, having context and also being able to do work, are really important characteristics of agents and I think will be important more broadly for AI adoption and making AI more useful for your products. So let's take a step back and just think about some of the places that your context lives and work gets done today. And these are apps that apply to you and your team and not just for AI agents. But the same problems still apply. So for example, you have all of your knowledge stored in internal documentation systems like Notion or Confluence Docs, even in PDF files and Google Drive and that sort of thing. All of that knowledge is really important for your teams to be able to get the context they need to do their work. You might have important team documents or engineering specifications or requirements in different places documented in those files. And if your AI agent doesn't have the ability to query that documentation, it's not going to help answer those specific internal questions that your team members might have. Second is systems of record. Your team today is using CRMs or project management tools or HR systems in order to track structured information about how work is happening or the status of an order or the status of a project, that sort of thing. And if your agent doesn't have access to those same systems of record that your co workers do, then it's not going to have all the context it needs to both understand the state of a project, understand the state of an order or a customer, and it's not going to be able to take actions on your behalf like a coworker would be able to do. And then lastly, of course, probably many of us experience this. Nothing is perfectly documented. And the rest of all that context that fits in the cracks is in Zoom meetings or Slack conversations. Things that are not well structured at all, but are still important for your own team members to have context in in order to, you know, do their work effectively. So all of that kind of knowledge that's scattered across all of these different apps is important, equally important for your AI agents to understand in order to be as effective as one of your team members and make sure that it can be a helpful source of information. So to apply that to the kinds of strategies that we can use to make agents more useful, there's kind of two approaches to context retrieval or pulling in external information. The first is called ingestion for rag, which really what that means is building up a search index for all of these documents and unstructured information that the agent needs to be able to query. So for example, we talked about all of those documents that are in Notion or those messages that are in Slack. If someone needs to search a topic by what are things related to this type of product that my team needs to look at Right now a vector database or a search database is really helpful for an AI agent to ask a question to that database and get all the related documents or related messages about that product and return it to the AI agent to generate an answer. So this is what we call an offline approach because all of that information is being replicated by your app in order for the AI agent to take advantage of it later on offline in a conversation. The second approach is what we call a gentic rag or it's an online approach which means that it fetches data in real time in order to pull in information context for a conversation. So for example, if I'm asking a sales copilot agent about different opportunities that are active and I need to get all contacts associated with that opportunity, I'm going to directly query the Salesforce CRM, or rather the agent is going to directly query the Salesforce CRM to pull in all of that information and bring it back to the AI agent in context. So that's another important example because in that case there's a structured query that we're looking for. We know exactly what types of context we want out of Salesforce. We, we don't need to do a natural language search in order to pull that information. So we can talk directly to the API to pull in all that information and get the user's context into the AI agent conversation. So those are the two approaches. There's ingestion for rag, there's agentic rag. Both are used for pulling in context for the user and solving that first problem of it not understanding all of the context outside of model or world knowledge. So the next approach that people are using to solve the second problem is really agentic actions or tool calling. So for example, here's an example conversation where a user is asking to book a call with a contact in their CRM. And essentially what an AI agent needs to be able to do to actually make that meeting happen or to actually book that call is it's going to have to reach out to at least three apps in order to make sure that it can book a meeting with the right person at the right time and send all the information to the relevant parties. So agentic actions essentially give the agent access to different individual tools that are exposed by Salesforce or Google Calendar or Gmail. So for example, searching for a contact by a specific company name or searching for availabilities on someone's calendar. All of these are individual actions that you can provide to your agent so that it can actually do work in those tools or query information real time about those tools so that the meeting can actually be booked and that the email can actually be sent, not just a draft created for your user as a text based conversation. So in that way the agent is actually working with you in your context instead of simply just being this chatbot off in the side that's doing its own thing, generating text. So next we'll move to some of the challenges, some of the key challenges that we see and some of our customers share around scaling some of these approaches and what some of the thorny problems around those approaches. So the first one around ingestion is that it's hard to implement correctly. And here are kind of the key challenges that we see. First is that connecting to third party data is A big problem. And that connecting to each individual API and implementing each individual connector is not only a huge implementation cost for your engineers, but it's a huge maintenance cost for your engineers as well. And we're an integrations company, so we know all about what it takes to maintain connections, to build out connections. There's a lot of work that goes into making reliable connectors. So that's challenge number one. Challenge number two is that once you actually have that data, you need to make it effective in order to have the AI agent be able to actually search on that data in your offline database. So, for example, you take a document and you have to actually make it searchable. There's all these different strategies around chunking and vector search that you'll have to fine tune in order to make sure that your agent is can more effectively get to the right snippets of information and pull that into its context. So that's challenge number two, is just ingesting and indexing is a hard project in and of itself. And then lastly, and really what I think is one of the most thorny problems in this is that if you're going to take a copy of everyone's Notions documents, everyone's Google Drive documents, all of their Salesforce CRM records, you better make sure that that copy of the data that you're creating for your app respects all of the permissions and business logic that those original systems of record actually enforce. So of course, you can imagine a disaster scenario where someone is just chatting with an AI chatbot and is able to gain access to some documents that they really shouldn't have access to because they're accessing a copy of the information that the agent had access to. So making sure that those permissions are actually correctly enforced by your app and stored correctly by your app so that the right people have access to the right documents is a really big challenge. So that's one. Those are one among three things to really watch out for. Ingestion and some of the technical challenges related there. Then moving to tool calling. Tool calling itself is difficult to implement simply because there are hundreds of different actions that your agent will need. It's, you know, somewhat difficult to predict or preempt what kinds of questions or tasks that your AI agent will need to do on behalf of users. So you'll have to implement it across a wide range of tools and make sure that the most common behaviors in each of those tools are exposed as capabilities to your agent. I mean, think about all of the SaaS apps that you are going to use just today. And all of the individual actions that you're going to take in those apps, each one of those clicks, each one of those filling out of a form, that's going to be an individual capability that has to be built into, implemented and given to an agent by your engineering team. So it's very difficult to scale out this large catalog of what we call integration actions. Fortunately, at Paragon, we actually built out a large catalog of these actions for a separate product for our workflow builder. And it turned out that these same definitions that we actually built for making it easy to construct workflows out of these integrations turns out to be very helpful capabilities to the LLM. And we'll cover a little bit more about that later. Okay, so now I'll turn it back over to Brandon to talk about how enterprise AI companies are leading the way with some case studies.

Great. Thank you, Ethan. And so next I wanted to share some real world examples to help contextualize how leading AI companies have implemented these strategies to bring their AI products to market. And so we'll start with what should be a familiar example to many. I'm sure most people here have used Intercom either as a customer or as as an end user using their chatbot. And so Fin is the name of Intercom's AI support agent. And so an end user could ask questions in the Intercom chat widgets and Fin would be an AI agent that is able to respond to customers questions and queries and requests. So early implementations of Fin are a good example of sort of the LLM wrapper problem. Because the early versions of Fin did not have access to any external context or external data, it really was only able to provide very surface level or generic responses to questions. So if you asked it a question, for example, like how do I process a refund? It give you an answer that would sort of look like an FAQ answer. You know, here's what you could find from the Intercom documentation. And because it didn't have access to any of specific context about that user or any of the external data, such as, you know, what's orders or purchases that user might have made, its responses are really limited to what the information is available in the Intercom help desk or FAQ system. So since then, Intercom has integrated Fin with a number of external data sources and actions to really solve this problem. And this is a great example of how they went sort of from the LLM wrapper approach to implementing forms of rag data ingestion and agentic actions. To deliver a far more relevant and useful experience for users. First of all, with ingestion, Fin is now able to connect to external data sources such as support help centers like Zendesk, internal documentation systems like Confluence and Notion, so that it could provide more detailed and relevant responses with context from many different data sources within a company about a company's products and how it works. So it's really able to utilize RAG data ingestion to collect all this context and make that available to its agents to pull from to deliver far more relevant and insightful responses. In addition to that, Fin now also has access to tool calling. Essentially it can make requests in real time to different systems such as status page, Stripe for billing, Shopify for e commerce order histories. So it can retrieve relevance data from these different systems in the context of the user that it's chatting with. So you could imagine now if a user might ask Fin about a return or how they can get a refund, could actually use its access to these different systems to get data about that user's specific order history, maybe even process the refunds on behalf of that user. All by nature of its ability to integrate and access the context and tools from these different systems. Next I'll share an example from Copy AI. So Copy AI is a AI go to market orchestration platform. Essentially it helps automate many of the sales and marketing activities that your team might perform today. And it utilizes ingestion and agentic automation in a slightly different way in a different context, but ultimately to deliver a far more valuable experience for its users. For example, it's able to integrate with a customer's gong meeting recording history to ingest content from call recordings, sales calls and use its AI, its LLM to extract important information from those recordings and then perform actions into customers systems such as their Salesforce CRM to update relevant information in their CRM records or send messages to the customers to Slack messaging channels. And so Copy AI is a Paragon customer and are today using Paragon to help power many of these integrations to deliver this far more relevant experience to its customers. Another example is tldb. TLDB is an AI meeting assistant and not only are they able to transcribe and record customers call recordings, they're able to also use agentic actions tools that its agents can call to take content from a call recording and similarly perform tasks automate tasks in its customers JIRA Project Management System or Trello or Asana to take relevant information from calls and actually make them actionable and automate the creation of those tasks in these different third party systems. Last we have supper, another Paragon customer who is integrating or they are an AI powered business intelligence platform. So what they do is integrate with various data sources such as CRM marketing platforms, billing systems like Stripe, such that users can ask it complex queries about their business and it can utilize the contextual data stored in those various systems to perform complex data analysis and, and return relevant responses to its users. So you could think of it as essentially an AI data analyst that is able to utilize data ingestion, connecting to all of its customers different systems of record and data sources to be able to provide these complex or be able to answer these complex queries and provide these far more relevant results based on a customer's specific business data. So hopefully these were helpful examples to Highlight how leading SaaS companies and startups are utilizing these strategies today to deliver a successful AI productized experience. And you know, we really believe that today the main constraints for mainstream adoption of AI is no longer model capability, but really the fact that agents and LLMs by default live in data isolation. And we believe that very quickly today the market is converging to a point where agents and LLMs will become default connected to all the different systems applications that their customers use. And that's exactly the problem that we're solving at Paragon today. So what happens after that point? Once all AI applications are inherently connected, then what are going to be the differentiating factors that separate the weight winners and losers in the market? So we believe that there's a few things that will be important here. One is response and action accuracy. So AI agents and LLMs can make mistakes. I think we've all experienced that. And so the ability for agents to select the correct tools, perform the correct actions and access the correct data for a specific context or a specific request is just as, if not more important than its ability to have access to that data and tools in the first place. If it's selecting the wrong tools, for example, it's really not going to be any more useful than if it didn't have access to that data. And of course we believe that user experience is still going to be a key differentiating factor in AI and AI products. AI should feel like a seamless and magical experience in the context of your customers existing workflows and the problems that they're trying to solve with your, your product. And I think what will be important is that products and engineering teams prioritize user experience over sort of the industry trends. And I think many companies today are building chatbots because obviously that's become very popular today. But chatbots are not necessarily the best form factor for solving every problem that your customer might want to use your products for. So I think that the teams that are most intentional about designing the form factor and the interface of their product, whether it's a chatbot or another form of application, will still be a key differentiating factor that will separate the winners and losers in the AI market. So last, if you are building an AI product, an AI agent, chatbots, we are solving this problem as I mentioned today. And Paragon is a developer platform for integrations. Today we've helped dozens of leading AI companies productize their AI strategy, both by helping them build scalable rag data ingestion pipelines to ingest contextual data from all of their customers different data sources, as well as with a new product that we're launching now called Action Kit, which is essentially a single API that will give any AI agent access to thousands of integration tools so that your agents can dynamically retrieve contextual data from any system, perform actions and automate actions in different applications. And so if you are building an AI products today, feel free to reach out and we'd love to see if we can help. You can learn more about paragon@use paragon.com I'm sure we'll drop some links after our call today and you can also feel free to email me brandonuseparagon.com Would love to chat about what you're building and see if we can help. Thank you everyone.

Thanks guys. I think we have time for one quick question. So Sandeep, let me see if you are still here to ask your question. Sandeep, do you want to come off mute? Okay, maybe I'll ask this question for him. Hold on there. No problem. Part of being live. Okay. His question was do you see the next evolution of sas?

Where do.

Do you see the next evolution of SAS tools with an agentic layer that replaces or uses APIs to provide wider context across the org.

Yeah, so I think agents will certainly utilize APIs and I think that's, you know, really the means by which they will be able to gain access third party systems to ingest contextual data. I think APIs will remain the sort of programmatic interface by which applications and AI agents by extension will use to communicate with different applications. I think the way in which developers implement APIs in the AI and Agenti context will look very different. And again, this is one of the problems that we've been focused on solving at Paragon, where With a product like Paragon, for example, you can provide your agents with access to hundreds or thousands of different function tools to call different applications different integrations. What would amount to implementing dozens if not hundreds of different APIs. But with Paragon you would be able to essentially provide a single API call to your agents to be able to dynamically understand Here are all the tools, here are all the APIs that are available to me and here's how I would use them in the context of a know real world chat conversation.

Awesome. Well thanks so much guys. Unfortunately we're out of time so thanks for joining us. I know we'll see you both back at SAS Annual this May. Excited to see you there too. Thanks everyone for tuning into AI Day for watching on the live stream as well. We'll see you for the next session. Brandon Ethan, thanks for sharing your emails as well so people can get in touch and thanks for doing this.

Thank you everyone. Thanks Aster. Thanks Jason.