SaaStr AI Day Streaming Live: Why Your Data Isn’t Ready for AI with Firebolt

Yeah. Awesome.

All right, I'm going to start to let folks in.

Okay. Okay.

Hello, everyone. Welcome to our final session of the day for Sastra's AI day. We're so excited to have Connor Carreras join us and talk a little bit more about your data and what you need for it to work in AI. So I'll just do a quick intro and just to remind you guys, I know this is the last session of the day, so you've been probably will verse in this. But just as a reminder, if you have any questions for Goner, please drop them into the chat and I'll help moderate Q and A at the end. And much like anything else I know you guys have been asking, like, where you can see recordings of this. It's on the Saster YouTube channel streaming live right now, so you can go back to one or if you want to reference this one for anything you missed, it's all there for you. So with that, we're excited to kick off our last session. And Connor, I'll let you take it away.

Awesome. Thanks so much, Melia. I am from Firebolt, so we're an analytical database. And so what I want to talk to you guys about today is all about data infrastructure and specifically how you can get your data infrastructure ready for AI. I love examples. We're going to be framing this in the context of one of our customers, a company called azora. They're a SaaS company and they serve enterprise customers like Soul Foods Group, so like kfc, Taco Bell, all of that. And so we're going to be using them as an example to talk about kind of how they evolved into being able to offer AI as a service using the Firebolt platform. There's of course, other data platforms that are well suited for AI, but obviously I'm going to talk about our product and we'll, we'll hear a little bit about what worked, the playbook that they used. But first, just to kind of set the scene, what we want to begin with. And I know this is probably super boring because you've been sitting in these meetings all day, so we'll speed through it fast. But wanted to talk a little bit about the core problem with data infrastructure in the AI era, and that really comes down to latency. So even pre AI, you know, you wanted your applications, your reports, your BI data products to be as quick as possible. Obviously, if your data product returns results quickly, customers are more happy, you have better retention rates, all of that. But AI has sort of upped the stakes on that. So if you're thinking now in know, maybe people were content with like a 10 second load in a dashboard, that's not the case in the agentic world. And if you start to think about agents, like you've probably heard a lot today, they're spawning hundreds of queries at a time as they're doing their exploration. So speed has kind of become table stakes. It's not necessarily a differentiator as it was like in the batch processing or the streaming world, but now when we're dealing with agentic systems that are, you know, constantly querying, constantly iterating, constantly adapting, speed is basically non negotiable. Your users are, you know, everyone's played with ChatGPT and they've trained themselves into this mode of like I'm asking a question and the thing is talking back to me and giving me answers in real time. And if your data infrastructure, the underlying platform that has the basis for all of these questions, isn't ready and isn't able to turn the results around within milliseconds, well, you don't have a great platform. And so it's actually interesting because we were positioning kind of the value of Firebolt as a data platform for agentic infrastructure for like a year or so. But a couple months ago, think it was end of August, beginning of September, UC Berkeley actually came out with some research entirely independent of us, that was talking about what the new requirements are for agentic infrastructure. And what they identified is that there's really like four, is it four on this slide. Four unique infrastructure demands that you need to have when you're building out your AI platform. The first obviously is subsecond responses. You know, your AI agents need to be able to give you immediate responses and those responses need to be accurate. Obviously if you don't have fast responses and people are, you know, waiting a long time or the conversation is timing out, like that's not a great experience. The next interesting observation about the behavior of AI agents was around the pattern of queries that they're producing. So if you have an agent that's, you know, maybe operating in this like reasoning and acting mode where you know, you don't just have a linear path for your queries, but like it's generating a lot of different nodes and pieces of your graph, you're ending up with dozens to hundreds of queries and a lot of times those queries overlap each other. And so if you need a, if you're building out your agentic infrastructure, you need to have a data platform that is Doing something intelligent about caching. Maybe there's queries that you can reuse either fully or in parts or like subqueries. It's kind of one of the big accelerators for being able to serve potentially thousands of queries without also spending thousands of dollars a minute for your infrastructure. Obviously reliability, this thing needs to support the load that you're expecting from both agents and users. And then finally compliance. Well, we've kind of classified this as compliance, but what it really comes down to is like traceability. So being able to understand the metadata that is actually driving these responses and kind of tracking the process of the agent actually interacting with your platform and what it's, what it's saying. So being able to produce that, that audit trail. And you know, as I said, these, these things are not just like our marketing team saying, these are the four things that you need to do in your agentic infrastructure because those happen to be the, you know, four of the things that Firebolt does. But these are actually backed up by research from, from Cal and I, I can find the link and we can certainly share it afterwards. It's a really interesting academic paper that, you know, if you are into reading academic papers, I, I'd highly recommend. And so you know, this slide has some technical jargon talking about Firebolt and you know, you'll see lots of words about, you know, how we are specifically serving those different, those different core characteristics of an AI platform fundamentally, you know, mixed workloads. We're doing everything in sub seconds, not just AI agentic work, but your traditional reporting as well. You've got query optimization. Obviously the queries that agents produce are not always the best queries. And so even if you have something coming through that's suboptimal, you want to still be able to get that sub. Second result. I talked a little bit about the importance of caching for agentic workloads. And so in firevault there's a lot of, of overpowered query reuse, tiered caching to get you that faster, cheaper performance. And then finally indexing in Firebolt, when we talk about indexing, this is sparse columnar indexes. So it's really designed for the type of analytical queries that would be generated by an agent. So it's not like traditional postgres or MySQL style indexing where you're getting like point single records, but it's indexing designed for groups of rows. So again we're, what we've seen is that a lot of, a lot of organizations, a lot of, you know, companies that are thinking of launching their own data products. You know, you get, if you Google, you know, agent first systems, there's a lot of things out there that, you know, it seems like this is all a work in progress. But the cool thing is that, like, we already have a platform that meets all these needs and it's proven in production today. And so the use case I wanted to talk about from one of our customers is, as I said, is Zora. So they're an analytical platform for food and beverage franchises. So some of their big brands in the US are kfc, Starbucks, Taco Bell, Pret. If you're overseas and, you know, a single franchisee can have multiple different locations, all those locations have different systems, then you have delivery partners. Like, there's a ton of data that Azora is providing to these users. And so they had a couple of huge problems. The first is that answering any kind of business question took a ton of reports. Their legacy data infrastructure that they were running on was Aurora Postgres. And postgres is great as a transactional system. It's a little bit less great as a data warehouse, especially once you start having a lot of data. It's just, you know, not designed for efficient OLAP style queries. And so, like most people to get performance, they did pre aggregations. They were limiting detail, kind of constraining what their users could actually do. And so if you are a franchise owner, the unfortunate outcome of this is that you might not have the full picture. If you're looking to understand complex questions like, did last month's price change improve margins across all of my stores? You might not get an answer very quickly. And so Azura was ending up with essentially a state where things were a little bit more reactive. The results were partial, and sometimes the results weren't trusted by their users. So obviously, AI can help answer some of these more complex problems, but the very first step is fixing latency at the infrastructure layer. And so that's how Firebolt became involved with Azora, is they wanted to be able to have a platform that was able to handle the scale of the data, their response times, and be able to return data really quickly. So once they started deploying Fireball, you started ending up with much faster ingestion into the database, much faster query response times, and then more critically, a faster time to market. But obviously we're all here because we want to talk about AI. And so after solving kind of the latency side of things, so having a data platform that could serve those subsecond queries now is Zora wanted to try to figure out how do you make cost effective actionable insights if you're a client? So somebody who wants to do a complex piece of analysis, if you're not using AI, you're probably going to hand higher people. And if you ask a complex question of an individual analyst, our champion at Azora actually said what you get back are a bunch of questions. You don't get answers, it takes a long time and it's expensive. So their bar for conversational AI is that it's fast, easy, cost effective and accurate. And so in order to actually get this into production, because a lot of conversational AI use cases are still in like an early development phase, Isora plugged Claude and Firebolt together using the MCP server. Now, the little diagram that we have here makes this look super straightforward and easy. I think everybody listening to this session knows that that's really far from the case because LLMs are going to hallucinate. So, you know, they make up tables, they invent joins, the SQL doesn't work. You might have gone through the process of trying to hack around the problems, you know, putting schema definitions in the prompt, adding a bunch of different layers, but it ends up really brittle, really expensive, tough to maintain. And so that's why you bring in MCP servers, so they expose different tools to your LLM. In this case, the Firebolt MCP server has a documentation tool, it has a connection tool, and it has a query tool. And it's able to give all of that context directly to the model so that the LLM isn't guessing and it's actually working on accurate data. And so this screenshot, we've blurred out some proprietary info, but this is a direct screenshot of what Azura ended up building. So this is directly querying via the Firebolt MCP server, and it's generating answers to those complex questions that are grounded in Azure's real data. So this is asking about pretz egg salad and how a price change for pretz egg salad impacted overall margins across all of the stores that were under management. So you can see this is a pretty complex piece of analysis that, you know, the agent, this isn't a live demo, but the agent was returning responses in milliseconds and then continuing to think, generating more questions and eventually getting to an output of key strategic insights. So again, if you're thinking of cost effective use of your data instead of, you know, bringing in analysts, letting the data infrastructure do a lot of that work, you know, These are really good insights. Things about demand patterns, market forecasting, price elasticity, and then it even ended up giving some strategic recommendations. So the proof of all of this is kind of what Alan o', Neill, the CEO of Azora, told us. And I've got his quote here. But he said that they deployed this live in production to Soul Foods and the leadership team was chatting live with their data in a strategy session. So it's not a demo, it's not like a proof of concept or a side project, but actually in a boardroom meeting asking questions of data using a proven in production platform. And so, you know, accuracy for them was also one of the most important things. And that's part of what you get from the MCP server. Now, obviously I don't have Azora's application or their code, but I did want to show you how all of this works when you put it into practice. So I have a side project. Like most people in tech in the winter, I am a competitive ski racer. So I compete in some of our US Ski and snowboard masters races and then I do some internationally as well. But for my local race league, I've built a results app. You know, people upload, you upload the raw results. It, you know, gives you nice dashboards. But this year I was like, well, let's incorporate some AI and let's do that on top of Firebolt. Because I work for Firebolt, I'll learn some things. So this is the code that I put together that mirrors what we built with Azora, very intentionally. So we've got some connection information. I'm using AWS bedrock just because my application is entirely on aws. I like the flexibility it gives you for picking different models. But I picked Claude Opus for this. And then this is our Firebolt MCP server. So the MCP server is a docker container that contains all of the code about the documentation, the different tools, how you can work with Firebolt. So this gives the context and the grounding to Claude so that it can come up with accurate answers. And then I'm using LangChain and Lang Graph to chain all of these different pieces together in code. Again, this is very similar to what Azora or other customers are probably doing in production, is you want to have full control over your code, full control over how your agents are interacting with different tools, what's happening at each stage. Probably easy to do that using some of these open source frameworks and libraries like LangChain. So I've got my agent that I create. I expose the different tools and then I've got some handling in here for the text. And so let me spin this up. I think the other piece that's relevant here is just to get this up quickly. I'm using chainlit, so just a nice UI for building this sort of agentic experience. And so now I can ask it a question. So I'm going to say in the 2024, 2025 season, which five female racers were closest to Connor Carreras in based on average place? Use the results by gender table and let's see what happens. So this is a live demo, so things might go awry, but I'm hoping that this, this works the way I expect it to. So you can see that we are. And it did not work, but that's okay. But you can see that we are providing Firebolt, you know, with the information to make this query. We're then generating the SQL query and then Firebolt is saying, okay, this is the output that we want, but there's an issue with the table. So let me say use name, maybe that exists. Let's see if that works. I could of course check this in Firebolt. Well, clearly I don't know my data very, very well and I probably need to fix this a little bit. But what we, what I had been hoping to show you is those sub second query results, the perils of the live demo. I'm going to give this one more try to get it working and then I'll wrap up because I really do want you guys to see how. Okay, so name and then race rank by gender. Okay, Use race rank by gender. Okay. Oh, use SQL query execution. Well, apparently I'm, I'm unique, but you get the idea what we've, what we've, what we've been able to do here is connect to Firebolt, generate queries. You know, you are getting this Response time was 60 0.63 seconds. Even though, you know, perhaps the data that I'm sending through could be refined a little bit more. So you're getting that immediate response. You're able to see the audit trail to inspect and understand exactly where all of this grounding data is coming from. And then my takeaway is that I need to do a little bit more work on my side project here to get it consistently generating the correct results. But the way that we wanted to wrap this up is really by, by focusing on the, the latency side of data infrastructure. Latency is absolutely the most important aspect of this. It's, you know, attacks on growth, attacks on margin, attacks on Trust some people as they move to the AI world will keep paying this tax and then others who adapt, who bring in, you know, the AI, the infrastructure that's really designed for these AI workloads are, are going to eliminate it. And so we did want to share that. You know, we have an AI agent Infrastructure readiness scorecard, so something actionable that we'll follow up with. It's got four questions that you can ask your team, it's got benchmarks, and it's really meant to be a starting place so that you can, you know, understand either before you begin your AI journey. If you're already rolling out agents, you know, you can see where you stand. And then, you know, if you, if you want to try this with Fireball, you can bring us a high impact workload and we'll try it out together. But really speed isn't just a differentiator. It's like absolutely the most critical part of setting up data infrastructure for the AI world. And so with that we can open for some Q and A. Yep.

There's a quick question on. In the example and demo you were showing, was that a. Just to clarify, was that a Firebolt agent connected to Claude for data fetching?

Yes. So what I've, what I've built here is, it is CLAUDE is the LLM and then I've granted Claude access to the to Firebolt via the Firebolt MCP server. So that has actually three different tools inside it. It's got the docs, the connect and the query tool. And those contain all of the context about how to work directly with firevault. And then I can also give the application itself some additional context about like what the tables are if I need to. And that's what I've done in this prompt here.

Now the related question, the query results cache feature, what control does the user have to refresh it if needed? Is this automated or is there a user option?

Yeah, great question. So the query results cache, there's actually two types of cache. There's the sub result cache and then there's full result, both you can toggle on and off as you want. The subresult cache happens automatically. So when you run a query in Firebolt, if there are any aspects of it that our query engine or query planner detects can be reused. So it tends to be things like hash tables for joins or intermediate results, those will get stored in sub result cache and can be leveraged by other queries that use those same patterns.

Cool. Next question from the live stream is for founders, what do you see as the top use cases for talking to our data.

Ooh, that's a good one. I mean, it's so dependent on your industry. What we're doing internally at Firebolt is we've set up a Revbot AI agent that we've tied into Slack and it has access to all of the data about our customer base and our salesforce and our meeting recordings. And so that's really good for getting fast information and summaries about what's going on with our business. And so actually our founder and our CEO are often chatting with Revbot and saying, hey, you know, Revbot, tell me, you know, summarize the top five deals this quarter and you know, tell me what some of the risks are. Instead of sending an email to the head of Ops and then the head of Ops, you know, reaching out to the AE or the sales engineer, just being able to get that data immediately and quickly and summarized through an agent is definitely our main internal use case as a SaaS company ourselves.

Yep, that's a good one. Related to that, are there use cases where fireable doesn't make sense? I. E? Like, is it overkill, maybe for founders that are more on the startup side?

This is a totally fair question. So obviously the biggest benefit you get from something with Firebolt is when your data's big. But I think there's also going back to some of the terminology in the session about pain attacks. Like if you start small with something like Postgres, but then imagine you're like lovable and your product takes off out of the blue, you may end up in a state where like the cost of re architecting or even the cost of maintaining something suboptimal is really high. So I would honestly, I'm like of the opinion that start with, start with something like fireable, even though, you know, we are like our historic positioning had been for really large data because as you reach that like hockey stick growth that we're all working for, you'll start to see that it's way more cost effective than starting on something optimized for smaller data and then trying to scale it.

Yep, there's a related question on that answer. Does Firebolt have the ability to migrate data from postgres to make it more agent friendly.

So Firebolt is, I guess I don't totally follow that question all the way. So the Firebolt syntax is Postgres compliant and so even if you don't connect to the MCP server, if you give the context of like Firebolt is a Postgres compliant SQL database. I'd say probably 95% of the time you're getting good SQL that runs in Firebolt out of the agent. In terms of migrating data. Some of our customers have actually been using AI to build out their, their pipelines. I mean AI is everywhere. I don't want to recommend five coding migration pipelines, but you can. I, I hope that, I hope that answers the question. I, I don't know if it does.

Yes, for, for. Because I know this has been our most technical session today for. And I know we're also at time. A good related one would be for those who maybe have more technical questions for you. What's the. Because there's like another one that just came in. What's the best way for them to get in touch with you to maybe riff on that a bit more?

Yeah, you can find me on LinkedIn and just shoot me a message. Also my email is just connorirebolt IO so you can also shoot me an email.

Awesome. Okay, cool guys, just send your technical questions to Connor directly. She'll answer them. She'll help you get set up in Firebolt, if that's the ultimate answer. And with that, thanks Connor for joining us and walking us through that. And thanks for doing a live demo. We know doing it live is always a little sweaty sometimes.

Well I'm, I'm more, I'm more upset with myself because I swear you guys it worked 10 minutes ago.

I could see it.

But all good.

Thanks so much. We'll see Firebolt in London. So if you guys aren't coming to south for London, you've got a few weeks your left. We'll see you there. Connor, hopefully you'll be there but I know the Firebolt team will be there, so thanks for joining. Thanks everyone for joining our AI. Hopefully this was helpful. This is our last session of the day and we'll see you guys next time. Thank you.