SaaStr AI Day Live: Why 95% of AI Pilots Fail with Flatfile

Lazarus, VP of AI Engineering at Flat File. He's leading a really great session here, taking a cue from some of the headlines in the news, but double clicking and going quite a bit deeper on why 95% of AI pilots are failing. In about 20 minutes we're going to go into Q and A following Nico's presentation. Please, you know, put your questions in the chat and we will get to them at the Q and A portion and without further ado, turning it over to you, Nico.

Hey, yeah, welcome everyone. I want to talk about why 95% of AI pilots are failing, or so we think. And this comes from a report from MIT where they talked about the divide within AI, the divide of AI within businesses and this kind of, some of these, some of these findings that they, that they wrote about in the report got surface as headlines such as 95% of generative AI pilots are failing. And this has been pointed out frequently as maybe we're in a bubble. And even today I saw a, an article about this 95% number and I want to encourage everyone to go, go ahead and read the report because it paints a very different picture. Some of the key findings, some of the interesting findings are around the actual wide use of wide adoption of AI systems within these enterprise companies. But the failure is on the custom built enterprise initiatives. So just as a background on the report itself, they analyzed 300 public AI initiatives. They surveyed 150 organization organizations and they interviewed 50 AI leaders around those organizations across those organizations and they found that 80% of those organizations piloted and used tools like ChatGPT and Copilot and 40% of those actually went into production. Meanwhile the like inter the enterprise grade systems, the custom built systems, the vendor sold systems, only 60% of those organizations evaluate those types of Systems and only 5% of those reach production. So the site, the failures here, the findings here as due to brittle workflows, lack of context and misalignment with day to day operations. But I think it actually leads to some really interesting takeaways from the report itself. So nearly everyone that was surveyed in this report was already using AI in some way in their work already. This points to a gulf between like the some consumer ready AI solutions and these enterprise ready solutions. So the challenge here is around the scale up and around the actual successful deployment of this within the enterprise itself. They also found that internal builds for enterprises were twice as likely to fail as something that might be built out with strategic partnerships with startups, with, with organizations that are fully focused on the success of these systems. And then they also found that context is still the bottom bottleneck to scaling. This is in contrast to a common belief that, you know, the technology is just not good enough yet. The infrastructure is not, is not good enough yet. They found that context, knowing what's useful to the enterprise, to the organization is actually the most important bottleneck. And this leads me, leads us to something that I think all of us believe is like the failure to pilot these AI programs is a capability problem and not a technology problem. I think that's why we're all here today is like learn how to be more capable with these systems and to scale it up for these larger organizations found. I found this chart from, of the survey from gain, which points to some interesting findings as well. Something that you can see here is despite a lack of, despite executive support for these AI initiatives, companies weren't, weren't ready. Organizations were not ready, are not ready to scale up to use generative AI and pilot them successfully. So we can take away from that what you will, but I like to bucket these until four different systems, four different themes. So the systems are not ready, the products are not ready, the people are not ready and the data is not ready. And rather than give you like a playbook of how to do these for each one because I think they're unique for each organization, I'd rather talk to you about how Flatfile has approached these different buckets and how we're thinking about how to scale them up and like the lessons that we've learned along the way. The first thing that want to talk about is the systems being ready for AI usage. What we've seen with LLM tools with AI is a large number of long running concurrent connections and this has put a huge load on our systems. How do you scale that? Super well. This is a different model from traditional request response systems and it requires a different architecture. So we've shifted our architecture, our systems to something that will support long running agent loops and we put a lot of intention and design behind how we think about scaling our architecture. The other thing that we think about is how do we, how do we scale up our different features in a way that's customizable but also also like within the context of our new features. So we think about how do we build once and deploy everywhere. This means a strong use of containers, but also release management and we bring in AI when we can to scale that up. The second thing that we think of at flat file when we think about shifting our mindset is the IS product. So what we what we saw in the report is folks already using consumer grade AI tools to get their work done. But that hasn't, hasn't happened, hasn't scaled up yet to enterprise organizations. And I think one of the things that we're thinking about is how do we bring the aha moment to these AI systems sooner and faster to our customers. So this has led to a shift of how we think about product. We, we've redesigned our first experience with users. We completely overhauled our onboarding flow and our configuration flow so that folks can have a quicker, faster enablement process with our systems. And this meant being shifting our our strategy to being a little bit more product led. This also means supporting something that's a little bit more hybrid self service and enablement model so that folks can can serve themselves when when they join and first interact with Flatfall. But then we'll have, we'll have folks in the loop to help enablement to help enable them and help improve their their success. We can talk about how we've thought about up leveling our our, our people internally, how we've improved the the muscles here at Flat File to give them more familiarity with AI systems. Right now familiarity with, familiarity with AI and agentic systems is just table stakes to build in an AI product. And so that means getting everybody used to building and getting everybody familiar with these concepts. So we've had a number of on site sprints where we get our reps in, we have hands on keyboards, we're building out agentic products, agentic tools internally internally so that folks have that full intent cycle of building out a out a feature or a product that is useful for the organization. Since I started I've been to five in person extended agent sprints or extended hackathons, another way to think about them. And that's really enabled our team to get familiar with these concepts. And I went from being like the first AI engineer to now the entire organization is capable of building AI features that so much so where our design team, our exec team is regularly shipping agent products. Lastly, I want to talk about the data flow or the data shift. So at Flat File we think a lot about the speed of data. How do we get it from one format that might not be useful to folks to another format that is built in and useful to your workflow. That's, that's what Flatfile does. We're a data migration platform platform and we make it. We focus a lot on the challenges that our enterprise customers will see with with messy data with hard to Reach data with unstructured data. And so we think a lot about this problem and how we might shift this for folks and on, on the data side we've shifted our mindset to looking for flat, for flywheel effects within data. So synthetic data might lead to the initial use case, initial custom model for, for folks. But that will allow us to, to get out there and get adoption, get usage which allows us to see where that model might fall down the the edges of what's capable for that model allows us to now generate better synthetic data and so on and so forth. Like will continually improve our models based off of what we see in the market. That leads us to a demo I'd like to show. So this is one of our AI features. You can see here that we have a dashboard with a bunch of prospect records and you can see some of the philosophies around our agent building. Our AI building first is to give users control and maintain user trust. And some of this comes from what's built into our product. So validations showing where things fall down, where things are not matching what's expected. So you can see where the red is showing some of those guardrails and inside of this, this is quite messy data and we want to clean it up quicker. And so this is where some of our AI features come into play. You can see that we can give it open ended instructions and then run a transformation on these columns and it allows us to create transformation easily in the context of our product. What's what we, what we see here is like a strong focus on, on usability, like on. On something that's extremely useful to folks in inside the context of where they're already working. I love this feature because it's, it's simple, it's straightforward of what the problem they're solving is and it's not, it's not outside of the context, it's not just another chat window. So this allows us to like clean up and improve the messiness of this data, the quality of this data and gives us, gives our users the ability to reproduce this again and again allows them to improve their processes to improve their existing workflows. What you can see is our philosophy on repeatability in context, in context improvements and then just the simplicity of some of the problem solving that we have. I want to move on to some of the key learnings from where we have been with the report, but also with what we're seeing as we build out more AI agent features. Firstly, AI features will get stuck without product led growth. AI is just another advanced form of problem solving where product needs to drive a lot of the adoption. Also, context is everything embedding directly within a workflow. Existing workflows is the best way to get the best results, but also means a lot of the technologies that are coming through the rag. Memory reasoning is all about how we manage context and that is still the bottleneck today of how we scale up well and how we improve the results for our customers. This means strong emphasis on context management, strong emphasis on control over what gets included, strong emphasis on memory and existing workflows within our agent prompts and then moving on. Like. We lean heavily on our enterprise partnerships. So this means getting grounded within our, within our build out by working with and partnering with some of our strong with some of our enterprise customers. This means going back and forth with what's capable versus what's useful, talking through some of the solutions, but also shining a light of what we can do tomorrow and then allowing us to grow together to build a solution that's more impactful to their workflow. And lastly we think about and we leverage a lot of our flywheel effects. So what's what we can do today with our data allows us to get down the road and further and improve our models and improve our data tomorrow. But that also applies to our culture, our systems and our people and our product. Everything that we've done today will allow us get down the road and scale up faster. That is it for the lessons. I think now it's time for Q and A. If any of you folks have questions.

Yeah. And Nico, great, that was super interesting, thank you. And so folks please feel free to add any questions into the chat. We will address them here. I do have a couple questions coming in from the live stream. A question, a little bit about, you know, data being AI ready versus just clean data. Do you know, could you go a little deeper into what is the difference? Nico?

Yeah, so we think about data in on two sides, right? One's ready for our data or our AI systems and one's ready for our enterprise customers. And that's how we like, that's how we like we encourage like a lot of our data migration is around the speed of data there. However, there is a big emphasis on curation and what might be included for custom models and custom and custom AI systems. I think it's, it's these are two different lanes, right? Like one's like speed within the organization and how we might help customers get data ready for their own systems and their own use cases. But then there's also how do we curate and, and improve data for our AI models? And we, we have different, like different techniques for either once. For one way we look at is our evals and our benchmarks for what's, what's a good data set for our AI models. And then on the other side we think about, you know, what's the right, what's some of the validations around the actual system itself and how do we, how do we guard and like how do we get to a result that is better for that AI, that use case built into enterprise workflow. What I mean by that is how do we think about the success case for our customers?

Makes sense. A couple questions coming in here. One from Abdel. Do you think the software layer will be replaced by agents provided the data puzzle is solved?

What was that last part?

Yeah, do you think the software layer will be replaced by agents provided the data puzzle is solved?

I don't think so. I think that software is still key and I think what everything that we've seen in the market right now is pointing towards the application layer being the important piece of the puzzle right now. So you, you look, you can look at some of the foundation model labs and how they're building out more application and more software for folks to use. I don't think that software is going anywhere anytime soon.

Another question here from Rick. What is the best practice for instructing users to verify the AI results to handle inaccuracy and hallucination?

Yeah, so we still have manual like Human in the loop previews. Right? So that's, that's not going away anytime soon is to review data is still part of the process. That's part of how we, part of our philosophy within flat files is still to have humans in the loop reviewing and ensuring high data quality. This data is critical to enterprise success and critical to enterprise users. And we can't just have automated AI systems that might hallucinate, that are non deterministic, without quality control, without inspecting. However, we think of AI as a, as an augmentation to get to a place where you can review quicker.

Okay, so, so, so Human in the Loop is a best practice. Really? Yeah, yeah, makes sense. We got another interesting question here from Gopal about what are the success success metrics? Are you, what success metrics are you using for your AI efforts? Do you measure this?

Yeah, there's a variety of success metrics but we, we do and we have a few different AI features that all have their own metrics. But I think overall it's it's customer adoption. Customer usage is our main driver, main success metric.

Okay, there was another question here in the live stream about a gartner prediction that 60% of AI projects unsupported by AI Ready data will be abandoned through 2026 through next year. Do you agree? Do you think that the failure rate will improve or get worse?

So I saw that report as well. I wanted to include that in this talk. I think that points to some of the things I touched on in this session. But I, I, I think, yeah, I'm pretty optimistic and I know it will get better.

Okay, so you think that it will improve, not get worse?

Yeah. We're at a moment where a lot of these systems are being built out and the data shift. Is an important piece of the puzzle. And we're going to get to a place where a lot of these enterprises will have that enterprise ready data for models.

Got it. Okay, I got another question here from the live chat. How are you controlling that your AI costs do not get out of control?

Yeah, that's a great question. We track our token usage, we track and trace all of our calls. We do a lot of deep dives into what might go and spin a lot of tokens and then we solve for that. We have logic in our platform that allows us to adapt and control some of these costs. A great example of that is model routers. That allows you to go to different models depending on costs and depending on complexity that's needed.

Nice. Another question here from this one from Alex. What is the benefit versus cost of having internal AI systems versus off the shelf or outsourced solutions?

What's the benefits or costs of using internal solutions versus external solutions? Well, internal solutions are going to be customized to your workflow. The risk there is the maintenance cost, but also the adoption of it. So as you can see in the report, there is a big drop off of what's been adopted within these custom builds. You're twice as likely to fail if you go at your own route. But working with strategic partners, you're way more likely to develop solutions that are more successful and more adopted. However, you're going to have more control and more customization. So my advice would be a hybrid approach of partnering with strong organizations that will help you get to somewhere where it's custom to your workflow, improving your workforce, but also not just, but also allowing you to like increase the adoption within your organization. Because maybe for whatever reason, stronger products and stronger product growth allows you to unlock some of the adoption within the wider organization.

I think we have time for one more question here, and this is maybe a good one to end on from. From Rick. I think this relates back to flat file a bit. When helping your customers migrate their unstructured data, do you also help them improve their systems and processes to improve how the data is created and organized in the future? That's something. Do you have any thoughts on that?

Yeah, I think we think deeply about data migration. So it means end to end. We aim to support every, every format we can. So we have a lot of different supported file types and input types, but we also work with our customers to inspect their workflows and improve their systems along the way. So as things change over time, also means our systems will adapt to your data needs. And. And yeah, it does improve over time.

Okay, great. Well, Nico, thank you so much for taking all of these really great questions. Listen, if folks have other questions for you or to get in touch, what's the best way to. To get in touch with you, Nico?

Yeah, no, thank you, David. And the best way to keep in touch is reach out to me at my email. Nico, plat file IO and yeah, say that you saw me at Saster.

Okay, great. Well, again, thank you everyone for joining this session. AI Day will continue over@sasterai day.com if you're going to join more sessions, recommend you head over there. And yeah, thank you everyone for joining the session. Thank you, Nico, for all the great insights and helping dispel some of the myths out there.

Yeah, of course. Thank you. Yes.

All right, thanks everyone. It.