5 Things That Are Working and 5 Things That Aren’t in B2B SaaS AI with Ironclad's CEO and a16z

So then the next one is around fully automating roles and agents. This is what everyone wants, right? But we're not there where like end-to-end processes are automated again yet. And there's a tweet, I think from Yann LeCun back in February that was saying, ask not what LLMs can do for planning, ask what planning can do for LLMs. The LLMs are not good at planning, right? They're good at predicting the next, step. And as part of that, you can't rely on LLMs to do the end-to-end processing yet. But what we are seeing as a flip side to that is there are LLM-powered processes as part of people's roles.

I think no matter what interactions you've had with the legal industry, would you rather have an AI lawyer or would you rather have a trained lawyer who's assisted by the best AI paralegal in the world? I think everyone would actually, if the price was equal, choose that the trained lawyer assisted by the best AI paralegal and this combination of human and machine will beat the machine only.

Please welcome Jason M. Lemkin, co-founder and CEO at Ironclad Inc., and Sima Aymbol.

Hey y'all, great to be here.

Yeah, thanks for, thanks for joining us. So we're gonna talk a little bit about 5 things that are working and not working yet in AI for SaaS. Maybe quick intros. Yeah.

So I'm Jason M. Lemkin, CEO, co-founder at Ironclad. We're a contract management platform, which means we help businesses with the entire lifecycle of their contracts. Could be sales contracts, vendor agreements, you name it. We're helping them make them and keep track of all the information on them.

And I'm Sima. I'm a partner at Andreessen Horowitz. I focus on early-stage investing and have been doing software investing for the last 15 years. I actually met Jason at one of the Ironclad rooftop law schools. Was that what they're called? Yeah. In, I think, 2016. And both of us have gone to law school, so nerded out over that. And it's, it's always fun to be back together.

Yeah. And excited to be here today. What we're gonna try to do is really talk through just some kind of like field notes on AI, what's working. What are we seeing, um, in with the lens that we have, and then some of the things that aren't working that I think are still to be figured out by the industry and hopefully we can get there together.

Yeah. So you'll get like what we're seeing from talking to early stage companies and then the like founder company builder experience as well. All right, let's dive in. So we're gonna start off talking about what's working. The first one is using AI to help with onboarding. Um, so I'd put onboarding along with customer support, which is two of the use cases you've probably heard a lot about. We're seeing a lot of companies have success getting— using AI either through tools they've built themselves or through SaaS vendors to help users understand a new behavior or use a product for the first time with the help of AI. So to put this concretely, there's a whole set of presentation software, AI-driven presentation software that's been Growing really quickly in the last year or two. And this category actually, I wouldn't say struggled, but it was growing much more slowly. It was harder to get folks to get used to like an AI-generated presentation. Once they started building out templates and the onboarding flow via AI themselves, the, that new user behavior was a lot easier to learn. And that same thing we're seeing across other categories as well. And a lot of times our companies are using SaaS vendors to help with that. Everything from honestly the demo all the way through the like support. And so onboarding is definitely one of them, but like it takes a lot of the, the learning and the like coaching and professional services that has to go in through the implementation period. And AI is able to help with that.

Yeah. And so something that drives me crazy as a founder is this just effort required to get a customer set up and like successfully using the platform. Um, and for us, like there's a lot of that. There's discovering the legal team's preferences, like how they like to negotiate a contract. When they see this language, they like to respond with this other language that's in their playbook. And the way we have done that historically is deliver professional services and scope it and ask a bunch of questions. What we're seeing early progress on is this ability of AI to help actually discover preferences, ingest a bunch of contracts, understand what the legal team we're working with likes to do, and then actually help implement that in the software so that when they go through a workflow, their preferences are already set up. I think early days, but extraordinarily excited about this because one, it can help expand the category. It can help expand the number of customers that we can successfully serve. And bring down our cost of service as we are a growth stage company, Series E+. We really wanna look at those unit economics and I think this can really help with that.

Yeah. And I think implementation and getting your customers live on products faster is a piece, is a, is a journey that a lot of SaaS companies struggle with. And if you can get it, if you can lessen the burden on yourselves and get the help of, uh, something that is learned off of, I get many other customers and that customer itself, why not use it? Yeah. The next point we want to talk about is what we're calling automating basic manual work processes. So what does that mean? One of the things that AI has obviously done very well is around like transcription, note-taking, and anything around this language use. And so what AI has done a very good job of, and I think helped a lot of SaaS companies drive usage and engagement with our customers is around this feature set. So anything from someone like Mailchimp using AI to auto-generate email copy, Zoom doing summarization notes, Dropbox having a search feature— these are all things that you guys are probably using, but it takes away that extra time that you would have spent, and it's just naturally part of the workflow. And so it's not really asking for something that's a big step for the user to really change, but it just makes your work a lot easier. And so that's where we're seeing it a lot, especially on the horizontal side. On the vertical SaaS side, a lot of— we're seeing a whole set of new companies that are popping up around these sort of basic manual work processes, because if you're a, a vertical provider, so if you're providing software to vet clinics, for example, there's a lot of time spent like transcribing notes, taking inbound phone calls, summarizing the notes, and then data entry, and then auto-filling out forms. Now you can do all of that and not need to hire someone necessarily, or hire 2 people, and you can just hire 1 to do that same work. And it's, again, it's very seamlessly fit into, into the workflow. But we're seeing a lot of vertical SaaS companies grow quickly with that wedge?

So I love that this is in our working category because to like rewind for a little bit, we started Ironclad in 2014 as ironclad.ai. And the basic idea around this was companies have lots of contracts, they don't really know what's in them. What if we used AI to read the contract and build a database? That turns out that didn't actually work in 2014. And so we had to go a different direction and build all these like workflow tools. But this is actually working now, and it's a special moment for me because we take our customers' contracts, we run them through AI, and we extract the key data points and put them in a database. Um, it's a great application. I think we're seeing it across industries, and it actually works now, 10 years later.

Yeah, no, and I, I think part of the reason of saying why is it working, I think goes back to my point around usage and engagement and love of your product. I think we're seeing a lot of SaaS companies that have successfully just added this into their feature set, like just yet either sell more seats or find, uh, more, you know, daily usage of their product. So the next point is around staying on the main branch for core model capabilities.

Yeah, there's kind of two approaches that we're seeing, at least in our space. One is you go to a foundational model provider, and again, this is for the use case of building AI that understands legal language, legal concepts. You can take the base models, GPT, and fine-tune them and build additional infrastructure like RAG around them. Or what the foundational model companies seem to be selling is the ability to build a fully custom model. And this would be something like GPT- stripping out the Reddit training data, adding in Delaware case law data, and having a, a model that understands Delaware case law better. I think what's interesting and what we found that, that maybe I didn't expect is staying on the main branch, staying on the core capabilities and supplementing it with fine-tuning and additional techniques like RAG seems to be the approach that's winning right now. And it, A good example of this is like GPT-3 hallucinated a lot of case law and was pretty scary. And you definitely didn't want to use the results from GPT. GPT-4.0 plus some fine tuning, plus some RAG on a Delaware case law dataset, a lot better. And if you went off of the main branch and built a custom model, you lost the benefits of that. So I think that this could be something we come back next year and say isn't working anymore, but. Right now, I think staying on that main branch has a lot of benefits because you're getting the, the rapid improvements to the foundational models as they're delivered.

Yeah. And I think one thing that we're seeing is this tension between behind like how much fine-tuning do you want to do? How much investment in the model do you want to do as a SaaS company versus just focusing on like the core company building for us as a SaaS company? And I think to Jason's point, where we are right now, there's still like, you still have to go through all the challenges and opportunities of building a SaaS company around go-to-market, product, et cetera. And so sort of making your life easier and getting most of the benefits of AI by staying on the main branch is where we are right now. And so that's like what we're seeing resonating with more of our companies. So the next thing, what we called was like leaning into vertical data and creating proprietary comment, but I'll let Jason kick it off.

Yeah. So this is really about figuring out what the vertical concepts are and building AI to understand those concepts. So again, with our touchpoint in the, the legal industry, we want AI that understands legal concepts. And it, I think this is one of the areas I'm most excited about, the applications that we're seeing that were just totally not possible before. To, to give a kind of practical real-world example, I'm sure y'all have negotiated a contract before or worked at a company that has a particular way that they like to negotiate contracts. And what happens is people basically get stuck. One company has their template that they like, the other company has their template they like. How do you morph those together? And it— redlining is the legal equivalent of doing a diff, comparing two documents. What happens if you compare one company's template versus another company's template? It's just all red. It's all in conflict. But really what you're looking for is the concepts and how different the concepts are. So if AI understands the vertical concepts, the legal concepts, you can actually do a conceptual redline, which compares the positions in two documents that otherwise would be completely red. And this is— groundbreaking. It helps attorneys negotiate contracts, multiples faster. And it, I think, is, is some of the really early applications of generating real business value. We're working with customers who are seeing tens of millions of dollars in attributable business value to speeding up attorney workflow, doing things in-house that they used to have to send out, out-house. And it, that's where we've put the effort as opposed to trying to like build a custom model is understand the legal concepts and feed those in.

Yeah. The legal, I spent a lot of time looking at sort of the intersection between professional services and AI. We've seen this a lot with legal, the legal field and legal precedent and ingesting data. Another area is accounting. I met an accounting AI company a couple weeks ago that had said they tested GPT-4.0 against their benchmarks and it was like 20% accurate. They then tested Claude and it was like 50 to 80%, and then a fine-tuned version of GPT-4.0, they were getting 90+ percent accuracy. And so I think to Jason's point, getting that level of accuracy is also super important in accounting. It's super important to the law. And so getting that context. From the vertical data is what really distinguishes, okay, I'm just using ChatGPT and unable to ingest files or that I have and data internally versus getting that fine-tuned model.

Okay. So yeah, we wrote this one before. Have y'all read the Founder Mode post? Do we know what that is?

Good show of hands.

Okay. That's, that's less than I thought actually. So most people have read the Founder Mode post. We wrote this before founder mode, and I do think that this is totally working in AI specifically. I actually don't agree with all of founder mode for what it's worth. I was at the YC event where that happened and I loved the passion and intensity, and I think that the core is something I do agree with. So we're, we have this like enterprise product, 9 figures of revenue, growing great. GoodClip, a lot of the company oriented around that. But as a founder, I'm like, I, there's a standalone product here that is an AI paralegal, AI assistant that's more of a general purpose standalone application of the technology we've built into this enterprisey product. And I pulled 5 of our best engineers and we just got in a room and pretended like we were a different company. And people were definitely like, what is Jason doing? Like we have, we are like stretched thin on resources. Why is this like room cordoned off and what are we working on? We had like a code name for the project.

More room for, uh, the new product.

Yeah. Um, but once we were able to like demo that and it have customer interactions that were positive and then connect those to opportunities we had. For our core products where this could be an accelerant. It really started picking up momentum. And so I think like getting the ball rolling as a founder is something that the founders in the room can uniquely do. And I would encourage you to, uh, and read Founder Mode if you haven't. Um, but this is kind of the concept we're, we're talking about here.

Yeah. And I think just to highlight something that Jason M. Lemkin just brought up, something that we are seeing a lot of our founders do is especially if you're launching an AI product, a standalone product, or it's not just like the feature top-up thing, summarization or autocomplete, but it's really a standalone product. You really do need that like founder level engagement and drive to push the product forward and really make it a center of the strategy as opposed to it being like the table stakes stuff that you have to have. 'Cause every SaaS company today is an AI company, but if you're really going to differentiate yourself in terms of your sort of AI-native product, you do definitely need the, like, founder-driven energy. We're going to move on now to what's not working. The really important here— thing here is the yet, because I think both of us are strong optimists, uh, on AI. And I think, as Jason M. Lemkin mentioned, in a year, if we came up here, I think we may have different points of view. And hopefully a lot of these things will actually be solved as the foundation models get a lot better, and, and then I think also how the user interacts with AI will change. But we wanted to give a state of how we're seeing things as they are today. So the first thing was going broad to narrow. What does this mean? What we are seeing is a lot of companies who are building AI-first SaaS applications start to narrow down either on a use case or a market versus being more horizontal. And I'll give you two, two examples of that. One is we've seen a ton of workflow automation come up in the last year because it's a very, it's a very tangible way to use LLMs to power SaaS. So that could be anything from filling out forms, doc extraction, et cetera. All of that is very powerful when powered by LLMs. We saw a number of these companies that all started on a very horizontal basis, and over time they've narrowed down serving specific industries.

Why?

Because partly it's a go-to-market motion piece, but partly it's also getting into the integrations within that vertical, understanding the user and exactly how they interact with the product and what pieces of data are most useful. And so that's actually I think, you know, it's instead now we're seeing people go narrow first versus trying to have a really broad application. I think the other piece too, another application of being narrow to, to begin with is now with AI, teams can be building their own software. Why buy an outside software vendor? That said, I think I talked to a finance team at a Fortune 500 company that you guys all probably know. They were originally gonna use a very, a very broad general use case AI product, and then they ended up realizing, actually, I, I need to just go with the narrow product that's set for finance because that's, that's really the way to go.

Yeah. My take on this one is, if y'all have seen the Gartner chart on the hype cycle where it's like the peak of lofty expectations and then the trough of disillusionment. Follows shortly thereafter, and then like a steady climb out into real-world application. I do think we are at the peak of inflated, lofty expectations in AI, and there's a, for good reason, there's a lot of excitement, but I do, I think the trough is coming and what we got to do to get out of the trough is find the repeatable value in AI. And it, I, I think there's a lot of work to do. We have 60% of our customers that have adopted AI. We're pretty far along in progress, but there's so much to do. And our best tool as an industry for finding that is focus. So I, I think this broad is going to be an issue and the way out is focus.

Yeah. And I would say it's, it's not just, it's not dissimilar from past like internet cycles, right? Often things, people try the most broad version and then they realize, okay, we actually probably have to start narrow to actually get, get adoption and get it into people's hands, especially as it's, it involves new behaviors. So the second thing is offering self-serve for the enterprise. And I think Jason has some specific anecdotes on this. Yeah.

I feel like my, my hot take on this one is PLG. Like how, how, how much did we all hear about PLG the past few years? I think my big learning from this is not everything needs to be a PLG product. And if I could go give myself advice 3 years ago, it would be go figure out a way to sell more million ARR repeatable deals. And that's your path to efficiency as opposed to try to tune the unit economics exactly correct on PLG. And I think this is playing out in AI products as well. The self-serve enterprise can be tough. Lean into the bigger deals.

Yeah. And I think just to marry that back to, we talked about onboarding. So how do you square the fact that there's AI used for onboarding and self-serve? There's onboarding help, which is like replacing or augmenting the implementation versus self-serve. And I think we're, you're still seeing enterprise-level products being sold versus like individual self-serve PLG products, but the enterprise version is getting assisted by AI. So, that's how I might square those two points. Okay, so then the next one is around fully automating roles and agents. This is what everyone wants, right? But we're not there where like end-to-end processes are automated again yet. And there's a tweet, I think from Yann LeCun back in February that was saying, ask not what LLMs can do for planning, ask what planning can do for LLMs. I really liked it and I think it got a lot of likes, so I wasn't unique in saying this, but the LLMs are not good at planning, right? They're good at predicting the next step. And as part of that, you can't rely on LLMs to do the end-to-end processing yet. But what we are seeing as a flip side to that is there are LLM-powered processes as part of people's roles. And I know Jason's seen some of that at, at Ironclad. Yeah.

I think no matter what interactions you've had with the legal industry, would you rather have an AI lawyer or would you rather have a trained lawyer who's assisted by the best AI paralegal in the world? I think everyone would actually, if the price was equal, choose that the trained lawyer assisted by the best AI paralegal and this combination of human and machine will beat the machine only.

So I think rely— I think related to that point, relying on the quality right now for a lot of AI-generated content for complex operations. Let me break down what that means. I think related to the fact right now we're not there with fully automated agents yet again with AI, I think for complex operations you still need a human in a loop, at least on the checking side, if not again, the human driving the operation right now. And where does this apply? Maybe 12 months ago, you all remember having AI-generated images where there would be a third arm in the human or a sixth finger, right? And I spent a lot of time looking at AI and marketing tools and no, no brand manager would approve an AI-generated image. Those hallucinate The hallucinations around appendages, the quality of images have gotten a lot better, but I think things around law, accounting, things where determinative answers are required and there's a complex sequence of operations where again, you can't, you can't afford to have the hallucinations and you need a right answer, otherwise you're gonna lose the trust of your buyer among other things. So we're not there yet. Again, I check in in a year. I'm, I'm sure there'll be many more use cases that weren't working now that aren't working now that will be then, but Yeah.

And I'd, I'd just like add on the unconstrained machine is, is not there, right? Especially for knowledge work. You don't want 5% of your contract work to be wrong, especially if you don't know what 5% might be wrong. Huge cost to that. And having the right human checkpoints there is still the best way to do things and getting ahead of those can be problematic.

So the last thing is around figuring out pricing and monetization. And this is something that's very much still an ongoing work in progress. Um, I'd say right now there are generally like 3 modes where people are incorporating AI into pricing and monetization. One is it's part of the core product, and that's usually if you're, you're trying to drive additional usage from your users or drive more seats. So Dropbox, it's like AI is part of the product. A second is like an upgrade, so you If you're offering additional features, you're a, you have an, another plan that allows, that has more AI-generated content, like MailChimp does this. A fourth, a third is it's an add-on, totally add-on module. Notion, I think right now has a $10 a month per person add-on where you can get a lot of the AI-generated features for Notion. And that's really suited for a power user. And I put the fourth bucket of where we're still very much exploring, which is like the, you've changed the paradigm around pricing altogether where you're paying for resolution or the, the solution to the problem versus just the service itself or for on an ongoing basis. So Intercom has its chat support, or sorry, its support agent that is called Finn. It's actually charging based on resolution. And our growth team, just to put in a plug, has written a great post on this that that I'm borrowing a lot of this content from. You should check it out if you want more. But the, the thing that I'd highlight here is every company that's looking at AI pricing has not really figured out which of these four, or maybe something else, is working. And that's, as you guys all know, for pricing is often a journey that every company every year or two is going back to the drawing board on. But it's still very much which of these buckets is successful? How do you drive value and How do you drive usage both from the AI?

Yeah, if any of you all have figured this out, let me know. I think it's exciting that we get to work on this together, but the variety of approaches is amazing. We're trying them all. We're iterating, we're experimenting, and it's one of the things I'm excited about working in AI right now is there are big problems out there with no clear answer yet.

Yeah. So that's just a little bit of what we've been seeing from the many companies that we interact with and from Jason's, uh, experience at Ironclad. I think things are changing and so we're excited to also see what this will look like in a year. But, um, those are just some of our observations in the field.

Yeah. Thanks for being here. It's great to be here at Saastr. Have a great one, y'all.