#80 Harvey AI’s GC: Lawyers Are 20x More Efficient... Now What?

It might not be the case that the billable hour is dead. Maybe it's time for us to start thinking about whether or not that's the right metric for people to be thinking about in how they measure performance. When I am using a generative AI system, it's not always going to give me the correct answer.

Hello everyone, and welcome to Pearls on Gloves Off. I'm your host, Mary o', Carroll, chief operating officer at Goodwin. Today my guest is is John Labar, the general counsel of Harvey, which unless you're new to this world, is a company you should all be aware of. As a major leader in legal AI, John's career has traced the evolution of technology itself. Before joining Harvey, he was Vice President and Deputy General Counsel at Snowflake, where he led teams across commercial product, ip, privacy and litigation. Basically all the cutting edge legal challenges that come with building a data driven company at scale. He's also been part of the startup world at Zymergen and and spent a decade at Google where we had the good fortune of working together when John led the consumer hardware patents team for products like Pixel, Chromebook and Nest. He also ran Google's patent transactions group and was part of the winning patent litigation team. And finally before that, he was a patent litigator at Gibson Dunn. In this episode, John and I talk about the moment we're living through, when he compares to the early days of the Internet and why he believes that AI will be even more productivity enhancing than that first digital revolution. We get into how lawyers need to change the way they interact with technology, moving from expecting perfect answers to building a more symbiotic relationship with these predictive tools. We also talk about the realities on the ground, how his team at Harvey actually uses Harvey, how to measure ROI when the tech is still evolving, what adoption looks like inside companies versus law firms, and of course, the question on everyone's mind, what happens to the billable hour incentives and the business of law? So much good stuff on the topic du jour, so let's dive right in. Don La Barre.

I'm excited to be here, Mary.

All right, so you have been at Google. We were there together, then you went to Snowflake and now you are at Harvey. Are you the secret to a company's meteoric rise at all times?

I have been very lucky and blessed about where I've been. Obviously I think Google, you know, I think it was already on a meteoric rise when we joined there. Candidly, Snowflake was fantastic. I started there maybe when they were around seven or 800 people. And it was also right around the time that they replaced a lot of their senior management with Frank Slickman and, you know, Mike Scarpelli. They brought in sort of a team to, you know, get it ready to take it public. And it was so fun being at a company where you could just feel the energy and that trajectory, you know, going off in that way. And it was just an amazing place to be at that time. And then I joined Harvey after leaving Snowflake. I was going to do this thing where, you know, I took six months off and tried to spend some time with my family and then, you know, jump back into it. And before I even left Snowflake, I got connected to the co founders here at Harvey and just got sucked into it from the get go and have basically been here ever since. You know, depending on how you count, I might be Employee 6 or something like that.

Wow.

Here at the company.

Wow. Okay. So it's just hypergrowth at all times. You're at Harvey now. Everyone who's listening probably knows what Harvey is. It's really the biggest name in legal AI, which is all anyone is talking about these days anyway. What is it like to be, you know, in the center of legal AI and what's the deal with it? You know, we talk about, is it real? Is it hype? So much attention on this space. Like, how does it feel to be in that?

It's really amazing. And one of the things that I find most amazing about this, Mary, is like co founders Gabe and Winston and I were sort of talking at the very beginning, before ChatGPT sort of broke. Everybody was talking about AI all the time. And they had this, you know, insight that the way the world was going to develop, at least in sort of like the short and intermediate term, was that there were going to be these large language models, right? There would be OpenAI and there'd be Gemini and there'd be, you know, a handful of other anthropic, but that most companies were going to have the resources to sort of build their own and that what was going to instead happen is that there would be a whole host of companies that kind of grow up in like the AI application layer, right, Whose job it is will be to provide generative AI experience and features to the particular vertical that they're trying to serve. So that was their thesis. And that's what Harvey does, right? Like, we are a generative AI company, but our job is to make generative AI useful to the legal profession. That's what we do. That's what we strive to do. And obviously, you know, it's incumbent upon the AI, large language model makers. They sell AI. That's what they sell right here at Harvey. We don't sell AI like we sell a tool that does useful, hopefully useful things for our customers. And it's not about selling them AI, it's about selling them this tool that does useful things, that makes them more efficient at what they do. And I think, you know, very early on I was like, why are we selling the lawyers? The most tech resistant, you know, group that you could possibly think of to imagine. Right, Lawyers, still facts. Right. They're the last ones to sort of get on the technological bandwagon. But then when you sort of solve these models in action and you started applying those models to legal tasks, it immediately became apparent that there was some really good efficiencies that could be brought to the legal profession using this technology. That's why we're doing it. It's almost as if the law is one of the low hanging fruit. The first low hanging fruit is coding and then the second low hanging fruit is the legal profession in terms of like where you can show off a tremendous amount of benefit to these kinds of tools and what they, how they can help make lawyers more efficient on what they do. Yeah. So it really went from why are we selling to lawyers and like to the aha moment. Like no, lawyers really are the optimal or a optimal source for this. And it's not just a flash in the pan. Right. Back in the late 90s, the early 2000s, everything was about the Internet. The Internet was this new technology. It's productivity enhancing, imbued across the universe. Right. You didn't know necessarily if you were working for Google or if you were working for pets.com, but you knew you were in the right space, you know, utilizing the right technology. I feel like that's where we are right now with AI tremendously productivity enhancing technology. There's so many different business models and permutations about how you can deliver the service and make it useful. That's just a really exciting time to be in the space. And I think what we're trying to do at Harvey is just make sure that we continue to build a product that does what we're trying to do, which is make lawyers more efficient.

So many interesting things you've said. I agree. It is like the early Internet days and timing is everything and lawyers are sort of slow to adopt tech. Is the timing going to be now? There was a lot of talk about legal will be the first, you know, area that will be disrupted. We don't know. We'll see. It's starting, but is it going to take off? And we saw with the Internet era, you know, Instacart succeeded later. Webvan didn't succeed in the beginning. Everything's gotta be in its right place. Also interesting, AI as a legal technology is actually pretty different from other legal tech that produces results that are concrete, that you know that they're right. You can kind of count on it. Right. This is a very different way of working with technology. And that's something especially for lawyers who don't like the wrong answers and, you know, don't like any risk. How do we get people comfortable with working with a tech tool that is very powerful in some ways and maybe also dangerous in some ways?

Yeah, you hit the nail on the head there, Mary. Right. We've all been brought up with technology. The way I was brought up with technology was kind of like my calculator, right? Like I have a calculator and I punch in buttons and I hit the equal, you know, at the end. And I know with absolute certainty. So much certainty. I don't even think about it. I have the right answer. The reason I know it's the right answer, because I'm using technology, it's perfect. It always gets the right answer. And we grew up with technology thinking about it in that magical way. Oh, it's a machine. It doesn't make mistakes. You know, there are things like bugs and, you know, whatever, but it's perfect. And now here we are, you know, many years later, the state of the art in technology breaks that paradigm. When I am using a generative AI system, which is, at its heart, a predictive system, right? That means by definition, it's not always going to give me the correct answer, right. It's predicting what it anticipates the right answer to be. Doesn't really understand. It doesn't really know. And it's doing these tremendous calculations at tremendous scale in order to deliver these results that often mimic understanding, but it's not real true understanding. All of a sudden, we have to go back and retrain ourselves in the way that we are interacting with technology. And we have to learn that it really now truly needs to be a symbiotic relationship. Lawyers using generative AI can't just take the output, not pay attention to it, and then ship it off to the court. Right. I think we've seen enough examples of lawyers trying to do that. That does not work. That's not the appropriate Be using these tools. These tools can be, you know, can do incredible things, but you still have to review the product, you still have to massage it, you still have to work with it, you still have to validate it. And all those things require legal input to get right, but they can still make you more efficient in the process. And so that's kind of where I think we are. It's like we have to retrain our brains on how to work with this kind of technology.

I think that's exactly right. Even just thinking through using AI, you know, just general efficiency AI as well as legal AI, I find that the use cases that work the best right now are when you as the user knows the answer you're looking for. In other words, like if you're even just asking it to draft an email response to somebody, you know, using this tone and that, and it gives you something back if you already know what you're trying to say and you read, oh, that's perfect. Sounds just like what I want to do. Proficient. If it starts to say, you know, per your email, you refer to this date and this thing and you're like, ah, did it. You know, then it starts to not save you time because you've got to like relook and you know, basically check it all.

Yeah, I would reframe it slightly differently. I think the low hanging fruit for generative AI systems today is anytime you have a project where you need to deal with large amounts of unstructured data. And as lawyers, we do that all the time. There's doc review, there's reviewing, you know, contracts, there's all these kinds of projects that require legal insight to do. But anytime you have one of those projects, there is a tool and a way to use a generative AI system to help make that project more efficient for you. It can do nothing more than just finding the needle in the haystack for you and putting it up on the screen for you and say, does this look right? Or pull out all the data that you're trying to extract and put it into a table and you can validate that it's been doing that correctly. These are all different things that you can do to make those specific processes work for what you're trying to achieve. But at the end of the day, that's what this is really good at right now. And there are other things that it's getting good at. But what's the thing it does the best right now? It's that.

Yeah, yeah, okay. In terms of AI literacy and teaching the lawyers how to use this, know when they have to check it, interpret it, how to write that prompt to get the right answer, who's responsible for that? That's something that's still emerging.

It's definitely still emerging. And you know, I'm a litigator by background, so I always go back to like court analogies. Right. But you know, I would never submit a document to the court that I didn't read and validate myself. Right. Like that would never happen. It's the same rule when you're using these kinds of tools, you to review, validate and make sure you believe in the output here. But it can do a lot to help shortcut you getting to good output. Right. And so the job is always, you know, for the foreseeable future at least it's going to be the responsibility of the lawyer to validate the output. Right. That then creates this, maybe a chicken and egg problem. Like some lawyers are like, oh, if it sometimes gets it wrong, I don't want to use it ever. Right. But I think if you can get over that hump where it's. Yeah, sometimes it gets it wrong. Just like sometimes when you're like a senior partner and you give a project to an associate, they might not hit the nail on the head the first time out there either. Right. You need to work with them and you need to figure out the right way to communicate with them. There's a difference in talking to a first year associate than like an eighth year associate. Right. You need to figure out where these models are. Right. You need to play around with this technology to figure it out. There aren't any bright lines, no magic wand that you can wave just to make it perfect. But you start using the technology, you get comfortable with it, you develop your own innate sense of, oh, this is how I talk to it to get it to do what I want. Or I know if I give it this kind of question, the responses are going to be fine and I can, you know, give them a cursory review. I know if I am asking about this, I need to be paying a lot more attention to the output in order to make sure I know it's getting it right. Like those kinds of things, you just develop a sense over time. But once you get in that flow and once you start using really is this thing that you interact with all day and makes you more efficient, you get more done at the end of the day and you know, sometimes it means you're leaving earlier or whatever, but it's a magical experience.

Yeah, I totally agree. And I think you said it. We're all kind of playing around with it, getting better, we're getting better at it. So everyone is still, you know, experimenting and learning. That said, these tools are expensive. They're investments. You know, everyone's getting asked, what's the roi? And all our clients are turning to us on the firm side and saying, oh, you know, is this impacting our bills? Should we see our bills get lower? How should we all be thinking about this right now as the ROI question.

Yeah. And obviously the ROI question is different depending on your context. Right. The ROI question is different for a law firm than it is for an in house situation. On the in house side, in some ways it's maybe a simpler way to think about it. Right. It's really not much more complicated than I have a legal department and the budget for that legal department is X. And if I bring this tool in, does that give me the ability to do more with that budget? In which case, great, is it worth it? $300,000 tool, does it save me at least $300,000 in budget? Hopefully it saves you a lot more than that. Try to make sense.

You are the GC event company. You have your own legal team and you want adoption. You've got the tool. I'm assuming it's free for you all. So you've got the tool, you want people to use it as much as possible. And the ROI is clear. The more you use it.

Yeah.

The better. And I would assume having been on the legal department side, you don't need to incentivize people. I'm assuming people want to use it. They want it to work well. You know, the more they use it, the earlier they can go home. Is that right?

We use it all day, every day, but we don't use it all day, every day for the purposes of making our lives more efficient, which we do.

Yeah.

Do it because we're dog fooding or you know, drink your own champagne or whatever euphemism you want to use for it. But we're like dog fooding this product because we sit, you know, adjacent to the product team. And I'm always like, hey, we should be doing this, or wouldn't it be cool if it do that? Or like, oh, solve this and we need to do better. There we are constantly having that discussion. I did a blog post about a little over a year ago now called How Harvey Uses Harvey. And within our legal department, it's hard to quantify exactly, but we save anywhere from three to five hours a week per attorney with this kind of tool. And again, the place where I probably to this day, it's changing a little bit. Back when I wrote the blog post about a year ago, I would say the place where I saw the most efficiencies, when you're in an in house position, it's not uncommon that engineering comes up to me and says, hey John, we're going to change the way something works in our backend infrastructure. Do we have any customer contracts that will be implicated by that change? And so like in the olden days, by which I mean three years ago, right. The way you would have done that is you would have taken every single customer contract, put it in a row on a spreadsheet, we assigned the first 20 rows to the first attorney, the second 20 row, and everybody would have been pulled out of what they were doing and they would have been asked, you know, to validate this particular question across the contracts that they've been assigned to. Today I have a repository on Harvey of all of our customer contracts. And when engineering comes to me with a question like that, I go to Harvey and Harvey does the first pass at that question for me. Then I go in and I validate to make sure I'm comfortable that we've gotten the right answer. But what could have been 30, 40, you know, attorney hour project turns into a two or three hour attorney project. But those projects don't happen every day, but happen enough that it really is great savings. Now, a year later, I think the tool where we're seeing or the part of this where we're seeing a lot of additional efficiencies is our word add on. Now all of a sudden, our commercial attorneys, who make up the vast bulk of the attorneys on my team, we have our playbooks in there and we're integrated with word. And now all of a sudden you can bring the power of generative AI into that environment and you can be faster at doing what's usually a very long and intense thing like that first pass of the redlined agreement always is the longest one, and now you can do it with AI. And again, you're still validating it, you're still reviewing it, you're still working with it on a case by case basis, red line by redline basis, but you can do it much more quickly. That first pass might take four hours the old way, and you can do it in an hour now. And so we're seeing a lot of efficiencies there, John.

These are like classic CLM examples. Is AI disrupting the CLM market And I know you're. You're a ironclad customer, so how do you look at that?

So I think these are complementary things. Right. I don't know where the world is going. I'm a lawyer, not a technologist. Right. One thing that seems realistic and rational to me is that one of the things that these tools are going to be able to do over time is they're going to allow folks to bring all of the context together into one single spot where it then makes sense for the lawyer to be working on it. Right. So everything IT firms is a matter. Right. There's a matter number for everything. Right. Everything the lawyer works on is a project. Historically speaking, you work on the project by working, you know, in LexisNexis for this part of the project, and then you get the cases that you need, and then you work in Word for this part of the project, and you do what you need there. And then if it's. And you have the emails back and forth with the client and, you know, the relevant documents, if it's a case you're working on and you're sort of switching context the whole time, if you can bring all of the context together, and then the AI systems have that same shared context. Right. And then you talk about doing what needs to be done in the confines of whatever. Whatever it is that you're working on, that becomes a very powerful paradigm. Harvey isn't a replacement for clm. You would use CLM or a document management system in conjunction with Harvey and let Harvey help bring the context together. And then you have something really that's not just, you know, additively more efficient, but multiplicatively more efficient for the people who are using that service.

That makes a lot of sense. So then pivoting to the law firm side.

Yeah.

Going back to sort of the ROI and adoption question, how are you guys seeing or thinking about that?

Yeah. So let's talk about you here for a second. You were the pioneer or one of the pioneers in breathing life into legal operations. And the notion of legal operations, there are many lenses that you can look through when you describe, like what legal operations. So bear with me for my particular metaphor for the moment. A way you could describe legal operations is to say you've got a. A department of lawyers, and in the olden days, all the work that needed to be done by that department would be done by the lawyers, regardless of whether or not it was legal work or not. The aha moment for legal operations, you know, kind of can be summed up as Maybe it doesn't make sense for lawyers to do all the work. Maybe lawyers are good at some things and other professionals are good at other things. And if you bring them together, you have a more powerful department. So in the context of generative AI, in some ways the metaphor holds. Its job is to do the things that it's really good at so that the lawyers can continue to do the things that they're really good at and don't have to worry so much about the stuff that they're not so good at or not as efficient at or not optimized for. And so when you think about AI in the law firm, it's exactly about that. It's about freeing up the lawyers to do the legal stuff, right? Like the stuff that, you know, you go to law school to learn the strategy, business empathy, ability to sort of understand ethics. Like all those things aren't embedded in these models, they're embedded in the lawyers that use them. And you can use these tools to do some of this other stuff that historically speaking, lawyers were asked to do because somebody had to do it, but now there's a more efficient way to get it done. It really has the ability to sort of change, I think, how. And I don't want to sound like everybody else who's always predicted that this is going to be the demise of the billable hour, but I think this is another nail in the coffin maybe of the billable hour. Right. And it's a pretty big step leap, right. I think this is another opportunity for us to say, hey, is there a different way that we can be delivering legal services that still makes a lot of sense with a different paradigm? It's almost a little bit like the innovator's dilemma, right? Long term, bill by the hour. But if they don't pay attention to what's going on in the world around them, somebody's going to figure out a better way and they're going to get stuck behind that. I love this story. So we have a customer of ours who told the story to us at an all hands recently. They're using Harvey and they got an RFP from a prospective customer. I forget exactly what the RFP was. Maybe it was something like we want you to pitch, you know, reviewing a thousand side letters and extracting 20 bits of information from each of those side letters or something like that. And they told the firm, our customer, we're going to go to two other law firms and we'll probably go with the lowest cost provider here. And so our customer said, okay, Great, thank you. Would you mind providing us all of the side letters now so that we can properly scope the rfp. And then, you know, later that afternoon they put those into a project and they had Harvey do a first pass at providing an answer. And then they went through and they validated the results. And then the next morning they sent the result to the other side. And they're like, what is this? They're like, it's your rfp. Here it is. The lawyer who was telling this said he's never used the word flabbergasted before in describing anything. But he said the client was flabbergasted. Right. They were like just totally amazed that we were able to turn what they thought was 100 hour project and turn it around overnight at no cost to the customer. And so the moral of that story is when that client had real matters, like substantive matters, they were so impressed that this law firm was forward thinking about how AI can improve the practice and deliver more efficient results to the customers, they started to get business from them.

Right.

And so I think we're in a stage now where we need to be thinking about how these tools can provide, you know, more efficient service to our customers. And I don't think that means it's not a zero sum game. I don't think we're living in an increasingly complex world. I don't think that's going away anytime soon. I don't think the work is going away anytime soon. But I think the way that we deliver it and the way that we can be efficient about delivering it, that's on the table here and now. And we should be thinking long and hard about how these tools can change those things.

Yeah, that's such a great example. What excites me about that is that there's multiple parties involved that can all win and it's not always about reducing cost. And that example is great and kind of what you walked through. What is the term I always use is right. Sourcing. Right. You think about the work and like, how do we match it? Is it technology? Is it an attorney, is it a non attorney? And we constantly want to think about client value. So there's certainly some areas where efficiency and technology and AI can reduce the cost, but there's other areas where technology and AI can actually add value. Because you couldn't do this before. You know, if you had a hundred hours and a bunch of humans, you couldn't get to the insights in that volume of data by your deadline or whatever is happening or, you know, I'm.

Guilty of this now that I'm on the other end of it. When I was in private practice, I can't tell you the number of times the client came to us and said, we need this in three hours, we need this in five hours. Right. So you do as much as you can in the time allotted. Now if you had a tool like Generative AI that you could apply to that, you know, limited time constraint project, now all of a sudden you can deliver more in the constraints of the time that you have. And so it's not just about delivering it more cheaply. Sometimes it gives you the power to get to greater depth on the answ in the amount of time or whatever other constraints you're working on to provide that answer to the client.

Yeah. When I also think about roi, I've been saying this for years. So I have the six measures of legal sort of ROI revenue, accelerated costs, avoided cost, reduced hours, reduced quality of work and quality of life. And that last one is actually really important.

Right.

Because we think about the ROI on an investment for a law firm and do you really want your attorneys working on that kind of like routine non value add time? You probably have to write off, write down. They're not getting any sort of professional development out of it. It's not why they went to law school. And if you're able to allow them to do the big picture thinking, interacting with the clients, using that industry knowledge and empathy and all the things, you're going to be able to attract better talent. And that's the product. That's the product of the firm, right?

Yeah. And you need good quality associates at every level, good quality lawyers at every level in order to do that. And if these tools can take away some of the drudge work, the non interesting legal work, and make that more efficient, that's just going to lead to better outcomes across all six of those dimensions.

So I do ask a lot of firms when they're talking about their AI tools, I say, what are you guys doing to encourage adoption? Because the billable hour paradox still exists where, you know, there's an efficiency component, but then there's also, I've got targets and this is how we make money. Are you seeing any changes? Are you seeing any conversation about that from your clients who are law firms?

My law firms are not calling me up and saying, hey John, what do you think our billable hour requirements should be this year? That's not a conversation that I'm privy to. But I can tell you it might not be the case that the Billable hour is dead. Maybe it's time for us to start thinking about whether or not that's the right metric for people to be thinking about in how they measure performance. I don't care how long you work, I care about the quality of the output that I get. I'm willing to pay the same for it if it took you 100 hours or if it took you 50. If it's still high quality output, it. I'm willing to pay for it. So stop making this artificial quasi arbitrary factor that's actually kind of misaligned with my interest as the client. Right. That doesn't really work for me. Now, I am quite familiar with the adage, you know, the billable hour is the worst way to deliver legal services to folks other than all the rest. But I actually don't even think that's true anymore. We should start having more conversations about success based billing or other types of ways that you can sort of get at the same thing that are better aligned with the interests of client. If you can find better alignment with the interest of the client, that's a good thing for everybody, right?

Yeah.

Billable hours, the worst that I can think of at that. I just can't think of a better one sometimes.

I agree. And I think that's the challenge. And we've been saying this, you know, for the last two decades, we need clients and law firms to have aligned incentives, but nobody knows how to price the work properly. And so we're getting there. I think that education on both sides, there's been eroded trust because of the former model. And so trying to get to what a success fee or value based or outcome based or subscription or what have you. Yeah, it's just going to take a lot of partnership and conversations.

Yeah. If you hear about things, if the wind of change is blowing, I think that would be a super interesting topic for one of your future podcasts.

Yeah, I think we talk about it a lot. At the end of the day, that's where it has to go because it's a win, win, win situation. You know, if we can get to a different type of billing structure that makes the law firms be more efficient, that lowers the price overall in probably most cases for the client. So that's a win for the client. We start to do things more efficiently. We talked about why that's a win for the attorneys. And if we are doing it right and leveraging our knowledge and our experience and our technology stacks and adding value to the client, then we should be getting as a law firm Better margins. And so everybody should come out ahead if we all do this right.

Could not agree more. Yeah.

What's the path to really awesome AI? We are still in this. It's good at some things, it could be better at others. I think that's just where we are in the state of development. But this is not a linear, you know, improvement trend line. It's sort of like steps up and up, and you just never know when the next step up is going to happen. What are the opportunities in the next three to five years? What is it going to take to make it look truly transformative? And how far are we from those big steps?

We talked about this in the beginning, like, output needs to be reviewed. But I think the bigger theme here is we need to make sure that we're talking about the right thing and that you're selling the right thing. If you're a generative AI company that is tailored to law firms. Right. The conversation really shouldn't be about, hey, we're going to sell you AI. The only people who really should be talking about selling AI are the large language model providers, because that is literally what they sell. What we're selling is a tool that makes lawyers more efficient. That, oh, by the way, happens to be powered in the back end by AI. So AI is helping us do the things that we're doing in trying to deliver more efficient processes to the lawyers. But it's not that we're selling them AI, we're selling them these tools. The focus needs to be on how are you building these tools, what are they going to do as these tools start to sort of slowly walk away from mimicking the interface of ChatGPT and sort of like the open, you know, ask me anything box, and instead start to permeate, like, specific workflows for doing this or workflows for doing that, where you sort of have an interactive session with the AI that iteratively gets you to the right output that you're looking for. If you're setting up those workflows the correct way, you're actually going to get to the work product that is reviewed and been validated by the lawyers because of the way those workflows were set up and the way you sort of interacted with the AI along the way to get there. And this is what I mean. We're like, we're symbiotic with the AI with these tools, and that's the way it's going to have to be for the foreseeable future. The models themselves will continue to improve and this is again, lawyer Punta advocating about technology. So take it with a grain of salt. But you know, I personally don't think that AGI is going to happen this year or next year. The transformer model, which is the sort of the insight that powers generative AI today, was a great thing that happened back in 2017 or 2018. That's when Google published it. I still think there's one to three more breakthroughs that have to happen like that are of that scale before we really get to something that's truly understands what's going on. Like an artificially intelligence that understands things and can prognosticate based on what it understands about the world. Right now, I think all we're doing is building a better prediction engine, which is great. And that can mimic. It could do a lot of wonderful things and it can predict a lot of great things, but it's still a prediction engine. It doesn't have true understandings. I still think we're a little bit away from that. And then, you know, again, not to anthropomorphize these things, but in some ways they are more like interacting with a human than interacting with a machine. I had this interesting discussion with somebody not too long ago and what they wanted was a system that would take a look at all of their historic litigation and tell them in the context of a new case what their likelihood of success in winning was. Right. They were saying, oh, we should use generative AI for that. I'm like, I actually don't think generative AI is, if that's what you want. I don't think generative AI is the right tool for that. That's really a statistics problem. Right? Like you want to build like a neural network that does, you know, that sucks up all that information and sucks up all the different variables and then you give it a new situation. It will predict the outcome based on all those different variables. That's not what generative AI is doing. Generative AI is more like, go ask Mary o' Carroll what her opinion is about this case. She's really smart, she knows a lot about litigation and she's going to have an opinion about maybe that court or that jurisdiction and this area of the law, and she'll give you her opinion on it. It's not going to be scientifically rigorous the way like a statistical model might be, but it still might be more valuable than the statistical model. Right. We need to continue to realize that these systems don't necessarily converge on a Magic 8 ball that does everything for us. Right. Like you're still going to be interacting with these systems on the generative AI side, kind of like the way you interact with these two humans in a way. And I want to hear your opinion, but I might not say, oh, it's definitively right because you know, I take everything a human tells me and I sort of filter it a little bit. Right. Versus if it's my calculator, I'm like, oh yeah, that's right. I know it's right.

No, I agree completely. Sometimes I think people use the term AI just to mean automation and technology and analysis and data and all those things. But I have another question, which is you guys are selling to both law firms and in house teams and I kind of wonder in this world, I always hear the it would be great if your tool could do this one day. This is the nirvana. Are the dream goals of the in house lawyers versus the law firms that you're talking to. Are they very different? Like what they would love this system to do one day?

The answer to that question is yes. We have, you know, a lot of internal feedback that we collect from all of our users about what they would love, what would make it. It's literally like you said, it's all over the place. A lot of times it's, oh, I wish it better integrated with some third party system, or I wish you had more knowledge sources, you know, in this jurisdiction or something like that. And also, you know, I wish your, your vault system could do this way instead of that way or something like that. So those things are all, I'm going to call them technological challenges. They're all within the realm of possibility through some engineering time at it. If we got the third party to sort of align with us that this was a thing to do, we could make those things happen. They're all tractable things. Then there's a whole nother class of things that people talk about. A good example of this is something I want. I keep telling my product team, I would love an AI agent that handled NDAs. My dream. And I don't think we're that far away from this. But my dream is one day I will have an agent. I will say, here is my NDA, here are the things I care about in an NDA, or here's my NDA playbook, or whatever you want to do, right? Somebody else has their NDA agent who they've also privately told it, like, here's what I care about, here's our playbook. And then those two NDA agents could go off and they could come back with an NDA. It would be one of two things. It would be. Here is an NDA that we think should be mutually acceptable to everybody. Right. Based on your parameters. So then all I have to do is review that real quick and hopefully I'm done. Or it would say, oh, this is as close as we could get. But based on your parameters were still unresolved on the choice of law provision or something like that. Right. And then instead of spending 30 or 40 minutes negotiating an NDA, which is not everybody's favorite thing to do, you would then have these agents do that. You spend five minutes on it and you're done and you moved on. Right.

Yeah.

We need to figure out how to make that work. That's not a nut that has been cracked yet. It's within the realm of crackability, but we need to still get there. There's feature requests and then there's a. Wouldn't it be great if AI could do this kind of thing that gets layered on top of it and there's no end to what people think AI should be able to do for them?

That's very true. I find it interesting, having worked with so many in house teams, it's always contracting. You know, contracting is the biggest pain point. It's what most teams are. The volume of the work is contracts.

And it's. And it's frictionful too. Right. By the time you're in contracts with somebody, there's kind of generally alignment that we want to give it to you and you want to use it, and it can still take weeks, months. And so you're in a place with both businesses are like, let's do this, let's do this. And the lawyers are doing their job, but sometimes it takes a while. A lot of it is just. I like my way of saying it. You like your way of saying it. And if we could get agents to sort of do that for this, using AI and using intelligence about the process and what we care about, what the other side cares about to come up with a solution. You almost think you would converge on things like, this is like AI would almost invent for itself. Like, here's the common, you know, universal NDA. You could imagine that those things would start to sort of spontaneously organize themselves. This is the universal SAS agreement. This is the universal, like, whatever agreement. But the NDA would still reflect the things that I care about or the things that the other side cares about. I think that's a really interesting place to get to. And I don't think you know what we're seeing with agents today, it suggests to me that we're not that far off. And part of it will be a little bit of a leap of faith. You have to, to trust these systems a little bit. But I think the technology is maturing enough that at least for NDAs, maybe not an NDA for an M and A or something like that, but like, we're just sort of a general, let's have commercial discussions type thing. Yeah, I would give that to an agent to do, assuming I could validate a couple of them to make sure that it was doing it right. I would be pretty trustful of that kind of situation.

It's funny, like, in my experience, trying to get four lawyers on the same team in the same company to agree on common language is hard enough. Like maybe an agent can actually do it and break the ties.

Yeah, it's the hard part. And then the business needs to be there to sort of say, yeah, no, we're good with that. If the business people always watched the legal wrangling happen, they would get very frustrated because there is a lot of that, oh, my way's better. No, your way's better. But if the legal is saying like, no, let the AI do it and if it works, we're done, they would love that. It would remove friction and it would allow lawyers to work on more interesting parts of the work that we do.

That's right. I think the law firm side, which you guys are tackling head on, is so much harder because there's not a common use case. You think about all the different practice areas, all the different industries, all the different, different things that we do all day. And it's not routine, as in house.

Definitely not. Right. That's kind of the, the magic of law firm though, right? They are there for all the other stuff that isn't routine, right?

No, that's right. Trying to figure out how to apply AI and what are all the use cases for each team. A lot more to figure out, a lot more to learn and structure there is. All right, John, this was super fun. Always great to catch up and this is such a fun, meaty topic. I really appreciated your time.

I really appreciate you finally inviting me on the podcast. This is your first podcast, like what, 10 years now?

I didn't know you wanted to be a guest. This is your first podcast ever, right?

This is my first podcast ever, but I can't imagine anybody better to share that experience with me.

And now I'm sure you're going to get invited non stop and you will wish that you never came back.

Thank you so much, Mary.

Great to see you, everyone. Thanks so much for listening. We'll catch you next time.