EP 140: How AI Will Transform The Business of Law

This is the Everyday AI show, the everyday podcast where we simplify AI and bring its power to your fingertips. Listen daily for practical advice to boost your career, business, and everyday life. One thing I always think about is what's going to happen now with lawyers and attorneys and just the law in general. Now that generative AI is getting better and better, are we still going to pay these really high legal fees? Is, is law going to become, you know, legal representation, going to become affordable and more accessible? I'm, I'm excited to actually have someone that can give us great insights on today's episode of Everyday AI. Welcome. Thank you for joining us. My name is Jordan Wilson. I'm your host. And if you don't know, Everyday AI is a daily livestream podcast and free daily newsletter helping all of us, all of us everyday people learn and leverage generative AI. So we're going to do that and I'm very excited to talk about today how AI will transform the business of law. I'm very excited for you all because we have a great guest with just extensive background in this field. So before we get to that, as we always do, let's go over the AI news and if you're joining this live, let us know right now. What are your questions? What are your questions? Get them in now so we can tackle them and answer them live on show. That's one thing about everyday AI that's a little unique, is we take your questions live and get them answers by, answered by experts. So, all right, let's jump into the AI news because there's actually some big, big things going on today. Let's start with Samsung. They're making some historic moves. So the electronics giant just launched a generative AI model made for its own Samsung devices called Samsung, called Samsung Goss. The presumption is that the Samsung Goss model will come to Samsung's popular smartphones such as the Galaxy S24. And looks like they're probably going to beat Apple to the punch. And the first to bring generative AI to a major mobile device, Samsung Goss is named after the legendary mathematician Carl Friedrich Goss. I think I'm pronouncing that right. And includes language, code and image functions.

All right.

Samsung's not the only big tech name making some moves. Amazon as well. Y' all have heard of that, right? This, this little company called Amazon, I think they sell books, but they've, they're entering now the large language model game reportedly. So they are working on their own large language model codenamed Olympus. According to a recent Reuters report. So Amazon is investing millions in training a large language model called Olympus with 2 trillion parameters, which would make it one of the largest large language models to date. Amazon has obviously already been investing in a lot of other companies and their large language models, specifically a reported $4 billion. That's billion with a B. Yes. Investment in Anthropic and their large language model. Claude2 all right, last but not least, Meta is requiring advertisers to disclose AI use in political ads. So a freshly released announcement from the Facebook and Instagram parent company is saying that this new policy applies to Facebook and Instagram ads and requires advertisers to disclose when AI generated or altered content is featured in political, electoral or social issue ads next year. Obviously, this decision comes as lawmakers and regulators are preparing to address the use of AI in political ads. As we get closer ahead to the 2024 presidential election. My only comment on that is like, what took so long? We've seen leaders from both parties already use AI generated ads, so I'm wondering what took so long. Like, we know that these shouldn't be or it should at least be disclosed in political ads, right? All right, but you did not come here to talk and chat about the AI news. Maybe you did, but you're probably wondering about how AI is going to transform the businesses of, of law. I'm extremely excited about that one. And for that, let's go ahead and bring on today's guest. Yes, I'm super excited to have Enam Hawk, consultant at Law Beta. Enam, thank you for joining the show. Appreciate it.

Yeah, yeah, yeah, no problem. It's early in the morning here, but we're ready to go.

Yeah, he makes, he makes the elite list of people that have joined the live show from the West Coast. So thank you for that, Enam. Maybe just let's start high level. Give everyone just a real quick background of, of your career and, you know, kind of some of your experience in law so we can kind of set the stage here.

Oh, of course, of course. So I spent the first 10 years of my career at a law firm in New York, you know, doing corporate finance. Spent the next maybe eight years or so at Moody's debt rating agency where I was studying legal risks and incorporating some of that into the ratings platforms and their own analysis. And we were doing a lot of manual work, but really turning this unstructured documents into structured scores, things like that that investors could use. So I spent almost 20 years doing that and really just I'm the type of lawyer that actually does not, has never been to court. You know, nothing like, you know, fancy like that. Like, not the ones you see on tv, more or less the negotiating lawyer, the one on the phone. But what's interesting about that, that experience was that I really just only did one type of law my whole career, and then I subsequently did the same thing at Moody. So I'm really, really focused on leverage finance, leveraged loans, and corporate finance. And that's sort of my legal background. But I've been a technologist pretty much all my life. It's kind of my passion, my love. I've been into it as far as I can remember. Maybe 5600 baud CompuServe, Internet in 1988 maybe. So it's been a while. And now I'm getting to. Now that the law and sort of artificial intelligence, machine learning and these technologies are sort of collid. It's everything I love now all packed together. So that's kind of the skinny of it.

It's.

It's.

It's great. And I, I kind of feel the same way. Right. Like, I've always, even for me personally, I've always been kind of a dork and I've always been fascinated by the creative side and the tech side, new Internet innovations. And, you know, for yourself, with your law background, I mean, maybe let's even talk like, how did this culmination of events and kind of your creation of Law Beta, how did that kind of all play out?

You know, the seed for it, honestly was probably in high school when I had done A. In 1997, I did like a whole presentation on a legal. A law firm that was going to be online. It's going to be like law.com, something like that. And so the seed was definitely planted there. I kind of had an idea that I was going to be a lawyer for a long time. I think what really accelerated all this for me personally was probably Covid, to be honest, being in New York City, sort of being trapped in an apartment for a year, I decided to really lean into that sort of area of my life that I hadn't had time to really focus on. So it was like learning Python, taking, you know, refreshing. Taking like Harvard's free CS50 class, like doing all those kind of things and getting, getting back into it a little bit, just because that was my passion. It's what I actually like to do. And so that that sort of culminated, obviously with perfect timing about, you know, these large language models and, you know, the paper from 2017 and then OpenAI putting it and everything just sort of came together. So that was, that was basically it.

Yeah. And just, you know, generally speaking, what is kind of the focus that you're doing with Law Beta, you know, is it more helping, you know, others kind of understand the kind of AI and the technical side of law and where it's going?

Yeah, yeah, yeah. And on a broad level too, I should say, because I know you've had a few speakers in the past maybe touching on law. So my particular perspective is really through the prism of sort of elite law practice in New York and you know, firms like internationally, things like that. But that's just one segment of the law, you know, that I happen to be specializing in. But I think if you look at it from all the sides, I mean, these technologies are going to improve the whole landscape. So for me, yeah, it's definitely sort of getting, I mean, lawyers have historically not been tech savvy like these firms. And some things, and I'm not going to say all lawyers, because there's an incredible ecosystem of lawyers, at least right now too, that are just gunning for these technolog pushing, you know, full pedal to the metal. But historically, if you look around, the IT departments were kind of lagged behind. There wasn't a huge investment in this stuff. Machine learning tools have been around for a while on Wall street and they pitched them to firms and rating agencies and things like that, but it didn't really hold, you know, it didn't, it didn't come through. And I think what's great about this time, and I think you can appreciate this too, is that, that I think, honestly, Chat GPT has just captured the imagination of people to be able to interact with it, to use it. You're not talking about having to go tr document or a system or something like that, but just to kind of see what it is is just incredible. And I think that's really the spark that was needed to accelerate the adoption of these technologies, which is my whole goal with LA Beta is to try to really accelerate the adoption of these technologies. I'm eager to work with partners and with clients that really want to take advantage of these systems and not just get to the middle ground, not just get to where everybody's going to be, everyone's going to have a large language model of some kind in their, in their firm and their systems, in their law practice, whatever. But it's almost like my brain space is always about, okay, and then where do we go? Right then what can we do? How can I Differentiate ourselves? How can we differentiate ourselves? How can we gain a competitive advantage here? How can we start using all these data lakes?

Are you still running in circles trying to figure out how to actually grow your business with AI? Maybe your company has been tinkering with large language models for a year or more, but can't really get traction to find ROI on gen AI. Hey, this is Jordan Wilson, host of this very podcast. Companies like Adobe, Microsoft and Nvidia have partnered with us because they trust our expertise in educating the masses around generative AI to get ahead. And some of the most innovative companies in the country hire us to help with their AI strategy and to train hundreds of their employees on how to use Gen AI. So whether you're looking for ChatGPT training for thousands or just need help building your front end AI strategy, or you can partner with us too, just like some of the biggest companies in the world do. Go to your everyday AI.com partner to get in contact with our team or you can just click on the partner section of our website. We'll help you stop running in those AI circles and help get your team ahead and build a straight path to ROI on gen. Yeah, and I'm. Oh, I can't wait for the rest of today's conversation. I have so many questions. And hey, as a reminder, everyone joining us live, Jay saying thanks for joining, joining us. Enam, thanks for joining us. Tara saying good morning from Nashville. Val, good morning. Rodrigue, good morning. But yeah, what questions do you have about kind of the, the future of law and, and what this means, kind of the business of law with AI? I'm going to start here. I'm going to start here. Let's go to the, let's go to the end. How do you think this is ultimately going to play out? Because I think, you know, I'm like, for, for good reasons. Right? Like, you know, probably big companies that are, have big, you know, law firms on retainer are probably hesitant to, to use large language models, but I know that there are even, you know, specific large language models now that are being developed specifically for the legal side. So how do you think this is all going to shake out? You know, not asking you to predict the future, but how do you think this is going to shake out with large language models becoming so popular, so powerful and so niche that can really perform well in a certain vertical?

Yeah, I mean, I think this is still just the beginning here, to be honest. And all the capabilities that people are dreaming of and actively building the whole ecosystem of legal startups Right now is incredible, and it's really just the beginning. But I would think that I think we're sort of overestimating what's going to be done in a year or two and underestimating what's going to be done in a decade here. And so I think that these technologies are just going to find, like, we can mark the calendar and talk in 10 years, but this is going to be everywhere. These, these technologies are so good at sort of getting rid of. And this is a theme that I think you've had with many of your guests on the podcast. But that, that gunky work, that, that stuff just feels like where you don't have a lot of autonomy, where you don't feel like a good, you know, employee or you're not having a good career because you're just kind of stuck in this. And especially with law, in this, in the world of what I call the world of small print, you know what I mean? Like the, if you ever go to a sports venue or something, you look on the other side of the ticket, there's like font. And three, that's the lawyer's world right there. You know, like these little, you know, things or, you know, you get a beach ball the other day and it said like, do not use as a flotation device. You know, like, okay, so there's a story. There's a story there, obviously, but there's also like, that's, that's the world that they live in. It's very, you know, unfortunately, I think with lawyers, they're very risk averse and they're also always talking about risk. They're always thinking about what could go wrong. That's why documents have ballooned in size. I've done studies just to see like, where a credit agreement started 20 years ago to today. It's just adding words, more and more words because it's thinking of all the hypotheticals that could go wrong. And you're going to be really glad if something goes wrong, that you had a lawyer there and you had all this stuff really prepared. But for the most part, some of this is not going to ever come up. It's all theoretical. And the best lawyers that I encounter and the ones that will, I think, use artificial intelligence tools and get creative and become sort of market intelligence platforms themselves. They're the ones that don't give, you know, a client. Okay, here's 100. Theoretical risk of everything that go wrong with the deal. They boil it down with their judgment, their own strategy, their own sort of experience to 10 items, 20 items, the ones that actually matter. And so I think that that's really where I think the practice is going to shift, at least from like sort of the corporate, you know, negotiation strategy, transactional angles. Sure.

And some, you know, speaking of some of that mundane work. And yes, we talk about that a lot on the everyday AI show. But one thing at least, you know, I'm an. I'm an outsider. Right. I have no clue. But, you know, when I think of lawyers, I think that there's obviously many different types, many different roles, you know, in. In law firms, but I think that they're spending a lot of time reading, you know, related case law. I think they're spending a lot of time analyzing it. I think they're spending a lot of time writing based on their analysis of the reading. So when I put those things together, I say, and I think, can't most large language models, and we've seen the testing, can't they consistently do that at the level of the best lawyers out there? I mean, is that the case? Is that not the case? Help. Help us all understand that?

I mean, I think, I think it is the case and that a lot of that sort of analysis about, you know, like, the work I'm doing right now is about, like, how can we sort of ingest these things and have the computer or the system sort of spit out that top 10 issues, list the top 20 to understand somehow the risks involved in there. And yeah, a lot of the work is that. But I will say that freeing up from that sort of rote, you know, you don't want to look through 300 pages of an agreement just to really care about the 10 pages. And lawyers spend a lot of time sort of getting there. To that point. You just kind of, kind of get to that point. A lot of legal documents, for the most part, are sort of boilerplate. You know, they're not negotiated. You don't have to deal with it. A lot of it, I would say, like 80% of it, the fireworks are all in that, you know, the 80, 20 principle. Same thing applies in law, right. It's the 20% of the words that actually have the most impact. And that's where imagine now you're free to just focus on that. It's being served to you really elegantly by a. By a system. And then you can just spend all your brain space there thinking about that, not worrying about sort of how to summarize something or extracting information or trying to find outliers or precedence or, you know, you're doing a lot of control f to find things that are all that's going to be gone away now. You'll have time to really think about. I mean, I actually think this is going to improve the practice a lot, quite a bit because you're just going to have. Right now everyone's kind of doing their own thing. Like, I know there's a big movements now and I love to see this like one NDA like everyone's trying to move to like, let's get to like forms that we can all agree on and get rid of all these, like millions of, you know, sort of firms that are forms that are out there. That's a step in the right direction for sure. But you know, it's just, it's that kind of work about like, okay, get rid of that and let's focus on the real stuff. Like a lawyer really should be involved. And again, I'm doing this all from like a corporate lens. But what's, what's, what's the company you're representing, your client? What industry are they in? What are the hot buttons there? All the research that goes into what kind of problems have they encountered? Not just like legal is not a vacuum. You know what I mean? Richard Troman's from Artificial Intelligence is a good friend of mine, holds a lot of conferences. He just did this great think piece about how just legal tech is not a vacuum. And at first, you know, I read it and I'm like, well, that doesn't seem that insightful. But if you really read his piece on LinkedIn, it's, it's, it's profound because it just means like this, legal is never just on its own. Right. It's always tied to a client, to an industry, to, you know, a government regulation. Whatever it is, it's so large that it'll, it'll have sort of tentacles everywhere. So you can't really just think about it in a vacuum. And lawyers will no longer be in a vacuum when all these tools are available.

Yeah, and, and it seems like to me that there's the, the very infamous case, you know, a couple of months ago of the, the New York attorneys who submitted essentially information to the courts that contained hallucinations from Chat GPT. So they, they use Chat GPT to help, you know, write a summary or a brief, submitted it and it turns out that they weren't using chatgpt correctly. It hallucinated it lied to. Do you see that as a potential ongoing problem? And also did that incident in its high visibility in the earlier parts of kind of the gen AI boom. Did that set the legal space back and you know, people were too afraid, man.

I, I heard a rumor that somebody thought that that was a plant by the New York Times just to slow this thing. But you know, I will say this. First of all, I don't think, I think that's all going to be solved. Right? They've already, if you look at ChatGPT today, it's got sources, it's doing real time data analytics on like actual found, you know, truthful foundation matters, ground truths, things like, things like that. That'll all be sorted out. I will say too, because I have to give an ode to the engineers that I work with a lot because I really try to study these technologies, not to program and engineer them, but to understand the theoretical limitations of them. I need to know what they can and can't do for my clients. And one thing the engineers always tell me is like everyone calls it hallucinations, but they call it creativity. Right. And you do want that system to be able to be creative, to give you ideas. You know, people think like, oh well, if it's trained on all this data, just going to know what it already exists, but it's really not the case. And again, this is delving a little bit into, I don't think this is a hot take, but it's delving a little bit into like, well, these things are like prediction machines. They can do probabilities, they're very good at that. And they can take, you know, all that to language. But you know, just remember that we communicate in language. It's the most effective way other than in person communication with language, probably the most effective way to communicate. But also like our thoughts themselves are just language right in your head. Most people have a little voice that talks to them that they think is them. So there's a lot of things about what patterns and predictions it saw there to recognize that, to be able to interact, you know. And you can see like OpenAI's announcements even from Monday. I think they're thinking the future of computing is not going to have one of these anymore. Right. I think it's, they're really leaning towards, you're just going to, you know, it's going to be like Star Trek, you know that you'll just talk to your computer and all that stuff. So it's definitely where I think things are going to head with that hallucination issue and all that. It probably was good for the space to take on a little bit of a break because I felt like law firms particularly were starting to panic. You jump into bed with the wrong vendor, you know what I mean? The wrong. You bet on the wrong horse, the wrong technology. You get dazzled by an AI, you know, demo. I always tell my clients like never just be like, oh, show me your AI, I want to see it. And they dazzle you with. And demos can be anything, right? Like I've done demos before. Like you can make them up and the text isn't even real behind it, you know, but sometimes, you know, when you put on that show, that's not the way to go. The best way to approach it. And I'll give this advice totally for free, you don't have to consult with me or anything. Is yes.

Saving money here on the show. All right, here we go, here we.

Go, here we go. Here's a 550advice, right? Always come with a problem, a real problem, because the first question is whether or not AI is even the solution here. I mean, yeah, it's the, it's the sexy thing right now, but like if it's a one off project and you just need it done most times a manual human, that's going to be the way to go until some technology comes that maybe improves it. But it's no sense of you spending that money and R and D or Capex or whatever it is to get that solution if it's not really needed. So you know, it's obviously works for repetitive things and it's going to get to be, it's going to start impacting work that's even not just repetitive anymore. I think you'll be able to. I heard somebody talk about how, oh, it'll probably never be able to do like a Supreme Court brief the way that like appellate lawyers do Supreme Court briefs. I don't think that that's the case, you know, because I could feed a machine every elegant Supreme Court brief going back to the 1800s, get, get principles extracted from them, incorporate that back in and then add. Because these systems, right, you can just add disparate knowledge and have it folded in. So you take a, you take a book like Robert Keanu's Persuasion, all the principles from that gunk in. You take whatever death by PowerPoint principles, throw that in and then all of a sudden it's going to generate for you an argument that's got psychological components, it's got the legal components, it's got the flash, that's got everything. And so that's where I think, like, why would you say that these briefs are not going to be able to be that good, it's just a very short sighted. Yeah.

And I'm gonna, I'm gonna throw in a personal story and then I want your take on this. So, you know, I started a, a venture company about nine months ago. And so when I was looking for attorneys, one of the first things I asked them is how do you use artificial intelligence in your practice? Do you use it? Obviously I hired the person that I thought was using it correctly because I believe that's the future. And even when you think, okay, well I want to pay an affordable rate, I don't care if I'm paying someone more an hour, if they're using their hours correctly. How do you see even the accessibility for the everyday, you know, the average person or the average small business, the average entrepreneur. Is it going to become more affordable to hire attorneys? Are we going to see kind of, you know, AI first or AI led companies that are very high quality and use this technology and they're going to start to create an own category? Or do you think it's still going to stay how it is for a while? What are your thoughts?

I mean, I think in the short term it's going to stay a little bit how it is for a while, but I think in the long term, I think personally what excites me the most about this space and it's not necessarily the work I'm doing right now for my clients, but I would really, I'm really trying to barrel into that space here and I've had great exposure to some of the leaders of that movement, you know, at Stanford or at Berkeley. Out here in Silicon Valley, in California is just this equal access to justice, right? That's going to be the thing here. So it's not really the business of law, but there's a huge unmet legal need in the United States just because people can't make it work. The numbers work, right? People don't have a lot to. Everybody really does need a lawyer or has legal issues, right? A lot. You know, you, it could be evictions, it could be just credit card debt, identity theft, whatever it is, right? You usually need a lawyer to kind of navigate and go through that, but a lot of times there's, there's all this unmet need because people, even people that are just like, maybe not working at firms that are charging, you know, 700 an hour, but something much more reasonable and they still can't really cater to that clientele. This is the most promise, this technology has to actually close that gap. But There's a bunch of hurdles in the way. One of them is sort of this, this general principle of the unauthorized practice of law in the United States, meaning that really lawyers are the only ones that should be practicing law. You know, and it's a regulatory scheme and it's, you know, some people think, well, maybe is that just a competitive sort of moat that they've created around lawyering, just like, you know, other kind of licenses and things like that, or is it more? But I think that's going to just have to be like really, really modified.

And changed because is there, is there a way to even, I'm just wondering now, is there a way to even police that? Right? Like let's say one, one lawyer starts a firm and they're tapping into artificial intelligence and maybe a large language model maybe specifically set up for legal. Maybe it's doing the majority of the work and it's just the lawyer signing off on that. Is that allowed? I mean, can you even govern something like that? You know, if, if now lawyers are spending, let's just say if they're tapping in and leaning into AI so hard and becoming over reliant, at what point are they still the lawyer versus maybe if they're really leaning heavily into generative AI and large language models.

That's a great question. That's a great question. Because I think that, that that's the million dollar question here because again, these technologies are so good that at a minimum, I think the language models are always going to get you average or better because it's going to take all of, all of that data, it's going to plot it somewhere in the middle. And then obviously with prompting and other, you know, knowledge bases and things like that, it can start to improve toward more of an expert level. But it's always going to start off as an average lawyer. So everyone's going to be able to get that at their fingertips. The issue is going to be ethical issues, right? So lawyers have, like, have you, have you actually done the diligence here? If you're just rubber stamping something that you think the machine could do and that's it, that that may be a competence issue for you, ethically speaking, especially if something goes wrong, you know, because these systems, let's, let's be totally clear about it. As much as I believe that they're going to just do everything that you could possibly imagine and more in terms of like sort of the future of law right now, the accuracy is not there, right? Let's call it 80% to start right. Which sounds good on its face. But imagine if only 80% of planes landed every day. It'd be millions of like, you know what I mean? There'd be problems. So it's not a good number. You need that number to be 99 or 6 Sigma 99.9999 whatever. But it needs to get improved and it will over time. But right now, you know, if you were just to like, let's say I have, I'm authorized to practice law in New York, New Jersey. If I just sort of hung out a shingle on the Internet and said, hey, bring it all in and I'll just, I'll write your complaints and I'll do that. Like, ah, if something goes wrong, I'm gonna find myself in front of the bar committee, you know, having to sort of say, like, what, what have I been doing? Is that zealous representation of my client? Am I actually doing the diligence? And that's what's, you know, every bar association in the United States is having this discussion about how to fold in generative AI. I sort of overheard a conversation two weeks ago where I listened in on the meeting of the state of California's sort of just a committee that's sort of addressing these issues. So it's not obviously the recommendation or it's not floated up yet, but when they're talking about it, they were talking about like the billing, like, okay, let's say like an average lawyer, it would have taken eight hours. Now I can do it in eight minutes. Right. What do I bill that as? And some people are saying, well, if you bill it at eight hours, that's fraud. And I'm like, is it really fraud if that's the market price for it? Is that going to impact hourly billing? You know, is this finally going to be the, the so called death of the billable hour that people have been talking about for 20 years? I don't know. But there's definitely something there where we'll have to think about because these tools are going to be incorporated so well. And that's why, you know that these tools are going to be huge if all the bar associates are already doing exploration, exploratory committees and adding rules and all that. So there's going to be an ethical landmine there. Yeah. Touch on like the bias of these systems either. Right.

For sure. And, and you know, this is, honestly, this has been a conversation and I'm glad we could have you on the show because this has been one I've been wanting to have for Months, because I do remember when OpenAI released GPT4, which was back in, in March. So it's, it's been a long time. And even at that time, their initial scores, I don't, I don't know what any of these scores mean, but apparently they took the bar exam, scored about a 298 out of 400, which placed it in the 90th percentile, which I think is pretty good. But it goes to your point if only 90% of planes land, not very good. But then I guess on the other side, that's a very old model. You know, OpenAI just, is just released or is releasing this week, GPT4 Turbo, which is supposed to be much more powerful, a much more capable model. Like, at what point do we as a society say, okay, well, if, if, if. Generative AI, if large language models, if, if. And this is always like the different exams, they always have them take the exams to see, oh, how good is it, how smart is it? But at what point do we as a society say, hey, if, if these large language models can score in the 99th or 99.5 like percentile? Like, at what point do we have to shift and say, okay, what are our odds of hiring the, the lawyer that's in that 0.1 percentile? Like, is that, is that a crazy thought to think that it might be safer to rely on this kind of technology? Or will lawyers just become better in the future when they're spending less time on this knowledge work and being able to think more creatively, more strategically?

Yeah, I don't think the professional law is going anywhere. So no matter what, like McKinsey says about the, you know, I think enterprising lawyers will always find a way and there's always going to be a need for. You're not, you're never going to do, bet the firm litigation and use a LLM. I'll say this too about benchmarks just really quickly, because I'm involved in one called Legal Bench. It's a Stanford paper. It's going to a machine learning conference this fall. And it's all about like real lawyers, real professionals like me that have been in this space have given to computer science majors, PhDs and sort of the tech side, all this data so that they could actually assess these models. One thing to make clear is that like, sometimes, even though it looks like a model, and I'm sure GPT4, they took all the precautions necessary, but as it swallows all that data, you never know if it swallowed an old exam. Right. And Just understood, just knew the answer because deep down somewhere in that neural network it pulled it out, right? So there's always like, benchmarking is really, really tricky, you know, because you always have to have something that's totally segregated from the Internet or not available yet to do a real time. What they do is hand score these things like open AI for all this technology of prowess and things like that. When they really get down to it, their hand score, they're grading these things by hand because that's like the old school way to make sure that this isn't. It's a new exam and all that. So I'll say that first. But second, that other question I think about, like, you know, I don't know, I wouldn't give much credence to like LSAT performance or bar exam performance translating into law at all. And I think any real lawyer would never, would never say, yep, yep, Every lawyer that got in the 99 percentile is an amazing Supreme Court litigator and a big law firm partner and all that. It's just not the case. Those things are so abstract and really not related to law at all. You know, maybe there's some sort of logical thinking that they're analyzing, but I wouldn't give it any credence. So again, it's going to lean into the future of. I think lawyers are just going to be better with these tools. They're going to be able to express their talent in different ways to people and really start exercising strategic judgment. And I think going back to sort of what a lawyer used to be, you know what I mean? Which is just, it's not a paper pusher. Just remember we live in a world where legal is really just looked on as a friction. Like in the work that I did, they never really respected lawyers in the sense of like it was just a thing in the middle of the other thing, you know, they wanted to do an M and A deal and so their financing lawyers had to come to get the money to buy the company. That's not their primary objective. They just want a check. You know, at the end of the day, all that paperwork, all that legal work, just so they can send a wire. Because really what they want is to buy a business. So again, everyone's looking at it like friction. That's why I think that if it's not happening. Yeah, just wait. But corporate clients of major law firms are going to start, especially if they're using AI in their own legal departments. You know, companies like Ironclad, they're doing great Things with contract management and all that life cycles. Once clients start demanding, hey, we're using this, we're having success, you better start showing us this on your, you know what I mean? I don't want to see bills inflated with associate hours and all that. So it's going to be the practice of law in that sense of like being trained initially by your, on your client's dime. That'll have to change. There's gonna have to be other, other methods that, that I go into my clients because we're building tools to even address that.

Oh, this has been a good one, Enon. Thank you so much. So as, as we wrap up here because we've taken this all over the place and I love that you've been able to give us fact based insights. The fact that you know, and we're going to share about the work that you're working on with Legal Bench in the newsletter today. But you know, just, just as we wrap everything up and as we look forward, what's the, what's the one takeaway that everyday people can, can, you know, use this conversation? Whether they're hiring an attorney for themselves. Maybe you know, someone that's going into the legal field, you know, maybe they just passed the bar, you know, what's I, I mean what should people be, be looking at and paying attention to to help make sense of generative AI in the future of the business of law?

I think for future lawyers, the people that just taken the bar exam or whatever know that these tools are coming. And so if you're not already conversant in them, get conversant in them because it is going to dominate the industry in 10 years. That being said, you don't have to necessarily don't. I think and this could go broader than just lawyers, but no one should feel like they missed the boat here that you know that, that these tools are just off and done and it's all, you know, like these large language models could only be created now. Like we're living in the now and this is when they were, they're available because of all the data and all the compute that was available. It just took this amount of time. So everyone's here together and we're at the ground floor. And I think no knowing how these technologies work is helpful. I don't think you need to say you have to program or engineering them. But getting some framework that you just kind of roughly understand how they work is a good thing. And obviously using these tools, you know, I feel like the lawyers that are going to be able to write emails quickly or, you know, what I do these days, it's incredible. Like, I just take notes, just kind of randomly, really. But it's so good at organizing them, packing them nice, making them more, you know, presentation worthy. So it's things like that start using these tools right away, you know, whether it's. I would obviously recommend Chat GPT, the premium version too. Do not get stuck in some other model. But, you know, look around. There's perplexity. There's other tools too. There's mid journey for art. There's so many incredible things that you should be leaning on. It's not just like, don't think about, is this a work thing? This work thing can become a lot of different things. You know what I mean?

So good, so many good insights, you know, thank you so much for joining the Everyday AI show. We really appreciate it. Thanks. Thanks for your time.

Thank you very much.

All right. And as a reminder, there was a lot here. We unpacked so much. I'm. I'm super excited. Yeah, like, people don't realize I'm still like, I'm a human. After I get off this conversation, I'm gonna get up re listen to it, type in all the insights. So we talked about Legal Bench. A lot of other great insights dropped in today's episode. So make sure you go to your everydayai.com Sign up for the free daily newsletter. If you're listening on the podcast, check in the show notes and we hope to see you back tomorrow and every day for more Everyday AI.

Thanks, y'.

All. And that's a wrap for today's edition of Everyday AI. Thanks for joining us. If you enjoyed this episode, please subscribe and leave us a rating. It helps keep us going for a little more AI magic. Visit your everydayai.com and sign up to our daily newsletter so you don't get left behind. Go break some barriers and we'll see you next time.