Generative Artificial Intelligence Explained In Plain Language

So good afternoon, everybody, and welcome to today's learning objectives series. And with us today is Ed Walters, and he's going to explain generative artificial intelligence in plain language. And Ed is the first person I ever had the pleasure of hearing explain GenAI, I think, in January right after ChatGPT hit the market in November of 2022. And I've never been so grateful in my entire life to have somebody explain stuff, because the Board of Governors of the NCBA then asked me to come and give a presentation to explain GenAI. And I actually had enough vocabulary and understanding to not just completely fall on my face, or, or, you know, read a million articles.

So you're being too honest, Catherine. I saw your presentation, it was way better than mine.

No, no, no. Anyway, Ed is the Chief Strategy Officer for VLECS Fast Case. He was the founder, uh, co-founder of Fast Case. Um, I've known Ed for probably longer than either one of us want to actually admit. Um, and in the, you know, in addition to just having been to ILTA, AALL, he, he makes the rounds. He also teaches at Cornell and Georgetown. Busy, busy, busy man and one of the smartest people I know. So I'm going to silence myself and let Ed take it away.

Thank you for that introduction, Katherine, and for the invitation to join y'all today to talk about large language models and generative pre-trained transformers, generative AI generally. As Katherine said, I am the co-founder of Fastcase and now the Chief Strategy Officer of VLECS. I also teach the Law of Robots at Georgetown Law and at Cornell Law School in the spring. This semester, I'm going to be teaching at Georgetown a new class called Generative AI and Big Law, where we'll explore using some of these legal AI tools, as well as thinking about what that means for the business of law, the practice of law, what it means for the leverage model and how we train associates. So I'm just going to jump right in today. And my goal is really that we together kind of talk about what these tools are, what they do, what they're good at, what they're maybe not so good at, how they work. I think everyone wants to sort of focus on the cautionary parts of it, and we'll do that. But I want to focus on the promise as well, and some of the more interesting parts of it. I'm going to do a couple of live demos. My engineering team hates it when I do this. It's kind of like a high wire act. So I'll just caution right away, like, they might blow up. But that's fun too. Like, I think that's really interesting, not just to talk about it in an academic sense, but to use the tools and, you know, kind of think forward a little bit. All right. So without kind of further ado, let's talk about generative AI first and foremost. So what are we talking about? Like, what does this— what do these AI tools do? Are they like kind of a robot lawyer? How should we think about them? Okay. And we're going to start with what they're made of. So these generative AI tools and large language models or GPT tools are really made from patterns. They're neural networks, but I think the thing that's important is that they recognize patterns and they interpret them and they can sort of create them. And to illustrate this point, I'd like to— you'll have to pardon me for fumbling around a little bit. I want to show a tool from Google. This is from like GPT-2, like a couple of generations back that illustrate this point. We are going to do Google Quickdraw and try not to blow that up. Let's see. Okay. Google Quickdraw was an experiment from the team at Google. And what they said was, we'll create a kind of a blind algorithm. On the, the left hand is going to say to the user, doodle something, draw it with your computer mouse. The right hand won't really know what's being drawn, but we'll try to guess what it is based on the doodle. I want to make sure that when I share this, by the way, I am sharing with sound because you'll have to hear it guessing. Okay. I'm going to start this and then you'll have to tell me in the chat, or Catherine, if you want to pop in and tell me if you can hear this as I'm doing it. Okay, so what's gonna happen is this is going to tell me I want you to draw something. I'm gonna try and draw it badly, and then the algorithm's gonna try and guess what I'm drawing. Okay, so it says draw a fireplace in under 20 seconds. Ready, go.

I see line or springboard or swing or table. I see computer keyboard. Oh, I know. I see its fireplace.

Okay, can y'all hear that?

We can.

Okay, good. As you can see, I'm not really much of an artist, but it did guess fireplace. All right, I'm going to draw a trumpet in under 20 seconds.

I see water or necklace or potato or dumbbell. I see computer mouse or trombone. Oh, I know, it's trumpet.

Okay. Draw an eraser in under 20 seconds.

I see line. I see river or ocean or green bean or belt. I see swimming pool. I'm stumped. I'm not sure what that is. I have no clue what you're drawing. Sorry, I couldn't guess it.

All right, draw a scorpion in under 20 seconds. I don't really know.

I see circle or bun or potato or mouse. Oh, I know, it's scorpion.

Okay, that's kind of unbelievable. Draw a cake.

I see bun or circle or umbrella or hockey puck or shoe. I see paint can. Oh, I know, it's cake.

I don't know how I'm going to draw a prison.

I see elbow or foot or suitcase or pillow or square. I see eraser or postcard. Oh, I know, it's prison.

Now it guesses eraser. Okay, so it is guessing what I'm drawing. How is it doing that? Like, how is Google guessing what I am drawing? Well, it is using a database of other doodles., the images that people drew in the past so it can figure out what the thing is when I start to draw it. So here are 135,000 boomerang drawings made by real people. Uh, when asked to draw a boomerang, this is what people draw. And in some way, um, you know, that creates like a platonic form of what, uh, people draw when they draw a boomerang. Or a chandelier in that case. So what this does is it basically looks at that pattern of drawing a chandelier across, like you can see, like a lot of different expressions of that. And then when the next person draws it, it can use that pattern to identify what the thing is that's being drawn, including if you draw terribly like I do. Okay. And that is effectively what the basis of large language models are. I mean, that's what they do. They effectively use patterns in the past to understand the future in a different way. And so these patterns, in my case, like a set of doodles, but in the case of generative AI that we're using, words and patterns of words. That is what large language models are made of. The specific tools are called generative pre-trained transformers. That's the GPT in ChatGPT or GPT-4. It's the generative pre-trained transformers. And the idea is that instead of training something the way we used to train software, If you see this, then do this. It is pre-trained and the outcomes are somewhat unpredictable. This is made most famous by ChatGPT from OpenAI, but there are lots of different tools, large language models or GPT tools, sometimes called the foundation models, that use generative AI and work in exactly this way, these neural nets. Here are some of the more famous ones, but there are increasingly lots of them. What are they good at? Well, I would say one thing that they're really good at is understanding text. So when you type in a question or a prompt into these foundation models, they're really good at understanding what you're asking for. It's kind of like recognizing the pattern that you make when you're making a doodle. So understanding the question and then creating an answer in a way that wasn't like pre-programmed. So kind of no matter what you ask, these tools will create a statistically likely answer. They will say, really starting from the very first word, a great answer to that question What would be the first word? What would the most likely first word be for an answer to that question based on everything we've seen in the past? And then it'll go through that chain, right? What would the most— based on that first word and the question, what would the most likely second word be, third word be, fourth word be? And it is really good at kind of predictively figuring out what the answer might be. Sometimes it'll get part of the way down the chain to realize looking back, that doesn't really make any sense, and kind of go back and start again. But what they're really great at is kind of creating very human-like answers. In my Law of Robots class, I used to teach about the Turing test. Alan Turing had this hypothesis about when artificial intelligence would be sufficiently smart, and they said, he said, like, if you had you know, a human being and a computer behind a curtain, and you ask a question, and they both kind of submit answers out, and you can't tell which one was submitted by the computer or the person, then the machine is kind of sufficiently smart. This was a thought experiment when he came up with it in the 1950s. But increasingly, machines in the 1990s and the 2000s and the 2010s We're getting better and better at it, although iteratively, for the most part, after a couple of questions, you could figure out that something was a computer and not a human. That is no longer true. I mean, the Turing test now is a history test because the large language models are kind of Turing complete. You can't really tell whether you are talking to a machine or a person. Or to the extent that you can tell, you can tell because the grammar from the large language model is better than what you're accustomed to if you're chatting with a human being. It's like smarter. And so you might recognize it as a machine because it's better. It's Turing-like supreme or something. I don't really know how to express that. But the idea is that they're very good at coming up with a statistically likely answer. I'm showing these models. There are others too. The Clem from 273 Ventures is one that is trained specifically for legal. It's sort of organic, if you will, because it's all organically sourced. They're guaranteeing there's no copyright issues with the training materials in there. So these large language models are really good at creating these statistically likely answers. But I want to emphasize here what they're doing and what they're not doing. So one of the biggest challenges of these large language models is that if you ask a question, they will always answer it. Right? I mean, sometimes they're trained not to. If you say, like, to fake or whatever, it might say, as a large language model, I can't really do that. But if you ask a question, it is programmed to create, for the most part, a statistically likely answer. Note, though, I'm saying a statistically likely answer, not a correct answer, not a true answer. The idea is statistical likelihood. It should read perfectly, right? But it's not designed to be correct or true. So I think one place where we get in trouble is people will ask these foundation models a truth question. And get back an answer, and we sort of assume it's true, right? It's not made to be true. It's made to be statistically likely. And sometimes the statistically most likely answer is not the true answer, right? And so you, you hear about this sometimes, uh, people talk about hallucinations. Um, and this is particularly a problem with law because when we ask legal questions, What is the limitations period for this thing? We might get back an answer that is based on statistical likelihood, but it's not looking for the true answer. To put it differently, like in the history of law, we learned in law school to use legal research engines like Fastcase or VLEX, or I can't think of any other, but we ask these questions and what happens is search engines will search a database of judicial opinions or statutes or regulations or law review articles. And it will pull from that database documents that answer the question. And we typically read them and, you know, kind of make decisions for ourselves. But it is searching in a database. When we ask the same question to these foundation models, they are not then searching a database. They're not reading through their training materials and finding the document that best answers the question. They're looking at the entire aggregate of the model, the entire world, including Reddit comments and YouTube videos and Twitter and, you know, whatever they can find on the web to create, you know, an amalgam of what the answer might look like. And famously, like, this has gotten people in trouble when trying to do legal research using these foundation models. The most famous example is Stephen Schwartz, who we all know now because it was a New York Times article about him. But let me just say a couple of words in defense of Stephen Schwartz. There's a, there's kind of a cottage industry right now in dunking on this guy. But some of the mistakes he made, I think, are relatively understandable. So a little bit of background, if I can deep dive on this for a second. 2019, um, uh, a man named, uh, Ronaldo Marais is flying from San Salvador, uh, to New York on Avianca Airlines. And in the middle of the flight, he's sitting on the aisle, the beverage cart, uh, cuts loose and screams down the aisle and smashes him in the knee. Um, it breaks his kneecap and he is in agony. He gets to New York. Um, you know, he goes to see a doctor. He has all this pain. He can't walk really correctly after this because of the damage done. But immediately after this happens, the world also plunges into COVID. And so Mata, like most people during that time, especially in big cities, was kind of locked down. So he doesn't do anything until like 2022, uh, when he late in the year goes to see Steve Schwartz and he says, look, I'm, I'm in pain, I still have all these medical bills that are unpaid, I'd like to recover against Avianca Airlines, I'd like for them to cover my medical bills and pay for my pain and suffering. Um, unfortunately Schwartz says, look, you know, when you sue an airline, it's covered by this treaty, the Montreal Convention, And the limitations period for doing that kind of a suit, for a civil suit against an airline, is 2 years. But Stephen Schwartz dimly remembers that, uh, under this Montreal Convention, if the defendant declares bankruptcy, the limitation period pauses. It is tolled for the duration of the bankruptcy. And in fact, Avianca Airlines had declared bankruptcy during COVID and so the limitation period was tolled. So when filing the complaint, he needs to cite something that says that the limitation period is told. Now, travel with me to the distant past of maybe January, February of 2023, when Steve Schwartz is drafting this complaint. ChatGPT had come out scarcely 6 weeks earlier. In November of 2022, pundits like me have been telling Stephen Schwartz for a couple of years, you're not going to lose your job to AI. You're going to lose your job to a lawyer using AI. And so he wants to be one of the lawyers who is using AI, not one of the lawyers who is losing his job to lawyers using AI. What tools does he have? January, February 2023. Does he have Vincent AI from VLEX? He does not. Does he have Lexus Plus AI or, you know, Co-Counsel from CaseText? Not even close. He doesn't have any of those tools. And so what does he do? He uses the AI tool that he has access to, ChatGPT. And, you know, in this case, he asked ChatGPT, like, Does the statute of limitations toll under the Montreal Convention during the duration of the bankruptcy of the defendant? And ChatGPT says, Stephen Schwartz, you are in luck. The law is exactly as you imagine it to be. There is a case, Varghese v. China Southern Airlines Co., Ltd., 925 F.3d 1339, 11th Circuit, 2019, that stands for exactly that proposition. Here's a quote from the case. That says exactly what you want it to say. Man, he thinks, this is perfect, right? I can't believe it's this easy in this magical AI future that I can type a question into ChatGPT and get back a fully formed and completely true answer to the question that I was asking. It's perfect. It's exactly what I'm looking for. So I think that you can sort of tell from the way I say it, I have some sympathy for Steve Schwartz, right? Remember too, for the whole history of this guy's practice, when you run a search in Westlaw or Lexis or Fastcase or VLEX and there's no results, you get back an answer that says there's no results, right? If you look for a web page that doesn't exist, you get 404 file not found. Yet all the software that you ask some sort of question to, when there's no answer, if it doesn't know, it comes back and says, error, I can't find anything, there's no results. For the first time ever in the history of him using software with ChatGPT, when ChatGPT doesn't have an answer to the question, instead of saying, I don't really have access to that sort of database to answer the question. Instead, it is designed to say, here is the exact answer to your question perfectly, and even fabricating a quote from the case to give him what he is looking for. I mean, it thinks it's doing the right thing, right? So this sort of became known as hallucinations, and this would be a surprise. How would Stephen Schwartz even know this was possible? We know it because the New York Times wrote this whole article about him. He doesn't have the benefit of this New York Times article to understand this could even happen. All right. So he's got this perfect answer for him, you know, and in fact, it turns out to be actually kind of a correct statement of the law. This is where I get off the train with Steve Schwartz, right? Because what he does next is he copies that from the ChatGPT screen and pastes it into a complaint before the court. Obviously, in hindsight, you know, you have to read those cases. If you remember nothing else from what I tell you today, it is that you should— the number one thing is you should always read the cases, read the statutes, right? You are responsible for it under the model rules. Whenever you use these tools, your duty of candor, um, your, uh, duty of competence require— duty of supervision require that you read all the cases. If you asked someone else at your firm, like an associate, to do some research, or a paralegal to do some research for you, and they came back with a section for the brief, you similarly can't just copy it from their Word document and drop it into your brief. You are responsible for reading all of that, right? And so I would say the rules are exactly the same with these, um, GPT engines, the general foundation models. You are responsible to make sure that the cases say what they're supposed to say. That is your responsibility, no matter who is handing you that research. And the Steve Schwartz case kind of illustrates that. Now, we all know what happened next, right? So opposing counsel for Avianca gets this complaint and says, Judge, we've got an issue. We can't find the cases that are cited in this complaint. And so the court writes Schwartz a letter and basically says, we've got the same problem. We've looked for these cases. We can't find them. Would you send them to us? And this is where Schwartz really kind of, you know, loses his way. He says, you know, I need some more time. I'm on vacation. He was not on vacation. What does he do when these cases are challenged? Does he log into Fastcase, which he gets as a member benefit for being a member of the New York State Bar Association? He does not. Does he go to a public law library or a public library and log into like a traditional legal research terminal or ask a friend to look them up for him? He does not, friends. He goes back to ChatGPT. These are some screenshots of his ChatGPT session from the sanctions brief. He asks ChatGPT, is Varghese a real case? Yes, ChatGPT says. Varghese versus China Southern Airlines Company Limited, 925 F. 3rd, 1339. 11th Circuit, 2019, is a real case. By the way, like, look at this citation. This is— this looks very much like a real case. Um, you've got the abbreviations using Table 6 of the Bluebook. The citation, um, like the range of 925 F.3rd actually kind of matches for 2019. 1339 is a page number, is like within the range of pages of a Federal Reporter. I mean, this Well, it looks pretty legit. What is your source? Schwartz asks ChatGPT. I apologize for the confusion earlier. Upon double-checking, double-checking, I found the case Varghese versus China Southern Airlines does indeed exist and can be found on legal research databases such as Westlaw and LexisNexis. I apologize for any inconvenience or confusion. My earlier responses may have caused. ChatGPT didn't just check Varghese. According to ChatGPT, it double-checked it. Right? I have, I have now double-checked this and according to this response, looked it up on Westlaw and LexisNexis and found it. By the way, complete fabrication, like none of this is true. Didn't check. Didn't double-check, has no idea whether it's available on Westlaw or LexisNexis. It is just creating a statistically likely, highly verified answer. Are the other cases you provided fake? Schwartz asks. No, ChatGPT reassures. The other cases I provided are real and can be found on reputable legal databases such as LexisNexis. And Westlaw. So does this mean we can never use the foundation models for legal work? Well, I would just say this is really a matter of using them for what they're good at and not what they're bad at. We certainly can't use them for legal research, as Stephen Schwartz has shown us. We can't use it for truth engine work. You know, I almost think about like the hearsay standard, like you can't admit something for the truth of the thing being admitted. And here, if you want a truthful answer, if the truth of the outcome really matters, these foundation models— I'm picking on OpenAI, but it could be Gemini or Llama or Claude from Anthropic— shouldn't be used for those purposes. But there are a million things you can use them for. If you want to use them to draft an email, if you want to use them to write a letter of recommendation or summarize a document or, you know, draft correspondence or something, as long as you are supervising it and making sure that there's no, like, kind of false statements in there, they're really good for that kind of work. We use them all the time for these kinds of tasks. I just want to say, like, let's be careful not to use them for things that are beyond that. All right. Does that mean that we can never use these large language models for legal work? Well, it does not mean that. And I want to make an important distinction here between kind of the general foundation models and the fit-for-purpose legal generative AI tools. So those tools— and here I'm not pitching, right? I'll say like Co-Counsel from Casetext is a good example of this. Vincent AI from VLEX is a good example of this. Lexis+ AI from LexisNexis is a good example of this. These legal-specific generative AI tools work differently than the foundation models. They are sort of made for these legal tasks. And the reason is that they use a technique called retrieval augmented generation. That might sound, I don't know, intimidating or something. Sometimes people call it RAG. But I want to explain there's nothing really very fancy about it. And I think to illustrate this, I will show you again. I hope you will forgive me somewhat for showing it in practice. I'm going to show my screen. I'll close out Quickdraw, which was fun. All right. So first, let's look at legal research the traditional way. Just remember how this works. We are told when we do legal research that we should imagine the language that a court or a legislature would use when discussing our topic. If we're trying to figure out what triggers the beginning of the limitations period for a workers' compensation case in North Carolina, we might use like one word within 4 of this other word and this exact phrase and this synonym or this synonym or this synonym. And we use these parentheses and quotation marks and proximity operators, all of the computational linguistics of a Boolean search to try and get a list of documents that sort of answers the question. And the way we normally get it is we get back a list of hundreds of documents, thousands of documents. You know, in this case, there's like probably 19,500 individual documents that match the Boolean expression that I used. Do we read them all? Of course not. How do we know how many of them to read? Man, I don't know. I gotta tell you, I mean, I, I've been doing this for 20 years. I've been building legal research engines. We've tried to put the best stuff close to the top with some cool ranking algorithms in the old Fastcase. But I can't tell you, do you read 8? Do you read 18? Do you read 88? No one knows. Like, there's not a person in the world who can tell you how many of these to read. And then, of course, there's always stuff that's not in your list. You know, you say it's within 4, but it's actually within 5. There's another term of art for the thing you're looking for, so you miss the important thing. This is how it's always worked. This is the state of the art as of 2022. I think Fastcase is pretty good at this for the, you know, kind of keyword search where you're searching using the words. But now let me illustrate retrieval augmented generation and why I think that is better. So this is Vincent AI in VLEX. And I'll just sort of show you, instead of creating like a Boolean expression using computational linguistics, here I'm just asking the question the way I might ask it to a colleague or a research assistant. What event triggers the limitations period in a workers' compensation case? The date of the accident or the discovery of the injury? And I'm asking this question in North Carolina. Now, what retrieval augmented generation does is it breaks this research process into 2 steps. The first step, it's going to retrieve some documents that help answer the question, looking in a database that helps— that contains some of the answers, right? Not like a generative AI foundation model. It is not going to try and create an average statistically likely answer. We're going to go and find documents in a database that help answer the question. Okay? And then in this case, you'll see them on the right-hand side. In addition, we're going to go through all of them. And we're going to figure out, do they help answer the question or not? Alright, this will take a couple of minutes. And then we take those documents that are retrieved that really help answer the question. We pass the question and the documents that answer the question along with some instructions to a large language model. We will say, answer this question using only these documents that I'm giving you. Link to them when you're using them. Show your work. Don't hallucinate. Everything that you're citing, I want you to verify that it actually exists and says exactly what you say it does. And then answer the question. Okay, so step 1, we retrieve the documents to help answer the question. And step 2, we pass them over and generate an answer based on those documents. Retrieval augmented generation. And everyone's really excited about the generative part, the generative AI part. But I gotta tell you, as a search geek, I'm more excited about the retrieval part. There are things possible here that I never imagined would be possible. So on the retrieval part, we're using a new technology called vector search. Instead of searching the individual words, When we index the legal research database, every document is indexed by its ideas, by the concepts used in it. Think about like the Old West headnotes. Here are the 7 headnotes for the case. These are the 7 things that the judicial opinion says. We're doing that for everything, for judicial opinions, for statutes, for law review articles, for treatises. And then we index all of those concepts and basically put them on a map. Okay? And the map is like a map of all of the ideas, the concepts in the entire legal research database. Ideas that are very similar are really close together on the map. Things that are different are like really far apart on that map. And what we're doing here in retrieval augmented generation is we're taking that question. We are converting that question to vector. And then putting it on the map. Here is the idea in the question being asked. And then we look at the documents that are most closely associated with that idea on the vector map, and we challenge them one at a time. You saw it happening on the right-hand side here. We evaluated 30, 40, 50 different documents, and we said, Some of them are valuable. In fact, this case, 19 of them are valuable. And then 31 of them not so valuable. You know, we just toss them. Right? And so on the right-hand side here, you see what was retrieved under legal authorities. I'm not trying to pitch, okay? I'm just showing how the retrieval augmented generation works. I'm not asking you to buy Vincent AI, but let me just show you. So here are 13 cases, 2 statutes, 4 administrative decisions, probably from a workers' comp board. Every one of them is hyperlinked, so I can open this case to verify it. There is a summary of the case here. This is a generated summary of what the case says, followed by a confidence score. And this confidence score says how likely it is that this case addresses the question being asked. Finally, in gray here, this is an exact quote from the Brown case that is verified. And if we open the case, we can see this exact quote in it. In the computer science nerd world, this is called an embedding. But this is the idea contained in that document in its quoted form, in the exact words used in the document. That is indexed for this idea. So instead of 19,000 documents, uh, from all over the world, most of them irrelevant, we get 19 that are super relevant, that are just bang on for the question that we're asking. Uh, and I can verify all of them. I can open the Taylor versus J.P. Stevens case And if I want to, I can even modify the list and say I don't want to include, you know, this statute or this individual case from the Court of Appeals of Longshore complaints. I can remove that and then rerun the answer and regenerate it. So by the way, I just think this is an important ethical part of these kind of retrieval augmented generation tools. We require nutrition labels on our food. We want the list of ingredients so that we can be informed consumers about the food that we consume. And I think the same thing is true for legal information. I think we should have, like, transparency as a matter of our ethical responsibility. We should have transparency into what is creating the answer. The foundation models I mentioned before, like ChatGPT, don't really do this. You ask a question, something magic happens behind the scene with, you know, some model, but not a bunch of documents, and then an answer pops out the other side. We don't really know why. We can't verify it. We can't figure out why that's the answer. Here, with this retrieval augmented generation model, and especially in the Vincent tool, We are being extremely transparent. These are the ingredients that are used to answer the question. You have full audit control over it. You can see all of them. You can click on them and verify all of them. You can even remove some of them and choose not to use all of them. But then when we pass that over to the generative engine, what you get on the other side is a really solid, well-substantiated answer to the question, even if it's a little nuanced, even if there's caveats, right? So it's kind of a trick question. I ask what triggers the limitations period, and the answer is it depends. Uh, for injuries by an accident, the limitations period is triggered by the date of the accident. But for occupational diseases, which might only be discovered years later, the limitation period begins when the employee is informed by a competent medical authority of the nature and work-related cause of the disease, or when the employee is rendered incapable of earning the wages they were receiving. This is a pretty nuanced answer, and you can see in here we are citing chapter and verse. Here is the section of the North Carolina General Statutes that talks about the limitations period. I can go read that by clicking this here. Here are cases that interpret the limitations period and administrative decisions. And then here's at the end some exceptions, some qualifiers, like the general rule is that the date of the accident is the trigger, but there's an exception where the last payment of medical compensation can extend the period for filing claims for occupational diseases or death benefits. So this is, I think, a good maybe illustration of retrieval augmented generation and why it is different. There's a bunch of cool stuff I could show you inside of Vincent AI. Maybe we can do that in a different webinar. But I just want to illustrate here the difference between foundation models, which are not designed to answer the question. They're designed to create like a statistical average but maybe not true answer to the question. And the legal-specific tools which use retrieval augmented generation, which are designed to answer the question, they're designed to search a database, find documents that answer the question. I'm pretty excited about this vector database idea where they find extremely good and then verify that they are the right documents answer the question, and then use those to create an answer to the question. Um, this is not, uh, 1,000% hallucination-proof. I'll still say the same thing here that I said with the foundation models. It is the lawyer's responsibility to verify all of these before putting them in a pleading, just like you would do if an associate or a paralegal or a colleague or an information professional, a law librarian, gave you a stack of papers or a research memo, you are still responsible for verifying. But I mean, if you compare this to the way that we used to do research, this is a revolution. I mean, this is really different. Instead of reading thousands of documents and trying to synthesize them and create an answer for yourself over days or weeks or months, we did this in about, I don't know, 3 minutes, 4 minutes., and have like highly accurate, very verifiable answers. So I think for summarization, for the kinds of tools or the kinds of uses that don't require like research or truth, it's fine to use the foundation models. There's no problem at all using it for, say, generating an email to a client or to create materials for the firm's website, create a bio or something. But I think for the legal-specific work, you should use the legal-specific tools and specifically tools that use retrieval augmented generation. I think the last big thing I'll just caution, we still have a duty of confidentiality to clients. We need to make sure that the foundation models or the legal-specific tools aren't using our questions and answers to train some model. I can tell you that the big ones don't. So Lexis AI, the co-counsel from CaseText, now Thomson Reuters, Vincent from Dilex, are using the locked-down foundation models. They are subscribing to the most expensive versions of OpenAI. AI and Gemini. They are designed never to train with materials that are passed in. And so at the end of the session, all of that is basically flushed. It's not used to train any models. With retrieval augmented generation, it's not like we are creating a foundation model. We're not creating our own LLM. What we're doing is we're running a search, in this case a vector search, but we're doing a search. We're getting a bunch of documents and we're passing it over into the best kind of foundation models in the world. But so we always have to make sure the tools are locked down in that way, protecting client confidences. And we always have to make sure that we are verifying the accuracy of things that we find through generative AI before submitting it to a court or in final documents. There are some people who say, well, if it, it doesn't, if it requires me to verify it, it's not worth my time. I might as well just go do it from scratch. I think that's fine. It's a very responsible way of doing it. And if you're paid by the hour, maybe it's financially more lucrative for you to spend 20 hours doing it yourself. I'm not sure it's ethical if you can get a better, more competent response in 3 minutes instead of 30 hours, but It's a changed world. So I think I'll just flag all of those things here. We have to be careful. We have to be ethical and responsible. We have to understand what the tools are good for and what they're not good for. It might seem like from my tone of caution, my encouragement to be ethical and responsible and to verify these tools, that I don't think they're very good. Maybe I think Maybe you could interpret that I think that they are too dangerous to use. Nothing could be further from the truth. I think that this is the most important development in legal technology since LexisNexis created the first legal search engine in 1972. Everyone says that their technology is a revolution. I actually do think that the retrieval augmented generation technology is a fundamental change in the way that lawyers will do their work. Certainly research work, but I think also drafting work. I didn't even show a bunch of the drafting tools. I showed one research tool. Vincent has like 20 tools. You'll see an announcement like this week or next. Again, not to pitch, but to say that the promise of these tools is extremely great. And the tools we're using right now feel like the future, but in many ways they are the past. You know, these are really good tools. They work extremely well. This is the dumbest, slowest, and most expensive they will ever be. These tools are getting better all the time. And I think that they have enormous potential to improve the practice of law, not just the day-to-day life of lawyers, but the quality of the services that we provide to our clients. I think that we'll be able to provide better services and at lower cost for clients from here on out. Okay, I wanted to leave about 10 minutes at the end here for questions. And what I'll do, I will stop sharing. If you want to ask a question, you can put your hand up. You will probably see that in the React. Button on the bottom of your screen, or I think you can put your hand up. You can signal to me that you want me to call on you for a question, or you can drop something in the chat, and I will try to answer questions in the chat as well. Marty Martin says in a direct message to me, but I think relevant to the whole group, how does this work with Fastcase? Available through the North Carolina Bar Association and Decisis through the Wake County Bar Association. What is the cost of Velex to solo practitioners? Uh, great question. So, uh, in 2023, Fastcase and Velex merged. It doesn't change the bar benefit through the NCBA, um, but we will be moving on to the Velex platform Sometime in the second half of this year, you'll have the ability to use both for some time, and then we'll migrate fully over to the VLECS platform. Same model. You still get free as a member benefit through the North Carolina Bar Association. But you'll get the upgraded version, which has some of the Vincent AI tools, the summarization tools, the translation tools, the find related documents tools from Vincent included for free. The, the generative AI tools cost us a ton of money. They used to cost about the cost of a flight to New York. Every time we ran them, the cost has come down some, but those we will make available by subscription. The subscription pricing will be sort of over and above the free benefit through the North Carolina Bar Association. Decisis is anyone's guess. That's a crippleware version of Lexis. I have no idea what Lexis is going to do. But so the cost of VLEX to solo practitioners, the VLEX research engine will be free just like Fastcase was, has been through the North Carolina Bar Association. Rosemary Florence asks, how does AI handle summarizing medical records used in legal matters? Well, I think summarizing is a great use case for these generative AI tools. And effectively, if you upload the document, what it will do is it will create like an AI model of what that document says and create like a summary based on that. When you use that document to say summarize it, even the foundation models are pretty good. I'll say that the legal-specific tools are probably a little better and might be able to create some additional context around that medical record. It might say like, here are some questions to ask about it. If you're creating a deposition about it, you know, here's some questions to ask the doctor. But for summarization, effectively a large language model has summarization built in. And so any of them, Gemini, OpenAI, ChatGPT, will do a pretty good job of summarizing. Again, I will say like before you submit that as work product, you should verify the accuracy of it.

And Ed, if I can interject, um, this is a question that I've been getting a lot about, is specifically summarization and analysis of medical records. And there are tools on the market specific to that, so I haven't fully investigated them, but I have just seen a lot of tools that are specific to that use case. And hopefully they'll put some guardrails around these tools and also be cognizant of the fact that some of the medical records could have images could have handwritten notes in the margins. And there's, you know, so it's also, can they handle something beyond just text? Similarly, with real estate research, some of the real estate title searches, I have yet to find anything that truly substitutes for a human being on a title search. Because many of the court records are not digitized and, and not available. So it's, you know, they don't have any way of going in and doing that. So there's a lot of practice-specific tools that are coming out, and you just have to put on that skeptic hat a little bit and think about what do I look at when I look at these things, and could a computer actually deal with it? But I do love the notion of RAG, and I think that RAG, the real-world augmented generation has a lot of applicability, not just in the legal research databases, but also thinking of things like Microsoft Copilot or knowledge management, where we're looking at specific— we want the answers to come from a specific data set, not made up by the foundational models. And thank you for giving me that language. But the, the other thing is we're starting to see GAI tools implemented within law practice management systems. And, you know, again, we're very early days with all of this stuff. I've also read articles about in terms of summarization. It, it's like humans, it remembers the first and the last thing we read, and sometimes it doesn't so good, so good with the middle stuff. So I am— I've had it try to summarize documents that I've written. And since I wrote them, I'm probably pretty adept at recognizing whether it did a good job or a bad job. And sometimes it does a good job, and sometimes it does not. And formatting matters. So I had it try to answer— Acrobat has this little, you know, GAI tool built into it. And I gave it a simple 3-page document and asked it a question. And because of the way the document was formatted, there were some bad charts. And it was literally, when is Katherine Reach speaking at this conference? And this was the conference grid. It couldn't answer the question. So we're definitely in a trust but verify situation. But it is great for tone, marketing, I generate every single blog post image in Windows Developer now because it's better than stock photography. There's so many great uses of this stuff.

I don't see other questions. If you have one, would you raise your hand or volunteer? Ah, Todd Garner says, to follow up on your prior answer regarding the availability of generative AI tools as an add-on for NCBA members. Can you give any kind of estimate or range of what that will cost? Will it be price tiers, and if so, is that based upon access to different types of tools or based upon amount of usage? Great question, Todd. I think the, the full answer to that is we are trying to push the cost down right now, and so I don't think we have set an exact price. Again, like the found— when we use OpenAI is like GPT-4.0. It's insanely expensive for us. So we wanna make sure that we're doing this in a way that's sustainable. We're here for the long haul. We don't wanna have an explosive burst one year where we pay, you know, $10 million and recover, you know, $40,000 and then go out of business the next year. So we wanna do this in a way that is sustainable and is a good business model. So the cost of the foundation models has come down. Like GPT-4 was originally very, very expensive, but GPT-4.0, which came out about 2 months ago, is less expensive. And we're actually pushing OpenAI to try and give us a discount even beyond that. And when we do, we're going to pass that through to members. The whole DNA of Fastcase and VLEX is to democratize the law. And if these generative AI tools are only available for the biggest firms in the world, total failure. That would be really against the ideals of Fastcase and VLEX. And so we are pushing right now to make it less expensive. I can't really quote a price just yet. But when we roll this out, we will have some pricing tiers available. I'm happy to announce it. Hopefully, as you know, little cost as possible. Can I show one more thing? This is kind of cool. This is like a little magic trick, I think. Alright, so this is one of the other tools that is available in Vincent, which is analyze a document. This is not like a legal research tool, and you can pull into it just about anything. So transactional documents, depositions, medical records, whatever you want. I am pulling into this an asset purchase agreement. It's a PDF. And what's going to happen is Vincent's going to read it and it's going to say, ah, I know it's a fireplace. Not a fireplace, but it says, I know this is an asset purchase agreement between Garden City Hospital and Prime Healthcare Services. Now, based on that, we didn't hardcode this in. Um, it's going to say like, what kinds of things do we do with asset purchase agreements? And those are all in purple. I'm going to identify client hostile language. I'm going to create a timeline of key dates. Uh, give me a compliance checklist. Um, and so you can choose any one of these things to do with an asset purchase agreement. Or you can ask whatever question you want, but we'll pull out from the asset purchase agreement, citing chapter and verse, what clause and what section, what the compliance obligations are after the purchase agreement, including areas of concern and recommendations. This is the kind of stuff that you know, a law firm would do after an asset purchase agreement is closed. And here we can see, like, here are all the requirements from this contract. I'm, I'm doing a contract here. I could do a complaint, I could do a law review article. And depending on what kind of document you put in here, you would get different kind of prompts. And different kinds of answers. So I put here the lawsuit, the complaint that Elon Musk filed against Sam Altman, and you can see all of the types of things in purple, all of the kinds of things we might do are different than they were with the contract. So give me a timeline of the facts, give me all the legal defenses for each claim, find cases that are like this that are filed in courts. And then based on the substance, what we find in the document, you know, one of the things that's raised in the complaint is a breach of fiduciary duty under California law. What are the elements required, you know, based on the real substance of this? And so if I say, give me a timeline of the facts, it will pull from the complaint, you know, a timeline of what happened in this case. In narrative format, and I can download this in Word, uh, or in Excel. Um, I can say, give me all of the defenses to all the claims, uh, or, you know, ask anything I want. Um, you know, how many entities are there under OpenAI? And it will pull that, uh, from the complaint.

It's pretty cool. I think this is a really good example also of where the tools are becoming more sophisticated so that the prompt engineering is not the struggle. Just like we went from complex Boolean searching to natural language searching, you're not going to have to create a prompt more. It's going to be smarter at helping you understand what you can do with something, which is going to be a relief, I think, for everybody because that big blank page syndrome and having to understand prompt engineering is, is kind of intimidating. Um, I, at first everybody was like, oh, well, there's going to be a whole new field of people who do nothing but prompt engineering. And I think the, the people who are actually developing the products are recognizing pretty quickly that's not going to be a field of study that we want to force people into to use it on a daily basis. Someone was asking about uploading documents and transcripts and asking for examples of parties failing to co-parent. I mean, so interestingly enough, this brings in a couple of things. A, the tool that you're using maintains confidentiality, doesn't remember the documents, doesn't train the tool itself. And then if, if all of that has been cleared, being familiar enough with the documents to understand whether it did a good job or not. I mean, to me, like the paid version of Claude comes to mind because you can upload multiple documents into it and have it look over that. But again, because it's foundational, it may not be the best use case for that.

Yeah, I think, I think you can, you can do this kind of work probably in foundation models where you say, here's a folder full of documents, find me things in the documents in this folder. I'll just point out, you know, Vincent does this as well. You can create a collection with a bunch of like a folder full of documents. Pretty soon you can pull this from iManage if you use that or NetDocs, but do it directly from your hard drive as well, and then ask questions about it. I don't think that's too much to ask. I think there's a lot of tools that should be able to do that.

And even what's cooler are the transcription tools themselves. So if you have an audio file, a lot of the transcription tools on the market, like Descript, are amazing at creating very, very accurate transcriptions. And because they match up the transcript to the audio recording, you can go through and read it and listen to it and make sure that it's right. But the amount of time that it saves is astounding. I mean, just nobody has time to listen to a recording and type it up. So if there's a tool that can do that, that's a great step in the right direction, I think.

Can I offer my postscript? Yes. OK, so here's my conclusion. These tools are really powerful. Especially if you know what they're good at and what they're not good at. But using these tools for their kind of recommended purposes really have the ability to supercharge the work that we do, not just make it faster, but make it more complete, make it better, think of things that we didn't think of. But I don't for a second believe this is like a robot lawyer or something. Sometimes I think people sort of want it to replace everything that lawyers do. And I don't think of these tools that way at all. I think the right way to think about it is that representing a client is more like a chain. You know, some of those links are arduous, like, um, discovery work, you know, reading through paper documents. And we've seen in our history that we can replace some of those individual links, the work that is the most time-consuming, has the most drudgery we can replace. But the last links of the chain have to be human. You know, the work that we're doing really is to replace individual links, not to replace the whole chain. We're not trying to create a robot lawyer. We're trying to build a suit of armor around lawyers and their work. And so the idea is to make us better, faster, and smarter in our work, but never to replace the judgment of the lawyer at the end of the day. And so this is not made to, you know, replace lawyers, but it's made to make us much better at our work and maybe create a couple of new kinds of products we can offer to clients.

Thanks, Ed. And I will— Todd, if you want to reach out to me, we can have a discussion about some of your questions, or reach out to Ed. I will stop the recording after I thank Ed profusely for spending so much time with us today and showing us what we should know, giving us again some vocabulary, some understanding of the differences between the foundational models, legal-specific models. You know, I guess what I've been telling people is use it for the low-hanging fruit stuff. Get used to it. See where it comes in handy. I've done it for like just for personal stuff a lot. I've used it a lot for like my— I'm the president of the HOA. We got RFPs for new management. 3 companies. I said compare and contrast the, the RFP responses from these 3 documents, stuff like that. You know, doesn't give me an excuse not to read them, but it certainly makes it easier to condense and suggest because I don't have time for all that. And, you know, though, so I think those are the kinds of things where they can help with your day-to-day task, administrative task, up to very sophisticated legal work, again, with verification and checking. So I'm going to stop the recording. And again, thank you all for coming. Next month we will be going through Windows 11. So if you have recently transitioned to Windows 11 or considering it, I'm going to show you some of the ins and outs of that.