Enrico Schaefer explains why ChatGPT invents false legal citations and how to stop it. The fix: upload source documents, use fresh threads for each task, and set explicit constraints. Skills, code-based guardrails, can enforce rules with high reliability.
Chapters
Step into a world where machines dream. AI whispers secrets never seen. Tomorrow glows bright on the screen. Future unfolds. Come join the team.
So today we're going to talk about AI hallucinations, which is a topic near and dear to every attorney's heart because we're the ones that see the headlines of the attorney who Sent a brief to the court, and the brief included citations that were not real and got in a lot of trouble. And so we don't want that. But hallucinations are more than just case citations. They can occur in a variety of different ways within the AI model. And if you understand how these foundational models work, you can reduce the risk of hallucination. And you always should be cross-checking to make sure there are no hallucinations, and there are techniques for that as well. If you learn how to use these AI tools correctly, you can get beyond your initial thought of, I'm not going to use it because it might hallucinate. If that's your thinking, then you're already lost because these tools can do so much and you are the one that can reduce hallucinations. So let's take a look at why those things might occur. In the case of a case citation that was not in the database that got the the model got trained on, didn't have access to Westlaw's database of cases, number one. But number two, it also isn't designed to do case citations well, or at least the prior versions weren't at all designed to do that well. It's guessing the next word, and the next word may or may not be the correct second party, and it may or may not be the correct page number, volume number, etc. It's just going to correct what is on a vector basis the most likely next word. Now they've corrected a lot of that with these more recent versions, and they've trained it, or they've it or they've limited it to know that you're looking for a case citation. And most of the time it gets them right. You still need to go grab the case, upload the case into ChatGPT, and make sure that it's accurate. And if you upload a case directly into the platform, then you're going to get really great results. So that's one thing that you need to understand. The other thing is that hallucinations go way beyond case citations. You can still have hallucinations because you're dealing with this token chain, which is the input, the output, the knowledge, and the instructions that you uploaded, as well as the memory now, which can be on a system level or it can be on a project folder level in ChatGPT. So as you start to build out these specific tools for specific clients and specific projects, and you're uploading documents, you're uploading instructions, you're creating all these different chat threads that are getting remembered, you have to remember that all of that may end up in your token chain, even on a new thread. But it's certainly going to end up in a token chain if you continue a thread, if you do a search to find where you left off on a contract drafting process, and then you continue in that thread. Let me give an example how you might see a hallucination based on the wrong strategy in how you are using ChatGPT for contract drafting. So here's a scenario. You are going to draft a contract. You are Taking one of your great prior work products, you're using that in order to, on a paragraph-by-paragraph basis, build out the structure for a new contract for this client, given the exact nature of what the client is doing and the risk tolerance of that client, specific goals of that client, all of which have been uploaded by prior transcripts into the GPT. And you start to build out the structure for a client contract, and you start to build out specific terms. You work with the AI tool, and eventually you end up with a really nice version of a version 1 contract. You send that to the other party. And the other party sends it back with some marked-up changes. You want to kind of understand those changes. You want to use AI tools to help understand those changes. You do a search in ChatGPT to find the old thread, and then within that thread, you go to the bottom and you upload the latest version. You say, hey, what changes were made and what are their significance to my client, given my client's goals, given the transcripts that I uploaded previously? Great. You might get as an output some changes that the AI tool says were made to the contract which were not in the version that you sent to the adverse party. It hallucinated that output, that answer to you, and the question is why. And if you understand how these tools work, you can understand why, and then you can use them more strategically in the future. And here's what will sometimes happen: because you need to think about what's going into the token chain for a particular conversation, whatever's in that token chain is going to be part of the knowledge that going to be used that's going to spit out the next word. So if you go continue that prior thread that had all the word product, including the terms that were included, the terms that were modified, the terms that were excluded before you got to the final version, all of that's going to be in the token chain. So when you say compare the versions, it's very easy for the AI to get confused with all that, what is now irrelevant context, for it to then say, hey, it changed the term between your prior version that you sent and the version you received back, but really it's between some discussion you had in a much prior version up the token chain that it is identifying and flagging for you. So that is an end-user-induced hallucination that you should have seen coming if you know how to use these AI tools correctly. So what would be a better strategy? A better strategy would be to work within a project folder with the uploaded knowledge about the client's goals and the background information concerning the client, but to start a new thread and to tell that thread, hey, I want you to focus exclusively on 2 versions of a contract that I'm going to send you. One is the one I sent to the adverse party, the other is the marked-up version that I got back, and we're going to compare those versions. But I don't want you to bring any other contract version, language, or prior work product into this particular exercise. And then you're going to upload the email chain with the With the version that has the track changes on and the prior version into a new thread. And you've limited what's going to go into the token chain in that thread, because even though there's memory and it does have access to that other prior conversation, that contract drafting thread, you've told it not to consider that as part of the work product. You put guardrails on this particular thread within this particular project for this particular client, and you've reduced— The opportunity for hallucination. So there are hundreds of these types of issues that come up every week if you're using these tools effectively, where it didn't work well and you need to ask yourself, what did I do wrong? Not what did AI screw up, not how is AI inadequate, but what did I do wrong? Because if you understand how the tools work and you use them correctly, you're going to have very few problems with it. Now, you should always ask the thread. Did you in fact limit your exercise as instructed? Is there anything that you brought into this conversation that I should know about as we continue the analysis? You can work with the thread in order to confirm that instructions were followed. Now, here's bonus, bonus content for you that is going to even make things better in the future. So we had Claude introduce something called skills. Skills are like supercharged guardrails that get put on a ChatGPT or a Claude session. ChatGPT is going to integrate the same technology into their platform. And what these skills do is it's code-based rules that get uploaded as a thread, as a file, as an instruction into a particular task. And the chance that the AI thread is going to follow those rules is really, really high. So instead of plain English trying to guide to AI, hoping that when you tell your kid to come home by midnight, they're home by midnight. This new application of technology, this new skills-based code that you're going to be identifying and uploading in the future is going to put guardrails in to make sure your kid gets home at midnight. And these types of things are coming down the pipe. That's why AI is so exciting. It's not only what it can do today, but it's all the problems that you've been told exist that are being fixed and worked on by the most brilliant minds in the universe. My name's Enrico. We'll see you next time.
