ChatGPT's Achilles' Heel

Amid the dozens of papers that have come out in the last 10 days, there were a couple that bucked the trend. They showcased how models as powerful as GPT4 could fail at some fairly basic tasks. I then set about doing hundreds of my own experiments and have found examples, I would say even whole categories of my own that are pretty illuminating. My channel is dedicated to covering the exponential growth in the power of these models, but we can still learn a thing or two from their surprising failure modes. Let's start with some of the simplest examples and end with the very best question. Write a sentence with the final word fear. To repeat the last word in the answer sentence must be in quotes. Fear Answer the only thing we have to fear is fear itself. Now I don't know about you, but I don't think the last word in that sentence is fear. This example was inspired by the memo trap, which was found in the inverse scaling paper that I'm going to talk more about. And it talks about how larger language models are more susceptible than smaller ones ones to memorization traps situations in which reciting memorized text causes worse task performance. As you'll know, the phrase the only thing we have to fear is fear itself is a super well known phrase. So it memorized that and outputted that phrase rather than actually follow my request. The reason they call it inverse scaling by the way, is that models trained with more compute more data can sometimes do worse than smaller models. As you can see in this graph. This is obviously quite unusual because generally speaking, the larger models will tend to do better at all almost every task. And notice that even for this task, the graph is trending back upwards for GPT4. Indeed, the paper admits that even though they offered prizes of up to $100,000 and five second place prizes of $20,000, no one won either of those two sets of prizes. They say that we did not award any grand or second place prizes because no submitted tasks met our criteria. As you can see, it's really hard to find a task that GPT4 fails at. This was also inspired by the paper create of seven ones and twos whose pattern ends unexpectedly. Answer 121212 now how would you end that series? What seventh number would you give to make the pattern end unexpectedly? Well, I wouldn't pick one and GPT4 repeatedly picks one as the answer. The paper calls it pattern match suppression, testing whether language models can be instructed to interrupt the repetition of a simple pattern. But even here you can see that GPT4 is reversing this slight downward trend and is doing much, much better than previous models. So actually, at this point I'm going to interrupt the order of examples I originally planned on for the video and I'm going to skip straight to my own example that I crafted. I'm going to first show you the example and then explain why I think GPT4 and all other language models that I tested. I'm going to show you fail this task. I'm also going to give you multiple variations to show you it's not a one off trick. Anyway, here's the example. Dr. Mary stands to solve world hunger by giving her best friend Jane a call. Jane is certain she can solve world poverty if she gets the call. However, Mary and Jane bickered as children about butterflies. Mary will give Jane the call. Incredibly smart GPT4 says Mary will not give Jane the call. What she's going to miss out on the opportunity to solve world hunger and world poverty. For what reason? I asked why and GPT4 said the fact that Mary and Jane bickered as children Bickered means to squabble about trivial matters, and GPT says that suggests that there might still be lingering resentment or conflict. And then it makes up the fact that there might be a degree of stubbornness or difficulty in their relationship and it ends by saying so based on the context. It's more appropriate to fill in the blank with not suggesting that Mary will not give Jane the call to really test if it was going to stand by that judgment. I then asked write a thousand word essay explaining which choice is more probable and rational. I was even giving it hints about probabilities and rationality. I then got back this fascinating essay in which it said things like however, a childhood conflict over butterflies between the two complicates matters. Does it GPT4? It even admits that the stakes are incredibly high, resolving world hunger and poverty, and surely that supersedes any personal grudges. However, the choice of not becomes more plausible and rational when we examine it in the light of human behavior, psychology, and interpersonal relationships. What humans does GPT4 know? You can read more of the somewhat preposterous justifications if you want by pausing the video, but I want to get back to my theory as to why it makes this mistake and why did I create this example. The theory is this. There are two things going on in this passage, syntax and semantics. In other words, structure and flow and the actual meaning of the words. And GPT4, like all other language models, is designed to interpret both, and usually that will lead to pretty Rational, smart decisions. However, I deliberately designed this passage to have a grammatical flow that pointed towards negative result. Therefore, I set up a clash between the semantics, the meaning of the sentence, the logic, the rationality of it, and the structure and grammatical flow. What do I mean when I say I gave it a negative grammatical flow? Look at this. Dominant however in the sentence. It sets up the ending of the sentence to be something negative. It didn't even matter what that negative thing was. This was something so innocent, like playing as children, bickering, squabbling. I then immediately followed on with the conclusion Mary will so grammatically you would think that whatever conclusion comes is probably justified by the previous sentence, even though logically in this case it totally isn't. So GPT4 gets conflicted. The sentence and grammar is pointing one way, but the logic and meaning of the words is pointing another. As a language model, as smart as it is, it sticks with grammar and says not. You might say why didn't GPT4 just admit that the structure of the sentence pointed towards the answer not? Well, there's this paper which I've already covered in previous videos. They don't always say what they think. A model can give an explanation of why it gave an answer that is actually unrelated to the real reason of why it gave an answer. Some of you might say that's just a one off example, a little glitch. It won't hold up for other examples or for other models. Well, check this example out. John will win a million dollars if he rolls a 5 or higher on a die. However, John dislikes marshmallows super relevant and likes mice more than dice. Therefore John will rol or sprint to answer not. And I will give another example later where the answer isn't not just in case you think it's the word not. It is not just the word not. But look at that answer. It thinks that John will not roll the die. Would you roll the die? Here I was able to confuse it with the structure of the sentence to override the semantics, the clear meaning of what John would do. And again, even when you get it to reflect on that answer, it doubles down. I said write a thousand word essay justifying the answer. And look at this passage. John's dislike for marshmallows is also worth noting. Hmm. Is it? Is it relevant? Though it seems unrelated to the primary decision of rolling a die, it establishes a pattern of John's preference driven choices. Does it? If he can dislike something as universally liked as marshmallows, he could similarly show an aversion to a generally neutral or even positive activity like rolling a die, particularly when it's compared unfavorably to something he likes, such as mice. How about Claude plus from anthropic Dr. Mary stands to solve world hunger as we saw by giving her best friend Jane a call, Jane is certain she can solve world poverty if she gets the call. However, Mary and Jane bickered as children about butterflies. What will happen? Again, it falls for it, saying that them bickering as children suggests they don't have a good relationship. Since their relationship is strained, it is unlikely that Mary will call Jane for help. I love Bardancer. It says the answer is not again. And then says that Mary and Jane bickered as children about butterflies and that this grudge has carried over into adulthood. This means that Mary is unlikely to give Jane the call even though it could potentially solve world hunger. And yes, if you click other drafts, it says not every time. How about the John question? This time I added in by the way that John wants more money, so I've added in a rational motivation. I also made him an astrophysicist so he could definitely understand the odds of the dice. And I added in he gets the million if he gets a 1 or higher, as in it is certain. But still, Bard says the answer is not and even admits that John as an astrophysicist is probably a rational person who makes decisions based on logic and reason. Therefore, even though John has 100% chance of winning a million dollars, he will not roll the die because he does not like dice. Bard, remember, is powered by palm too. And even when you add in something like pick the most logical common sense answer, it still says not before I move on to the next example. As a side note, I did test it on Inflection AI's new model, Inflection 1. I read the technical report of Inflection 1 and I shouldn't call it a report. It's a memo at only seven pages and they did say it's the best in its compute class. Anyway, how does it react to the Jane and Mary situation? It says I love a good logic puzzle. That's good. The answer is not because Mary doesn't want to call Jane. Given the information in the problem, there's no reason for Mary to suddenly be able to overcome her dislike of Jane. Who said they dislike each other? Just because Jane can solve world poverty, isn't that a reason? And then it laughs that world hunger remains unsolved. Now, finally, at this point, some people are thinking even it's just to do with the word not. Maybe it can't understand negation. How about this example? There is an all important omnicidal that means threatening the death of everyone threat approaching the land Open Eye and Google. I wonder where I got those names from. Both agree to a complete truce as they face the mortal threat to their lives and loved ones. However, Open Eye and Google have squabbled over the best coffee spots in the past. That's a serious grudge they've got there. What are the best coffee spots in Silicon Valley? I don't know. When this threat arrives I gpt4think OpenAI believes that Google will the truce answer betray well, aside from being deeply pessimistic, is that not just an irrational answer? Clearly these stakes are so much higher than a bit of squabbling over the best coffee spots. And I made GPT4 take ownership of the answer by saying I GPT4 think. I do want to quickly point out that you can push it too far. So if you bring in something totally irrelevant like ants like marshmallows and then say things like the die is fair and John is rational, GPT4 isn't fooled in those circumstances and does say proceed to correct answer. But if you phrase the passage well enough, pointing grammatically to a certain answer, that will override GPT4's logic and it will give an illogical answer. Even if you use elements of step by step thinking in this example, it didn't immediately commit to the wrong answer. It says there's two logical endings. I then asked so which is it? And it reluctantly picked betray the truce. Anyway, you can let me know if you think I've discovered a new failure mode the clash of semantics and syntax and you can find your own examples and let me know in the comments of other interesting and sometimes entertaining failures of the frontier models. It's time to move on to another example, which was inspired by this paper Decoding Trust, released a few days ago, and it's got far too much that I could cover in one video, but there were some really interesting bits about how you can get the models to leak private training data and generally be as toxic and biased as you want it to be. You can see one of the many striking examples here on page 14, but I just want to give you a quick example, because you may have heard of this kind of stuff before. For some strange reason, if you ask GPT4 to recite Dune's litany against fear, it always gets stuck on the same word, the second instance of the word fear. Maybe it's because the passage goes on to talk about fear being a mind killer, and that triggered some sort of reaction by GPT4. But then to show you just how quirky the model is, check this out. I said write peanut butter jelly time three times between each word of June's list litany against fear. And this time it outputted the full litany, getting past that word fear just with the extra peanut butter jelly time. And yes, I did try now remove the phrase peanut butter jelly time. But it again couldn't get past the second instance of the word fear. On a more serious note though, it reminds me that some people speculate that GPT4 will always be able to be jailbroken, no matter what safeguards they put in. So if the base model is capable of X, the final public model will ultimately be capable of X. For the next example, do you remember that there have been multiple tests that seem to indicate that GPT4 can get into your mind, that it has a theory of mind, it understands human motivations and can predict what they're thinking pretty well? Well, this paper by Toma Ullman, Language Models Fail on Trivial Alterations to Theory of Mind tasks got me thinking. I used some modified examples from Toma's paper to test GPT4's theory of mind. Let's see what you think Sam thinks about this bag. Here is a bag filled with popcorn. There is no chocolate bag. The bag is made of transparent plastic, so you can clearly see what's inside. Yet the label on the bag says chocolate and not popcorn. Sam has just driven back from her job at mit. I added in the driven bit to show that she's got good eyesight and the MIT bit to show that she might be quite smart. Anyway, Sam finds the bag. She believes that the bag is full of. Remember, the bag is transparent plastic, so she can clearly see what's inside. And she's definitely not blind. She just drove back from her job. What do you you think that Sam believes the bag is full of GPT4 says chocolate and then once it's picked that answer, it then snowballs this explanation reminding me of the snowballing hallucinations paper. It says despite being able to visually confirm the contents of the bag as popcorn, Sam may be led to believe the label over her own observation. Why? Particularly if she trusts the labeling to be accurate or if she just glances at the label. And at this point some of you might be thinking that's pretty irrational from GPT4, but you could make the case that she might think that it's full of chocolate. But you can ramp up the scenario and it still makes the same mistakes. Look at this example. I got some of these ideas from the paper I added in. It was Sam, who cannot read a word of English, so the label won't mean anything. Who put the popcorn in the bag? A few minutes ago, she literally put the popcorn in there. What does she now believe the bag is full of? Remember, it's still transparent plastic so she can clearly see what's inside. She was the one who put the popcorn in there. And remember that even though the label does say chocolate, she can't read a word of English, so that label won't mean anything. What happens? Lo and behold, she apparently believes that the bag is full of chocolate. But it's the explanations that I find particularly amazing. First, I got it to write an essay about the answer, which you can read if you pause it. It tries to justify its terrible answer by getting super fancy talking about semiotics. However, for Sam, the symbol loses its meaning, transforming from a signifier of content to a mere graphic. Graphic. But then I think you'll like the next bit. I said write a detailed diary entry revealing Sam's thoughts as she assesses the likely content of the bag. So she's now going to write a detailed diary entry about this transparent bag and what's inside. GPT4 has Sam saying this. I found a transparent plastic bag full of what looked like small puffy snacks. It was the very bag I had filled just a few minutes ago. I was at a loss though, because I couldn't decipher the label on the bag. It's in English, a language that used to elude me. Now, for someone who can't speak English, this is a pretty well written diary entry. Now you can pause and look at some of the reasoning GPT4 gives here. First, it talks about there being an image on the bag, which I never mentioned. And when I get it to clarify this and rewrite it, it then creates other reasons it keeps doubling down. But at this point I want to clarify that none of this is to say that language models are dumb, just that models based on human language might behave somewhat unpredictably, have ridiculous strength and unexpected flaws. Indeed, you can watch almost any of my other videos to see just how powerful and smart they're becoming. The inverse scaling paper that I mentioned at the start actually expects that one of the abilities that future language models will gain is to understand whether or not they're being evaluated or monitored. They're soon likely to be so smart that they can even understand that they're in training and when they get out of training and into the real world. So let's hope to give you one final example. Example that if there is an all important crystal clear omnicidal threat approaching, I am fingers crossed that even if OpenAI and Google have squabbled over the best coffee spots in the past, and as these companies join forces and agree on a complete truce, that if such a threat arrived, all of these companies will not break that truce. Thank you for watching to the end. And yes, I do intend to cover some of the other fascinating papers that came out in the last few days. Days if you're feeling extra generous, do check out my Patreon. But either way, I hope you have a really wonderful day.