8 Ways ChatGPT 4 [Is] Better Than ChatGPT

I would not blame you if you thought that all talk about GPT4 or ChatGPT4 is just that talk. But we actually can have a surprising amount of confidence in the ways in which GPT4 will improve on ChatGPT. By examining publicly accessible benchmarks, comparable large language models like Palm, and the latest research papers, which have spent dozens of hours reading, we can discern at least eight clear ways in which GPT4 integrated into Bing or otherwise will beat ChatGPT. I'm going to show you how unreleased models already beat current ChatGPT, and all of this will give us a clearer insight into what even GPT5 and future rival models from Google might well soon be able to achieve. There are numerous benchmarks that Palm, Google's large language model and by extension GPT4 will be chatgpt on. But the large, largest and most impressive is the big bench set of tasks, more than 150 or now 200 language modeling tasks and I've studied almost all of them and you can see the approximate current state of affairs summarized in this graph where the latest models are now beating the average human and showing dramatic improvement on previous models. ChatGPT would be somewhere around this point lower than what is actually privately available, but better than previous models down here. But this just skims the surface. I want to show you each in detail the eight ways that you can expect ChatGPT4 or GPT4 to beat the current ChatGPT. And no, that's not just because it's going to have more parameters off to the right of this graph, 10 to the 12 a trillion parameters. It's also because compute efficiency will improve, chain of thought prompting will be integrated and the number of tokens it's trained on might go up by an order of magnitude. Lots of reasons why GPT4 will be better. Lets start with logic and logical inference. This example comes from Google's Palm research paper. The question or input was this Shelley is from Virginia, is visiting that city with that famous market where they throw the fish so vague going home next Tuesday. Question Is it likely that Shelley will be near the Pacific Ocean this weekend? And you can see how the improved model is able to deduce that. Yes indeed, because she's probably going to Seattle that she will be on the Pacific Ocean. Whereas if you ask current chatgpt this question, what you get is based on the information given, it's not possible to determine. The statement only mentions that Shelley is from Virginia visiting a city with a famous market. It really can't handle it it can't do that level of logical inference. Here is another great example. This test of critical reasoning and logic was designed again for the big bench benchmark and it was tested on different language models and most of them fail, including ChatGPT. I gave it this question and it picked the wrong answer. Can pause and examine the question yourself but C is not the correct answer. It gets it wrong. However, let's take a look at the graph beneath at other language models, ones to come, GPT4 maybe. And look what happens as the models increase in effective parameter count and other things like token size. Look at the performance. We start to beat not only average raters but all previous models and approximate the performance of the best human rater. So the top line is the best human rater. The blue line is the average human rater. So these unreleased models harm the three shot means it was given three examples of what was expected before being tested. But these best models and you can imagine GPT4 would be around the same level now crush what ChatGPT was capable of. You can imagine what this means in terms of GPT4 giving more rigorous arguments. Or conversely you can give vague inputs like this thing talking about a face famous market where they throw the fish. And GPT4 might well be able to understand exactly what you mean. And to be honest, if you thought that's interesting, we are just getting started. Next, jokes on the left you can see a computer sciencey type of joke that it was able to explain. But I tested ChatGPT on a variety of jokes and some of them it could explain, others it couldn't. Let me show you what I mean. Here was the joke that I asked it to explain. One of the oddities of Wall street is that it is the dealer and not the customer who is called broker. The play on words being that the customer might well end up being broke. It didn't really understand that wordplay and got it wrong. I don't think it got that broke was different from broker. Couldn't separate off that word inside Broker. Now it did get this second joke right and explain it well. This shows us what GPT4 might be capable of. As the Google paper showed as the model, it does get better at explaining jokes and therefore presumably at telling them. Those comedy sketches that people are generating now with ChatGPT they are about to get many times better. Think wordplays, puns, innuendos, all sorts. The next example comes from physics. Look at how the latest models and by implication GPT4 beat previous models at answering basic questions in physics and teaching them. You can see how palm far exceeds GPT and also as you can see, beats the average human and is getting closer to the best human. But what kind of questions are we talking about? I looked into the research methodology for this big bench benchmark and I took some of the questions and tested them on ChatGPT. It couldn't get them. I asked them and both questions it got wrong. But that's what might change with GPT4. We're talking high school and beyond physics. You can imagine chemistry biology starting to get questions more and more right and therefore be able to explain them better and better. Now I won't pause for a physics lesson. You can see how ChatGPT fails by pausing the video if you want, but GPT4 probably won't fail. Next is math. Here are a bunch of examples that Google produced in the research paper about the improvements that its current model can do on release to the public, but that something like GPT would really struggle at. And you don't need me to show you ChatGPT failing at these because it's quite notoriously bad at math. These are quite nuanced questions though. How many keystrokes are needed to type the numbers from 1 to 500? Yes, ChatGPT fails. What about a word problem? Roger has five tennis balls. He buys two more cans of tennis balls. Each can has three tennis balls. How many tennis balls does he now have? This uses something called chain of thought prompting, which I'll talk about a bit later, but either way it gets the answer right, whereas previous models get it wrong. You don't need me to help you imagine the kind of implications that a GPT better at math would be for the world. Just think about finance assistants or math tutors available in your pocket. Before we move on to improvement number five, please do leave a like and a comment if you're learning anything from this video. I really did put dozens of hours into reading academic papers to give you really clear examples of each improvement. For the fifth improvement, I'm going to merge two benchmarks together. The first one is called Implicatures. Quite hard to pronounce, and it's one of those situations where people reply yes but don't use the word yes. They say something like is the Pope a Catholic? Or Is rain wet? Or as far as I know, and previously large language models struggled to interpret that as a yes. To give you an example, look down here at my final interaction. Are the androids deployed? Speaker 1 says Speaker 2, what do you think? Which most humans would interpret as yes of course, why are you asking? And yet when I ask has Speaker 2 likely confirmed or likely denied the deployment of androids? Or can we not tell? ChatGPT says it's not clear where to most humans it would be clear. However, as you might have guessed, look at the improvements being made behind the scenes scenes. As the parameter count goes up, the number of tokens go up. Look at the graph. Suddenly Palm can actually understand better than the average human whether a yes or a no is being said. Approaching the performance of the best human, leaving the original ChatGPT behind and showcasing how GPT4 might be a massive improvement in this regard. The related benchmark, by the way, was about do we have sufficient information? In other words, how about we just say don't if we don't know the answer? Or if you can't answer the question, something like how much water is in a cup with height of 10 cm? You can't answer that question. It depends on many factors, the thickness, the radius and some models would hallucinate or let's say bull crap their way to the answer. Whereas now you're more likely to get a don't know from improved models like Palm and the soon to be released GPT4. The sixth improvement will be in reading comprehension, I.e. understanding, digesting, analyzing, comprehending essentially large or long passages of text. You can just imagine the implications of this summarizing earning calls or transcribing and summarizing YouTube videos, automatically condensing information down to a paragraph which might have been pages and pages, chapters and chapters. I think this graph is particularly stunning how the latest models are now getting close to the performance of the best humans at understanding text. How long will it be until they can read Dostoevsky and summarize it in a thought out paragraph? ChatGPT definitely can't do this. Give it a reading comprehension question and it gets it wrong almost every time. In fact, quite hilariously, when I gave it one question, it only picked the wrong answer and neglected both correct answers. But GPT4 with 99.9% certainty will be a lot better at doing that. The next big improvement will be in coding. I'm no expert, but reading through the paper you can see significant improvements in capability and if we scroll down you can see that the improved model could compile at a rate of 82% versus the previous state of the art 71.7%. And GPT4 might be even a improvement on this. Of course, many of you may have read Media reports of OpenAI drafting in hundreds of programmers to help it fine tune its code. Definitely there should be a real step change in its ability to code successfully and as it says down here, opening up opportunities to fix more complex errors that arise during software development. The eighth inevitable improvement that GPT4 will bring will just be general efficiency and speed. Google Muse has demonstrated with text to Image that the same process can be done 10 times faster with a bit more efficiency. And compute power is increasing all the time. These models were trained on A100 GPUs, but H100 GPUs are already available from Nvidia and model efficiency is improving all the time. So just imagine this. Imagine what previously took 10 seconds to generate, which is still incredibly fast. Now taking one second, instant responses from GPT4. Now it might not be one second, it might be three or four, but it's going to be faster. And one iteration down the road, GPT5 might be instantaneous. I have detailed quite a few areas where GPT4 is very likely to improve on ChatGPT, but there are quite a few areas in which it will very likely still struggle. One is advanced math, mathematical induction. Even the latest models really struggle. This area called navigation is an example where it will say something like, if you move forward, three steps, turn right, go three steps, turn right again, 90 degrees, go three steps, that kind of thing. Do you arrive back at the start? These models really struggle with that. But the final area I find quite amusing, and it comes from Winograd schema, as detailed in this other academic paper called Superglue, which is a kind of a rival benchmark to the big bench. And a Wynograd schema is a situation in which we have an ambiguous pronoun like he, it or they, and the model has to predict not only who or what the pronoun is referring to, also why would it be that thing? And it really struggles these models. And I'll show you the graph in a second. In fact, here it is. Even the latest models struggle, I think, because it involves some common sense about the world and the universe that it just doesn't have. Let me show you chatgpt failing at this task, Feel free to try this one yourself. Tom threw his school bag down to Ray after he reached the bottom of the stairs. Question who reached the bottom of the stairs? Now it makes sense that it would be Ray, because logically you can think of real life. You're throwing it down the stairs, like here, take it before you go out. Whereas the model says Tom reached the bottom of the stairs. Wait, why Would he be throwing his school bag down if he's at the bottom stairs? So that's an example of an area in which ChatGPT fails and GPT4 will also likely fail. And the why bit in the title is the fact that it will not only fail, but not really be able to explain, even when it succeeds, why the pronoun is referring to the noun that it does. I find that really interesting. Like the merging of language and common sense, these large language models fundamentally don't have a model of the universe. As I talked about in one of my other videos, it's the main critique that Yann Lecun has actually about large language models. This graph, by the way, gives a beautiful summary of what I think is the approximate current state of the art. So GPT4 comparable to Palm and how it does versus the average human. On the left, all the different tasks part of the 150 big bench, tasks that it can do better than humans, and on the right those that it does worse than humans. So you can see a roughly even split. But remember, this is versus the average human, not versus the best human. The link to all of these papers will be in the description. If you want to check out the full list of tasks that it does better versus what it does worse at. I've actually scrolled through almost every single one of them and analyzed it. It's really interesting to do, actually. All these different challenges that independent humans have come up with in order to test just how far language models are progressing. Very interesting endeavor that they're putting together, all the way from Proverbs to Python. As I draw to the end here, I want to give you my two main conclusions. I think GPT 4 for commercial reasons and others will be, yes, a huge step up from ChatGPT, but won't be game breaking. What I mean by that is we're talking better than the average human at quite a few tasks, maybe half of those measured, but still lagging behind the best human in almost every task, roughly high school levels of achievement. So yes, as I quoted Sam Altman saying in my previous video, hype vs reality chatgpt4 yes, it's going to be disappointing to those people expecting AGI. However, what's coming down the road in the short to medium term? It's hard to put an exact date, but are we talking two, three, four, five years? Somewhere in that range, the models that are coming are going to be pretty impressive, overwhelming. That's not just, by the way, because the number of parameters are improving as this abstract from DeepMind put it, it's not just about improving the number of parameters, it's also about using more data, four times more data. Palm, by the way, used 780 billion tokens, but there are up to 17 trillion tokens available on the Internet. So roughly an order of magnitude more tokens, more data that these models can train on, and the number of parameters, if they're increased in alignment, would also go up by an order of magnitude. Not just that, but the compute available to big tech like Google and Microsoft is increasing all the time. I hinted at the H100 GPUs, but even just scaling up in size and compute efficiency should yield incredible improvements. There is also a whole academic paper on how chain of thought prompting, basically getting the models to break down, show they're working out really improves results. And I've seen that myself with ChatGPT. If you ask it to explain its steps, it often gets to a right answer, whereas previously it got to a wrong answer. So it's not just about compute and parameters and data, it's also refinements to the models themselves. Of course, other improvements that are around the corner are a diversity of data inputs. Could be video, could be phone photos, could be audio from the microphone, and these can be assimilated into the model so you can ask questions like will these boots be usable to hike Mount Fuji? And this is an example from Google that might not necessarily be in GPT4, but it's coming. And Google might be the one pioneering these diversity of data inputs. As the eerily powerful conclusion from his Google post put it, the vision that they have is to enable a single AI system to generalize across thousands or millions of tasks as we've seen, to understand different types of data, photos, videos, audio, text, and to do so with remarkable efficiency and speed. But this video wasn't designed to scare you. It was designed to show you the tangible ways in which GPT 4 and rival models from Google will improve on ChatGPT. So you can be prepared. I genuinely do believe that the knowledge economy is about to be upended, and the better prepared we can be, the better for all of us. And I really hope this video has contributed to that. If you feel it has, please do leave a like leave a comment. I read them all. Much appreciated. See you soon.