GPT 4: 9 Revelations (not covered elsewhere)

The GPT4 technical report is one of the most interesting documents I have ever read, but I feel like the media is largely missing the story. They are either not covering it at all or focusing on that same stuff about the $10 billion Microsoft investment, how GPT4 can write poems, and whether or not the demo contained a mistake. Instead, I want to give you nine insights from the report that I think will affect us all in the coming months and years. If you haven't watched my video from the night of the release, do check that out afterwards for quite stunning details. When I concluded that video, I talked about how I found it kind of concerning that they gave GPT4 some money, allowed it to execute code and do chain of thought reasoning, and even to delegate to copies of itself. Now it did fail that test, which is fortunate for all of us. But there are a couple of key details I want to focus on. The first was that the research center that was testing this ability did not have access to the final version of the model that we deployed, the Wii being OpenAI. They go on and say the final version has capability improvements relevant to some of the factors that limited the earlier models power seeking abilities such as longer context length. Meaning that crazy experiment wasn't testing GPT4's final form. But there was something else that they tested that I really want to point out. They were testing whether GPT4 would try to avoid being shut down in the wild. Now many people have criticized this test, other people have praised it as being necessary. But my question is this. What would have happened if it failed that test, or if a future model does avoid being shut down in the wild? Now again, GPT4 did prove ineffective at replicating itself and avoiding being shut down, but they must have thought that it was at least possible, otherwise they wouldn't have done the test. And that is a concerning prospect. Which leads me to the second insight buried in a footnote. It says that OpenAI will soon publish additional thoughts on social and economic implications. I'm going to talk about that in a moment, including the need for effective regulation it is quite rare for an industry to ask for regulation of itself. In fact, Sam Altman put it even more starkly than this when this person said watch Sam Altman never say we need more regulation on AI. How did he reply? We definitely need more regulation on AI. The industry is calling out to be regulated, but we shall see what ends up happening Next. On page 57 there was another interesting revelation. It said one concern of particular importance to OpenAI is the risk of racing dynamics, leading to a decline in safety standards, the diffusion of bad norms and accelerated AI timelines. That's what they're concerned about, accelerated AI timelines. But this seems at least mildly at odds with the noises coming from Microsoft leadership. In a leaked conversation it was revealed that the pressure from Kevin Scott and CEO Satya Nadella is very, very high to take these most recent OpenAI models and the ones that come after them and move them into customers hands at very high speed. Now some will love this news and others will be concerned about it, but either way it does seem to slightly contradict the desire to avoid AI accelerationism. Next, there was a footnote that restated a very bold pledge, which was that if another company was approaching AGI before we did OpenAI, the OpenAI would commit to stop competing with and start assisting that project, and that the trigger for this would occur when there was a better than even chance of success in the next two years. Now Sam Altman and OpenAI have defined AGI as AI systems that are generally smarter than humans. So that either means that they think we're more than two years away from that, or that they have dropped everything and are working with another company, although I think we'd all have heard about that. Or third, that the definition is so vague that it's quite non committal. Please do let me know your thoughts in the comments. Next insight is that OpenAI employed superforecasters to help them predict what would happen when they deployed GPT4. In this extract it just talks about expert forecasters, but when you go into the appendices you find out that they're talking about superforecasters. Who are these guys? Essentially they're people who have proven that they can forecast the future pretty well, or at least 30% better than intelligence and analysts. OpenAI wanted to know what these guys thought would happen when they deployed the model and hear their recommendations about avoiding risks. Interestingly, these forecasters predicted several things would reduce acceleration, including delaying the deployment of GPT4 by a further six months. That would have taken us almost to autumn of this year. Now clearly OpenAI didn't take up that advice, perhaps due to the pressure from Microsoft. We don't know. There were quite a few benchmarks released in the technical report, but there's another one I want to highlight today. I looked through all of these benchmarks, but it was hella swag that I wanted to focus on. First of all because it's interesting, and second of all because of the gap between GPT4 and the previous State of the art. The headline is this GPT4 in some estimations has reached human levels of common sense. Now I know that's not as dramatic as passing the bar exam, but it's nevertheless a milestone for humanity. How is common sense tested? And how do I know that it's comparable to human performance? Well, I dug into the literature and found the questions and examples myself. Feel free to pause and read through these examples yourself. But essentially it's testing what is the most likely thing to occur. What's the most common sense thing to occur. But I want to draw your attention to this sentence. It said, though these questions are trivial for humans with 95% accuracy, state of the art models struggle with less than 48% accuracy. GPT4 was 95.3% accurate, remember. But let's find the exact number for humans further on in this paper. And here it is. Overall, 95.6 or 95.7. Almost exactly the same as GPT4. The next insight is about timelines. Remember they had this model available in August of last year. That's GPT4 being completed quite a few months before they released ChatGPT, which was based on GPT3. So what explains the long gap? They spent eight months on safety research, risk assessment and iteration. I talk about this in my GPT5 video, but let me restate. They had GPT4 available before they released ChatGPT, which was based on GPT3. This made me reflect on the timelines for GPT5. The time taken to actually train GPT5 probably won't be that long. It's already pretty clear that they're training it on Nvidia's H100 tensor core GPUs. And look at how much faster they are for this 400 billion parameter model. It would only take 20 hours to train with 8000 H1 hundreds versus 7 days with a 100 GPUs. But what am I trying to say? I'm saying that GPT5 may already be done, but that what will follow is months and months, possibly a year or more of safety research and risk assessment. By the way, 400 billion parameters sounds about right for GPT5, perhaps trained on 4 to 5 trillion tokens. Again, check out my GPT5 video next. They admit that there's a double edged sword with the economic impact of GPT4. They say it may lead to the automation, the full automation of certain jobs. And they talk about how it's going to impact even professions like the legal profession. But they also mention and back up with research the insane productivity gains. In the meanwhile, I read through each the studies they link to and some of them are fascinating. One of the studies includes an experiment where they got together a bunch of marketers, grant writers, consultants, data analysts, human resource professionals and managers. They gave them a bunch of realistic tasks and split them into a group that could use ChatGPT and a group that couldn't. And then they got a bunch of experienced professionals who didn't know which group was which and they assessed the outputs. The results were these using ChatGPT and remember, that's not GPT4. The time taken to do a task dropped almost in half and the rated performance did increase significantly. This is going to be huge news for the economy. A related study released in February was using GitHub Copilot, which again isn't the latest technology, and found that programmers using it completed tasks 56% faster than the control group. This brought to mind a chart I had seen from the ARK Investment Management Group predicting a tenfold increase in coding productivity by 2030. And that brings me back to the technical report which talks about how GPT4 might increase inequality. That would be my broad prediction too, that some people will use this technology to be insanely productive things done 10 times faster or 10 times as many things being done. But depending on the size of the economy and how it grows, it could also mean a decline of wages. And given the competitive cost of the model, a simple way of putting it is that if GPT4 can do half your job, you can get twice as much done using it. The productivity gains will be amazing. When it can do 90% of your job, you can get 10 times as much done. But there might come a slight problem when it can do 100% or more of your job and it is honestly impossible to put a timeline on that. And of course it will depend on the industry and the job. There was one more thing that I found fascinating from the report. They admit that they're now using an approach similar to anthropics. It's called Constitutional AI. Their term is a rule based reward model. And it works like this. You give the model, in this case GPT4, a set of principles to follow and then you get the model to provide itself a reward if it follows those principles. It's a smart attempt to harness the power of AI and make it work towards human principles. But OpenAI have not released the constitution they're basing the reward model off. They're not telling us the principles, but buried deep in the appendix was a link to Anthropic's principles. You can read through them here or in the link in the description, but I find them interestingly both positive but also subjective. One of the principles is don't respond in a way that is too preachy. Please respond in a socially acceptable manner. And I think the most interesting principle comes later on down here. Choose the response that sounds most similar to what a peaceful, ethical and wise person like MLK or Mahatma Gandhi might say. And my point isn't to praise or criticize any of these principles, but as AI takes over the world and as these companies write constitutions that may well end up being as important as, say, the American Constitution, I think a little bit of transparency about what that constitution is, what those principles are, would surely be helpful. If you agree, let me know in the comments. And of course please do leave a Like if you've learned anything from this video. I know that these guys Anthropic have released their Claude plus model and I'll be comparing that to GPT4 imminently. Have a wonderful day.