Orca: The Model Few Saw Coming

Do you remember this paper? Less than two weeks old, it made waves by concluding that open source models can mimic the style but not the factuality of ChatGPT. Overall, we can conclude they say that model imitation is a false promise. Well, 48 hours ago we have this a 51 page report on Orca based on a small 13 billion parameter model. I don't often comment on open source models because they're simply not competitive with open source AIs models. But Orca is not just competitive with GPT 3.5, it beats it in quite a few well established benchmarks and even matches GPT4 in a couple of tests of reasoning. As always, I've read both papers in full and can also bring in just released comments from Sam Altman and Ilya Sutskever on competition from open source models. But let's start with orca, named presumably because orcas or killer whales are frequent visitors to South American coastlines and South America is of course the land of llamas and vicunas. But all the research was done by Microsoft, which I find interesting and I'll come back to that at the end. But why did they make Orca and why does it perform better than models like Llama, Alpaca and Vicuna? Well, they say here in the abstract that those other models lack rigorous evaluation resulting in overestimating the small models capability as they tend to learn to imitate the style but not the reasoning of LFMs large foundation models. To address these challenges we develop Orca, a 13 billion parameter model that learns to imitate the reasoning process of the larger models. ORCA learns by looking at GPT4's step by step thought processes and is guided by teacher assistants from ChatGPT, which is GPT 3.5. And to give you a taste of what's to come, ORCA surpasses conventional state of the art models such as Vicuna by more than 100% in complex 0 shot reasoning benchmarks like the Big Bench Hard, which I'll talk about, and by 40 on AGI eval it goes on. Orca reaches parity with ChatGPT on the big Bench Hard and shows competitive performance in professional and academic examinations like the sat, lsat, GRE and gmat. And I know many of you will be interested in this footnote. We are working with our legal team to publicly release a diff of the model weights in accordance with lama's release policy. So if this is anything like llama, it's going to be leaked across the Internet imminently. I'm going to show you so many tests and benchmarks in a moment, but just to give you a sample, here is Orca outperforming ChatGPT in the Vicuna evaluation set and matching text DaVinci3 in the Sat, LSAT, GRE and GMAT. And as I'll touch on later, this was zero shot without chain of thought or any advanced methods. You can watch pretty much any of my other videos to see how advanced prompt engineering would probably boost those results still further. Further, for those who didn't know, 13 billion parameters is about 7% the size of GPT3, which is 175 billion parameters and possibly around 1 or 2% of GPT4's size. That gives you an indication of the difference in size between ORCA and these models that it's competing with. And if that doesn't make any sense, a smaller size means it can be run on much smaller devices like a desktop or even possibly a laptop. The authors start off by giving a little slap to the other paper. You know that one that said model imitation is a false promise. And they continue that contrary to this assertion, it is possible to reduce the gap with proprietary LLMs on multiple zero shot benchmarks that require sophisticated reasoning. As we'll see, models like Vicuna claim to have 90% of ChatGPT's quality, but when it came to reasoning tasks or more technical tasks, it basically flopped. Here's a chart I'll come back to outlining some of the more technical challenges you can give a language model. We should remember that Vicuna is a fine tuned version of the Llama model and it's competitive or even better than Palm 2. But give it some of the harder challenges for a language model and it really struggles as you can see in this column. Take Logical Deduction where it only scored 1.2%. Well, this Orca model was 2,900% better than that, scoring 36% competitive with ChatGPT. I'm going to come back to the big bench benchmark, but look for a second at Causal judgment where Orca, a 13 billion parameter model matches GPT4, which is about 100 times the size. But back to how they actually did it. Models like Alpaca and Vicuna were given lots of query and responses from ChatGPT or GPT4. But what they did is they leveraged system instructions asking models like GPT4 and ChatGPT to think step by step. This gave ORCA access to detailed responses from the model that explained the reasoning process of the teacher. As it generates a response, it allowed the these parent models of GPT 3.5 and GPT 4 to be much better tutors for this young orca. Also, they let the teachers of ChatGPT, which is 3.5 and GPT 4, give far more examples to their student. 5 million and 1 million examples respectively. That compares to the other models you may have heard of, like alpaca, uizard, vicuna, etc. Which had tens of thousands or the low hundreds of thousands of examples. But again, the key difference is the explanations, the step by step thinking that the smaller ORCA could then imitate. They give a quick demo here of how the other open source models learn from their GPT parents with a simplistic question and answer format. In contrast, the authors leverage system Messages to get ChatGPT and GPT4 to think step by step, leading to much richer explanations. As you can see in this diagram, it wasn't just let's think step by step by the way, also things like explain like I'm five. They also wanted the task to be as complex and diverse as possible, so they used the flannel collection. This was released by Google in February and focused on balancing the kind of prompts and tasks that you fine tune the language models on. You can see here the 16 system messages that they give to ChatGPT and GPT4 and you can see here the kind of difference that that makes. Imagine a language model trying to learn from this human. The human is asked pick which sentence is not logical. Sentence A, people in the desert often look forward to flood or Sentence B, people in the desert often look forward to rain. The human responds there is no reason to look forward to a flood because floods cause damage. The answer is sentence A. Now yes, a language model can learn from that, but by leveraging those system assistant messages, look at the kind of response that GPT4 gives now. Orca can learn a lot more from that explanation. And that's one of the main reasons it's better than all the other open source models. Because remember, Vicuna is the best of the open source models in this leaderboard. It has an ELO of 1054, better even than Palm 2 biceps. All the models higher than it are proprietary. But there is another reason why ORCA performs so much better. You might have wondered why didn't they just use only GPT4? Well, yes there were cost and time considerations, but there was another factor that they found. They were able to use ChatGPT or GPT 3.5 as an intermediate teacher. That teacher ChatGPT was able to reduce the gap in capabilities. So ORCA got smarter and better able to learn a bit like progressive learning where you first learn from easier examples then followed by harder ones. After that they gave IT outputs from GPT4. Notice by the way what happens if you skip the ChatGPT teaching assistant and only train on those 1 million examples from GPT4. What happens is a bit like a student struggling in a class that's too advanced for them. ORCA actually performs worse in those circumstances, averaging 37%, but with that intermediate teacher beforehand it gets 41.7%. Speaking of time, it only took about 200 hours to train Orca on 20 A1, 100 GPUs. They did take a few weeks to collect the data from ChatGPT and GPT4, but presumably if they're planning to open source this, which they say they are, then that step could be skipped by the wider community. Let's now look at some more of the results. First, for open ended generation, not multiple choice, Orca is 95% of ChatGPT quality and 85% of GPT4's quality as assessed by GPT4. But they wanted to quickly move on to some more definitive tasks because there is a problem of using GPT4 as an assessor. For example, they observed that there is a positive bias in GPT4 evaluation toward the response of the first model in the comparison set. This reminded me of the unfaithful reasoning paper that I talked about in one of my recent videos. You can't always trust GPT4 to give its true reasoning, but here it is in more objective multiple choice questions. And notice how much harder many of these tests are for even these advanced language models. I am fortunate and proud to have attained a perfect score in some of the tests in this chart. Like the GRE and gmat, they were part of the Aquarat test that they gave the models. So I can say that they really are quite challenging, hence why GPT4 only gets a 40%. But you can see that throughout ORCA outperforms Vicuna by quite a margin and is very competitive with text DaVinci3. Of course overall it does lag behind GPT4, but this is all zero shot. A bit later on I'll come back to the range of methods that we could use to further improve improve on Orca. The percentages by the way are the improvements on Vicuna. Again the second best open source model so far. We've looked at human centric benchmarks like the GMAT and gre. These are grouped with the Lovely name AGI Eval. And as we've seen, even the top models lag behind the top human performers. But what about a benchmark specifically for language models? It's called Big Bench Hard. The original big bench had 207 tasks, but language models got so good that they had to narrow down the benchmark to just the 23 challenging tasks where human raters still did better than language models. Now it turns out when you add chain of thought prompting to the models, they do even better, and there are even fewer tasks that humans are better at. But anyway, all you have to remember is that these are 23 of the hardest tasks for language models, and I'll just let you compare the results for yourself. But the trend is really quite clear. Walker massively outperforming the previous best open source model, Vicuna, beating even ChatGPT on average, but still of course lagging behind GPT4 except for a few tasks. Look at Web of Lies where Orca outperforms GPT4. That would be a question like this Alexis says Shaunda tells the truth. Jim lies. Antwan says Jim tells the truth. Shaunda says Antwan lies. Does Alexis tell the truth? Or what about temporal sequences where ORCA absolutely crushes Vicuna and doubles ChatGPT's performance? That would be a situation like this. Now I'm not going to read it all out, but essentially you have to figure out when the timings match up, basically keeping track of time and ORCA does really well and ChatGPT flops getting it wrong. Interestingly, they also tested all four models on that common sense reasoning question that I demonstrated for SmartGPT about hanging the clothes to dry. As you might remember, you can use prompt engineering to nudge the models to almost always get it right. Which is partly why I view these results more as a baseline rather than a cap. And the authors admit this too. ORCA has been trained on data that simulate zero shot setting with standard prompts the model's performance in other contexts such as multi turn conversations like the DIRA paper I talked about on the channel in context learning and few shot learning or advanced prompting techniques that Smart GPT or Tree of Thoughts for example. And they say light chain of thought prompting remains untested. These results are a baseline, not a cap. They mention other ways that ORCA could be improved, for example through tool augmentation. And that's not just calculators, calendars, Bing or AutoGPT. I was going to do a separate video on this paper, but I'll just mention it here. This paper from last week demonstrated that larger models can create tools that smaller models can then use more efficiently. Once the best language model, say GPT4 has created a generic python function which is the tool, and then written some unit tests, it can then wrap and hand over those tools to smaller models like GPT 3.5 or in this case Orca and check out the tool making row to see the improvement for ChatGPT or in our case Orca. When they're given these tools created by GPT4 or better language models, their performance across a range of tasks goes dramatically up. And we haven't even talked about using a process based reward model like in the let's verify step by step paper. That of course could further improve or performance. Of course when this model becomes publicly available I will test all of this out, but it hasn't been open sourced yet and they do say this model is solely designed for research settings. That does seem a little bit naive to me. I mean that's what Meta said when they released Llama, but then everyone and their grandma just use the language model for whatever. I do wonder what it means when they say we are working with our legal team and it is particularly interesting to me this was all done by Microsoft. I'm going to go into a little bit of speculation here about why I think they conducted this research. You might remember that leaked memo from Google. We have no moat and they even mentioned Vicuna and talked about how it circumvented restrictions on the OpenAI API by using ShareGPT. And my theory is that the Microsoft researchers were testing this point from the memo. The point was that training giant models from scratch not only throws away the pre training, but also any iterative open source improvements that have been made on top. It doesn't take long for those improvements to dominate, making the full retrain extremely costly. Maybe Microsoft is hesitating about Future investments in GPT5 or GPT6 and they really want to test out if it's easy to imitate those large models on the cheap. If it is, then why would Microsoft invest billions in a new giant model? That's my own theory as to why Microsoft is working on this, but let me know in the comments what your theory is. In the conclusion. The authors state that Orca suggests that learning from step by step explanations could significantly improve the quality of models regardless of their size, and that they hope these insights will inform the design of more robust evaluation methods compared to those used for Vicuna, for example, and the advancement of alignment and post training techniques and the more effective use of powerful models like GPT4 as teachers. And maybe they should have said and also with ChatGPT as an intermediate teacher. I'm going to end with the thought of the leaders of OpenAI, Ilya Sutskeva and Sam Altman on open source models, and I think there is a bit of a contrast between the two. Answers Ilya Sutskeva thinks that the gap is growing ever wider to the open.

Source versus non open source models question. You don't want to think about it in binary black and white terms where like there is a secret source that will never be rediscovered. What I will say or whether GPT4 will ever be reproduced by open source models. Perhaps one day it will be, but when it will be, there will be a much more powerful model in the companies. So there will always be a gap between the open source models and the private models. And this gap may even be increasing with time. The amount of effort and engineering and research that it takes to produce one such neural net keeps increasing. And so even if there are open source models, they will never be they will be less and less produced by small groups of dedicated researchers and engineers and it will only be the providence of a company, a big company.

While Sam Altman seems to say that even if open source models do catch up, OpenAI will always have a different kind of moat.

What are your thoughts about the we have no Motes document that was released.

Released lately.

A leaked document? The thing that is special about OpenAI, and I think the thing that is so misunderstood by that document, aside from the fact that we have a gigantic number of users and people that have formed some sort of relationship with us and our products is what OpenAI is special about is figuring out what comes next. It is easy to copy something once you know it can be done. And in that sense, sure, it is very hard to go figure out what to do next. And the ideas, the big ideas, the medium sized ideas, the small ideas and the careful execution on them that it takes to get from here to superintelligence. That's what our mode is anyway.

This video could have been at least three times longer. There was so much I had to edit out for brevity. If you're interested in me talking more about open source models, do let me know in the comments. I've got much more to say. Always. Thank you so much for watching to the end and have a wonderful day.