Microsofts New 'ORCA' Takes Everyone By SURPRISE! (Now ANNOUNCED!)

So Microsoft Research recently released a new research paper called orca, and this video will demonstrate exactly why. This is largely one of the biggest new recent research papers by a very large company like Microsoft. Now, many people have said that this is game changing, but a lot of people are missing the true reason as to why ORCA is so incredible. So the paper starts out by saying ORCA progressive learning from complex explanation with traces of GPT4. And this was done by Microsoft Research. So essentially the abstract explains this well, but I'm going to explain it in more detail later. It says recent research has found on enhancing the capability of smaller models through imitation learning, drawing on the outputs generated by large foundation models. So essentially the problem here is that these large language models are really good. But when building smaller language models, what these people do when they build them, they tend to overestimate the small model's capability as they learn to imitate the style but not the reasoning process of large language models. So that's why ORCA is here. And the reason this is going to be so crazy is because this completely changes how new people are going to train large language models and smaller models that are going to be on other devices. Now, if you want to truly understand just how great ORCA is, it actually surpasses models like Vicuna's 13 billion parameter large language model. And if you don't know what Vicuna is, then you're like, well how do I know it's even good? Because I've never heard of Vicuna. So Vicuna was actually a open source large language model that was trained by fine tuning the LLAMA model on the conversation data shared by users from ShareGPT. And why this is so crazy is that you have to remember this model was trained primarily to serve as an LLM and Chatbot and it was actually fine tuned by the version that Meta created. Now of course you may have not heard of Akuna, but if we talk about something else that you know, which is ChatGPT and GPT4, you're likely to understand these references much more clearly because Orca actually reaches parity with ChatGPT, which on the BBH benchmark and shows competitive performance in professional and examinations like the sat, the LSAT and in zero shot settings. And if you don't know what zero shot settings are, that's essentially where you train a large language model in a certain environment without seeing specific kinds of examples or styles and then you test it on new things, which is really cool because if it's able to handle these things without seeing them before then we know that the effectiveness of this large language model is likely to be greater than we imagined in the introduction of this paper. They also do show us the evaluations with GPT4. So essentially what that means is that GPT4 evaluated answers by these large language models. So from the left to the right we have LLAMA all the way up to ChatGPT and Orca, and of course Google's Bard. And what we can see here is the evaluations. So what the score represents is how effective these were against a range of different tests. And what's incredible is that ORCA actually manages to surpass ChatGPT. And not only that, it also manages to excel a small amount, which is 103% compared to ChatGPT's 100% and Bard's. Now what I also do find particularly interesting is that when we do come to the professional and academic exams like the sat or the LSATs, orca is quite similar to text davinci03, which is ChatGPT's earlier version, without all the fine tuning that we now love in ChatGPT. Which means that ORCA is much more effective than prior open source models. And when compared to Vicuna, this is actually a 40% increase in effectiveness with the same number of parameters. The reason this is so incredible as well is that if we take a look at the parameter count for both of these large language models, we know that there is a huge difference. You see, Orca is 13 billion parameters, whereas ChatGPT has 175 billion parameters. But here's why this is so cool. You see, if we have a smaller large language model, one that is around seven times more smaller and is just as effective, this makes it more lightweight and more integratable into certain devices, which is something that many of these large language model creators are doing. You see, if you remember Google's IE speech a couple of weeks ago, what they actually announced was truly incredible. They actually announced they were going to make some large language models available even when offline. Now the reason this is incredible is because, as you know, currently ChatGPT is powered by a bunch of servers and is probably located somewhere on a data farm. But here's where things get interesting. Google has stated that will be making Palm 2 available in four sizes, from the smallest to the largest. Gecko, Otter and bison and unicorn.

Palm 2 builds on our fundamental research in our latest infrastructure. It's highly capable at a wide range of tasks and easy to deploy. We are announcing over 25 products and features powered by Palm 2 today, Palm 2 models deliver excellent foundational capabilities across a wide range of sizes. We have affectionately named them Gecko, Otter, Bison and Unicorn. Gecko is so lightweight that it can work on mobile devices fast enough for great interactive applications on device, even when offline.

If these large language models now have a new framework in order to train and become more effective, it means that even the smallest of large language models that we now have in future devices such as phones and laptops are going to be seven to eight times more effective, which is one of the true applications of this, because as you know, large language models, one of the limitations is their sheer size because even GPT4 is 1 trillion parameters. So this is going to be something very, very interesting as I've seen countless individuals currently running LLAMA, the 13 billion parameter large language model on their laptop offline. And if people manage to get ORCA offline, it's going to present a very interesting opportunity. And the reason this could also pose some insane applications is because as you know, OpenAI is currently developing their humanoid robot in conjunction with a company called 1X Robotics. And the reason this is so incredible is because what happens when we do manage to get these small large language models integrated into these robots and let's say they no longer need their cloud based servers to process and generate responses and can do it instantly, it's going to present a very, very interesting turning point where these large language models are pretty much standard. Just imagine having ChatGPT on your phone without Internet access. Just how great would that be? Further on into the paper, they also show the complex zero shot reasoning tasks that we talked about before. You can see in the Big Bench Hard test, it scored even better than ChatGPT, scoring a 49.7% accuracy in the Big Bench Hard. Now this was where I found the data to be very interesting. From the research page, these are the system messages and there are 16 different messages that they use to train this large language. Now not only are these 16 system messages very good, but you can actually personally use these if you are trying to understand a concept. One thing that I do personally do is use some of these system messages myself when I'm trying to understand complex concepts that I don't particularly understand. For example, it says you are an AI assistant. Provide a detailed answer so the user doesn't need to search outside to understand the answer. You are an AI assistant. You will be given a task. You must generate a detailed and long answer. You are an assistant who always provides an explanation. Think like you are answering to A five and trust me when I say that using this framework to get very very difficult concepts down is quite impressive. And this is one of ChatGPT's very unique features. Now this explained like I'm 5 technique isn't something that is just used by US users. This was something that they originally did to ChatGPT and to OpenAI's DaVinci 003 early on when they were trying to make the system more better. A research paper that was actually Originally produced on 24 May 2022 which was recently revised, actually talked about how stating let's think step by step before each answer actually increases the level of Results and outperforms 0 shot large language models on diverse reasoning tasks, including arithmetic and other logical reasoning tasks. They were able to improve it from anywhere from 10 to 40% by simply adding those few words. Let's think step by step now. One thing I did find largely interesting that many people didn't mention and this isn't to say that these models are bad, but it is interesting to see the kinds of limitation that these large language models do you possess. Because so often when these large language models are released, we're often so shocked by what we do see and wonder how these capabilities are even possible. But here we have some tests that show that GPT4, GPT3, ChatGPT and of course this new model ORCA just can't understand. And the question is simply asking I tried five clothes to dry out in the sun. It took them five hours to dry completely. How long would it take to dry 30 clothes? So this is a common sense question which possesses a really simple answer. But a lot of these large language models, and I find it interesting because I've seen this myself, sometimes they tend to overcomplicate simple things. You can see Orca, Vicuna and ChatGPT all doing insane math to figure out how it would actually work. And this was actually taken from this TED talk where they said AI is incredibly smart and also shockingly stupid.

Okay, one more Would I get a flat tire by bicycling over a bridge that is suspended over nails, screws and broken glass? Yes, highly likely. GPT4 says, presumably because it cannot correctly reason that if a bridge is suspended over the broken nails and broken glass, then the surface of the bridge doesn't touch these sharp objects directly. Okay, so how would you feel about an AI lawyer that a's the bar exam, yet randomly fails at such basic common sense? AI today is unbelievably intelligent and then.

Shockingly stupid for a better example right here we can see this question. Now this question is shockingly basic. It says I have a 12 liter jug and a 6 liter jug and I want to measure 6 liters. How do I do it? The simple answer obviously is to Simply take the 6L jug to measure the 6L because you already have a 6L jug. But Orca decides to completely fluff this and give out a strange answer. The same with ChatGPT and incredibly the same with GPT4. Now I definitely think you should watch this TED talk because it does dive into what we don't really understand about large language models when we realize that they over complicate very, very simple concepts and common sense concepts. So it's interesting to note that this is one shocking limitation of not just orca, but many different large language models. And I still find it incredible that we haven't managed to solve this problem yet. So with that being said, I do think that in the future, as these large language models continue to get more efficient, more nimble and more effective at different sorts of reasoning tests and different sorts of questions, that eventually we will have some state of the art offline model that is used widely. The only thing is that what will its primary purpose be? I think that will be very interesting to see.