Microsoft's Phi-1.5 - Shocking Power with Just 1.3B Parameters (Crushing Llama 2 7B)

So Microsoft has just released a new language model called Phi 1.5, which is a Transformer model with only 1.3 billion parameters. Even though it's smaller, it performs better than some of the larger models in the industry, such as Meta's Llama 27B in specific tests. It also has some unique features that make it ideal for tasks like question answering, chat, and code generation. In this video, I'm going to explain what Phi 1.5 is, how it works, what it can do, and why it's so awesome. So first, what's a transformer model? It's a neural network that understands context in data like words or pixels by using attention. This lets it focus on important parts and process long sequences better than older models like RNNs or CNNs. Google introduced transformers in 2017 with their paper Attention Is all youl Need Now. They're widely used in language tasks like translation, text summarization, and sentiment analysis. They're also used in areas like computer vision and speech recognition. Transformers are often used in large language modeling. LLMs learn from huge data sets and can produce text from prompts or answer questions. And while powerful, they require a lot of computing power and can sometimes provide incorrect or harmful results, lack common sense, or have difficulty with complex subjects. This is where Phi 1.5 comes in. Microsoft Research introduced Phi 1.5 as a new LLM to tackle some challenges and explore the potential of Transformers. It's built on the foundation of the GPT2 architecture, but has unique features and enhancements. For starters, Fi 1.5 doesn't just rely on the usual web data for training like many LLMs. Instead, it gets its knowledge from multiple sources. This includes pieces of Python codes from Questions and answers on Stack, Overflow, coding, competition samples from codec contests, synthetic Python textbooks, and exercises made by GPT 3.5 Turbo 0301, which is a smaller version of GPT 3. All these different data sources give PHI 1.5 a solid understanding of programming languages and logic, making it handy for tasks like code generation and fixing code errors. Moreover, Phi 1.5 has another interesting data source, various NLP synthetic texts. These are pieces of text made by other LLMs or computer algorithms that follow specific rules. Some examples are texts where words are swapped with their synonyms or opposites, texts where words are intentionally left out, and texts where the arrangement of words or sentences is reversed. The purpose of these synthetic texts is to teach PHI 1.5 more about the intricacies of language, aiding in tasks related to understanding and creating text. Lastly, to make its training better and faster. PHI 1.5 uses some cutting edge software tools. One notable tool is DeepSpeed, which is a freely available tool to boost deep learning on pytorch. Thanks to DeepSpeed, Phi 1.5 can learn quicker using multiple GPUs while using less memory and reducing data exchange time. Another crucial tool in the mix is flash attention. This algorithm arranges the attention computation differently and uses established methods like tiling and recomputation. The result? Attention processes become faster and memory use goes from being based on the square of the sequence length to just the length itself. Using these data sources and tools tools PHI 1.5 has shown impressive outcomes on different tests that measure its common sense understanding of language and logical thinking. Here's what happened on the AGI Eval benchmark, which assesses foundation models using exams like SAT, LSAT and math competitions. Phi 1.5 does better than the average person and also outperforms Meta's Llama 2.7B. This latter model is a set of text models that have been trained and adjusted, and they have between 7 billion to 70 billion parameters. Specifically, PHI 1.5 scores a 95% on the SAT math test and gets a 92.5% on the English section of the Chinese College entrance exam. On the GPT4ALL's benchmark suite with LMEVAL harness, which tests language models on over 200 tasks, Phi 1.5 performs on par with Llama 20.7B. It does better than other models like GPT3, Chat, GPT and Text DAVINCI003 in most of these tasks. These outcomes tell us that PHI 1.5 is a top performer among models that have fewer than 10 billion parameters. It also indicates that the model is not just skilled at creating texts, but it's great at understanding and making sense of them too. PHI 1.5 is a base model, and while it's powerful, it has its limitations. One issue is that it might give you more information than you ask for. For example, if you ask who the US President is, it could tell you it's Joe Biden and then add extra details about his election and age, which you might not want. Also, if you're asking it something really complicated, especially in fields like math or engineering, it might not give you a great answer. This is because some topics need more than just understanding language they require specialized skills and knowledge. Safety is another concern. With Phi 1.5 and similar models, AI can sometimes produce content that's biased or offensive. This happens because of the data they were trained on, even though Microsoft tried to make PHI 1.5 safer by being careful with its training data. It's not perfect. We need to remember that it might have biases and always use it responsibly. Microsoft has also released this model as an open source model under a research license, which means that anyone can download and use it for research purposes only. This is a great contribution to the research community as it provides a non restricted small model to explore vital safety challenges and improve our understanding of large language models. Alright, I hope you enjoyed this video and if you did, please give it a thumbs up and share it with your friends. And if you have any questions or comments, feel free to leave them down below. Thank you so much for watching and I'll see you in the next one.