Is Vicuna the ChatGPT Killer?

Since the release of Alpaca Meta's large language model and the leak of its weight, there has been an explosion of open source models. First we had Alpaca from Stanford Group, followed by Dall E from Databricks and then a bunch of models from Cerebris and most recently GPT for all from Nomic AI. But there's a new model which claims to be as good as ChatGPT. It's called re Cunya. It's an open source chatbot that claims to be 90% of ChatGPT's quality. They have used a very interesting training data set and evaluation strategy. More on that in a minute. In this video we are going to do a one to one comparison between ChatGPT and Vicunia on these tasks. Just a spoiler alert, it's very impressive. So watch till the end for detailed comparison before looking at the model and how to use it and potentially how you can retrain it by accessing its weights. Let's look at how it was trained. So vicuna has a 13 billion parameters and it's trained by fine tuning LLAMA on user shared conversations collected from shared GPT. Now if you don't know what sharegpt is, it's a website where people can share their chatgpt conversations. So if you go to sharegpt.com they have an introductory video. But I would encourage you to actually look at these examples that people have shared. So for example, here is an example conversation that this person had with ChatGPT. You can learn a lot about prompt engineering just by studying these examples. Share GPT has a chrome extension and you can install this and share your own conversation with the community as well. So this team of researcher collected data from shared GPT and fine tune the LLAMA model on that data set. And I think that is why it is performing as good as ChatGPT because it's specifically trained on a data collecting from ChatGPT. Right before looking at the evaluation, let's look at the training data set. So in this case they have collected 70,000 conversations from shared GPT and it was trained on eight A100 GPUs for one day. So to compare Vicunia with different models, what they have done is they selected 80 diverse questions and then used GPT4 to judge the output of these different models including Llama, Alpaca, chatgpt and Rocunia. We will look at the evaluation in a minute, but let's first look at the comparison between different models. So in terms of data set, Llama was trained on 1 trillion tokens, Alpaca is trained on 52,000 samples and Vicuna is trained on 70,000 samples. In terms of training code, Alpaca and Vicuna code is available. Llama's code is not available. Now for evaluation, Llama used academic benchmarks, Alpaca is using author's evaluation and Laconia is using GPT for assessment. In terms of cost for the 7B 7 billion parameter model it was the training cost was around $140 and for 13B parameter models is only 300 model $300. Now keep in mind, in this case they didn't really had the training data cost because they were simply using shared GPT API to get their training data. So Alpaca has a maximum context length of 512, whereas in this case they increase it to 2048, which also substantially increases their GPU memory requirement. That's why you need a lot beefier GPU to actually use this. But they also did some memory optimizations on top of it. It is very interesting to look at their evaluation strategy. So they came up with eight different strategies including Fermi problems, role play scenarios, writing, coding and math tasks. Then they used GPT4 to actually generate different questions. And then they selected 10 questions per category and collected answers from five chatbots. And then they used GPT4 for evaluation again. So it rated the answer based on helpfulness, relevance, accuracy and detail. This is actually a very interesting use case for GPT4. So basically it can assess the quality of answers as well. So all the students, not only you can generate your essays with GPT4 or ChatGPT, but in future it's going to also score your essays and assignments. One thing they found was it's not really good at judging math or coding tasks, which is kind of strange because now they have a code interpreter as well in GPT4. So based on the results in terms of predatory response quality assessed by GPT4, vicuna can be as good as 92% of ChatGPT responses. So it is very close to ChatGPT. But let's look at the actual responses to see whether it's actually close to ChatGPT or not. Now they have provided a demo where you can actually access Vicunia 13B and play around with it. So here I simply ask it, can you create a three day travel plan for la? And it gave me a pretty detailed answer. So before further playing with this, let's look at the evaluation responses that were judged by GPT4. So this is a very Interesting website. They have provided some questions in these the 10 different categories that they mentioned and then response from Vicuna along with the other models. And the great thing is they have also provided evaluation from GPT4. So let's look at a few examples before we actually experiment with the Vicunia model itself. First let's look at the writing category and then there are 10 different questions. So for example, for this question, compose an engaging travel blog post about a recent trip to Hawaii highlighting cultural experiences and must see attraction. So GPT 4 give 9 by 10 and 9 by 10 both to Vicuna and ChatGPT. So here is the response from Chat GPT. But I think we are interested in Vicunia. So I'm going to read I think a paragraph of it and let's see how good the flow is. So aloha fellow travelers. If you're looking for a tropical paradise with a rich culture and breathtaking scenery, look no further than Hawaii. My recent trip to Aloha State was an unforgettable adventure filled with exciting cultural experiences and must see attractions. Of course, NOAA trip to Hawaii could be complete without spending some time at the beach with this crystal clear waters, golden sand and gentle waves. That actually looks pretty nice. I would say if you Compare it with ChatGPT responses, I think it's, it's. It's up there. Right. I'm going to put a link to this page web page so you can look at these different examples. Let's look at something else. Let's say write a script for a YouTube video exploring the history and cultural significance of jazz. Right. So hello and welcome to our YouTube video on the history. Right. Jazz is. Yeah, I think it doesn't look bad at all. And in this case GPT4 thinks it's 8 by 10. And here's the response. So Assistant 1, which is ChatGPT provided a well structured script for YouTube video including visual cues and narration which made it more engaging and informative. Right. Assistant 2, Vicuna also provided a good response discussing the history and cultural significance of jazz and influence on the other. Okay. It's role in civil rights movements. However, Assistant 2's response was less engaging and lack the virtual cues and structure for a video script. Okay, so there are. They are actually comparing three different things, ChatGPT versus VIR and then you can even see the ability of GPT4 to assess these. So which is great, let's say in terms of role playing, GPT4 thinks that Vicunia is as good as ChatGPT, which is pretty awesome. Let's see coding examples. Now one thing that we notice, GPT4 is not good at evaluating math and coding. So but so far from what I have seen, I can't say that the responses are pretty nice. How do social media platforms influence the way people consume? Share? Okay, so in this case GPT4 thinks that actually give a better response. So again I'm going to put a link to this. You can actually go and experiment with this. I think it's a pretty nice tool to simply compare the responses and see how good this model is. But let's go and play with it ourselves. Okay, so I'm going to put a link to it how you can use it. But here is a question that I asked Write an email explaining why GPT4 should be open source. Now keep in mind it doesn't have access to Internet, so it doesn't know what G GPT4 is. But here's the response. As you may know, GPT4 is a groundbreaking language model that has the potential to revolutionize the way we communicate and interact with technology. At its core, GPT4 is an open source project and we believe that its continued development improvement should remain in the hands of the community. Right. So it doesn't know what GPT4 is, but it's writing a letter about it. I'll also show you a response from ChatGPT so that we can compare it. Right then there are several reasons. So one is collaboration and innovation. Open source project Foster collaboration and innovation among a community of developers, researchers and enthusiasts by allowing others to continue contribute to the project then improved quality. An open source project can draw on the collective expertise of a diverse community which can lead to more accurate and reliable models. This is particularly important for a complex and powerful tool like GPT4. Then transparency. With an open source project, the source code and development processes process are transparent which allows for greater accountability and trust in technology. It's making actually some really good points. Flexibility. So open source project can be adopted and modified to meet specific needs and use cases which can lead to more flexible and customizable solution. I think if it had access to GPT4 Internet and knew its capabilities, GPT4's capability, it could come up with some other arguments as well. Now same question and response from ChatGPT. So as you may know, the development of AI and machine learning model is a rapidly advancing field that has the potential to revolutionize industries and societies as well. So you see like that text is pretty similar, like it's using similar words and it's because that the training data is derived from chat GPT. So one such model, so GPT4 is a set to be released in near future. Okay. So it kind of have a sense of that. Okay, what is GPT into what GPT4 is? So open sourcing GPT4 would mean that the open source of data used to train the model would be freely available. Right. So it's a more descriptive, I would say, but let's look at the reasons. Right. So advancements of research. In this case it's talking about advancement of research, but Vicuna was talking about collaboration and innovation. So innovation is basically advancement of research. Right. Then increase trust. So we had improved quality, transparency. So it's basically looking at the same points and then flexibility, which I think is accessibility. Right. The date could be relevant. So from the responses it does seem to be similar to ChatGPT, if not right there yet. Okay, next I asked both of them to create a list of things I need to do to plan for my wife's birthday party. Okay. So it came up with a list and they look pretty similar. ChatGPT Saturday. Well, it's a birthday party, so. Okay, decide on a theme, Guest list, venue, invitation, food and drinks, decoration, entertainment, gifts, timeline, setup and cleanup. Right. And here is the list. Setup, budget, very smart. Choose a theme, make a guest list, choose a venue. Right. So decide on a location for the party, whether it's at home or a rented hall or whatever. Right. Plan the menu, plan the decorations. So it's pretty similar. List, send out invitations, purchase supplies, make a timeline, plan for transportation, hire entertainment, order flowers. Right. Okay, good point. Right. Make a shopping list and plan for emergencies. So and if you look at the more detailed description, it actually does a pretty nice job. And I think, like I would say in some cases you won't really see the difference between the two. Although in the strand it. It was a little more descriptive and it simply said, all right, here's the list of things you can do. Right. So but I would say like both of them are pretty good. Okay. One thing to note from time to time you might encounter this network error is regenerate or refresh this page. Right. So I have noticed that it happens. A lot of people are trying to access it, so they are simply prioritized. Some folks, which kind of makes sense now I asked it to actually write a Python program and the great thing is it actually formatted it. Right. Let's look at it whether it's actually correct or not. Okay. And wow, this is great because it also explains the Code like chat GPT. So this function takes in integer num as an input and check if it's prime or not. Right. And then it explained the logic as well. So this is pretty awesome, I would say. Okay, it's still writing, so it's giving examples as well. I'm actually impressed. This. This definitely could be as good as ChatGPT. Now, since it was able to actually solve that coding problem, I'm going to ask it to actually write a code for a website that has three buttons and whenever each button should display a random joke when pressed. So let's see whether it can actually come up with that or not. It came up with the code just by looking at it. Seems to be working. I'm going to run it though. One thing I noticed is that it actually stopped producing the code. So it seems like there is a limit to it, like how much textures can generate. But let's run this in a web browser and see if we can create a working website. Okay, so when I ran the code, it did make a website. However, if I click on this joke one or any of the joke, so it displays this message, this, sorry, the joke doesn't exist. So let's ask it if it actually can fix the issue or not. Okay, so it gave me the code again. So let's run it again. So here's the website and actually it works, which is crazy. This is the type of thing that ChatGPT can do. Like, I had to do a little bit of tweaking, but it actually works, which is impressive. Okay, let's get a little philosophical with this and we will ask it, what is the meaning of life? So one thing you will notice is that I'm in a queue, there are some people ahead of me, so it could be a little slow in response, but still it's free. Okay, so here's the response. The meaning of life is a philosophical and existential question that has been debated and explored by various thinkers, scientists and religious leaders throughout history. It's a complex and subjective topic, and different people may have different answers. Okay. In general, the meaning of life can be thought of as the overarching purpose or significance that gives our lives direction and fulfillment. Some people believe that the meaning of life is to seek happiness, love and fulfillment, while others believe that it is to serve higher power or achieve spiritual enlightenment. There is no correct answer to the question of the meaning of life, and maybe something that each individual needs to discover for themselves based on their beliefs and experiences. Cool. Based on my experience, this is the only open Source Chatbot, which come close to ChatGPT. Actually there are some other parameters as well. So for example, right now we were limiting the output tokens to 512, but you can take it to double the size. So I think the one example where I saw that it stopped responding, we could do that. And then you can change the creativity and everything using the temperature as well. So overall it's a very impressive tool. It's definitely much better than Laminal Packer and even the GPT for all that I posted in another video. So I'll post this link again. You can explore it now. There's definitely one question that you might be thinking, can you access it locally? Right, before that I want to actually show you one more thing. There is another model called Koala which I don't know if it's available or not. We can actually test it now. Let's return it. So it's another model which is going to be released pretty soon. It's a very similar. Supposed to be very similar in responses, but I'm going to keep you posted for that. Okay. It doesn't seem to be assemblies that signal is. The answer was not really that long, although it's. It makes sense. Okay, now you might be thinking, can you use this on your local machine? So to answer this, let's look at the release plan. So initially they released the training serving an evaluation code on GitHub repo. Right. We plan to release the model weights by providing a version of delta weights that build on the original Llama weights. But we are still figuring this out. Right. So they haven't released the weights. They might release it, but I think I'll look at the Discord server to see when that happens. I'll definitely make a video on it now in order to train it. The training is going to be very tricky because it needs around eight A100 GPUs to train it. So it's amazing to see how far we have come just in a span of two or three weeks since Alpaca was released and it's simply astonishing to look at the progress. Thanks for watching. Hope you liked the video. If you did, please consider liking the video and subscribing to the channel. Thanks for watching. See you in the next one.