Google's Gemini Pro jumped to second place on Chatbot Arena with an Elo rating around 1215, beating GPT-4 in some matchups. The Bard version has internet access and handles recent factual questions better, though it still hallucinates occasionally. The Arena ranks models through human votes on head-to-head comparisons.
Questions this video answers
January 28, 2024.
1215.
December 18, 2023.
January 14, 2024.
(1) Gemini Pro — Vertex AI API on Google Cloud; (2) Gemini Pro Dev — developer API on Google AI Studio; (3) Bard January 24 Gemini Pro (latest Bard edition).
Chapters
There is a major upgrade to Google Bard. It has surpassed GPT4 to the second spot on the arena leaderboard, the large model system. Chat Arena Leaderboard is a crowdsource open platform for LLM evaluation. Here they have collected over 200,000 human preference votes to rank LLMs with the Elo ranking system and right now Bard with Gemini Pro is holding the second position. The LMSYS organization who maintains this leaderboard tweeted about this and even Google retweeted the tweet saying we're excited to see how everybody is using Bard. Now this was extremely surprising to everybody because on the same leaderboard Gemini Pro previously held the eighth position, but with the latest update it is sitting on the second position which is a huge improvement. Before talking about how this happened, let's have a quick look at the Chat arena leaderboard and see how it actually works. Unlike the open LLM leaderboard from Hugging Face, which keeps track of the performance of different models on existing benchmarks, the Chatbot arena board is actually crowdsourced. This is benchmarking LLMs in the wild. Now how does it work? So the user asks a question and then it randomly picks two models from more than 20 plus models. These contain both proprietary as well as open source large language models. When you provide a prompt all you see is model A and model B and you don't know which model it is. Once the responses are generated by the model, the user can pick which response they like best. For example, I can say in this case B is better and after that it will show me which model it was. So in this case it's Bard Gen 24 Gemini Pro. And on the other side we have the latest GPT4 preview. Now based on the user feedbacks it keeps track of the average ELO score for different models. So currently Bard with Gemini Pro is setting at the second position that has an Elo score of 1215. It has relatively few votes because I think it's the latest update of Bard. Now this is purely based on human preferences, so I think it will also depends on what type of questions and queries people are asking. This is not the only Gemini Pro API results available. There is the Dev API for Gemini Pro which is sitting on the ninth position and there is even a Gemini Pro which is sitting on the 12th position. So there was a lot of confusion whether this version is different than what we have on the Bard web interface. So the bar that you get on Google website that had the Last update on 18 December 2023, so nearly a month ago. But the one that we get here is based on the version in January 2024. So this seems to be the latest version. These three different versions of Gemini Pro were explained by the LMSYS group, which was great. The Gemini Pro is the Vertex AI API on Google Cloud. The Gemini Pro Dev is the developer API on Google AI Studio. You can actually get access to this for free if you keep it under 60 queries per minute. And the third one is the Bard January 24 Gemini Pro, which is the latest version, but I think it's not available to everybody yet. Now the question is, how does Gemini Pro work so well with this new update? And the answer might be partially that it has access to the Internet via the API. So for GPT4 Turbo and other GPT4 variants, they are relying solely on their training data when they're generating responses to user queries. However, this new version of Bard with Gemini Pro through their API has access to the Internet. Just to confirm this, I asked this question. Who won the Republican primary in Iowa? GPT4 says the winner of 2020 Republican primary in Iowa was Donald Trump. However, when I ask the same question for the Bard, so it says in the 2024 Republican caucuses held on January 14, Donald Trump won and then it goes on to actually list different details. So for example, he captured over 58% of the total votes. Then Ron Desantis, Nikki Hadey, those were the other two competitors. Here are the results for Iowa caucuses from the New York Times. Donald Trump seems to got 51%, descendants was 21% and Nikki Haiti was 19%. So I don't know if it's the final tally, but this is 95% of the votes in, so seems like it probably missed a little bit on the details, but it does have access to Internet, so the comparison might be a little unfair. However, there are actually other models on this Chatbot arena leaderboard, such as the models from Perplexity AI who also has access to the Internet. Irrespective, I think it's a huge leap for the Bard team. I modified my prompt this time. I asked when was the last Republican primary in Iowa and who won that? And I'm using the gemini Pro Bard 1-24- version and asked the same question from the perplexity AI 70 billion model that has Internet access. Now. This time it got the date right when I specifically asked for when this was held. So it says January 15th and it also got this right that Donald Trump won it, but it doesn't have any information regarding who came second and who came third for Perplexity AI. It also got the date right and it also got the the percentages right as well. So seems like this is doing a pretty good job at rank. So as we saw here in the first example, Bard Gemini Pro with Internet access could still hallucinate. So you need to be very careful and don't trust the results that you get. You need to verify them independently. Another great thing about the Chatbot arena is that you can actually run two models side by side. So click on this arena side by side here. I selected the Bard January 24th edition and we're going to run it against GPT4 Turbo, which is the current leader on the arena. I'm going to test this on a couple of quick prompts and probably it has seen these prompts in the training data. So the first prompt is that there are three killers inside a room. Somebody else enters the room, kills one of the original occupants. Nobody leaves or enters the room. How many killers are in the room? And we are comparing responses from GPT4 Turbo and Bard Gemini Pro. So according to GPT4, when somebody comes in, kills one of the person, there are two possible scenarios. Now, if the person who entered and shot one of the killers has never killed anyone before, then after shooting, there would be three killers in the room, the two original killers who were not shot, plus the person who committed murder by shooting one of the original killers. And the second scenario is if the person who entered the room and shot one of the killers was already a killer before that. Right. So that means he's already a murderer. Then after the shooting, there would still be three killers in the room. Now here's the response from Barred with Gemini Pro. The answer is true. While the newcomer certainly committed murder, they don't qualify as a killer in the context of the initial definition. So while the riddle might play on wordplay or unexpected interpretation, it is important to remember that the language should be used responsibly and avoid framing actions in ways that normalize or trivialize violence. Let's focus on on more positive and creative uses of language instead. I ran the same prompt again because previously I gotten better responses, and here it tells me two different situations in which either there will be two killers in the room or three. However, then it goes on to say at the end. Perhaps a more interesting question to ponder is what motivates someone to enter a room full of killers and take a life? Exploring the reason behind such action can lead to deeper insights into human behavior and the complexity of violence. Now you can sense that Bard kind of has this personality baked into it, which is not present in GPT4 Turbo. If you want to experiment with this new version of Bard, powered by the latest Gemini Pro, you will have to use the Chatbot Arena. It's not yet integrated as part of Bard or the Gemini Pro API that is available through the Google AI Studio. This is very exciting. Google seems to be catching up and we'll see what happens with the Gemini Ultra that might be on par with GPT4 Turbo. Even better than that, only time will tell. We'll have to wait and see. I hope you found this video useful. Thanks for watching and as always, see you in the next one.
