Enter PaLM 2 (New Bard): Full Breakdown - 92 Pages Read and Gemini Before GPT 5? Google I/O

Less than 24 hours ago, Google released the Palm 2 Technical Report. I have read all 92 pages, watched the Palm 2 presentation, read the release notes, and have already tested the model in a dozen ways. But before getting into it, all my four main takeaways are these. First, Palm 2 is competitive with GPT4 and while it is probably less smart overall, it's better in certain ways. And that surprised me. Second, Google is saying very little about the data it used to train the model, or about parameters parameters or about compute, although we can make educated guesses on each. Third, Gemini was announced to be in training and will likely rival GPT5 while arriving earlier than GPT5. As you probably know, Sam Altman said that GPT5 isn't in training and won't be for a long time. Fourth, while dedicating 20 pages to bias, toxicity and misgendering, there wasn't a single page on AI impacts. More broadly, Google boasted of giving Gemini planning abilities in a move that surprised as I am to say, it makes OpenAI look like paragons of responsibility. So a lot to get to. But let's look at the first reason that Palm 2 is different from GPT 4. On page 3 they say we designed a more multilingual and diverse pre training mixture extending across hundreds of languages and domains like programming, mathematics, etc. So because the text that they train Palm2 on is different to the text that OpenAI trained GPT4 on, it means that those models have different abilities and I would say Palm 2 is better at translation and linguistics and in certain other areas which I'll get to shortly. If that's data, what about parameter count? Well, Google never actually say. They only use words like it's significantly smaller than the largest Palm model which was 540 billion parameters. Sometimes they say significantly, other times dramatically. Despite this, it significantly outperforms Palm on a variety of tasks. To all the references you may have seen to imminent 100 trillion parameter models were bogus. Skipping ahead to page 91 out of 92 in the model summary, they say further details of model size and architecture are withheld from external publication. But earlier on they did seem to want to give hints about the parameter count inside Palm 2, which OpenAI never did. Here they present the optimal number of parameters given a certain amount of compute flops. Scaling this up to the estimated number of flops used to train Palm2, that would give an optimal parameter count of between 100 and 200 billion. That is a comparable parameter count to GPT3 while getting competitive performance with GPT4. Bard is apparently now powered by Palm2 and the inference speed is about 10 times faster than GPT4 for the exact same prompt. And I know there are other factors that influence inference speed, but that would broadly fit with an order of magnitude fewer parameters. This has other implications, of course, and they say that Palm 2 is dramatically smaller, cheaper and faster to serve. Not Only that, Palm Palm 2 itself comes in different sizes. As Sundar Pichai said, Palm 2 models.

Deliver excellent foundational capabilities across a wide range of sizes. We've affectionately named them Gecko, Otter, Bison and Unicorn. Gecko is so lightweight that it can work on mobile devices fast enough for great interactive applications on device, even when offline.

I would expect Gecko to soon be inside the Google Pixel phones Going back to data, Google cryptically said that their pre training corpus is composed of a diverse set of sources, documents, books, code, mathematics and conversational data. I've done a whole video on the data issues that these companies face, but suffice to say they're not saying anything about where the data comes from next. They don't go into detail, but they do say that Palm 2 was trained to increase the context length of a model significantly beyond that of Palm, as of today can input around 10,000 characters into Bard. But they end this paragraph with something a bit more interesting. They say without demonstrating our results show that it is possible to increase the context length of the model without hurting its performance on generic benchmarks. The bit about not hurting performance is interesting because in this experiment published a few weeks ago about extending the input size in tokens up to around 2 million tokens, the performance did drop off. If Google have found a way to increase the input size in tokens and not affect performance, that would be a breakthrough on multilingual benchmarks. Notice how the performance of Palm 2 in English is not dramatically better than in other languages. In fact, in many other languages it does better than in English. This is very different to GPT4, which was noticeably better in English than in all other languages. As Google hinted earlier, this is likely due to the multilingual text data that Google trained Palm2 with. In fact, on page 17, Google admit that the performance of Palm2 exceeds Google Translate for certain languages, and they show on page four that it can pass the mastery exams across a range of languages like Chinese, Japanese, Italian, French, Spanish, German, etc. Look at the difference between Palm 2 and Palm in red. Now before you rush off and try Bard in all of those languages, I tried that and apparently you can only use Bard at the moment in the following languages English, US English. What a pity. And Japanese and Korean. But I was able to test Bard in Korean on a question translated via Google Translate from the MMLU dataset. It got the question right in each of its drafts. In contrast, GPT4 not only got the question wrong in Korean when I originally tested it for my SmartGPT video, it got the question wrong in English. In case any of my regular viewers are wondering, I am working very hard on Smart GPT to understand what it's capable of and getting it benchmarked officially. And thank you so much for all the kind offers of help in that regard. I must admit it was very interesting to see on page 14 a direct comparison between Palm 2 and GPT 4. And Google do admit for the Palm 2 results they use chain of thought prompting and self consistency. Reading the self consistency paper did remind me quite a lot actually of SmartGPT where it picks the most consistent answer of multiple outputs. So I do wonder if this comparison is totally fair if Palm 2 used this method and GPT4 didn't. I have to talk about these benchmarks more in another video, otherwise this one would be too long. But a quick hint is that winogrand is about identifying what the pronoun in a sentence refers to. Google also weighed into the emerging abilities debate, saying that Palm 2 does indeed demonstrate new emerging abilities. They say it does so in things like multi step arithmetic problems, temporal sequences, and hierarchical reasoning. Of course I'm going to test all of those and have begun to do so already. And in my early experiments I'm getting quite an Interesting result. Palm2 gets a lot of questions wrong that GPT4 gets right, but it can also get questions right that GPT4 gets wrong. And I must admit it's really weird to see Palm 2 getting really advanced college level math questions right that GPT 4 gets wrong. And yet also when I ask it a basic question about prime numbers, it gets it kind of hilariously wrong. Honestly, I'm not certain what's going on there, but I do have my suspicions. Remember though that recent papers have claimed claim that emergent abilities are a mirage. So Google begs to differ. When Google put Palm 2 up against GPT4 in high school mathematics problems, it did outperform GPT4, but again it was using an advanced prompting strategy not 100% different from SmartGPT. So I wonder if the comparison is quite fair. What about coding? Well, again, it's really hard to find a direct comparison that's fair between the two models Overall, I would guess that the specialized coding model of Palm, what they call Palm 2s, is worse than GPT4. It says its pass at 1 accuracy as in pass first time is 37.6%. Remember the sparks of AGI paper? Well that gave GPT4 as having an 82% zero shot pass at 1 accuracy level. However, as I talked about in the Sparks of AGI video, the paper admits that it could be that GPT4 has seen and memorized some or all of human eval. There is one thing I will give Google credit on, which is that their code now sometimes references references where it came from. Here is a brief extract from the Google Keynote presentation How would I use.

Python to generate the Scholars Move in Chess? Okay, here Bard created a script to recreate this chess move in Python and notice how it also formatted the code nicely, making it easy to read. We've also heard great feedback from developers about how Bard provides code citations. And starting next week you'll notice something right here we're making code citations even more precise. If Bard brings in a block of code, just click this annotation and Bard will underline the block and link to the source.

As always, it seems the appendix contained more interesting information sometimes than the main body of the technical report. For example, we get a direct and fair comparison between GPT4 and Palm 2. Or I should say Flan Palm 2. That is the instruction fine tuned version of Palm 2. Essentially, that's the version where it's been fine tuned to get better at following a question and answer format. But Anyway, the original Palm 2 scored 78.3% and Flan Palm 2 scored 81.2%. That's below the 86.4% of GPT4. And that's why my broad conclusion is that GPT4 is a bit smarter than Palm 2. But as I'll be showing over the coming days and weeks, there are genuinely quite a few areas in which Palm 2 is better than GPT 4. What about the Big Bench, which was designed to be particularly tough for language models? I talked a lot about this in my earliest videos. Well, the graph is going to look pretty weird because Palm 2 has improved upon Palm while reducing the number of parameters. So the graph kind of doubles back on itself back up here up to around 69% according to the technical report. I would say this is quite a major moment in human history. There is now virtually no language task that the average human can do better than Palm 2. Of course, expert humans can do better in individual domains. The average human is now worse in virtually every domain of language. Here you can see that confirmation of the Big Bench hard results for Flan Palm 2 69.1%. Interestingly, in the original chart, Palm 2 is even claimed to have higher performance than that at 78.1% if you remember. The reason we can't compare that to GPT4 is that in the technical report for GPT4 they admit that during their contamination check we discovered that portions of Big Bench were inadvertently mixed into the training set and we excluded it from our reported results before we get to Gemini, Google show off in the latter half of the technical report with examples of linguistic ability like writing paragraphs in Tajiki and then translating them into Persian. They go on to show examples in Tamil and they are really making a big point of showing off its multilingual capabilities at this point. And I'm going to admit this is my personal opinion. Google then strays into dozens of pages on bias, toxicity and gender. Interestingly, some of the people paid to assess these risks were paid only 1.5 cents per judgment. These things do need to be addressed, of course, but it was somewhat shocking to me to see 20 pages of that and not a single page on the broader AI impacts. As many of you may know, I have criticized OpenAI plenty of times on this channel, but compare their technical report, which goes into far more detail about what we need to monitor. The closest Google got was showing how their universal translator could be used for deep fakes, universal translators and experimental AI video dubbing service that helps experts translate a speaker's voice while also matching their lip movements. Let me show you how it works with an online college course created in partnership with Arizona State University. What many college students don't realize is that knowing when to ask for help and then following through on using helpful resources is actually a hallmark of becoming a productive adult. Muchos universitarios no comprende que saber cuando pedira yuda Yusar recursos utiles para convert una dulto productivo. It just seems a massive black hole when one of their recent former employees, Geoffrey Hinton, had this to say this week on cnn.

You've spoken out saying that AI could manipulate or possibly figure out a way to kill humans.

How could it kill humans?

If it gets to be much smarter than us, it'll be very good at manipulation because it will have learned that from us. And very few examples of a more intelligent thing being controlled by a less intelligent thing. And it knows how to program, so it'll figure out ways of getting round restrictions we put on it. It'll figure out ways of manipulating people to do what it wants. It's not clear to me that we can solve this problem. I believe we should put a big effort into thinking about ways to solve the problem. I don't have a solution at present. I just want people to be aware that this is a really serious problem and we need to be thinking about it very hard.

This all seems particularly relevant when Google made this announcement about Gemini, their rival to GPT5.

All this helps set the stage for the inflection point we are at today. We recently brought these two teams together into a single unit, Google DeepMind. Using the computational resources of Google, they are focused on building more capable systems safely and responsibly. This includes our next generation foundation model Gemini, which is still in training. Gemini was created from the ground up to be multimodal, highly efficient at tool and API integrations, and built to enable future innovations like memory and planning.

That ability to plan may ring a Bell from the GPT4 technical report which said this novel capabilities often emerge in more powerful models. Some that are particularly concerning are the ability to create and act on long term plans. Remember, Google didn't identify planning as a risk, but as a selling point for Gemini. Next Google talked about accelerating their progress, which was again directly mentioned in the GPT4 technical report. It said one concern of particular importance to OpenAI is the risk of racing dynamics leading to a decline in safety standards, the diffusion of bad norms and accelerated AI timelines, each of which heightens societal risks associated with AI. We refer to these here as acceleration risk. And make no mistake, Gemini will be very accelerated. From Palm 2. It looks set to use the TPU V5 chip which was announced back in January of last year. And on page 91 of the Palm 2 technical report they say that that model used TPU V4. Now it should be said that Palm 2 is leading to some impressive medical applications, as I actually first reported on seven weeks ago without quite realizing it. Here's MedPalm2 we believe large language models have the potential to revolutionize healthcare and benefit society. Medpom is a large language model that we've taken and tuned for the medical domain.

You know, medical question answering has been a research grand challenge for several decades, but till date the progress has been kind of slow. But then over the course of the last three to four months, first with Med, Palm and MedPalm 2 we have kind of like broken through that barrier.

Unlike previous versions, MedPom 2 was able to score 85% on the USMLE medical licensing exam.

Yeah, this is immensely exciting because people have been working on medical question answering for over three decades. And finally we are at a stage where we can say with confidence that AI systems can now at least answer USMLE questions as good as experts.

As many of you may know, the CEO of Google as well as the CEO of Microsoft and Sam Altman and the CEO of Anthropic all went to the White House to discuss AI risk and opportunity. But given that the main outcome from that seems to be 140 million to establish seven new AI research institutes, that feels a little slow given all the acceleration that's occurring. Because as Google somewhat soberly conclude their report, we believe that further scaling of both model parameters and data set size and quality, as well as improvements in the architecture and objective, will continue to yield gains in language, understanding and generation. They are not slowing down and the world hasn't yet caught up. Thank you so much for watching to the end and have a wonderful day.