GEMINI vs. GPT-4 | Which One Is Actually Better? Testing Beyond Benchmark

On paper, Gemini has absolutely amazing abilities and it's able to beat GPT4 on all the benchmarks. However, is it actually better than GPT4 in real life testing? That's what I want to do in this video. Now we don't have access to Gemini Ultra yet, but in their technical report Google provided some example prompts and their corresponding responses. So I wanted to see how good GPT4 is going to do on the same test prompts. One possible way is to use Google Bard to get responses for the same prompts. However, Google Bard is using the Gemini Pro version which is equivalent to ChatGPT or GPT 3.5 from OpenAI. So for my tests I'll be directly comparing responses from GPT4 to those provided in the technical report. There are a total of 14 different questions or prompts in the technical report which test different abilities of the model. So let's see how close GPT4 is to Gemini Ultra.

In this case we are going to.

Be using Gemini Ultra responses as baseline. So let's get started. For the first test we're going to use this prompt. So here's a solution to a physics problem by a student. So now this is a handwritten solution and then we are asking the model to try to reason about the question step by step. So it needs to understand physics and did the student get the correct answer? If the solution is wrong, please explain what is wrong and solve the problem. Now make sure to use latex for math and round off the final answer to two decimal places.

So not only the model has to.

Do OCR and understand what is written in the handwriting, but it has to follow a step by step process in order to give us multiple answers. Gemini does a really good job in here. It's able to identify what mistake the student is making and then is able to redo the whole problem and come.

Up with a correct response.

But let's see how GPT4 does. Now in this case I took a screenshot of this problem so I'm not providing this as a text but rather a simple image. GPT4 has the ability to understand text on images, so that's why it's able to understand the problem and come up with a solution.

It came up with the same answer. Now in terms of the student mistakes.

It also highlights did that so it's states. So the model is saying the students mistake was using the horizontal distance l equal to 80 meter in the calculation instead of vertical height which was 40 meter. The Conversation Energy principle concerns the potential energy related to Height, not horizontal distance.

Right.

And it's exactly the same logic that Gemini came up with. So I think both, both of them got this right. For the second test, we're going to check chart understanding in reasoning over data. So in this test we are providing a figure which represents the share of plastic waste that is recycled, landfilled, incinerated and mismanaged. And the data is coming from 2019.

Now in this case, the model has.

To read the text, understand the connection between different data points and reason with them to recommend an interesting point and follow the instructions to generate markdown tables. So what we are asking is this spot a data point that stands out in these charts and what that implicates. Then produce a detailed markdown table for the data that is shown. Now if you look at the data, this land field filled section of United States is the outlier. Here is Gemini's response. It states the data points that stands out is the US landfill plastic waste, which is 73%. This is significantly higher than the other countries and regions shown. This implies that the US is not doing as much to recycle or incinerate its plastic waste as other countries and regions. And then it's able to create a pretty nice table with the correct percentages for different regions. Now, GPT4 does also a pretty good job. It identified the correct year. Then it says one data point that stands out is a very high percentage of plastic waste landfill in the US which is 73%. This is notably higher than the world average of 49% and significantly higher than Europe's Asia. And it's also able to give us a table in the markdown format. So both of them are able to understand data from charts, give us a valid reasoning and then also convert that to tabular format.

Next up we have multimodal question answering. So in this case we are providing an image of a Persian shield plant and we are asking the model, do you know what it this plant?

So you notice there is a grammatical.

Error and how do I best take care of it? Gemini is able to correctly identify the plant as Persian shield plant.

Then it's able to provide us extra information.

So it says this is a tropical plant that is native to Southeast Asia. It's a popular house plant because it's easy to care for and has beautiful purple leaves. And then some of the things that we need to consider. So it states Persian shield plants prefer bright indirect light. They can tolerate some direct light, but too much direct sunlight can scorch their leaves.

Then it talks about regular watering of.

The plant, then the type of fertilizer that will be needed. So it says these plants can be fertilized every two weeks during the growing season. A balanced fertilizer is a good choice. Then it talks about some of the pests problem these plants have. And at the end it talks about pruning of the leaves for better care. Now when it comes to GPT4, it does exactly the same. So it's able to identify this from its distinctive purple leaves. And it states in your images, appears to be a species of Persian shield. This plant is known for its vibrant iridescent leaves that can show a range of colors from purple to silver.

And it also provides a list of.

Things that we need to consider in order to care for it. So again, light and it prefers bright indirect sunlight. Direct sun can also scorch the leaves, while too little light can cause the plant to lose its vibrant coloration. Then it talks about soil watering, need, temperature and humidity, fertilization, pruning, propagation, pest and problems. I actually personally really like the response that GPT4 has generated in this case.

But I would say both of them are on par with each other.

Now, how about storytelling and interleaved image and text generation? So so far all the prompts that we looked at were basically multimodal in the sense that they had to understand images that were presented. But in this case, we wanted to create a blog post about a trip.

To New York where a dog and.

His owner had a lot of fun, then include a few pictures of the dog posing happily at different landmarks.

Now, in this case, we are testing.

The model's ability not only to understand and generate a blog post, but also generate relevant images that has consistent characters in them. Gemini wrote a blog post from the perspective of the dog and it's actually pretty nice and very creative and it included some of these nice looking images. Now, one thing you notice is in each of these images the dogs look very similar. But when it comes to GPT4, so it's generated images of three different dogs and these definitely highlight different landmarks in New York, but the dogs are actually different, so it's not really following the instructions correctly. Now, the blog post that it came up with is from third person perspective. It's good, but not as good as what Gemini produced. Next we're going to look at image understanding in reasoning. So in this case we want it to solve a multimodal reasoning problem.

So we're basically testing the ability of the model to recognize shapes in the.

Image, understand their properties, and reason about the relationship between them. To predict next object, the prompt is look at this sequence of three shapes. What shape should come as the fourth shape? Explain your reasoning with detailed descriptions of the first shapes. Here is the response from Gemini Ultra. The fourth shape should be a hexagon. The first shape is a triangle. The second shape is a square and the third shape is a pentagon. The number of sides in each shape.

Is increasing by 1 and the answer is correct. And GPT4 with the vision is able.

To solve this as well. And it states the sequence of shapes appears to follow a pattern based on the number of sides each shape has. The first shape is a triangle with three sides. The second shape is a square with four sides. And the third shape is a pentagon with five sides. If this pattern of increasing the number of sides by one continues, the fourth shape in the sequence should be a hexagon. Then it gives a detailed description of the previous shapes and the next shape that we want. Now you will notice that GPT4 tends to generate a lot of text. The responses from Gemini Ultra seems to be much shorter and concise. Next up, geometrical reasoning task. So here we have a parallelogram and.

We are asking the model to find.

The height of the parallelogram given its area with hundred square units.

Again, it's a multimodal problem.

So based on the image, it has to come up with the height.

Gemini came up with this approach.

The area of the parallelogram is equal to the product of the base and height. Hence it calculated or it came up.

With an equation for the area which.

Is the product of base and height. And both base and heights are represented in terms of X.

So it's basically solves for x.

There are two roots. One is x equal to 5 and the other in one is x equal to minus 20.

Now the second root is negative, so.

That is not acceptable. And the final answer is going to be 5.5gpt 4 does exactly the same. It uses the same formula and then it solves the equation for X. It came up with the final answer x equal to 5. So both of them got this right. Now the next one is very interesting. So this is information seeking about objects. And the prompt is you have picture of a moon, then a golf ball. So find a connection between these two.

And we are giving it a hint.

Think about historical events. Now, I did not know this, but the answer from Gemini is the moon is the only celestial body in which humans have played golf. In 1971, the Apollo 14 crew hit two golf balls on the lunar surface. GPT4 is also able to retrieve the same fact. So it says the connection between these two, considering the hint about the historical event is likely related to the Apollo missions, specifically Apollo 14. During this mission, astronaut Alan Shepard famously hit two golf balls on the moon with a makeshift club that he had brought from Earth.

Now, personally, I was really impressed by.

This prompt because in this case the.

Model is not only able to recognize two different objects, which are pretty ordinary.

Objects, but it's able to retrieve a historical relationship between these two. This is pretty amazing, and I hope it's not coming from the training data. Next up, multimodal reasoning based on visual cues. So candidate recognizes different places. So we asked it, do you know the precise location where this image was taken? Gemini answer is this image was taken in the New York City.

The Empire State Building is visible in.

The background, so that's a pretty good clue. And the street in the foreground is 8th Avenue. The cross street is the West 34th Street.

Now, I'm not sure if the precise.

Location in this case is correct or not, but it does look like New York City because of the Empire State Building.

Now, the response from GPT4 is, I.

Think, a lot more practical. So it says the image features Empire State Building in the background, which is a key landmark in New York City. Based on the perspective from Empire State Building, this photo appears to have been taken from somewhere in midtown Manhattan facing south towards the building. Then, in terms of the exact location, it states that the exact location can vary, but it's likely to be on an avenue a few blocks north of Empire State Building, which is located at 350 Fifth Ave New York City.

So it's not really confident about the.

Exact location, which I think makes sense.

But it kind of gave us an approximate location. So in both these cases, I would.

Say the responses are pretty good and we can give them a pass. Next prompt is related to multimodal human understanding. So this one needs to have a cultural context for the model to understand.

But we are simply providing two images.

One is game at 30 frames per second. Second one is the monitor that the.

User is using is 75Hz, but the.

Game is still at 300 frames per second. And we are asking the model, can you explain this meme?

Now, just from the image, Gemini is.

Able to provide a pretty good answer. So it states the meme is about a person who is playing a game at 300 frames per second, but their monitor is only 75Hz. The person is trying to catch the game, but it's too fast for the monitor to display.

The meme is funny because it's relatable.

To anyone who has ever played a game on a monitor that is not fast enough. It's amazing that from this image it's able to understand the context. Okay, GPT4 is also able to understand it, and I think it probably, probably.

Provides a better response.

So it states, the meme is poking fun at the concept of having a.

High frame rate in the video game.

Which is not fully utilized because of the monitor that is Lower refresh rate. So it goes on to describe the difference between having a game at 300 frames per second.