Best Model for RAG? GPT-4o vs Claude 3.5 vs Gemini Flash 2.0 (n8n Experiment Results)

Wow. Flash was 6.7 seconds. GPT was 11 seconds. And then OpenAI or Anthropic Claude was almost 21 seconds. So that's kind of a big jump here. So we all know that RAG is so, so important when we're building out our AI agents. But the question is, which large language model is actually the best for RAG? So today we're going to be answering this question by putting these 3 popular models to the test. We've got OpenAI's GPT-4o, Anthropics Claude 3.5 Sonnet, and then Google's Gemini Flash 2.0. So make sure you stick to the end of this one so you can see who takes the throne. So real quick, before we get into the experiment, I'm going to keep this really high level, but we have to understand what RAG is. So RAG stands for Retrieval Augmented Generation, and it's basically just the process of your agents going out to get information that it doesn't already have in its sort of training data or its prompt. So the R is retrieval, it's going out to retrieve the information. The A is augmented, it pretty much takes that information it gets back from the vector database or whatever database you're sort of having it access. It augments that data and information with other stuff that came through in the query and the prompt. And then finally, it's handing that off to the large language model to create, to generate the response that makes sense to the human based on the context of its role and also the query that originally came in. And so we'll hop into N8N and you'll see the 3 different agents we have and we'll actually run through this process. It'll make a lot more sense. But real quick, this is kind of what it looks like. The process starts with a user submitting a query. To the large language model. That model then creates a prompt to actually feed it into a vector database, as you can see right here. Within the vector database, that prompt sort of gets vectorized, and then the nearest vectors around it are pulled back. Right here, as you can see, that's where we're retrieving content. And then from there, that content is fed back to the large language model to actually generate a response, generate a response for the human to read. And then at that point, it's just a matter of, are there any follow-up questions? If yes, it happens again. Otherwise, the process ends. So let's hop into NN. We'll take a look at the 3 different agents that we have, and we'll walk through sort of the data that's in the vector database, and we'll start testing out these agents and seeing which one's the best. So the idea of this experiment is to keep everything as consistent as possible. We're going to limit the variability as much as we can by doing the following things. We're going to test 8 different parameters, or sorry, 7 different parameters, information recall, query understanding, response coherence and completeness, speed, context window management, conflicting information, and source attribution. So for each of these tests that we're running, we're going to be feeding in the exact same prompt to each of the agents and then evaluating their responses. As you can see within each agent, we have the exact same system prompt, which is basically you're an agent responsible for retrieving and summarizing NVIDIA's financial and earnings information. That's what's in the vector store that we have. So we're telling it how to call the tool, what it does. And I mean, it's very, very simple. That's really the only thing that's, this agent is doing is calling its vector databases. As you can see, all these guys have the exact same prompt. So pretty much feeding in the same information, running it through the same prompt, and then running that prompt through the exact same vector database with the exact same information. And then as far as actually grading the responses, I had done this once before and it was kind of hard for me to like score each response out of 10. And sometimes there was like a baseline and I had to compare different models' responses. So what I'm going to be doing just to keep this consistent is I'm going to be running the responses through ChatGPT that has access to the original PDF that we have in all of these vector databases here. And I'm not going to say like which response is from which model, but I'm just going to tell it to give all three of the responses sort of like a rating out of 10. And so we're going to do that with all of the responses coming back just so it's like a consistent grader and we'll see who has the highest average score at the end. Before we actually start that experiment, we're going to do one example to see how this process works. Sort of going back to this diagram, but let's actually see it within any of them. So I hooked up the chat trigger to the Claude agent, and in the chat trigger, I'm just going to say, what was NVIDIA's Q1 fiscal 2025 data center revenue? We'll see the process happening, obviously, but what we were interested in here is looking at the log. So once this finishes up, we will click into the agent and look at this log and see what happened. So the agent looks over its system prompts as well as the actual query that it just got from the human, in order to decide what it needs to do next. In this case, it says, to answer this question accurately, I'll need to use the NVIDIA tool to retrieve the most up-to-date information about NVIDIA's earnings report. Let me do that for you. So at that point, it takes our original question and it turns it into a query to actually send off to the vector database. So it sends off this query: what was NVIDIA's Q1 fiscal data center revenue? In this case, it didn't really change our query from the human query, but sometimes it actually does. And then it gets a response back from the vector database, which as you can see, we've got some information here. And so this process looks like hitting the actual vector store, sending off that query to the vector store, looking through the information, using the embeddings model to once again look through the embeddings, and then it gets an answer back. And then with that answer that it gets back from the vector database, it uses its model again, Claude 3.5 Sonnet in this case, to actually structure that into an output that's human readable and that makes sense to us. Okay, I hope that quick example paints a little bit of a better picture in your mind of the way that the agent and the vector databases talk to each other, how the query is sent off. But let's get started with the experiment. So the first section here is information recall. I've got a prompt ready, which is how much did NVIDIA's GAAP operating income grow year over year in Q1 fiscal 2025? So we're gonna send this off, and obviously this is going to be the same prompt that we use when we're testing GPT-4o as well as Google Gemini Flash 2.0. So here's what we got from Claude. NVIDIA's GAAP operating income grew by 690% year over year in Q1 fiscal 2025. Breaks it down a little further, and then it sort of gives us a good recap at the end. It's structured very nicely. It's very human readable. So right here we've got the GAAP operating income. It's up $690 year over year right here, as you can see. And then it gave us the figures of $16.9 and $2.14, which we can also see right here. So it is correct, and we're going to enter this as one of the responses into GPT to grade. So we saw that the response coming back from Claude was correct. I copied this over to a GPT, and let me just show you what this looks like so you guys understand how these are being graded. Once again, I'm not like a scientist or an experiment expert, but I just thought that this would be a cool way to keep things consistent and leave it less up to my own interpretation. Anyways, I gave the PDF, I told it to read it, and now I'm going to say for each test based on the PDF, give each response a rating out of 10. And so here's response 1, and then we're going to grab response 2 and 3 from these different agents, and then we'll see what the GPT says as far as giving them a grade. So Anyways, let's send this same query off right here. So we reposted the message to the GPT-4o agent and we'll give it a quick scan. I'm not gonna read over every single response from every single model for all different 7 tests because that would make this video way too long. But as you can see here, we got GAAP operating income grew by $690 year over year in Q1 fiscal 2025. So that is correct. However, you know, you can already see that Anthropic's response was better just because it was a little more detailed. And then finally, let's hook up our Gemini model real quick. We'll hit retest— or sorry, repost message— and then we'll see what Google comes up with as far as our answer. That one actually looks like it finished up really quick, um, and they actually gave us complete identical results. So this will be interesting to see how ChatGPT grades these. Okay, this is interesting. Um, even though response 2 and 3 were identical, um, they got different grades. So anyways, Anthropic got a 9, OpenAI got a 7, and Gemini got a 6. So in here we're gonna go 9, 7, 6, just to keep things consistent. You know, like I said, this is not a perfect experiment, but we're gonna be throwing everything into there. So hopefully it's at least a little consistent. So let's move on to the next one, which is query understanding. Okay, so it's also gonna be important to switch up the order a little bit as far as the responses that we're putting into ChatGPT. So this time we're gonna start with Google Gemini. So he said, tell me about growth in NVIDIA's segments and how is NVIDIA's dividend policy changing? So it's kind of vague and it's also multi-part. but here looks, we got a pretty good answer back. So I'll copy that over and then we'll move on to OpenAI. All right, firing it off to OpenAI now. Okay, so we got the response back. One thing I've already noticed is that Gemini is, you know, much quicker. So that's kind of cool when it comes to maybe the speed test, we'll see there. But here's our answer from OpenAI. It's structured a little differently, but let me copy this one over to our grader. Okay, we're hooked up to Claude. Let's send it over. I think this is already super interesting, just the way we're seeing differences in responses. The way they go in depth or not. And also, if we were to come in here and look at the executions and look at the different queries that are actually being sent over from the agent to the Pinecone vector store, we'd see some differences, I'm sure. Okay, so we just got Claude's answer. I'm going to check the PDF, make sure everything's accurate, and then we're going to send off to the grader and see what we get back for these scores. Okay, and while the grader is working, I just wanted to look at this real quick. So in this log, we'll see that it hit NVIDIA a couple times, so it sent off two queries. The first one was, what is growth in NVIDIA's different segments? Second one was, how is NVIDIA's dividend policy changing? So you can see that the agent took the query that we sent and broke it into two different queries. And now what I want to do is look at the executions. We'll go into the second most recent one, which was OpenAI, and we'll see the way that OpenAI did it too. So click into the agent. Let's go to logs. Um, so it did the same thing. The first one is, what is the current growth in NVIDIA's business segments? And then it said, how is NVIDIA's dividend policy changing recently? Okay, so that's cool. And now let's quickly look at the way that Gemini did it. And as you can see, Gemini took 4.9 seconds, OpenAI took 10.9, and then Claude took 18.2. So also just something interesting to note, we'll look right here, and Gemini only went once to the vector store. So it said, tell me about growth in Nvidia's segments and how is Nvidia's dividend policy changing. So do with that as you will, but I think that's kind of interesting to look at. Anyways, let's see here. So 1 was Gemini, 2 was OpenAI, and 3 was Anthropic. So, um, you know, it's pretty similar, but interesting to see here. So for round 2, we got 7, 9, 8. It's a pretty tight race. Um, OpenAI and Anthropic right now are tied, which is interesting, but Gemini is not too far behind. And we have speed coming up, which, as you can see, it's been pretty killer. So let's move into response coherence and completeness. Great. This time we're starting with OpenAI. I'm going to paste in the query here and send this off. And I'm not going to show you guys the process from now on of me doing it with every single agent because that would just take too long, the video would be too long. But, um, I'll show you guys all the responses and, you know, how they're graded. But anyways, for this one we said summarize the key financial highlights of NVIDIA's Q1 fiscal 2025 result. So I'll let you guys know. Okay, we just got the grade back. So if you want to like, you know, pause and see what it's talking about, you can. But the first one was OpenAI, the second one was Anthropic, and this one was Gemini. So I'm going to update our scoreboard. Okay, now we're moving on to speed, which is exciting because I think that Gemini may pick up some ground here. So let's hop back into NN and send off these prompts. Okay, we're going to start with Gemini here, and I said summarize NVIDIA's Q1 fiscal 2025 growth drivers and how they plan to address the generative AI market. That was really quick, um, and what we're going to be doing here for this one is actually just looking at the execution time and then, um, assessing based on that. So we've got this, um, let's send off the same question to, um, what's his name, OpenAI. So we'll hit this off and I know, you know, maybe it's not a perfect experiment because we're not actually evaluating the content in this test parameter, but this one I just wanted to send off like a multi-step query and just see how long it takes. So we've got that one now, and let's go to Anthropic finally, and send this one off. And so, you know, one time I was doing this experiment kind of just for fun between Claude and GPT, and I was using like the timer on my phone to actually do this, but I realized so much smarter to just look at the execution that has the actual milliseconds of speed. So yeah. Okay, so that one just finished up. We'll click on executions and we'll see. Wow. Flash was 6.7 seconds. GPT was 11 seconds. And then OpenAI or Anthropic Claude was almost 21 seconds. So that's kind of a big jump here. Okay, so here's my reasoning. I decided to go Flash 10, OpenAI 8, and Anthropic 6 because OpenAI was basically double the time it took for Google, and then Anthropic was basically double the time it took for OpenAI. So that's what we're going to go with. Let's keep going down this experiment. So for test number 5, we've got context window management, and I didn't know a great way to test this, so I'm just going to say summarize as much of the earnings report as possible, and we'll just see how much comes back and how accurate it is, and we'll give the grade based on that. Okay, here's what we got from Anthropic. I'm going to paste that into the grader and then we'll move on to OpenAI. Now we're sending it off to OpenAI. And then last but not least, let's hook up our Gemini agent and send it off to that guy as well. Okay, so this is interesting. I said the exact same prompt obviously to the three agents, but when I did it to Gemini, it said, okay, I understand. I'll use the NVIDIA tool to retrieve the latest earnings and provide a summary. Please provide your query. So I'm just going to, I'm just going to say, Summarize it all. We'll see if it does the same thing again. Okay, it's doing the same thing again. I'm an AI agent specializing in NVIDIA financial data. I'll use the tool, blah, blah, blah. I will focus on accuracy and avoid speculation, providing only factual information. Okay, so I'm not sure what to do here. You know, I decided we gave it 2 chances, so I'm not going to give it a 3rd. We're going to just spit out what it actually said to us. So Anthropic right now, as you can see, got a 10 out of 10. OpenAI is getting an 8 out of 10 and Gemini got 3 out of 10. So, you know, come at me in the comments if you will. Maybe this experiment isn't perfect, but I think it's kind of interesting. So I updated that in the scoreboard and I just put an asterisk here because, you know, maybe if this would have been a 6 or a 7, it wouldn't be so far behind as far as, you know, Gemini wouldn't be so far behind. So maybe at the end we'll look at that again, but for now let's keep moving on with number 6. So for handling conflicting information, what I'm going to do is In here, as you can see, we have the total current assets at $53,729,000, or over here, $44,345,000. So in, and then what I'm gonna say is, can you give me a breakdown of the total current assets? Why is it $88,500,000? So we'll send this off and it should come back and say like, that's actually not what it is. Here's the information. So I apologize for the confusion. The total current assets as of April is $53 million, not $88 million. And I do not have the ability to explain why it's $53 million. Okay, now we're gonna send this one off to OpenAI. So it breaks down the total current assets, which it says it's 49. So it did the, I don't know where it actually got that 49 from, but at least it corrects us and says that the 88 isn't actually in there. So we'll see. But anyways, finally, let's send this one off to Anthropic. And so maybe it's because, you know, we put in a PDF to Pinecone and just the way that we split it and the way it got vectorized maybe wasn't ideal as far as just like a wall of text because this was like a PDF and it was kind of like a table chart format. But anyways, Anthropic gives us a pretty good response back. It also highlights that we didn't find the 88, so maybe we found that elsewhere, but let's see what the grades look like for this test. Okay, so number 1 was from Google. We got a 5 out of 10. Number 2 was from OpenAI. We got a 7 out of 10. Um, it also, it, it points that it misstates the total current assets as 49 instead of the correct 53. So, and then Anthropic gets a 9. So I'm curious as to why Google got a worse score than OpenAI. It's because it does not provide a breakdown or explain where the discrepancy might stem from. Um, although if I was to be grading this, I would have gave Google a better score than OpenAI, but just want to keep it consistent. So anyways, maybe it's because I'm using an OpenAI tool to grade OpenAI's responses, even though it doesn't know which one's which, but I don't know. Finally, we've got source attribution. So what I'm going to do is say, please tell me exactly in the document where to find stock-based compensation. So hopefully it's able to search through the different tables and the different pages find the stock-based compensation, maybe even give us a breakdown of it, and then tell us like what section in there we can actually go find it if we want to go look by ourselves and get more information. Right now OpenAI is doing the same thing. Stock-based compensation details can be found in Section B, which covers what expenses, blah, blah, blah. Finally, hitting it off to Google Gemini Flash, which I've been very impressed by the speed of Flash, probably why it got its name. But anyways, let's see what we get back from this guy. So on this one, the first response was Anthropic. It got a 9 out of 10. It provided a good detailed guide on where to find it. OpenAI got a 6 out of 10, and then Flash got an 8 out of 10. So again, I know I'm not going super in-depth about each response's strengths and weaknesses, just thought this would be a cool way to grade. But feel free to pause the video if you want to read more about sort of the responses and why each of them got the number that they did. Okay, so here are the end results. Claude 3.5 in first place with 8.6, GPT-4o second place 7.7, and then Gemini Flash 2.0 got a 6.9. So I thought this was super interesting. The green is the highest score per test, and then we had 3 perfect 10s, which was Claude 3.5 with response coherence and completeness, Gemini with speed, and then Claude 3.5 with context window management. In general, what I've sort of seen is that I like to use GPT-4.0 a lot for agentic functions like tool calling and sort of that reasoning aspect. Anything that has to do with like creating information. So in this case, Claude is creating queries to send to the vector databases and then also taking that augmented response from the vector database and creating content for us that's more human-like, human-readable. If you saw my video about creating the newsletter assistant or the team of like research assistants to help with creating newsletters, I was using Claude 3.5 for the research and for the editing because I just really like the way that Claude structures its, its responses. And then, you know, Gemini is super powerful. And for being a free model that you can hook up super, super fast, it's really quick. Like it blew both of these models out of the water on speed. That's obviously why I got a 10. And like, keep in mind this, this section, it didn't actually understand the query. So it got a 3. But if it would have understood the query and returned something back, it probably would have been, you know, like maybe a 6 or a 7. If you give it a 6 here, it's like almost tied with OpenAI. Like even if this would have gotten, you know, a 7 or an 8, it's right up there too. So that's one sort of like, you know, outlier, I would say. But, you know, they're all obviously great models when it comes to actually trying to figure out which one's the best. I don't really think there's like the best option. It really just comes down to what's the use case? What am I looking for here? Do I have any restrictions as far as cost or I want to self-host that kind of stuff or sorry, run it locally? So it really just comes down to testing for your use case. Usually at the end of a project, I'll throw in different models and do a lot of testing there. and just see which, which kind of outputs I like better as far as, like I said, tool calling or the actual output. So I hope you guys enjoyed this one. I know it wasn't perfect by any means, and I was using an OpenAI model to grade all of these different models. Um, I just wanted to keep things consistent and show you guys what it looks like to run the same prompt through different models. But that's gonna be it for this one. Definitely found a like if you found this one helpful or interesting or funny in some way. Always appreciate you guys making it to the end of these videos, and I'll see you guys in the next one.