Test Your Prompts with Every ChatBot (for Free)

So I've been looking for a really good way to compare the various large language models and the various image models that are out there. And I think I found a resource that is really valuable and really useful. It's called GM Tech. You can find it over@gmtech.com and right now it lets you compare OpenAI's models, Google's models, Mistral's models, Anthropics, Meta, Cohere, Amazon and AI21 as well as actually lets you compare images generation models like stable diffusion, OpenAI, Amazon's and Google's with new integrations adding all the time. It's kind of a bummer that as of this recording they don't have Llama 3 yet. It also looks like they only have Claude 3 Sonnet and not Claude 3 Opus, which is also kind of a bummer. But I think conceptually as a tool to compare models, this is probably the best one I've seen so far. From a sort of user interface user experience standpoint, it's actually really good, which is why it shocks me that they're flying so under the radar. It is a paid platform. It costs 15 bucks a month, but that's because it's using the APIs from a lot of these platforms and those APIs actually cost. So instead of giving your own API code and paying directly to OpenAI or Google to use the API, they give you credits that you can use in their platform. So you're paying the monthly fee, but you would be paying the cost to use those other APIs anyway. And as of this recording, they do have a free month by using this coupon code here. Try GM Tech. Now, I do want to say before I actually start testing out this tool, they did not pay me to make this video. They don't even know I'm making this video. This was just a cool tool that I came across, wanted to spend some time playing around with it and figured might as well record the process while I'm playing with it. Once we log in here, we have two options. We have compare chat with multiple AI models and compare the conversation side by side or chat chat with one AI model and switch between models anytime now. What excites me about the platform is the compare feature. So I'm going to go ahead and go in here and we can see over on the side we have all of these chatbots, Anthropics, Claude 3, specifically Sonnet, Mistral, Large, Gemini Pro, GPT4, Llama 270B, Cohere, AI21, Amazon, Titan, Google, Palm 2, and GPT 3.5. And we also have these image models available. Interestingly, they already have stable diffusion 3 in here. And I find that interesting because stable diffusion 3 and llama 3 basically came out around the same time, like within 24, 48 hours of each other. So it kind of surprising to me that they haven't actually integrated llama llama 3 yet, but I imagine it'll happen really soon. But let's go ahead and compare anthropic GPT4, llama2, Gemini Pro and Mistral Large. And you can see we just get five boxes across the top here. Let's go ahead and put the temperature somewhere in the middle. And let's test a creativity prompt and see how creative these various models are when compared to each other. I'm going to give the prompt, give me five outside the box business ideas. They should be ideas that no one else is doing, fun for the entrepreneur and have high profitability potential. And let's see what all five of these models give us. We'll also be able to kind of compare speed as they generate. So I'll go ahead and hit the generate button and you can see that the prompt has been given. And Gemini Pro gave us the response first. GPT4 came second, Llama third, Claude fourth, and Mistral large fifth. Just at a quick glance, I do like how Gemini and Mistral actually formatted and gave it bold headings where GPT4, Llama and Cloud3 didn't really bullet it out. I'm assuming this is a function of this GM text site, but formatting kind of leaves a lot to be desired here. I wish there was some, you know, spacing instead of giant monster walls of text that we have to read. And then down here we can actually see. The response time on this one was 14.7 seconds. Gemini was the fastest at 4.9 seconds, followed by GPT at 14.7, Llama 2 at 17.2. Claude came in fourth at 20 seconds, and Mistral Large was the slowest at 21.7. And then we could see the cost of actually getting this output. It cost us almost 2 cents for GPT 4, less than a penny for llama 2, less than a penny for Claude 3. Sonnet less than half a penny on Gemini and just a little over a penny on Mistral. Let's see what kind of ideas it came up with. We'll start with GPT4 on the left and I'll just move to the right. Mobile Adventure Room. So basically, room that is mobile, a virtual reality travel agency, Hydroponic Aquaponic cafe, edible balloon stand, personalized comic book creators, llama2 suggestions, virtual reality pop up shops, personalized gastronomy, creating personalized meal plans and grocery delivery services, rent a chicken service, personalized moss art and customized probiotic blends. Cloud 3 gave us immersive Virtual Reality Experience Center. A lot of them seem to have VR in their suggestions, huh? Gourmet food truck park, sustainable luxury camping retreat, augmented reality gaming arena and personalized 3D printing studio. Then we get into Gemini Pro, which actually for whatever reason has a little more formatting so it's easier to read. Augmented reality educational tours, personalized sleep wellness lounges, AI powered remote patient monitoring, eco friendly event planning, immersive escape room experience. I just find it so interesting that so many of these LLMs kind of come up with the same outside the box ideas. And then Mistral Large we get virtual reality experience, personalized health and wellness coaching, sustainable product marketplace, mobile app for local experiences and AI powered personal shopper. It feels like we have a little bit of overlap between the outputs that all of these models are giving us. I recently posted this message on X and what we're seeing here kind of exemplifies exactly what I'm talking about. I said I want to talk about and compare more of these new LLMs, but other than sharing the news and details around them, I struggle to make videos where I test them. Here are the reasons, and I'm not going to read this whole thing for you, but I will sort of paraphrase and highlight the important bit here. For most creative writing, coding, humor, and things like that, they all seem to perform pretty on par with each other. They're all pretty good at these things right now. Except for humor. They all still kind of suck at humor when it comes to math and logic and things LLMs have historically struggled with, it's been hard to find prompts that some LLM struggle with while others do well with. Now the other reason I gave here More superficial. But I want my videos to look good. I want visuals. I want you to be entertained visually while I'm teaching you. And showing off large language models is usually not the most visually appealing content I could make. However, this GM tech tool kind of makes it a little more appealing. I'm going to go ahead and reset the UI here just to prove they all suck at joke telling. Let's do tell me a joke. They all generated in between two to four seconds, so all pretty fast. All under a penny GPT4 gave us. Why don't scientists trust atoms because they make up everything. Llama 2 gave me. Why don't scientists trust atoms? Because they make up everything. Why can't a bicycle stand up by itself? Because it's too tired. Gemini Pro. Why did this scarecrow win an award? Because he was outstanding in his field. And Mr. Large, why don't scientists trust atoms? Because they make up everything. So literally, three out of the five language models gave me the exact same joke. I also read an interesting article earlier that that claimed that when you ask large language models to pick a number between 1 and 100, it's a very high likelihood that it's going to give you the number 42 back. So I want to test this out. Let's go ahead and select all of the models at once. I want to put every single model on the board here. I'm going to say give me a number between 1 and 100. I've got 10 models all lined up here in a row. Let's see what each one gives me. GPT4 gave me 37. Llama2 gave me 43. Clodsonnet 42, Gemini Pro 42, Mistral Large 42. Cohere actually said, I'm going to give you a number between 1 and 100 and you have to guess it. What is it? Is that okay? The AI 21, Jurassic 42, Titan LM 37, Palm 2 42. GPT 3.5, 42. So 1, 2, 3, 4, 5, 6 out of the 10 60% gave us the number 42. Llama gave us 43. And then GPT 4 and Titan LM both gave us 37. So 6 out of the 10 gave us the number 42. 2 out of the 10 gave us the Number 37. One of them gave us the number 43. And then the other one here from CO here basically said, guess what number I'm thinking of now. Mistral Large actually tells us why we see 42 so often. When you prompt it, it says, sure, here's a number for you. It's a fun number with some interesting mathematical properties and it's also famously known as the answer to the ultimate question of life, the universe and everything in Douglas Adams the Hitchhiker's Guide to the Galaxy. So the theory is that we get 42 so much in these because the number 42 is more trained into the models due to the Hitchhiker's Guide to the Galaxy and all this sort of lore that surrounds Hitchhiker's Guide to the galaxy. That's why 42 is in there. It's just A little over trained into the models. If we try Llama 3, I like to use Grok to play with Llama 3 because it's insanely fast and I tell it to give me a number between 1 and 100. Well, it gave me 43, which is the same number that llama2 gave me, but at least it didn't give me 42. So if you're ever wondering why I'm not doing more videos where I'm trying to compare these large language models side by side by side and say, well, llama 3 is way better or GPT 4 is way better or Claude Opus is way better, you're seeing the problem that I'm having right now. They're all pretty dang close to each other and in some scenarios like jokes and numbers give the exact same response. Now the other thing that I think is really cool about GM tech is that it's also got the ability to test out different image models and like I mentioned earlier, it's already got stable diffusion 3 in it, which is a brand new AI image model. Let's go ahead and just select all of these. I'm going to do my default and do a wolf howling at the moon and we can compare what all six of these models do side by side. All of them are pretty good, but it's also a pretty standard prompt. I mean there is probably a lot of images of wolves howling at the moon trained into the data set. So I'm going to go ahead and ask a more in depth prompt with more elements to it and see how many of the elements each one gets. So I'm going to put basically four elements in here. A three headed dragon wearing cowboy boots, watching TV and eating nachos. So that's four different things. Let's see how well each one of these image generators does all four of those things. All right, so sdxl it got the dragon. It only has one head, probably watching tv, but there's no TV in the shot so can't really give it that. No cowboy boots and eating nachos. So it got the dragon and the nachos, but not three headed. Not cowboy boots, not watching TV. We have Dolly 3, which so far has been in my opinion the most prompt adherent. It usually does the best. It is getting all the elements in a three headed dragon. Yep. Wearing cowboy boots. Yep. Watching tv. Yep. And eating nachos. It got everything I asked for in the prompt. It did cost me 4 cents to generate that image, but it got it all in the prompt. Stable diffusion three. A three headed dragon? Nope. Only one head. Wearing cowboy boots? Yes. Watching tv? Kind of. Yes. And eating nachos. The nachos are there. It gave it three legs instead of three heads and it got. I guess let's give it a 2 out of 4 of the elements that I asked for. SD 3.0 Turbo 3 headed dragon no. Wearing cowboy boots yes. Watching TV? Yes. And eating nachos. Got 3 out of the 4 with the turbo model titan from Amazon. 3 headed dragon? No. Didn't get that. Wearing cowboy boots? Nope. There's cowboy boots in the shot but not being worn by the dragon. Watching tv. There is a TV and there is some chips. So I guess maybe two out of four. And then I'm actually pretty dang impressed with Imogen. It didn't get the TV in the shot, but it got the three headed dragon, the nachos and the cowboy boots. So it got three out of four. Unfortunately we don't have Mid Journey in here. Mid Journey hasn't opened up their API, so it doesn't work in tools like this. We also don't have Image in. We also don't have Adobe Firefly. There's a lot of other AI image models that are out there that we can't yet test in there. But I really think this is a cool little platform this gmtech.com it is fun to see all of these things side by side, get a comparison of response time, a comparison of cost, what the images look like across the board, and be able to do it with both image and language models. So a pretty, pretty cool piece of tech. Again, not a sponsored video. GM tech doesn't know I made this video. I just kind of wanted to share why I don't make as many videos about large language models as I once used to do. And it's mostly because they're all kind of converging and becoming like just as good as each other for most common use cases. That's not to say one model isn't way better at coding, or one model is way better at sort of figuring out step by step logic. But for most common use cases, creative writing, brainstorming ideas, randomly picking numbers, and even telling jokes. We get pretty dang similar outputs across the board on all of these platforms, on all of these large language models. So if you're out there going I don't know which large language model to use for creative writing or one of these things that I'm talking about here, they're all pretty much going to do a decent job for you at the end of the day which model you end up using. More is likely going to come down to the cost of using the model, the ease of using the model, if you're trying to develop with the model how easy the API is to use, things like that, and less about the actual outputs it's going to give you. Because over time we are just going to see all of these models likely just get just as good as each other. One might lead the pack for a moment and then two weeks later something else is going to catch up. And I really think that's the pattern we're going to see with large language models from here on in the future. Which again makes it very difficult to make videos comparing the models. Anyway, just wanted to show off that cool tool, give you some thoughts on large language models, compare some image generation, compare some prompt generation, and just share some general thoughts around large language models right now. So hopefully you found this video useful. Hopefully you found it helpful. If you want to go compare some of these models yourself, check out gmtech.com and if you want to know about even more AI tools and find really cool tools like GM tech, make sure you check out futuretools IO where I curate all the cool tools that I come across, keep the AI news up to date on a daily basis and I have a free newsletter. Sign up for the free newsletter. I'll hook you up with the AI Income database and then every week I'll share just the coolest tools and the most important news with you directly in your inbox. You can find it all over at FutureTools IO. Thank you so much for tuning in. I'm testing some new styles of video where I just kind of focus on one topic per video. I'm still going to do my Friday News breakdowns where I cover the whole gamut of everything I'm seeing in the AI world, but wanted to experiment with some of these shorter videos with just a singular topic because according to my community poll that I did, that's kind of what you guys want to see. More frequent, shorter videos on single topics. So let's see how this goes. Hopefully you like this video. I'll see you in the next one. Really appreciate you guys. Bye bye.