Benchmarking: NVIDIA DGX Spark & NVIDIA RTX 4090 | Part 4 - Comparing the results

We're going live. Let's see how we are on YouTube. If you're joining the chat, let me know. Stream settings. Okay, let's see. There we go. Okay, we can hear it. If you're joining the chat, say hello, say yo. Let's now— so over the past few streams, we've been comparing results for the, uh, DGX Spark, which is the NVIDIA GB10, and the GeForce RTX 4090. So we can't run all of the benchmarks on the RTX 4090 because— why? Well, it can't handle some of the models, so I'm running this locally. This is on my Mac Mini, by the way, so we're not running any—

anything new.

What I might do is run. Let me pop out the chat actually. Hello Namal, how are we? So I got the benchmark scripts and then I want to move these benchmark compare, benchmark report benchmark_report.md, benchmark Singles, benchmark utils. Maybe I move these into benchmark comparisons or something like that. Alex, hello! Am I using Claude code? No, we're using Codex. TreeFrog, have you asked that question several times in the chat? This is a new app. What's the update here? So I'm running Codex on my Mac Mini, and what is this?

Codex automatically runs commands in a sandbox.

Hi, so I want to move these all into a single script, I believe, a single folder. So let me just show you the benchmark so far. This is what I got. This is generated by Codex, um, but it's just comparing, going through all the CSV files in here, right? To do that would take me ages manually. So that's what AI is really good for, is going through and doing like analysis tasks. And because I've got the ground truth, I can always refer back to that. So this is LLM/VLM inference. Now we use Llama server as well as VLLM, and then we also have LLM training and image generation and object detection. So 5 major or 4 major benchmarks, but frameworks. So let me zoom in on that so that people can see it.

There we go.

So random question, will the old black hoodie profile picture make a comeback? Did you like that one? I like that profile picture too. But that's a couple of years ago now. Maybe I need to get a new black hoodie. Or no, I use the same black hoodie but update it for like a modern one. So if I take a photo from here, I just be in a black hoodie. Black hoodie photo from there, that's a new profile picture. So this is GPT-OSS-20B, uh, Unslough Dynamic Quantization for, uh, XL-GGUF 100 samples. So if we look at the prompt throughput, basically the, the, the TL;DR of the results here is that the DGX Spark compared to the RTX 4090, the DGX Spark is slower on anything that requires compute, real, real powerful compute, um, but it can actually run benchmarks with larger models. So missing— so you notice that the NVIDIA RTX 4090 isn't there. So there's several benchmarks here where there isn't any results for the RTX 4090 because it simply can't run those models. But basically on every other benchmark where, uh, compute power makes a difference, which is everything else, the RTX 4090 is faster. And so total runtime— this is lower is better, of course— is— what's this one? So this is Llama Server Quen 3 VL. Multimodal samples. So we have image, 115 seconds versus 389. So what's that?

389 divided by 1115. You can't zoom in on calculator, it's only that big.

3.3x. And then here, if we look at generation output for GPT-OSS, 242 divided by 88, 2.75 faster. So what I want to figure out today— Arav Jan, how are you? Good, good evening or good morning. It's morning here in Brisbane time. Got a cup of tea. Today we're compiling all of our results. So yeah, there we go, Nimotron 3. So let's write down some benchmarks we want to do. I might move those into, uh, a README.

We'll call it Benchmark Analysis, and then let's get Codex to Move all the @benchmark reporting files, not the benchmark results, into the benchmark analysis folder. Change. The benchmark_report.md to be the README.md in this folder. In essence, all benchmark reporting files should be updated to reflect this new pathing system. Final product benchmark results folder with results from different tests. Benchmark Analysis folder with analysis and reporting tools to inspect.

Hey, just want to ask you, if I'm at college and I need an entry-level position, where does RTX 2050 will get me to there? I need an entry-level position anywhere. Does RTX 2050 will get me there? What do you mean? You're in college and you need a job somewhere and and you have an RTX 2050, is that what you're saying? Okay, so RTX 2050, entry level 2021. 4 gigabytes of memory, that'll hold you back a bit on running LLMs, but you can definitely do a lot of machine learning experiments on still 4 gigabytes of memory. You just have to be smart about— you run smaller models on that. So, and plus, I wouldn't worry too much about what GPU you have in terms of locally. I would— you could, you could use Google Colab or something like that. And it also depends on what entry-level position you're going to. Okay, this is good. Now we've got benchmark analysis. And if we look at our README— nice. So if you want to get a job anywhere, Daniel Bourke, how to get an ML job, read this article. This is what I would personally do if I want— I'm not looking for a job at the moment, but if I was, I would do something like this.

And my internet is going so slow at the moment. There we go.

So this is a little— this is a couple years old now, but I would still do the same thing that's listed in here. All the techniques are timeless in there. So let's see what we got. How's Codex going? Okay, it's thinking. That's all right. So we need to think of some benchmark comparisons we want. We've got the baseline comparisons. I don't use Cursor. I don't use Claude Code, I use mostly VS Code and just a combination of different agent chat windows. What are we doing today on stream? We are benchmarking the NVIDIA DGX Spark and NVIDIA RTX 4090. Well, we've actually done that, so now we're comparing the results. So if we have Oh no, mostly focused now on comparing the results because I want to make a YouTube video about this. So if we look at LLM training, we have total train time 81 seconds. So if we go to calculator, 3, 4, 3 divided by 81. 4.2 times faster on the LLM training. And then generation. So this is Flux-2 Klein 4B, which is Klein is stands for small. How long did it take? Seconds. So 420 divided by 128 to generate 100 images, 3.2 times faster. So the ratio, or the rough ratio for— and here we go, here's another good one. This is object detection. We have 800 divided by 222, 3.6. So that's a rough ratio. Yes, you were just in time to see the results. So the rough ratio for any computing that the, uh, RTX 4090 needs to do compared to the DGX Spark is about 3 to 4 times faster in terms of compute numbers. So let's have a look at why that might be. NVIDIA DGX Spark specs. I want the spec sheet. Here we go. And then we go RTX 4090 NVIDIA. Okay, here's a good comparison. So if we look at Let's go Cuda cores. Cuda cores, Blackwell generation. Does it say how many? Why don't you say how many Cuda cores? Just says Blackwell generation. How many CUDA cores does the NVIDIA DGX Spark have? Okay, so coup de cause is—

why is my calendar opening up? Come on.

I think we're going to find that the ratio is basically just down to the number of CUDA cores. So 2.66, and then— but we also have some bandwidth here. Because yeah, the bandwidth is 273, whereas the bandwidth on here. Is there a full spec sheet here? I am here to thank you. I recently started my PhD in AI at University of Edinburgh, and in my deep learning journey started with your amazing PyTorch course. Oh, thank you so much. Oh, thank you so much for the course. Um, that's really, that's really cool to hear. Congrats on your PhD. I want a full spec sheet. I don't want this. Yeah, thank you, Arav Jan. Nvidia RTX 4090 memory bandwidth. There we go. Where is this from? So we get 3x the memory bandwidth on— or 4x, thereabouts— as well. So the 4090 is still a good card, like a really good card. Like the 5090 is about 30% faster. So to me, the 3090 to the 4090— the 4090 was a bigger jump than the 5090 is to the 4090. So let's, let's get Chad, ChatGPT, to, um, Create a table comparing the NVIDIA RTX 4090 to the NVIDIA DGX Spark across different AI and ML compute metrics. Research these to include sources. Okay, so we'll let that run for a few minutes. But now we need to write down a list. So I just want to go to the— let's write down a takeaway from this.

Several benchmark— main goal of this repo is to benchmark the NVIDIA DGX Spark first against the Nvidia RTX 4090 on a series of ML AI tasks. Why these two GPUs? They are the two GPUs I own, but the tests could easily be extended to other GPUs. Most of the analysis in was performed by Codex, or just GPT Codex, simply to compare the results in benchmark results. This holds the ground truth for a series of tests done in, in CSV files.

So now I'm just going to write a bit more about this summary. So let's write this.

Nvidia DGX Spark is like a minivan, whereas the Nvidia RTX 4090 is like a Ferrari. What I mean by this is that the is that for any compute-intensive task, the RTX 4090 is generally faster than the DGX Spark on an order of 3 to 4x.

Good itchy eye.

Um, however, there are several benchmarks, e.g., running, where the RTX 4090 simply can't because it doesn't have enough space.

VRAM. So, yeah.

Can run much larger models.

Performance. What is the summary for training models?

Same thing, faster on training and inference. One line: RTX 4090 faster on training and inference by 3 to 4x, but cannot run larger models like the Spark. 24GB VRAM versus 128GB VRAM.

Setup.

DGX Spark is one plug and play. Go from unboxing to running models in 30 mins, especially with, um, Nvidia DGX Spark website, tutorials. Uh, RTX 4090 requires a custom-built PC. This took me and my friend 4 to 5 hours— 3 to 4 hours actually— as well as, um, software setup. 1 to 2, or maybe 2 to 3 hours. On your PC build, we'll have a much larger footprint.

So DL gradient boosted trees assemble ensembles. You could probably on the Spark— I actually haven't done those tests. That would be an extension of my own personal benchmarks. But you can use CUDA X for data science, but these run, I believe, on So here we go. QML, popular compute-intensive machine learning algorithm in scikit-learn. So linear SVC, UMAP, HDBSCAN— I don't know if any of them relate to what you're doing or if you use any of those. Uh, accelerates operations for data preparation and core data processing of 8 gigabytes of string data with no code changes. So, um, gradient boosting trees with CUDA. Yeah, XGBoost has Kuda support. So again, my guess is you'll probably see the same, same results. That's my high-level guess. Nice! Cool, let's do a benchmark breakdown of what we tested. I want to write— actually, we've done this, we haven't done Transformers. They could be notable extensions. Notes and extensions.

LM/VLM inference. Compare LLM VLM inference speed with Llama CPP and VLM inference engines across various models. Um, LLM training Compare LLM, um, training speed with a relatively LLM full fine-tuning. And then we want image generation.

That how you spell it?

Klein?

What models we got today on Hugging Face? Step 3.5 Flash, Paddle OCR VL2, or VL1.5, sorry. Deep, DeepSeq OCR 2, Quen 3 Coder Next. Oh, was that just updated? I think we're going to see a lot of models in February. Wow, Quen 3 Coder next is on par with Claude Sennette 4.5, apparently on benchmarks. Of course, 80 bil— I think you could run this on the Spark. Oh, maybe not. 40 times 4, maybe the unslough version you'll be able to run that. Where's flux? We need flux. Flux to Klein 4B.

Okay, and then we got object detection. Train, fine-tune, and object detection.

MV FP4 should be fastest on the Spark, right? No, none of these were FP4, they were FP8. We can technically— I actually tried to run some FP4, but I had some issues, so Let me see what Codex has done here.

Yeah, that's a good point, setup.

Uh, yes, do I have a go-to LLM? Um, I alternate as well, so I guess I just vibe LLM depending on what I need. ChatGPT is probably the best for research. Gemini is the best for multimodal stuff and fast things like with Flash. And then, um, Claude is definitely the best coder for me, but Codex is doing pretty good. So let's, let's find out NVFP4 on the DGX Spark. It is capable, yeah, but I found basically found there was a lot of takeaways of like the setup of running NVMe FP4. I just wanted the simplest benchmarks. What are we saying? Okay, let's see what this guide says. 1-hour TensorRT model optimizer.

What do we get here?

I'm just going to put that in extensions. I'm happy with my benchmark so far. Could we do better? Yeah, of course, but we'll let, we'll let the YouTube comments tell us how much better we could do.

Extension: get better, get some numbers on NVFP4. In my research, brief research, it's still not as fleshed out as I'd like. See, example.

Unsloth brought Quen 3 Coder next down to 46GB in 4-bit and 85GB in 8-bit. Yes, of course they did. So let's go have a look at that. Thank you, Naveen, or Arav Jain. Um, Hugging Face, UnSloth. Who wants to test this on the Spark? We've done GPT-OSS 120 gigabyte. So let's try— I like this quantization, Q4. So we're going to download this in the background while we write up some information. Oh no. Okay, so that's going to take a while, and my internet might be fried because that's downloading. But we're going to test Quentree next in QuantEyes 4. If you want to see GPT-OSS 120B, here we go. DGX Spark can do token Generation, where is it? Output token throughput, 116 tokens per second. That's pretty fast. Here, this is with VLLM, by the way. Did I do it with Llama? We can run it with Llama. That's not 4090. This is NVIDIA GB10. 4090 can't run GPT-OSS 120B. This is on the NVIDIA GB10, which is the DGX Spark. So the output token is 116 tokens per second. And we can actually try GPT-OSS-TGUF. We're gonna try one model, so it's fast. It is fast. We're going to try Quentrecoder next with Llama and see how that goes. So what could we try with Quen 3 coder next? We need a good test. So let's get— oh, we'll just use one of our scripts and pass it that. So I want to update this README so we know Okay, we're waiting for that to download, and now I want to Let's do VLLM first. I am running. Okay, so it's running in a Docker container. I'm using whatever version of VLLM the, uh, NVIDIA Docker container 2512-Post1-Pi3 is running. So I'm running it through a Docker container. Is there a difference between VLM nightly or 0.15? Have they done Yeah, see, that's the hard part about benchmarking is I've just used a Docker container. I can't, um, I can't always guarantee the same, uh, I guess, versions across different tests and whatnot. So I've bench— I've pinned the Docker container that I used, and then that way this is more so not necessarily to be in It's more of an exploratory benchmark, not necessarily supposed to be like battle-hardened and intestined, but we're seeing the same results and probably results you would expect, which is good. As in, the RTX 4090 is faster at compute, but it doesn't have as much memory. So how's my download going? Nice. Okay. So let's now run a llama server.

Um, I want to go So we don't need multimodal projection here, but we do want coder I forgot what I saved it as already.

Okay, let's see how our server goes. So this is running Quen 3 Next Coder on the DGX Spark.

We'll wait for this to load.

We'll give it a few minutes.

If it doesn't work— oh no, we lost connection to the Spark. Oh no, offline error. So the poor Spark just, I don't know, shut down maybe. Okay, so we'll give it a minute to reboot. Let's see if the DJI Spark has come back to life. We're back. CUDA out of memory. Something's taking up a lot of memory on the Spark. Have I got another Spark window open somewhere? VLM. So I need to find a way to cancel the, the LLM engine or server I've got running, because see how you see that's using This is what I like Gemini for. That's the problem with the Spark GPU RAM is also system RAM, so the SSH hung-ups are possible. Yeah, you're exactly right.

That didn't work.

Still there.

Okay, it worked.

So let's now run our Llama server. Okay, that's much better. We're getting further already.

Um, write me a GPT-2 architecture and training loop. In a single Python script.

Wow. 40 tokens per second, thereabouts, output. You ever run Llama Benchy? No. Nice. That can be in the extension. Thank you for that. So let's create a new chat. And then go— let's convert this 200 lines of code into a Python script. Convert this Oh, llama bench. Cool. I'll use that next time.

I'm happy with the benchmarks that we've done it so far. Let's see.

How was this? So, Quen 3 Coder Next, we're getting about 38 tokens, 40 tokens per second, and about 800 tokens per second. So that was generation. And then on the prompt processing speed, we get about 832 tokens per second. Nice. So that's workable, very workable.

Okay, now write a step-by-step README to explain the usage.

This is excellent. Okay, so Quen 3 Coder Next is fast. I would say what, what's kind of the generation token per second that's like usable? I'd say like 30 per second is like minimal. And then it's faster than reading, that's for sure. But then we want for like If you could 3x this, that would be great. So 100 per second is like, that's— you're, you're cooking with gas at 100 per second. And then obviously faster is better at that, but like 100 per second I think is like a good benchmark of like, that's, that's where you want to get to. See, I could have done this in FP or Q4.

Wait, I've already got Quen 3, so this is Lama.

I don't need this. Okay, so we've done that. Okay, this is good. These are much easier to run. So DJX Spark is good at preprocessing, so would be great at ingesting entire GitHub repos. Do you also consider content length in your benchmarks? As in, so I've got context length as randomly varying and concurrency test for 1, 2, 4, 8 users. I've got, um, I just set it So a lot of the benchmarks I set to the default VLLM benchmark settings, so max concurrency 10. Um, that's— I haven't got, I haven't got benchmarks across concurrency for users, so this is mostly targeted at, um, I guess These benchmarks are mostly targeted at single user usage rather than multiple users at the same time.

For example, a single developer working on their own machine.

Does that make sense? So you can see all the settings that I used here and in the testing. So if we go to the VLM benchmark page I use most of these settings here. You would be able to support significant longer context lengths for a single user compared to serving 10 concurrent users though, right? Yes, of course. But I was just mostly testing like raw tokens per second on various different prompts. So context length, that's a good, that's a good extension. Louder Than Thunder. So what bench, Louis, what bench, or Lewis, what benchmarks would you, um, so you mean you would like to see a benchmark on how much can it handle, how does it perform when it's got 50k tokens in context or something like that? Extension.

Upgrade the amount of tokens used in context to see how the devices perform under higher load, e.g., um, large codebase in the context window.

So benchmarks at the— basically the Spark is if you want to— yeah, anything for— let's go, which one you want fastest GPU for fine-tuning/running smaller models equals RTX 4090.

You want to run large context large models, e.g., um, and want lots of context equals DGX Spark. You want to, um, create large amounts of synthetic data with large models plus distill, um, synthetic data into smaller models via fine-tuning equals get both.

So the specific use cases that Spark highlights— that provides for is large models. Lots of context. The specific use case that the RTX 4090 provides for is fast, smaller models. Does that make sense? PyTorch, Transformers. Now, where was I?

Okay, how we looking? So what do we need here?

Do I have a Mac Studio? Have I tried pairing a Mac with the DGX Spark via Exo to create a hybrid compute cluster? Uh, no, I haven't. I don't have a Mac Studio, I have a Mac Mini. So this is my setup, and then I've got an RTX 4090 down below. So in the future, maybe when the Mac Studio like M5 comes out Um, that might be worth looking at, but I know what you're talking about. XO Max Studio plus DGX Spark. Yes, a prefill is compute-bound. Decode is memory-bound. This is cool. I do have a Mac Mini with M4 Pro, and I do need to get into MLX. So MLX will probably be next. So this is just focused on NVIDIA. I'm going to be getting into MLX this month because I want to build something with, uh, an on-device, um, VLM. So yeah, MLX will probably be a future stream. I'll be right back. See you, Alex.

Thanks for joining in.

Yeah, Louie, that would be an awesome stream. Lots of people have Mac Minis now because of OpenClaw and would absolutely find MLX model benchmarks useful. Yeah, so that's probably— that's an upcoming stream for sure. Okay, we're starting to get somewhere. So now we'll do llama. Hey, Maguela, always a treat to see you live streaming stuff. Oh, thank you very much, that's very kind. What's XO got on here? How do I just get to their blog? 4 Mac Studios. What is the price of a Mac Studio? Right, if I am— if I'm in Australia and we want an M3 Ultra and we want the big dog, oof, 512GB of RAM is $6,000 AUD. So I could spend $15,000 on a Mac Studio and I'm still— I still don't have storage. I'd obviously want like 8 terabytes. $20,000 AUD. So what's that, $13,000 USD? That's crazy. But what you are getting though is 512 gigabytes of RAM. Now if we compare this price, this price is actually not that crazy with, um, how do we find it out? Oh yeah, RTX 6000 Pro by Oz, right? So if we wanted to buy this This is a 15K GPU, 96GB of memory. So 512, we actually have— where's our calculator? Let's just say 5 times as much memory. But then we have to— if we bought this, we'd have to whatchamacallit compare this. We'd have to— sorry, we'd have to build a system, and we could easily spend $5,000 on building a system to house this. Because like, what's the best-in-class CPU at the moment? Let's go AMD, and we'd want to— we'd want a beast CPU, right? How do I sort by Can we just get a 96-core, 16— maybe not that big. 16-core, $1,000. That sounds cheap, but let's just say that, $1,000. And then, uh, storage, we'd want 8 terabytes or 4 terabytes. So what's that, about $1,000 as well? And then RAM, we'd want 128 gigabyte RAM.

What?

128 gigabyte of RAM is $2,500. That is nuts. 64GB of RAM is $1,400. I didn't know it was this bad. Honestly, this price is not that outlandish at the moment now. Seriously, like, you're not going to get H100 speeds, but what's it— what does a H100 cost? Which you probably— we can't even buy. Oh, 6. That is nuts. I saw a funny tweet. Thanks to AI Slop. RAM prices are bad. Yeah, I saw a funny tweet the other day. It's like a cat in a gym. Just on like a rowing machine, just going, right, like terrible video. But the caption was the funny part. It's like, RAM is $3,000 because of this, and it's just a cat in a gym, some AI video. I'm not, I'm not a fan of AI videos. Sora, never downloaded the app. I just saw that as like the ultimate, the ultimate level of brain rot. And look at this, this is beautiful. But yeah, once you get— once you start adding things up, right? And then a case. I'm keen to see the M5 Ultra. What are the rumors on that? Mid-2026. You reckon they're gonna, um, bring out M5 Pro and M5 Max to launch imminent? Should I do another MLX Uh, sorry, not MLX. Um, oh yeah, MLX would be a big benchmarking video when the M5 comes out, M5 Pro and stuff. I should do an MLX stream series. Apparently it's coming soon, Tuesday, February 3rd. That's— that was yesterday, but this is American time. Okay, so the Mac M5 new updates are supposed to come out with the new Xcode 26.3 or whatever. That's crazy. I'm sticking with my hardware. I think we can go pretty far this year with the hardware that's here. Yeah, MacBook Pro M5 Pro supposedly next week. Honestly, my M1 Pro is going fine for day-to-day usage, right? Like video editing, everything that I do. But the only reason I would be interested in an M5 Pro— I mean, I've got an M4 Pro, I could just use that— is for MLX now, because MLX is like— I haven't tried it enough hands-on But it's like all reports I'm seeing is like it's getting, it's getting real good. So let's have a look at our README now and write this down. TK note: this table is AI. Generated by ChatGPT.

I will sort out soon, but after looking and comparing to actual specs, it seems pretty legit.

I'm gonna move the benchmark overview to the top. How do you do a note in GitHub README? Okay, I think that's pretty good. Oh, I've got a spelling mistake in the title. That's how you know this is not AI written. A lot of it is, but this— a lot of this isn't. I made this. So, and then we go benchmark analysis.

Beautiful.

Okay, now let's think about charts or plots that we want to compare. So a cool thing that I was making before with Codex was So I've got the Codex app running locally, by the way. It's pretty good, but I want— I want to show you this video. So I want to create some of these graphs. What do we think of this video? Okay, very simple, but this is the type of video I want to make. But I think it's only comparing— it's only comparing a one benchmark. So I'd like to turn this into a vertical but also collate the VLM. So, uh, keep the blue and orange, but then on the bottom, uh, so that'll be the key, and then vertical bars, tokens per second over here, and then the model will be down the bottom. So let's, let's write that out, because based on this README, I want Let's commit this. Why is tab not working?

Cannot run larger models or, uh, Very long context.

So we have 4 benchmarks with 2 inference engines. So we want—

there are 4 benchmarks. LLM, VLM inference, um, LLM training, image generation, object detection, and we want to visualize the results. Easily. So plots we want, one will be So we want, um, inference engine via LLM metric to plot token throughput per second.

Y-axis— oh, this is good—

x-axis equals model name, and then we go, um, key blue, RTX 4090 blue, um Oh, maybe that's, that's orange, so I get orange, blue, matplotlib colors instructions.

So I think that's what we want. Let's see if we can create this. Let's see what happens here. We'll let that run in the background. Let's see if we can get them to run some data analysis for us. Nice! That is beautiful. So we're trying to make some animations now. Let's see what this looks like.

Wow!

These aren't that great, are they? Hello there, thanks for the fine-tuning video. I'm fine-tuning Liquid 1.2 right now. Oh, that's so cool! How's it going? I don't think that output the new one. So that looks good, but I want the text to be— for some reason the text isn't that high quality, and the bars Let's try. Gathering, generating synthetic data right now. It's for my N8 agent. To— for Gmail to sort all my 20K mails. That's awesome.

Update script 2 to use the same text styling as Let's see if this works.

I'm not familiar with the animation library, um, Manim. I've only seen it. I like it. But did this actually work? Did Claude actually change anything? That's better. How do we like this plot? I think that's good enough. I just don't want it to disappear at the end. That's awesome. Okay, let's see, how do I stop the animation from— oh, it's got a fade out option. Let's just see what happens, eh? So we'll make fade out optional. And then we'll create— you got to start your own repos, okay? There we go. Let's see if this works. Okay, that's great. So we want— that's VLLM. We want to create our other plots. That's our reference script. Let's see if it can do for the input. We've got the output. So let's see if this is correct. If we go to benchmark results, did it extract what we want?

Is this number in here?

No, it's not. So does VLLM measure I'm gonna update the report. Let's look up VLM bench. So this number is this number minus this number. I say.

So we're just going to save this to here for a second because I want to add columns. Let's get Codex to go So we're using codecs here.

Yeah, you've got— you can do it here now. I've got the code live.

Let's see if this looks correct.

Yo, Westsider, how are you? We are, we are going through benchmarks. So we're looking at the results of how our models have performed and trying to make some animations. So once we get this— no, this is not for a course. This is just for a YouTube video. So I'm trying to collate all the results here and then I'm going to make a YouTube video putting them to the test.

I don't like the spacing there.

What's your opinion on vibe coding solutions for building apps and solely focusing on good models to solve business problems? These vibe coding apps are actually good for building UI if you're an independent app developer. Oh, for sure. I think just use the tools that you have, but just remember you're always responsible for quality, right? So that's what I'm doing. I don't know I don't know manim, but I know what I want. And so I'm refining this iteratively until I get it. Like, so this is pretty good to me now. Like, this looks pretty good. What do you think of this visualization? I'd be happy to keep this style now. So yeah, I think like just— I think people overthink it. Use AIs if they're good for your use case, but don't expect to— like, I know for a fact I don't know Manam very well, right? And I'm not learning anything by just copying what the AI does. But if I wanted to, I could just re— go back through all this code and write it myself. So the bar to starting is much lower, but the bar to learning is still the same as what it's ever been. So I want to move this down a little bit. See, now I'm just playing around with these animations.

I don't know if it's going to work.

Okay, there we go.

That looks good to me.

Nice. Okay, so now we want Scripps to do So I'm liking this at the moment. Now I'm going to copy these scripts into Now let's see if we can get Let's see how this goes. When you run Llama versus VLM for GPT OSS 20B, do you use the same top K P for both? And that's a great question. Let's find out. So that would be in the README. So Llama. No, I use different settings, I believe. Whatever the defaults are with VLLM, I believe. You can see all the testing on GitHub anyway if you want to check them out. Yeah, so the code will be here if you want to check it out. These are the scripts I used. So it's probably— there are probably settings in here that, uh, this is more of an exploratory, um, benchmark rather than— I guess if I was to do it really hardcore, I would spend a lot more time making sure all the settings are exactly the same across it. But this is more just a Okay, let's see what we can figure out. Sweet, I'm starting to be quite happy with this. I think it's time for me to have some lunch, but So I'm going to end this stream there. Um, thank you all for tuning in. I'm happy with the benchmark so far. We could take this further, but I want to move on to different projects, to be honest. And it's kind of got to the point where it's like, okay, I know what these, um, uh, devices are best useful for my certain use case. So If you want to see any benchmarks in the future, leave a pull request or an issue on the GitHub or something like that, or leave a comment on this video wherever you're watching it. So I'm gonna— my next step is I'm gonna turn this into a YouTube video, and then that way it'll be a bit of a breakdown of what's actually here. But this is just— the streams were essentially to just stream them live with different codes. So, um, yeah, sorry, stream them live so you could see me running, actually running the benchmarks. But what is Codex created here? I like it. I'm having fun with Codex as well with this running live. This is— I guess this is kind of like Claude Code, right? Let's see what happens. Nice! Okay, so I'm going to clean these up. Yeah, that's a good idea, but that is for another time. I'm gonna go for a walk, have some lunch, and, uh, yeah, I'll see you on the next stream. Hey, thanks for tuning in, and wherever you are in the world, have an amazing day and happy machine learning.