Benchmarking: NVIDIA DGX Spark & NVIDIA RTX 4090 | Part 3 - Running LLMs on the RTX 4090

Hello, hello. Just waiting for this to refresh. Hopefully my internet is okay. Should all be hardwired now. I've changed my setup around. We'll put a yo yo yo in the chat. Okay, good, that sounded good to me. All right, lovely. So if you're joining in, please say hello and let me know how the audio and video sounds and looks. So what we're doing today is we're going to benchmark the Spark and RTX 4090. Now I've got some tests running— or not running, sorry, but if we go to GitHub, I just added a repo called GPU Benchmarking. So we have a bunch of tests in here. So there's, uh, Llama CPP, uh, LLM Fine Tune, Object Detection, Z-Image Turbo, and so that's ImageGen, and then VLM— VLLM, sorry. So I'm installing— I just ripped out WSL2 or Windows from my deep learning PC. Oh, I forgot that the camera's up here this time. I just ripped out Windows and WSL2 from my deep learning PC. So now it's just a fresh install of Ubuntu and already it's running smoother. So I had Windows with WSL2, which is Windows Linux system or something, Windows Subsystem for Linux. And it was just so buggy, like the Linux system worked, but Windows was just incredibly annoying. So I'm very happy that that is done. So now we've got the RTX 4090, Daniel Deep Learning PC over there. NVIDIA SMI. So there we go, RTX 4090. And over here, these are all running locally, by the way. So NVIDIA SMI, there's the DGX Spark, which is an NVIDIA D— or NVIDIA GB10. So let's check out some benchmarks. What are we gonna do? Yo, who is that next to me? That's my brother, he's in the corner of the stream. Josh can wave, this is a camera up here. That's my brother, so he is an iOS developer. So he's working on an app for a client. Um, we've got a small little office here that we run together. So, um, yeah, he's just going to be coding away and hopefully not too annoyed by me talking out loud here. So if you want to see what tests we're running, there's GPU benchmarking here. Now this is pretty messy. But, um, I'm gonna clean it up over the next day or so, or in today's stream as well. But today's main goal of the stream is to run the LLAMA CPP tests on the RTX 4090 and the VLLM benchmarking as well. So I wonder if I should run it with the Docker container. So let's look up how to install Dock, Dock, Docker on Linux Ubuntu. Oh, James, hello! Alex, what's going on? Chop chop on DMS, Josh. I think, I think James is in the stream. Josh is locked in, James, don't worry. What? James BB411 says chop chop on DMS, Josh. He's working hard, don't worry. I wonder if I can just run VLM without Docker. All right, let's see if we can run VLLM without Docker first, and then if we need Docker, well, we can, we can get that later. So llama cpp, which one we want to do first? I'm gonna do Llama CPP. So we're testing a number of different large language models here, but the thing is, the benefit of the DGX Spark is that it's got 128GB of memory, so we can actually run much larger LLMs on it versus the RTX 4090 is— has incredibly, like, much more horsepower, but but not as much storage. So a good way to put it would maybe be the RTX 40— I keep forgetting that I've just rearranged the cameras up here. The RTX 4090 is more like a Ferrari, right? It's fast, um, but it doesn't have much storage. Now the NVIDIA DGX Spark is like a minivan— plenty of storage but not so fast. Wow, that's a great analogy. I'm going to write that down. So that's how I can explain them. So analogy for DGX Spark versus RTX 4090 equals RTX 4090 is a Ferrari. Fast but not much storage. And then DGX Spark is like a minivan— plenty of storage, not that fast. So let's run, uh, Llama CPP first because I just installed that. And by the way, I'm just doing it versus via the docs. Right. So, VLLM. Let's see if my own notes can help me run this. Okay, so I need to run a server. Nvidia actually have some good documentation on setting up Llama CPP, so they cloned it. Ah, I see. So let's download If we go to Quen 3 VL on Sloth, we're going to do Quen 3 VL 8B because we can't run a larger model. GGUF. So this is how we run a GGUF. Model file. Now this could be a lot better automated, but I'm just doing it sort of a hacky way. We need— for images, we need the multimodal projection, so in, uh, bfloat16.guuf. And to keep things consistent, I'm going to use the Q4 quantized 4 version, so we'll get that as well. And then we can run hf_download, and then we pass in this model path. So that's Hugging Face download, and then we want this one. And then I want to go to local dir, and then I'll put it in models, uh, Quen 3 VL 8B. There we go. Now let's see how we run Llama CPP. How to run local server. They have really good docs. So I need to do it from source. Okay. Let's get Claude to write some, um, create me a series of steps to install, uh, okay, beautiful. We've got the GGUF file. Now we're going to update this to get the multimodal file. This allow us to do images because right now we've downloaded the text-only version. So we'll get the multimodal projection layer, which is this. Too many GGUFs. There we go. Can I just run the server now? Okay, cool. So we should have So, unslav have a good guide on Qwen 3 VL. I want to use port 30,000 as well. Just to keep things simple. So now we're going to run a multimodal server. Now I just want to show you, like, Nano 3— I'll put a note here. Note: requires— even if we go to this model, I don't think we can run this on the RTX 4090. Yeah, even in quantization 4, it's still 22.8 gigabytes. So that's one of the limits of the RTX 4090. So let's see if this works. What do we get? Okay, I need to compile. Actually, I'll put it into code. So I just did a fresh install of Linux on my RTX 4090, so that's probably why I have to Kudo Toolkit not found. I probably should have installed all of this before the stream, but at least we can see. 7 gigabytes, my goodness. Yeah, Llama Server is what we're using. Okay, beautiful. We've got CUDA Toolkit. So this is the docs for the, um, NVIDIA DGX Spark. And that worked. What other playbooks do we have here? Fine-tune with PyTorch, specular decoding. Jax, nice. Does Jax have a V1.0 yet? There we go. Okay, this looks like it's got better progress. So I learned a cool thing the other day that there's a lot of different models in the vision language model space which use, like, for example, where is it? Light on OCR was the one I looked at. This one. So if we look at the paper, does it have Vision Encoder? No. Architecture? No. So while this is installing over here, if we look at the architecture So it's a 1B parameter vision language model of 3 main components. So a vision encoder, a multimodal projector, and a language model decoder. We train the model to perform OCR without task prompts at inference time, so the extraction behavior is embedded in the weights rather than controlled by an explicit prompt. So it's just Page in, OCR out. So we employ a native resolution vision transformer initialized from the pre-trained Mistral Small 3.1 vision encoder weights. This encoder handles variable size images while preserving spatial structure, which is critical for documents with diverse aspect ratios and fine topographic details. So then they bridge the vision encoder with a multimodal projector, 2-layer MLP with Jell-O activation. It turns the vision features into the language model's embedding space before projection. So we apply spatial merging with a factor of 2, effectively grouping 2x2 patches, and while reducing the number of visual tokens for X. This keeps the overall token count traceable for— or tractable for high-resolution inputs while preserving sufficient spatial granularity. Hey, there we go, built the server. The projector is randomly initialized and trained from scratch. Nice. And then the decoder is QuEN-3. I like it. I wouldn't mind replicating this type of, uh, VLM for my own tasks because a lot of the pre-trained knowledge is in the backbones there, isn't it? Okay, so we've got our— let's see if we can run a server. Hey, build llama server. Now the next command they use here I don't think I have flash attention though. Nice, there we go. Okay. So we've got Llama CPP running. So write me a detailed recipe for a beef ragu and spaghetti with garlic bread, um, entree. Return in markdown. Oh my gosh. That was fast. Generation speed, 142 tokens per second. So Maybe port. Now let's try it with an image. Wow! Okay, let's get a PDF. Prompt processing speed, 2,000 tokens per second. Output speed, 126 tokens per second. So let's try— let's do an experiment of that on the exact same model on the DGX Spark. I'll do this on a different port. So— This is the DJI Spark. I'll delete it so it's not I'll get a new chat so it's not in— bro, Hizzle, how you going, my friend? Okay, so this is DGX Spark on the left, 30002, and this is the, uh, RTX 4090. Let's compare. So 3, 2, 1. Wow. Okay, so this might have been cached. Wonder if I can clear the cache. Let's just get a new PDF. The spark's going good, but what's another paper we can use? Hmm. I need a text dense paper. What paper did I like reading the other day? Malmö 2. This is a great model. So we've got the Spark running here, and on the right we have the RTX 4090. So let's compare the pair. Which model did they use here? The Connector MLP uses the same intermediate dimension as the LLM. All models use the SigLIP2 shape optimized 14384 Vid. What do I like about Quen versus Nemo Tron-a-Lama? Uh, Quen has vision inputs. So let's— so this is a lot of text. Let's just see how it goes getting all the text from this image. Like, this, this is all text, I would say. I don't know, 300 words maybe. So this is not cached. Same model here. Yeah, Quinn, NemoTron, I haven't tried enough, but NemoTron Nano 3 is a really good model. Um, and it's fast. Problem is I can't run it on the RTX 4090. So let's compare. We've got DGX Spark over here and we got RTX 4090 here. So we go click, go, click, go. Okay, something may have frozen on the RTX 4090. Okay, so that took 30 seconds on the DJI Spark. I'm not sure what happened here. Let's just try again. I'll stop the server over here. Wow, so there we go. We get about— so the DJI Spark is 619 tokens per second reading, but the RTX 4090 is 1,900 tokens per second reading. And then generation, we get 130 tokens per second, and on the DGX Spark we get 35 tokens per second. So about 4x faster— 3x faster reading and 4x faster generation on the RTX 4090. Same model, same prompt, same input on vision as well. The reason I do OCR is because you kind of have a ground truth of like you're expected them to both output the same thing. That's how we can kind of control the generation output. GPT-OSS tokens per second. I haven't done GPT-OSS. I wonder if I could run both of those on— should we try GPT-OSS? Let's see. Unslough, uh, GPT OSS 20B. Oh yeah, this is why I didn't run it, because it had a bunch of processing issues. Just gonna get Claude to merge all these together. What's up, Nick T? If you had to choose between the Spark and a 4090 for dabbling and fine-tuning, which way would you go? Well, I'll give you these. I'll get— I'll get you some raw numbers rather than just— and you can decide. Object Detection fine-tuning here, LLM fine-tuning as well. So these are both on the RTX 4090. I'll put the RTX 4090 on the right and the DGX Spark on the left. So we'll go LLM fine-tune— where is that? And then Object Detection fine-tune. So LLM bench. Okay, so we got LLM fine-tune RTX 4090. And, uh, sorry, LLM fine-tune. This is Gemma 3, 270 mil, on 1,000 or so samples. And the total train time on the GB10, which is the NVIDIA Spark, was 343 seconds versus the RTX 4090, 81 seconds. Total train time, 5 minutes. Total training time, 1.3 minutes. So in terms of training, the RTX is like 4 times faster in just raw speed, but the trade-off you're going to get is you won't be able to run as larger models on the RTX 4090. So if you want to do some LoRA fine-tuning as well, you could do that on the RTX 4090, but just beware. So in raw compute power the RTX 4090 is better, but in memory, uh, the DGX Spark is better. So yes, you can still run experiments on the DGX Spark, uh, but just know that the raw compute will just take longer. But oftentimes, like, for me, I don't think this is that bad because what I would do most of the time is spend a lot of— or most of the time that I had during the day, I would spend that preparing a dataset to be used. And then I would run that, the training or something overnight. So this time is actually— it doesn't really matter because you can just run that while you're away from keyboard or something like that. So let's see if we can try What Unslaught models do they have? GGUF. Oh, here we go. Hello, Amir. Okay, while this is downloading, I'll be right back. Oh yes, so, um, the Spark— another benefit of the Spark is, yeah, it is plug and play. So if we look at stream, this is my setup, right? So Mac Mini, Spark. The Spark is basically the size of a Mac Mini. A little bit, a little bit bigger footprint, but you get the point. You can just plug that in, you can connect to your Spark, um, just plug it in via Ethernet or something like that, and you're off to the races. Like, it's basically 30 minutes from zero, from plug-in to running and training models could be 30 minutes. It's really that simple. So big benefit to the Spark there. Whereas my deep learning PC, I basically spent all day yesterday, um, cleaning out Windows from it and setting up, uh, normal Linux because, uh, Windows was just giving me so much pain. Okay, let's see if we can do this. So we're on the RTX 4090 there. We got GPT-OSS. Write me a detailed recipe for a Beef Wellington in Markdown. Wow. That is cool. So 1,100 tokens per second processing speed and 248 tokens per second generation speed. I hope you— I hope God gives you everything you want in life. I really appreciate your hard work and courses, especially your TensorFlow course. That was nuts. Oh, thank you so much. That's very nice to say. So let's do the same thing except this time on the DGX Spark. So GPT-OSS, there we go. I can't run the 120B version on— so excuse me, I need to go to Visual Studio Code output. So we're going to download Download a new model to— or GPT-OSS-20B to DJI Spark. Because now we got scripts here. Yeah, text only. BenchLlama.cpp text only. So we'll run this in a live server next. On the DGX Spark, and then we'll see, we'll see how they look face to face with the exact same prompt. And then we will— I should really move the RTX 4090 over here. And then I can bring that over here. Okay, RTX 4090 from now on is on the right and DGX Spark is on the left. I'm very impressed by this. What models do we have today on the Hugging Face page? Step 3.5, int 4. I haven't done enough with SAM, SAM3 lately. Alpha Genome, all folds. Have any courses in the works, mate? What do you mean? Do I have any courses in the works? So I do. I have— these are on the channel already, but I'm going to formalize this a bit more. So where's my— so we've got a couple new projects here. LLM fine-tuning, LLM fine-tune, a VLM build, a multimodal RAG pipeline. I'd like to turn those into a course, but I'd also like to create like an LLMs from Scratch course and a VLM from Scratch course. So yeah, we'll probably stream some stuff for that. I've got some streams planned where I'm going to build LLMs from scratch, and then I'll turn that into a, a video or a course or something like that. So now we want to run a server. I want to write this down first before we lose what we've run. But yeah, let me know in the comments if you'd like to see any— anything in particular. But I want to try more streaming this year, see how that goes. That's my experiment for the year, is actually just getting hands-on with all these new tools, right? So now I run a server on the Spark. Okay, this is on the DJI Spark. So we're going to just do the same prompt. Okay. See, this is still fast to me. That's incredibly fast. Wow, so we have— wow, so this is a big difference, right? The, the RTX 4090 just has the raw horsepower. Like, look at that, 294 tokens per second. I mean, it didn't take long because our input prompt was not large, but it's 4, 4 times faster, or thereabouts, 4-ish. So 1138 tokens per second processing speed versus the Spark, 294 tokens per second. And then, uh, generation speed is 2.5 times. But now we can run some benchmarks. Let's see what we can do. But notice how the output is slightly different as well, right? Why don't we put a big input prompt in there? Let's just go, um, Wikipedia and go transformer architecture. All right, and we'll just copy all this text, put a big prompt in there. I wonder if— actually, I wonder if this can even handle this much. Um, extract the key points from this article and summarize it in 200 words. So this is a big input prompt. And then a controlled output prompt: extract the key points from this article and summarize it in 200 words. Okay, DJI Spark, RTX 4090. Oh no. Okay, so we need a bigger context size. That's okay. I'm going to increase the context size to 32K. I don't know if this will work on the RTX 4090, but let's just see what happens. So Nick asked a good question. Oh no. Nick asked a good question. The previous streams have been amazing. Oh, thank you so much. With regards to gathering datasets for fine-tuning, is it acceptable to use synthetic LLM-generated data, or does that introduce risk and quality? So I would have said that yes, like a year and a half ago. But now I'm— I actually want to do a series where I do a lot of like, um, that can be another live stream in the future, is creating a dataset for LLM fine-tuning. And, uh, I believe now like you, you, your data mix should be quite broad. And what I mean by that is yes, you want— of course you always want real data, but these days the synthetic data that you can create is also high enough quality to use in your model. And so these are all levers. I wouldn't worry too much about degrading quality. If you're starting from a good baseplate and then the synthetic data you're feeding in is aligned with the problem you're trying to solve, you can definitely use synthetic data. I actually think like the, all the unlocks for this year in terms of data will mostly be from synthetic. And by synthetic, that could be you just go, hey, here's this text and you get the LLM to rewrite it like 4 or 5 times into different ways. And now you've got diverse input data. That could be one example or just JSON. You might have an example of where you want a— here's a good one. All of the recent OCR models are trained with synthetic data. So what you can do, large amounts of synthetic data, actually, you can create, say, a PDF generator. And what that does is it automatically creates different PDF layouts with known input text. And then what you can do is because you have the known ground truth of that PDF, you can turn it into an image and then get the model to reproduce the text from that because you know it, because you just put it into a generation engine. And what I mean by that is that could be like a hardcoded Python script which just creates like random tables, random paragraphs, random headings, in different locations around the PDF. And so then you fine-tune your— or train your dedicated OCR model on the exact text it should read from that. And then that's been shown to generalize quite well to the real world because at the end of the day, many PDFs are quite similar layout. Like it's— yes, you get many diverse layouts, but if you have a synthetic data engine, you can generate a million PDFs. In a couple of hours. And then that's a good base dataset to train on. So you can use that sort of mindset to transfer to your own target problem. So let's go test this now. We'll refresh our server. So extract the key points from this article and summarize them in I'm typing in American with a Z in summarize instead of, uh, S, which is Australian. So again, DGX Spark, and let's send this off. Enter. And then this is RTX 4090. Holy smokes. Okay, right, not the exact same outputs. Yeah, the variance in the data is what the synthetic bits can cover. Yeah, you're exactly right. So the prompt is 25K tokens, and we get a reading speed here of 3,500 tokens per second on the DGX Spark, and then on the RTX 4090 we get about 2.5-ish, so 9,000 tokens per second thereabouts. And generation speed is 208 tokens per second versus 71, so again about 2.5 or closer to 3 actually on the generation speed. 3x output. This also outputted more tokens, something to note. So I hope this exemplifies the— now, of course, we're not, we're not doing quality assessment here. We are just strictly doing raw benchmarking on tokens per second. So quality would be another test entirely. We are just doing raw tokens per second here. So clearly there's a discrepancy between the RTX 4090 and the DGX Spark. The benefit of the Spark is that we could run GPT-OSS 120B, albeit it would probably be slower, but it will actually run, whereas on the RTX 4090 we couldn't run that model. So this is a pretty clear case of, um, they actually have their strengths on their own, but then if you would— if you had to pick one, well then it depends on what your, uh, use case is. So for my use case, I'm personally probably going to use the DGX Spark to generate synthetic data for my use cases because it can run a larger model, and generally the larger model is going to be better on in terms of just quality, right? So my specific use case, what I'll do is I'll generate synthetic data on the DGX Spark. But what I want to do with that synthetic data is to fine-tune a smaller model so that I can deploy that for really specific tasks. I don't— I'm not interested in building like generalist models. I want to build specific models for specific tasks. So generate synthetic data on the Spark with a large model, the largest model that can fit into memory, much larger than what can fit on the RTX 4090. and then I'll take that data and fine-tune a small model with the RTX 4090, which will be faster at doing that. And then that'll be my loop. My personal loop is data generation on the Spark, training on the RTX 4090 for a smaller model, then deploying that smaller, smaller model. So for example, one of our goals is to replace Gemini in our app this year. So instead of using Gemini for these predictions, right? This is the app my brother and I build. So Josh is sitting right there. He builds this iOS app. But I want to— we use Gemini on the backend to identify foods, but I want to create a custom VLM that we can deploy and just run it on device. So then we have no inference cost and we can just— yeah, we don't have— the app just essentially is a piece of software now instead of hooked up to the Gemini API. Now, of course, don't get me wrong, the Gemini API is very handy and easy to run. But if you're like me, you kind of like making your own stuff and deploying that. So that's our goal for the year, or one of the goals. Like, we have many things we want to do. So let's now run. Okay, we're going to keep our server running, actually. So is this on port 30,000? And this is on port 30,000. Okay, so let's see, let's focus on the RTX 4090 for now because I have— I create a new terminal, conda activate ai. Let's see if we can run our benchmark script. Ls cd gpu-benchmarking. So I think I should just be able to run this. Oh, excuse me, Python. There we go, nice. And then this is the benchmarking script that I've got. So we've got our RTX 4090 over there. Let me just clean this up. What it is, it's just going to run, uh, 100 or so prompts on GPT-OSS-20B, randomize things, right? But the, the input tokens are going to vary randomly between 50 and 1,000, and the output tokens are going to vary randomly 50 and 1,000. So they're all just synthetic, but we're just seeing how fast can we get input and output tokens. This is quite quick. And then we should be able to just run the same test on the Spark. That's my goal. So as you can see, we've got, uh, preprocessing is about 6,000 tokens per second and generation is about 250, which is quite close to what we used— saw here. So I guess reading speed will go up when we have more tokens in the input prompt. We're capping our inputs at 1,000, by the way, for the benchmark. And then the output is around 200 or 250 or so per second. So Gian Carlos asks, is it possible to run the GLM 4.7 full model? Let's have a look. So do you mean Flash? Oh, what the hell. Flash or, um, non-Flash. So GLM 4.7, yeah, this is probably too big for the Spark. 358 billion parameters. So if you just look at the number of files here, yeah, 92— what's that, 92 times 8? So about 720 gigabytes or something like that. So that's, that's far too big for the DGX Spark. But then if we— what about the quantized version? So if we go, um, Unsloth GLM 4.7. So even GLM 4.7 in the— um, if we go UnSloth GLM, here we go. So they've converted it. So even if you convert it to Quantize 4, yeah, it's still, uh, what's that, 5 times 50, 200 and— oh, 206 gigabytes in Quantize 4. That's still, that's still too large to run. So you could definitely do GLM 7 Flash because this model is about 50 gigabytes in FP16, but then if we go GLM 4.7 Flash GGUF So Quantize 4, we could probably run that in 17 gigabytes. We could probably run that on, on the, uh, RTX 4090 as well. So yeah, so Nick, the goal for getting the model running on iOS would be to have it just in the app. So then when you download the app, it comes with the model. Or maybe the model would be like one on the setup. You click download because the model might be like a gigabyte, for example. And the app bundle, when you publish to the App Store, has a limit on how large your app can be. And so what you would do, you download the app, which would basically be like a skin, and then you would have the model file, say, hosted on Hugging Face or something like that. And then you would click download and then the model would download to the device and then it would just run on the device then. So there would be no API calls. So you could even turn on airplane mode and the inference would still work. So if we go to, um, MLX Swift examples. Yeah, here we go. So you— we've converted to MLX and then this is maybe a future stream. This is running on, uh, directly on a Mac app. So MLX is the format that Apple use for running— it's like PyTorch but for Mac. And so then they would— we'd convert it to MLX and then it would run on iOS and macOS, all Mac platforms basically. So this is why I think Apple are kind of like the dark horse in uh, AI world because they're building out MLX, and that just means every open source model that comes out can just run natively on your own hardware. Like right now, what I'm going through is a little bit of advanced setup for this, but you've got apps like LM Studio, which is just releasing, um, LLMs that you can just run on your computer. Locally. So my, my goal for my upskill for the next few months is to work out how to create really good, um, VLMs and LLMs that you can just run locally on your own apps, for example. Okay, so we did our benchmark. Now let's compare. So we got about an average of 6,000 tokens per second throughput. Which is crazy. And then generation throughput, 6— 242 tokens per second on the RTX 4090. Let's compare that same model on the, um, Spark. So I said the RTX 4090 was going to go on the right, but we can just go a new terminal here and conda activate ai. We'll cd ls, cd into gpu-benchmarking, ls, and then we should just be able to do— there we go. So now we're running the same, same benchmark on the DGX Spark. So we've got our output, and we do, we do 10, 10 for warm-ups. So that way we sort of, I don't know, we don't include the first 10 to just give it a sec to warm up. GPUs generally like to warm up. Yeah, so clearly we're getting slower outputs on the DJI Spark. Let's get, um, Chad, can you research the specs of the, um, DJI Spark? And the RTX 4090 GPUs and compare them line by line in terms of all relative, uh, tech specs for AI/ML. Return results in a table with a column for explaining what the particular spec is valuable for. Have I tried NVFP4 models? Not, not enough. I have tried, um, FP8 models and they work well. I've tried Quantize 4 models, but NV— NVFP— I can't even say it. NVFP4 is probably what I would have to— if I was to keep running these benchmarks all the time, I would probably, yeah, do some more NVFP4. But for now, I'm just doing— I've got a list of— I've got to tidy this up, but I've got a list of different models that we're running in here. So we're running GPT-OSS. This is— which version did I get of this? This one. Oh no, I don't want to update this because— oh, I've got too many computers running at the same time here. So let's run another benchmark on the RTX 4090. We're gonna run quen-vl, and we're gonna run a Python Let's do image and text. There we go. So let's see how long this takes. Yes, I'm doing FP8 because, uh, FP4 is not supported on the RTX 4090. Yeah, so CUDA cores, we've got nearly 3x Memory bandwidth. Wow. Okay, see you, Alex. Thanks for tuning in. You have a good night. So next will be VLLM on the RTX 4090. I wonder if I can do On the app, is there another model involved in judging the volumes of the foods for calorie macro estimations? Or you're just seeing that bit being RAG? So if I go to here, so we just basically prompt, we keep it as simple. This is going to be as simple as possible. We just prompt Gemini to estimate weights and servings. So we just get Gemini to output structured data and then we display it nicely in the app. And then on the app we have a database of about 1,070 foods and then we just key match the output from Gemini to our database of foods. And then we have the macros already pre-calculated on. So the response from Gemini is only a JSON of title, information, weight, and servings. So then on device, we hook that up to a database and then we display the macros based on like what a pineapple generally is. So we're focused on Whole Foods. We don't do barcodes because, um, I mean, Nutrify's philosophy is to— whole foods are best. I mean, that's my background. I studied food science and nutrition, and basically what I learned of a 3-year degree plus like 10 years of practicing it is that just avoid eating crap. And the best way to do that is to just eat things that don't come with barcodes, which is not to say I'm not crazy about it. Like, of course I eat foods with barcodes barcodes all the time, but they are generally more, much more processed than Whole Foods. So in a nutshell, yeah, we get the simplest prompt back we can from Gemini or simplest return back from Gemini, and then all of the number crunching gets done on device because it's just a lookup in a dictionary basically. But you could in practice, if you wanted to, just go from straight image to calorie estimation out. We just get Gemini to estimate weights and serving size and then map those values to a database of macronutrients. So why do we do that? Because one, pricing. I mean, the more tokens you're generating, the more it's going to cost. So if you're generating macros every time, then that's going to take cost of just sheer volume of tokens. And then 2 is latency. So for every token you have to generate, that's a few milliseconds, right? And so if you're generating, I don't know, 1,000 tokens instead of 200 because you have to generate macros for every food, well then that's— that could potentially take 2 to 3 times as long to do versus ours, which we try to keep quick. We want it to be fast and we want it to be even faster. That's why we want it to move onto device. Okay, there we go. So we've done some benchmarks now, uh, we've already done some here. This is Quen 3 VL Pineapple Unit. So this was for GPT-OSS. Now we can compare I should have benchmark results here. GB10, GPT-OSS. Okay, so we've now done an official benchmark, and I'm probably going to get Claude to just tie these all up and make them look nice later on. Because it'll probably do that way faster than I could. And I mean, they're just CSVs, right? It's not, it's not life-changing, um, code that we need to write. So GPT-OSS, there we go. So if we compare the pair Prompt throughput 2700, prompt throughput 6000, gen throughput 88, uh, gen throughput 244. So again, we're seeing the same, uh, they're about the same results we saw in practice, is the RTX 4090 is about 2.5 or 2 to 2.5 times as fast for throughput in terms of reading. And then generation is, yeah, at least 2.5 times faster. This is across 100 different prompts with 10 practice prompts with an input range of 50 to 1000 and an output range of 50 to 1000 as well. So that's pretty cool to test. But then when we add vision into the mix, so this is Quen, Quen 3 VL AB. Let's see the results over here. Now do I have two of these? Do I? Ah, multimodal. There we go. What do we get? So prompt throughput 799 and prompt throughput 2500. Generation 33, uh, generation 122. So That's pretty cool. I'm glad that these benchmarks work and they kind of line up with where we'd expect them to because why we have memory bandwidth is 4x. So bandwidth is critical for inference training. How many steps? Bandwidth latency sensitive. And then we also have 16,000 CUDA cores versus 6100. So notice how the DGX Spark— this is the metric that the Spark wins out— is 128GB of unified memory times, well, 4x as big as an NVIDIA RTX 4090. So what other tests do we need to run? Let's run text only. Oh wait, did I just do that? Oh no, that was on the Spark. Sorry, we can run this again. So we're doing text only now with Quen3VL. Hello, Hussein. Nah, Nick, I like that. It's not hogging questions at all. Like, you're here. That's why I'm streaming, to talk to, to talk to people. Okay, so now we're running. Hussein, I have been excellent. I'm very happy that I have now ripped out Windows from my RTX 4090 and I am running, uh, pure Linux. So I'm getting much less errors on it, and I've also hardwired everything now. So, um, yeah, new setup. So I rearranged the desk and whatnot. There's a window over here, so I'm getting hopefully better lighting in the stream. Um, but I've hardwired everything up, so, um, the internet's over there, and now there's Ethernet cables running through here because I just wanted to make the setup a bit more reliable. Um, and yeah, I'm looking forward to doing more streams and more videos. So today my goal is to finish off the, uh, benchmarking on the RTX 4090 and the Spark. And then tomorrow I want to compile all the results and then I'll, uh, turn that into a video. So that's my goal. So next is VLLM. Oh, thank you, Hussein. I appreciate that. So for VLLM, I need to— let me get this README. Now I did use Docker on Oh no, that's gonna break that, isn't it? We'll just keep that running. Which one is winning, uh, in terms of raw speed? The 4090 is definitely much faster, but in terms of, um, like ability to hold a larger model, the Spark wins. So basically my, my way of describing it is the RTX 4090 is a Ferrari It's in terms of the world of ML, it's fast, but it doesn't have much room to sit in, if that makes sense, because you only have 2 seats in a Ferrari. But it's really fast. However, the DJX Spark is like a minivan. It's great, it can hold so much stuff, but you're not going to win races with it. It's, it's a bit slower. Now it just depends on what you want to do, right? Like if you want fast inference, you don't mind running smaller models, you actually want to just train smaller models, um, definitely get an RTX 4090 or something like that. But if you want to just explore the AI ecosystem, you don't mind, um, waiting a little bit longer for things to run, DGX Spark. And then again, another trade-off is simplicity of setup. If you're someone who just wants to buy one device, plug it in, and run AI, Spark. Whereas my RTX 4090 I haven't got— I wonder if I can take a photo of the machine. One sec, I'll take a photo of it. I'm going to measure the size of this versus the Spark. Okay, this is the— look, that measurement's not very right, but this is my Deep Learning PC for— and that one's probably skewed a bit, um, but you see how big this is, right? Like, it's, it's, it takes up some room on my desk, uh, whereas the Spark— sorry, on the floor. You couldn't, you couldn't store that on the desk, or you could, but you'd lose a lot of desk space, whereas the Spark is that. So that's something to just consider, right? Yeah, so I think the Spark just runs a, um, a version of Ubuntu, which is what I would recommend a lot of people to run if you're just looking to have like your own deep learning PC. Run Ubuntu. I switched my— yeah, so the Corsair, look, it's a great case, it's a great setup, but it is just big, right? Like it just takes up so much physical space and you can't necessarily just put it to the side of your desk. Like the Spark, if I really wanted to, I could take that if I go on holidays. And I could just plug that into a PowerPoint. And then, uh, or if on the road, and then you just plug it into your laptop via Ethernet, and all of a sudden you've got a little deep learning machine that comes with you. So yeah, there are pros and cons. Like, it is, it is kind of like there is like arguments for and against either one. Like, there's no, oh, I'm definitely going to get this, I'm definitely going to get that, you know. Like, it kind of depends on what you need to run. So there we go, we just ran some text-only benchmarks on— where is it? There we go. So this is text only. And yeah, wow, we get a faster throughput. Not as fast generation output, which is strange. But that's all right. I'm happy with that. So what can we run next on the The— did that not save? Oh no, this is the RTX 4090. Hey Daniel, anything coming up that you're excited for, both personally and AI-wise? Um, what I'm excited for, uh, my brother and I are working on updating our apps for the year, and I'm doing a talk on March 5th at the Queensland AI Meetup. So that'll be an in-person talk on the power of small language models or small models in general. And so, yeah, I'm excited for that. So I'm just gathering ideas for, for getting that ready. And my wife and I are planning our honeymoon, so I'm very excited for that. But obviously, AI, new things are happening every day, so that kind of just keeps me addicted as well. And yeah, I'm excited to do more live streaming and more YouTube videos in 2026. So I'm having a lot of fun. At the moment. Let's see if we can get VLM running. How about you? What are you excited for? VLM. I want to do VLM on the RTX 4090. How do we get this running? So I did use Docker on the DGX Spark, but I almost can't be bothered installing Docker at the moment, so maybe that's something I do off stream. I'll just see if I can get one benchmark running with VLM. How about, how about that? Because I don't really want the stream to just turn into hours of me trying to turn install Docker, and plus I'm meeting my dad for lunch soon. So, um, what I might do is if we can get one VLM test running— where's my README? Here we go. Okay. Do I have just vllm here? Okay, we do have vllm. I'm just going to run quenvl. I can't see— that's the problem, right? All these benchmarks that can't run on the RTX 4090. This can't run on the RTX 4090. So even if we go to the Quen page, we are limited by 24GB of memory. So if you wanted to run Quen VL32B instruct FP8, which is what I want to do for, uh labeling a large image dataset, and then I want to fine-tune that into a smaller model. We have 35.5 gigabytes of files, so that's too big for the RTX 4090. So what we can run though is— do I have— there's VL30B, VL8B instruct. Let's see if we can run this. Vlm serve without Docker. This is probably going to download that vlm. Hey, cool, that's a cool output. I love these outputs. Even a 4090 can't run it. Yeah. So you know what my dream GPU is? And, uh, like, this is— we're just going to manifest this into the world, all right? So— oh no, we've got an error. We'll fix that in a sec. But this is, this is what I'm manifesting: RTX 40— RTX 6000 Pro. I'm manifesting NVIDIA sending me one of these because they sent me the, uh, the DGX Spark, right? Which I'm very grateful for, but I'm making a lot of videos in return, so that's fair's fair. This is, this is what they're going to send me next. That would be so cool. But then again, like the numbers of whether you should own one of these or not is probably not worth it, honestly, because if you— a lot of the, a lot of the— if you just have that sitting there all day, unless you're doing training all the time, it amortizes out to like you just rent a H100 for 4 hours at a time, right? Then you don't need to pay this large cost if that's just going to be sitting there all the time. Okay, what do we get? So on startup is less— what have I got running here? Oh, have I still got my server running? That's my llama server. So we'll serve that. But look at the— look at the specs of this bad boy. 96 gigabytes of GDR7. That's what we want, right? It's still not Spark level memory, but that's, that's some, that's some stuff you could get there. Let's throw in that to the mix. Add in RTX Pro 6000 to the comparison. All right, let's see what we can do in the comparison. Here we go, we're downloading models. Yeah, can we use Flash Attention? We may run into an error here. I don't know if I have Flash Attention installed on the RTX 4090. So that's what we're manifesting here, everyone, is to get one of these sent to us. Not that we need it, but it would be cool, right? Wow, 200— 2.5x faster than RTX 6000 Ada for training. Yeah, building Flash Attention gives me nightmares too. So that's probably— I'm probably— if this, if this crashes out because of Flash Attention, I'll probably end the stream and go get lunch with my dad. And then probably have to build Flash Attention off stream because that might just be, I don't know, 2 to 3 hours of just building wheels. So let's see if this runs. Okay. If this runs, I'll be very happy. So what we do for this benchmark is we run, we run the server, and then we have a bash script which pings the server with synthetic generation. Oh no, what do we get? A warning. I guess we'll see. I'm trying to use as default settings as possible because if we go to VLM, VLLM I need to get out of some tabs. Here we go. Blackwell, wow, 4,000 AI TOPS CUDA cores. So not as big as a leap on CUDA cores compared to the RTX Sorry, the DJI Spark. More teraflops, about 50% more. This is the best one, 96GB of memory, and then faster bandwidth, nearly 2x. You can combine them, that's cool. Requires more power. Okay, we should be running here. Let's see if we can run our test. Oh, excuse me, I need to create a new terminal. cd Uh, oh no, what happened? Ah, Docker. I might have to install Docker. Yeah, I probably should do that because that's, that's a fair comparison. I should use NVIDIA's container for, uh, whatchamacallit, NVIDIA Docker container vlm, because this is what I use for the DGX Spark, so I should really use that same container. Oh, there's already 26.01. Do I have Docker already? I don't think so. Okay, so that'll be— maybe the next stream is I'll run this, I'll get VLM installed, and then the next stream we can start to compare them all. So I'll collate the results. So once I've run all the benchmarks that I can I'll collate them. So behind the scenes, I'll install Docker and VLM and all that sort of jazz, and then get it running so that we have the same, like, level of benchmarking here. So I actually need to get rid of this because they don't have any results in there. And then I'll run them, and then we can compare them. We can compare them with graphics in the next stream because this stream was mostly just looking at text side by side. But the good news is they're all running except for VLM so far. So thank you all for tuning in for the stream. Stay tuned for part 4. We will combine all that or collate all the results, probably using Claude or, I don't know, OpenCode or something like that, because otherwise it's just going to be a lot of CSV matching. And then we'll get some nice graphics to go along with that. So I'll see you all next time and keep machine learning. Okay.