Benchmarking: NVIDIA DGX Spark & NVIDIA RTX 4090 | Part 2 - Formalizing the tests

Testing 1, 2, 3. Okay, beautiful. Looks like we're live. Let me just see. Testing 1, 2, 3. Sweet sound, looks like it's coming through. If you're joining the chat, say hello. We are benchmarking some results. Yugi, what's up? Hussein, hey, what's up? I think this is running. There we go. Alright, so we're running some benchmarks across the GB10, which is the Spark. We're benchmarking throughputs on the NVIDIA RTX 4090. Well, not yet actually. We're actually doing all of the work on here first, and then we'll copy this over to there. And run the test on the RTX 4090 as well. Let's see if this works. So we're using VLLM on the DGX Spark. Oh nice. I have 5080 and H1000. Can we do this together? So might take a bit of setup, but I'm running this— I'm running this documentation, but basically I'm putting it into a bash script. And so what you also need is, uh, I'm running NVIDIA's container for VLLM. So I need to write this down. Steps. We're using a multimodal benchmark so that way we can just turn off images and get text throughput as well. Excuse me. So we're going to update the bash script to save the outputs to So right now I've got a server running here. Cache hit rate. So then we get outputs like this. Oh, did it save anything? Doesn't look like it saved anything to the CSV. Maybe we have to wait till it's fully complete. So we've nearly got a working benchmarking script. Okay, beautiful. So we're running Quen. Gwen Vielle, 3, 32B. So if we go back to So if I could get all the VLM benchmarking done today, that would be great. So we run it on 0 images first. Hmm, I don't think that's Saving anything to file though. So VLM— VLLM works well on the Spark with the NVIDIA container. Okay, it looks like the output CSV is not actually capturing Color Lace, hello. So we are running the following benchmarks at the moment. We're starting with VLLM because we're going to do image generation next, or later on. So this is probably the order we'll do things in, actually. So LLM inference, and we can do VLLM inference. LLM fine-tuning, image generation, and object detection model training. So they're the 5 benchmarks, and we have a series of different models to run. So models, Quen3 VL, Quen3 VL, um, 30B, and Quen 3 VL32B. So I could do GPT-OSS, but you got to draw the line somewhere. So we're going to run, uh, Nemotron V3 and then the Quen models and then VLM inference. Uh, we'll do frameworks. We're going to use, um, Laminet CPP and VLLM. VLM inference, we're going to use the these three frameworks. vlm or llama.cpp.transformers. And maybe GPT OSS. 20B/GPT OSS, 120B. So let's look into— oh, here we go. And also I want to Okay, let's see if this update saves what we need. This is great. So once you have a VLLM server running, you can then ping it. So you run a server on your own computer, and then you can— there we go— encode it to 64, get media type. And then you can send it a request. And get the outputs from your model. Okay, did this work? There we go. Okay, beautiful. So what we might do next is see if we can run Nemotron V3 with VLLM with our text only. Or let's go OpenAI, um, GPT-OSS. Where's Nemo-tron? Oh wow, they've just updated some more. Nvidia on an absolute tear at the moment with their releases. Oh, no way, they've got Nvidia NemoTron NVV port. M— NV FP4. So we lose some performance as we drop in precision. We might actually be able to use That's cool. So should we try FP— NV— FP4? I'm gonna try FP8 because I've had that working. Nice! So let's now try the text-only benchmark. And once we get that one running, we can also do gpt-oss They had a blog post on this somewhere. So the throughput is supposed to be Quite fast. Let's see if we can get it to run. So to run it, we need to close the server. So this is where we're running our container. Might update the max model length to be 128,000. Oh, we don't need that actually. I think 32K was enough actually. So this Nemo model that we're going to use is going to be FB8. Wow, April 1V, V1.6 15B Thinker. I wonder if that's capable with VLLM. Let's hook up that next. We're having fun here. So I want to get Might take a little while to download. Does it have VLL— VLLM settings? Okay, so I need the, the Nano V3 reasoning parser. FP— so, uh, FP8. So 30B means 30 billion total parameters, A3B means active 3 billion parameters, and FP8 means the precision. So it's got 30 billion total parameters, but only 3 billion of those are active. A3B stands for active 3 billion. Let's see if we can download. Oh, what do we get wrong? Oh no. No. So the LoRA is not 3 billion. It's because the original model is 30B, and then— but only 3 billion of those 30 billion are active. So that's called mixture of experts. So if it has 30 billion total parameters and there's 8— sorry, 10 experts at 3 billion parameters each, So let's have a look at MOE architecture. So this is probably So what happens here is you have an input and then you have— if it's 30B with a bit active, 3 billion. So each of these experts is 3 billion parameters. So it's actually 10 smaller models in one big model. And what happens here is that an input comes in and it gets routed to the right expert and then it gets turned into the output. So at any one time a normal model, if it was a dense model, the input would go through every one of these modules. But in an MOE, mixture of experts, it goes into here. This network chooses where it should go. Now the important thing is that we don't actually tell it what to learn and where to put the input. It's just going to learn that based on all the data that it's looked at. So yeah, when our— a mixture of experts model, you kind of get the benefit of having larger number of parameters total, but you get the efficiency of only using a certain number of those at inference. So, um, while a mixture of experts model 30B will have active 3B, at inference time you're only using 3 billion parameters to infer on. So versus Aquin 332B is a dense model, so it's going to inference on all 32 billion parameters. Does that make sense? So MOE, you get the efficiency of a smaller number of inference parameters, but you get the benefit of having a total number of more parameters when you train so that it could potentially learn more patterns. However, dense models are usually still better than MOE on raw benchmark results, but they do take more compute to inference. So there's trade-offs there. Okay, so I kind of knew we would run into these issues. That's what I've had a lot in the past with NVIDIA models, is they're incredibly great, but they do take a bit of setup. So why can't we find this? Yeah, Color Lace, you're right. Interesting. See, these are the type of things that LLMs are incredibly good at. I would never— like, that would take so long to find. Let's see what happens here if we just run this. So I'm not familiar with Docker. Beautiful. There we go, enabled separate CUDA stream for MOE shared experts. Okay, nice. So we're now downloading So I'm just gonna— while this model is downloading, I'm just gonna start updating our README. So same with this model, Qwen 3 VL30B A3B Instruct. I'm using the FP8 versions here. Moses, hello. You know what model just came out that I'm excited to try about— try out, sorry. Tencent just released U2 VL. Let's see if we can get this running in Colab. Hey brother, what's going on? Oh, did I just lag out? If I lag out, it's because I'm downloading a model. Can I get a H100? You really like my context? Content. Thank you so much, that's very nice. Oh, we need an image path. We might not be able to get an H100. Oh damn, we didn't get an A100— H100. Okay, we've downloaded the model. All right, we're an hour in. We're going to try to do some push-ups and some squats. Who's joining in? Oh, what'd we miss over here? 10 push-ups, 10 squats. Who's joining in? Whatever we get now. Let's try benchmark. So we'll make it ex— executable. Oh no, incorrect. Oh, there we go. Oh no, what's happened here? Let's see how they detect. Don't tell me this is going to download. Let's see if the U2 VL can detect. So this is interesting. I think we have to download a model each time we run food. The life of machine learning engineer is waiting for models to download. Okay, we should be— please have loaded correctly. There we go, we got VLM running. Let's see if this works now. What was our raw output here? So something's slightly off about our conversion. 1, 2, 3. But that's pretty good. Wow, the NVIDIA model's pretty quick. You're right. May I get that PDF? This one? This what you're talking about? This is the model that we're using— oh, sorry, we're trying out in here. Wow, that's pretty darn good. 1, 2, 3, 4, 5, 6, 7, 8. Now let's get Broccolini in there. Okay, so we've got some results. Total token throughput. Am I building a website? No, I'm not. Wow. Wonder if we can handle like a group. My gosh, we have a lot of outputs here. Might just have to get rid of all these. Your code looks like you have declared a lot of variables. I have. Okay, so we can now— what did they benchmark it here? I don't really want to copy NVIDIA's benchmarks because I want to do my own. Training a Laura? No, I'm not. So we're getting distracted over here running the U2VL models, but they are quite good. So now we've got a way to do text only. Every time I get a Hugging Face token, I feel obligated to be ethical. What do you mean by be ethical? So let's start ticking off some tests. With VLM. I'd be happy. But I can't always guarantee my datasets are not violating some sort of GDPR rules. Ah, I get you. So Well, AI in general has basically— it doesn't— all the AI models don't really follow the data ethics, do they? Okay, let's run the— so If we wanted someone to be able to reproduce this, how could we do that? That's not too bad. So we got, for a data scientist looking to grow in my AI/ML knowledge and host locally, is now a good time to buy the DGX Spark or is it worth waiting future better hardware? I saw the ASUS was the same with only 1TB of storage for $2,999 USD. So do you have a link to the ASUS? ASUS DGX Spark. Is it the GX10? This one? Well, mine's got 4 terabytes, I believe. I've already used 600 gigabytes, but I'm, I'm being pretty reckless with my storage. Like, I'm not monitoring things. I've downloaded so many models, it's not funny. So I would say, yeah, the DGX Spark is a great, great starter computer. Like, for just to just be able to plug in and run basically so many things. And look, the— by the end of the year, the ecosystem is going to be better. I would say it's probably like 80% there in terms of software ecosystem. The hardware can always be better, but kind of like, that's, I guess it, uh, what's it called, chicken and egg problem. As in, like, would I get the DJI Spark 2? Yes, I probably would. But do I know when that's going to come out? No idea. So they could release the DJI Spark 2 at the end of the year like they did the DJI Spark this year. So I don't know. But yeah, if you want a plug-and-play computer, I definitely recommend the DGX Spark. I mean, you can run— I'm running right now a 30 billion parameter model and it's going quite quick. Like, yes, it is 2.3, 2.3 seconds per iteration, but if you were creating a synthetic dataset, you could leave this running for a few hours and you you could get some incredible results. So yeah, we're using a lot of GPU memory here. I think that's the total available, and this is what VLLM is using. But I would say, just like, if you're worried about storage That— this is what I've used here. I've almost used close to a terabyte. Have I found it useful compared to your 4090? Put it this way, I haven't used the 4090 since I got the DGX Spark, and in fact I'm trying to find more ways to just use the Spark. Yes, the 4090 will be quicker, but I like the flexibility. So probably the way I'll go in the future is— my take on at the moment is that they're not like— they are complementary, but if you could only choose one, I'd probably choose the Spark for me personally, um, because just the setup for the 4090 took way, way more setup than the DGX Spark. Because you have to build a whole computer. And I mean, look at the footprint of the DGX Spark. That's it. You could take this on the road with you. So yes, it is slower in terms of raw compute, but it does have incredible memory, so you can run a lot more, um, a lot more bigger models. Is external SSD or setting up a NAS going to inhibit LLM performance if a model is stored there? So, um, for LLM performance, I would say no, SSD is not. But if you want to like have a larger dataset, you'll probably want an SSD. Depends how big the datasets you are working with, right? Like, as I said, I've been pretty reckless with my storage. I haven't been, um, I've downloaded probably 10 models and they're all 30, 40 gigabytes each. And so if you were being sort of, you're like, okay, I'm going to use this model for the next couple of weeks, and then you clear them off as you go, then yeah, 1 terabyte would be plenty enough. But as for running a model, um, as long as I would say anything you're running, keep it on the the storage that's inbuilt to the Spark. Like, I don't know what the ports are on this in terms of— like, they're USB-C, right? So I'm not sure what the throughput would be in terms of loading that in, but I'm sure it'd be fast enough, right? I don't think you'll take that big of a hit. So if you're just looking to like run some fine-tuning experiments use different LLMs, create your own datasets, that sort of thing, I think, yeah, you'd be fine with, with this. But just be cognizant that, like, if you're downloading— if you have 10 to 20 different models on your 1TB hard drive plus half a dozen datasets that are about 100GB in total, or depends what data you're using, of course. If you only have text datasets, then you maybe may have plenty of space to work with. Um, but yeah, if this is the same chip, I mean, I'm, I'm really enjoying the DGX Spark. So we got some results here. So this is NVIDIA's NemoTron Nano 3 30B A3B FP8. So a lot of stuff there, but we're getting an average token throughput of about 566 tokens per second, which is quite quick. And for 100 samples, it took about 4 minutes. So, and we varied it from 1,000— yeah, there you go. Total input tokens, 100,000. Now this is inference. Am I running this correctly? Do I want to do some voice synthesis? So Usman offering a dataset. At the moment, I'm not— I haven't touched voice in a while. I'm going to stick with text and image for now. I've got enough work to do there, but thank you for the offer. So let's get out of these. So that is cool. Peak concurrent requests, 13. So I'm pretty happy with that. So now let's run. That was text only. Now let's run I'm gonna run Quen, QuenVL 32-bit instruct. So what I do, I'll show you. Let me get rid of some extra terminals. So we first close the server. Anything using memory? No. Beautiful. So this is our Terminal 1, is our server that we set up. So if we want to set up a server for Okay, we're getting somewhere here. Next, we should run You host hardware? I don't— I don't personally host hardware. So let's do the big one, QNVL32B, next. Ah, I think it's important to note that we're using the FP8 version. So this is a dense model now, much more active parameters. So we'll let this download. I wonder if I got that error last time. Have you found the Docker bottlenecks at performance when running models? That's a great question. I, I'm going to put that as a, as a caveat. I'm using NVIDIA's Docker image, so I guess hopefully they've optimized it. I'm going to put that down as notes. Does Docker slow down, um, inference, e.g., using NVIDIA's So I'm using— where's the Docker container? This one. I'll be back in a minute. Just gonna wait while this downloads. I'm supposed to be getting a delivery. Oh, just some stuff, random stuff at home, and I'm just seeing whether it's, uh, arrived or not. I'm also waiting on a, a Switch, not a— I know, I think it's delivered. So an internet switch, Ethernet switch, something like this. I'm getting one of these delivered because I want to be able to— oh no. What did I get wrong here? Yeah, so we have to make sure we have enough GPU memory to serve— it's a 32 billion parameter model, so it's a big model. Performance tuning with chunked prefill. Matched max number batch tokens. So how much VRAM? Here we go, 30— oh no, 3.6 gigabytes at the moment, but that'll probably go up soon because it's still downloading. Let's read about caching. That's something I'll have to look into. So, okay, we're loading the model into memory now. Oh, did it crash? Oh no. My little Spark is chugging along. So once we get a VLLM server running, we can perform some inference. Okay, there we go. Capturing the CUDA graphs. Ladies and gentlemen, we have a model running! So now we can run. Learning about bash scripts here. Nice. Okay, so the memory usage— there we go, 89 gigabytes at the moment. And I'll show you what this looks like as we go. So I might start recording as well. And if I say purple elephant, that way when I download the stream later I can say, okay, we're— now we're going to run tests on Quen 3 VL 32B instruct FP8. And so to do that, we have this script here, and we can update this. So we could go vlm image and text testing.sh, and then we could put in model name, and then we could put in num prompts. But we actually Want to leave both those by default. So let's run this. Then we get some outputs here. There's the model that we're running. Now the model has to be the same model we've got running on the VLLM server. So there's our, our settings hostname, and there's the GPU we're using. So it's going to start to warm up first, and we should be able to watch this. Yeah, there we go. There's the GPU percentage that has jumped right up to the top. So it's using 89 gigabytes at the moment. I believe GPU is using 96% of memory. Wait for it to start inferencing as well. So here's what we're doing. So we're sampling from the input length. So we're doing random prompts input, 7— between 700 and 1300 input and output length of random between 210 and 390 and different image sizes. Now we're starting, we're doing, um, one image at a time and no images. So this is effectively doing LLM only and VLM, so with one image and with no images. And then we can compare what the difference inference speed is with and without images. So it's about to run some warm-ups, so they don't get included in the, in the benchmarks. We just want to have the raw performance outputs. There we go. So the warm-ups are done. Now we're going to do 100 prompts. So behind the scenes, VLLM is— there we go. So the first one takes a while, but now it's warmed up. As everything starts to get cached and put into memory. Okay, and some are going to be faster and slower than others because we have a random input length and output length, and random image sizes. So the different image sizes are of course going to require more preprocessing, but I wanted to do these tests because that reflects a bit more of real life where not all your images are normalized. But of course in the future we could normalize them. Let's write this down. Note for different image sizes. Sweet. So there we go. As you can see, it's running here, and this is telling us how many seconds per iteration it's taking. And we're using a whole bunch of GPU memory right here. NVIDIA SMI LMS 500. Okay. What did you want done with the LMS? LMS is not a recognized command. There we go. GPU utilization 96%. VLLM engine core is using 89 gigabytes of memory. So I may not be running this the best possible way. But the beauty of when I post this to YouTube is that people are going to watch this and tell me that I'm not doing it right and then offer better ways to do it. So then it gets improved. So that's, that's the reverse internet theory, is that if you want the correct answer, often the best way to get it is to post it to the internet in the wrong form, and then someone will correct you. So while this is running, I'm going to do some push-ups and squats. Just 10 of each. 10 squats, 10 push-ups every hour. I'm slowly getting back into more push-ups cuz I hurt my shoulder. Well, not hurt, but like just strained a little in jiu-jitsu, so I was laying off it a bit, but now it's feeling pretty good. Now let it rest and recover, but the key is also to just keep moving. It's still a little tight when I do that. Maybe I need to get the gymnastic rings back out. So we're currently running this test here, QUEN 3 VL32B. Oh, you want to loop every 500 milliseconds? We've got nvtop running here. See, it's just a straight line, it's not really changing. But if you want, you can do watch n1-nvidia-smi. This is going to refresh every 1 second. But VLLM holds it pretty good at 96% usage. For some reason you don't get— because it's, I think it's CPU and GPU— you don't get memory usage here. But yeah, there we go, VLLM engine core is using 89 gigabytes. So we are currently running Quen 3 VL32B instruct. Now The important point is, note, you could only run this on the NVIDIA DGX Spark. It's too large for the RTX 4090, so these benchmarks are already kind of biased towards the Spark because we have larger models. Essentially, this is the only one we'll be able to run on this— on the RTX 4090. This is too big, this is too big, and this is too big. So as you can see, yeah, text only, zero images. So how long is this going to take? About 10 minutes. And then we'll do one image at a time. And then I'd love to— I I think I'll be able to get these. Yeah, so now we're going to start. This is our bash script, goes through 1 image and 2 images— oh sorry, 0 images and 1 image. There we go, single image, 1 image, and it's going to do some warm-ups. But now we should have benchmark results, NVIDIA GB10, and there we go, 100 samples, 0 images. And we're capturing the whole output trace here as well. And this is straight from VLLM as well. So if we compare Now this is NemoTron 3 and this is QuenVL 32-bit instruct. Same settings. Yeah, see, total token throughput Almost 3x. And now we're going to use some images, so we'll probably see images put a slowdown as well. There we go, that's really cool. Okay, so the NemoTron 3 Nano is about 3x faster with VLLM than QN. Not that much faster than I— what I thought it would be. Everyone put in the chat where you're watching from. I love to hear where people are from. And also put in what your current project is, or the thing you're having the most fun with at the moment working on. Where you're from and what you're having the most fun with working on. I'm gonna get a cup of tea. Who wants a cup of tea? Hey, Plexus, what's going on? Radelade. Oh, Leona's from Brisbane. What's up? Working for Queensland Health project now for an agent. That is very cool. Working on conversational voice models. I love it. We got a good diversity. Los Angeles, Brisbane, Queensland Health. That's very nice. Is it AI related? Are you allowed to tell me what project it is? So I have kicking off a test and reverse internet theory. I'm just saving that clip. So I probably got about half an hour left of streaming time. Okay, you see what— yeah, there we go. That's a good example of a sample that took a bit longer than usual because in my script, we sample different images. So 512, 512, 720, 1280, and these are going to be random images, right? VLLM's benchmark. And what I've done actually is stick as close as possible to the default benchmarks. There we go. They try to do some AI things, but the AI response is hard to control. Lol. Oh, isn't that the definition of AI? The benchmarks come out and they're incredible, but then when you deploy them to an actual system I'm doing your PyTorch DL course right now, but some Jupyter issues came up, so hopping on stream while I debug on the side. This course is so dope. Oh, thank you very much, Arav— Arav— Arav Jan. Do I say that correctly? Arav Jan. Arab Jane. Yeah, get the stream up because we're doing the coding vibes here. I'm happy to talk to chat at the moment because, well, always happy to talk to chat. That's why I stream, right? But we kind of got to wait for some benchmarks to run. We're doing 100 samples because I think that's a good enough amount of prompts. So after we finish these And then image generation. And then we want optimizer step, step, step. Haha, you got the PyTorch training loop jingle stuck in your head. That is incredible. Well, do you know that that jingle is the result, or what trains every single LLM in existence right now, or every single neural network? Is you do the forward pass, you calculate the loss, optimize a zero grad, loss backward, optimize a step, step, step. Once you remember that song, you know all the steps to train a neural network in the current paradigm, at least. Okay, where's some articles we can read? Should we try GPT-OSS 20B or 120B? I wonder. Yeah, let's just try the 20B model. World-changing song, lol. Okay, we just finished the test now. So take a note here of this is no images, right? Let's just go the big number, total token throughput 231. What do we get when we add images in, random images? Okay, so we get about a 30% token decrease or 20% thereabouts. So images give us a little bit of a hit on throughput, and the duration for 100 prompts instead of about 100 seconds more. So what's that, about 20% more with images? It's not that much more, to be honest. So that is— we can now tick off. We're going to get all the VLM tests done on the DGX Spark, and I'm going to do all the DGX Spark tests first, sorry, and then I'll go do them on the RTX 4090. So we're kind of batching it because I've ordered an internet switch, because what I want to do is bring my RTX 4090 GPU is over there. I don't know if you can see that. It's over there. I want to bring that over to here and basically remove all this desk so I have a bit more room in this space. So I want to upgrade my streamer setup, if you know what I mean. And that way I don't have to— if something goes wrong over there, I don't have to go sit over there. I want to put it right here in front of me, and then I want to hardwire it all together. So silly me, I've been using SSH over Wi-Fi, and that connection is not great, and I'm in the same office, so might as well connect. And I think I'm lagging out, so that's the problem with Wi-Fi. I want to hardwire this whole setup. Okay. Um, I have been blessed with great Wi-Fi, but I think hardwire can be better. Let's now do Quen 3 VL 30B Active 3B. So this is a mixture of experts one, so we'll expect this one to be— it's using 10x less active parameters. So will we get an increase in Optimizer 0. Great, now you've got it stuck in my head. So we're using the instruct versions, not the thinking, because I mean, you have to draw the line somewhere, right? Can't benchmark everything. Or we could become a company like Artificial Analysis and make millions of dollars benchmarking everything. Arguably that is a good company to start, is you like choose a vertical and you run benchmarks on your vertical. Let me know if you're lagging or the stream is lagging. Okay, we're doing Quen 3 VL now. I'm sorry, we're lagging. Okay, thank you for the updates, everyone. I appreciate it. So the stream is not— it's not too bad, audio is fine. That's good. Yeah, I could try GLM Flash. Yeah, VLLM and SG Lang support GLM's 4.7 Flash on their main branches, so I would do that in a future one, probably, because what we're doing is, at the time of recording, late January 2026, VLM support GLM 4.7 Flash on their main branches, but I'm using NVIDIA's VLLM container. Yeah, so They have an FP8 version though. I'd expect it to be similar then, uh, to— oh no, sorry. That was the full one. Oh, they don't have a flash FP8 yet. I'd expect it to be similar to, um Quen 3 VL. Oh wait, Unslaught have one. Okay, we could try that in Llama CPP. Now there was also Aki. See, these guys inspire me because they've started— they've just started up and they have incredibly powerful models out there already. I just heard my DJI Spark make the coolest fan noise. Okay, we're running. Now we can run our benchmark. So to run our benchmark— You have GLM 4.7 Flash Q3 KS. I think it probably will be. These days a lot of models are trained with compression in mind. So Unslaught also have— so yeah, you have this one. Where is it? There. So what do these magic numbers mean? Like Q2K, Q2KM. What's the bloke up to these days? 26,000 followers. So he made all of these GGF models and now is gone. What does the S mean? KS. Q3 KS. Oh no, my internet. Okay, so this is a— yeah, a much faster This is about what, what we would expect, right? S is small. Oh, nice. Yeah, you're right, Arav Jan. So we're nearly finished testing for QN3VL. Is your reason for switching to an Ethernet switch due to SSH not being stable? Yes. So when you connect directly, like, this could all run offline now. Right, I've got the models downloaded and I could just perform inference offline as long as I didn't need any, um, any external data. And then the— so the SSH connection, if it's connected to via cable, is then not dependent on the internet. But if I was to, I guess rely on Wi-Fi, then the SSH connection could be shaky. Um, not only if the Wi-Fi jumps, then the SSH connection could drop, which I've experienced several times in the past. Wi-Fi is helpful, of course, when you're downloading things and whatnot, internet's downloading things, but once you've got your data and whatnot running locally, you don't actually need internet connection. So I'd rather rely on a stable connection. And then only need— only use the internet when it's required, if that makes sense. I don't want to have to rely on having an internet connection just to connect two of my computers that I'm working on right here. So I'm going to hardwire up everything over the next couple of days. But it's 3 hours in, it's time to do, uh, 10 push-ups, 10 squats. Let's see if we can get it done before this finishes. We've got about a minute. Beautiful, we beat the inference. Okay, let's check. This is QN32B. And then we compare it. This was with no images. Let's do no images first. And this is, uh, 30B active 3B. Yeah, okay. So that's pretty good. So over 2x throughput. That's basically what I'm looking at. Duration time, so half the time or less than half the time. Because what my main goal would be is to run these models these larger models to do synthetic data labeling and then distill them into a smaller model. So that's probably a future series that we're going to do. But that is incredible. The Active 30B— or sorry, the 30B Active 3B, the mixture of experts one— less than half the time than the dense model. But the dense model is not far off. Maybe we could increase this. I don't know how much higher this could go. Maximum request concurrency. I'll put that in the notes. Uh, right now we're running exclusively on the DJI Spark. I'm going to do all the tests on the Spark first and then go to the RTX 4090 next. Are you running a Mac Mini setup? I'm curious how you plan to wire it up if you're using a laptop. Yeah, I'm using a— I'm using a Mac Mini. This is my setup. This is my setup right now. This is what I can see in front of me. So I got the DJI Spark right there, and then there's a Mac Mini, and behind me is the deep learning PC with the RTX 4090. But I'm going to put that over here and then hardwire it all together, and then hopefully the connection is a lot more stable. And I'm also going to— I'm working on basically cleaning out by deleting everything on the deep learning PC and rebuilding it with just pure Linux. Right now it runs Windows and Linux and using WSL2, but Windows is just terrible. So I'm going to switch to just pure Linux because I've never had to use Windows. Uh, I want to use just pure Linux. Okay, let's run one more test and then I gotta go. I want to write up all the tests that we're doing and then save it to, um, GitHub, and then I can just run them on my RTX 4090. So now we're doing QWEN, QWEN3 VL8B instruct FP8. I love the Mac Mini. But it's quite hot at the moment too. The DJI Spark is nice and cool, so this download should be quicker. But now we can put a tick on this. I'm getting good at running benchmarks now. See, at the start it was quite confusing to set this all up, but after a good few hours of practice, I'm confident using VLLM now. And I think I want to run a large batch inference job, and I'm going to use VLLM for that. I'm going to do it with Quen32B Instruct FP8 because it's the slowest model but it's the best quality. I think it's the best VL model we can run on the DGX Spark. And then let's have a look at with the image. So if we go down, we lose about 20% throughput with the image. Is that the same on— yeah, we lose about 20% throughput when we add multimodality. So the upcoming stream series will probably be— I want to run a large inference job. That'll probably be— I'll do that offline just because that's going to take a while. And then I want to create a VLM, a small VLM, to replicate what the large VLM did. And then I want to figure out how to deploy it to device. So that'll be the goal. But I need to run this test. I wonder, because the 8B model will have 8B parameters, 8 billion, all dense. Will it be as fast or slower than the A3B model? So this does have more total, but it has less, uh, active than the 8B model. VLM concurrency values. So I wonder— So now we're going to benchmark. We'll do a final benchmark. Well, final benchmark for today. Run Anywhere is good for putting SLMs onto phones. Run Anywhere. Cool. Matthew Neumann, 4535. I was thinking about DGX as well. What's your opinion? Would you rather stick with a consumer GPU like the 4090? Well, Matthew, here's my problem. I'm torn, and I'm kind of biased because I already have a 4090, but I think if I was in the market for a single computer these days, I would get a DGX Spark because of the simple plug and play. It's one plug and it's more memory. So if you want to train things fast, the RTX 4090. But to me, a lot of the most work you do is like data processing and getting things ready. And then the training, I mean, you could just run that overnight. So, uh, I think the DGX Spark is worth it for the upgraded memory of being able to large— run large models if I had to choose one. But obviously I'm biased, I'd say get both because to me they complement each other in a sense that, um, my takeaway is already basically, um, for pure brute force computing, RTX 4090. But for generally exploring the AI space, doing a little bit of fine-tuning here, trying new models, big models, definitely the DGX Spark. The 128GB of memory is outstanding. Uh, what is the ultimate goal here? Uh, just to basically run tests, run a series of tests across them so we have some concrete numbers. My gut is already telling me what I just said, like the RTX 4090 is clearly better at brute force compute. So if you need fast stuff, do that. But if you need run, run all the latest models, uh, another advantage of the DGX Spark is that it's Grace Blackwell architecture, so you can run, um, NVFP4, right? So It's a newer, newer version of the hardware. So for inference, like, all the times are going to go down on the RTX 4090, but it won't even be able to run certain models. So it won't be able to run Quen 3 VL32B instruct, it won't be able to run 30B Active 3B. I've already tried on there. It wouldn't be able to run NemoTron V3 Nano. It may not even be able to run Flux 2 client. But so the end goal of this sort of series of streams is to basically put some numbers behind what I'm saying. So then you could just go to the repo and go, okay, the RTX 4090, sure, all the numbers are lower, but um, in terms of raw speed, but it actually couldn't run half the tests because half the tests use a larger model. Yeah, well, if you want to purely build and train ML models, if you want the fastest possible thing, I'd get an RTX 4090. But Big caveat, just know that you will run into VRAM issues if you want to run bigger models over 8 billion parameters. That's a big but. But if you're like, oh, I want to train some object detection models, RTX 4090. I actually used my RTX 4090, um, in a client project recently where I was training on 400,000 images for an object detection model. Sorry, not 400,000, 40,000. But there was 700,000 in the classification. This is cool. Thanks for this, um, Arav— Arav Jain. I'm going to send this to my brother. Yeah, Ibrahim is correct. They really do complement each other, especially for robotics. You can simulate with RTX series GPUs and load the models directly on Nano hardware, for example. So there we go, we've run it on the Quen. Let's just do one last check and then I've got to rock and roll because I've got to start getting some food for dinner. Cook dinner for the family. Now we got Quen. Here we go. So the 8B instruct version versus the 30B, AB, A3B. Wow. So the Quen 8B model had the highest throughput and the, the shortest runtime. That's interesting to me, right? Same test, but it has more parameters, a dense model. Then, hmm, again, maybe there's some settings I could change, but I just wanted to do apples to apples, the same comparison. All we're changing is a different model. But that's something of note, right? Okay, when we add an image, we lose about 20% of throughput. But it's still faster than QN30B Active 3B. Okay, so that's been a successful stream to me. Thank you everyone for tuning in. Um, tomorrow— what's tomorrow? Friday. Tomorrow we'll continue with the benchmark results. So we've done a lot of the VLM— VLLM testing They're most of the tests I wanted to run. Maybe we want run one more tomorrow with GPT. We can do that with Llama. Hmm, maybe we have to do it with Transformers as well. Okay, that's what we want to do next. Um, GPT-OSS. That's a good idea, just copy the settings from the VR. And then we want to use image generation, object detection model fine-tuning, and was there one more? LLM fine-tuning. Okay, so thank you all for tuning in wherever you are in the world. Los Angeles, Brisbane, Adelaide, you have an amazing evening, morning, slash lunchtime, and I'll see you all tomorrow. Well, tomorrow for me might be the same day for you. Peace.