Benchmarking: NVIDIA DGX Spark & NVIDIA RTX 4090 | Part 1 - Creating a series of tests

We're going live. Let me just see if this sounds all right. We're going live. Beautiful. As always. Do a shout out if you're in the chat. Where are you from? Let's do a LinkedIn post. I'm gonna go here. Going live on YouTube comparing the NVIDIA DGX Spark to the NVIDIA, uh, RTX 4090 across a series of of tests. This is part 1 where we'll figure out what tests to run. I wonder if that'll work. Sweet! View post. I don't know if it generates a preview. Oh well. Okay, so this is the paper of the day, Prompt Repetition Improves Non-Reasoning LLMs. So if we look at this, this is a very simple technique to improve results. And now they have the caveat here of non-reasoning LLMs. But essentially, if you just put in query followed by the same query, you get better results. And as we can see on all the benchmarks, the one here— where's it? Yeah, prompt repetition either gets better or on par with just one prompt. So we're still figuring out how to best prompt engineer LLMs. Now, On this one was a pretty cool one. Name index 2.0 flashlight Gemini goes from whatever that is, about 22%. Hey, how are we? Desktop or mobile 4090? Desktop. Hey Hassan, how are you? Or Hussein. This goes from 22% to about 90%. And then 2.0 Flash, not Lite, goes from 65% right up to basically 95% or so on this task. But if we go down to see how they did it, it's literally just repeat the prompt. So prompt repetition query. Let me repeat that query. I love how they, they have an example, example table about this. Here's the question. Let me repeat that. Here's the question again. Exact thing that I just said above. And that improves results in non-reasoning LLMs. Now it also didn't increase latency because the prompt, the prefill stage is the fast stage. You can parallelize that. They even said down here, if you prompt 3 times, it also works good. Prompt repetition times 3. Alex, what's up, my friend? We got query. Let me repeat that query. Let me repeat that one more time. Query. And you get improved results. That's what I'm talking about. That's cool. Second time asking is the charm. Yeah, asking again is all you need. This is what that paper should have been. So today, where, as you can see, we're going to benchmark the RTX 4090, or we're going to start. I've been having some terrible issues connecting to my RTX 4090, uh, via SSH, but I'm going to fix that. But I think today's stream, we're going to figure out, or we're going to think of a series of tests that we can run. Correction, yeah, third time is the charm. So That's— this is paper of the day. And they have recommendations here. So where's their recommendations? Future directions. There we go. A conclusion. We showed that repeating the prompts consistently improves model performance for a range of models and benchmarks when not using reasoning. In addition, latency is not impacted. And then prompt repetition can affect latency for long prompts and might be impossible for very long ones. So something to keep in mind depending on how long your prompt is. Prompt repetition does not change the length or format of the generated outputs, and it might be a good default for many models and tasks when reasoning is not used. Hey, Faith, how are you? Great to see you as well in the chat. Future directions: fine-tune the model with repeated prompts, train a reasoning model with prompt repetition to increase efficiency. The model might learn to avoid repeating the prompt. Periodically repeat the last generation tokens during generation, as well as explore applicability to multi-turn scenarios. Only keep the second repetition in the KV cache, uh, that's been completely performance neutral for the generation stage. Repeat only parts of the prompt, especially for longer prompts. Reorder the prompt with a smaller model instead of repeating everything. Uh, applicability to non-text modalities. So they only tested it with text only, no images. Further analyze different variants. More than 2 repetitions might be advantageous. All right, so put in the chat, what are some tests you'd like to see Let me make a new document. Actually, I'll get a Keynote going. Keynote. New document. Or maybe we just get some code running. Use code as your ground truth. By the way, if you want to read this paper, I'll put it in the chat. That's what we're reading. So let's go to the DGX Spark stream and we'll go folder, um, GPU benchmarking. Beautiful. Readme.md. So we want some goals. Compare GPUs on different tasks. Goal: run several, uh, ML tasks and measure performance on each, then collate the results into a report. So Tess, what should we do? LLM inference, of course. VLM inference. And our measures will be tokens per second, tokens per second, maybe image generation, and then probably LLM fine-tuning samples per second or tokens per second. Uh, what else can we do? Total time. Total time, image generation. And then maybe we also do— what else could we do? Maybe, uh, I do have some object detection model. I do have some code there. Object detection model training. Who likes that? So let's, let's find an image generation model that we can use. I've actually got an ImageGen. Let's put that in here. Which model did I use? Klein. G'day, Drasai, how are you? So can we run this yet? Let's upgrade to the latest version of Diffusers. There we go. Then we'll have a look at the image. So what are you using CV for? What do you mean by CV? Computer vision. I did see that Hugging Face released V5 of Transformers. There was a blog post I read that Transformers V5, um, blog. There we go. Transformers is installed 3 million times every day. That's incredible. Okay, we're downloading Flux2 Klein. So what test should we run? Who has a favorite image generation model? So let's go huggingface.co/models. Today we're going to do some image generation because that's what I feel like doing. And also KIMI just released K25. 2.5. That is an incredible model. So how many parameters does Flux2Dev have? 32 billion parameter model. Holy smokes. 64 gig. I wonder if we can run that on the DJI Spark. NVIDIA's Cosmos— what is that model? See, this is— there's too many models out here these days. Text to image, that's what I want. So we're going to try a few. Models, and we can use, uh, ZImage Turbo. So yeah, we need to look at Black Forest Labs. Ah, okay, they have NVFP4 and FP8. Okay, so that's much smaller. 32B open white model. Okay, who wants to try this? Flux2dev mv fp4. This might take a little while to download though. Oh no, what happened? And let's find out, NVIDIA Cosmos. So we'll try Klein 4B because this is Apache 2.0. And then give that an upvote. We are spoiled for choice here, aren't we? Oh, we've got ZImage Turbo and ZImage now, not Turbo. This is brand new. Nvidia's Cosmos. Let's see. Oh, Radio V4! My gosh, we are getting some Now, what is Cosmos? Cosmos 2. Okay, these aren't really image generation. We're focusing on image generation at the moment. So improves visual feature extraction from radio. So it combines SigLIP2, DynoV3 7B, and SAM3. Embeddings generated by this model expected to be used by downstream application, for example, image-level understanding, dense processing, integration into a vision language model. That is cool. And Radio V4. I gotta put this into I'm crafting my Commercially available as well. So I have been looking to build a VLM. Hi, Ayuki, how are you? So by leveraging multi-teacher distillation Algo, algomorative vision backbones provide a unified student model that retains and improves the distinct capabilities of multiple teachers. In this work, we describe the most recent release of the C-Radio family of models, which builds upon AM Radio, Radio v0.2.5 in design, offering strong improvements on key downstream tasks at the same compute computational complexity. Re-release SO 400 mil, so 400 million parameters, shape optimized that is, and huge variants both trained with updated set of teachers SigLIP2, DynoV3, and SAM3. In addition to improvements on core metrics and new capabilities from imitating SAM3, the SeaRadioV4 model family further improves any resolution support, brings back the vit-det option for drastically enhanced efficiency at high resolution and comes with a permissive license. Oh no, I'm lagging. How's the stream? Okay, I think I'm back. Let me know if we're back in the chat. So Radio can replicate SAM3 results while also having— that is cool. Shoe, shoe, helmet. Okay, this is great. I like this technical report, nice and straightforward. Are we still stabilized? Let me know. Thanks, Alex, for shouting out. Okay, let's go back. Probably because we were downloading a model. Did we get a model out? Image. Sorry. Hello world. Okay, how long did that take? So let's turn off enable model to go to CPU, and we just want to see if we can just work straight on the GPU. Cannot generate a CPU tensor from a generator type of CUDA. So we're loading the model into GPU memory here. There might be a better version. Let me get out of some of these. We should compare the NVFP4 version. And then we'll also do ZImage Turbo. Oh, it's much quicker on the GPU only. So let's now try another pipeline, this time with So that was using the basic Klein 4B. Should we do FP4 now? Let's see how we go there. We gotta figure out how to use this. Okay, we're gonna load Flux2Klein. How did I get the DGX Spark? NVIDIA sent me one, so very lucky for that. What do we get wrong? So we're going to try and figure out how to run the, uh, NV. FP4 version. How long does ZImage Turbo take? Wow, that took a bit while longer. Try it again. Use the NVFP4 model in ComfyUI. Let's go build. Use ComfyUI to generate images. Ah, I see. So this is a guide on how to set up I kind of want to use the Flux 2 models. Look at that, you can get 55 tokens per second generation. So they're using Flux 0.1, but I want to use Flux 2. Let's try Flux. Let's download the 9B version. Or can we run the FP8 version, I wonder? Okay, well, it doesn't look like there's many options here for the FP8. Versions, and the FP4 versions require quite a bit of setup. Okay, so it's only available with ComfyUI. 100K stars, that's outstanding. The most powerful and modular visual AI engine application. Has anyone in the chat used ComfyUI? T Sizzle, have you considered using NVTop to monitor GPU utilization rather than NVIDIA SMI? Tensor LLM support for GB10 seems lacking. It's the community on NVIDIA forums pushing fixes, not the actual NVIDIA team. Are you connecting to the DGX Spark via VS Code? Yes, I am. So this is running on the DGX Spark, as you can see down here. Okay, NVTop looks good. Let me install it. Very cool. Yeah, it seems that, that the, the support for the GB10 is fairly lacking at the moment. And the GV10 is the GPU, uh, GPU that's in the DGX Spark. All of the base things work, but the optimization steps are still— they still require a bit of setup. Apparently this model has been censored. Let's see if we can run this. So I might lag out here if we're going to start downloading a new model file. Are you grinding LeetCode every day? What is your opinion about it? No, I've never done a LeetCode problem. What is my opinion of LeetCode? If you want to use it, go ahead. Seems like a great way to practice code, but my, my way of practicing code is by doing things like this, building things I'd like to use in the real world. So, okay, we've done enough testing here. We've got our models that we're going to use. So the ImageGenTest— no, requires ComfyUI. Perhaps. So we're going to do I don't think it's Apache 2.0, actually. No, it's not. Maybe do some transcription. Super appreciate your long tutorials on YouTube, mate. Went from zero to conversant in coding and all thanks to you. Oh, thank you so much, that's very nice. Let me know if there's any other kind of tutorials you'd like to see, long form. So we're just waiting on Flux 2 9B to download. There's also GLM image, but we've got to draw the line somewhere, you know. I might be starting to lag out because this is downloading a model quite fast. How many parameters is this? Okay, so the memory is climbing up here. Stream is surprisingly fine. Okay, that's good. Yeah, web browsing is really being, uh, slowed down at the moment. So the model's loading into memory here, and we are an hour in. Who's going to do 10 squats? I've got a bit of a sore shoulder at the moment, so I'm skipping push-ups. Do I have any thoughts? On combining more than 2 DGX Sparks, say 4, to run larger models? Where were we? We just had a blog post up here of— yeah, I'm not sure if you can combine 4. Maybe you can if someone's seen you combining 4. So you can run 2 together. I know that for sure. It may— depends how much better, like, you get outputs from Qwen 3 235B for 5x less output speed. It does have quite the high throughput speed. So the model I would probably run is NVIDIA, um, NemoTron V3. Yeah, this one. What issues do we have here? So you can run this one with— and you don't actually need 2 Sparks for that. Yeah, NemoTron 3 with Lama. CCP, CPP. So that'll probably be one of our tests. And probably a Quen 3 as well, right? So right now we're using— oh, that's cool— our GPU memory is using about 35 gigabytes. And how long did that take? Which cat is better? I think I prefer the 9B image. What do you think? Just try to run it again. Okay, about 8 seconds, whereas So what we could do is we need about 1,000. I think we see how long it takes us to generate 1,000 images. And then we measure, we benchmark that, and then we could also compare them, right? So I wonder how it does with a more complex prompt. So let's go to Gemini. So this is what I had for dinner the other night, and we're going to do reverse image prompt. So caption this image in detail. So an image Generation model can replicate it. Let's just see what happens. Oh, I went to reload the model. My bad. So we need How does that sound? A thousand images? Johnny, you saying agreed to generating 1,000 images? So that's what I think I'd like to do. I agreed to the left cat being the better one. Yeah, this is Flux Klein 9B, and this is Flux Klein 4B. Hey, okay, how close is that to this? Obviously a lot more detail. And now let's compare— or compare Gemini. We'll compare Nano Banana Pro and then maybe ChatGPT. Okay, a bit more detail here with Nano Banana Pro. That looks a lot more realistic, right? So this is FluxKline, but the benefit of this is that it's local. And then this is ChatGPT. Okay, that was pretty good. This is the original image. I think it's Nano Banana Pro and ChatGPT. Which makes sense, right? They're much bigger models. Let's try Z-Image Turbo. Okay, that was all right, but not as high quality. And then why don't we try one more? GLM image. Oh, which one is winning? Out of the RTX or DGX Spark, is that what your question is? Okay, so that's taking a fairly long time to generate. Okay, maybe this model won't finish. All right, so let's create a dataset for testing, and maybe we can do so using— I've got an idea how we can like test multiple things simultaneously. So we're going to load our dataset. So we'll maybe do Quentry 8B. I think that's a pretty good model to focus on. Okay, I'll be back in a sec. Okay, let's See if we can take this and go Let's just go, based on this recipe markdown, can you please, um, convert it to an image generation prompt which showcases the final outcome? Let's see if this works. Oh, that's pretty good actually. What are we thinking? Wait, did that just run? Oh no. That's pretty good. Flux Klein. Oh, that's even better. Okay, that's good too. So, steps. Um. Get recipe input. So we're going to create Beautiful. So we're gonna see— Okay, let's see if we can pass this to NVIDIA. So that's our goal. And let's now restart this because we want to free up some memory. And now let's So I need to activate So we're going to try and get Llama or Nimatron 3 running. There we go, we should be able to run this. Here we go, load tensors. Nice. Okay, format, and we want Check out that. Now this is running on the Spock. Now we're using these to test. I wonder how I turn off Thinking. Wow, that's a nice image. There we go. Artichoke hearts wrapped in bacon. So let's now pass this to— or I wonder if Quen— Quen 38B can do it. Okay, nice. So let's figure out how we can get this. Is there a way to turn reasoning off? Okay, so we can turn reasoning off. So we've got a workflow that's going to work. So 50 tokens a second is pretty good. But it did take 20 seconds for a single output. Okay, this is good. Okay, so we get the timing. Oh, that's great. So what are the current settings here? So we get back the timings. This is great. So it spent much of its time Reasoning. Then we can turn reasoning off. So we get about 50 tokens per second. This is a good model. So whether or not we put thinking on— Now we're going to go random. What's a guilt-free— guilt-free blueberry LaQuanda? So let's— Then we're going to get our model. Now we're doing See what this looks like. That's cool. Okay, so we've got a way to create synthetic prompts. So has Quinn Tree VL on Sloth. So you can run it with Transformers, or With— Let's read about Lama CCP, or CPP. I keep saying CCP, that's the Chinese Communist Party, isn't it? We get a web UI. Lama Ban. Oh, this is awesome. It works on mobile too. Oh, this is so cool. We might need to have more of a llama CPP stream. That's awesome. Okay, so we've got some ways to do things here. What about OpenAI? I'm not sure if I can run this on Hey, Adam. So we are We are currently setting up tests of benchmarking. So, uh, LLM inference, LLM fine-tuning, uh, VLM inference, transcription, image generation, and object detection model tuning. So what have we got? 1, 2, 3, 4, 5, 6 tests, and We're going to do, uh, we'll test a couple of models for each one, and then we'll try and run the same test from the NVIDIA DGX Spark and, uh, to the RTX 4090, and then we'll compare and see which, which, which GPU is better for which use case. So running just a bunch of AI native tests. Oh, it's squat time. Who's joining in for 10 squats? Let me know if that makes sense. Okay, so someone's getting a GPT OSS 20B running on RTX 4090. Because we'll, we'll try to run that. Okay, so we got some good options. So image generation we can do. LLM inference we can do. LLM fine-tuning. Maybe we stick to 1, 2, 3, for 5 tests. I think that's enough. So let's start That's incredibly fast. Spanish chicken and rice. Let's see what this comes out with. So we've got our tests that we want to run. Next will be figuring out Ah, we get cache. That's cool. If we rerun the same prompt Okay, we don't get any cash there. Let's figure out— AI and ML Monthly for this month? As in the video version? There'll be the— there'll be the text-based version, that's for sure. But the video versions, I'm going to put on pause for now because I want to focus on doing other types of videos. Is there a quick, like, sample method from here? So let's get Claude to ride us. Let's see if Claude can create us a test script for what we'd like to do. Beautiful. Sir, can you be my mentor? What makes my guru Daniel Bourke so good at AI and ML? I'm not a guru, I just like to practice. I just find it fun. I would really like to have you as my mentor. How can I mentor you? What would you like me to do as your mentor? Let's see if this works. Okay, this is cool. Maybe we just try it for 100 to begin with. You're an AI researcher and you like to— you would like to have you— me mentor you. If you want to become good at PyTorch, you got to code a lot of PyTorch. What are you doing day to day at the moment? Are you practicing? Okay, good. So we've got a way to do testing LLM inference. So I think for this And then we're going to go Options, Use Plus Transformers. Simplest methods. Let's see what this saves out to. Bro Hizzle, how you going? So I want you to provide me with the research or project idea I can build on and have you as a mentor for guiding me. I'm not available for that at the moment because I've got my own projects to work on. But What projects do you have, or what projects are you working on at the moment? Is there anything you've been trying yourself? Okay. That's nice. Let's see if we can get Let's see if we can get Quen 3 VL running. Let's see if we can get Quen 3 VL running. So we want to download this model. Aha, which one do I want to use here? Who knows what all of these— who knows what all of these mean. So that must be, yeah, the original. Would you recommend running Gemma locally on an enterprise environment? Depends what for. Let's see what happens here. So that's Q8, UnSloth Dynamic Q8. Let's read about GGUF. Ideally train Gemma with our own data only for a company-related prompt, example to look up certain things from a project. Oh yeah, in that case, 100%. Depends which Gemma you want to run. So there's a fair few questions I would ask is what, what do you want it to do? Um, do you want it to be like a generalist model you can just chat to? Um, do you want it to perform a specific task? What kind of data do you have? So there'd be a fair few questions I would ask for. Um, but definitely if you want to run it locally, it's a great model to run locally. So UnSloth does Dynamic 2.0 quantization. Okay, we're learning a lot here, so let's get We want the 4— this one. There we go, downloading. This should be quite quick, and then What am I doing? So we're trying to get Quen 3 VL to run with Llama CPP. Cuz we are doing benchmarks. Let's see if this works. Beautiful. That is quick. Images require vision models to be processed. Let's have a look at So we've got a series of tests that we're trying to run, but for some reason we can't upload an image. So I need a multimodal projector. Okay, cool. Hey, when are you going to start the LLM from scratch series streams? When would you like me to start it? I'm going to do one comparison video. Hey, do you know where I can find investors? I don't know where you can find investors. Let's check this out. Do we have image support? Nice! That's cool. So now we have an official way. Oh, time for 10 squats. That is really cool. So let's write down some notes here. So with Lama I really like your teaching style. That's why I'd like to learn LLM building. Plus, I really want to build on— well, and put it on my profile. What do I think about vibe coding? Vibe coding? I love vibe coding. It's really helpful to learn. Why aren't you using Cursor, Claude code right now? Because I don't necessarily need to. I will eventually try them out, but at the moment I haven't got a need for them. We're doing a lot of exploratory stuff, and plus I got Claude code right here. But Hussein, um, I appreciate that comment on the teaching style, but it'll take me— just let me let you know, it'll take a while to sort of get the materials ready, and then by the time we have a polished video, might be, might be a little while, but we'll definitely practice it live on stream, that's for sure. So I got some reading to do. GGF format. I wonder, is this working on my phone? Because we're on the same network, right? Let me go to localhost:30000 on my phone. Plexus AU, it would save the copy pasting to Claude. It's nice for it to always have your current context. You mean like it could use information from here and here? It would need to be the local IP of your computer, not localhost. Oh, you're right, but I'm gonna try it anyway so I get an error. Learn MLOps or ML security? They're, they're much of the same thing. Okay, so let me just try this. Okay, it's not working on the computer— on the phone. So, can perform inference with Llama CPP. Actually, I might have used the wrong IP address. Yeah, I'm also already using the host. One sec. Or maybe— I think it might be because I'm using port forwarding. Let me give that a shot. Wow! We got it working. Hey man, I got job watching your YouTube tutorial. Love from India. Oh, thank you very much. God bless you as well. Hello from India. What are we doing today? Just joined. We are doing some, uh, LLM inference benchmarking on the RTX 4090 and the NVIDIA DGX Spark, but we're kind of doing a bit of research at the moment on how best to run these models. And I just got it running on my phone, so let's go images. Please list all. So I don't know if you can see this, that's Llama CPP running on the phone. Now let's do it. So we've got this chat here. So this is it running on Safari, but if we press go, we also get it here. That is epic! Thank you, Plexus, appreciate the help. And then if I refresh this— wow, we're getting 40 tokens per second. So there's an LLM running locally and on the phone. Well, sorry, it's not running on the phone. I've connected to the DJI Spark from Okay, this is officially my goal now. We've got it running, an LLM running on— we've got a server running and we've connected to the server via the phone. But now my next goal is to get this same model running natively on the phone. So that's a longer term thing. But that's where we might have to look up MLX. Do we have examples? MLX examples? Here we go. MLX Swift examples. There's a few. So MLX is a way of running— yeah, here we go, run LLMs with Swift. It's a little bit convoluted, this repo. MLX Swift examples, MLX examples, MLX Swift. MLX Swift LM, MLX LM. We're just going to try them all. I don't think it's that. Here we go. MLX Chat Example. That's going to be a future stream. Really appreciate your help, Lexus. Okay, so let's— we need to write some potential setup. Because if we have a look at nvtop, what do we got left? Not actually using that much memory, which is good. On the contrast, let's see what happens if we were to use Quen 3 VL in Transformers and compare, compare the speed. Because we could now run Quen 3 VL GGUF— sorry, the 30B one. Where's the Q4? 17 gigabytes. And it's also got 32B. Ah, I see. Hardware compatibility. I wonder how many tokens per second we can get on— let's download this. Do they use the same Multi-modal projector. And then next we got to download the multimodal projector. So we're downloading another model now. We might be able to just run this locally if this gets a good throughput. See, what were we getting with— so let's write this down. So let's try our next model. We might get faster inference on this because it's got 3 billion active parameters rather than Nice, there we go. So now we've got Quen 3 VL 30B A3B, and so we want to use— which image do we use before this one? Now that it's cached, let's try again. So this is on the DJI Spark. Wow. Ah, it was cached to the prompt. Let's try another image, and this time we'll get it to do a few more things. Holy smokes! This is excellent. How much memory are you using here? This feels very similar to some of the work we do at Instacart. So GPU memory here is about How do you work at Instacart, Plexus? Oh, that's awesome. As in, like, I'm imagining you guys would, uh, basically enrich enriched datasets for search capabilities. Are you US-based or Australia-based? I see AU in your name. I didn't know, was— is Instacart in Australia or they, uh, US only? I don't know if I've used Instacart in Australia. Look at that. enrich catalog data with attributes and also do image gen. But some Coles in Melbourne have Instacart. Ah, the Instacart AI shopping trolleys. I've seen those. So this is a lot of what we want to do for Nutrify. So what one of my main goals is at the moment, we use Gemini for this, right? But I want to basically distill a model like Qwen3VL into a local model so we can just run it on device and we don't need to use Gemini. So if we can train a small VLM to deploy onto that. So this is the app my brother and I build. Um, the goal is to basically take one of these large models and distill it to be an on-device model, because then you have inference for free and it can run without an internet connection. I mean, not that you don't often need an internet connection these days, but you'd probably— example this, a great example is when you're in a shopping center or a Woolworths or a Coles or something like that. Um, I don't know about you, but whenever I'm, I don't know, looking up something and I'm like in 3 aisles deep, for some reason the signal in shopping centers is terrible. And so if you had, uh, an offline model, it would be able to run in a shopping center just locally on device. This is incredibly fast. Wi-Fi is dodgy in the store. We do local inference on the trolley. That is cool. So let's go Instacart trolley AI. This what you're talking about? Capacart. So do you work on these, or do you work on like the ML systems or the backend? You know what would be a cool video? Is to rebuild this just with some hacked together Raspberry Pis. Like, I would, I would retrofit a Raspberry Pi to like any shopping trolley, so something like this. Right, Raspberry Pi on that thing, and then scan Jetson Nano. Okay, hey, now we're talking. This guy's got cameras and screens connected to it. That's cool. Okay, so we can write down that we've got incredibly fast generation. So what do we get? This is on just an image in, image out. Thanks, Plexus. Appreciate this. So they're both fast on the reading, but then what about the output? Okay, we get about double the throughput, which makes sense, right? Because it's only got 3 billion active parameters versus 8 billion. 75 tokens per second. So this is with, uh, Llama CCP or CPP. No worries. Your recent video on multimodal RAG gave me some inspo for my SaaS. Oh, that's really cool. So okay, we got some experiments to run. I need to go for a walk, get some fresh air. I'm working on an AI-automated QA regression testing for enterprise desktop software. Software— you record your test in a Windows VM, AI segments the recording into steps, and then replays it on newer versions. That is cool. Taking UI screenshots plus step descriptions, and then I can group that together to form a graph of the UI states of the software and then can automatically build new test— new tests. 100%. If you've got screenshots and step descriptions, you can definitely use multimodal RAG for that. So let's do Next, implement, uh, inference with— how does it go against VLLM? Okay, VLM looks like it's faster. Yeah, given the pace that these push out updates, I'd be mindful that these comments are from 7 months ago. Yeah, that's why I kind of jumped. Off that. This is from November. I don't know if this— this kind of looks AI-generated. I don't know if it's correct. Maybe we're just going to have to try it. I'm not sure. Can you run VLM on NVIDIA DGX Spark. Okay, apparently you can. Oh, okay. Last updated. Wow, that's it. That's like a week ago. Okay, they don't have Quen 3 VL though. Wow, this is an absolute minefield. It's like, do you use Llama CPP? Do you use VLLM? Do you use native Transformers? Obviously Transformers is the best, uh, sorry, the easiest to get started, but it has significantly slower, I'm guessing, inference time. How to get good, good at PyTorch? Any suggestions? Read the documentation and just practice, practice, practice. And just go to like Claude or ChatGPT and go turn on learn mode. Help me learn PyTorch from a beginner, step by step, one concept at a time. Write something for me to repeat and think about before moving on to the next stage. Once I've nailed a concept, ask me if I'm ready to move on. My goal is to go from nothing to building an LLM like GPT-2 from scratch. I might actually do a live stream of doing this exactly. That could be a live stream. That's how you get good at PyTorch, would be to go through this, and then you would implement like a research paper, which is GPT-2, right? Okay, so we need some tests. We can do it on several different forms of inference. What we're going to probably try and do is I think that's a fairly good target to do. GLM 7 Flash— 4.7, sorry. Function Gemma as well. Okay, we've got some good direction for the next time. We can kind of combine this VLM inference into LLM inference. Okay, we'll finish off with some squats and then we'll end the stream. This is set 4. I'm feeling good. Okay, thank you all for tuning in. We did some— we actually didn't do too much benchmarking actually, but we discussed some options of what we're going to do and we played around uh, with llama.cpp, and we got it working. We even got it working on our phone. So, uh, we had some image generation models. I really, really liked those. So over the next couple of streams, what we'll do is we'll formalize all these tests and benchmarks, and then I'll get them running on the RTX 4090 as well. So, but by the end of this, we'll have 1, 2, 3, 4, 5 categories um, with several inference engines and two machines. So I think it'll be a pretty comprehensive test. So, um, thanks to everyone, and big shout out to Plexus. Appreciate it. Notifications on, will be good to have it go in the background and learn more with you while I work. Awesome, appreciate it. I'll see you all next time, eh? Have a good day.