NVIDIA DGX Spark vs RTX 4090 | LLM inference, training speed and more

About a month ago, NVIDIA sent me a DGX Spark, and it's this beautiful little gold box on my desk with a GB10 inside. And I've got another deep learning PC which has an RTX 4090 GPU inside, which I've been using for the past 2 years or so for local AI development, client work, building models for our startup, you name it. And I wanted to see how the DGX Spark can pit against the RTX 4090. So naturally, I've been using it almost every day, but I wanted to put some real numbers behind it. And so I had an inkling of— because I've been using it, as you might have seen in the live streams— how it might perform. But these benchmarks show basically head-to-head on several different use cases, the use cases I'm mainly interested in for modern machine learning and AI work, how they go against each other. The NVIDIA DGX Spark has about this many CUDA cores versus the RTX 4090 has about this many CUDA cores. So on raw CUDA core count, which is compute processing, the RTX 4090 wins. And then on memory bandwidth, which is another important thing— by the way, if you want to see a blog post or a recent case study on the difference between how compute helps with modern LLMs versus memory bandwidth, I'd highly recommend the post by Exolabs. Basically, for pre-fill, you want high compute power in an LLM, and for decoding, you want high memory bandwidth. And so the RTX 4090 has both higher number of CUDA cores and a higher memory bandwidth, but the DGX Spark has 128GB of memory total, whereas the RTX 4090 has 24GB of VRAM. So what that means in practice is that on the DGX Spark, although you do run smaller models slower, you have the capacity to run much larger models such as GPT-OSS 120B, which just simply would not fit on the RTX 4090. So if you want a quick summary, I'll get into the actual numbers in a second. By the way, all the code's on GitHub. I'll leave the link below. You can see it and inspect the numbers there. But the RTX 4090 is like a Ferrari, right? It's fast, but it doesn't have much space. Whereas in terms of VRAM, but 24GB is enough to run most models, 8 billion parameters or so and under, and certainly plenty of capacity to run smaller encoder models that are around 200 million parameters or so, which are like the food vision models we use in Nutrify. But the RTX or sorry, the DGX Spark is more like a minivan. You've got plenty of space with the 128GB of memory, but it's not that fast in terms of if you wanted raw compute speed. Aside from just compute speed, if we're talking about footprint, the DGX Spark is unmatchable in terms of just plug and play. I've got it set up on my desk right next to my Mac Mini. And in my setup video, you might have seen that I plugged it in and within about half an hour, I was running deep learning jobs and AI models. So from unboxing to getting started with AI models, the DGX Spark is about an hour. Whereas the RTX 4090 requires an entire PC to be built around it. So you need to get a CPU, get some RAM, get some storage, get a case, get a power supply. And this, this takes a few hours. My friend Michael helped me build mine. So big shout out to Michael. Thank you very much. And that took about 3 to 4 hours on the hardware side of things. And then on the software side of things, the DGX Spark comes with basically all you need for modern AI and machine learning out of the box. Runs Ubuntu. My deep learning PC took about 2 to 3 hours to set up in terms of software. Now, I was running Windows with WSL2, so Windows Subsystem Linux 2 or something like that, but that caused me a lot of headaches. So I've just recently wiped my deep learning PC to run just pure Ubuntu, and it is much better. So yeah, maybe if you're looking to build in your own PC, probably just run Ubuntu if you want the most reliable option. Okay, so let's do a benchmark overview. We've got LLM and VLM inference with llama.cpp, which is an inference engine, as well as VLLM. We've got image generation model inference with Transformers or Diffusers. We have LLM fine-tuning with Hugging Face Transformers and TRL, and we have object detection model fine-tuning. So 2 inference benchmarks, 2 inference engines, and 2 fine-tuning benchmarks. All of these benchmarks are focused on speed. I'm not evaluating the quality of these models. I'm just focused on raw speed, whether that be tokens per second for input and output and overall training time for the fine-tuning benchmarks. Time for the results. If we go to LLM and VLM inference, and if we start off with VLLM as our inference engine, if you just looked at the raw specs of the hardware, these kind of transferred directly to the benchmarking results with the RTX 4090 being, uh, 3 to 4 times faster on input and output in terms of raw tokens per second. And so that's naturally attributed to the more CUDA cores and the higher memory bandwidth. Whereas of course the RTX 4090 couldn't run some benchmarks for certain models such as GPT-OSS-120B or Nemotron V3, because these just simply could not fit in the VRAM that the RTX 4090 offered at 24GB. Tokens per second, take this with a grain of salt, was slower on paper than the RTX 4090, but definitely still usable if you were using it in, say, a Llama CPP server. So if you're using it in like a locally running chat window, if you wanted to just interact with the model, you certainly can do that locally. And then of course, the DGX Spark really shines on just being able to run these large models at all. So previously over the last couple of years when using my deep learning PC, if a new model came out that was over 8 billion parameters, I would kind of neglect that if I wanted to run it locally, just because I knew I wouldn't have the, um, just the memory capacity to run it locally. So my models or my favorite models for the last couple of years have been around the 8 billion parameter mark and under. So if you want to translate this to a use case, you might use the DGX Spark to generate synthetic data or run a much larger model over a dataset. If you're doing these things overnight, it sort of doesn't necessarily matter of what the raw tokens per second are, because you can just leave it running on a large inference job. And then you could take those labels from a larger model, which generally is higher quality, and fine-tune a smaller model to reproduce those labels. Moving on to image generation, it was much the same story here. The RTX 4090 was about 3 to 4x faster on both of the models that I tested, which were just two of the popular ones on the Hugging Face models page. Uh, Flux Klein 4B, which is Apache 2.0, supposed to be quite fast. And then Z-Image Turbo, which is a 9 billion parameter model. Uh, the RTX 4090 definitely had more images per second there. But again, um, the DGX Spark could run much larger models such as Flux 2 Dev. However, I'll leave that testing for, uh, a future benchmark. So if we move on to LLM fine-tuning, this was from, uh, a recent video slash recent stream that I did where we fine-tune an LLM with Hugging Face Transformers and Hugging Face TRL, Transformers Reinforcement Learning, using supervised fine-tuning. It's a small language model, Gemma 3 270 mil, which I'm really loving the small language model space. And these days you can get such good results out of even small language models. So in terms of raw training time, again, similar pattern to what we're seeing here. It is a smaller model, 270 mil parameters. So the RTX 4090 just simply loved training on that. Again, 3 to 4x faster overall speed than the DGX Spark. But this is something to keep in mind that a lot of the time as a machine learning engineer, I'm not necessarily worried about raw training speed. My focus is I'm not building large extreme foundation models. I'm more focused on building specialized models for specialized tasks. So in the case of Nutrify, I build specialized computer vision models to deploy on device. For the case of a client I recently worked with, I was building specialized object detection models to run in an offline environment. And now much of my time was spent on crafting a dataset. And so the training, I would just run overnight, come in the next day and see how it went. So in this case, yes, the RTX 4090 is faster in raw training speed. But if you're running that overnight or over a couple of days where you don't necessarily need to interact with your computer, that kind of negates the sort of— is not as important that it's extreme, much faster in raw training speed. And if we move on to object fine-tuning, we have again a similar story to what we've seen in terms of raw compute speed. The RTX 4090 definitely wins. This is a relatively small dataset of 1,000 images for 10 epochs, but you can still get great results if you have high-quality data and a good base model with just 1,000 data samples. And that goes across image and text and LLM modern workflows. High-quality data with a good foundation model can get great results. So if we're going to summarize it and you're going to ask, which one should I get? Personally, if I was starting from no computers, I'd probably lean towards a DGX Spark because I love the ability to just plug and play and start running AI models. Training time, yes, you would take a hit on how long that takes. Inference time, yes, you do take a small hit. But in terms of just ease of use, the DGX Spark definitely wins there. And being able to run the much larger models that are coming out these days from 30 billion parameters right up to 120 billion parameters. That is really cool. But if you are after just raw training speed and you wanted something— hey, I don't need to run the latest and greatest 30 billion parameter models, I'm happy with 8 billion parameters and below— then the RTX 4090 is something you might want to opt for. But just keep in mind, you will need to build basically a whole PC around that because it's just a GPU card. It is not a full system like the DGX Spark. And again, price points. A DGX Spark is $3,999, but an RTX 4090 is this price. I'll put it on screen. I'll look up the, the correct price because I only know it in AUD. Um, but then you have to add all the peripherals, so they probably work out to be roughly the same price if you want, uh, quite a fast deep learning PC. They do, however, have a complementary workflow. For example, in my personal workflow, I'm using a large model such as Qwen3 VL32B Apache 2.0 model to label a large image dataset. I'll then take that labels from that to train a smaller model on the RTX 4090. So I could have them running in, in tandem. The DGX Spark just does inference for labeling because it can run a larger model. And then I'll pass all of those labeled samples to the RTX 4090 to fine-tune a smaller model that we eventually want to deploy in the Nutrify app so it can run locally. I've left a bunch of notes in the GitHub repo of different benchmarks and different other tests and different avenues and things I could have tried. Of course, these are biased towards the two GPUs that I have and the workflows that I have. So if you'd like to see anything in the future, please leave a comment. Or if you inspect the benchmarks, you're like, hmm, Daniel, you could have flipped this switch and it would have made things faster. Please let me know, and I'd love to hear that from you. Otherwise, happy machine learning, and I'll see you next time.