DeepSeek's new speculative decoding method, called DeepSpark, speeds up AI inference by 60 to 85 percent using a small draft model to predict tokens before verification. The approach works best for code and math but requires access to the target model's probabilities and a matching draft model.
What if an AI could see into its own future? Not perfectly, no. Just a little. Deepseek did something absolutely amazing in their new research work. You see, when you ask an AI how to rewrite a difficult email, it answers you one word, or one token at a time. That can get pretty slow. So, why not put out multiple words, like a whole sentence at the same time? That would be much faster, and that would be a game changer, because it would boost many slow AI systems into usability. Maybe one right on your phone, in your pocket. Well, yes. Except that it doesn't work. A big smart AI is like a senior editor. Brilliant, but expensive. If you ask it to write 5 words one by one, it still has to think 5 times. Slow, expensive. So, what do you do instead? The solution is unexpectedly brilliant. What is that? Well, you hire a junior writer. The junior writer writes the next few words quickly. Of course, these words are not always accurate, so you ask the senior editor to verify them. It looks and says yes, yes, yes, no, no. And everything after the first no is thrown away. If the junior writer guessed well, we get our 5 words for cheap quickly. Hooray! We call this speculative decoding. But, not so fast. There is a problem. The writer is not that experienced, it messes up. For instance, it forgets things, it starts a phrase and finishes another. And, I don't want this little writer to get fired, so here is DeepSeq's incredible new work. Called DeepSpark, with 3 new tricks to make this relationship work a bit better. 1. DeepSeq adds just a tiny bit of memory to the junior writer so it doesn't forget everything. With this, a drafted word can influence the next one. Doesn't need a huge brain, no, just enough for these few words not to fall apart. 2. You have to realize that some words are just doomed. If we ask, Hey little AI, what planet do we live on? And if the answer starts with lasagna, I think you can throw the rest away without looking. Although, it would be great. Yum yum. So here, DeepSpark estimates which words are unlikely to survive and saves the senior editor some time on these. And 3, DeepSpark also asks, is checking this extra word worth precious GPU time right now? Well, sometimes yes. If we talk about code or math, the next word may be very predictable. The writer is excellent there. But, hmm, if the prompt is open-ended, like writing a funny wedding speech, then, oof, there are a million good and bad answers. In those cases, the draft gets risky much faster. Maybe the first few words are fine, but the later ones are much more likely to fall apart. Ladies and gentlemen, today we are gathered here to celebrate the ancient potato. No. Just no. And DeepSpark predicts that in advance. That is amazing. So, what does all this magic get us? Now hold on to your papers, Fellow Scholars, because they report a 60-85% speedup on their own Flash and Pro models. And this is measured against MTP-1, their old production multitoken prediction baseline. Wow! But careful, Fellow Scholars. In the paper, you also get a number that says 661% throughput, heh, what about that? Now that is not the normal everyday speedup. That only happens in corner cases where the old MTP-1 system is running out of room. Just trying to help you avoid some pitfalls. In practical cases, we get 60-85% And that is amazing. And here is the best part. In principle, it can be implemented into many AI assistants. But not magically from the outside. No, you need a matching draft model, access to the target model's probabilities, and a serving system that can do this efficiently. So, once again, not a new smarter AI model. Therefore, you don't see many headlines about it. No. Instead, this makes a busy brain even faster. And we get all this for free. Open science! And I think that is amazing. What a time to be alive! Now, not even this technique is perfect. One, it is not a magic switch you can put into any closed API. We just noted that. But it gives you an AI that kind of sees a bit into its future. And two, the gains depend on the workload. Code and math, great. Open-ended chat, not so much. This is where that lasagna strikes back. I use Lambda to reproduce AI research papers, often in minutes. It's also great to train your own models or fine-tune an existing one. Run inference or text to image or video? Easy peasy. Running a DeepSeq chatbot or agent? Super fast, super reliable. Lambda gives you powerful NVIDIA GPUs to run your own experiments. I test ideas from the papers I cover, and moments later, results. Love it. Seriously, try it out now at lamda.ai/papers.
