Dylan PatelFounder and Chief Analyst, SemiAnalysis · The semiconductor-AI stack

Makora's Sequential Monte Carlo Speculative Decoding keeps multiple draft tokens alive instead of re

6 Jun 2026 · 1 min

Watch on YouTube

1:23

Dylan Patel explains Sequential Monte Carlo Speculative Decoding, a new inference algorithm that keeps multiple draft tokens alive at once instead of discarding them. It scores all drafts with importance sampling, kills the weak ones, and duplicates the strong ones to avoid expensive rewinds and boost throughput.

Chapters

One new inference algorithm that we're very excited about is called Sequential Monte Carlo Speculative Decoding. And so the way that works is that, you know, in normal speculative decoding, you have a draft model. So you draft one token at a time, and then you take K tokens and you verify them in parallel with a larger model. And that's how you get speedup because you're using that large model in parallel instead of autoregressively. However, one problem is that any tokens that don't match the target model, You rewind the draft and then you start again. And so this rewinding is just a big performance hit. In sequential Monte Carlo, what we do instead is that we keep N drafts alive. So we have N drafts going in parallel. And instead of verifying with the target model and looking for an exact match, we're actually just scoring those N drafts with the target model. And based on that importance sampling, based on that importance score, we do some kind of sequential Monte Carlo sampling where we can evict the ones with a very low score and duplicate the ones, the drafts, with a very high score.
So if I'm understanding correctly, increasing the number of draft tokens, well, but more specifically in parallel, while guaranteeing a higher acceptance rate?
Yeah, you're exactly right.