In this Two Minute Papers episode, Dr. Károly Zsolnai‑Fehér examines Google’s newly announced TurboQuant method for compressing the KV‑cache (short‑term memory) of large language models. The presenter first outlines the media hype – claims of 4‑6× memory reduction and 8× attention speedup with no quality loss – and notes the broader context of rising hardware costs. He then breaks down the three core techniques behind TurboQuant: (1) quantization (truncating low‑precision values), (2) random rotation of vectors before quantization to spread energy evenly, and (3) a Johnson‑Lindenstrauss (JL) transform to preserve pairwise distances after dimensionality reduction. All three ideas are decades‑old, but TurboQuant combines them in a novel pipeline. The video emphasizes the importance of a formal proof and reproducibility. Subsequent independent reproductions confirm the method works, but the actual gains are more modest: 30‑40% KV‑cache memory savings and roughly 40% prompt‑processing speedup, rather than the advertised 4‑6×. The presenter cautions against over‑hyping the results, likening them to idealized benchmark conditions. He also discusses controversy: several researchers argue TurboQuant overlaps with prior work and question whether the novelty claim was fully addressed, though the paper has been accepted for publication. Finally, the episode highlights practical implications for users running long‑context LLMs (e.g., large PDFs, codebases), where a few gigabytes of memory can be saved, making AI deployment cheaper. The video ends with a sponsor plug for Lambda GPU cloud.
Chapters
Google made a huge announcement about their new method that lets us run AI techniques cheaper. The news took the world by storm. This came at the best possible time, because we have a worldwide memory shortage. So, the prices for capable laptops and GPUs and anything that can run these AI systems is up by an insane amount. And this work would make them much cheaper to run. They call it TurboQuant. Roughly speaking, they claim 4 to 6 times less memory. That is insane. And 8 times faster computation for a part of the neural network called attention. No meaningful loss in output quality. And it works on top of existing models as is. Whew. If true, that is a total game changer. The news was so huge, it even moved the stock price of huge semiconductor companies. Because of that, I did not want to publish an early video on this huge sensation. I really wanted to wait a bit and find out whether it actually works in practice. I'll tell you about that. And I'll also tell you that not everyone is happy about it. So, 3 questions: what does it do, does it work, and 3, what is the controversy about? Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér. Man, it feels so good to do it like this. Well, this compresses the KV cache of AI systems, like large language models. This is the short-term memory of an AI assistant. If you would look into that, you would see tons and tons of these numbers. These numbers relate to what you are currently talking about. Movies, a bunch of documents, or a huge codebase. Now, personally, I am a research scientist. What caught my eye was not the media hype, but This. Ooh, a formal mathematical proof that it works. Now we're talking! Scientists propose that we chop off the end of the numbers to save memory. Is that a new idea? No. Is that a good idea? No. Unless you are very careful. Because you can lose a lot of information, and your neural network might output Nonsense. So, how do you do that? Well, imagine a vector. This is like an arrow pointing somewhere. Sometimes that arrow points mostly along one axis. This is the axis. So most of its energy is in one direction, and a little in other directions. And when you chop that information off, it snaps to the grid. You basically lose everything. Except that one direction. That is not useful. Now here is a brilliant idea. Before chopping it off, rotate the arrow in a random direction. Now, the energy spreads more evenly across all directions. So when you round off parts of it, you lose a little from everywhere, instead everything from most places. The result? Much less information lost. Is this idea new? No. This is a very old idea. Now, they do one more thing. They use the Johnson-Lindenstrauss transform to compress the data. What is that? Remember, we have a bunch of numbers representing arrow directions. And we want fewer numbers to describe these directions. But very carefully. You do that in a way that guarantees that the distances between these arrows is roughly the same after squishing. If you want to sound really cool, just call it the JL transform. Is that new? Not really. 40-year-old technique. And I think this is the key. Everyone loves to invent shiny new stuff. But here, quantization is not new. Rotating things around is not new. This transform is not new. These are three age-old ideas combined together to great effect. Look, sometimes you don't need to invent grand new theories. Sometimes you need a smart combination of existing methods. Now, to conclude that it really works in practice, I wanted to see other scientists reproduce the technique and benchmark it for themselves. This is why the video appears later than most others, but I think it makes it more truthful. So, were other scientists able to reproduce this technique? Yes. Did they also benchmark it? Yes. Does this technique help? Yes. But, not so fast. The first tests reveal that it decreased the memory cost of the KV-Cache short-term memory by 30-40%. That is fantastic. I would have been very happy with this, but it doesn't end there. Typically, you have a tradeoff where you decrease memory usage at the cost of something. So something needs to slow down. Now, hold on to your papers, Fellow Scholars, because it also sped up processing the prompts by about 40% as well. Huh? What? That? My brain crashed. We get faster assistants that need less memory at almost zero cost. That is insane. In a world where it's harder and harder to own things, this is a blessing. Thank you so much! It is also remarkable that the paper has barely been out for a week, and some of you Fellow Scholars already coded it up. Nice work! Link is in the description. So, it's not quite like the media says. Based on the results, we cannot conclude that every AI machine suddenly needs 6 times less RAM. No, that is a bit idealistic and only true for some corner cases. You know, when you see an official benchmark of a phone battery or electric car mileage with somewhat idealized conditions, It is a bit like that. So, careful with the media hype. But experienced Fellow Scholars like you know that in your mind, you have to tone these numbers down a little. This is why we wait for more data and analyze experiments here to get the highest quality information for you. But it's still good. Really good. It helps most people who run AI systems with very long contexts. When you chuck in a huge PDF document, or a movie, or a huge codebase for an AI to analyze, yes, you will be able to do that cheaper, with meaningfully less memory. Often, a few gigabytes less. Wow. And I think that is absolutely amazing news. Third, I will note that other researchers point out that the paper overlaps with previous techniques. They felt that it has similarities that should be discussed more thoroughly. And there was more. Eventually, the paper was accepted for publication, though not all researchers agree the concerns were fully addressed. I put a link to all of these in the video description. But this proves that even in modern AI, there are still basic things we haven't invented yet. And that makes this a very exciting area to be in. What a time to be alive! And if you agree that this is the way of talking about papers, please consider subscribing and hitting the bell icon. Here you see me running the full Deepseek AI model through Lambda GPU cloud. 671 billion parameters running super fast and super reliably. This is insane. I love it, and I use it on a regular basis. Lambda provides you with powerful NVIDIA GPUs to run your own chatbots and experiments. Seriously, try it out now at lambda.ai/papers or click the link in the description.
