DeepMind's Gemma 4 is a tiny multimodal model that runs on laptops by treating images and audio as token patches fed directly into one transformer, skipping separate vision and audio encoders. It's 99% smaller than typical models yet delivers strong image, video, and audio understanding, with over 300 million downloads.
Finally, something amazing happened. You see, today, AI models have become almost comically large. Some of the biggest open and free AI models, like Deepseek, reached over 1.6 trillion parameters in size. And that's not even the biggest one. So these cost hundreds of thousands of dollars to run, and then, you find out something crazy. You show them an image, and you ask, what does this image depict? And, you will be surprised to hear the answer. It doesn't know. Yes, it cannot see. Now, what if I told you that they promise you a model that is 99% smaller? A speck of dust comparatively. And yet, this tiny guy can see. That sounds impossible. Yes, maybe in our dreams, wishful thinking. But it actually exists. It is made by DeepMind and is called Gemma 4. This runs on your laptop. It is an absolute gem, free and open, downloaded more than 300 million times by us Fellow Scholars. That is insane. I love it. And then, something amazing happened. Now hold on to your papers Fellow Scholars, because—yup, they gave us the secret sauce. They finally told us the architecture they used for Gemma 4 to see. And it's kinda crazy. So, we finally understand how this can pull off things so easily, like talking as a medieval bard while identifying objects in your video. And so much more. So how did they add vision and multimodal reasoning into an unusually small local model? The secrets are finally out. Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér. So, a conventional AI is really several neural networks connected together. And if you have an image, you need a dedicated vision model just for that. Okay, what about audio? Giving it ears. Yup, that needs an audio encoder. A specialized part for each of these tasks. But this is not really looking or listening. This is just passing a translation around that is created by a different neural network. Their smallest models do that, but when we upgrade to the 12 billion model, things get crazy. Scientists at DeepMind say throw that all away. Out. Right now. Instead, hmm, it cuts your picture into small patches. Then, it projects those pixels directly into the model's internal representation. So, it knows where each patch came from. Okay, but what is the point here? Why do that? Well, you don't need a separate neural network, a vision transformer to interpret the image for you. No, throw it out! Same for audio. Slice it up into 40-millisecond chunks, and then comes the magic. You just pour all these tokens into your main transformer, and then what happens? Well, this system is forced to learn to be the eyes, ears, and brain at the same time. And I think this is one of the most important architectural ideas in Gemma 4. It removes hundreds of millions of specialist parameters, and it blurs the boundary between perception perception and thinking. And the result is an AI system that punches way above its weight. It handles images, it handles audio, and it is bloody smart. And this is something that you can own. Now, two more points. One, the Gemma 4 ecosystem continues to get improvements to make it faster and better. I'll give you a link to this in the description. Two, they shared the secret sauce there, so Gemma 4 is not just amazing in and of itself, but it can help Deepseek, and other systems learn to see better and more efficiently. That is absolutely amazing. What a time to be alive! And please, do not take it for granted that these amazing open models will just keep coming in the future. These are gifts to all of us. And these gifts may stop coming as capabilities increase. It is not a law of nature that we just get these models for free in the future too. So, to everyone who is working on open models, wherever you are in the world, you are heroes. You help scientists, students, and millions of other people to do their work better. Thank you so much. And we as a community have to come together and do everything we can to support these open systems. I use Lambda to reproduce AI research papers, often in minutes. It's also great to train your own models, or fine-tune an existing one. Run inference, or text to image or video? Easy peasy. Running a DeepSeq chatbot or agent? Super fast, super reliable. Lambda gives you powerful NVIDIA GPUs to run your own experiments. I test ideas from the papers I cover, and moments later, results. Love it. Seriously, try it out now at lambda.ai/papers.
