Tim DaviesCEO Projekt Rising - The expert AI partner for SMEs

Differences between dense and sparse AI models

2 Apr 2024

Watch on YouTube

A short YouTube explanation (by Tim Davies) contrasting dense and sparse AI models. The speaker defines dense models as systems where most or all internal parameters/weights are active for each query, while sparse models activate only subsets of parameters relevant to the question—an efficiency and specialization tradeoff.

What's the difference between dense and sparse when we're talking about AI? So when we talk about these models, they are huge sets of weights within a file which is run on large compute to guess an output, to guess that output in sequence. The cat sat on the mat because it was flat or being extra guesses that the model is making. A dense model like GPT3 is effectively all of that brain fires every time it gets a question. Okay, which is good in that you're using the complete capability of that mini brain, if you like, but it's not efficient in terms of compute and it's not necessarily the best way of getting an answer. Sparse models, sections of that model fire in order to solve a question. So instead of the whole brain needing to work out, where am I now? You know, in a little bit of your brain, I'm sat in Stanford. And that doesn't take your whole brain to work out the answer to that question. Sparse models have that same principle, effectively, that's being answered with a subset of the data.