AI labs spend most of their effort on training data quality, not model size. StableLM's performance tanked when too much Reddit data went in. The real question now: can synthetic data from reinforcement learning replace real-world text entirely?
Chapters
The quiet work of extracting, filtering, and curating what the models read has driven 3x more of the gains.
You are what you eat, right? I remember when we trained StableLM, one of the first open-source big large language models, we had too much Reddit data and it broke the scaling curves because it turned a bit stupid and nasty. So it really does matter what's in there. What every lab is doing is just this constant train to get the best input data, the general ingredients. And then what post-training is, is the garnish. And so when you think what pre-training actually is, it's a pressure cooker. So you're moving the latents back and forth, you're tenderizing the meat, like literally you're breaking down these kind of very static bonds between them, finding the latent spaces. So this is really not a surprise. In fact, you could argue it's almost all data because when you distill a model, what do you do? The input from one model to the other is literally digested compressed data, right? And so that's why now one of the big questions we had was, could you train on synthetic data? And RL environments, things like that, and the latest leap forward is giving some indication of that one.
