Hamel Husain warns that LLMs generating synthetic data tend to produce homogeneous examples that follow the same patterns, masking errors because similar inputs get similar outputs. You need to deliberately diversify inputs across multiple dimensions instead of relying on a single LLM source.
The things that we teach in the class, like, okay, you wanna be thoughtful, like, what dimensions to explore. If you put a large language model across some, like, initial examples, your expansion is gonna be, like, very homogeneous. You're not gonna really be, like, exploring.
The first thing I say in lecture, when you generate synthetic queries, do not just ask an LLM for synthetic queries because it's gonna give you queries that all sound the same. Then you're gonna run your system prompt on it and all the outputs are gonna sound more or less the same and you'll be like, there are no errors. And then, well, yeah, of course, because you didn't really cover any sort of diversity of inputs.
