Hamel Husain shows how to build synthetic data when you don't have real users yet. Use structured inputs that vary features and personas, seed generation with real examples from actual use, enforce output schemas to filter junk, and avoid naive prompts that produce homogeneous results.
Looking at data is really important for evals, but what if you don't have any data? This is part 9 of a 12-part series on evals. So today we're going to talk about how you can generate synthetic data if you don't have any data. So there's some things you should keep in mind when generating synthetic data. The first thing is use structured input for diversity. Don't just prompt an LLM to say, hey, give me 10 outputs. Don't do that. Give it structured input. So think about dimensions that are important to vary. An example of such dimensions could be feature, persona, and scenario. Your dimensions are going to be specific to your product. The next thing you want to think about is you want to seed your generation with actual real examples. So those real examples can come from you using your application, trusted friends, design partners, but you want to have some real data in there as examples to make it as real as possible. Because what LLMs tend to produce can be overly polished and not indicative of what's going to actually happen. The next thing you want to think about is having structured output and enforcing that structure. And what you want to do is have a defined schema you want to output and then filter bad examples. So what you want to do is curate really good examples, especially ones that challenge your system and that are difficult. The last thing that you want to do with synthetic data generation is do a zero-shot request that just says generate 10 examples. That tends to give you homogeneous, uniform output that's not going to be as diverse as you need. If you enjoy learning about evals, take a look at the comments where I'll put more resources as well as a link to all of these flashcards.
