Hamel Husain walks through four ways to debug AI systems when you can't access the real data. Get permissioned access to a test subset, partner with domain experts who have clearance, redact sensitive information from model traces, or use synthetic data as a last resort. Real data inspection is irreplaceable, but these alternatives work when it's off limits.
There is no replacement for looking at real data. So the fact that you can't look at data is not ideal, but there's several things you can do about it. Number one, try to find real data that you have the permission to look at. A customer might agree to give you access to test users or test datasets or a certain subset of users that you are allowed to look at. Even limited access is better than nothing. If you can't inspect the data yourself, then the next best thing you can do is work with domain experts that have authorized access to the data. This might be lawyers, doctors, etc. And if you do this, you need to make sure that your product makes it easy for those experts to verify the correctness of the AI outputs. The next thing you can do is redact or edit the traces to remove the sensitive information. There's a lot of different approaches to doing this. There are lib— software libraries that help with this. Of course, you can get an LLM to do this for you. You have to rigorously test this to make sure that the sensitive data is in fact being stripped out. If none of those options are available, your last resort is to synthetic data. It's good to know that synthetic data is very limited in its ability to catch errors. So it's usually only good for catching those initial low-hanging fruits, but it's better than nothing. If you want to read more about these topics and how to deal with sensitive data, check out this link so you can learn more.
