Hamel Husain shows how to test document search systems without manual work: have an AI generate a question for each document that only that document answers, then feed those questions back into your search engine to see if it retrieves the right result. It's fast and catches major problems, though not every edge case.
If you're building AI that searches through documents, here's a quick way to test it without writing hundreds of test questions or labeling a bunch of examples by hand. What you want to do is flip the problem. Point an AI at each document and say, make up a plausible question that only this document can answer. Now you've got a question paired with the document it came from. You can now feed that question back into your search and measure, did it find the right document? You just turned a hard problem, which is having a bunch of label data, into a much easier one. Now this won't catch everything, and it isn't a perfect test, but it's a fast and easy test that surfaces big problems, and it's a really high return on investment given the amount of effort it takes. If you want to learn more techniques like this, I left some additional resources in the comments.
