Hamel Husain and a co-speaker break down how to test a document question-answering system by isolating three stages: OCR, retrieval, and answer generation. They built a 77-page lab with 22 questions and bounding-box citations for instant visual verification. OCR emerged as the main bottleneck, causing ten-fold slowdowns and most errors.
Let's actually open one of your blog posts. Let's try that.
Yeah, write a Twitter post about this blog post and focus on the pieces of the question answering pipeline that can be tested. Okay. I asked a model, what's the underwriting fee on this claim estimate? It said checking that by rereading the document saves no time. Every answer cites a bounding box on the page. Once checking takes seconds, So you can test each stage of the pipeline separately. Split it into 3 parts and swapped one at a time across 22 questions. Reading, the OCR. Finding, GREP versus semantic search. Answering, the prior harness. I actually built a small lab for question answering over documents. That's 77 pages, 22 questions, citations that draw a box around the exact spot on the page where the answer came from. So checking an answer takes seconds instead of a reread. Cheap checking means he can swap out one stage of the pipeline at a time, rerun the same questions, and see what changed. The pipeline has 3 stages, and in case each one of its own reading, placing the OCR model with raw page images and letting the agent run its own OCR. Tesseract and ImageMagick fixed 3 questions. OCR failed. Each question took 10 times longer though. Every failure in the text runs traced back to the OCR stage, and the citations made that easy to see.
I don't like it.
Yeah, it's pulling the reading, finding, and answering steps probably from the post, but I don't think it explains it very well. I like B better so far. A model answers 1,097 for the underwriting fee on a loss estimate. Is that right? If you have to reread the whole document to find out, the answer saved you nothing. Citations as bounding boxes with the OCR text drawn as an orange rectangle in the cage. Checking takes seconds. Once checking is cheap, experiments are cheap. That lets him split the pipeline of 3 testable stages, swap one at a time against the same. Any questions? I feel like it's very similar to the Unmannered, but a little bit more wordy so far. I think the unmannered one did a little bit better because it said 17 over 22 versus 14 over 22. That made it a little bit easier to know. And it also included one last fact about the cost, which is nice. And then like the finding, grep versus semantic search, same accuracy, cheapest run, is a lot better, I think, than grep versus semantic search over the same OCR output, same accuracy, cheapest run. The bottleneck was never search. I mean, it's true, but it's like too wordy. It's not good. I think B is the best. I think so too.
