An AI practitioner shares the interview question "What are you running overnight?" and explains why it matters. A serious AI system should run nightly sweeps testing cheaper models, prompt variations, and guardrails against 100-200 real examples, scoring each on correctness, cost, latency, and failure modes. This automated loop keeps systems improving without active monitoring.
What are you running overnight? This is what my friend who runs a company called 10x, he asked this during interviews. Now, not only did I learn during the interview process, it's not just show me what you built and let's nerd out on it. Show me your Codex, show me your Claude code, show me your Hermes, show me your OpenCLAW. Now it's what are you running overnight? Here's, we have an overnight eval loop here for every serious AI system build. So what is the pattern? It builds with the best model first. It captures 100 to 200 real examples and runs overnight sweeps, test cheaper models, prompt variants, routing logic, guardrails against the dataset, score on business outcomes, not just in accuracy. Correctness, cost per run, monthly cost, latency, failure modes, human review required, regression risk. The whole idea here is that you're constantly having this thing work even when you're not around. I got 2 things from it. Not only do I have a question I can ask in my interview questions now, now I have an overnight eval loop that I'm working on.
