Reliability of AI Agents: Rater Effects, Drift, and the Return to an Evaluation Program
This working paper investigates how firms can continuously evaluate the reliability of deployed AI agents by using repeated human rubric scores on production voice‑and‑video interviews. Analyzing 2,611 interviews collected between March and August 2026, the authors model reliability as a latent state observed with rater‑specific noise. They estimate reviewer fixed effects (e.g., Reviewer B scores ~0.79 SD lower than Reviewer A) and fit a local‑level state‑space model to quantify drift (innovation variance q). The study examines three production deploy regimes, showing that forecast uncertainty reaches the magnitude of the largest observed deploy contrast in roughly five weeks. Table 10 disaggregates trends by rubric family, revealing significant improvements in conversation structure, latency, A/V sync, and interview outcome after accounting for reviewer and regime fixed effects, while other families show negligible change. Operational corroboration comes from support‑ticket data, which indicate a ~10 % monthly reduction in interview‑related tickets relative to technical‑issue tickets. The paper concludes with three practical checks for production evaluation: (1) ensure score comparability across raters, (2) assess evidence currency given drift, and (3) validate evaluation changes against consequential operational outcomes.
Topics
Rater Fixed Effects and Score Normalization
Quantifies systematic differences between human reviewers (e.g., Reviewer B scores ~0.79 SD lower than Reviewer A) and applies normalization to make rubric scores comparable across raters. This adjustment underpins the first of three production‑evaluation checks—ensuring rater comparability.
State‑Space Drift Modeling and Evaluation Interval Guidance
Uses a local‑level (random‑walk) state‑space model to treat AI‑agent reliability as a latent state observed with rater‑specific noise, estimating the innovation variance q that captures temporal drift. The drift estimates inform practical recommendations on how frequently agents should be re‑evaluated in production.
Deploy‑Regime Fixed Effects, Rubric‑Family Trends, and Forecast Uncertainty
Separates performance changes attributable to distinct production deploy regimes and examines family‑level rubric trends (Table 10). Highlights significant improvements in conversation structure, latency, A/V sync, and interview outcome after accounting for regime and reviewer fixed effects, and shows that forecast uncertainty reaches the magnitude of the largest deploy contrast in roughly five weeks.
Operational Corroboration through Support‑Ticket Metrics
Links evaluation findings to real‑world outcomes by demonstrating a ~10 % monthly reduction in interview‑related support tickets relative to technical‑issue tickets, providing empirical validation for the operational impact of the observed performance gains.