When Should Humans Step In? Optimal Human Dispatching in AI-Assisted Decisions

Abstract

AI systems increasingly assist decision making by producing cheap factor-level assessments of complex inputs, but these assessments can be biased or incomplete. We study selective human evaluation: given AI signals, which factors should be dispatched to human experts? We formulate this human--AI collaboration problem as a factor-level information-acquisition problem. Under squared loss, the optimal dispatching rule maximizes a contextual reward; under a linear model, this reward admits a closed-form decomposition into two interpretable terms: predictive relevance and residual uncertainty in the human evaluation after conditioning on the AI signal. We instantiate the framework on peer review, decomposing papers into ten aspects and evaluating up to 3,408 ICLR submissions across three LLMs and multiple regression heads. In this retrospective proxy, where human aspect signals are extracted from existing reviews, predictions that query three of the ten aspects under the optimal dispatching rules match the full ten-aspect human benchmark. The finding holds when the human signal comes from a single noisy reviewer, and across review years. These results position principled dispatching as a practical building block for scalable human--AI decision systems. We also discuss safeguards under which such a rule can serve as decision support that leaves the final decision with a human.