Human-in-the-loop AI in government: a case study
Open accessAbstract
In this paper, we present a novel application where Human-Computer Interaction (HCI) meets Artificial Intelligence (AI) and discuss obstacles that need to be resolved on the long journey from research to production. Unlike academia and industries that have been at the forefront of automation for decades, government is a new player in the field, though an important one. We build systems that are used on a large scale, we collect data to inform policymakers. Using the example of the Household Budget Survey, we demonstrate how government agencies can apply Human-in-the-Loop AI to automate the production of official statistics. The aim is time and resource saving on repetitive, labour-intensive tasks which machines are good at, allowing humans to focus on value added tasks requiring flexibility and intelligence. One major challenge is the human factor. How will the users, who are accustomed to manual tasks, react to the complexity of AI? How should we design the interface to give them a good user experience? How do we measure success? Indeed, one key step towards production is to secure funding, which requires presenting potential success in a way that the stakeholder can understand. Stressing the importance of formulating problems from a practical business viewpoint, we hope to bridge the communication gap and help the research community reach out to more potential users and help solve more novel real-world problems.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Workload management in HITL systems involves not only monitoring current workload but also designing task allocation strategies that maintain sustainable human engagement.
“Workload management in HITL systems involves not only monitoring current workload but also designing task allocation strategies that maintain sustainable human engagement.”
From Human-in-the-Loop Artificial Intelligence: A Systematic Review of Concepts, Methods, and Applications (2026) · cites this paper as [112]
The Introduction describes partitioning tasks between humans and machines and designing handovers. The User Interface section also emphasizes low cognitive load and keeping coders active in the workflow. Our copy does not describe monitoring current workload or task-allocation strategies specifically designed to sustain human engagement; the claim goes beyond what the paper says. (LLM, conf=0.84)
Experimental — accuracy not yet measuredBenedikt et al. found that early deployments underestimated human workload demands, leading to bottlenecks in system performance.
“The study revealed that early deployments were found to underestimate human workload demands, leading to bottlenecks in system performance.”
From Human-in-the-Loop Artificial Intelligence: A Systematic Review of Concepts, Methods, and Applications (2026) · cites this paper as [112]
The User Interface section discusses human needs and possible effects on productivity, while the Conclusion and Future Work section says larger-scale tests and a production decision are still planned. Searches for early deployments, underestimated human workload, and bottlenecks found no passage reporting the claimed finding. The indexed passages provide no page numbers. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016 · see citing works
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · see citing works
Enriching Word Vectors with Subword Information
Bojanowski, P., Grave, E., Joulin, A., & Mikolov, T. (2017) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00051 · Cited by 9,645 · see citing works
Scikit-learn: Machine Learning in Python
(2011) · Journal of Machine Learning Research · doi:10.5555/1953048.2078195 · Cited by 8,539 · see citing works
The random subspace method for constructing decision forests
Ho, T. K. (1998) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.709601 · Cited by 6,874 · see citing works
Norman, D. A. (2016) · Vahlen eBooks · doi:10.15358/9783800648108 · Cited by 6,390 · see citing works
A study of the behavior of several methods for balancing machine learning training data
Batista, G. E. A. P. A., Prati, R., & Monard, M. (2004) · SKDD · doi:10.1145/1007730.1007735 · Cited by 4,271 · see citing works
A model for types and levels of human interaction with automation
Parasuraman, R., Sheridan, T. B., & Wickens, C. D. (2000) · IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans · doi:10.1109/3468.844354 · Cited by 4,195 · cited 1× · see citing works
“Examples include flight simulators[45], helpdesk chatbots[7], photo tagging[36], self-driving cars[16], to name only a few.”
BackgroundAn Overview of the Tesseract OCR Engine
Smith, R. W. (2007) · Proceedings of the International Conference on Document Analysis and Recognition · doi:10.1109/icdar.2007.4376991 · Cited by 2,457 · see citing works
API design for machine learning software: experiences from the scikit-learn projectPreprint
Buitinck, L., Louppe, G., Blondel, M., Pedregosa, F., Mueller, A., Grisel, O., Niculae, V., Prettenhofer, P., Gramfort, A., Grobler, J., et al. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1309.0238 · Cited by 2,387 · see citing works
Beyond Accuracy, F-Score and ROC: A Family of Discriminant Measures for Performance Evaluation
Sokolova, M., Japkowicz, N., & Szpakowicz, S. (2006) · Lecture Notes in Computer Science · doi:10.1007/11941439_114 · Cited by 1,974 · see citing works
(2010) · doi:10.5555/1756006.1859921 · Cited by 1,924 · see citing works
United Nations Statistics Division
(2006) · Choice Reviews Online · doi:10.5860/choice.43sup-0473 · Cited by 1,202 · see citing works
Deep Learning using Linear Support Vector MachinesPreprint
Tang, Y. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1306.0239 · Cited by 891 · see citing works
Robi Polikar, A. (2012) · Springer eBooks · doi:10.1007/978-1-4419-9326-7_1 · Cited by 812 · see citing works
A Framework for Reasoning About the Human in the Loop
Cranor, L. F. (2018) · Figshare · doi:10.1184/r1/6620651 · Cited by 256 · see citing works
Tree Boosting With XGBoost - Why Does XGBoost Win "Every" Machine Learning Competition?
Nielsen, D. (2016) · BIBSYS Brage (BIBSYS (Norway)) · Cited by 230 · see citing works
Information-Based Evaluation Criterion for Classifier's Performance
Kononenko, I., & Ivan Bratko, A. (1991) · Machine Learning · doi:10.1023/a:1022642017308 · Cited by 208 · see citing works
LOCAL GOVERNMENT STRUCTURE AND PERFORMANCE: LESSONS FROM AMERICA?
BOYNE, G. A. (1992) · Public Administration · doi:10.1111/j.1467-9299.1992.tb00942.x · Cited by 187 · see citing works
Deep Learning using Support Vector MachinesPreprint
Tang, Y. (2013) · arXiv (Cornell University) · Cited by 184 · see citing works
Performance Evaluation in Machine Learning: The Good, the Bad, the Ugly, and the Way Forward
Flach, P. A. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33019808 · Cited by 159 · see citing works
The Fourth Annual Test of OCR Accuracy
Rice, S. V., Jenkins, F. R., & Nartker, T. A. (1995) · Cited by 143 · see citing works
Synthesis for Human-in-the-Loop Control Systems
Li, W., Sadigh, D., Sastry, S., & Seshia, S. (2014) · International Conference on Tools and Algorithms for Construction and Analysis of Systems · doi:10.1007/978-3-642-54862-8_40 · Cited by 128 · see citing works
Understanding Chatbot-mediated Task Management
Toxtli, C., Monroy-Hernández, A., & Cranshaw, J. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173632 · Cited by 127 · see citing works
The early origins of the logit model
Cramer, J. S. (2004) · Studies in History and Philosophy of Science Part C Studies in History and Philosophy of Biological and Biomedical Sciences · doi:10.1016/j.shpsc.2004.09.003 · Cited by 104 · see citing works
Mitigating the paucity-of-data problem
Banko, M., & Brill, E. (2001) · doi:10.3115/1072133.1072204 · Cited by 95 · see citing works
Fast and compact regular expression matching
Bille, P., & Farach-Colton, M. (2008) · Theoretical Computer Science · doi:10.1016/j.tcs.2008.08.042 · Cited by 71 · see citing works
A comparison of standard spell checking algorithms and a novel binary neural approach
Hodge, J., & Austin, J. (2003) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2003.1232265 · Cited by 66 · see citing works
Contextual spelling correction using latent semantic analysis
Jones, M. P., & Martin, J. (1997) · Conference on Applied Natural Language Processing · doi:10.3115/974557.974582 · Cited by 65 · see citing works
Unsupervised Transfer Learning for Spoken Language Understanding in Intelligent Agents
Siddhant, A., Goyal, A. K., & Metallinou, A. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33014959 · Cited by 50 · see citing works
Effective Spell Checking Methods Using Clustering Algorithms
de Amorim, R. C., & Zampieri, M. (2013) · Cited by 44 · see citing works
Improving optical character recognition performance for low quality images
Brisinello, M., Grbić, R., Pul, M., & Anđelić, T. (2017) · International Symposium ELMAR (ELMAR) · doi:10.23919/elmar.2017.8124460 · Cited by 35 · see citing works
Human-in-the-Loop Simulations: Methods and Practice
Rothrock, L., & Narayanan, S. (2011) · Digital Access to Libraries (Université catholique de Louvain (UCL), l'Université de Namur (UNamur) and the Université Saint-Louis (USL-B)) · doi:10.1007/978-0-85729-883-6 · Cited by 28 · see citing works
Designing the Human in the Loop of Self-Adaptive Systems
Gil, M., Pelechano, V., Fons, J., & Albert, M. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-48746-5_45 · Cited by 27 · see citing works
Fivez, P., Šuster, S., & Daelemans, W. M. P. (2017) · Computational Linguistics in the Netherlands Journal · doi:10.48550/arxiv.1710.07045 · Cited by 14 · see citing works
Improving Optical Character Recognition Techniques
Ramesh, N., Srivastava, A., & Deeba, K. (2018) · International Journal of Engineering & Technology · doi:10.14419/ijet.v7i2.24.12085 · Cited by 7 · see citing works
Fast and Compact Regular Expression MatchingPreprint
Bille, P., & Farach-Colton, M. (2005) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/0509069 · Cited by 7 · see citing works
Value for money in defence: Strategic choices and efficiency savings
Hartley, K. (1986) · Chartered Institute of Public Finance and Accountancy Public Money · doi:10.1080/09540968609387356 · Cited by 3 · see citing works
Respondent Burden: A First Measurement Effort
Sharp, L. M. (1981) · VS Verlag für Sozialwissenschaften eBooks · doi:10.1007/978-3-322-87749-9_17 · Cited by 2 · see citing works
Flagged first, then by citations
Research works printed but not linked: 4
- [10]10. Wenyuan Dai, GuiRong XueQiang, and Yang Yong Yu. 2007. Trans-ferring naive Bayes classifiers for text classification. Proc. of Assoc. for the Adv. of Art. Int. (AAAI 07) (Aug. 2007). http://new.aaai.org/Papers/AAAI/2007/AAAI07-085.pdf
- [19]19. Brink Henrik, Richards Joseph, and Fetherolf Mark. 2016. Real-World Machine Learning. Manning Publications Co. 1sr Ed. (Jan. 2016). https://doi.org/1617291927, 9781617291920
- [33]33. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representa- tions. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (June 2018). https://arxiv.org/pdf/1802.05365.pdf · arXiv
- [39]39. Ingersoll Grant S., Thomas S. Morton, and Andrew L. Farris. 2013. Taming Text: How to Find, Organize, and Manipulate It. Manning Publications Co. (Jan. 2013). https://doi.org/193398838X,9781933988382
Cited by
Sorted by citations
A survey of human-in-the-loop for machine learning
Wu, X., Xiao, L., Sun, Y., Zhang, J., Ma, T., & He, L. (2022) · Future Generation Computer Systems · doi:10.1016/j.future.2022.05.014 · Cited by 800 · see citing works
Constructing Ethical AI Based on the “Human-in-the-Loop” System
Chen, X., Wang, X., & Qu, Y. (2023) · Systems · doi:10.3390/systems11110548 · Cited by 45 · see citing works
Exploring citizens’ stances on AI in public services: A social contract perspective
Schmager, S., Grøder, C. H., Parmiggiani, E., Pappas, I., & Vassilakopoulou, P. (2024) · Data & Policy · doi:10.1017/dap.2024.13 · Cited by 35 · see citing works
Mapping Citizen Science through the Lens of Human-Centered AI
Rafner, J. F., Gajdacz, M., Kragh, G., Hjorth, A., Gander, A. J., Palfi, B., Berditchevskiaia, A., Grey, F., Gal, K., Segal, A., et al. (2022) · Human Computation · doi:10.15346/hc.v9i1.133 · Cited by 32 · see citing works
Lazaros, K., Vrahatis, A. G., & Kotsiantis, S. (2026) · Entropy · doi:10.3390/e28040377 · Cited by 50 · see citing works
Toward Value Scenario Generation Through Large Language Models
Jung, H., Seo, W., Song, S., & Na, S. (2023) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3584931.3606960 · Cited by 13 · see citing works
Enabling people-centric climate action using human-in-the-loop artificial intelligence: a review
Debnath, R., Tkachenko, N., & Bhattacharyya, M. (2025) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2025.101482 · Cited by 12 · see citing works
Lee, J., Jung, K., Newman, E., Chow, E., & Chen, Y. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713244 · Cited by 9 · see citing works
Dong, M., Bonnefon, J., & Rahwan, I. (2025) · Nature Communications · doi:10.1038/s41467-025-62440-3 · Cited by 6 · see citing works
Governing AI through interaction: situated actions as an informal mechanism for AI regulation
Papyshev, G. (2024) · AI and Ethics · doi:10.1007/s43681-024-00446-1 · Cited by 6 · see citing works
Food Assistant for Consumer Behaviour Change through Citizen Science and AI
Casado–Mansilla, D., Gómez–Carmona, O., Fernández–de–Retana, M., Muzzioli, L., Kušar, A., Vandevijvere, S., & López–de–Ipiña, D. (2024) · International Conference on Smart and Sustainable Technologies (SpliTech) · doi:10.23919/splitech61897.2024.10612651 · Cited by 7 · see citing works
Semi-Automatic Dataset Annotation Applied to Automatic Violent Message Detection
Botella-Gil, B., Sepúlveda-Torres, R., Bonet-Jover, A., Martínez-Barco, P., & Saquete, E. (2024) · IEEE Access · doi:10.1109/access.2024.3361404 · Cited by 5 · see citing works
Human-in-the-Loop Optimization for Artificial Intelligence Algorithms
Farhood, H., Saberi, M., & Najafi, M. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-14135-5_7 · Cited by 3 · see citing works
Human-Centered Machine Learning
Schillizzi, C. (2025) · Springer eBooks · doi:10.1007/978-981-95-3658-0_25 · Cited by 2 · see citing works
False Consensus Biases AI Against Vulnerable StakeholdersPreprint
Dong, M., Bonnefon, J., & Rahwan, I. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4876227 · Cited by 2 · see citing works
What Do We Already Know? Towards Mindful AI for Climate Change Policymaking
Berns, K., Marttila, T., & Gaziulusoy, İ. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3720060 · Cited by 1 · see citing works
AI in Government: Future Trends and Recommendations
Azhar, A. (2026) · Springer eBooks · doi:10.1007/978-3-032-12344-2_16 · Cited by 1 · see citing works
IoT Sensor Data Analysis and Fusion
Sohail, M. (2020) · Studies in computational intelligence · doi:10.1007/978-3-030-52067-0_17
Combining Document Understanding and Action Center in UiPath for Human-In-The-Loop Claims ProcessingPreprint
Kotla, P. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5379346
Transforming Government Through AI: Navigating Adoption, Barriers and Solutions
Aikins, S. K., & Dimitrijevska‐Markoski, T. (2026) · Springer eBooks · doi:10.1007/978-3-032-12344-2_2
Leveraging Phenomenon-Based Learning (PhBL) for Teaching AI Skills to Students
Uden, L., & Francis, J. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/407553
Kim, S. Y. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6906954
Schepers, T. G. J., Bex, F., Kempeneer, S., & Meijer, A. J. (2026) · Data & Policy · doi:10.1017/dap.2026.10095
- Authors
- Benedikt, L., Joshi, C., Nolan, L., Henstra-Hill, R., Shaw, L., & Hook, S.
- Affiliations
- Office for National Statistics (GB)
- Venue
- Proceedings of the 25th International Conference on Intelligent User Interfaces
- Publisher
- ACM
- Published
- 2020-03-17
- Pages
- 488-497
- OpenAlex
- W3009126165
- Cited by
- 24
- References
- 38
- Citations by year
- 2026: 6 · 2025: 6 · 2024: 5 · 2023: 3 · 2022: 3 · 2020: 1
- FWCI
- 8.26
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion)
- Topics
- Social Sciences › Political Science and International Relations › E-Government and Public Services
- Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- human-in-the-loop AI, government agencies, household budget survey, task automation, user experience design, human factors, stakeholder communication
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
