Human-Centered Evaluation and Auditing of Language Models
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
HCI scholarship has a distinctive role in translating abstract principles into concrete, citizen-centred practices by offering insights into participatory design methods for AI systems and policy, tools to facilitate stakeholder engagement, and socio-technical evaluation.
“HCI scholarship has a distinctive role here: by offering established insights into participatory design methods for AI systems and policy [87, 103, 183], tools to facilitate stakeholder engagement [97], and socio-technical evaluation [12, 110, 131], it helps translate abstract principles into concrete, citizen-centred practices.”
From “It’s Just a Wild, Wild West”: Harnessing Public Procurement as an AI Governance Mechanism (2026) · cites this paper as [110]
Cites
Fairness and Abstraction in Sociotechnical Systems
Selbst, A. D., Boyd, D., Friedler, S. A., Venkatasubramanian, S., & Vertesi, J. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287598 · Cited by 1,631 · see citing works
Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372873 · Cited by 1,372 · cited 1× · see citing works
“It is also known that practitioners engage in extensive internal auditing (e.g., red teaming [16]) to mitigate the safety concerns and other risks of LLMs, although there is a lack of trans-parency and shared practices in how such audits are performed [39, 43, 44, 46].”
Questioning the AI: Informing Design Practices for Explainable AI User Experiences
Liao, Q. V., Gruen, D., & Miller, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376590 · Cited by 925 · see citing works
Self-citationZhang, Y., Liao, Q. V., & Bellamy, R. K. E. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372852 · Cited by 895 · see citing works
Self-citationImproving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?
Holstein, K., Wortman Vaughan, J., Daumé, H., Dudik, M., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300830 · Cited by 777 · see citing works
Taxonomy of Risks posed by Language Models
Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P.-S., Mellor, J., Glaese, A., Cheng, M., Balle, B., Kasirzadeh, A., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533088 · Cited by 764 · cited 1× · see citing works
“Meanwhile, these models also raise significant risks for individuals and society [51].”
"I always assumed that I wasn't really that close to [her]"
Eslami, M., Rickman, A., Vaccaro, K., Aleyasen, A., Vuong, A., Karahalios, K. G., Hamilton, K., & Sandvig, C. (2015) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702556 · Cited by 665 · see citing works
Self-citationGhazal, A., Rabl, T., Hu, M., Raab, F., Poess, M., Crolotte, A., & Jacobsen, H. (2013) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/2463676.2463712 · Cited by 343 · see citing works
Holistic Evaluation of Language Models
Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.09110 · Cited by 255 · see citing works
Dynabench: Rethinking Benchmarking in NLP
Kiela, D., Bartolo, M., Nie, Y., Kaushik, D., Geiger, A., Wu, Z., Vidgen, B., Prasad, G., Singh, A., Ringshia, P., et al. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.324 · Cited by 254 · see citing works
Auditing Algorithms: Understanding Algorithmic Systems from the Outside In
Metaxa, D., Park, J., Robertson, R. E., Karahalios, K., Wilson, C., Hancock, J., & Sandvig, C. (2021) · Found. Trends Hum. Comput. Interact. · doi:10.1561/1100000083 · Cited by 218 · see citing works
AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap
Liao, Q. V., & Wortman Vaughan, J. (2024) · Harvard Data Science Review · doi:10.1162/99608f92.8036d03b · Cited by 210 · see citing works
Self-citationAI Transparency in the Age of LLMs: A Human-Centered Research Roadmap
Liao, Q. V., & Vaughan, J. W. (2023) · arXiv · doi:10.48550/arxiv.2306.01941 · Cited by 209 · cited 2× · see citing works
“Collectively, members of the organizing committee contributed to HCI and AI on various topics, including Explainable AI [29, 31, 58 Wei Xu is an Associate Professor in the College of Computing at Georgia Tech.”
Self-citationXiao, Z., Zhou, M. X., Liao, Q. V., Mark, G. J., Chi, C., Chen, W., & Yang, H. (2020) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3381804 · Cited by 206 · see citing works
Self-citationRaji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · arXiv · doi:10.48550/arxiv.2001.00973 · Cited by 179 · see citing works
Designing for Responsible Trust in AI Systems: A Communication Perspective
Liao, Q. V., & Sundar, S. S. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533182 · Cited by 165 · see citing works
Self-citationTowards Accountable AI: Hybrid Human-Machine Analyses for Characterizing System Failure
Nushi, B., Kamar, E., & Horvitz, E. (2018) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v6i1.13337 · Cited by 126 · see citing works
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text
Gehrmann, S., Clark, E., & Sellam, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13715 · Cited by 123 · see citing works
Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance
Raji, I. D., Xu, P., Honigsberg, C., & Ho, D. (2022) · Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3514094.3534181 · Cited by 122 · see citing works
Exploring How Machine Learning Practitioners (Try To) Use Fairness Toolkits
Deng, W. H., Nagireddy, M., Lee, M. S. A., Singh, J., Wu, Z. S., Holstein, K., & Zhu, H. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533113 · Cited by 99 · see citing works
Self-citationDeVos, A., Dhabalia, A., Shen, H., Holstein, K., & Eslami, M. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517441 · Cited by 98 · see citing works
Self-citationHaving Beer after Prayer? Measuring Cultural Bias in Large Language Models
Naous, T., Ryan, M. J., Ritter, A., & Xu, W. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.862 · Cited by 96 · see citing works
Self-citationSupporting Human-AI Collaboration in Auditing LLMs with LLMs
Rastogi, C., Ribeiro, M. T., King, N., Nori, H., & Amershi, S. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604712 · Cited by 87 · see citing works
Deng, W. H., Guo, B., Devrio, A., Shen, H., Eslami, M., & Holstein, K. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581026 · Cited by 78 · see citing works
Self-citationLam, M. S., Gordon, M., Metaxa, D., Hancock, J. T., Landay, J. A., & Bernstein, M. S. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555625 · Cited by 78 · see citing works
Self-citationYurrita, M., Draws, T., Balayn, A., Murray-Rust, D., Tintarev, N., & Bozzon, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581161 · Cited by 73 · see citing works
Young, M., Magassa, L., & Friedman, B. (2019) · Ethics and Information Technology · doi:10.1007/s10676-019-09497-z · Cited by 71 · see citing works
Shen, H., Deng, W. H., Chattopadhyay, A., Wu, Z. S., Wang, X., & Zhu, H. (2021) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3442188.3445971 · Cited by 71 · see citing works
Self-citationDiscovering and Validating AI Errors With Crowdsourced Failure Reports
Cabrera, Á. A., Druck, A. J., Hong, J., & Perer, A. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479569 · Cited by 61 · see citing works
Deng, W. H., Yildirim, N., Chang, M., Eslami, M., Holstein, K., & Madaio, M. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3594037 · Cited by 57 · see citing works
Self-citationKim, T. S., Lee, Y., Chang, M., & Kim, J. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606833 · Cited by 56 · see citing works
Self-citationHuman-Centered Responsible Artificial Intelligence: Current & Future Trends
Tahaei, M., Constantinides, M., Quercia, D., Kennedy, S., Muller, M. J., Stumpf, S., Liao, Q. V., Baeza-Yates, R. A., Aroyo, L. M., Holbrook, J., et al. (2023) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3544549.3583178 · Cited by 52 · see citing works
Self-citationZeno: An Interactive Framework for Behavioral Evaluation of Machine Learning
Cabrera, Á. A., Fu, E., Bertucci, D., Holstein, K., Talwalkar, A., Hong, J. I., & Perer, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581268 · Cited by 51 · see citing works
The Model Card Authoring Toolkit: Toward Community-centered, Deliberation-driven AI Design
Shen, H., Wang, L., Deng, W. H., Brusse, C., Velgersdijk, R., & Zhu, H. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533110 · Cited by 45 · see citing works
Self-citationSociotechnical Audits: Broadening the Algorithm Auditing Lens to Investigate Targeted Advertising
Lam, M. S., Pandit, A., Kalicki, C. H., Gupta, R., Sahoo, P., & Metaxa, D. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610209 · Cited by 44 · see citing works
Self-citationLENS: A Learnable Evaluation Metric for Text Simplification
Maddela, M., Dou, Y., Heineman, D., & Xu, W. H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.905 · Cited by 38 · see citing works
Self-citationReducing Privacy Risks in Online Self-Disclosures with Language Models
Dou, Y., Krsek, I., Naous, T., Kabra, A., Das, S., Ritter, A., & Xu, W. H. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.741 · Cited by 29 · see citing works
Self-citationSearch atlas: Visualizing divergent search results across geopolitical borders
Ochigame, R., & Ye, K. (2021) · Designing Interactive Systems Conference 2021 · doi:10.1145/3461778.3462032 · Cited by 23 · see citing works
Human-Centered Evaluation and Auditing of Language Models
Xiao, Z., Deng, W. H., Lam, M. S., Eslami, M., Kim, J., Lee, M., & Liao, Q. V. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3636302 · Cited by 20 · see citing works
Self-citationHuman-in-the-loop Evaluation for Early Misinformation Detection: A Case Study of COVID-19 Treatments
Mendes, E., Chen, Y., Xu, W. H., & Ritter, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.881 · Cited by 10 · see citing works
Self-citationDancing Between Success and Failure: Edit-level Simplification Evaluation using SALSA
Heineman, D., Dou, Y., Maddela, M., & Xu, W. H. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.211 · Cited by 10 · see citing works
Self-citationECBD: Evidence-Centered Benchmark Design for NLP
Liu, Y. L., Blodgett, S. L., Cheung, J. C. K., Liao, Q. V., Olteanu, A., & Xiao, Z. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.861 · Cited by 8 · see citing works
Self-citationOpportunities for Human-centered Evaluation of Machine Translation Systems
Liebling, D. J., Heller, K., Robertson, S., & Deng, W. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.17 · Cited by 7 · see citing works
Self-citationResponsible AI Considerations in Text Summarization Research: A Review of Current Practices
Liu, Y. H., Cao, M., Blodgett, S. L., Cheung, J. C. K., Olteanu, A., & Trischler, A. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.413 · Cited by 6 · see citing works
Self-citationThresh: A Unified, Customizable and Deployable Platform for Fine-Grained Text Evaluation
Heineman, D., Dou, Y., & Xu, W. H. (2023) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2023.emnlp-demo.30 · Cited by 3 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Developing a Framework for Auditing Large Language Models using Human-in-the-loop
Amirizaniani, M., Lavergne, A., Snell Okada, E., Chadha, A., Roosta, T., & Shah, C. (2025) · Proceedings of the 2025 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region · doi:10.1145/3767695.3769514 · Cited by 16 · see citing works
“It’s Just a Wild, Wild West”: Harnessing Public Procurement as an AI Governance Mechanism
Hudig, A. I., Kallina, E., & Singh, J. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791968 · Cited by 3 · see citing works
T2IBias: Uncovering Societal Bias Encoded in the Latent Space of Text-to-Image Generative Models
Sufian, A., Distante, C., Leo, M., & Salam, H. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-16886-3_4 · Cited by 3 · see citing works
OntoScope: Using a Divergent-Convergent Interaction Framework to Support LLM-based Ontology Scoping
Zhao, Y., Meroño-Peñuela, A., & Simperl, E. P. B. (2026) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3742413.3789225 · Cited by 2 · see citing works
Muteract: Interactive and Iterative Prompt Mutation Interface for LLM Developers and Evaluators
Meena, Y. K., Mondal, S., & Potta, M. P. (2025) · International Conference of Human-Computer Interaction Design & Research · doi:10.1145/3768633.3770129
Third Workshop on Human-Centered Evaluation and Auditing of Language Models: AI Agents-in-the-Loop
van der Maden, W., Deng, W. H., Liu, Y. L., Jiang, H., Chen, V., Li, H., Kim, J., Liao, Q. V., Xu, W., Eslami, M., et al. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3778709
Self-citationDeletion Isn't Enough: Auditing RAG for Selective Forgetting
Tavakoli, L., & Sanderson, M. (2026) · Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3805712.3808545
- Authors
- Liu, Y. L., Deng, W. H., Lam, M. S., Eslami, M., Kim, J., Liao, Q. V., Xu, W., Novikova, J., & Xiao, Z.
- Affiliations
- Johns Hopkins University (US)
- Carnegie Mellon University (US)
- Stanford University (US)
- Korea Advanced Institute of Science and Technology (KR)
- Microsoft Research Montréal (Canada) (CA)
- Georgia Institute of Technology (US)
- AI Risk and Vulnerability Alliance, Seattle, Washington, USA
- ORCID
- Venue
- Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2025-04-25
- Pages
- 1-7
- OpenAlex
- W4409720313
- Cited by
- 7
- References
- 44
- Citations by year
- 2026: 5 · 2025: 2
- FWCI
- 7.23
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) (publishedVersion)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- language model evaluation, human-centered evaluation, auditing, large language models, human-in-the-loop
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
