Trust in AutoML: exploring information needs for establishing trust in automated machine learning systems
Abstract
We explore trust in a relatively new area of data science: Automated Machine Learning (AutoML). In AutoML, AI methods are used to generate and optimize machine learning models by automatically engineering features, selecting models, and optimizing hyperparameters. In this paper, we seek to understand what kinds of information influence data scientists' trust in the models produced by AutoML? We operationalize trust as a willingness to deploy a model produced using automated methods. We report results from three studies - qualitative interviews, a controlled experiment, and a card-sorting task - to understand the information needs of data scientists for establishing trust in AutoML systems. We find that including transparency features in an AutoML tool increased user trust and understandability in the tool; and out of all proposed features, model performance metrics and visualizations are the most important information to data scientists when establishing their trust with an AutoML tool.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Tasks that involve inherent uncertainty are often those where humans tend to rely on AI systems for advice, such as loan approval, recidivism prediction, house price estimation, and student admission.
“Tasks that involve inherent uncertainty are often those where humans tend to rely on AI systems for advice, such as loan approval [27, 34, 124], recidivism prediction [31, 47, 86], house price estimation [2, 13, 25], and student admission [13, 24].”
From Dealing with Uncertainty: Understanding the Impact of Prognostic Versus Diagnostic Tasks on Trust and Reliance in Human-AI Decision Making (2024) · cites this paper as [34]
Cites
Making sense of cronbach’s alpha.
Tavakol, M., & Dennick, R. (2011) · International Journal of Medical Education · doi:10.5116/ijme.4dfb.8dfd · Cited by 15,126 · see citing works
Exploratory and confirmatory factor analysis: Understanding concepts and applications.
Thompson, B. (2004) · doi:10.1037/10694-000 · Cited by 3,469 · see citing works
Guidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439 · see citing works
Questioning the AI: Informing Design Practices for Explainable AI User Experiences
Liao, Q. V., Gruen, D., & Miller, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376590 · Cited by 925 · see citing works
Zhang, Y., Liao, Q. V., & Bellamy, R. K. E. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372852 · Cited by 895 · see citing works
Golovin, D., Solnik, B., Moitra, S., Kochanski, G., Karro, J. E., & Sculley, D. (2017) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3097983.3098043 · Cited by 585 · see citing works
Understanding the Effect of Accuracy on Trust in Machine Learning Models
Yin, M., Wortman Vaughan, J., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300509 · Cited by 527 · cited 1× · see citing works
“However, much of the literature states a lack of transparency in AI [5, 64, 65].”
BackgroundDeep feature synthesis: Towards automating data science endeavors
Kanter, J. M., & Veeramachaneni, K. K. (2015) · IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2015.7344858 · Cited by 441 · see citing works
(2018) · Singapore Management University Institutional Knowledge (InK) (Singapore Management University) · Cited by 390 · see citing works
Exploration and Explanation in Computational Notebooks
Rule, A., Tabard, A., & Hollan, J. D. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173606 · Cited by 347 · see citing works
Human-AI Collaboration in Data Science: Exploring Data Scientists' Perceptions of Automated AI
Wang, D., Weisz, J. D., Muller, M., Ram, P., Geyer, W., Dugan, C., Tausczik, Y., Samulowitz, H., & Gray, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359313 · Cited by 342 · see citing works
Self-citationAlgorithmic Bias: From Discrimination Discovery to Fairness-aware Data Mining
Hajian, S., Bonchi, F., & Castillo, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2945386 · Cited by 334 · see citing works
How Data Science Workers Work with Data
Muller, M. J., Lange, I., Wang, D., Piorkowski, D., Tsay, J., Liao, Q. V., Dugan, C., & Erickson, T. D. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300356 · Cited by 262 · see citing works
Self-citationAffective Processes in Human–Automation Interactions
Merritt, S. M. (2011) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720811411912 · Cited by 260 · see citing works
Hoffman, R. R., Johnson, M., Bradshaw, J. M., & Underbrink, A. J. (2013) · IEEE Intelligent Systems · doi:10.1109/mis.2013.24 · Cited by 238 · see citing works
Passi, S., & Jackson, S. J. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274405 · Cited by 225 · see citing works
Kery, M. B., Radensky, M., Arya, M., John, B. E., & Myers, B. A. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173748 · Cited by 216 · see citing works
Feature Engineering for Predictive Modeling Using Reinforcement Learning
Khurana, U., Samulowitz, H., & Turaga, D. S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11678 · Cited by 205 · see citing works
Cognito: Automated Feature Engineering for Supervised Learning
Khurana, U., Turaga, D. S., Samulowitz, H., & Srinivasan, P. (2016) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw.2016.0190 · Cited by 159 · see citing works
Weitz, K., Schiller, D., Schlagowski, R., Huber, T., & André, E. (2019) · Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents · doi:10.1145/3308532.3329441 · Cited by 136 · see citing works
van der Aalst, W. M. P., Bichler, M., & Heinzl, A. (2017) · Business & Information Systems Engineering · doi:10.1007/s12599-017-0487-z · Cited by 132 · see citing works
How Data Scientists Use Computational Notebooks for Real-Time Collaboration
Wang, A. Y., Mittal, A., Brooks, C. A., & Oney, S. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359141 · Cited by 131 · see citing works
Feinberg, M. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025837 · Cited by 115 · see citing works
Wang, Q., Ming, Y., Jin, Z., Shen, Q., Liu, D., Smith, M. J., Veeramachaneni, K. K., & Qu, H. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300911 · Cited by 113 · see citing works
Guo, P. J., Kandel, S., Hellerstein, J. M., et al. (2011) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/2047196.2047205 · Cited by 110 · see citing works
Towards Effective Foraging by Data Scientists to Find Past Analysis Choices
Kery, M. B., John, B. E., O'Flaherty, P., Horvath, A., & Myers, B. A. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300322 · Cited by 96 · see citing works
Olson, J. S., Wang, D., Olson, G. M., & Zhang, J. (2017) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3038919 · Cited by 89 · see citing works
Self-citationPractitioners Teaching Data Science in Industry and Academia
Kross, S., & Guo, P. J. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300493 · Cited by 86 · see citing works
Towards human-guided machine learning
Gil, Y., Honaker, J., Gupta, S., Ma, Y., D'Orazio, V. J., Garijo, D., Gadewar, S. P., Yang, Q., & Jahanshad, N. (2019) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3301275.3302324 · Cited by 80 · see citing works
Hou, Y., & Wang, D. (2017) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3134688 · Cited by 67 · see citing works
Self-citationThe many faces of data-centric workflow optimization: a survey
Kougka, G., Gounaris, A., & Simitsis, A. (2018) · International Journal of Data Science and Analytics · doi:10.1007/s41060-018-0107-0 · Cited by 45 · see citing works
Conditional Parallel Coordinates
Weidele, D. K. I. (2019) · Visualization · doi:10.1109/visual.2019.8933632 · Cited by 19 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
What is Human-Centered about Human-Centered AI? A Map of the Research Landscape
Capel, T., & Brereton, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580959 · Cited by 228 · see citing works
How to Evaluate Trust in AI-Assisted Decision Making? A Survey of Empirical Methodologies
Vereschak, O., Bailly, G., & Caramiaux, B. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3476068 · Cited by 229 · see citing works
Wang, D., Wang, L., Zhang, Z., Wang, D., Zhu, H., Gao, Y., Fan, X., & Tian, F. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445432 · Cited by 209 · see citing works
Self-citationWhither AutoML? Understanding the Role of Automation in Machine Learning Workflows
Xin, D., Wu, E. Y., Lee, D. J.-L., Salehi, N., & Parameswaran, A. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445306 · Cited by 110 · see citing works
Investigating and Designing for Trust in AI-powered Code Generation Tools
Wang, R., Cheng, R., Ford, D., & Zimmermann, T. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658984 · Cited by 87 · see citing works
“Specifically, some emphasize the value of transparency [22] and predictability [19], which helps a user build proper expectations when interacting with AI [35].”
BackgroundBetter Together? An Evaluation of AI-Supported Code Translation
Weisz, J. D., Muller, M., Ross, S. I., Martinez, F., Houde, S., Agarwal, M., Talamadupula, K., & Richards, J. T. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511157 · Cited by 69 · see citing works
Self-citationExploring the Use of Personalized AI for Identifying Misinformation on Social Media
Jahanbakhsh, F., Katsis, Y., Wang, D., Popa, L., & Muller, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581219 · Cited by 42 · see citing works
Self-citationCheng, R., Wang, R., Zimmermann, T., & Ford, D. (2024) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3651990 · Cited by 47 · see citing works
“[17] found that transparency features in an AutoML tool can help users understand the system and build trust with the system.”
BackgroundSalimzadeh, S., He, G., & Gadiraju, U. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641905 · Cited by 41 · see citing works
Humans, AI, and Context: Understanding End-Users’ Trust in a Real-World Computer Vision Application
Kim, S. S. Y., Watkins, E. A., Russakovsky, O., Fong, R., & Monroy-Hernández, A. (2023) · 2023 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3593013.3593978 · Cited by 37 · see citing works
MetaWriter: Exploring the Potential and Perils of AI Writing Support in Scientific Peer Review
Sun, L., Tao, S., Hu, J., & Dow, S. P. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637371 · Cited by 37 · see citing works
- Authors
- Drozdal, J., Weisz, J., Wang, D., Dass, G., Yao, B., Zhao, C., Muller, M., Ju, L., & Su, H.
- Affiliations
- Rensselaer Polytechnic Institute (US)
- IBM (United States) (US)
- IBM
- Venue
- Proceedings of the 25th International Conference on Intelligent User Interfaces
- Publisher
- ACM
- Published
- 2020-03-17
- Pages
- 297-307
- OpenAlex
- W4288083546
- Cited by
- 119
- References
- 32
- Citations by year
- 2026: 10 · 2025: 16 · 2024: 27 · 2023: 17 · 2022: 21 · 2021: 23 · 2020: 5
- FWCI
- 6.21
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Keywords
- automated machine learning, trust in AI, data scientists, transparency, explainability, model evaluation metrics, model visualization, model deployment, information needs
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
