How Do Data Analysts Respond to AI Assistance? A Wizard-of-Oz Study
Open accessThe DOI matches its Crossref record; 100 of the 100 references it cites were found in a registry or an index.
Abstract
Data analysis is challenging as analysts must navigate nuanced decisions that may yield divergent conclusions. AI assistants have the potential to support analysts in planning their analyses, enabling more robust decision making. Though AI-based assistants that target code execution (e.g., Github Copilot) have received significant attention, limited research addresses assistance for both analysis execution and planning. In this work, we characterize helpful planning suggestions and their impacts on analysts’ workflows. We first review the analysis planning literature and crowd-sourced analysis studies to categorize suggestion content. We then conduct a Wizard-of-Oz study (n=13) to observe analysts’ preferences and reactions to planning assistance in a realistic scenario. Our findings highlight subtleties in contextual factors that impact suggestion helpfulness, emphasizing design implications for supporting different abstractions of assistance, forms of initiative, increased engagement, and alignment of goals between analysts and assistants.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The granularity of control might be high-level (strategic direction [61]) or fine-grained (parameter tuning [208]) depending on the phase.”
From Exploring Collaboration Patterns and Strategies in Human-AI Co-creation through the Lens of Agency: A Scoping Review of the Top-tier HCI Literature (2025) · cites this paper as [61]
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1×
“The improved capabilities of large language models (LLMs) for coding, language, and visualization generation [21, 22, 29, 31, 35] offer an opportunity to make such planning and execution data analysis assistants a reality.”
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · cited 2×
“Further, the unconsidered flexibility in decision-making (and the biases inherent therein) contributes to the scientific reproducibility crisis [1, 10, 33].”
BackgroundWohlin, C. (2014) · Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2601248.2601268 · Cited by 4,234 · cited 1×
“We searched Google Scholar with the keywords "sensemaking data analysis" from which we iteratively snowball sampled [117].”
Method1,500 scientists lift the lid on reproducibility
Baker, M. (2016) · Nature · doi:10.1038/533452a · Cited by 4,209 · cited 2×
“As data analysis often informs high-stakes real-world decisions [1, 9, 31], it is essential that analysts be able to understand and think critically about their analyses, regardless whether they received execution or planning assistance.”
BackgroundTraining language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · cited 2×
“More recently, general-purpose LLMs, such as InstructGPT [92] and GPT-4 GPT-4 [ However, ChatGPT requires users to actively prompt it for information (Fig.”
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · cited 2×
“LLMs trained on large-scale code corpora [29, 31, 43, 73, 89] have shown proficiency in learning programming concepts, solving programming challenges [29], and covering various domains like web development [130] and data science problems [27, 71].”
Guidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439 · cited 4×
“Finally, we simulated an assistant that could learn over time what suggestions the analyst found helpful–following common Human-AI guidelines [6].”
BackgroundMethodInfluentialBox, G. E. P. (1976) · Journal of the American Statistical Association · doi:10.1080/01621459.1976.10480949 · Cited by 2,114 · cited 2×
“The iterative nature of data analysis demands the nuanced interpretation of prior results, statistics, and domain knowledge to inform the subsequent decisions [17, 46, 57, 111], which is typically unaccounted for in execution-only assistants (Fig.”
BackgroundMethodProceedings of the 2020 CHI Conference on Human Factors in Computing Systems
(2020) · doi:10.1145/3313831 · Cited by 1,951
Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy
Shneiderman, B. (2020) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2020.1741118 · Cited by 1,743 · cited 1×
“In general, designing AI-based user-facing systems is challenging since designs must face issues that include explainability [65, 75], trust [69, 76, 110], user control [50, 105], and user expectations [68, 82].”
BackgroundMapping Species Distributions: Spatial Inference and Prediction
Franklin, J. (2010) · doi:10.1017/cbo9780511810602 · Cited by 1,534
Principles of mixed-initiative user interfaces
Horvitz, E. (1999) · Proceedings of the SIGCHI conference on Human factors in computing systems the CHI is the limit - CHI '99 · doi:10.1145/302979.303030 · Cited by 1,490 · cited 4×
“All analysts appreciated execution assistance, occasionally rating it negatively only when it came too late or differed from what they had intended.”
BackgroundMethodInfluentialIncreasing Transparency Through a Multiverse Analysis
Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016) · Perspectives on Psychological Science · doi:10.1177/1745691616658637 · Cited by 1,372 · cited 4×
“Inadequate analysis planning can complicate this flow and can causeanalyststoriskmakingarbitraryandill-foundeddecisions[108, 109]forahostofreasons.”
BackgroundInfluentialWhy Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts
Zamfirescu-Pereira, J. D., Wong, R. Y., Hartmann, B., & Yang, Q. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581388 · Cited by 1,274
Luger, E., & Sellen, A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858288 · Cited by 1,176
Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124
Variability in the analysis of a single neuroimaging dataset by many teams
(2020) · doi:10.1038/s41586-020-2314-9 · Cited by 1,097 · cited 3×
“Determining analysis correctness is inherently challenging [12, 15, 18, 35, 86, 103, 107], with there often being no single “correct” answer, but rather multiple defensible interpretations.”
BackgroundCompetition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043 · cited 1×
“LLMs trained on large-scale code corpora [29, 31, 43, 73, 89] have shown proficiency in learning programming concepts, solving programming challenges [29], and covering various domains like web development [130] and data science problems [27, 71].”
BackgroundCompetition-Level Code Generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Lago, A. D., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.07814 · Cited by 1,043 · cited 1×
“LLMs trained on large-scale code corpora [29, 31, 43, 73, 89] have shown proficiency in learning programming concepts, solving programming challenges [29], and covering various domains like web development [130] and data science problems [27, 71].”
BackgroundG-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · arXiv · doi:10.48550/arxiv.2303.16634 · Cited by 948
G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.153 · Cited by 948
Questioning the AI: Informing Design Practices for Explainable AI User Experiences
Liao, Q. V., Gruen, D., & Miller, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376590 · Cited by 925 · cited 1×
“In general, designing AI-based user-facing systems is challenging since designs must face issues that include explainability [65, 75], trust [69, 76, 112], user control [50, 107], and user expectations [68, 82].”
BackgroundMany Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results
Silberzahn, R., Uhlmann, E. L., Martin, D. P., Anselmi, P., Aust, F., Awtrey, E., Bahník, Š., Bai, F., Bannard, C., Bonnier, E., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245917747646 · Cited by 784 · cited 6×
“To characterize assistant suggestion content ( RQ 1 ), we conducted a literature review and critically assessed data from a crowd-sourced analysis study [107], wherein 29 analyst teams separately analyzed the same dataset to answer the same research question (Sec.”
BackgroundMethodInfluentialWizard of Oz Studies in HRI: A Systematic Review and New Reporting Guidelines
Riek, L. (2012) · Journal of Human-Robot Interaction · doi:10.5898/jhri.1.1.riek · Cited by 779 · cited 1×
“When a new suggestion occurs, the side panel shows a loading animation of variable duration to call attention to it; variability in duration aims to indicate that the system is working and simulate a real system computing and generating a suggestion [42, 99].”
BackgroundDoes the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445717 · Cited by 757 · cited 2×
“Nevertheless, determining the best practices for designing explanations remains an open field of research [11, 25, 38, 115].”
BackgroundDoes the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. S. (2020) · arXiv · doi:10.48550/arxiv.2006.14779 · Cited by 754 · cited 2×
“Nevertheless, determining the best practices for designing explanations remains an open field of research [11, 25, 38, 115].”
BackgroundWizard of Oz studies: why and how
Dahlbäck, N., Jönsson, A., & Ahrenberg, L. (1993) · Proceedings of the 1st international conference on Intelligent user interfaces - IUI '93 · doi:10.1145/169891.169968 · Cited by 742
Evaluating large language models trained on code
Chen, M. (2024) · TIB Data Manager · doi:10.57702/wbv3e61b · Cited by 737
Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
(2023) · doi:10.1145/3544548 · Cited by 735
The cost structure of sensemaking
Russell, D. M., Stefik, M. J., Pirolli, P., & Card, S. K. (1993) · Proceedings of the SIGCHI conference on Human factors in computing systems - CHI '93 · doi:10.1145/169059.169209 · Cited by 726 · cited 1×
“…are closely related to cognitive theories of sensemaking , whereby analysts seek and integrate new observations to build mental models [67, 94, 101] that inform their decisions for collecting and analyzing data to elicit findings; these findings, in turn, inform their mental models in an…”
BackgroundVaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
Understanding the Effect of Accuracy on Trust in Machine Learning Models
Yin, M., Wortman Vaughan, J., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300509 · Cited by 527 · cited 1×
“This example of overreliance on AI is well documented in other scenarios [25, 26, 115, 127].”
BackgroundWhat Makes a Good Conversation?: Challenges in Designing Truly Conversational Agents
Clark, L., Pantidi, N., Cooney, O., Doyle, P., Garaialde, D., Edwards, J., Spillane, B., Gilmartin, E., Murad, C., Munteanu, C., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300705 · Cited by 461
Fraser, N. M., & Gilbert, G. N. (1991) · Computer Speech & Language · doi:10.1016/0885-2308(91)90019-m · Cited by 442 · cited 1×
“When a new suggestion occurs, the side panel shows a loading animation of variable duration to call attention to it; variability in duration aims to indicate that the system is working and simulate a real system computing and generating a suggestion [42, 99].”
BackgroundThe Role of Explanations on Trust and Reliance in Clinical Decision Support Systems
Bussone, A., Stumpf, S., & O'Sullivan, D. (2015) · 2015 International Conference on Healthcare Informatics · doi:10.1109/ichi.2015.26 · Cited by 441 · cited 1×
“This example of overreliance on AI is well documented in other scenarios [25, 26, 115, 127].”
BackgroundEnterprise Data Analysis and Visualization: An Interview Study
Kandel, S., Paepcke, A., Hellerstein, J. M., & Heer, J. (2012) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2012.219 · Cited by 439 · cited 2×
“challenges [62, 78] to be able to fully benefit from analysis planning in its current form.”
BackgroundSelf-citationGrounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3586030 · Cited by 436 · cited 2×
“” This is akin to the two modes, acceleration and exploration, found in AI-assisted programming [12].”
BackgroundExplanations Can Reduce Overreliance on AI Systems During Decision-Making
Vasconcelos, H., Jörke, M., Grunde-McLaughlin, M., Gerstenberg, T., Bernstein, M. S., & Krishna, R. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579605 · Cited by 402 · cited 5×
“From a cost-benefit perspective [115], execution assistance is less intrusive, with its time-saving benefits more easily recognizable.”
BackgroundInfluentialSelf-citationLai, V., & Tan, C. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287590 · Cited by 377
Predicting human interruptibility with sensors
Fogarty, J., Hudson, S. E., Atkeson, C. G., Avrahami, D., Forlizzi, J., Kiesler, S., Lee, J. C., & Yang, J. (2005) · ACM Transactions on Computer-Human Interaction · doi:10.1145/1057237.1057243 · Cited by 344 · cited 2×
“For instance, prior works have used probalis-tic utility modeling [50] and sensors to model and predict human interruptibility [38].”
BackgroundExploration and Explanation in Computational Notebooks
Rule, A., Tabard, A., & Hollan, J. D. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173606 · Cited by 335 · cited 2×
“In this process, the wizard acts as the interface between the analyst and the LLM, deciding on the content and timing of the suggestion.”
BackgroundPrototyping an intelligent agent through Wizard of Oz
Maulsby, D., Greenberg, S., & Mander, R. (1993) · Proceedings of the SIGCHI conference on Human factors in computing systems - CHI '93 · doi:10.1145/169059.169215 · Cited by 307 · cited 1×
“We conducted an exploratory lab study using a Wizard of Oz protocol [33, 83] to understand the types of suggestions analysts prefer and how these suggestions might affect their workflow.”
MethodProxy tasks and subjective measures can be misleading in evaluating explainable AI systems
Buçinca, Z., Lin, P., Gajos, K. Z., & Glassman, E. L. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377498 · Cited by 307
The relationship between trust in AI and trustworthy machine learning technologies
Toreini, E., Aitken, M., Coopamootoo, K., Elliott, K., Zelaya, C. G., & van Moorsel, A. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372834 · Cited by 296 · cited 1×
“In general, designing AI-based user-facing systems is challenging since designs must face issues that include explainability [65, 75], trust [69, 76, 112], user control [50, 107], and user expectations [68, 82].”
Kocielnik, R., Amershi, S., & Bennett, P. N. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300641 · Cited by 277
Ross, S. I., Martinez, F., Houde, S., Muller, M., & Weisz, J. D. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584037 · Cited by 258 · cited 1×
“This is consistent with the preferences found in prior works that studied code execution assistants [74, 100, 116].”
ResultHelp Me Help the AI": Understanding How Explainability Can Support Human-AI Interaction
Kim, S. S. Y., Watkins, E. A., Russakovsky, O., Fong, R., & Monroy-Hernández, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581001 · Cited by 233 · cited 1×
“In general, designing AI-based user-facing systems is challenging since designs must face issues that include explainability [65, 75], trust [69, 76, 112], user control [50, 107], and user expectations [68, 82].”
BackgroundInteractive notebooks: Sharing the code
Shen, H. (2014) · Nature · doi:10.1038/515151a · Cited by 224 · cited 1×
“We focused on notebooks because they are an extremely common medium for data analysis [100, 104].”
Breznau, N., Rinke, E. M., Wuttke, A., Nguyen, H. H. V., Adem, M., Adriaans, J., Alvarez-Benjumea, A., Andersen, H. K., Auer, D., Azevedo, F., et al. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2203150119 · Cited by 220 · cited 3×
“Determining analysis correctness is inherently challenging [12, 15, 18, 35, 86, 103, 107], with there often being no single “correct” answer, but rather multiple defensible interpretations.”
BackgroundThe Story in the Notebook: Exploratory Data Science using a Literate Programming Tool
Kery, M. B., Radensky, M., Arya, M., John, B. E., & Myers, B. A. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173748 · Cited by 214
Productivity Assessment of Neural Code Completion
Ziegler, A., Kalliamvakou, E., Li, X. A., Rice, A., Rifkin, D., Simister, S., Sittampalam, G., & Aftandilian, E. (2022) · Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming · doi:10.1145/3520312.3534864 · Cited by 200 · cited 2×
“Effective planning suggestions can not be achieved by simply feeding the current analysis context into an LLM as it requires selective context, higher-level scaffolding, targeted prompts, and sensitive triggers.”
BackgroundProductivity Assessment of Neural Code Completion
Ziegler, A., Kalliamvakou, E., Simister, S., Sittampalam, G., Li, A., Rice, A., Rifkin, D., & Aftandilian, E. (2022) · arXiv · doi:10.48550/arxiv.2205.06537 · Cited by 200 · cited 2×
“Effective planning suggestions can not be achieved by simply feeding the current analysis context into an LLM as it requires selective context, higher-level scaffolding, targeted prompts, and sensitive triggers.”
BackgroundSupporting the analytical reasoning process in information visualization
Shrinivasan, Y. B., & van Wijk, J. J. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357247 · Cited by 191 · cited 1×
“Therefore, we did not focus on problems with specific systems [3, 108] or address common analysis challenges related to software and debugging [28, 48, 62], communication [117, 132], and data provenance [63, 85, 124].”
BackgroundInCoder: A Generative Model for Code Infilling and Synthesis
Fried, D., Aghajanyan, A., Lin, J., Wang, S., Wallace, E., Shi, F., Zhong, R., Yih, W.-T., Zettlemoyer, L., & Lewis, M. (2022) · arXiv · doi:10.48550/arxiv.2204.05999 · Cited by 187 · cited 1×
“LLMs trained on large-scale code corpora [29, 31, 43, 73, 89] have shown proficiency in learning programming concepts, solving programming challenges [29], and covering various domains like web development [130] and data science problems [27, 71].”
BackgroundConceptual Metaphors Impact Perceptions of Human-AI Collaboration
Khadpe, P., Krishna, R., Fei-Fei, L., Hancock, J. T., & Bernstein, M. S. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415234 · Cited by 183 · cited 1×
“We described what the system can do (e.g., make suggestions based on the notebook context), being careful not to anthro-pomorphize the assistant to facilitate accurate evaluations [64].”
BackgroundSchweinsberg, M., Feldman, M., Staub, N., van den Akker, O. R., van Aert, R. C. M., van Assen, M. A. L. M., Liu, Y., Althoff, T., Heer, J., Kale, A. M., et al. (2021) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2021.02.003 · Cited by 173 · cited 3×
“In crowd-sourced data analysis studies [13, 17, 20, 37, 86, 105, 109], numerous independent teams of analysts conduct analyses answering the same research question with the same dataset.”
Self-citationDutilh, G., Annis, J., Brown, S. D., Cassey, P., Evans, N. J., Grasman, R. P. P. P., Hawkins, G. E., Heathcote, A., Holmes, W. R., Krypotos, A.-M., et al. (2019) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1417-2 · Cited by 171 · cited 3×
“In crowd-sourced data analysis studies [13, 17, 20, 37, 86, 105, 109], numerous independent teams of analysts conduct analyses answering the same research question with the same dataset.”
BackgroundManaging Messes in Computational Notebooks
Head, A., Hohman, F., Barik, T., Drucker, S. M., & DeLine, R. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300500 · Cited by 165 · cited 1×
“Thus, the burden is on the analyst to decide what they want, where to find relevant context in a likely messy analysis [49, 63, 100], and craft a prompt or series of prompts.”
BackgroundWhat's Wrong with Computational Notebooks? Pain Points, Needs, and Design Opportunities
Chattopadhyay, S., Prasad, I., Henley, A. Z., Sarma, A., & Barik, T. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376729 · Cited by 159 · cited 1×
“Therefore, we did not focus on problems with specific systems [3, 108] or address common analysis challenges related to software and debugging [28, 48, 62], communication [117, 132], and data provenance [63, 85, 124].”
BackgroundBastiaansen, J. A., Kunkels, Y. K., Blaauw, F., Boker, S. M., Ceulemans, E., Chen, M., Chow, S.-M., de Jonge, P., Emerencia, A. C., Epskamp, S., et al. (2019) · doi:10.31234/osf.io/c8vp7 · Cited by 158 · cited 3×
“Determining analysis correctness is inherently challenging [12, 15, 18, 35, 86, 103, 107], with there often being no single “correct” answer, but rather multiple defensible interpretations.”
BackgroundPromptMaker: Prompt-based Prototyping with Large Language Models
Jiang, E., Olson, K., Toh, E., Molina, A., Donsbach, A., Terry, M., & Cai, C. J. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3503564 · Cited by 157 · cited 1×
“Prior work found that non-AI experts often struggle to write effective prompts [129] and experience significant cognitive load [54].”
BackgroundLet Me Explain: Impact of Personal and Impersonal Explanations on Trust in Recommender Systems
Kunkel, J., Donkers, T., Michael, L., Barbu, C.-M., & Ziegler, J. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300717 · Cited by 156
Threats of a replication crisis in empirical computer science
Cockburn, A., Dragicevic, P., Besançon, L., & Gutwin, C. (2020) · Communications of the ACM · doi:10.1145/3360311 · Cited by 155 · cited 2×
“As data analysis often informs high-stakes real-world decisions [1, 9, 31], it is essential that analysts be able to understand and think critically about their analyses, regardless whether they received execution or planning assistance.”
BackgroundThe Impact of Placebic Explanations on Trust in Intelligent Systems
Eiband, M., Buschek, D., Kremer, A., & Hussmann, H. (2019) · Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290607.3312787 · Cited by 152 · cited 1×
“Identifying explanation best practices remains an open area of research [10, 23, 36, 113].”
BackgroundSurfacing Visualization Mirages
McNutt, A. M., Kindlmann, G., & Correll, M. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376420 · Cited by 134 · cited 2×
“Adhering only to analysts’ execution preferences may perpetuate poor analysis decisions and practices, leading to misguided inferences or biased results [84].”
BackgroundSelf-citationFutzing and Moseying: Interviews with Professional Data Analysts on Exploration Practices
Alspaugh, S., Zokaei, N., Liu, A., Jin, C., & Hearst, M. A. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2865040 · Cited by 118 · cited 2×
“These included: domain expertise information [18, 57, 78, 119, 121], additional statistical knowledge [57, 78, 79], guidance for statistical modeling [16, 47, 51, 57, 119], guidance for analysts’ hypothesis derivation [2, 5, 14, 56], alternative decisions [61, 78, 79], and data analysis specific execution assistance […”
BackgroundAI-Assisted Code Authoring at Scale: Fine-Tuning, Deploying, and Mixed Methods Evaluation
Murali, V., Maddila, C., Ahmad, I., Bolin, M., Cheng, D., Ghorbani, N., Fernandez, R., Nagappan, N., & Rigby, P. C. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3643774 · Cited by 114
A Cognitive Interpretation of Data Analysis
Grolemund, G., & Wickham, H. (2014) · International Statistical Review · doi:10.1111/insr.12028 · Cited by 108 · cited 3×
“…included: domain expertise information [16, 57, 78, 116, 118], additional statistical knowledge [57, 78, 79], guidance for statistical modeling [14, 46, 51, 57, 116], guidance for analysts’ hypothesis derivation [2, 4, 13, 56], alternative decisions [61, 78, 79], and data analysis specific…”
BackgroundMethodInvestigating How Practitioners Use Human-AI Guidelines: A Case Study on the People + AI Guidebook
Yildirim, N., Pushkarna, M., Goyal, N., Wattenberg, M., & Viégas, F. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580900 · Cited by 104 · cited 1×
“Though these resources address common challenges in Human-AI interaction design, designing a system still requires support for domain-specific examples and proper problem formulation [126].”
BackgroundInteractive Model Cards: A Human-Centered Approach to Model Documentation
Crisan, A., Drouhard, M., Vig, J., & Rajani, N. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533108 · Cited by 101
Kang, Y., & Stasko, J. (2011) · IEEE Conference on Visual Analytics Science and Technology (VAST) · doi:10.1109/vast.2011.6102438 · Cited by 95 · cited 1×
“These included: domain expertise information [18, 57, 78, 119, 121], additional statistical knowledge [57, 78, 79], guidance for statistical modeling [16, 47, 51, 57, 119], guidance for analysts’ hypothesis derivation [2, 5, 14, 56], alternative decisions [61, 78, 79], and data analysis specific execution assistance […”
Analytic Provenance for Sensemaking: A Research Agenda
Xu, K., Attfield, S., Jankun-Kelly, T. J., Wheat, A., Nguyen, P. H., & Selvaraj, N. (2015) · IEEE Computer Graphics and Applications · doi:10.1109/mcg.2015.50 · Cited by 83 · cited 1×
“Therefore, we did not focus on problems with specific systems [3, 108] or address common analysis challenges related to software and debugging [28, 48, 62], communication [117, 132], and data provenance [63, 85, 124].”
BackgroundWang, A. Y., Wang, D., Drozdal, J., Muller, M., Park, S., Weisz, J. D., Liu, X., Wu, L., & Dugan, C. (2022) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3489465 · Cited by 81 · cited 1×
“Therefore, we did not focus on problems with specific systems [3, 106] or address common analysis challenges related to software and debugging [26, 47, 62], communication [115, 127], and data provenance [63, 85, 121].”
BackgroundOn the Design of AI-powered Code Assistants for Notebooks
Mcnutt, A. M., Wang, C., Deline, R. A., & Drucker, S. M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580940 · Cited by 77 · cited 5×
“Prior LLM-based assistants have explored practical approaches to execution assistance [12, 55, 85, 114, 133], however planning remains unexplored (Fig.”
BackgroundMethodInfluentialSelf-citationHow Data Workers Cope with Uncertainty: A Task Characterisation Study
Boukhelifa, N., Perrin, M.-E., Huron, S., & Eagan, J. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025738 · Cited by 76 · cited 1×
“These included: domain expertise information [16, 57, 78, 116, 118], additional statistical knowledge [57, 78, 79], guidance for statistical modeling [14, 46, 51, 57, 116], guidance for analysts’ hypothesis derivation [2, 4, 13, 56], alternative decisions [61, 78, 79], and data analysis specific…”
BackgroundDesign Exposition with Literate Visualization
Wood, J., Kachkaev, A., & Dykes, J. A. (2018) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2864836 · Cited by 76 · cited 1×
“” Similarly, A2 wanted suggestions that could provide a whole analysis plan: “Given the dependent variables and the data structures, what are some of the analyses I could run? If I need to run different analyses, how would I need to transform the data, with suggestions or sample code of how I can do that?” In these si…”
MethodData analysis and statistics: an expository overview
Tukey, J. W., & Wilk, M. B. (1966) · Proceedings of the November 7-10, 1966, fall joint computer conference on XX - AFIPS '66 (Fall) · doi:10.1145/1464291.1464366 · Cited by 74
Boba: Authoring and Visualizing Multiverse Analyses
Liu, Y., Kale, A., Althoff, T., & Heer, J. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2020.3028985 · Cited by 73 · cited 5×
“Furthermore, these assistants could be seamlessly integrated with existing multiverse analysis systems [48, 81, 104] to help analysts conduct multiverse analyses.”
InfluentialSelf-citationDecision-Making Under Uncertainty in Research Synthesis: Designing for the Garden of Forking Paths
Kale, A. M., Kay, M., & Hullman, J. R. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300432 · Cited by 61
Data Analysis: What Can Be Learned From the Past 50 Years
Huber, P. J. (2011) · Wiley series in probability and statistics · doi:10.1002/9781118018255 · Cited by 57
Understanding the Role of Alternatives in Data Analysis Practices
Liu, J., Boukhelifa, N., & Eagan, J. R. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2934593 · Cited by 56
Zheng, C., Wang, D., Wang, A. Y., & Ma, X. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517615 · Cited by 52
Kang, Y., & Stasko, J. (2012) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2012.224 · Cited by 49 · cited 1×
“Therefore, we did not focus on problems with specific systems [3, 108] or address common analysis challenges related to software and debugging [28, 48, 62], communication [117, 132], and data provenance [63, 85, 124].”
Tea: A High-level Language and Runtime System for Automating Statistical Analysis
Jun, E., Daum, M., Roesch, J., Chasins, S., Berger, E., Just, R., & Reinecke, K. (2019) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3332165.3347940 · Cited by 49
Datamations: Animated Explanations of Data Analysis Pipelines
Pu, X., Kross, S., Hofman, J. M., & Goldstein, D. G. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445063 · Cited by 43 · cited 1×
“For example, visualizations or animations of the data during and after data transformations could help analysts understand the assistant’s code actions [96, 123].”
BackgroundExploring the Learnability of Program Synthesizers by Novice Programmers
Jayagopal, D., Lubin, J., & Chasins, S. E. (2022) · Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3526113.3545659 · Cited by 40 · cited 1×
“As in other triggerless systems [53], participants received suggestions from the system without needing to take specific action.”
BackgroundLi, X., Zhang, Y., Leung, J., Sun, C., & Zhao, J. (2023) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3545995 · Cited by 36
Tisane: Authoring Statistical Models via Formal Reasoning from Conceptual and Data Relationships
Jun, E., Seo, A., Heer, J., & Just, R. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501888 · Cited by 35 · cited 1×
“Prior work developed systems to simplify the planning of statistical models and statistical tests [58, 59].”
MethodSelf-citationAI-assisted Code Authoring at Scale: Fine-tuning, deploying, and mixed methods evaluation
Murali, V., Maddila, C., Ahmad, I., Bolin, M., Cheng, D., Ghorbani, N., Fernandez, R., Nagappan, N., & Rigby, P. C. (2023) · arXiv · doi:10.48550/arxiv.2305.12050 · Cited by 33
Visualizing the Scripts of Data Wrangling With Somnus
Xiong, K., Fu, S., Ding, G., Luo, Z., Yu, R., Chen, W., Bao, H., & Wu, Y. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3144975 · Cited by 31
DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation
Lai, Y., Li, C., Wang, Y., Zhang, T., Zhong, R., Zettlemoyer, L., Yih, S., Fried, D., Wang, S., & Yu, C. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.11501 · Cited by 31 · cited 2×
“, Consider a (6,7,8) shape array, what is the index (x,y,z) of the 100th element? [71]), this work focuses on situations where the problem statement and methods to apply are less clear, unspecified by the user, or at a higher level of abstraction.”
Hypothesis Formalization: Empirical Findings, Software Limitations, and Design Implications
Jun, E., Birchfield, M., de Moura, N., Heer, J., & Just, R. (2022) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3476980 · Cited by 30
Self-citationGenLine and GenForm: Two Tools for Interacting with Generative Language Models in a Code Editor
Jiang, E., Toh, E., Molina, A., Donsbach, A., Cai, C. J., & Terry, M. (2021) · Adjunct Proceedings of the 34th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3474349.3480209 · Cited by 27 · cited 1×
“Prior LLM-based assistants have explored practical approaches to execution assistance [11, 48, 55, 85, 112, 128], however planning remains unexplored (Fig.”
BackgroundGrounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2022) · arXiv · doi:10.48550/arxiv.2206.15000 · Cited by 23 · cited 2×
“” This is akin to the two modes, acceleration and exploration, found in AI-assisted programming [12].”
BackgroundUnderstanding and Supporting Debugging Workflows in Multiverse Analysis
Gu, K., Jun, E., & Althoff, T. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581099 · Cited by 19 · cited 2×
“Furthermore, these assistants could be seamlessly integrated with existing multiverse analysis systems [48, 81, 104] to help analysts conduct multiverse analyses.”
BackgroundSelf-citationRaghunandan, D., Cui, Z., Krishnan, K., Tirfe, S., Shi, S., Shrestha, T. D., Battle, L., & Elmqvist, N. (2022) · arXiv.org · doi:10.48550/arxiv.2204.07876 · Cited by 12 · cited 4×
“” In these situations, analysts might benefit from workflow scaffolding recommendations similar to those in Lodestar [96] or litvis [119].”
InfluentialToward a Deeper Understanding of Data Analysis, Sensemaking, and Signature Discovery
Jolaoso, S., Burtner, R., & Endert, A. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-22668-2_36 · Cited by 8 · cited 1×
“These included: domain expertise information [18, 57, 78, 119, 121], additional statistical knowledge [57, 78, 79], guidance for statistical modeling [16, 47, 51, 57, 119], guidance for analysts’ hypothesis derivation [2, 5, 14, 56], alternative decisions [61, 78, 79], and data analysis specific execution assistance […”
BackgroundProblem Solving-A Statistician's Guide.
Bissell, D., & Chatfield, C. (1989) · Journal of the Royal Statistical Society Series D (The Statistician) · doi:10.2307/2348324 · Cited by 2
Research works printed but not linked: 25
- Jacob Andreas. 2022. Language Models as Agent Models. ArXiv abs/2212.01681 (2022). · arXiv
- Leilani Battle and Jeffrey Heer. 2019. Characterizing Exploratory Visual Analysis: A Literature Review and Evaluation of Analytic Provenance in Tableau. Computer Graphics Forum 38 (2019).
- Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, John A. Gehrke, Eric Horvitz, 2023. Sparks of Artificial General Intelligence: Early experiments with GPT-4. ArXiv abs/2303.12712 (2023). · arXiv
- Katharina Buchholz. 2023. Time to One Million Users. https://www.statista.com/chart/29174/time-to-one-million-users/. Accessed June 5, 2023.
- Shubham Chandel, Colin B. Clement, Guillermo Serrato, and Neel Sundaresan. 2022. Training and Evaluating a Jupyter Notebook Data Science Assistant. ArXiv abs/2201.12901 (2022). · arXiv
- Liying Cheng, Xingxuan Li, and Lidong Bing. 2023. Is GPT-4 a Good Data Analyst?ArXiv abs/2305.15038 (2023). · arXiv
- Victor C. Dibia. 2023. LIDA: A Tool for Automatic Generation of Grammar-Agnostic Visualizations and Infographics using Large Language Models. ArXiv abs/2303.02927 (2023). · arXiv
- Google. 2022. ML-Enhanced Code Completion Improves Developer Productivity. Google AI Blog. https://ai.googleblog.com/2022/07/ml-enhanced-code-completion-improves.html Accessed: June 3, 2023.
- Andrew D Gordon, Carina Negreanu, José Cambronero, Rasika Chakravarthy, Ian Drosos, 2023. Co-audit: tools to help humans double-check AI-generated content. arXiv preprint arXiv:2310.01297 (2023). · arXiv
- Ken Gu, Ruoxi Shang, Tim Althoff, Chenglong Wang, and Steven Mark Drucker. 2023. How Do Analysts Understand and Verify AI-Assisted Data Analyses?ArXiv abs/2309.10947 (2023). https://api.semanticscholar.org/CorpusID:262064320 · arXiv
- Hannah Rose Kirk, Bertie Vidgen, Paul Röttger, and Scott A. Hale. 2023. Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback. ArXiv abs/2303.05453 (2023). · arXiv
- Jenny Liang, Chenyang Yang, and Brad A. Myers. 2023. Understanding the Usability of AI Programming Assistants. ArXiv abs/2303.17125 (2023). · arXiv
- Qingzi Vera Liao and S. Shyam Sundar. 2022. Designing for Responsible Trust in AI Systems: A Communication Perspective. 2022 ACM Conference on Fairness, Accountability, and Transparency (2022).
- Stephanie C. Lin, Jacob Hilton, and Owain Evans. 2022. Teaching Models to Express Their Uncertainty in Words. Trans. Mach. Learn. Res. 2022 (2022). https://api.semanticscholar.org/CorpusID:249191391
- Albert J. Menkveld, Anna Dreber, Felix Holzmeister, Juergen Huber, Magnus Johanneson, 2021. Non-Standard Errors. SSRN Electronic Journal (2021).
- Swaroop Mishra and Elnaz Nouri. 2022. HELP ME THINK: A Simple Prompting Strategy for Non-experts to Create Customized Content with Models. ArXiv abs/2208.08232 (2022). · arXiv
- Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Haiquan Wang, 2022. A Conversational Paradigm for Program Synthesis. ArXiv abs/2203.13474 (2022). · arXiv
- Fernando Pérez and Brian E. Granger. 2007. IPython: A System for Interactive Scientific Computing. Computing in Science & Engineering 9 (2007).
- Deepthi Raghunandan, Aayushi Roy, Shenzhi Shi, Niklas Elmqvist, and Leilani Battle. 2022. Code Code Evolution: Understanding How People Change Data Science Notebooks Over Time. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (2022).
- Abhraneel Sarma, Alex Kale, Michael Moon, Nathan Taback, Fanny Chevalier, 2021. multiverse: Multiplexing Alternative Data Analyses in R Notebooks. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (2021). https://api.semanticscholar.org/CorpusID:238541538
- Uri Simonsohn, Joseph P. Simmons, and Leif D. Nelson. 2015. Specification Curve: Descriptive and Inferential Statistics on All Reasonable Specifications. MKTG: Methodological Issues in Consumer Research (Topic) (2015).
- Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones 2017. Attention is All you Need. In NIPS.
- Chris J. Wild and Maxine Pfannkuch. 1999. Statistical Thinking in Empirical Enquiry. International Statistical Review 67 (1999).
- Kanit Wongsuphasawat, Yang Liu, and Jeffrey Heer. 2019. Goals, Process, and Challenges of Exploratory Data Analysis: An Interview Study. ArXiv abs/1911.00568 (2019). · arXiv
- Beiqi Zhang, Peng Liang, Xiyu (Thomas) Zhou, Aakash Ahmad, and Muhammad Waseem. 2023. Practices and Challenges of Using GitHub Copilot: An Empirical Study. ArXiv abs/2303.08733 (2023). · arXiv
Cited by
The Metacognitive Demands and Opportunities of Generative AI
Tankelevitch, L., Kewenig, V., Simkute, A., Scott, A. E., Sarkar, A., Sellen, A., & Rintel, S. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642902 · Cited by 493
“Correlation between confidence and objective accuracy can estimate confidence calibration [127, 202]; Meta-d’ is a derived metric capturing users’ prospective confidence resolution independent of their calibration [66, 115] (see also [65, 145]).”
BackgroundMethodData-Copilot: Bridging Billions of Data and Humans with Autonomous WorkflowPreprint
Zhang, W., Shen, Y., Tan, Z., Hou, G., Lu, W., & Zhuang, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.07209 · Cited by 126
“We quantitatively assess Data-Copilot in processing real-time user requests, where it could invoke the interfaces designed in the first phase.”
Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decomposition
Kazemitabaar, M., Williams, J., Drosos, I., Grossman, T., Henley, A. Z., Negreanu, C., & Sarkar, A. (2024) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3654777.3676345 · Cited by 112
“Consequently, research has investigated in detail the chatbot paradigm for AI assistance in data analysis and visualization [17, 27, 34, 41, 80, 101].”
Kazemitabaar, M., Huang, O., Suh, S., Henley, A. Z., & Grossman, T. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712104 · Cited by 92
“Ultimately, designing AI tools that support deeper cognitive reflection remains crucial to maintaining both short-term productivity and long-term skill development in programming.”
Ironies of Generative AI: Understanding and Mitigating Productivity Loss in Human-AI Interaction
Simkute, A., Tankelevitch, L., Kewenig, V., Scott, A. E., Sellen, A., & Rintel, S. (2024) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2024.2405782 · Cited by 89
“Otherwise, during this stage, suggestions should carefully align with users’ flow (Gu et al., 2023), in line with ecological interface design.”
Influential
Metadata
- Authors
- Gu, K., Grunde-McLaughlin, M., McNutt, A., Heer, J., & Althoff, T.
- Affiliations
- University of Washington (US)
- Venue
- Proceedings of the CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2024-05-11
- Pages
- 1-22
- OpenAlex
- W4396832182
- Semantic Scholar
- 2fe74a07051649cb5dc41da319fef547647e9171
- Cited by
- 40
- References
- 94
- Citations by year
- 2026: 17 · 2025: 19 · 2024: 4
- FWCI
- 8.26
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- OSF Preprints (OSF Preprints) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Bill & Melinda Gates Foundation (INV-004841)
- NSF (IIS-1901386)
- NSF CAREER (IIS-2142794)
- Topics
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Keywords
- AI assistants, data analysis, AI-assisted decision-making
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.