What Makes a Well-Documented Notebook? A Case Study of Data Scientists’ Documentation Practices in Kaggle
The DOI matches its Crossref record; 24 of the 25 references it cites were found in a registry or an index.
Abstract
Many data scientists use computational notebooks to test and present their work, as a notebook can weave code and documentation together (computational narrative), and support rapid iteration on code experiments. However, it is not easy to write good documentation in a data science notebook, partially because there is a lack of a corpus of well-documented notebooks as exemplars for data scientists to follow. To cope with this challenge, this work looks at Kaggle — a large online community for data scientists to host and participate in machine learning competitions — and considers highly-voted Kaggle notebooks as a proxy for well-documented notebooks. Through a qualitative analysis at both the notebook level and the markdown-cell level, we find these notebooks are indeed well documented in reference to previous literature. Our analysis also reveals nine categories of content that data scientists write in their documentation cells, and these documentation cells often interplay with different stages of the data science lifecycle. We conclude the paper with design implications and future research directions.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Longstanding definitions of data quality have viewed good quality data as”
From A Survey of Data Quality Requirements That Matter in ML Development Pipelines (2023) · cites this paper as [74]
In the citing paper
“These dimensions are subsumed by higher-order characterisations that capture the intrinsic, contextual, accessibility, and representational aspects of datasets [74].”
From A Survey of Data Quality Requirements That Matter in ML Development Pipelines (2023) · cites this paper as [74]
In the citing paper
“Below we will draw on Wang & Strong's [74] categorisation of the intrinsic, contextual, accessibility, and representational aspects of datasets to illustrate some of the ways in which previously established data quality categories already apply to ML problems.”
From A Survey of Data Quality Requirements That Matter in ML Development Pipelines (2023) · cites this paper as [74]
In the citing paper
“Merino et al. [47] draw on Wang and Strong [74]'s canonical distinction between the intrinsic, contextual, accessibility, and representational aspects of datasets when using the above mentioned industry standards ISO/IEC 25012 and ISO/IEC 25024.”
From A Survey of Data Quality Requirements That Matter in ML Development Pipelines (2023) · cites this paper as [74]
Citations
Cites
Why Jupyter is data scientists’ computational notebook of choice
Perkel, J. M. (2018) · Nature · doi:10.1038/d41586-018-07196-1 · Cited by 352 · cited 1×
“The data science community has widely adopted computational notebooks under the premise that it can help data scientists to efectively create, communicate, and collaborate on their analysis work [22].”
BackgroundHuman-AI Collaboration in Data Science: Exploring Data Scientists' Perceptions of Automated AI
Wang, D., Weisz, J. D., Muller, M., Ram, P., Geyer, W., Dugan, C., Tausczik, Y., Samulowitz, H., & Gray, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359313 · Cited by 342 · cited 1×
“For example, Wang et al. [31] summarized a typical data science workfow as three high-level phases: data preparation, model building, and model deployment, where each phase contains many sub-goals and activities; Similarly, [30] proposes a taxonomy of ten stages and six user personas to describe…”
BackgroundSelf-citationExploration and Explanation in Computational Notebooks
Rule, A., Tabard, A., & Hollan, J. D. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173606 · Cited by 335 · cited 16×
“Previous studies [13, 25] suggest that data scientists are hesitant to follow the practice of literate programming to create documentation during the development of code, despite they acknowledge the benefts that these informal notes have for recalling the analysis path.”
BackgroundMethodInfluentialA study of the documentation essential to software maintenance
de Souza, S. C. B., Anquetil, N., & de Oliveira, K. M. (2005) · Proceedings of the 23rd annual international conference on Design of communication: documenting & designing for pervasive information · doi:10.1145/1085313.1085331 · Cited by 305 · cited 1×
“Writing clear and comprehensive documentation is critical to software projects’ development and maintenance [4, 11, 18, 24, 26].”
BackgroundA Neural Model for Generating Natural Language Summaries of Program Subroutines
LeClair, A., Jiang, S., & McMillan, C. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00087 · Cited by 294 · cited 2×
“This is promising as there are many breakthroughs in NLP and ML for automatically generating natural language descriptions for code snippets [16].”
BackgroundMethodHow do Data Science Workers Collaborate? Roles, Workflows, and Tools
Zhang, A. X., Muller, M., & Wang, D. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3392826 · Cited by 268 · cited 2×
“Our Labs) have become the predominant coding environment for data analysis also reveals nine categories of content that data scientists scientists and machine learning engineers [13, 32].”
BackgroundSelf-citationHow Data Science Workers Work with Data: Discovery, Capture, Curation, Design, Creation
Muller, M., Lange, I., Wang, D., Piorkowski, D., Tsay, J., Liao, Q. V., Dugan, C., & Erickson, T. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300356 · Cited by 261 · cited 1×
“3451617 cle [30], and has a wide spectrum of usage scenarios [20].”
BackgroundSelf-citationThe Story in the Notebook: Exploratory Data Science using a Literate Programming Tool
Kery, M. B., Radensky, M., Arya, M., John, B. E., & Myers, B. A. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173748 · Cited by 214 · cited 3×
“Our Labs) have become the predominant coding environment for data analysis also reveals nine categories of content that data scientists scientists and machine learning engineers [13, 32].”
BackgroundPatterns of Knowledge in API Reference Documentation
Maalej, W., & Robillard, M. P. (2013) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2013.12 · Cited by 183 · cited 1×
“Writing clear and comprehensive documentation is critical to software projects’ development and maintenance [4, 11, 18, 24, 26].”
BackgroundHow do professional developers comprehend software?
Roehm, T., Tiarks, R., Koschke, R., & Maalej, W. (2012) · 2012 34th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2012.6227188 · Cited by 178 · cited 1×
“Writing clear and comprehensive documentation is critical to software projects’ development and maintenance [4, 11, 18, 24, 26].”
BackgroundWhat's Wrong with Computational Notebooks? Pain Points, Needs, and Design Opportunities
Chattopadhyay, S., Prasad, I., Henley, A. Z., Sarma, A., & Barik, T. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376729 · Cited by 159 · cited 1×
“This known challenge of having incomplete or no documentation indeed hinders the readability and reusability of these notebooks, when they are shared with other collaborators or with one’s future self [2].”
BackgroundA Survey of Documentation Practice within Corrective Maintenance
Kajko‐Mattsson, M. (2004) · Empirical Software Engineering · doi:10.1023/b:lida.0000048322.42751.ca · Cited by 129
Listening to programmers — Taxonomies and characteristics of comments in operating system code
Padioleau, Y., Tan, L., & Zhou, Y. (2009) · 2009 IEEE 31st International Conference on Software Engineering · doi:10.1109/icse.2009.5070533 · Cited by 86 · cited 1×
“Programmers often write comments in their source code to make the code easily understood by both themselves and other developers [21].”
BackgroundAutoDS: Towards Human-Centered Automation of Data Science
Wang, D., Andres, J., Weisz, J. D., Oduor, E., & Dugan, C. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445526 · Cited by 75
Self-citationAn Empirical Study on Evolution of API Documentation
Shi, L., Zhong, H., Xie, T., & Li, M. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-19811-3_29 · Cited by 70 · cited 1×
“Writing clear and comprehensive documentation is critical to software projects’ development and maintenance [4, 11, 18, 24, 26].”
BackgroundHacking with NPOs: Collaborative Analytics and Broker Roles in Civic Data Hackathons
Hou, Y., & Wang, D. (2017) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3134688 · Cited by 67
Self-citationHow Much Automation Does a Data Scientist Want?Preprint
Wang, D., Liao, Q. V., Zhang, Y., Khurana, U., Samulowitz, H., Park, S., Muller, M. J., & Amini, L. D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.03970 · Cited by 66 · cited 5×
“…[31] summarized a typical data science workfow as three high-level phases: data preparation, model building, and model deployment, where each phase contains many sub-goals and activities; Similarly, [30] proposes a taxonomy of ten stages and six user personas to describe the complex research space.”
BackgroundMethodInfluentialSelf-citationUnderstanding the Role of Alternatives in Data Analysis Practices
Liu, J., Boukhelifa, N., & Eagan, J. R. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2934593 · Cited by 56 · cited 1×
“Some researchers attribute the cause of this challenge to the iterative and explorative nature of data science projects; data scientists often need to explore multiple hypotheses and candidate coding solutions [14, 17, 23], many of which will be discarded later.”
BackgroundGeiger, R. S., Varoquaux, N., Mazel-Cabasse, C., & Holdgraf, C. (2018) · Computer Supported Cooperative Work (CSCW) · doi:10.1007/s10606-018-9333-1 · Cited by 32 · cited 1×
“(especially open source projects) have low quality documentation, partially due to the low level of intrinsic enjoyment for doing documentation [6].”
BackgroundCrowdsourcing Data Science: A Qualitative Analysis of Organizations’ Usage of Kaggle Competitions
Tauchert, C., Buxmann, P., & Lambinus, J. (2020) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2020.029 · Cited by 30 · cited 1×
“Tauchert et al. [28] studied the motivation of organizers hosting Kaggle competitions.”
BackgroundJupyter Notebooks on GitHub: Characteristics and Code Clones
Källén, M., & Wrigstad, T. (2021) · The Art Science and Engineering of Programming · doi:10.22152/programming-journal.org/2021/5/15 · Cited by 24
Building Community Knowledge In Online Competitions: Motivation, Practices and Challenges
Cheng, R., & Zachry, M. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415250 · Cited by 20 · cited 1×
“Cheng and Zachry [3] interviewed why users participate in Kaggle competitions, and what challenges they have.”
BackgroundTowards Understanding Data Analysis Workflows using a Large Notebook Corpus
Rehman, M. S. (2019) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3299869.3300107 · Cited by 10 · cited 1×
“Some researchers attribute the cause of this challenge to the iterative and explorative nature of data science projects; data scientists often need to explore multiple hypotheses and candidate coding solutions [14, 17, 23], many of which will be discarded later.”
BackgroundProblem-driven teaching activities for the capstone project course of data science
Bai, L., & Hu, Y. (2018) · ACM Turing Celebration Conference - China (ACM TURC) · doi:10.1145/3210713.3210745 · Cited by 4 · cited 1×
“We chose these two competitions because they are the most popular competitions (5,280 notebooks submitted for House Price and 6,300 notebooks submitted for Titanic Survival), and because many data science courses use these two competitions as tutorials [1, 5].”
Background
Research works printed but not linked: 5
- Jesus Fernandez-Bes, Jerónimo Arenas-García, and Jesús Cid-Sueiro. [n.d.]. Energy generation prediction: Lessons learned from the use of Kaggle in Machine Learning Course. Group 7, 8 ([n. d.]), 9.
- 10.1145/3173574.3173748 · DOI · Paper page
- Yaoli Mao, Dakuo Wang, Michael Muller, KUSH VARSHNEY, IOANA Baldini, CASEY Dugan, and ALEKSANDRA MOJSILOVIĆ. 2020. How Data Scientists Work Together With Domain Experts in Scientific Collaborations. In Proceedings of the 2020 ACM conference on GROUP. ACM.
- Krishna Subramanian, Nur Hamdan, and Jan Borchers. 2020. Casual Notebooks and Rigid Scripts: Understanding Data Science Programming. In 2020 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE, 1–5.
- Ge Zhang, Mike A Merrill, Yang Liu, Jeffrey Heer, and Tim Althoff. 2020. CORAL: COde RepresentAtion Learning with Weakly-Supervised Transformers for Analyzing Data Analysis. arXiv preprint arXiv:2008.12828(2020). · arXiv
Cited by
A Survey of Data Quality Requirements That Matter in ML Development Pipelines
Priestley, M., O’donnell, F., & Simperl, E. (2023) · Journal of Data and Information Quality · doi:10.1145/3592616 · Cited by 86
“One possible way of doing this is through the use of interactive notebooks to weave together code and documentation [72].”
BackgroundCausalvis: Visualizations for Causal Inference
Guo, G., Karavani, E., Endert, A., & Kwon, B. C. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581236 · Cited by 36
“These notebooks have been widely adopted for their ability to support exploratory data analysis [39, 57], collaboration [70], rapid iteration and workflow documentation [60].”
MethodOrtloff, A.-M., Martius, F., Meier, M., Sans-Raimbault, T., Geierhaas, L., & Smith, M. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713671 · Cited by 30
“…CHI has investigated aspects of research methods from various stages of the research process, from literature search [98], recruiting [119, 131], types of participants [81, 88] and study methods [40, 111, 119] to data analysis of quantitative [70, 84, 87, 110, 121, 147], and qualitative data [95].”
Code Code Evolution: Understanding How People Change Data Science Notebooks Over Time
Raghunandan, D., Roy, A., Shi, S., Elmqvist, N., & Battle, L. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580997 · Cited by 27
“Part of the problem is the inconsistent notebook quality on GitHub [39].”
BackgroundInfluentialHow Data Scientists Review the Scholarly Literature
Mysore, S., Jasim, M., Song, H., Akbar, S., Randall, A. K. C., & Mahyar, N. (2023) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3576840.3578309 · Cited by 24
“Furthermore, in examining the literature review practices of data scientists our work also extends a recent body of work shedding light on the work practices of data scientists ranging from: teaching practices [68], code and data workflows [79, 88, 120], to documentation practices [130], and others.”
Background
Metadata
- Authors
- Wang, A. Y., Wang, D., Drozdal, J., Liu, X., Park, S., Oney, S., & Brooks, C.
- Affiliations
- University of Michigan (US)
- IBM (United States) (US)
- Rensselaer Polytechnic Institute (US)
- MIT Computer Science and Artificial Intelligence Laboratory (US)
- Venue
- Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2021-05-08
- Pages
- 1-7
- OpenAlex
- W3161776805
- Cited by
- 31
- References
- 25
- Citations by year
- 2026: 2 · 2025: 6 · 2024: 7 · 2023: 11 · 2022: 1 · 2021: 4
- FWCI
- 4.77
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- computational notebooks, documentation practices, data scientists, Kaggle, computational narrative, machine learning competition, code documentation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.