The DOI matches its Crossref record; 24 of the 25 references it cites were found in a registry or an index.
What Makes a Well-Documented Notebook? A Case Study of Data Scientists’ Documentation Practices in Kaggle
Abstract
Many data scientists use computational notebooks to test and present their work, as a notebook can weave code and documentation together (computational narrative), and support rapid iteration on code experiments. However, it is not easy to write good documentation in a data science notebook, partially because there is a lack of a corpus of well-documented notebooks as exemplars for data scientists to follow. To cope with this challenge, this work looks at Kaggle — a large online community for data scientists to host and participate in machine learning competitions — and considers highly-voted Kaggle notebooks as a proxy for well-documented notebooks. Through a qualitative analysis at both the notebook level and the markdown-cell level, we find these notebooks are indeed well documented in reference to previous literature. Our analysis also reveals nine categories of content that data scientists write in their documentation cells, and these documentation cells often interplay with different stages of the data science lifecycle. We conclude the paper with design implications and future research directions.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Longstanding definitions of data quality have viewed good quality data as”
From A Survey of Data Quality Requirements That Matter in ML Development Pipelines (2023) · cites this paper as [74]
In the citing paper
“These dimensions are subsumed by higher-order characterisations that capture the intrinsic, contextual, accessibility, and representational aspects of datasets [74].”
From A Survey of Data Quality Requirements That Matter in ML Development Pipelines (2023) · cites this paper as [74]
In the citing paper
“Below we will draw on Wang & Strong's [74] categorisation of the intrinsic, contextual, accessibility, and representational aspects of datasets to illustrate some of the ways in which previously established data quality categories already apply to ML problems.”
From A Survey of Data Quality Requirements That Matter in ML Development Pipelines (2023) · cites this paper as [74]
In the citing paper
“Merino et al. [47] draw on Wang and Strong [74]'s canonical distinction between the intrinsic, contextual, accessibility, and representational aspects of datasets when using the above mentioned industry standards ISO/IEC 25012 and ISO/IEC 25024.”
From A Survey of Data Quality Requirements That Matter in ML Development Pipelines (2023) · cites this paper as [74]
Citations
Cites
Why Jupyter is data scientists’ computational notebook of choice
Perkel, J. M. (2018) · Nature · doi:10.1038/d41586-018-07196-1 · Cited by 352 · cited 1×
“The data science community has widely adopted computational notebooks under the premise that it can help data scientists to efectively create, communicate, and collaborate on their analysis work [22].”
BackgroundHuman-AI Collaboration in Data Science: Exploring Data Scientists' Perceptions of Automated AI
Wang, D., Weisz, J. D., Muller, M., Ram, P., Geyer, W., Dugan, C., Tausczik, Y., Samulowitz, H., & Gray, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359313 · Cited by 342 · cited 1×
“For example, Wang et al. [31] summarized a typical data science workfow as three high-level phases: data preparation, model building, and model deployment, where each phase contains many sub-goals and activities; Similarly, [30] proposes a taxonomy of ten stages and six user personas to describe…”
BackgroundSelf-citationExploration and Explanation in Computational Notebooks
Rule, A., Tabard, A., & Hollan, J. D. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173606 · Cited by 335 · cited 16×
“Previous studies [13, 25] suggest that data scientists are hesitant to follow the practice of literate programming to create documentation during the development of code, despite they acknowledge the benefts that these informal notes have for recalling the analysis path.”
BackgroundMethodInfluentialA study of the documentation essential to software maintenance
de Souza, S. C. B., Anquetil, N., & de Oliveira, K. M. (2005) · Proceedings of the 23rd annual international conference on Design of communication: documenting & designing for pervasive information · doi:10.1145/1085313.1085331 · Cited by 305 · cited 1×
“Writing clear and comprehensive documentation is critical to software projects’ development and maintenance [4, 11, 18, 24, 26].”
BackgroundA Neural Model for Generating Natural Language Summaries of Program Subroutines
LeClair, A., Jiang, S., & McMillan, C. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00087 · Cited by 294 · cited 2×
“This is promising as there are many breakthroughs in NLP and ML for automatically generating natural language descriptions for code snippets [16].”
BackgroundMethodHow do Data Science Workers Collaborate? Roles, Workflows, and Tools
Zhang, A. X., Muller, M., & Wang, D. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3392826 · Cited by 268 · cited 2×
“Our Labs) have become the predominant coding environment for data analysis also reveals nine categories of content that data scientists scientists and machine learning engineers [13, 32].”
BackgroundSelf-citationHow Data Science Workers Work with Data: Discovery, Capture, Curation, Design, Creation
Muller, M., Lange, I., Wang, D., Piorkowski, D., Tsay, J., Liao, Q. V., Dugan, C., & Erickson, T. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300356 · Cited by 261 · cited 1×
“3451617 cle [30], and has a wide spectrum of usage scenarios [20].”
BackgroundSelf-citationThe Story in the Notebook: Exploratory Data Science using a Literate Programming Tool
Kery, M. B., Radensky, M., Arya, M., John, B. E., & Myers, B. A. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173748 · Cited by 214 · cited 3×
“Our Labs) have become the predominant coding environment for data analysis also reveals nine categories of content that data scientists scientists and machine learning engineers [13, 32].”
BackgroundPatterns of Knowledge in API Reference Documentation
Maalej, W., & Robillard, M. P. (2013) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2013.12 · Cited by 183 · cited 1×
“Writing clear and comprehensive documentation is critical to software projects’ development and maintenance [4, 11, 18, 24, 26].”
BackgroundHow do professional developers comprehend software?
Roehm, T., Tiarks, R., Koschke, R., & Maalej, W. (2012) · 2012 34th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2012.6227188 · Cited by 178 · cited 1×
“Writing clear and comprehensive documentation is critical to software projects’ development and maintenance [4, 11, 18, 24, 26].”
Background
Research works printed but not linked: 5
- Jesus Fernandez-Bes, Jerónimo Arenas-García, and Jesús Cid-Sueiro. [n.d.]. Energy generation prediction: Lessons learned from the use of Kaggle in Machine Learning Course. Group 7, 8 ([n. d.]), 9.
- 10.1145/3173574.3173748 · DOI · Paper page
- Yaoli Mao, Dakuo Wang, Michael Muller, KUSH VARSHNEY, IOANA Baldini, CASEY Dugan, and ALEKSANDRA MOJSILOVIĆ. 2020. How Data Scientists Work Together With Domain Experts in Scientific Collaborations. In Proceedings of the 2020 ACM conference on GROUP. ACM.
- Krishna Subramanian, Nur Hamdan, and Jan Borchers. 2020. Casual Notebooks and Rigid Scripts: Understanding Data Science Programming. In 2020 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE, 1–5.
- Ge Zhang, Mike A Merrill, Yang Liu, Jeffrey Heer, and Tim Althoff. 2020. CORAL: COde RepresentAtion Learning with Weakly-Supervised Transformers for Analyzing Data Analysis. arXiv preprint arXiv:2008.12828(2020). · arXiv
Cited by
A Survey of Data Quality Requirements That Matter in ML Development Pipelines
Priestley, M., O’donnell, F., & Simperl, E. (2023) · Journal of Data and Information Quality · doi:10.1145/3592616 · Cited by 86
“One possible way of doing this is through the use of interactive notebooks to weave together code and documentation [72].”
BackgroundCausalvis: Visualizations for Causal Inference
Guo, G., Karavani, E., Endert, A., & Kwon, B. C. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581236 · Cited by 36
“These notebooks have been widely adopted for their ability to support exploratory data analysis [39, 57], collaboration [70], rapid iteration and workflow documentation [60].”
MethodOrtloff, A.-M., Martius, F., Meier, M., Sans-Raimbault, T., Geierhaas, L., & Smith, M. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713671 · Cited by 30
“…CHI has investigated aspects of research methods from various stages of the research process, from literature search [98], recruiting [119, 131], types of participants [81, 88] and study methods [40, 111, 119] to data analysis of quantitative [70, 84, 87, 110, 121, 147], and qualitative data [95].”
Code Code Evolution: Understanding How People Change Data Science Notebooks Over Time
Raghunandan, D., Roy, A., Shi, S., Elmqvist, N., & Battle, L. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580997 · Cited by 27
“Part of the problem is the inconsistent notebook quality on GitHub [39].”
BackgroundInfluentialHow Data Scientists Review the Scholarly Literature
Mysore, S., Jasim, M., Song, H., Akbar, S., Randall, A. K. C., & Mahyar, N. (2023) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3576840.3578309 · Cited by 24
“Furthermore, in examining the literature review practices of data scientists our work also extends a recent body of work shedding light on the work practices of data scientists ranging from: teaching practices [68], code and data workflows [79, 88, 120], to documentation practices [130], and others.”
Background
Metadata
- Authors
- Wang, A. Y., Wang, D., Drozdal, J., Liu, X., Park, S., Oney, S., & Brooks, C.
- Affiliations
- University of Michigan (US)
- IBM (United States) (US)
- Rensselaer Polytechnic Institute (US)
- MIT Computer Science and Artificial Intelligence Laboratory (US)
- Venue
- Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2021-05-08
- Pages
- 1-7
- OpenAlex
- W3161776805
- Cited by
- 31
- References
- 25
- Citations by year
- 2026: 2 · 2025: 6 · 2024: 7 · 2023: 11 · 2022: 1 · 2021: 4
- FWCI
- 4.77
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- computational notebooks, documentation practices, data scientists, Kaggle, computational narrative, machine learning competition, code documentation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: What Makes a Well-Documented Notebook? A Case Study of Data Scientists’ Documentation Practices in Kaggle. https://app.cochat.ai/papers/10.1145/3411763.3451617
- The paper
- Wang, A. Y., Wang, D., Drozdal, J., Liu, X., Park, S., Oney, S., & Brooks, C. (2021). What Makes a Well-Documented Notebook? A Case Study of Data Scientists’ Documentation Practices in Kaggle. Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3411763.3451617
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.