Managing Messes in Computational Notebooks
Open accessAbstract
Data analysts use computational notebooks to write code for analyzing and visualizing data. Notebooks help analysts iteratively write analysis code by letting them interleave code with output, and selectively execute cells. However, as analysis progresses, analysts leave behind old code and outputs, and overwrite important code, producing cluttered and inconsistent notebooks. This paper introduces code gathering tools, extensions to computational notebooks that help analysts find, clean, recover, and compare versions of code in cluttered, inconsistent notebooks. The tools archive all versions of code outputs, allowing analysts to review these versions and recover the subsets of code that produced them. These subsets can serve as succinct summaries of analysis activity or starting points for new analyses. In a qualitative usability study, 12 professional analysts found the tools useful for cleaning notebooks and writing analysis code, and discovered new ways to use them, like generating personal documentation and lightweight versioning.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
AI Modifiers can yield unexpected results due to the inherent unpredictability of AI models.
“Unlike traditional menu buttons which have predictable outcomes, AI Modifiers can yield unexpected results due to the inherent unpredictability of AI models [40,64,95,96].”
From ABScribe: Rapid Exploration & Organization of Multiple Writing Variations in Human-AI Co-Writing Tasks using Large Language Models (2024) · cites this paper as [40]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Weiser, M. (1981) · doi:10.5555/800078.802557 · Cited by 699 · see citing works
How We Refactor, and How We Know It
Murphy-Hill, E., Parnin, C., & Black, A. P. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.41 · Cited by 547 · see citing works
Callahan, S. P., Freire, J., Santos, E., Scheidegger, C., Silva, C. T., & Vo, H. T. (2006) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/1142473.1142574 · Cited by 480 · see citing works
Enterprise Data Analysis and Visualization: An Interview Study
Kandel, S., Paepcke, A., Hellerstein, J. M., & Heer, J. (2012) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2012.219 · Cited by 443 · see citing works
Exploration and Explanation in Computational Notebooks
Rule, A., Tabard, A., & Hollan, J. D. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173606 · Cited by 347 · see citing works
Kery, M. B., Radensky, M., Arya, M., John, B. E., & Myers, B. A. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173748 · Cited by 216 · see citing works
Refactoring Tools: Fitness for Purpose
Murphy-Hill, E. R., & Black, A. P. (2008) · IEEE Software · doi:10.1109/ms.2008.123 · Cited by 155 · see citing works
Hartmann, B., Yu, L., Allison, A., Yang, Y., & Klemmer, S. (2008) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/1449715.1449732 · Cited by 152 · see citing works
Kery, M. B., Horvath, A., & Myers, B. A. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025626 · Cited by 146 · see citing works
Interactive record/replay for web application debugging
Burg, B., Bailey, R. P., Ko, A. J., & Ernst, M. D. (2013) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/2501988.2502050 · Cited by 129 · see citing works
Finding causes of program output with the Java Whyline
Ko, A. J., & Myers, B. A. (2009) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1518942 · Cited by 123 · see citing works
Exploring exploratory programming
Beth Kery, M., & Myers, B. A. (2017) · 2017 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2017.8103446 · Cited by 123 · see citing works
Systematizing pragmatic software reuse
Holmes, R., & Walker, R. J. (2012) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/2377656.2377657 · Cited by 89 · see citing works
BURRITO: Wrapping Your Lab Notebook in Computational Infrastructure
Guo, P. J., & Seltzer, M. (2012) · Digital Access to Scholarship at Harvard (DASH) (Harvard University) · Cited by 83 · see citing works
Interactions for Untangling Messy History in a Computational Notebook
Kery, M. B., & Myers, B. A. (2018) · Proceedings/Proceedings -- IEEE Symposium on Visual Languages and Human-Centric Computing · doi:10.1109/vlhcc.2018.8506576 · Cited by 77 · see citing works
FireCrystal: Understanding interactive behaviors in dynamic web pages
Oney, S., & Myers, B. A. (2009) · Proceedings/Proceedings -- IEEE Symposium on Visual Languages and Human-Centric Computing · doi:10.1109/vlhcc.2009.5295287 · Cited by 76 · see citing works
Aiding Collaborative Reuse of Computational Notebooks with Annotated Cell Folding
Rule, A., Drosos, I., Tabard, A., & Hollan, J. D. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274419 · Cited by 59 · see citing works
Software history under the lens: A study on why and how developers examine it
Codoban, M., Ragavan, S. S., Dig, D., & Bailey, B. P. (2015) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsm.2015.7332446 · Cited by 57 · see citing works
Foraging Among an Overabundance of Similar Variants
Ragavan, S. S., Kuttal, S. K., Hill, C., Sarma, A., Piorkowski, D., & Burnett, M. (2016) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858469 · Cited by 49 · see citing works
Servant, F., & Jones, J. A. (2012) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2393596.2393646 · Cited by 43 · see citing works
A code reuse interface for non-programmer middle school students
Gross, P. A., Herstand, M. S., Hodges, J. W., & Kelleher, C. L. (2010) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/1719970.1720001 · Cited by 37 · see citing works
Explaining Visual Changes in Web Interfaces
Burg, B., Ko, A. J., & Ernst, M. D. (2015) · doi:10.1145/2807442.2807473 · Cited by 37 · see citing works
Drag-and-Drop Refactoring: Intuitive and Efficient Program Transformation
Lee, Y. Y., Chen, N., & Johnson, R. E. (2012) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign) · Cited by 34 · see citing works
Supporting selective undo in a code editor
Yoon, Y., & Myers, B. A. (2015) · doi:10.5555/2818754.2818784 · Cited by 33 · see citing works
Hibschman, J., & Zhang, H. (2015) · doi:10.1145/2807442.2807468 · Cited by 29 · see citing works
Hempel, B., Lubin, J., Lu, G., & Chugh, R. (2018) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3180155.3180165 · Cited by 27 · see citing works
An exploratory study of backtracking strategies used by developers
Yoon, Y., & Myers, B. A. (2012) · doi:10.5555/2663638.2663669 · Cited by 26 · see citing works
Interactive Extraction of Examples from Existing Code
Head, A., Glassman, E. L., Hartmann, B., & Hearst, M. A. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173659 · Cited by 25 · see citing works
Self-citationHibschman, J., & Zhang, H. (2016) · doi:10.1145/2984511.2984570 · Cited by 23 · see citing works
Identifying Code of Individual Features in Client-Side Web Applications
Maras, J., Štula, M., Carlson, J., & Crnković, I. (2013) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2013.38 · Cited by 21 · see citing works
Oezbek, C., & Prechelt, L. (2007) · Proceedings/Proceedings - Conference on Software Maintenance · doi:10.1109/icsm.2007.4362619 · Cited by 19 · see citing works
Fine-Grained Provenance Collection over Scripts Through Program Slicing
Pimentel, J. F., Freire, J., Murta, L., & Braganholo, V. P. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-40593-3_21 · Cited by 16 · see citing works
Design and Use of Computational Notebooks
Rule, A. (2018) · eScholarship (California Digital Library) · Cited by 10 · see citing works
Interactive and collaborative source code annotation
Suzuki, R. (2015) · doi:10.5555/2819009.2819173 · Cited by 7 · see citing works
Tempura: temporal dimension for IDEs
Lee, Y. Y., Marinov, D., & Johnson, R. E. (2015) · doi:10.5555/2818754.2818783 · Cited by 3 · see citing works
Flagged first, then by citations
Research works printed but not linked: 7
- Steven P. Callahan, Juliana Freire, Emanuele Santos, Carlos E. Schei- degger, Cláudio T. Silva, and Huy T. Vo. 2006. VisTrails: Visualization meets Data Management. In Proceedings of the ACM International Con- ference on Management of Data. ACM, 745-747.
- Brian Hempel, Justin Lubin, Grace Lu, and Ravi Chugh. 2018. Deuce: A Lightweight User Interface for Structured Editing. In Proceedings of the ACM/IEEE International Conference on Software Engineering. ACM/IEEE, 654-664.
- Joshua Hibschman and Haoqi Zhang. 2015. Unravel: Rapid Web Appli- cation Reverse Engineering via Interaction recording, Source Tracing, and Library Detection. In Proceedings of the ACM Symposium on User Interface Software and Technology. ACM, 270-279.
- Joshua Hibschman and Haoqi Zhang. 2016. Telescope: Fine-Tuned Discovery of Interactive Web UI Feature Implementation. In Proceed- ings of the ACM Symposium on User Interface Software and Technology. ACM, 233-245.
- Mary Beth Kery, Amber Horvath, and Brad Myers. 2017. Variolite: Sup- porting Exploratory Programming by Data Scientists. In Proceedings of ACM Conference on Human Factors in Computing Systems. 1265-1276.
- Mary Beth Kery and Brad A. Myers. 2017. Exploring Exploratory Pro- gramming. In Proceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing. IEEE, 25-29.
- Adam Rule, Aurélien Tabard, and James D. Hollan. Data from: Exploration and Explanation in Computational Notebooks. https://doi.org/10.6075/J0JW8C39. · DOI
Cited by
Sorted by citations
What's Wrong with Computational Notebooks? Pain Points, Needs, and Design Opportunities
Chattopadhyay, S., Prasad, I., Henley, A. Z., Sarma, A., & Barik, T. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376729 · Cited by 166 · see citing works
Self-citationHow Data Scientists Use Computational Notebooks for Real-Time Collaboration
Wang, A. Y., Mittal, A., Brooks, C. A., & Oney, S. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359141 · Cited by 131 · see citing works
Collaboration challenges in building ML-enabled systems
Nahar, N., Zhou, S., Lewis, G. A., & Kästner, C. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510209 · Cited by 164 · see citing works
Wang, A. Y., Wang, D., Drozdal, J., Muller, M. J., Park, S., Weisz, J. D., Liu, X., Wu, L., & Dugan, C. (2022) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3489465 · Cited by 82 · see citing works
On the Design of AI-powered Code Assistants for Notebooks
McNutt, A. M., Wang, C., DeLine, R. A., & Drucker, S. M. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580940 · Cited by 81 · see citing works
Self-citationWu, Y., Hellerstein, J. M., & Satyanarayan, A. (2020) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3379337.3415851 · Cited by 70 · see citing works
Understanding the Role of Alternatives in Data Analysis Practices
Liu, J., Boukhelifa, N., & Eagan, J. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2934593 · Cited by 58 · see citing works
Facilitating Knowledge Sharing from Domain Experts to Data Scientists for Building NLP Models
Park, S., Wang, A. Y., Kawas, B., Liao, Q. V., Piorkowski, D., & Danilevsky, M. (2021) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3397481.3450637 · Cited by 61 · see citing works
Fork It: Supporting Stateful Alternatives in Computational Notebooks
Weinman, N., Drucker, S. M., Barik, T., & DeLine, R. A. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445527 · Cited by 57 · see citing works
Self-citationZheng, C., Wang, D., Wang, A. Y., & Ma, X. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517615 · Cited by 54 · see citing works
Nahar, N., Zhang, H., Lewis, G. A., Zhou, S., & Kästner, C. (2023) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1109/cain58948.2023.00034 · Cited by 58 · see citing works
Reza, M., Laundry, N. M., Musabirov, I., Dushniku, P., Yu, Z. Y. “., Mittal, K., Grossman, T., Liut, M., Kuzminykh, A., & Williams, J. J. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641899 · Cited by 52 · see citing works
UMLAUT: Debugging Deep Learning Programs using Program Structure and Model Behavior
Schoop, E., Huang, F., & Hartmann, B. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445538 · Cited by 52 · see citing works
Code Duplication and Reuse in Jupyter Notebooks
Koenzen, A. P., Ernst, N., & Storey, M. (2020) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl/hcc50065.2020.9127202 · Cited by 51 · see citing works
Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decomposition
Kazemitabaar, M., Williams, J., Drosos, I., Grossman, T., Henley, A. Z., Negreanu, C., & Sarkar, A. (2024) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3654777.3676345 · Cited by 55 · see citing works
Symphony: Composing Interactive Interfaces for Machine Learning
Bäuerle, A., Cabrera, Á. A., Hohman, F., Maher, M., Koski, D., Suau, X., Barik, T., & Moritz, D. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502102 · Cited by 50 · see citing works
Self-citationThe Design Space of Computational Notebooks: An Analysis of 60 Systems in Academia and Industry
Lau, S., Drosos, I., Markel, J. M., & Guo, P. J. (2020) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl/hcc50065.2020.9127201 · Cited by 46 · see citing works
Fine-grained lineage for safer notebook interactions
Macke, S., Gong, H., Lee, D. J., Head, A., Xin, D., & Parameswaran, A. (2021) · Proceedings of the VLDB Endowment · doi:10.14778/3447689.3447712 · Cited by 48 · see citing works
Self-citationCallisto: Capturing the "Why" by Connecting Conversations with Computational Narratives
Wang, A. Y., Wu, Z., Brooks, C. A., & Oney, S. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376740 · Cited by 44 · see citing works
ToonNote: Improving Communication in Computational Notebooks Using Interactive Data Comics
Kang, D., Ho, T., Marquardt, N., Mutlu, B., & Bianchi, A. (2021) · International Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445434 · Cited by 44 · see citing works
How Do Data Analysts Respond to AI Assistance? A Wizard-of-Oz Study
Gu, K., Grunde-McLaughlin, M., McNutt, A., Heer, J., & Althoff, T. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641891 · Cited by 40 · see citing works
Notable: On-the-fly Assistant for Data Storytelling in Computational Notebooks
Li, H., Ying, L., Zhang, H., Wu, Y., Qu, H., & Wang, Y. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580965 · Cited by 38 · see citing works
How Do Analysts Understand and Verify AI-Assisted Data Analyses?
Gu, K., Shang, R., Althoff, T., Wang, C., & Drucker, S. M. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642497 · Cited by 45 · see citing works
Self-citationSlide4N: Creating Presentation Slides from Computational Notebooks with Human-AI Collaboration
Wang, F., Liu, X., Liu, O., Neshati, A., Ma, T., Zhu, M., & Zhao, J. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580753 · Cited by 38 · see citing works
Li, X., Zhang, Y., Leung, J., Sun, C. P., & Zhao, J. (2022) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3545995 · Cited by 37 · see citing works
Albireo: An Interactive Tool for Visually Summarizing Computational Notebook Structure
Wenskovitch, J. E., Zhao, J., Carter, S., Cooper, M., & North, C. L. (2019) · IEEE Visualization in Data Science (VDS) · doi:10.1109/vds48975.2019.8973385 · Cited by 35 · see citing works
Diff in the Loop: Supporting Data Comparison in Exploratory Data Analysis
Wang, A. Y., Epperson, W., DeLine, R. A., & Drucker, S. M. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502123 · Cited by 34 · see citing works
Self-citationAspirations and Practice of ML Model Documentation: Moving the Needle with Nudging and Traceability
Bhat, A., Coursey, A., Hu, G., Li, S., Nahar, N., Zhou, S., Kästner, C., & Guo, J. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581518 · Cited by 33 · see citing works
Imprimer: Computational Notebooks for CNC Milling
O'Leary, J. T., Benabdallah, G., & Peek, N. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581334 · Cited by 31 · see citing works
Data Leakage in Notebooks: Static Detection and Better Processes
Yang, C., Brower-Sinning, R. A., Lewis, G. A., & Kästner, C. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3556918 · Cited by 32 · see citing works
CoPrompt: Supporting Prompt Sharing and Referring in Collaborative Natural Language Programming
Feng, L., Yen, R., You, Y., Fan, M., Zhao, J., & Lu, Z. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642212 · Cited by 29 · see citing works
Dead or Alive: Continuous Data Profiling for Interactive Data Science
Epperson, W., Gorantla, V., Moritz, D., & Perer, A. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3327367 · Cited by 28 · see citing works
Subtle Bugs Everywhere: Generating Documentation for Data Wrangling Code
Yang, C., Zhou, S., Guo, J., & Kästner, C. (2021) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678520 · Cited by 28 · see citing works
Eliciting Best Practices for Collaboration with Computational Notebooks
Quaranta, L., Calefato, F., & Lanubile, F. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3512934 · Cited by 28 · see citing works
Ayobi, A., Hughes, J., Duckworth, C. J., Dylag, J. J., Gordon James, S., Marshall, P., Guy, M., Kumaran, A., Chapman, A., Boniface, M., et al. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581424 · Cited by 26 · see citing works
TweakIt: Supporting End-User Programmers Who Transmogrify Code
Lau, S., Ragavan, S. S., Milne, K., Barik, T., & Sarkar, A. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445265 · Cited by 26 · see citing works
Self-citationStickyLand: Breaking the Linear Presentation of Computational Notebooks
Wang, Z. J., Dai, K., & Edwards, W. K. (2022) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3491101.3519653 · Cited by 25 · see citing works
NBSearch: Semantic Search and Visual Exploration of Computational Notebooks
Li, X., Wang, Y., Wang, H., Wang, Y., & Zhao, J. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445048 · Cited by 24 · see citing works
Using Annotations for Sensemaking About Code
Horvath, A., Myers, B. A., Macvean, A., & Rahman, I. (2022) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3526113.3545667 · Cited by 26 · see citing works
Glinda: Supporting Data Science with Live Programming, GUIs and a Domain-specific Language
DeLine, R. A. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445267 · Cited by 23 · see citing works
Self-citationRahman, S., & Kandogan, E. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502068 · Cited by 20 · see citing works
multiverse: Multiplexing Alternative Data Analyses in R Notebooks
Sarma, A., Kale, A. M., Moon, M. J., Taback, N., Chevalier, F., Hullman, J. R., & Kay, M. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580726 · Cited by 26 · see citing works
Wang, F., Lin, Y., Yang, L., Li, H., Gu, M., Zhu, M., & Qu, H. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642865 · Cited by 20 · see citing works
Workflow analysis of data science code in public GitHub repositories
Ramasamy, D., Sarasua, C., Bacchelli, A., & Bernstein, A. (2022) · Empirical Software Engineering · doi:10.1007/s10664-022-10229-z · Cited by 19 · see citing works
Code Code Evolution: Understanding How People Change Data Science Notebooks Over Time
Raghunandan, D., Roy, A., Shi, S., Elmqvist, N., & Battle, L. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580997 · Cited by 19 · see citing works
Unravel: A Fluent Code Explorer for Data Wrangling
Shrestha, N., Barik, T., & Parnin, C. (2021) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3472749.3474744 · Cited by 19 · see citing works
Self-citationWang, C., Lee, B., Drucker, S. M., Marshall, D., & Gao, J. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713296 · Cited by 30 · see citing works
Self-citationBolt-on, Compact, and Rapid Program Slicing for Notebooks
Shankar, S., Macke, S., Chasins, S., Head, A., & Parameswaran, A. (2022) · Proceedings of the VLDB Endowment · doi:10.14778/3565838.3565855 · Cited by 18 · see citing works
Self-citationBug Analysis in Jupyter Notebook Projects: An Empirical Study
de Santana, T. L., Neto, P. A. D. M. S., de Almeida, E. S., & Ahmed, I. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3641539 · Cited by 18 · see citing works
In, S., Krokos, E., Whitley, K. N., North, C. L., & Yang, Y. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642932 · Cited by 18 · see citing works
Splitting, Renaming, Removing: A Study of Common Cleaning Activities in Jupyter Notebooks
Dong, H., Zhou, S., Guo, J., & Kästner, C. (2021) · IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) · doi:10.1109/asew52652.2021.00032 · Cited by 17 · see citing works
ReSplit: Improving the Structure of Jupyter Notebooks by Re-Splitting Their Cells
Titov, S., Golubev, Y., & Bryksin, T. (2022) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner53432.2022.00066 · Cited by 17 · see citing works
ChatDBG: Augmenting Debugging with Large Language Models
Levin, K. H., van Kempen, N., Berger, E. D., & Freund, S. N. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3729355 · Cited by 16 · see citing works
Visualising data science workflows to support third-party notebook comprehension: an empirical study
Ramasamy, D., Sarasua, C., Bacchelli, A., & Bernstein, A. (2023) · Empirical Software Engineering · doi:10.1007/s10664-023-10289-9 · Cited by 21 · see citing works
Exploring Organization of Computational Notebook Cells in 2D Space
Harden, J., Christman, E., Kirshenbaum, N., Wenskovitch, J. E., Leigh, J., & North, C. L. (2022) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl/hcc53370.2022.9833128 · Cited by 16 · see citing works
Sánchez‐Peña, M., Vieira, C., & Magana, A. J. (2022) · Computer Applications in Engineering Education · doi:10.1002/cae.22580 · Cited by 16 · see citing works
Restoring the Executability of Jupyter Notebooks by Automatic Upgrade of Deprecated APIs
Zhu, C., Saha, R. K., Prasad, M. R., & Sarfraz Khurshid, D. (2021) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678889 · Cited by 15 · see citing works
GridBook: Natural Language Formulas for the Spreadsheet Grid
Srinivasa Ragavan, S., Hou, Z., Wang, Y., Gordon, A. D., Zhang, H., & Zhang, D. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511161 · Cited by 15 · see citing works
Gori, J., Han, H. L., & Beaudouin-Lafon, M. (2020) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3379337.3415830 · Cited by 15 · see citing works
TRACTUS: Understanding and Supporting Source Code Experimentation in Hypothesis-Driven Data Science
Subramanian, K., Maas, J., & Borchers, J. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376764 · Cited by 14 · see citing works
Edit - Run Behavior in Programming and Debugging
Alaboudi, A., & LaToza, T. D. (2021) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl/hcc51201.2021.9576170 · Cited by 14 · see citing works
Meta-Manager: A Tool for Collecting and Exploring Meta Information about Code
Horvath, A., Macvean, A., & Myers, B. A. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642676 · Cited by 14 · see citing works
NotePlayer: Engaging Computational Notebooks for Dynamic Presentation of Analytical Processes
Ouyang, Y., Shen, L., Wang, Y., & Li, Q. (2024) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3654777.3676410 · Cited by 13 · see citing works
Notational Programming for Notebook Environments: A Case Study with Quantum Circuits
Arawjo, I., DeArmas, A., Roberts, M., Basu, S., & Parikh, T. S. (2022) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3526113.3545619 · Cited by 13 · see citing works
Colaroid: A Literate Programming Approach for Authoring Explorable Multi-Stage Tutorials
Wang, A. Y., Head, A., Zhang, G., Oney, S., & Brooks, C. A. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581525 · Cited by 12 · see citing works
Self-citationLoops: Leveraging Provenance and Visualization to Support Exploratory Data Analysis in Notebooks
Eckelt, K., Gadhave, K., Lex, A., & Streit, M. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3456186 · Cited by 11 · see citing works
A collaborative semantic-based provenance management platform for reproducibility
Samuel, S., & König‐Ries, B. (2022) · PeerJ Computer Science · doi:10.7717/peerj-cs.921 · Cited by 11 · see citing works
Error identification strategies for Python Jupyter notebooks
Robinson, D. J. S., Ernst, N., Vargas, E. L., & Storey, M.-A. D. (2022) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3524610.3529156 · Cited by 11 · see citing works
Refactoring in Computational Notebooks
Liu, E. S., Lukes, D. A., & Griswold, W. G. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3576036 · Cited by 11 · see citing works
ElasticNotebook: Enabling Live Migration for Computational Notebooks
Li, Z., Gor, P., Prabhu, R., Yu, H., Mao, Y., & Park, Y. (2023) · Proceedings of the VLDB Endowment · doi:10.14778/3626292.3626296 · Cited by 10 · see citing works
Casual Notebooks and Rigid Scripts: Understanding Data Science Programming
Subramanian, K., Hamdan, N. A.-H., & Borchers, J. (2020) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl/hcc50065.2020.9127207 · Cited by 10 · see citing works
Agility in Software 2.0 – Notebook Interfaces and MLOps with Buttresses and Rebars
Borg, M. (2022) · Lecture notes in business information processing · doi:10.1007/978-3-030-94238-0_1 · Cited by 10 · see citing works
How I Met Your Data Science Team: A Tale of Effective Communication
Roy, A., Raghunandan, D., Elmqvist, N., & Battle, L. (2023) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl-hcc57772.2023.00032 · Cited by 9 · see citing works
Persist: Persistent and Reusable Interactions in Computational Notebooks
Gadhave, K., Cutler, Z. T., & Lex, A. (2024) · Computer Graphics Forum · doi:10.1111/cgf.15092 · Cited by 9 · see citing works
Prototyping Text Mining and Network Analysis Tools to Support Netnographic Student Projects
Леонидович, М. И., & Bulygin, D. (2020) · International Journal of Emerging Technologies in Learning (iJET) · doi:10.3991/ijet.v15i10.12313 · Cited by 9 · see citing works
Talaria: Interactively Optimizing Machine Learning Models for Efficient Inference
Hohman, F., Wang, C., Lee, J., Görtler, J., Moritz, D., Bigham, J. P., Ren, Z., Foret, C., Shan, Q., & Zhang, X. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642628 · Cited by 8 · see citing works
Self-citationBetween Scripts and Applications: Computational Media for the Frontier of Nanoscience
Nouwens, M., Borowski, M., Fog, B. V., & Klokmose, C. N. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376287 · Cited by 8 · see citing works
Themisto: Towards Automated Documentation Generation in Computational Notebooks.Preprint
Wang, A. Y., Wang, D., Drozdal, J., Muller, M. J., Park, S., Weisz, J. D., Liu, X., Wu, L., & Dugan, C. (2021) · arXiv (Cornell University) · Cited by 8 · see citing works
Elevating Jupyter Notebook Maintenance Tooling by Identifying and Extracting Notebook Structures
Jiang, Y. Y., Kästner, C., & Zhou, S. (2022) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme55016.2022.00047 · Cited by 7 · see citing works
A Theory of Scientific Programming Efficacy
Pertseva, E., Chang, M., Zaman, U., & Coblenz, M. J. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639139 · Cited by 7 · see citing works
Between Principle and Pragmatism: Reflections on Prototyping Computational Media with Webstrates
Borowski, M., Fog, B. V., Griggio, C. F., Eagan, J., & Klokmose, C. N. (2022) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3569895 · Cited by 7 · see citing works
Enhancing Computational Notebooks with Code+Data Space Versioning
Fang, H., Chockchowwat, S., Sundaram, H., & Park, Y. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714141 · Cited by 6 · see citing works
Tooling for Developing Data-Driven Applications: Overview and Outlook
Weber, T., & Hußmann, H. (2022) · Mensch und Computer (MuC) · doi:10.1145/3543758.3543779 · Cited by 6 · see citing works
How Do Data Science Workers Communicate Intermediate Results?
Pang, R. Y., Wang, R., Nelson, J., & Battle, L. (2022) · IEEE Visualization in Data Science (VDS) · doi:10.1109/vds57266.2022.00010 · Cited by 6 · see citing works
The Product Beyond the Model - An Empirical Study of Repositories of Open-Source ML Products
Nahar, N., Zhang, H., Lewis, G. A., Zhou, S., & Kästner, C. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00006 · Cited by 11 · see citing works
Computational notebooks to support developers in prototyping IoT systems
Corno, F., Russis, L. D., & Sáenz, J. P. (2022) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2022.102850 · Cited by 6 · see citing works
InterLink: Linking Text with Code and Output in Computational Notebooks
Lin, Y., Yang, L., Li, H., Qu, H., & Moritz, D. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714104 · Cited by 5 · see citing works
Harrison, G., Bryson, K., Bamba, A. E. B., Dovichi, L., Binion, A. H., Borem, A., & Ur, B. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642755 · Cited by 5 · see citing works
Keelawat, P. (2023) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3584931.3607020 · Cited by 5 · see citing works
The Role of Organizational Structure in Project Management
Nizma, C., Bangun, R., Benhur, B., Cahyoginarti, C., & Zuardi, M. H. (2024) · Jurnal Syntax Transformation · doi:10.46799/jst.v5i2.928 · Cited by 5 · see citing works
It's a Complete Haystack: Understanding Dependency Management Needs in Computer-Aided Design
Cheng, K., Olechowski, A., & Zhou, S. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757617 · Cited by 4 · see citing works
Pagebreaks: Multi-Cell Scopes in Computational Notebooks
Rawn, E., & Chasins, S. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713620 · Cited by 4 · see citing works
How Scientists Use Jupyter Notebooks: Goals, Quality Attributes, and Opportunities
Huang, R. L., Ravi, S., He, M., Tian, B., Lerner, S., & Coblenz, M. J. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00232 · Cited by 4 · see citing works
Facilitating Dependency Exploration in Computational Notebooks
Brown, C., Alhoori, H., & Koop, D. (2023) · Workshop on Human-In-the-Loop Data Analytics (HILDA) · doi:10.1145/3597465.3605222 · Cited by 4 · see citing works
Chen, A., Wong, C., Sharif, B., & Peruma, A. (2025) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc66645.2025.00043 · Cited by 4 · see citing works
multiverse: Multiplexing Alternative Data Analyses in R NotebooksPreprint
Sarma, A., Kale, A. M., Moon, M. J., Taback, N., Chevalier, F., Hullman, J. R., & Kay, M. (2021) · doi:10.31219/osf.io/yfbwm · Cited by 4 · see citing works
Multiverse Notebook: Shifting Data Scientists to Time Travelers
Sato, S., & Nakamaru, T. (2024) · Proceedings of the ACM on Programming Languages · doi:10.1145/3649838 · Cited by 3 · see citing works
Ciência de Dados com Reprodutibilidade usando Jupyter
Pimentel, J. F., Oliveira, G. P., Silva, M. O., Seufitelli, D. B., & Moro, M. M. (2021) · doi:10.5753/sbc.6757.3.1 · Cited by 3 · see citing works
Quaranta, L. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510454.3517055 · Cited by 3 · see citing works
Odyssey: An Interactive Workbench for Expert-Driven Floating-Point Expression Rewriting
Misback, E., Chan, C. C., Saiki, B., Jun, E., Tatlock, Z., & Panchekha, P. (2023) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3586183.3606819 · Cited by 3 · see citing works
- Authors
- Head, A., Hohman, F., Barik, T., Drucker, S. M., & DeLine, R.
- Affiliations
- University of California, Berkeley (US)
- Georgia Institute of Technology (US)
- Microsoft (United States) (US)
- Venue
- Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2019-05-02
- Pages
- 1-12
- OpenAlex
- W2941232686
- Cited by
- 165
- References
- 35
- Citations by year
- 2026: 11 · 2025: 19 · 2024: 22 · 2023: 30 · 2022: 45 · 2021: 23 · 2020: 12 · 2019: 3
- FWCI
- 16.40
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Data Visualization and Analytics
- Keywords
- computational notebooks, code versioning, code reuse, data analysis pipeline, code documentation, Jupyter notebooks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review