The DOI matches its Crossref record; 70 of the 70 references it cites were found in a registry or an index.
How do Data Science Workers Collaborate? Roles, Workflows, and Tools
Open accessAbstract
Today, the prominence of data science within organizations has given rise to teams of data science workers collaborating on extracting insights from data, as opposed to individual data scientists working alone. However, we still lack a deep understanding of how data science workers collaborate in practice. In this work, we conducted an online survey with 183 participants who work in various aspects of data science. We focused on their reported interactions with each other (e.g., managers with engineers) and with different tools (e.g., Jupyter Notebook). We found that data science teams are extremely collaborative and work with a variety of stakeholders and tools during the six common steps of a data science workflow (e.g., clean data and train model). We also found that the collaborative practices workers employ, such as documentation, vary according to the kinds of tools they use. Based on these findings, we discuss design implications for supporting data science team collaborations and future research directions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Doyle, T. (2017) · The Information Society · doi:10.1080/01972243.2017.1354593 · Cited by 3,071
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681 · cited 1×
“Recent work in this space similarly argues for more documentation to improve transparency, as well as greater standardization around documentation [27, 65], particularly important when it comes to publicly-released datasets and models.”
BackgroundGuidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439
Communities of practice: A brief introduction
Wenger, É. (2009) · Scholars' Bank (University of Oregon) · Cited by 2,349
Bellamy, R. K. E., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilovic, A., et al. (2018) · arXiv · doi:10.48550/arxiv.1810.01943 · Cited by 1,581
Coordination in software development
Kraut, R. E., & Streeter, L. A. (1995) · Communications of the ACM · doi:10.1145/203330.203345 · Cited by 1,080
Social coding in GitHub: transparency and collaboration in an open software repository
Dabbish, L., Stuart, C., Tsay, J., & Herbsleb, J. (2012) · Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work · doi:10.1145/2145204.2145396 · Cited by 999
Questioning the AI: Informing Design Practices for Explainable AI User Experiences
Liao, Q. V., Gruen, D., & Miller, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376590 · Cited by 925
Zhang, Y., Liao, Q. V., & Bellamy, R. K. E. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372852 · Cited by 895
AI Fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias
Bellamy, R. K. E., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilović, A., et al. (2019) · IBM Journal of Research and Development · doi:10.1147/jrd.2019.2942287 · Cited by 884
Wrangler: interactive visual specification of data transformation scripts
Kandel, S., Paepcke, A., Hellerstein, J., & Heer, J. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979444 · Cited by 581
Where does AlphaGo go: from church-turing thesis to AlphaGo thesis and beyond
Wang, F.-Y., Zhang, J. J., Zheng, X., Wang, X., Yuan, Y., Dai, X., Zhang, J., & Yang, L. (2016) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2016.7471613 · Cited by 342
Exploration and Explanation in Computational Notebooks
Rule, A., Tabard, A., & Hollan, J. D. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173606 · Cited by 335
Algorithmic Bias: From Discrimination Discovery to Fairness-aware Data Mining
Hajian, S., Bonchi, F., & Castillo, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2945386 · Cited by 334
An empirical study of global software development: distance and speed
Herbsleb, J. D., Mockus, A., Finholt, T. A., & Grinter, R. E. (2001) · doi:10.5555/381473.381481 · Cited by 332
Example-centric programming: integrating web search into the development environment
Brandt, J., Dontcheva, M., Weskamp, M., & Klemmer, S. R. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753402 · Cited by 324
Agency plus automation: Designing artificial intelligence into interactive systems
Heer, J. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1807184115 · Cited by 279
Latent social structure in open source projects
Bird, C., Pattison, D., D'Souza, R., Filkov, V., & Devanbu, P. (2008) · Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of software engineering · doi:10.1145/1453101.1453107 · Cited by 276
Heer, J., & Shneiderman, B. (2012) · Queue · doi:10.1145/2133416.2146416 · Cited by 269
Voyagers and voyeurs: supporting asynchronous collaborative information visualization
Heer, J., Viégas, F. B., & Wattenberg, M. (2007) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240781 · Cited by 261
How Data Science Workers Work with Data: Discovery, Capture, Curation, Design, Creation
Muller, M., Lange, I., Wang, D., Piorkowski, D., Tsay, J., Liao, Q. V., Dugan, C., & Erickson, T. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300356 · Cited by 261
Self-citationGarcia, M. (2016) · World Policy Journal · doi:10.1215/07402775-3813015 · Cited by 257 · cited 1×
“We speculate that Managers may become more involved in this stage in the future, as bias issues become more salient in industry and academia [1, 26, 35].”
BackgroundThe emerging role of data scientists on software development teams
Kim, M., Zimmermann, T., DeLine, R., & Begel, A. (2016) · Proceedings of the 38th International Conference on Software Engineering · doi:10.1145/2884781.2884783 · Cited by 231
Passi, S., & Jackson, S. J. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274405 · Cited by 225
The Story in the Notebook: Exploratory Data Science using a Literate Programming Tool
Kery, M. B., Radensky, M., Arya, M., John, B. E., & Myers, B. A. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173748 · Cited by 214
The Politics of Measurement and Action
Pine, K. H., & Liboiron, M. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702298 · Cited by 194
Disempowered by Data: Nonprofits, Social Enterprises, and the Consequences of Data-Driven Work
Bopp, C., Harmon, E., & Voida, A. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025694 · Cited by 167
Mikut, R., & Reischl, M. (2011) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.24 · Cited by 158
Vartak, M., Subramanyam, H., Lee, W.-E., Viswanathan, S., Husnoo, S., Madden, S. R., & Zaharia, M. (2016) · Workshop on Human-In-the-Loop Data Analytics (HILDA) · doi:10.1145/2939502.2939516 · Cited by 157
Early engagement and new technologies: Opening up the laboratory
Doorn, N., Gorman, M. E., van de Poel, I., & Schuurbiers, D. (2013) · Philosophy of engineering and technology · doi:10.1007/978-94-007-7844-3 · Cited by 151
How Data Scientists Use Computational Notebooks for Real-Time Collaboration
Wang, A. Y., Mittal, A., Brooks, C. A., & Oney, S. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359141 · Cited by 138
AI and HCI: Two Fields Divided by a Common Focus
Grudin, J. T. (2009) · AI Magazine · doi:10.1609/aimag.v30i4.2271 · Cited by 126
ICICLE: groupware for code inspection
Brothers, L., Sembugamoorthy, V., & Muller, M. (1990) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/99332.99353 · Cited by 124
Self-citationData Vision: Learning to See Through Algorithmic Abstraction
Passi, S., & Jackson, S. (2017) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2998181.2998331 · Cited by 123
Proactive wrangling: mixed-initiative end-user programming of data transformation scripts
Guo, P. J., Kandel, S., Hellerstein, J. M., & Heer, J. (2011) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/2047196.2047205 · Cited by 116
Feinberg, M. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025837 · Cited by 115
Who’s Got the Data? Interdependencies in Science and Technology Collaborations
Borgman, C. L., Wallis, J. C., & Mayernik, M. S. (2012) · Computer Supported Cooperative Work (CSCW) · doi:10.1007/s10606-012-9169-z · Cited by 111
Shared waypoints and social tagging to support collaboration in software development
Storey, M., Cheng, L.-T., Bull, I., & Rigby, P. C. (2006) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/1180875.1180906 · Cited by 108
How to Make Sense of Team Sport Data: From Acquisition to Data Modeling and Research Aspects
Stein, M., Janetzko, H., Seebacher, D., Jäger, A., Nagel, M., Hölsch, J., Kosub, S., Schreck, T., Keim, D. A., & Grossniklaus, M. (2017) · Data · doi:10.3390/data2010002 · Cited by 102
Towards Effective Foraging by Data Scientists to Find Past Analysis Choices
Kery, M. B., John, B. E., O'Flaherty, P., Horvath, A., & Myers, B. A. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300322 · Cited by 102
Olson, J. S., Wang, D., Olson, G. M., & Zhang, J. (2017) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3038919 · Cited by 101
Self-citationCollaborative Approaches Needed to Close the Big Data Skills Gap
Miller, S. (2014) · Journal of Organization Design · doi:10.7146/jod.9823 · Cited by 98
Kross, S., & Guo, P. J. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300493 · Cited by 91
Why do people tag? motivations for photo tagging
Nov, O., & Ye, C. (2010) · Communications of the ACM · doi:10.1145/1785414.1785450 · Cited by 90
Towards human-guided machine learning
Gil, Y., Honaker, J., Gupta, S., Ma, Y., D'Orazio, V. J., Garijo, D., Gadewar, S. P., Yang, Q., & Jahanshad, N. (2019) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3301275.3302324 · Cited by 84
Huang, H., & Trauth, E. M. (2007) · ACM SIGCPR Conference on Computer Personnel Research (SIGCPR) · doi:10.1145/1235000.1235008 · Cited by 79
Designing task visualizations to support the coordination of work in software development
Halverson, C. A., Ellis, J. B., Danis, C. M., & Kellogg, W. A. (2006) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/1180875.1180883 · Cited by 71
Layered participatory analysis: new developments in the CARD technique
Muller, M. J. (2001) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/365024.365054 · Cited by 70
Self-citationAn ADMM Based Framework for AutoML Pipeline Configuration
Liu, S., Ram, P., Vijaykeerthy, D., Bouneffouf, D., Bramble, G., Samulowitz, H., Wang, D., Conn, A. R., & Gray, A. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.5926 · Cited by 68
Self-citationHou, Y., & Wang, D. (2017) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3134688 · Cited by 67
Self-citationData Scientists Aren't Domain Experts
Viaene, S. (2013) · IT Professional · doi:10.1109/mitp.2013.93 · Cited by 66
Unakite: Scaffolding Developers' Decision-Making Using the Web
Liu, M. X., Hsieh, J., Hahn, N., Zhou, A., Deng, E., Burley, S., Taylor, C., Kittur, A., & Myers, B. A. (2019) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3332165.3347908 · Cited by 66
Weidele, D. K. I., Weisz, J. D., Oduor, E., Muller, M. J., Andrés, J., Gray, A., & Wang, D. (2020) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3377325.3377538 · Cited by 63
Self-citationTools for spreadsheet auditing
Davis, J. S. (1996) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1996.0061 · Cited by 62
Developing a Research Agenda for Human-Centered Data Science
Aragon, C., Hutto, C. J., Echenique, A., Fiore-Gartland, B., Huang, Y. K., Kim, J., Neff, G., Xing, W., & Bayer, J. (2016) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2818052.2855518 · Cited by 53
Exploring the ecosystem of software developers on GitHub and other platforms
Wu, Y. “., Kropczynski, J., Shih, P. C., & Carroll, J. M. (2014) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2556420.2556483 · Cited by 52
Callisto: Capturing the "Why" by Connecting Conversations with Computational Narratives
Wang, A. Y., Wu, Z., Brooks, C. A., & Oney, S. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376740 · Cited by 47
Data Diff: Interpretable, Executable Summaries of Changes in Distributions for Data Wrangling
Sutton, C. A., Hobson, T., Geddes, J., & Caruana, R. (2018) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3219819.3220057 · Cited by 43
HyperSource: bridging the gap between source and code-related web sites
Hartmann, B., Dhillon, M., & Chan, M. K. (2011) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979263 · Cited by 35
Cooperation between developers and operations in software engineering projects
Tessem, B., & Iden, J. (2008) · International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE) · doi:10.1145/1370114.1370141 · Cited by 32
Confronting Social Criticisms: Challenges when Adopting Data-Driven Policing Strategies
Verma, N., & Dombrowski, L. S. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174043 · Cited by 20
Translation between software designers and users
Williams, M. G., & Begg, V. (1993) · Communications of the ACM · doi:10.1145/153571.214831 · Cited by 18
Effecting change: coordination in large-scale software development
Begel, A. (2008) · International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE) · doi:10.1145/1370114.1370119 · Cited by 18
Post-literate Programming: Linking Discussion and Code in Software Development Teams
Park, S., Zhang, A. X., & Karger, D. R. (2018) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3266037.3266098 · Cited by 13
Self-citationPredict Saturated Thickness using TensorBoard Visualization
Nguyen, V. T., Dang, T., & Jin, F. (2018) · Eurographics · doi:10.2312/envirvis.20181135 · Cited by 12
Empirical Studies on Collaboration in Software Development: A Systematic Literature Review
Treude, C., Storey, M., & Weber, J. H. (2009) · Cited by 9
The Semantic Snake Charmer Search Engine
Grappiolo, C., van Gerwen, E., Verhoosel, J. P. C., & Somers, L. (2019) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3295750.3298915 · Cited by 5
Managing collaborative activities in project management
Zhang, S., Zhao, C. G., Zhang, Q., Su, H., Guo, H., Cui, J., Pan, Y., & Moody, P. (2007) · Computer Human Interaction for Management of Information Technology · doi:10.1145/1234772.1234791 · Cited by 4
The Science of Managing Data Science
Matsudaira, K. (2015) · Queue · doi:10.1145/2756506.2767971 · Cited by 3
Parnin, C. (2013) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2013.6645270 · Cited by 1
Cited by
“Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI
Sambasivan, N., Kapania, S., Highfill, H., Akrong, D., Paritosh, P., & Aroyo, L. M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445518 · Cited by 781
Wang, D., Churchill, E., Maes, P., Fan, X., Shneiderman, B., Shi, Y., & Wang, Q. (2020) · Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3334480.3381069 · Cited by 271
Self-citationHuman factors in model interpretability: Industry practices, challenges, and needs
Hong, S. R., Hullman, J., & Bertini, E. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3392878 · Cited by 219
How AI Developers Overcome Communication Challenges in a Multidisciplinary Team: A Case Study
Piorkowski, D., Park, S., Wang, A. Y., Wang, D., Muller, M., & Portnoy, F. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449205 · Cited by 151
Self-citationStudying Up Machine Learning Data: Why Talk About Bias When We Mean Power?
Miceli, M., Posada, J., & Yang, T. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3492853 · Cited by 125
Metadata
- Authors
- Zhang, A. X., Muller, M., & Wang, D.
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2020-05-28
- Volume
- 4
- Issue
- CSCW1
- Pages
- 1-23
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W3029504795
- Cited by
- 268
- References
- 70
- FWCI
- 37.19
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.