How Data Science Workers Work with Data: Discovery, Capture, Curation, Design, Creation
The DOI matches its Crossref record; 82 of the 82 references it cites were found in a registry or an index.
Abstract
With the rise of big data, there has been an increasing need for practitioners in this space and an increasing opportunity for researchers to understand their workflows and design new tools to improve it. Data science is often described as data-driven, comprising unambiguous data and proceeding through regularized steps of analysis. However, this view focuses more on abstract processes, pipelines, and workflows, and less on how data science workers engage with the data. In this paper, we build on the work of other CSCW and HCI researchers in describing the ways that scientists, scholars, engineers, and others work with their data, through analyses of interviews with 21 data science professionals. We set five approaches to data along a dimension of interventions: Data as given; as captured; as curated; as designed; and as created. Data science workers develop an intuitive sense of their data and processes, and actively shape their data. We propose new ways to apply these interventions analytically, to make sense of the complex activities around data practices.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Reflective Practitioner: How Professionals Think in Action
Checkoway, B., & Schon, D. A. (1985) · Journal of Policy Analysis and Management · doi:10.2307/3324262 · Cited by 21,268
The reflective practitioner: How professionals think in action
Stewart, C. J. (1984) · Patient Education and Counseling · doi:10.1016/0738-3991(84)90022-3 · Cited by 19,752
(2014) · Cited by 15,747
Basics of Qualitative Research (3rd ed.): Techniques and Procedures for Developing Grounded Theory
Corbin, J., & Strauss, A. (2008) · SAGE · doi:10.4135/9781452230153 · Cited by 12,136 · cited 3×
“These are sometimes called selective codes [21], and they indicate deliberate interpretive choices by the analyst(s) [16], [68].”
(2008) · Cited by 7,674
boyd, D., & Crawford, K. (2012) · Information, Communication & Society · doi:10.1080/1369118x.2012.678878 · Cited by 5,479
(1998) · Medical Entomology and Zoology · Cited by 2,953
What is Wrong with Social Theory?
Blumer, H. (1954) · American Sociological Review · doi:10.2307/2088165 · Cited by 2,535
(2014) · doi:10.4324/9781315799681 · Cited by 2,174
Grounded Theory and Sensitizing Concepts
Bowen, G. A. (2006) · International Journal of Qualitative Methods · doi:10.1177/160940690600500304 · Cited by 1,894
The Fourth Paradigm – Data-Intensive Scientific Discovery
Hey, T. (2012) · Communications in Computer and Information Science · doi:10.1007/978-3-642-33299-9_1 · Cited by 1,585
What Is Wrong with Social Theory?
Blumer, H. (2017) · Sociological Methods · doi:10.4324/9781315129945-8 · Cited by 1,180
(2013) · The MIT Press eBooks · doi:10.7551/mitpress/9302.001.0001 · Cited by 1,160
Value Sensitive Design and Information Systems
Friedman, B., Kahn, P. H., & Borning, A. (2008) · The Handbook of Information and Computer Ethics · doi:10.1002/9780470281819.ch4 · Cited by 1,136
Proceedings of the 18th European conference on information systems (ECIS2010)
Tervonen, T., Brock, D. B., Graeff, D. P., & Hillege, H. L. (2010) · Cited by 836
Value Sensitive Design and Information Systems
Friedman, B., Kahn, P. H., Borning, A., & Huldtgren, A. (2013) · Philosophy of Engineering and Technology · doi:10.1007/978-94-007-7844-3_4 · Cited by 678 · cited 1×
“Scholars in the Value Sensitive Design tradition have studied older cases in which the design of a system influenced the kind of data that could be represented, and the reality that was consequently asserted by that system [31].”
BackgroundPotter's Wheel: An Interactive Data Cleaning System
Raman, V., & Hellerstein, J. L. (2001) · Very Large Data Bases · Cited by 667
Editorial—Big Data, Data Science, and Analytics: The Opportunity and Challenge for IS Research
Agarwal, R., & Dhar, V. (2014) · Information Systems Research · doi:10.1287/isre.2014.0546 · Cited by 610
Wrangler: interactive visual specification of data transformation scripts
Kandel, S., Paepcke, A., Hellerstein, J., & Heer, J. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979444 · Cited by 581
The Fourth Paradigm: Data-Intensive Scientific Discovery [Point of View]
Tolle, K. M., Tansley, D. S. W., & Hey, A. J. G. (2011) · Proceedings of the IEEE · doi:10.1109/jproc.2011.2155130 · Cited by 528
Expertise in a complex skill: Diagnosing x-ray pictures.
Lesgold, A. M., Rubinson, H., Feltovich, P. J., Glaser, R. A., Klopfer, D. S., & Wang, Y. (1988) · Cited by 527
The place of the literature review in grounded theory research
Dunne, C. (2011) · International Journal of Social Research Methodology · doi:10.1080/13645579.2010.494930 · Cited by 488 · cited 2×
“[63] and Dunne [26] noted that researchers need to be able to discern new research questions, and to shape their questions based on what is already known.”
BackgroundDeep feature synthesis: Towards automating data science endeavors
Kanter, J. M., & Veeramachaneni, K. (2015) · 2015 IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2015.7344858 · Cited by 421
Grounded theory research: literature reviewing and reflexivity
McGhee, G., Marland, G. R., & Atkinson, J. (2007) · Journal of Advanced Nursing · doi:10.1111/j.1365-2648.2007.04436.x · Cited by 374 · cited 1×
“McGhee et al. [63] and Dunne [26] noted that researchers need to be able to discern new research questions, and to shape their questions based on what is already known.”
BackgroundThe challenges and limits of big data algorithms in technocratic governance
Janssen, M., & Kuk, G. (2016) · Government Information Quarterly · doi:10.1016/j.giq.2016.08.011 · Cited by 363 · cited 1×
“Janssen and Kuk called for enhanced algorithmic transparency in “unravelling the imperceptibility, material-ity and governmentality of how algorithms work ” (em-phasis in original) [46].”
BackgroundThe marks are on the knowledge worker
Kidd, A. (1994) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/191666.191740 · Cited by 325
From interaction to trajectories: designing coherent journeys through user experiences
Benford, S., Giannachi, G., Koleva, B., & Rodden, T. (2009) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1518812 · Cited by 314
Dix, A. (2007) · Electronic Workshops in Computing · doi:10.14236/ewic/hci2007.53 · Cited by 304
Gray, J., Gerlitz, C., & Bounegru, L. (2018) · Big Data & Society · doi:10.1177/2053951718786316 · Cited by 277
Intelligent services for Big Data science
Dobre, C., & Xhafa, F. (2014) · Future Generation Computer Systems · doi:10.1016/j.future.2013.07.014 · Cited by 273
Transparent to whom? No algorithmic accountability without a critical audience
Kemper, J., & Kolkman, D. (2019) · Information, Communication & Society · doi:10.1080/1369118x.2018.1477967 · Cited by 260 · cited 5×
“We take a similar interest to make visible how data work in contemporary data science [51], [74], [79].”
BackgroundInfluentialThe emerging role of data scientists on software development teams
Kim, M., Zimmermann, T., DeLine, R., & Begel, A. (2016) · Proceedings of the 38th International Conference on Software Engineering · doi:10.1145/2884781.2884783 · Cited by 231
Investigating How Experienced UX Designers Effectively Work with Machine Learning
Yang, Q., Scuito, A., Zimmerman, J., Forlizzi, J., & Steinfeld, A. (2018) · Proceedings of the 2018 Designing Interactive Systems Conference · doi:10.1145/3196709.3196730 · Cited by 226
Broken data: Conceptualising data in an emerging world
Pink, S., Ruckenstein, M., Willim, R., & Duque, M. (2018) · Big Data & Society · doi:10.1177/2053951717753228 · Cited by 205
Data journeys: Capturing the socio-material constitution of data objects and flows
Bates, J., Lin, Y.-W., & Goodale, P. (2016) · Big Data & Society · doi:10.1177/2053951716654502 · Cited by 199
The Politics of Measurement and Action
Pine, K. H., & Liboiron, M. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702298 · Cited by 194
Data Scientist: The Engineer of the Future
van der Aalst, W. M. P. (2014) · Proceedings of the I-ESA Conference · doi:10.1007/978-3-319-04948-9_2 · Cited by 180
Curiosity, Creativity, and Surprise as Analytic Tools: Grounded Theory Method
Muller, M. J. (2014) · Springer eBooks · doi:10.1007/978-1-4939-0378-8_2 · Cited by 173
Self-citationNeff, G., Tanweer, A., Fiore-Gartland, B., & Osburn, L. (2017) · Big Data · doi:10.1089/big.2016.0050 · Cited by 173
Completing Design in Use: Closing the Appropriation Cycle
Carroll, J. M. (2004) · AIS Electronic Library (AISeL) · Cited by 161
Grounding Interactive Machine Learning Tool Design in How Non-Experts Actually Build Models
Yang, Q., Suh, J., Chen, N.-C., & Ramos, G. (2018) · Proceedings of the 2018 Designing Interactive Systems Conference · doi:10.1145/3196709.3196729 · Cited by 156
Donald Schon's Philosophy of Design and Design Education
Waks, L. J. (2001) · International Journal of Technology and Design Education · doi:10.1023/a:1011251801044 · Cited by 154
Data-in-Place: Thinking through the Relations Between Data and Community
Taylor, A. S., Lindley, S., Regan, T., Sweeney, D., Vlachokyriakos, V., Grainger, L., & Lingel, J. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702558 · Cited by 152
Methodology for materiality: interaction design research through a material lens
Wiberg, M. (2013) · Personal and Ubiquitous Computing · doi:10.1007/s00779-013-0686-7 · Cited by 145
Uhlir, P. F., & Schröder, P. (2007) · Data Science Journal · doi:10.2481/dsj.6.od36 · Cited by 133
Investigating statistical machine learning as a tool for software development
Patel, K., Fogarty, J., Landay, J. A., & Harrison, B. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357160 · Cited by 129
Good Data Won't Guarantee Good Decisions
Shah, S., Horne, A., & Capella, J. (2012) · Harvard business review · Cited by 122
Feinberg, M. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025837 · Cited by 113
Proactive wrangling: mixed-initiative end-user programming of data transformation scripts
Guo, P. J., Kandel, S., Hellerstein, J. M., & Heer, J. (2011) · Proceedings of the 24th annual ACM symposium on User interface software and technology · doi:10.1145/2047196.2047205 · Cited by 109
Structures, forms, and stuff: the materiality and medium of interaction
Gross, S., Bardzell, J., & Bardzell, S. (2013) · Personal and Ubiquitous Computing · doi:10.1007/s00779-013-0689-4 · Cited by 108
Information ecology: open system environment for data, memories, and knowing
Baker, K. S., & Bowker, G. C. (2007) · Journal of Intelligent Information Systems · doi:10.1007/s10844-006-0035-7 · Cited by 104
Data Vision: Learning to See Through Algorithmic Abstraction
Passi, S., & Jackson, S. (2017) · Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing · doi:10.1145/2998181.2998331 · Cited by 100
Rethinking empirical social sciences
Ruppert, E. S. (2013) · Dialogues in Human Geography · doi:10.1177/2043820613514321 · Cited by 99
Gender, Productivity, and Prestige in Computer Science Faculty Hiring Networks
Way, S. F., Larremore, D. B., & Clauset, A. (2016) · Proceedings of the 25th International Conference on World Wide Web · doi:10.1145/2872427.2883073 · Cited by 90
Predictive Interaction for Data Transformation
Heer, J., Hellerstein, J. M., & Kandel, S. (2015) · Conference on Innovative Data Systems Research · Cited by 82
Crowdsourcing Ground Truth for Medical Relation Extraction
Dumitrache, A., Aroyo, L., & Welty, C. (2018) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3152889 · Cited by 81 · cited 1×
“I-04 and colleagues were acutely aware that the initial derivation lost the subjective data of the surveys, and the second derivation omitted anyone whose mobile was not actively communicating with a cell-tower (similar to integration issues in [25]).”
BackgroundUnderstanding agency in interaction design materials
Tholander, J., Normark, M., & Rossitto, C. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2208417 · Cited by 77
Digital Entanglements: Craft, Computation and Collaboration in Fine Art Furniture Production
Cheatle, A., & Jackson, S. J. (2015) · Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing · doi:10.1145/2675133.2675291 · Cited by 76
Crowd Truth: Harnessing disagreement in crowdsourcing a relation extraction gold standard
Aroyo, L. M. (2013) · Figshare · doi:10.6084/m9.figshare.679997 · Cited by 68
What Makes a Great Software Engineer
Li, P. L. (2016) · ResearchWorks at the University of Washington (University of Washington) · Cited by 65
Data care and its politics: designing for local collective data management as a neglected thing
Baker, K. S., & Karasti, H. (2018) · Participatory Design Conference · doi:10.1145/3210586.3210587 · Cited by 60
Sensitizing Concepts for Socio-spatial Literacy in HCI
Krogh, P. G., Petersen, M. G., O'Hara, K. P., & Grønbæk, J. E. S. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025756 · Cited by 55
Rosner, D. K., Ikemiya, M., & Regan, T. W. (2015) · International Conference on Tangible, Embedded, and Embodied Interaction (TEI) · doi:10.1145/2677199.2680587 · Cited by 52
Designing as a conversation with digital materials
Dearden, A. (2005) · Design Studies · doi:10.1016/j.destud.2005.11.004 · Cited by 47
Data Diff: Interpretable, Executable Summaries of Changes in Distributions for Data Wrangling
Sutton, C., Hobson, T., Geddes, J., & Caruana, R. (2018) · Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3219819.3220057 · Cited by 42
Crafting the Image in Surgical Telemedicine
Mentis, H. M., Rahim, A., & Theodore, P. R. (2016) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2818048.2819978 · Cited by 41
Assessing Human Error Against a Benchmark of Perfection
Anderson, A., Kleinberg, J. M., & Mullainathan, S. (2016) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2939672.2939803 · Cited by 37
Ribes, D. (2017) · Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing · doi:10.1145/2998181.2998344 · Cited by 36
Assessing Human Error Against a Benchmark of Perfection
Anderson, A., Kleinberg, J. M., & Mullainathan, S. (2017) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3046947 · Cited by 36
Data scientists as game changers in big data environments
Chatfield, A. T., Shlemoon, V. N., Redublado, W., & Rahman, F. (2014) · Tuwhera (Auckland University of Technology) · Cited by 30
Abt, S., & Baier, H. (2014) · doi:10.1145/2666652.2666663 · Cited by 23
A story without end: writing the residual into descriptive infrastructure
Feinberg, M., Carter, D. L., & Bullard, J. (2014) · Designing Interactive Systems Conference · doi:10.1145/2598510.2598553 · Cited by 22
Russell, D. M., Pirolli, P. L. T., Furnas, G. W., et al. (2009) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/1520340.1520732 · Cited by 18
The Poetics of Socio-Technical Space: Evaluating the Internet of Things Through Craft
Lingel, J. (2016) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858399 · Cited by 17
Lingel, J., & Regan, T. W. (2014) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2531602.2531614 · Cited by 13
Introduction: Fundamental Confusion
Michael, J. (2000) · Duke University Press eBooks · doi:10.1215/9780822381396-001 · Cited by 9
Evaluation of manually created ground truth for multi-view people localization
Kiss, Á., & Szirányi, T. (2013) · doi:10.1145/2501105.2501106 · Cited by 8
The work of software development as an assemblage of computational practice
Sim, S. E., Cohn, M. L., & Philip, K. (2009) · International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE) · doi:10.1109/chase.2009.5071419 · Cited by 7
The Future of Data Integration
Miller, R. J. (2017) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3097983.3105809 · Cited by 6
Conceptualization of Computer-Supported Collaborative Sensemaking
Hong, M.-T., & Müller-Birn, C. (2017) · doi:10.1145/3022198.3026351 · Cited by 5
Durrant-Whyte, H. F. (2015) · ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2783258.2785467 · Cited by 4
Translating Texture: Design as Integration
Feinberg, M., Carter, D. L., Bullard, J., & Gursoy, A. (2017) · Designing Interactive Systems Conference · doi:10.1145/3064663.3064730 · Cited by 3
Cited by
Questioning the AI: Informing Design Practices for Explainable AI User Experiences
Liao, Q. V., Gruen, D., & Miller, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376590 · Cited by 925
Self-citation“Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI
Sambasivan, N., Kapania, S., Highfill, H., Akrong, D., Paritosh, P., & Aroyo, L. M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445518 · Cited by 781
“ship have long examined the practices of collaboration, problem formulation, and sensemaking, by humans behind the datasets, including data collectors and scientists, [69, 86, 127], and are designing computational artefacts for dataset development [53].”
BackgroundKaur, H., Nori, H., Jenkins, S., Caruana, R., Wallach, H., & Wortman Vaughan, J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376219 · Cited by 564
Human-AI Collaboration in Data Science: Exploring Data Scientists' Perceptions of Automated AI
Wang, D., Weisz, J. D., Muller, M., Ram, P., Geyer, W., Dugan, C., Tausczik, Y., Samulowitz, H., & Gray, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359313 · Cited by 342
Self-citationOverview and Importance of Data Quality for Machine Learning Tasks
Jain, A., Patel, H., Nagalapatti, L., Gupta, N., Mehta, S., Guttula, S., Mujumdar, S., Afzal, S., Sharma Mittal, R., & Munigala, V. (2020) · Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3394486.3406477 · Cited by 293
Metadata
- Authors
- Muller, M., Lange, I., Wang, D., Piorkowski, D., Tsay, J., Liao, Q. V., Dugan, C., & Erickson, T.
- Venue
- Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2019-05-02
- Pages
- 1-15
- OpenAlex
- W2941766203
- Cited by
- 261
- References
- 82
- FWCI
- 71.20
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.