Data scientists in software teams: State of the art and challenges
Abstract
The demand for analyzing large scale telemetry, machine, and quality data is rapidly increasing in software industry. Data scientists are becoming popular within software teams, e.g., Facebook, LinkedIn and Microsoft are creating a new career path for data scientists. In this paper, we present a large-scale survey with 793 professional data scientists at Microsoft to understand their educational background, problem topics that they work on, tool usages, and activities. We cluster these data scientists based on the time spent for various activities and identify 9 distinct clusters of data scientists, and their corresponding characteristics. We also discuss the challenges that they face and the best practices they share with other data scientists. Our study finds several trends about data scientists in the software engineering context at Microsoft, and should inform managers on how to leverage data science capability effectively within their teams.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The research method is based on existing systematic methods.
“The research method designed for our purpose is based on the existing systematic methods [7,9].”
From Software engineering challenges for machine learning applications: A literature review (2020) · cites this paper as [7]
Combining data is not always straightforward since different systems can have other objectives for storing data, hence leading to data semantics heterogeneity and data integration problems.
“In addition, combining data is not always straightforward (Kim et al., 2017;Lwakatare et al., 2019) since different systems can have other objectives for storing data, hence leading to data semantics heterogeneity and data integration problems.”
From A software engineering perspective on engineering machine learning systems: State of the art and challenges (2021) · cites this paper as Kim et al., 2017
Requirements to be fulfilled by ML components are generally specified using quantitative measures (such as accuracy, precision, recall, F measure) that are new for many stakeholders.
“Requirements to be fulfilled by ML components are generally specified using quantitative measures (such as accuracy, precision, recall, F measure) (Kim et al., 2017;Vogelsang and Borg, 2019;Wan et al., 2019) that are new for many stakeholders.”
From A software engineering perspective on engineering machine learning systems: State of the art and challenges (2021) · cites this paper as Kim et al., 2017
Developers work on extracting and selecting informative features for models.
“Developers work on extracting selecting informative features for models (Amershi et al., 2019;Hill et al., 2016;Kim et al., 2017).”
From A software engineering perspective on engineering machine learning systems: State of the art and challenges (2021) · cites this paper as Kim et al., 2017
The presence of diverse and incompatible programming and data tools is a severe concern for engineering ML systems.
“The presence of diverse and incompatible programming and data tools is a severe concern for engineering ML systems (Kim et al., 2017).”
From A software engineering perspective on engineering machine learning systems: State of the art and challenges (2021) · cites this paper as Kim et al., 2017
While there are generic roles such as data scientist, ML engineer, data analyst, ML software engineer, ML researcher, and software engineer, there are more specific roles, such as data evangelist, data preparer, data shaper, data analyzer.
“While there are generic roles such as data scientist, ML engineer, data analyst, ML software engineer, ML researcher, and software engineer, there are more specific roles, such as data evangelist, data preparer, data shaper, data analyzer (Kim et al., 2017).”
From A software engineering perspective on engineering machine learning systems: State of the art and challenges (2021) · cites this paper as Kim et al., 2017
Practitioners and academicians already started to identify emerging roles to classify skill sets and have proper people in teams to engineer ML systems.
“Practitioners and academicians already started to identify emerging roles to classify skill sets and have proper people in teams to engineer ML systems (Kim et al., 2017;Workera, 2020).”
From A software engineering perspective on engineering machine learning systems: State of the art and challenges (2021) · cites this paper as Kim et al., 2017
It is even challenging to have all these people on the same page.
“It is even challenging to have all these people on the same page (Kim et al., 2017).”
From A software engineering perspective on engineering machine learning systems: State of the art and challenges (2021) · cites this paper as Kim et al., 2017
Cites
Situated Learning: Legitimate Peripheral Participation.
Bloch, M. E. F., Lave, J., & Wenger, É. (1994) · Man · doi:10.2307/2804509 · Cited by 39,323 · see citing works
Lave, J., & Wenger, E. (1991) · Cambridge University Press eBooks · doi:10.1017/cbo9780511815355 · Cited by 33,219 · see citing works
Silhouettes: A graphical aid to the interpretation and validation of cluster analysis
Rousseeuw, P. J. (1987) · Journal of Computational and Applied Mathematics · doi:10.1016/0377-0427(87)90125-7 · Cited by 21,392 · see citing works
Finding groups in data: an introduction to cluster analysis
(1991) · Choice Reviews Online · doi:10.5860/choice.28-4558 · Cited by 10,537 · see citing works
(2011) · Cited by 5,203 · see citing works
Testing Statistical Hypotheses
Niewiadomska-Bugaj, M., & Bartoszynski, R. (2021) · Wiley series in probability and statistics · doi:10.1002/9781119243830.ch12 · Cited by 5,188 · see citing works
Finding Groups in Data: An Introduction to Cluster Analysis.
Sarle, W. S., Kaufman, L., & Rousseeuw, P. J. (1991) · Journal of the American Statistical Association · doi:10.2307/2290430 · Cited by 3,653 · see citing works
Questionnaire Design: How to Plan, Structure and Write Survey Material for Effective Market Research
Brace, I. (2004) · Choice Reviews Online · doi:10.5860/choice.51-0965 · Cited by 1,166 · see citing works
Building knowledge through families of experiments
Basili, V. R., Shull, F. J., & Lanubile, F. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799939 · Cited by 857 · see citing works
Analytics at Work: Smarter Decisions, Better Results
Davenport, T. H., Harris, J. G., & Morison, R. (2009) · Cited by 522 · see citing works
Motivating salespeople: what really works.
Steenburgh, T. J., & Ahearne, M. (2012) · PubMed · Cited by 519 · see citing works
Enterprise Data Analysis and Visualization: An Interview Study
Kandel, S., Paepcke, A., Hellerstein, J. M., & Heer, J. (2012) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2012.219 · Cited by 443 · see citing works
Interactions with big data analytics
Fisher, D., DeLine, R., Czerwinski, M. P., & Drucker, S. M. (2012) · interactions · doi:10.1145/2168931.2168943 · Cited by 363 · see citing works
Self-citationDoing Data Science: Straight Talk from the Frontline
Schutt, R., & O’Neil, C. (2013) · O'Reilly Media, Inc. eBooks · Cited by 287 · see citing works
The emerging role of data scientists on software development teams
Kim, M., Zimmermann, T., DeLine, R. A., & Begel, A. (2016) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2884781.2884783 · Cited by 238 · see citing works
Self-citationAnalyze this! 145 questions for data scientists in software engineering
Begel, A., & Zimmermann, T. (2014) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2568225.2568233 · Cited by 212 · see citing works
Self-citationEthical issues in empirical studies of software engineering
Singer, J., & Vinson, N. G. (2002) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2002.1158289 · Cited by 194 · see citing works
Conducting on-line surveys in software engineering
Punter, T., Ciolkowski, M., Freimut, B., & John, I. (2003) · International Symposium on Empirical Software Engineering · doi:10.1109/isese.2003.1237967 · Cited by 189 · see citing works
Information needs for software development analytics
Buse, R. P. L., & Zimmermann, T. (2012) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2012.6227122 · Cited by 189 · see citing works
Self-citationImproving developer participation rates in surveys
Smith, E., Loftin, R., Murphy-Hill, E. R., Bird, C., & Zimmermann, T. (2013) · International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE) · doi:10.1109/chase.2013.6614738 · Cited by 133 · see citing works
Self-citationSoftware Analytics in Practice
Zhang, D., Han, S., Dang, Y., et al. (2013) · IEEE Software · doi:10.1109/ms.2013.94 · Cited by 91 · see citing works
Zimmermann, T. (2016) · Elsevier eBooks · doi:10.1016/b978-0-12-804206-9.00027-1 · Cited by 91 · see citing works
Self-citationAnalyzing the Analyzers: An Introspective Survey of Data Scientists and Their Work
Harris, H. A. H., Murphy, S. P., & Vaisman, M. (2013) · CERN Document Server (European Organization for Nuclear Research) · Cited by 62 · see citing works
Analytics for software development
Buse, R. P. L., & Zimmermann, T. (2010) · FSE/SDP Workshop on Future of Software Engineering Research (FoSER) · doi:10.1145/1882362.1882379 · Cited by 60 · see citing works
Self-citationData smart: Using data science to transform information into insight
Reed, D. (2014) · Journal of Direct Data and Digital Marketing Practice · doi:10.1057/dddmp.2014.33 · Cited by 57 · see citing works
The New Know: Innovation Powered by Analytics
May, T. A. (2009) · Cited by 51 · see citing works
Testing Statistical Hypotheses
Ross, S. M. (2010) · Elsevier eBooks · doi:10.1016/b978-0-12-374388-6.00009-0 · Cited by 28 · see citing works
Are Software Analytics Efforts Worthwhile for Small Companies? The Case of Amisoft
Robbes, R., Vidal, R., & Bastarrica, M. C. (2013) · IEEE Software · doi:10.1109/ms.2013.92 · Cited by 15 · see citing works
Software analytics: achievements and challenges
Zhang, D., & Xie, T. (2013) · doi:10.5555/2486788.2487049 · Cited by 7 · see citing works
Software analytics: Achievements and challenges
Zhang, D., & Xie, T. (2013) · Proceedings/Proceedings - International Conference on Software Engineering · doi:10.1109/icse.2013.6606753 · Cited by 5 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
“Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI
Sambasivan, N., Kapania, S., Highfill, H., Akrong, D., Paritosh, P., & Aroyo, L. M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445518 · Cited by 781 · see citing works
Challenges in deploying machine learning: A survey of case studies
Paleyes, A., Urma, R.-G., & Lawrence, N. D. (2023) · ACM Computing Surveys · doi:10.1145/3533378 · Cited by 638 · see citing works
Lwakatare, L. E., Raj, A., Crnkovic, I., Bosch, J., & Olsson, H. H. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106368 · Cited by 241 · see citing works
Data Lifecycle Challenges in Production Machine Learning: A Survey
Polyzotis, N., Roy, S., Whang, S. E., & Zinkevich, M. (2018) · ACM SIGMOD Record · doi:10.1145/3299887.3299891 · Cited by 214 · see citing works
Giray, G. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111031 · Cited by 187 · see citing works
How does Machine Learning Change Software Development Practices?
Wan, Z., Xia, X., Lo, D., & Murphy, G. C. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2937083 · Cited by 181 · see citing works
Software engineering challenges for machine learning applications: A literature review
Kumeno, F. (2020) · Intelligent Decision Technologies · doi:10.3233/idt-190160 · Cited by 81 · see citing works
Hopkins, A., & Booth, S. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462527 · Cited by 40 · see citing works
From a Data Science Driven Process to a Continuous Delivery Process for Machine Learning Systems
Lwakatare, L. E., Crnkovic, I., Rånge, E., & Bosch, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-64148-1_12 · Cited by 28 · see citing works
Data Science Project Barriers—A Systematic Review
Labarrère, N., Costa, L., & Lima, R. M. (2025) · Data · doi:10.3390/data10080132 · Cited by 2 · see citing works
- Authors
- Kim, M., Zimmermann, T., DeLine, R., & Begel, A.
- Affiliations
- University of California, Los Angeles (US)
- Microsoft (United States) (US)
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2018-11-01
- Volume
- 44
- Issue
- 11
- Pages
- 1024-1038
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- 2326-3881
- OpenAlex
- W2757656223
- Cited by
- 199
- References
- 30
- Citations by year
- 2026: 7 · 2025: 19 · 2024: 30 · 2023: 28 · 2022: 25 · 2021: 33 · 2020: 29 · 2019: 15 · 2018: 11 · 2017: 2
- FWCI
- 29.00
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Software Engineering (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Decision Sciences › Information Systems and Management › Big Data Technologies and Applications
- Physical Sciences › Computer Science › Information Systems › Information Systems Education and Curriculum Development
- Keywords
- data scientists, software development teams, data science tools, telemetry data analysis, Microsoft
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
