When the Machine Meets the Expert: An Ethnography of Developing AI for Hiring
Abstract
The introduction of machine learning (ML) in organizations comes with the claim that algorithms will produce insights superior to those of experts by discovering the “truth” from data. Such a claim gives rise to a tension between the need to produce knowledge independent of domain experts and the need to remain relevant to the domain the system serves. This two-year ethnographic study focuses on how developers managed this tension when building an ML system to support the process of hiring job candidates at a large international organization. Despite the initial goal of getting domain experts “out the loop,” we found that developers and experts arrived at a new hybrid practice that relied on a combination of ML and domain expertise. We explain this outcome as resulting from a process of mutual learning in which deep engagement with the technology triggered actors to reflect on how they produced knowledge. These reflections prompted the developers to iterate between excluding domain expertise from the ML system and including it. Contrary to common views that imply an opposition between ML and domain expertise, our study foregrounds their interdependence and as such shows the dialectic nature of developing ML. We discuss the theoretical implications of these findings for the literature on information technologies and knowledge work, information system development and implementation, and human–ML hybrids.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Alavi, M., & Leidner, D. E. (2001) · MIS Quarterly · doi:10.2307/3250961 · Cited by 10,176 · see citing works
Judgment under Uncertainty: Heuristics and Biases
(1982) · doi:10.1017/cbo9780511809477 · Cited by 8,240 · see citing works
Strategies for Theorizing from Process Data
Langley, A. (1999) · The Academy of Management Review · doi:10.5465/amr.1999.2553248 · Cited by 7,008 · see citing works
Business intelligence and analytics: from big data to big impact
Chen, H., Chiang, R. H. L., & Storey, V. C. (2012) · MIS Quarterly · doi:10.2307/41703503 · Cited by 6,250 · see citing works
The Duality of Technology: Rethinking the Concept of Technology in Organizations
Orlikowski, W. J. (1992) · Organization Science · doi:10.1287/orsc.3.3.398 · Cited by 4,957 · see citing works
The Black Box Society: The Secret Algorithms That Control Money and Information
Pasquale, F. (2015) · doi:10.4159/harvard.9780674736061 · Cited by 3,796 · see citing works
Explaining Development and Change in Organizations
van de Ven, A. H., & Poole, M. S. (1995) · The Academy of Management Review · doi:10.5465/amr.1995.9508080329 · Cited by 3,762 · see citing works
Knowledge and Organization: A Social-Practice Perspective
Brown, J. S., & Duguid, P. (2001) · Organization Science · doi:10.1287/orsc.12.2.198.10116 · Cited by 3,391 · see citing works
How the machine “thinks”: Understanding opacity in machine learning algorithms
Burrell, J. (2016) · Big Data & Society · doi:10.1177/2053951715622512 · Cited by 3,196 · see citing works
Understanding Machine Learning: From Theory to Algorithms
Shalev-Shwartz, S., & Ben-David, S. (2014) · doi:10.1017/cbo9781107298019 · Cited by 3,115 · see citing works
Barocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048 · see citing works
Knowing in Practice: Enacting a Collective Capability in Distributed Organizing
Orlikowski, W. J. (2002) · Organization Science · doi:10.1287/orsc.13.3.249.2776 · Cited by 3,021 · see citing works
Cook, S. D. N., & Brown, J. S. (1999) · Organization Science · doi:10.1287/orsc.10.4.381 · Cited by 2,894 · see citing works
Artificial Intelligence and Management: The Automation–Augmentation Paradox
Raisch, S., & Krakowski, S. (2021) · Academy of Management Review · doi:10.5465/amr.2018.0072 · Cited by 2,755 · see citing works
Big Data, new epistemologies and paradigm shifts
Kitchin, R. (2014) · Big Data & Society · doi:10.1177/2053951714528481 · Cited by 2,355 · see citing works
The firm as a distributed knowledge system: A constructionist approach
Tsoukas, H. (1996) · Strategic Management Journal · doi:10.1002/smj.4250171104 · Cited by 2,011 · see citing works
Fairness and Abstraction in Sociotechnical Systems
Selbst, A. D., Boyd, D., Friedler, S. A., Venkatasubramanian, S., & Vertesi, J. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287598 · Cited by 1,631 · see citing works
Artificial Intelligence in Human Resources Management: Challenges and a Path Forward
Tambe, P., Cappelli, P., & Yakubovich, V. (2019) · California Management Review · doi:10.1177/0008125619867910 · Cited by 1,609 · see citing works
The State of the Notion: Knowledge Management in Practice
Ruggles, R. (1998) · California Management Review · doi:10.2307/41165944 · Cited by 1,298 · see citing works
Levina, N., & Vaast, E. (2005) · MIS Quarterly · doi:10.2307/25148682 · Cited by 1,281 · see citing works
What is Organizational Knowledge?
Tsoukas, H., & Vladimirou, E. (2001) · Journal of Management Studies · doi:10.1111/1467-6486.00268 · Cited by 1,148 · see citing works
Power to the People: The Role of Humans in Interactive Machine Learning
Amershi, S., Cakmak, M., Knox, W. B., & Kulesza, T. (2014) · AI Magazine · doi:10.1609/aimag.v35i4.2513 · Cited by 1,084 · see citing works
Why Information Technology Inspired but Cannot Deliver Knowledge Management
McDermott, R. (1999) · California Management Review · doi:10.2307/41166012 · Cited by 985 · see citing works
Interactive machine learning for health informatics: when do we need the human-in-the-loop?
Holzinger, A. (2016) · Brain Informatics · doi:10.1007/s40708-016-0042-6 · Cited by 906 · see citing works
Knowledge management and innovation: networks and networking
Swan, J. A., Newell, S., Scarbrough, H., & Hislop, D. W. (1999) · Journal of Knowledge Management · doi:10.1108/13673279910304014 · Cited by 845 · see citing works
Re-Embedding Situatedness: The Importance of Power Relations in Learning Theory
Contu, A., & Willmott, H. C. (2003) · Organization Science · doi:10.1287/orsc.14.3.283.15167 · Cited by 818 · see citing works
Maanen, J. V. (2011) · University of Chicago Press eBooks · doi:10.7208/chicago/9780226849638.001.0001 · Cited by 788 · see citing works
Systematic Analysis of Breast Cancer Morphology Uncovers Stromal Features Associated with Survival
Beck, A. H., Sangoi, A. R., Leung, S. C. Y., Marinelli, R. J., Nielsen, T. O., van de Vijver, M. J., West, R. B., van de Rijn, M., & Koller, D. (2011) · Science Translational Medicine · doi:10.1126/scitranslmed.3002564 · Cited by 768 · see citing works
Schultze, U., & Leidner, D. E. (2002) · MIS Quarterly · doi:10.2307/4132331 · Cited by 668 · see citing works
Editorial —Big Data, Data Science, and Analytics: The Opportunity and Challenge for IS Research
Agarwal, R., & Dhar, V. (2014) · Information Systems Research · doi:10.1287/isre.2014.0546 · Cited by 611 · see citing works
A Confessional Account of an Ethnography About Knowledge Work1
Schultze, U. (2000) · MIS Quarterly · doi:10.2307/3250978 · Cited by 563 · see citing works
Are we there yet? What's next for HR?
Ulrich, D., & Dulebohn, J. H. (2015) · Human Resource Management Review · doi:10.1016/j.hrmr.2015.01.004 · Cited by 495 · see citing works
Social practices and the management of knowledge in project environments
Bresnen, M., Edelman, L. F., Newell, S., Scarbrough, H., & Swan, J. A. (2003) · International Journal of Project Management · doi:10.1016/s0263-7863(02)00090-x · Cited by 469 · see citing works
Practice as the Site of Knowing: Insights from the Field of Telemedicine
Nicolini, D. (2011) · Organization Science · doi:10.1287/orsc.1100.0556 · Cited by 451 · see citing works
Situating methods in the magic of Big Data and AI
Elish, M. C., & boyd, D. (2017) · Communication Monographs · doi:10.1080/03637751.2017.1375130 · Cited by 414 · see citing works
IT to Support Knowledge Sharing in Communities, towards a Social Capital Analysis
Huysman, M., & Wulf, V. (2005) · Journal of Information Technology · doi:10.1057/palgrave.jit.2000053 · Cited by 406 · see citing works
Self-citationAgile requirements engineering practices and challenges: an empirical study
Ramesh, B., Cao, L., & Baskerville, R. (2007) · Information Systems Journal · doi:10.1111/j.1365-2575.2007.00259.x · Cited by 386 · see citing works
The Rise of Big Data: How It’s Changing the Way We Think about the World
Cukier, K. N., & Mayer‐Schönberger, V. (2014) · Princeton University Press eBooks · doi:10.1515/9781400865307-003 · Cited by 364 · see citing works
Learning from notes: Organizational issues in groupware implementation
Orlikowski, W. J. (1993) · The Information Society · doi:10.1080/01972243.1993.9960143 · Cited by 342 · see citing works
Knowledge management technology and the reproduction of knowledge work practices
Schultze, U., & Boland, R. J. (2000) · The Journal of Strategic Information Systems · doi:10.1016/s0963-8687(00)00043-3 · Cited by 308 · see citing works
Substituting Human Decision-Making with Machine Learning: Implications for Organizational Learning
Balasubramanian, N., Ye, Y., & Xu, M. (2022) · Academy of Management Review · doi:10.5465/amr.2019.0470 · Cited by 299 · see citing works
Why Heideggerian AI Failed and How Fixing it Would Require Making it More Heideggerian
Dreyfus, H. L. (2007) · Philosophical Psychology · doi:10.1080/09515080701239510 · Cited by 294 · see citing works
Lindebaum, D., Vesa, M., & den Hond, F. (2020) · Academy of Management Review · doi:10.5465/amr.2018.0181 · Cited by 290 · see citing works
Information Systems Development and Data Modeling
Hirschheim, R., Klein, H. K., & Lyytinen, K. (1995) · Cambridge University Press eBooks · doi:10.1017/cbo9780511895425 · Cited by 286 · see citing works
Utilizing big data analytics for information systems research: challenges, promises and guidelines
Müller, O., Junglas, I. A., Brocke, J. V., & Debortoli, S. (2016) · European Journal of Information Systems · doi:10.1057/ejis.2016.2 · Cited by 282 · see citing works
Mechanical versus clinical data combination in selection and admissions decisions: A meta-analysis.
Kuncel, N. R., Klieger, D. M., Connelly, B. S., & Öneş, D. S. (2013) · Journal of Applied Psychology · doi:10.1037/a0034156 · Cited by 262 · see citing works
The Process and Product of System Design
Boland, R. J. (1978) · Management Science · doi:10.1287/mnsc.24.9.887 · Cited by 254 · see citing works
Make Way for the Algorithms: Symbolic Actions and Change in a Regime of Knowing
Pachidi, S., Berends, H., Faraj, S. A., & Huysman, M. (2020) · Organization Science · doi:10.1287/orsc.2020.1377 · Cited by 238 · see citing works
Self-citationPractices of managing knowledge sharing: towards a second wave of knowledge management
Huysman, M., & de Wit, D. (2004) · Knowledge and Process Management · doi:10.1002/kpm.192 · Cited by 218 · see citing works
Self-citationToward a Better Understanding of Tacit Knowledge in Organizations: Taking Stock and Moving Forward
Hadjimichael, D., & Tsoukas, H. (2019) · Academy of Management Annals · doi:10.5465/annals.2017.0084 · Cited by 218 · see citing works
Beyond design and use: How scholars should study intelligent technologies
Bailey, D. E., & Barley, S. R. (2019) · Information and Organization · doi:10.1016/j.infoandorg.2019.100286 · Cited by 215 · see citing works
Representations and actions: the transformation of work practices with IT use
Vaast, E., & Walsham, G. (2004) · Information and Organization · doi:10.1016/j.infoandorg.2004.10.001 · Cited by 206 · see citing works
Schildt, H. A. (2016) · Innovation · doi:10.1080/14479338.2016.1252043 · Cited by 203 · see citing works
The Politics of Measurement and Action
Pine, K. H., & Liboiron, M. (2015) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702298 · Cited by 201 · see citing works
Engineering Knowledge: The Construction of Knowledge in Artificial Intelligence
Forsythe, D. E. (1993) · Social Studies of Science · doi:10.1177/0306312793023003002 · Cited by 167 · see citing works
Bechmann, A., & Bowker, G. C. (2019) · Big Data & Society · doi:10.1177/2053951718819569 · Cited by 162 · see citing works
Rai, A., Constantinides, P., & Sarker, S. (2019) · MIS Quarterly · doi:10.25300/misq/2019/431e0 · Cited by 153 · see citing works
Mitchell, T. M., Carbonell, J. G., & Michalski, R. S. (1986) · Kluwer international series in engineering and computer science · doi:10.1007/978-1-4613-2279-5 · Cited by 136 · see citing works
Leonardi, P. M. (2009) · Communication Theory · doi:10.1111/j.1468-2885.2009.01344.x · Cited by 127 · see citing works
What we talk about when we talk about (big) data
Jones, M. (2018) · The Journal of Strategic Information Systems · doi:10.1016/j.jsis.2018.10.005 · Cited by 123 · see citing works
Keeping the Wheels Turning: The Dynamics of Managing Networks of Practice
Agterberg, M., van den Hooff, B., Huysman, M., & Soekijad, M. (2009) · Journal of Management Studies · doi:10.1111/j.1467-6486.2009.00867.x · Cited by 120 · see citing works
Self-citationManaging Knowledge and Managing Knowledge Work: What we know and what the Future holds
Newell, S. (2014) · Journal of Information Technology · doi:10.1057/jit.2014.12 · Cited by 104 · see citing works
High Reliability in Digital Organizing: Mindlessness, the Frame Problem, and Digital Operations1
Salovaara, A., Lyytinen, K., & Penttinen, E. (2019) · MIS Quarterly · doi:10.25300/misq/2019/14577 · Cited by 99 · see citing works
Why Artificial Intelligence Will Not Outsmart Complex Knowledge Work
Pettersen, L. (2018) · Work Employment and Society · doi:10.1177/0950017018817489 · Cited by 93 · see citing works
Anthony, C. (2018) · Academy of Management Review · doi:10.5465/amr.2016.0334 · Cited by 84 · see citing works
Seidel, S., Berente, N., Lindberg, A., Lyytinen, K., & Nickerson, J. V. (2018) · Communications of the ACM · doi:10.1145/3210753 · Cited by 72 · see citing works
Data Science in Radiology: A Path Forward
Aerts, H. J. W. L. (2017) · Clinical Cancer Research · doi:10.1158/1078-0432.ccr-17-2804 · Cited by 67 · see citing works
Building truths in AI: Making predictive algorithms doable in healthcare
Henriksen, A., & Bechmann, A. (2020) · Information, Communication & Society · doi:10.1080/1369118x.2020.1751866 · Cited by 58 · see citing works
Rethinking organizational learning: analyzing learning processes of information system designers
Huysman, M. (2000) · Accounting Management and Information Technologies · doi:10.1016/s0959-8022(99)00018-1 · Cited by 56 · see citing works
Self-citationVaast, E., & Pinsonneault, A. (2021) · MIS Quarterly · doi:10.25300/misq/2021/16024 · Cited by 56 · see citing works
Four integration patterns: a socio‐technical approach to integration in IS development projects
Bygstad, B., Nielsen, P. A., & Munkvold, B. E. (2009) · Information Systems Journal · doi:10.1111/j.1365-2575.2007.00280.x · Cited by 52 · see citing works
Hiring algorithms : An ethnography of fairness in practice
van den Broek, E., Sergeeva, A., & Huysman, M. (2019) · Data Archiving and Networked Services (DANS) · Cited by 50 · see citing works
Self-citationA Dramaturgical Model of the Production of Performance Data1
da Cunha, J. V. (2013) · MIS Quarterly · doi:10.25300/misq/2013/37.3.03 · Cited by 40 · see citing works
Datafied knowledge production: Introduction to the special theme
Thylstrup, N. B., Flyverbom, M., & Helles, R. (2019) · Big Data & Society · doi:10.1177/2053951719875985 · Cited by 36 · see citing works
Artificial Intelligence and Video Game Creation: A Framework for the New Logic of Autonomous Design
Seidel, S., Berente, N., Lindberg, A., Lyytinen, K., Martinez, B., & Nickerson, J. V. (2020) · Journal of Digital Social Research · doi:10.33621/jdsr.v2i3.46 · Cited by 29 · see citing works
Prediction, Judgment, and Complexity
Agrawal, A. K., Gans, J., & Avi Goldfarb, B. (2019) · University of Chicago Press eBooks · doi:10.7208/chicago/9780226613475.003.0003 · Cited by 28 · see citing works
Reframing Information System Design as Learning Across Communities of Practice
Gallagher, K. P., & Mason, R. M. (2007) · International Journal of Technology and Human Interaction · doi:10.4018/jthi.2007100102 · Cited by 6 · see citing works
Flagged first, then by citations
Research works printed but not linked: 8
- Brynjolfsson, 2014, The Second Machine Age: Work, Progress, and Prosperity in a Time of Brilliant Technologies
- Davenport, 2016, Only Humans Need Apply: Winners and Losers in the Age of Smart Machines
- Domingos, 2015, The Master Algorithm: How the Quest for the Ultimate Learning Machine Will Remake Our World
- Ford, 2018, Architects of Intelligence: The Truth About AI from the People Building It
- Lehr, 2017, Playing with the Data: What Legal Scholars Should Learn About Machine Learning, U.C. Davis Law Review, 51, 653
- McCarthy, 1969, Some Philosophical Problems from the Standpoint of Artificial Intelligence, 431
- Miles, 1994, Qualitative Data Analysis: An Expanded Sourcebook
- Siegel, 2013, Predictive Analytics: The Power to Predict Who Will Click, Buy, Lie, or Die
Cited by
Sorted by citations
Lebovitz, S., Lifshitz-Assaf, H., & Levina, N. (2022) · Organization Science · doi:10.1287/orsc.2021.1549 · Cited by 632 · see citing works
Managing artificial intelligence
Berente, N., Gu, B., Recker, J., & Santhanam, R. (2021) · MIS Quarterly · doi:10.25300/misq/2021/16274 · Cited by 898 · see citing works
Toward AI Governance: Identifying Best Practices and Potential Barriers and Outcomes
Papagiannidis, E., Enholm, I. M., Dremel, C., Mikalef, P., & Krogstie, J. (2023) · Information Systems Frontiers · doi:10.1007/s10796-022-10251-y · Cited by 191 · see citing works
Waardenburg, L., Huysman, M., & Sergeeva, A. V. (2022) · Organization Science · doi:10.1287/orsc.2021.1544 · Cited by 165 · see citing works
Self-citationHuman–artificial intelligence collaboration in prediction: a field experiment in the retail industry
Revilla, E., Saenz, M. J., Seifert, M., & Ma, Y. (2023) · Journal of Management Information Systems · doi:10.1080/07421222.2023.2267317 · Cited by 63 · see citing works
Establishing and evaluating trustworthy AI: overview and research challenges
Kowald, D., Scher, S., Pammer-Schindler, V., Müllner, P., Waxnegger, K., Demelius, L., Fessl, A., Toller, M., Mendoza Estrada, I. G., Šimić, I., et al. (2024) · Frontiers in Big Data · doi:10.3389/fdata.2024.1467222 · Cited by 59 · see citing works
Ramaul, L., Ritala, P., Kostis, A., & Aaltonen, P. (2026) · Journal of Management Studies · doi:10.1111/joms.13246 · Cited by 41 · see citing works
Herath, S., Bashardoust, A., Bole, Y., & Shrestha, Y. R. (2026) · European Journal of Information Systems · doi:10.1080/0960085x.2026.2616042 · Cited by 16 · see citing works
Sandeep, M. M., & Lavanya, V. (2026) · Journal of Management Development · doi:10.1108/jmd-03-2024-0082 · Cited by 8 · see citing works
- Authors
- van den Broek, E., Sergeeva, A., & Huysman, M.
- Affiliations
- Vrije Universiteit Amsterdam, De Boelelaan 1105, 1081 HV Amsterdam, The Netherlands
- Vrije Universiteit Amsterdam (NL)
- Venue
- MIS Quarterly
- Publisher
- MIS Quarterly
- Published
- 2021-09-01
- Volume
- 45
- Issue
- 3
- Pages
- 1557-1580
- ISSN
- 0276-7783 (ISSN-L)
- 2162-9730
- OpenAlex
- W3181262202
- Semantic Scholar
- e1059da9abe216bb536118be710031c76e10e0dd
- Cited by
- 330
- References
- 77
- Citations by year
- 2026: 66 · 2025: 94 · 2024: 68 · 2023: 63 · 2022: 29 · 2021: 7 · 2020: 1 · 2019: 2
- FWCI
- 57.86
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- MIS Quarterly (publishedVersion)
- VU Research Portal (publishedVersion)
- Journal of the Association for Information Systems (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- Topics
- Social Sciences › Sociology and Political Science › Information Systems Theories and Implementation
- Social Sciences › Business, Management and Accounting › Organizational Behavior and Human Resource Management › AI and HR Technologies
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Keywords
- machine learning, personnel selection, domain knowledge, human-AI collaboration, hybrid practices, mutual learning, knowledge work, information system development, algorithmic decision support
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
