The Stakes of Uncertainty: Developing and Integrating Machine Learning in Clinical Care
Open accessAbstract
The wide‐spread deployment of machine learning tools within healthcare is on the horizon. However, the hype around “AI” tends to divert attention toward the spectacular, and away from the more mundane and ground‐level aspects of new technologies that shape technological adoption and integration. This paper examines the development of a machine learning‐driven sepsis risk detection tool in a hospital Emergency Department in order to interrogate the contingent and deeply contextual ways in which AI technologies are likely be adopted in healthcare. In particular, the paper bring into focus the epistemological implications of introducing a machine learning‐driven tool into a clinical setting by analyzing shifting categories of trust, evidence, and authority. The paper further explores the conditions of certainty in the disciplinary contexts of data science and ethnography, and offers a potential reframing of the work of doing data science and machine learning as “computational ethnography” in order to surface potential pathways for developing effective, human‐centered AI.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
More research is needed to understand clinician views on black box machine learning models that do not explain their predictions in a way that humans can understand.
“Despite a growing body of literature on both physician perspectives of sepsis CDS implementation [26-28] and sepsis machine learning algorithms, more research is needed to understand clinician views on black box machine learning models that do not explain their predictions in a way that humans can understand [29,30].”
From Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study (2020) · cites this paper as [29]
In the Evidence section (page unavailable), the paper discusses physicians’ trust in an uninterpretable Sepsis Watch model and says technical interpretability does not necessarily make a model understandable or trusted by doctors. In “Machine Learning as Alchemy” (page unavailable), it calls for broader understanding of machine-learning systems’ limits, but does not specifically call for more research into clinicians’ views of black-box models. (LLM, conf=0.82)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
How the machine “thinks”: Understanding opacity in machine learning algorithms
Burrell, J. (2016) · Big Data & Society · doi:10.1177/2053951715622512 · Cited by 3,196 · see citing works
Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685 · see citing works
The social power of algorithms
Beer, D. (2017) · Information, Communication & Society · doi:10.1080/1369118x.2016.1216147 · Cited by 945 · see citing works
Flagged first, then by citations
Research works printed but not linked: 33
- Bundles in healthcare refer to a set of predetermined clinical elements of care based on evidence-based practice guidelines and which have been demonstrated to work effectively in concert. The treatment of sepsis with bundles, and the adoption of related protocols as formal federal and state regulation was largely driven by the global health initiative, "Surviving Sepsis Campaign" (2002) which began in 2002. REFERENCES CITED Asad, Talal 1973 Anthropology & the Colonial Encounter. Ithaca, NY: Ithaca Press.
- Beane, Matthew 2018 Shadow Learning: Building Robotic Surgical Skill When Approved Means Fail. Administrative Science Quarterly 33(January), 000183921775169-37.
- Bedoya, Armando, Meredith Clement, M. Phelan R.C. Steorts, Cara O'Brien, and B.A. Goldstein 2018 Minimal Impact of Implemented Early Warning Score and Best Practice Alert for Patient Deterioration. American Journal of Respitory and Critical Care Medicine 197:A4295.
- Cabitza, Federico, Raffaele Rasoini, and Gian Franco Gensini 2017 Unintended Consequences of Machine Learning in Medicine. JAMA 318 (6), 517-2.
- Clifford, James, and George E. Marcus 1986 Writing Culture: The Poetics and Politics of Ethnography. Berkeley, CA: University of California Press.
- Citron, Danielle 2008 Technological Due Process. Washington University Law Review 85, 1249-1313.
- Crawford, Kate and Jason Schultz 2013 Big Data and Due Process: Toward a Framework to Redress Predictive Privacy Harms. Boston College Law Review 55, 93-128.
- Dreyfus, Herbert 1972 What Computers Still Can't Do: A Critique of Artificial Reason. Cambridge, MA: MIT Press.
- Fiore-Gartland, Brittany, and Gina Neff 2015 Communication, Mediation, and the Expectations of Data: Data Valences Across Health and Wellness Communities. International Journal of Communication, May, 1466-84.
- Faubion, James and George E. Marcus 2009 Fieldwork Is Not What It Used To Be: Learning Anthropology's Method in a time of Transition. Ithaca, NY: Cornell University Press.
- Forsythe, Diana 2002 Studying Those Who Study Us: An Anthropologist in the World of Artificial Intelligence. Stanford, CA: Stanford University Press. 1993 Engineering Knowledge: The Construction of Knowledge in Artificial Intelligence. Social Studies of Science 23(3), 445-477.
- Futoma, Joseph, Sanjay Hariharan, Katherine Heller, Mark Sendak, Nathan Brajer, Meredith Clement, Armando Bedoya, and Cara Obrien 2017 An Improved Multi-Output Gaussian Process RNN with Real-Time Validation for Early Sepsis Detection. Proceedings of Machine Learning for Healthcare 68 (August), 1-12.
- Gershgorn, Dave 2017 The world's biggest and most important AI conference. Quartz.com, Dec 12. https://qz.com/1152156/inside-the-worlds-biggest-and-most-important-ai-conference/ (Aug 24, 2018).
- Guidi, Jessica L, Katherine Clark, Mark T Upton, Hilary Faust, Craig A Umscheid, Meghan B Lane-Fall, Mark E Mikkelsen, et al. 2015 Clinician Perception of the Effectiveness of an Automated Early Warning and Response System for Sepsis in an Academic Medical Center. Annals of the American Thoracic Society 12 (10): 1514-19.
- Gupta, Akhil & Ferguson, James 1997 Anthropological Locations: Boundaries and Grounds of a Field Science. Berkeley, CA: University of California Press.
- Haugeland, John 1985 Artificial Intelligence: The Very Idea. Cambridge, MA: MIT Press.
- LeCun, Yann 2017 My take on Ali Rahimi's "Test of Time" award talk at NIPS. Facebook.com/yann.lecun. Facebook post. https://www.facebook.com/yann.lecun/posts/10154938130592143 (August 24, 2018).
- Leslie, Myles, Elise Paradis, Michael A Gropper, Simon Kitto, Scott Reeves, and Peter Pronovost 2017 An Ethnographic Study of Health Information Technology Use in Three Intensive Care Units." Health Services Research 52 (4), 1330-48.
- Lustig, Caitlin, Katie Pine, Bonnie Nardi, Lilly Irani, Min Kyung Lee, Dawn Nafus, and Christian Sandvig 2016 Algorithmic Authority. CHI , 1057-62.
- Maiers, Claire 2017 Analytics in Action: Users and Predictive Data in the Neonatal Intensive Care Unit. Information, Communication & Society 20 (6), 915-29.
- Malinowski, Branislaw 1984 Argonauts of the Western Pacific. Prospect Heights, IL: Waveland Press.
- Mitchell, Tom 1997 Machine Learning. New York: McGraw Hill.
- Mol, Annemarie 2002 The Body Multiple: Ontology in Medical Practice. Durham, NC: Duke University Press.
- Moran, Bruce 2006 Distilling Knowledge: Alchemy, Chemistry, and the Scientific Revolution. Cambridge, MA: Harvard University Press.
- Pasquale, Frank 2015 The Black Box Society: The Secret Algorithms that Control Money and Information. Cambridge, MA: Harvard University Press.
- Rahimi, Ali. 2017. Ali Rahimi's talk at NIPS (NIPS 2017 Test-of-time award presentation). Online video, https://www.youtube.com/watch?v=Qi1Yry33TQE.
- Rhee, Chanu, Raymund Dantes, Lauren Epstein, David J Murphy, Christopher W Seymour, Theodore J Iwashyna, Sameer S Kadri, et al. 2017 Incidence and Trends of Sepsis in US Hospitals Using Clinical vs Claims Data, 2009-2014." JAMA 318 (13), 1241-49.
- Smith, G.B., D.R. Prytherch, P. Meredith, P.E. Schmidt, and P.I. Featherstone 2013 The ability of the National Early Warning Score (NEWS) to discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and death. Resuscitation 84(4): 465-70.
- Shapin, Steve and Simon Schaffer 1985 Leviathan and the Air-Pump: Hobbes, Boyle and the Experimental Life. Princeton, NJ: Princeton University Press.
- Stevens, Hallam 2017 A Feeling for the Algorithm: Working Knowledge and Big Data in Biology. Osiris 32(1), 151-174.
- Suchman, Lucy 2007 Human-Machine Reconfigurations: Plans and Situated Actions, 2 nd ed. New York: Cambridge University Press.
- Taussig, Michael 2011 I Swear I Saw This. Chicago: University of Chicago Press.
- Vinkhuyzen, Erik, Luke Plurkowski, and Gary David 2012 Implementing EMRs: Learnings From a Video Ethnography. EPIC Ethnographic Praxis in Industry Conference, February, 235-48.
Cited by
Sorted by citations
Petersson, L., Larsson, I., Nygren, J. M., Nilsen, P., Neher, M., Reed, J. E., Tyskbo, D., & Svedberg, P. (2022) · BMC Health Services Research · doi:10.1186/s12913-022-08215-8 · Cited by 434 · see citing works
Sendak, M. P., Ratliff, W., Sarro, D., Alderton, E., Futoma, J., Gao, M., Nichols, M., Revoir, M., Yashar, F., Miller, C., et al. (2020) · JMIR Medical Informatics · doi:10.2196/15182 · Cited by 287 · see citing works
Self-citation"The human body is a black box": supporting clinical decision-making with deep learning
Sendak, M., Elish, M. C., Gao, M., Futoma, J., Ratliff, W., Nichols, M., Bedoya, A., Balu, S., & O'Brien, C. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372827 · Cited by 196 · see citing works
Self-citationIntegrating a Machine Learning System Into Clinical Workflows: Qualitative Study
Sandhu, S., Lin, A. L., Brajer, N., Sperling, J., Ratliff, W., Bedoya, A. D., Balu, S., O'Brien, C., & Sendak, M. P. (2020) · Journal of Medical Internet Research · doi:10.2196/22421 · Cited by 168 · see citing works
Sokol, K., Fackler, J., & Vogt, J. E. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01725-9 · Cited by 105 · see citing works
Schwartz, J. M., George, M., Rossetti, S. C., Dykes, P. C., Minshall, S. R., Lucas, E., & Cato, K. D. (2022) · JMIR Human Factors · doi:10.2196/33960 · Cited by 88 · see citing works
“In addition, many have highlighted the importance of engaging clinical end users throughout development and implementation [7,35-38].”
Background"We Would Never Write That Down": Classifications of Unemployed and Data Challenges for AI
Petersen, A. C. M., Christensen, L. R., Harper, R., & Hildebrandt, T. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449176 · Cited by 37 · see citing works
Conceptualising Contestability: Perspectives on Contesting Algorithmic Decisions
Lyons, H., Velloso, E., & Miller, T. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449180 · Cited by 4 · see citing works
- Authors
- ELISH, M. C.
- Affiliations
- Data & Society Research Institute (US)
- Corresponding author
- ELISH, MADELEINE CLARE
- Venue
- Ethnographic Praxis in Industry Conference Proceedings
- Publisher
- Wiley
- Published
- 2018-10
- Volume
- 2018
- Issue
- 1
- Pages
- 364-380
- ISSN
- 1559-890X (ISSN-L)
- 1559-8918
- OpenAlex
- W2911336825
- Cited by
- 59
- References
- 3
- Citations by year
- 2026: 7 · 2025: 6 · 2024: 3 · 2023: 5 · 2022: 13 · 2021: 10 · 2020: 10 · 2019: 4 · 2018: 1
- FWCI
- 6.47
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Ethnographic Praxis in Industry Conference Proceedings (publishedVersion)
- SSRN Electronic Journal (submittedVersion)
- Funders
- Duke Institute for Health Innovation
- Centers for Disease Control and Prevention
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- machine learning in healthcare, trust in AI, epistemology of AI, clinical authority, uncertainty, human-centered AI, emergency department
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
