CoChat
Sign Up
Cited by 5Artificial Intelligence in Healthcare and EducationTechnology and Human Factors in Education and HealthElectronic Health Records Systems

Repairing Innovation: A Study of Integrating AI in Clinical Care

Open access
Elish, Madeleine Clare, Watkins, Elizabeth Anne·2020
DOI 10.69985/vcvu5761
Open this paper in CoChat — free accountOpen PDF

Abstract

Over the past two years, a multi-disciplinary team of clinicians and technologists associated with Duke University and Duke Health system have developed and implemented Sepsis Watch, a sociotechnical system combining an artificial intelligence (AI) deep learning model with new hospital protocols to raise the quality of sepsis treatment. Sepsis is a widespread and deadly condition that can develop from any infection and is one of the most common causes of death in hospitals. And while sepsis is treatable, it is notoriously difficult to diagnose consistently. This makes sepsis a prime candidate for AI-based interventions, where new approaches to patient data might raise levels of detection, treatment, and, ultimately, patient outcomes in the form of fewer deaths.As an application of AI, the deep learning model tends to eclipse the other parts of the system; in practice, Sepsis Watch is constituted by a complex combination of human labor and expertise, as well as technical and institutional infrastructures. This report brings into focus the critical role of human labor and organizational context in developing an effective clinical intervention by framing Sepsis Watch as a complex sociotechnical system, not just a machine learning model.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.69985/vcvu5761

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 74

  1. Chanu Rhee, Raymund Dantes, Lauren Epstein, David J. Murphy, Christopher W. Sey- mour, Theodore J. Iwashyna, Sameer S. Kadri et al. "Incidence and trends of sepsis in US hospitals using clinical vs claims data, 2009-2014." Jama 318, no. 13 (2017): 1241-1249.
  2. Amber E. Barnato, Sherri L Alexander, Walter T Linde-Zwirble, and Derek C Angus, "Racial variation in the incidence, care, and outcomes of severe sepsis: analysis of population, patient, and hospital characteristics," American Journal of Respiratory and Critical Care Medicine 177, 3(2008):279-284, doi:10.1164/ rccm.200703-480OC; · DOI · Paper page
  3. Justin Xavier Moore, John P. Donnelly, Russell Griffin, Monika M. Safford, George Howard, John Baddley, and Henry E. Wang, "Black-white racial disparities in sepsis: a prospective analysis of the Reasons for Geographic And Racial Differences in Stroke (REGARDS) cohort," Critical care 19, 1 (2015): 279, https://doi.org/10.1186/s13054-015-0992-8. · DOI
  4. Christelle Tan, Kristin Corey, Mark Sendak, Michael Gao, Marshall Nichols, Mike Revoir, Armando Bedoya, Suresh Balu, and Cara O'Brien, "Characterizing Sepsis En- counters Across Community and Quartnerary Hospitals Within an Academic Health System," Abstract published at Hospital Medicine, March 24-27, 2019. National Harbor, MD. Abstract 188. See also: A.L. Lin, M.P. Sendak, A.D. Bedoya, M. Clement, J. Futoma, M. Nichols, M. Gao, K. Heller, C. O'Brien, "What Is Sepsis: Investigating the Heterogeneity of Patient Populations Captured by Different Sepsis Definitions," American Journal of Respiratory and Critical Care Medicine 2020, 201: A3299, doi:197:A3299.
  5. Steven J. Jackson, "Rethinking repair: breakdown, maintenance and repair in media and technology studies today," Media meets technology, MIT Press 68 (2013).
  6. Hamid Ekbia and Bonnie Nardi. "Social Inequality and HCI: The View from Political Economy," Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16), ACM (2016), http://doi.acm.org/10.1145/2858036.2858343; · DOI
  7. Lara Houston, Steven J. Jackson, Daniela K. Rosner, Syed Ishtiaque Ahmed, Meg Young, and Laewoo Kang, "Values in Repair," Proceedings of the 2016 CHI Confer- ence on Human Factors in Computing Systems (CHI '16), ACM, (2016): 1403-1414, https://doi.org/10.1145/2858036.2858470. · DOI
  8. Anselm Strauss, Shizuko Fagerhaugh, Barbara Suczek, and Carolyn Wiener, Social Organization of Medical Work (Chicago: University of Chicago Press, 1985).
  9. Susan Leigh Star and Anselm Strauss, "Layers of silence, arenas of voice: The ecol- ogy of visible and invisible work," Computer supported cooperative work (CSCW) 8, 1-2 (1999): 9-30.
  10. J. R. Elliott and R. A. Smith, "Race, Gender, and Workplace Pow- er," American Sociological Review 69, 3 (2004): 365-386, https://doi. org/10.1177/000312240406900303; · DOI
  11. Christine L. Williams, "The Glass Escalator: Hidden Advantages for Men in the 'Female' Professions," Social Problems 39, 3 (August 1992): 253-267, https://doi.org/10.2307/3096961; · DOI
  12. M. O. Mosley, "Beginning at the beginning: a history of the professionalization of Black nurses in America, 1908-1951," Journal of cultural diversity 2, 4 (1995): 101-9;
  13. John F. Dovidio, Kerry Kawakami, and Samuel L. Gaertner, "Implicit and explicit prejudice and interracial interaction," Journal of personality and social psychology 82, 1 (2002): 62; Kimberle Crenshaw. " Mapping the margins: Intersectionality, identity politics, and violence against women of color," Stan. L. Rev. 43 (1990): 1241.
  14. Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan, "Dissecting racial bias in an algorithm used to manage the health of populations," Science 366, 6464 (2019): 447-453;
  15. Milena A. Gianfrancesco, Suzanne Tamang, Jinoos Yazdany, and Gabriela Schmajuk, "Potential biases in machine learning algorithms using electronic health record data," JAMA internal medicine 178, 11 (2018): 1544-1547;
  16. Kun-Hsing Yu and Isaac S. Kohane, "Framing the challenges of artificial intelligence in medicine," BMJ quality & safety 28, 3 (2019): 238-241;
  17. Merlin Chowkwanyun and Adolph L. Reed Jr., "Racial health disparities and Covid-19- caution and context," New England Journal of Medicine (2020);
  18. Zinzi D. Bailey, Nancy Krieger, Madina Agénor, Jasmine Graves, Natalia Linos, and Mary T. Bassett, "Structural racism and health inequities in the USA: evidence and interventions," The Lancet 389, 10077 (2017): 1453-1463;
  19. Matthew Beane, "Shadow Learning: Building Robotic Surgical Skill When Approved Means Fail," Administrative Science Quarterly 64, 1 (2019): 87-123, https://doi. org/10.1177/0001839217751692. · DOI · Paper page
  20. Federico Cabitza, Raffaele Rasoini, and Gian Franco Gensini, "Unintended consequences of machine learning in medicine," Jama 318, 6 (2017): 517-518, doi:10.1001/jama.2017.7797. REFERENCES Abbott, Andrew. The system of professions: An essay on the expert division of labor. Chicago: University of Chicago Press, 1988. · DOI · Paper page
  21. Angus, Derek C., Christopher W. Seymour, Craig M. Coopersmith, Clifford Deutschman, Michael Klompas, Mitchell M. Levy, Greg S. Martin, Tiffany M. Osborn, Chanu Rhee, and R. Scott Watson. "A framework for the development and interpretation of different sepsis definitions and clinical criteria." Critical care medicine 44, no. 3 (2016): e113
  22. Anteby, Michel, and Curtis K. Chan. "A self-fulfilling cycle of coercive surveillance: Workers' invisibility practices and managerial justification." Organization Science 29, no. 2 (2018): 247-263
  23. Aronowitz, Robert. Risky medicine: our quest to cure fear and uncertainty. University of Chicago Press, 2015.
  24. Ash, Joan S. Marc Berg, Enrico Coiera. Some unintended consequences of information technology in health care: the nature of patient care information system-related errors. J Am Med Inform Assoc. 2004;11(2):104-112. doi:10.1197/jamia.M1471 · DOI · Paper page
  25. Barley, Stephen R. "Technology as an occasion for structuring: Evidence from observations of CT scanners and the social order of radiology departments." Administrative science quarterly (1986): 78-108. doi: 10.2307/2392767 · DOI · Paper page
  26. Beane, Matthew. "Shadow learning: Building robotic surgical skill when approved means fail." Administrative Science Quarterly 64, no. 1 (2019): 87-123.
  27. Bedoya, Armando D., Meredith E. Clement, Matthew Phelan, Rebecca C. Steorts, Cara O'Brien, and Benjamin A. Goldstein. "Minimal impact of implemented early warning score and best practice alert for patient deterioration." Critical care medicine 47, no. 1 (2019): 49.
  28. Bossen, Claus, Kathleen H. Pine, Federico Cabitza, Gunnar Ellingsen, and Enrico Maria Piras. "Data work in health care: An Introduction." Health Informatics Journal 25: 3 (2019): 465-474. doi.org/10.1177/1460458219864730. PMID: 31405353 · DOI
  29. Bowers, John. "The work to make a network work: studying CSCW in action." In Proceedings of the 1994 ACM conference on Computer supported cooperative work, pp. 287-298. 1994.
  30. Bowker, Geoffrey C. and Susan Leigh Star. Sorting things out: Classification and its consequences. Cambridge: MIT Press, 2000.
  31. Cabitza, Federico, Raffaele Rasoini, and Gian Franco Gensini. "Unintended consequences of machine learning in medicine." Jama 318, no. 6 (2017): 517-518. doi:10.1001/jama.2017.7797 · DOI · Paper page
  32. Cefkin, Melissa. "Numbers may speak louder than words, but is anyone listening? The rhythmscape and sales pipeline management." In Ethnographic Praxis in Industry Conference Proceedings, vol. 2007, no. 1, pp. 187-199. Oxford, UK: Blackwell Publishing Ltd, 2007.
  33. Chowkwanyun, Merlin, and Adolph L. Reed Jr. "Racial health disparities and Covid-19-caution and context." New England Journal of Medicine (2020).
  34. Citron, Danielle Keats, and Frank Pasquale. "The scored society: Due process for automated predictions." Wash. L. Rev. 89 (2014): 1.
  35. Cohen, Jonathan, Jean-Louis Vincent, Neill KJ Adhikari, Flavia R. Machado, Derek C. Angus, Thierry Calandra, Katia Jaton et al. "Sepsis: a roadmap for future research." The Lancet infectious diseases 15, no. 5 (2015): 581-614. Doi:10.1016/S1473-3099(15)70112-X De Swaan, Abram. "The reluctant imperialism of the medical profession." Social science & medicine 28, no. 11 (1989): 1165-1170. · DOI
  36. Downing, Norman Lance, Joshua Rolnick, Sarah F. Poole, Evan Hall, Alexander J. Wessels, Paul Heidenreich, and Lisa Shieh. "Electronic health record-based clinical decision support alert for severe sepsis: a randomised evaluation." BMJ quality & safety 28, no. 9 (2019): 762-768.
  37. Dyrda, Laura. 100 of the largest hospitals and health systems in America: 2019. Becker's Hospital Review. (2019).
  38. Elish, Madeleine Clare, and danah boyd. "Situating methods in the magic of Big Data and AI." Communication monographs 85, no. 1 (2018): 57-80.
  39. Elish, Madeleine Clare, and Tim Hwang. "Praise the Machine! Punish the Human! The contradictory history of accountability in automated aviation." SSRN (2015).
  40. Evans, Laura. "A Closer Look at Sepsis-Associated Mortality." JAMA network open 2, no. 2 (2019): e187565-e187565.
  41. Fleuren, Lucas M., Thomas LT Klausch, Charlotte L. Zwager, Linda J. Schoonmade, Tingjie Guo, Luca F. Roggeveen, Eleonora L. Swart et al. "Machine learning for the prediction of sepsis: a systematic review and meta-analysis of diagnostic test accuracy." Intensive care medicine (2020): 1-18.
  42. Freidson, Eliot. Profession of medicine: A study of the sociology of applied knowledge. University of Chicago Press, 1988.
  43. Futoma, Joseph, Sanjay Hariharan, and Katherine Heller. Learning to Detect Sepsis with a Multitask Gaussian Process RNN Classifier. Proceedings of the 34th International Conference on Machine Learning. Sydney, Australia, PMLR 70 (2017).
  44. Futoma, Joseph, Sanjay Hariharan, Katherine Heller, Mark Sendak, Nathan Brajer, Meredith Clement, Armando Bedoya, and Cara Obrien. An Improved Multi-Output Gaussian Process RNN with Real-Time Validation for Early Sepsis Detection. Proceedings of Machine Learning for Health care 68 (August 2017), 1-12.
  45. Gianfrancesco, Milena A., Suzanne Tamang, Jinoos Yazdany, and Gabriela Schmajuk. "Potential biases in machine learning algorithms using electronic health record data." JAMA internal medicine 178, no. 11 (2018): 1544-1547.
  46. Glomb, Theresa M., John D. Kammeyer-Mueller, and Maria Rotundo. "Emotional labor demands and compensating wage differentials." Journal of applied psychology 89, no. 4 (2004): 700. doi:10.1037/0021-9010.89.4.700. PMID 15327355. · DOI
  47. Hochschild, Arlie Russell. The managed heart: Commercialization of human feeling. University of California Press, 2012.
  48. Houston, Lara. Steven J. Jackson, Daniela K. Rosner, Syed Ishtiaque Ahmed, Meg Young, and Laewoo Kang, "Values in Repair," Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16), ACM, (2016): 1403-1414, https://doi. org/10.1145/2858036.2858470. "Improving the prevention, diagnosis and clinical management of sepsis." World Health Organization Report by the Secretariat. (2017). · DOI
  49. Islam, Md Mohaimenul, Tahmina Nasrin, Bruno Andreas Walther, Chieh-Chen Wu, Hsuan-Chia Yang, and Yu-Chuan Li. "Prediction of sepsis patients using machine learning approach: a meta-analysis." Computer methods and programs in biomedicine 170 (2019): 1-9.
  50. Jackson, Steven J. "Rethinking repair: breakdown, maintenance and repair in media and technology studies today." Media meets technology, MIT Press 68 (2013).
  51. Kitchin, Rob. "Big data, new epistemologies and paradigm shifts." Big data & society 1, no. 1 (2014): 2053951714528481.
  52. Leonelli, Sabina. "What difference does quantity make? On the epistemology of Big Data in biology." Big data & society 1, no. 1 (2014): 2053951714534395.
  53. Lin, A.L. Mark Sendak, Armando D. Bedoya, Meredith E. Clement, Joseph Futoma, M. Nichols, M. Gao, Katherine A. Heller, Cara L. O'Brien. What Is Sepsis: Investigating the Heterogeneity of Patient Populations Captured by Different Sepsis Definitions. American Journal of Respiratory and Critical Care Medicine (2020); 201: A3299. doi:197:A3299.
  54. Lin, A.L., Mark Sendak, Armando D. Bedoya, Meredith E. Clement, Nathan Brajer, Joseph Futoma, Hayden B. Bosworth, Katherine A. Heller, and Cara L. O'Brien. "Evaluating sepsis definitions for clinical decision support against a definition for epidemiological disease surveillance." bioRxiv (2019): 648907. doi: https://doi.org/10.1101/648907 · DOI
  55. Maiers, Claire. "Analytics in action: users and predictive data in the neonatal intensive care unit." Information, Communication & Society 20, no. 6 (2017): 915-929.
  56. Mazmanian, Melissa, Ingrid Erickson, and Ellie Harmon. "Circumscribed time and porous time: Logics as a way of studying temporality." In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing, pp. 1453-1464. 2015.
  57. Mitchell, Tom M. Machine Learning. (New York: McGraw Hill, 1997): p. 2. ISBN 978-0-07-042807-2.
  58. Mittelstadt, Brent Daniel, and Luciano Floridi. "The ethics of big data: current and foreseeable issues in biomedical contexts." Science and engineering ethics 22, no. 2 (2016): 303-341.
  59. Obermeyer, Ziad, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. "Dissecting racial bias in an algorithm used to manage the health of populations." Science 366, no. 6464 (2019): 447-453.
  60. Orlikowski, Wanda J., and JoAnne Yates. "It's about time: Temporal structuring in organizations." Organization science 13, no. 6 (2002): 684-700.
  61. Pine, Kathleen H., and Max Liboiron. "The politics of measurement and action." In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pp. 3147-3156. 2015.
  62. Pine, Kathleen H., and Melissa Mazmanian. "Artful and contorted coordinating: The ramifications of imposing formal logics of task jurisdiction on situated practice." Academy of Management Journal 60, no. 2 (2017): 720-742.
  63. Pine, Kathleen H., and Melissa Mazmanian. "Institutional logics of the EMR and the problem of 'perfect' but inaccurate accounts." In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing, 2014 pp. 283-294.
  64. Procter, Rob, Joe Wherton, Trish Greenhalgh, Paul Sugarhood, Mark Rouncefield, and Sue Hinder. "Telecare call centre work and ageing in place." Computer Supported Cooperative Work (CSCW) 25, no. 1 (2016): 79-105.
  65. Rhee, Chanu, Kathleen Chiotos, Sara E. Cosgrove, Emily L. Heil, Sameer S. Kadri, Andre C. Kalil, David N. Gilbert et al. "Infectious Diseases Society of America Position Paper: Recommended Revisions to the National Severe Sepsis and Septic Shock Early Management Bundle (SEP-1) Sepsis Quality Measure." Clinical Infectious Diseases 2020. doi:10.1093/cid/ciaa059 · DOI · Paper page
  66. Rhee, Chanu, Raymund Dantes, Lauren Epstein, David J. Murphy, Christopher W. Seymour, Theodore J. Iwashyna, Sameer S. Kadri et al. "Incidence and trends of sepsis in US hospitals using clinical vs claims data, 2009-2014." Jama 318, no. 13 (2017): 1241-1249.
  67. Rhee, Chanu, Michael Filbin, Anthony F. Massaro, Amy Bulger, Donna McEachern, Kathleen A. Tobin, Barrett Kitch et al. "Compliance with the national SEP-1 quality measure and association with sepsis outcomes: a multicenter retrospective cohort study." Critical care medicine 46, no. 10 (2018): 1585.
  68. Sendak, Mark, Madeleine Clare Elish, Michael Gao, Joseph Futoma, William Ratliff, Marshall Nichols, Armando Bedoya, Suresh Balu, and Cara O'Brien. "The human body is a black box" supporting clinical decision-making with deep learning." In Proceedings of the 2020 ACM Conference on Fairness, Accountability, and Transparency, pp. 99-109. (2020).
  69. Seymour, Christopher W., Foster Gesten, Hallie C. Prescott, Marcus E. Friedrich, Theodore J. Iwashyna, Gary S. Phillips, Stanley Lemeshow, Tiffany Osborn, Kathleen M. Terry, and Mitchell M. Levy. "Time to treatment and mortality during mandated emergency care for sepsis." New England Journal of Medicine 376, no. 23 (2017): 2235-2244.
  70. Star, Susan Leigh, and Anselm Strauss. "Layers of silence, arenas of voice: The ecology of visible and invisible work." Computer supported cooperative work (CSCW) 8, no. 1-2 (1999): 9-30.
  71. Strauss, Anselm and Juliet Corbin. "Grounded theory methodology." Handbook of qualitative research 17, no. 1 (1994): 273-285.
  72. Strauss, Anselm, Shizuko Fagerhaugh, Barbara Suczek, and Carolyn Wiener. Social Organization of Medical Work. Chicago: University of Chicago Press, 1985.
  73. Timmermans, Stefan, and Marc Berg. The gold standard: the challenge of evidence-based medicine. Temple University Press, 2010.
  74. Yu, Kun-Hsing, and Isaac S. Kohane. "Framing the challenges of artificial intelligence in medicine." BMJ quality & safety 28, no. 3 (2019): 238-241. Data & Society is an independent nonprofit research institute that advances new frames for understanding the implications of data-centric and automated technology. We conduct research and build the field of actors to ensure that knowledge guides debate, decision-making, and technical choices.

Cited by

Sorted by citations

Authors
Elish, M. C., & Watkins, E. A.
ORCID
Publisher
Data & Society Research Institute
Published
2020-09-30
DOI
10.69985/vcvu5761
OpenAlex
W7123968246
Cited by
5
References
0
Citations by year
2026: 4 · 2025: 1
Type
report
Language
en
Open access
Licence
cc-by-nc-sa
Locations
Issue Lab (Candid) (submittedVersion, cc-by-nc-sa)
Funders
John D. and Catherine T. MacArthur Foundation
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Technology and Human Factors in Education and Health
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
Keywords
artificial intelligence integration, healthcare innovation, clinical decision support, AI adoption in medicine

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review