Batman and Robin in Healthcare Knowledge Work: Human-AI Collaboration by Clinical Documentation Integrity Specialists
Open accessThe DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Abstract
This article describes the successful collaboration “in the wild” between Clinical Documentation Integrity Specialists (CDIS) and an Artificial Intelligence (AI)-embedded software to conduct knowledge work. CDIS review patient charts in near real-time to improve clinicians’ documentation, with the goal to make medical documentation more accurate, consistent and complete. CDIS collaborate with an AI-embedded “Computer Assisted Coding” (CAC) system that scans records from the Electronic Healthcare Record and auto-suggests codes based on natural language processing. CDIS find the CAC's suggestions are often inaccurate—often humorously so. Still, they find the CAC to be a useful helper, like Robin is to Batman. This human-AI collaboration is contingent on several factors: the flexible integration of the AI into the workflow similar to the notion of unremarkable AI; supporting the CDIS’ sensemaking; the CDIS’ knowledge about the CAC being predictably unreliable, an experience by the CDIS of the AI's value; humans remaining in control; and ability to experiment with the AI, which spurs reflection and learning for these knowledge workers.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Artificial intelligence and the future of work: Human-AI symbiosis in organizational decision making
Jarrahi, M. H. (2018) · Business Horizons · doi:10.1016/j.bushor.2018.03.007 · Cited by 2,772 · cited 7×
“…to understand how AI-embedded sociotechnical systems should be designed, imlemented, and managed to augment human work and extend human cognition [Jarrahi 2018 ]. owever, human-AI collaboration is still under-researched, and empirical studies of how humans ollaborate with AI “in the wild” are…”
InfluentialGuidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439
The practical implementation of artificial intelligence technologies in medicine
He, J., Baxter, S. L., Xu, J., Xu, J., Zhou, X., & Zhang, K. (2019) · Nature Medicine · doi:10.1038/s41591-018-0307-0 · Cited by 2,246
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445717 · Cited by 757
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. S. (2020) · arXiv · doi:10.48550/arxiv.2006.14779 · Cited by 754
Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
(2023) · doi:10.1145/3544548 · Cited by 735
Cai, C. J., Winter, S., Steiner, D., Wilcox, L., & Terry, M. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359206 · Cited by 594
Artificial Intelligence as Augmenting Automation: Implications for Employment
Tschang, F. T., & Almirall, E. (2021) · Academy of Management Perspectives · doi:10.5465/amp.2019.0062 · Cited by 409
Human-AI Collaboration in Data Science: Exploring Data Scientists' Perceptions of Automated AI
Wang, D., Weisz, J. D., Muller, M., Ram, P., Geyer, W., Dugan, C., Tausczik, Y., Samulowitz, H., & Gray, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359313 · Cited by 342
The Future of Human-AI Collaboration: A Taxonomy of Design Knowledge for Hybrid Intelligence Systems
Dellermann, D., Calma, A., Lipusch, N., Weber, T., Weigel, S., & Ebel, P. (2019) · Proceedings of the Annual Hawaii International Conference on System Sciences · doi:10.24251/hicss.2019.034 · Cited by 279
Yang, Q., Steinfeld, A., & Zimmerman, J. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300468 · Cited by 247
Working Algorithms: Software Automation and the Future of Work
Shestakofsky, B. (2017) · Work and Occupations · doi:10.1177/0730888417726119 · Cited by 186 · cited 2×
“…longer period, which has allowed for the software, the AI, and CDIS’ work practices to oevolve as they interact, and thus the infrastructural and socio-technical re-arrangements necssary when implementing AI have already taken place [Shestakofsky 2017 ; Sendak et al. 2020 ; lish and Watkins 2020 ].”
You Complete Me: Human-AI Teams and Complementary Expertise
Zhang, Q., Lee, M. L., & Carter, S. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517791 · Cited by 85
AI in Context: The Labor of Integrating New Technologies
Mateescu, A., & Elish, M. C. (2019) · doi:10.69985/xtet4824 · Cited by 49
Clement, J., & Maldonado, A. Q. (2021) · Frontiers in Immunology · doi:10.3389/fimmu.2021.694222 · Cited by 48 · cited 1×
“…decision support systems that use AI are designed to draw n vast quantities of data to produce insights that are employed for decision-making in a variety f contexts, pointing to the best diagnosis or course of action given a multitude of data points Clement and Maldonado 2021 ; Kim 2018 ].”
Cited by
Yildirim, N., Richardson, H., Wetscherek, M. T., Bajwa, J., Jacob, J., Pinnock, M. A., Harris, S., Coelho De Castro, D., Bannur, S., Hyland, S., et al. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642013 · Cited by 82
Effect of Explanation Conceptualisations on Trust in AI-assisted Credibility Assessment
Pareek, S., van Berkel, N., Velloso, E., & Goncalves, J. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3686922 · Cited by 42
Fraile Navarro, D., Kocaballi, A. B., Dras, M., & Berkovsky, S. (2023) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3569893 · Cited by 28
“An example of proposed approaches using NLP that led to implementation is described by Pine and Bossen [98] where clinical documentation integrity specialists use NLP in combination with document statistics for a computer-aided coding of the EHR.”
MethodPanchanadikar, R., & Freeman, G. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3677082 · Cited by 43
Panchanadikar, R., Freeman, G., Li, L., Schulenberg, K., & Hu, Y. (2024) · Extended Abstracts of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613905.3650845 · Cited by 17
Metadata
- Authors
- Bossen, C., & Pine, K. H.
- Venue
- ACM Transactions on Computer-Human Interaction
- Published
- 2023-03-17
- Volume
- 30
- Issue
- 2
- Pages
- 1-29
- ISSN
- 1073-0516 (ISSN-L)
- 1557-7325
- OpenAlex
- W4307639462
- Cited by
- 63
- References
- 15
- FWCI
- 15.41
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.