Collaboration, not Confrontation: Understanding General Practitioners’ Attitudes Towards Natural Language and Text Automation in Clinical Practice
The DOI matches its Crossref record; 127 of the 127 references it cites were found in a registry or an index.
Abstract
General Practitioners are among the primary users and curators of textual electronic health records, highlighting the need for technologies supporting record access and administration. Recent advancements in natural language processing facilitate the development of clinical systems, automating some time-consuming record-keeping tasks. However, it remains unclear what automation tasks would benefit clinicians most, what features such automation should exhibit, and how clinicians will interact with the automation. We conducted semi-structured interviews with General Practitioners uncovering their views and attitudes toward text automation. The main emerging theme was doctor-AI collaboration, addressing a reciprocal clinician-technology relationship that does not threaten to substitute clinicians, but rather establishes a constructive synergistic relationship. Other themes included: (i) desired features for clinical text automation; (ii) concerns around clinical text automation; and (iii) the consultation of the future. Our findings will inform the design of future natural language processing systems, to be implemented in general practice.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“which is another common obstacle to accessing and using some of these tools [40,45,65,104]”
From Talk2Care: An LLM-based Voice Assistant for Communication between Healthcare Providers and Older Adults (2024) · cites this paper as [40]
In the citing paper
“older adults generally have lower technology proficiency compared to younger generations”
From Talk2Care: An LLM-based Voice Assistant for Communication between Healthcare Providers and Older Adults (2024) · cites this paper as [40]
In the citing paper
“lack of digital literacy is a widely known issue [40,91,104]”
From Talk2Care: An LLM-based Voice Assistant for Communication between Healthcare Providers and Older Adults (2024) · cites this paper as [40]
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 2×
“In particular, the discovery of new deep learning language algorithms including statistical language models based on continuous vector representations of words [87] with subsequent development of transformer models, such as BERT [35], GPT-3 [14], and their specific derivatives, e.g., trained on clinical text corpora […”
BackgroundHighly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764 · cited 1×
“Progress in areas such as computer vision [63] and developments that spread beyond what humans are capable of, such as protein folding [61] or real-time weather prediction [106], demonstrate that AI systems performing cognitively complex tasks may become a reality soon.”
BackgroundLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 2×
“…models based on continuous vector representations of words [87] with subsequent development of transformer models, such as BERT [35], GPT-3 [14], and their specific derivatives, e.g., trained on clinical text corpora [2], have allowed their application for complex NLP tasks including named…”
BackgroundHow Many Interviews Are Enough?: An Experiment with Data Saturation and Variability
Guest, G., Bunce, A., & Johnson, L. (2006) · Field Methods · doi:10.1177/1525822x05279903 · Cited by 20,808
High-performance medicine: the convergence of human and artificial intelligence
Topol, E. J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0300-7 · Cited by 10,450 · cited 3×
“Applications of AI in medicine recently have focused on high-stake decisions, such as ruling out diseases and providing diagnosis [124] or question-answering systems and conversational agents [94].”
BackgroundTransformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294 · cited 1×
“Although there is a certain degree of limitation of black box AI systems (Adadi & Berrada, 2018), more recent approaches and developments shed some light on the inner workings of the algorithms (Adadi & Berrada, 2018).”
BackgroundClinical Prediction Models: A Practical Approach to Development, Validation, and Updating
Steyerberg, E. W. (2019) · Statistics for Biology and Health · doi:10.1007/978-3-030-16399-0 · Cited by 3,281
What is AI Literacy? Competencies and Design Considerations
Long, D., & Magerko, B. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376727 · Cited by 3,241 · cited 1×
“Previous studies have highlighted the importance of addressing AI literacy [73] and this is reflected by our approach of dedicating the initial part of our interviews to creating a shared understanding of the technology and its capabilities.”
BackgroundChaudhry, B., Wang, J., Wu, S., Maglione, M., Mojica, W., Roth, E., Morton, S. C., & Shekelle, P. G. (2006) · Annals of Internal Medicine · doi:10.7326/0003-4819-144-10-200605160-00125 · Cited by 3,159 · cited 1×
“The adoption of automated technologies in various aspects of the clinical pipeline is becoming more and more common over the years, especially concerning health information technologies where it can increase several quality indicators such as adherence to guidelines [22], increase surveillance and monitoring [22], as…”
BackgroundAn overview of clinical decision support systems: Benefits, risks, and strategies for success
Sutton, R. T., Pincock, D., Baumgart, D. C., Sadowski, D. C., Fedorak, R. N., & Kroeker, K. I. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0221-y · Cited by 3,019 · cited 2×
“Ranging from decision support systems ( DSS ) [118] to clinical imaging [43] and clinical prediction models [117], it spans a wide range of optimizations surrounding the clinical pipeline [112].”
BackgroundA simple method to assess and report thematic saturation in qualitative research
Guest, G., Namey, E., & Chen, M. (2020) · PLOS ONE · doi:10.1371/journal.pone.0232076 · Cited by 2,929 · cited 1×
“To calculate saturation, we performed a retrospective analysis to assess data saturation following the methodology of Guest et al. [47].”
MethodA survey of named entity recognition and classification
Nadeau, D., & Sekine, S. (2007) · Lingvisticae Investigationes · doi:10.1075/li.30.1.03nad · Cited by 2,507 · cited 1×
“Additional clarification and explanations were provided on specific NLP tasks that were discussed: information extraction [5], named entity recognition [91] and summarization [81].”
BackgroundTerry, G., Hayfield, N., Clarke, V., & Braun, V. (2017) · The SAGE Handbook of Qualitative Research in Psychology · doi:10.4135/9781526405555.n2 · Cited by 2,448
Explainability for artificial intelligence in healthcare: a multidisciplinary perspective
the Precise4Q consortium, Amann, J., Blasimme, A., Vayena, E., Frey, D., & Madai, V. I. (2020) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-020-01332-6 · Cited by 2,203 · cited 1×
“Aspects like explainability [3], automation bias [76], fairness [20], and other ethical challenges need to be addressed [129] in parallel with improving the precision and accuracy of AI.”
BackgroundAlsentzer, E., Murphy, J., Boag, W., Weng, W.-H., Jindi, D., Naumann, T., & McDermott, M. (2019) · Proceedings of the 2nd Clinical Natural Language Processing Workshop · doi:10.18653/v1/w19-1909 · Cited by 1,736 · cited 1×
“, trained on clinical text corpora [2], have allowed their application for complex NLP tasks including named entity recognition [49], summarization [110], or question answering [51], also in the medical domain and the health industry.”
BackgroundAttia, Z. I., Noseworthy, P. A., Lopez-Jimenez, F., Asirvatham, S. J., Deshmukh, A. J., Gersh, B. J., Carter, R. E., Yao, X., Rabinstein, A. A., Erickson, B. J., et al. (2019) · The Lancet · doi:10.1016/s0140-6736(19)31721-0 · Cited by 1,613 · cited 2×
“challenges and opportunities for automation [70], especially in the domain of NLP, as opposed to other specialities that rely more on imaging or other diagnostic technologies such as ECGs [7] or EEGs [97].”
Background2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text
Uzuner, Ö., South, B. R., Shen, S., & DuVall, S. L. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000203 · Cited by 1,252
Machine Learning and Prediction in Medicine — Beyond the Peak of Inflated Expectations
Chen, J. H., & Asch, S. M. (2017) · New England Journal of Medicine · doi:10.1056/nejmp1702071 · Cited by 1,171 · cited 1×
“Several authors have also risen questions surrounding the use of machine learning in clinical prediction models and the challenges this entails [23], especially given the potential for biases and the lack of transparency [92].”
BackgroundSkilful precipitation nowcasting using deep generative models of radar
Ravuri, S., Lenc, K., Willson, M., Kangin, D., Lam, R., Mirowski, P., Fitzsimons, M., Athanassiadou, M., Kashem, S., Madge, S., et al. (2021) · Nature · doi:10.1038/s41586-021-03854-z · Cited by 1,024
(2019) · Cited by 986
Arndt, B. G., Beasley, J. W., Watkinson, M. D., Temte, J. L., Tuan, W.-J., Sinsky, C. A., & Gilchrist, V. J. (2017) · The Annals of Family Medicine · doi:10.1370/afm.2121 · Cited by 972 · cited 1×
“Previous studies have confirmed the growing time clinicians were tethered to the EHR [6] spending more time with the EHR than with the patients [95].”
BackgroundMachine learning in medicine: Addressing ethical challenges
Vayena, E., Blasimme, A., & Cohen, I. G. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002689 · Cited by 912 · cited 1×
“Aspects like explainability [3], automation bias [76], fairness [20], and other ethical challenges need to be addressed [129] in parallel with improving the precision and accuracy of AI.”
BackgroundRoberts, K., Dowell, A., & Nie, J.-B. (2019) · BMC Medical Research Methodology · doi:10.1186/s12874-019-0707-y · Cited by 867 · cited 1×
“Previous literature on qualitative research has shown that for investigating phenomena surrounding experience, a sample size of six participants is typically sufficient [109].”
BackgroundWhat can natural language processing do for clinical decision support?
Demner-Fushman, D., Chapman, W. W., & McDonald, C. J. (2009) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2009.08.007 · Cited by 688 · cited 1×
“Studies have evaluated the use of various automated systems such as Decision Support Systems to guide prescription decisions [57] combined with earlier NLP developments integrated into the EHR [33].”
MethodMiddleton, B., Bloomrosen, M., Dente, M. A., Hashmat, B., Koppel, R., Overhage, J. M., Payne, T. H., Rosenbloom, S. T., Weaver, C., & Zhang, J. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2012-001458 · Cited by 618 · cited 1×
“The GP interviews showed a growing desire to modernize the EHR interface and interaction with health information and EHR software, which have been highlighted previously [4,86].”
BackgroundCai, C. J., Winter, S., Steiner, D., Wilcox, L., & Terry, M. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359206 · Cited by 594 · cited 2×
“Equally, it has also shown the importance of addressing clinicians’ informational needs regarding the overall model objectives and attributes [18].”
BackgroundResultDeep learning in clinical natural language processing: a methodical review
Wu, S., Roberts, K., Datta, S., Du, J., Ji, Z., Si, Y., Soni, S., Wang, Q., Wei, Q., Xiang, Y., et al. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz200 · Cited by 491 · cited 1×
“The potential uses of NLP in medicine have been explored in previous reviews [138] as well as in the form of computational challenges addressing specific tasks [125].”
BackgroundAutomation bias and verification complexity: a systematic review
Lyell, D., & Coiera, E. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw105 · Cited by 451 · cited 1×
“Previous work has described this type of error as automation bias [75].”
BackgroundVirtual and augmented reality applications in medicine: analysis of the scientific literature
Yeung, A. W. K., Tosevska, A., Klager, E., Eibensteiner, F., Laxar, D., Stoyanov, J., Glisic, M., Zeiner, S., Kulnik, S. T., Crutzen, R., et al. (2021) · Journal of Medical Internet Research · doi:10.2196/25499 · Cited by 428
Coiera, E. (2015) · CRC Press eBooks · doi:10.1201/b13617 · Cited by 415 · cited 1×
“One of the potential ways of addressing these issues relies on the use of automation and information technologies in Health Care which has a long and outstanding history [26,36,123].”
BackgroundAndaur Navarro, C. L., Damen, J. A. A., Takada, T., Nijman, S. W. J., Dhiman, P., Ma, J., Collins, G. S., Bajpai, R., Riley, R. D., Moons, K. G. M., et al. (2021) · BMJ · doi:10.1136/bmj.n2281 · Cited by 415 · cited 1×
“Several authors have also risen questions surrounding the use of machine learning in clinical prediction models and the challenges this entails [23], especially given the potential for biases and the lack of transparency [92].”
BackgroundMelnick, E. R., Dyrbye, L. N., Sinsky, C. A., Trockel, M., West, C. P., Nedelec, L., Tutty, M. A., & Shanafelt, T. (2020) · Mayo Clinic Proceedings · doi:10.1016/j.mayocp.2019.09.024 · Cited by 413 · cited 1×
“Previous research has shown that the perceived usability of EHRs is an important factor influencing clinician burnout [85].”
BackgroundThieme, A., Belgrave, D., & Doherty, G. (2020) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3398069 · Cited by 406
Human-Centered Tools for Coping with Imperfect Algorithms During Medical Decision-Making
Cai, C. J., Reif, E., Hegde, N., Hipp, J., Kim, B., Smilkov, D., Wattenberg, M., Viegas, F., Corrado, G. S., Stumpe, M. C., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300234 · Cited by 405 · cited 2×
“[17] highlighted the concept of human-AI collaboration, but equally stressed the need for upfront information and the strength and limitations of the AI models to be used, similar to our findings regarding clinicians’ desire for a transparent AI.”
BackgroundResultPatient Protection and Affordable Care Act of 2010
(2018) · The SAGE Encyclopedia of Surveillance, Security, and Privacy · doi:10.4135/9781483359922.n327 · Cited by 400
Dearnley, C. (2005) · Nurse Researcher · doi:10.7748/nr2005.07.13.1.19.c5997 · Cited by 382 · cited 1×
“Semi-structured interviews offer specific advantages and have been used extensively in qualitative research, specifically when interviewing health professionals [31].”
BackgroundThe Barriers to Electronic Medical Record Systems and How to Overcome Them
McDonald, C. J. (1997) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.1997.0040213 · Cited by 378
Blease, C., Kaptchuk, T. J., Bernstein, M. H., Mandl, K. D., Halamka, J. D., & DesRoches, C. M. (2019) · Journal of Medical Internet Research · doi:10.2196/12802 · Cited by 346 · cited 4×
“…factors, targeting primary care and the needs of general practitioners could inform the design needs and functionalities of text processing automation, in one of the areas of medicine with a stronger potential for adoption of AI, especially for improving time-consuming and low-value tasks [9].”
BackgroundResultInfluentialDoes Japan really have robot mania? Comparing attitudes by implicit and explicit measures
MacDorman, K. F., Vasudevan, S. K., & Ho, C.-C. (2009) · AI & SOCIETY · doi:10.1007/s00146-008-0181-2 · Cited by 322 · cited 1×
“However, past literature has covered more often the challenges of automation related to the use of mechanical machinery or robots [77] or concerning the use of traditional computational approaches and their interaction with human agents, for instance in finance [74].”
BackgroundChapman, W. W., Nadkarni, P. M., Hirschman, L., D'Avolio, L. W., Savova, G. K., & Uzuner, O. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000465 · Cited by 296 · cited 1×
“Beyond technical limitations encountered in applying NLP methods to clinical text [21], there are additional challenges and questions around the most appropriate use cases for assisted automated technologies in real-world settings [126], especially in scenarios where high safety is required [128].”
MethodWaitzkin, H. (1989) · Journal of Health and Social Behavior · doi:10.2307/2137015 · Cited by 264 · cited 1×
“This medical-record-keeping structure coexists and influences the traditional clinical consultation format described by Waitzkin [131].”
BackgroundKim, M. O., Coiera, E., & Magrabi, F. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw154 · Cited by 263 · cited 1×
“Conversely, there have been numerous examples of poor implementation of technologies, and in particular, EHRs where it had a disruptive effect on clinical workflow, delaying care delivery [64] and increasing the time that clinicians spent in silence during the consultation with patients [42] as well as on decreasing s…”
BackgroundMedical records, medical education, and patient care;: The problem-oriented record as a basic tool
Weed, L. L. (1970) · Cited by 259
Ten Ways Artificial Intelligence Will Transform Primary Care
Lin, S. Y., Mahoney, M. R., & Sinsky, C. A. (2019) · Journal of General Internal Medicine · doi:10.1007/s11606-019-05035-1 · Cited by 248
Jacobs, M., Pradier, M. F., McCoy, T. H., Perlis, R. H., Doshi-Velez, F., & Gajos, K. Z. (2021) · Translational Psychiatry · doi:10.1038/s41398-021-01224-x · Cited by 247 · cited 1×
“How these algorithms issue their recommendations and how this information is displayed to the user may have a strong effect on their success [58].”
BackgroundThe influence of electronic health record use on physician burnout: cross-sectional survey
Tajirian, T., Stergiopoulos, V., Strudwick, G., Sequeira, L., Sanches, M., Kemp, J., Ramamoorthi, K., Zhang, T., & Jankowicz, D. (2020) · Journal of Medical Internet Research · doi:10.2196/19274 · Cited by 241 · cited 2×
“Concerning the specific use of automation technologies in the EHR, a few studies have addressed the attitudes of clinicians towards those, and in particular, showing the influence of the systems lacking optimization [59] as well as its influence on clinicians’ burnout [119,130].”
BackgroundOverhage, J. M., & McCallie, D. (2020) · Annals of Internal Medicine · doi:10.7326/m18-3684 · Cited by 238
de Bruijn, B., Cherry, C., Kiritchenko, S., Martin, J., & Zhu, X. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000150 · Cited by 235
History and future of human-automation interaction
Janssen, C. P., Donker, S. F., Brumby, D. P., & Kun, A. L. (2019) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2019.05.006 · Cited by 221 · cited 1×
“Research into human-automation interaction [60] and specifically AI has gained more attention recently [25].”
BackgroundEHRs: The Challenge of Making Electronic Data Usable and Interoperable.
Reisman, M. (2017) · PubMed · Cited by 216
Wang, D., Wang, L., Zhang, Z., Wang, D., Zhu, H., Gao, Y., Fan, X., & Tian, F. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445432 · Cited by 209 · cited 3×
“[132] explored the subject of Human AI collaboration in relation to its HCI components in medicine, and in particular in the case of rural GPs who, despite some concerns, expressed a generally positive attitude toward AI decision support systems [133].”
BackgroundImpact of the rise of artificial intelligence in radiology: What do radiologists think?
Waymel, Q., Badr, S., Demondion, X., Cotten, A., & Jacques, T. (2019) · Diagnostic and Interventional Imaging · doi:10.1016/j.diii.2019.03.015 · Cited by 191 · cited 1×
“Other medical areas show conflicting evidence, such as radiology, where there is a discrepancy between the future expectations of the technology and the confidence of clinicians in AI-based results [62], although other studies have reported a more positive view [134].”
ResultPrivacy Risks of General-Purpose Language Models
Pan, X., Zhang, M., Ji, S., & Yang, M. (2020) · 2020 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp40000.2020.00095 · Cited by 188 · cited 1×
“It has been recently explored in NLP, where concerns about the widespread use and potential misuse of pre-trained language models have been raised [8,84,96].”
BackgroundMoutinho, L., & Smith, A. (2000) · International Journal of Bank Marketing · doi:10.1108/02652320010339699 · Cited by 182 · cited 1×
“A wealth of literature has explored the attitudes toward different forms of automation [56,108] in diverse, highly specialized and high-risk environments such as aviation [55] and banking [89].”
BackgroundDeng, L. (2018) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2762725 · Cited by 180 · cited 1×
“Since the advent of the so-called third rise of AI [34], there has been an explosion of deep learning algorithms [44] capable of performing tasks that until recently, were believed to only be doable by humans.”
BackgroundJacobs, M., He, J., F. Pradier, M., Lam, B., Ahn, A. C., McCoy, T. H., Perlis, R. H., Doshi-Velez, F., & Gajos, K. Z. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445385 · Cited by 177 · cited 1×
“Studies have evaluated the use of various automated systems such as Decision Support Systems to guide prescription decisions [57] combined with earlier NLP developments integrated into the EHR [33].”
MethodEvans, C., & Lewis, J. (2018) · SAGE Publications Ltd eBooks · doi:10.4135/9781526439284 · Cited by 174 · cited 1×
“Inductive thematic analysis is commonly used for synthesising and analysing ethnographic data [12], including interviews [38].”
MethodIntroduction to information extraction
Appelt, D. E. (1999) · AI Communications · doi:10.5555/1216155.1216161 · Cited by 168
Automation fears: Drivers and solutions
Ivanov, S. H., Kuyumdzhiev, M., & Webster, C. (2020) · Technology in Society · doi:10.1016/j.techsoc.2020.101431 · Cited by 164
Challenges of developing a digital scribe to reduce clinical documentation burden
Quiroz, J. C., Laranjo, L., Kocaballi, A. B., Berkovsky, S., Rezazadegan, D., & Coiera, E. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0190-1 · Cited by 162 · cited 1×
“Preliminary work has been carried out, in areas such as generating an automated textual summary of clinical encounters [27], practical challenges faced by its development and translation [105], as well as clinicians’ [66] and patients’ [90] perceptions of such a system.”
BackgroundSelf-citationGeneral Practitioners’ Attitudes Toward Artificial Intelligence–Enabled Systems: Interview Study
Buck, C., Doctor, E., Hennrich, J., Jöhnk, J., & Eymann, T. (2022) · Journal of Medical Internet Research · doi:10.2196/28916 · Cited by 147
Technological Inertia in Economic History
Mokyr, J. (1992) · The Journal of Economic History · doi:10.1017/s0022050700010767 · Cited by 142 · cited 1×
“This is particularly relevant in the case of proprietary record software and interoperability with legacy systems, especially if stakeholders do not perceive the need for improving them, so technological inertia [88] and market consolidation of the EHR industry [102] may hamper progress.”
BackgroundScale, Causes, and Implications of the Primary Care Nursing Shortage
MacLean, L., Hassmiller, S. B., Shaffer, F. A., Rohrbaugh, K., Collier, T., & Fairman, J. A. (2014) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-032013-182508 · Cited by 135 · cited 1×
“Moreover, in the specific setting of Primary Care, it is unrealistic to consider that a medical scribe workforce could be easily implemented to accompany every primary care, physician, and consultation while most health care systems struggle to retain their workforce [78].”
BackgroundReading patterns and usability in visualizations of electronic documents
Hornbæk, K., & Frøkjær, E. (2003) · ACM Transactions on Computer-Human Interaction · doi:10.1145/772047.772050 · Cited by 125 · cited 1×
“Previous work also has explored the design and implementation challenges of the EHR [115] as well as elements surrounding electronic document visualization [52].”
BackgroundReimagining Clinical Documentation With Artificial Intelligence
Lin, S., Shanafelt, T. D., & Asch, S. M. (2018) · Mayo Clinic Proceedings · doi:10.1016/j.mayocp.2018.02.016 · Cited by 117 · cited 1×
“Although doctors have previously made calls to reimagine EHR and design it with AI in mind [71] as well as considering certain elements of the EHR design and content structure such as the problem’s list [141], still very limited literature exists on the design and key features that an NLP text system needs to provide;…”
BackgroundMedical Information Extraction in the Age of Deep Learning
Hahn, U., & Oleynik, M. (2020) · Yearbook of Medical Informatics · doi:10.1055/s-0040-1702001 · Cited by 115 · cited 1×
“However, the advantages that can be obtained from information extraction and text summarization could be useful across the health system [48,114] and could be used by a wide range of clinicians, specialties, and health settings.”
BackgroundArtificial intelligence in pulmonary medicine: computer vision, predictive model and COVID-19
Khemasuwan, D., Sorensen, J. S., & Colt, H. G. (2020) · European Respiratory Review · doi:10.1183/16000617.0181-2020 · Cited by 113 · cited 1×
“Progress in areas such as computer vision [63] and developments that spread beyond what humans are capable of, such as protein folding [61] or real-time weather prediction [106], demonstrate that AI systems performing cognitively complex tasks may become a reality soon.”
BackgroundGellert, G. A., Ramirez, R., & Webster, S. L. (2014) · JAMA · doi:10.1001/jama.2014.17128 · Cited by 112 · cited 12×
“This gradual implementation of technology has shown positive results previously as in the case of EHRs [113] where it helped overcome initial concerns that patients, clinicians and staff had regarding EHRs.”
BackgroundInfluentialKocaballi, A. B., Ijaz, K., Laranjo, L., Quiroz, J. C., Rezazadegan, D., Tong, H. L., Willcock, S., Berkovsky, S., & Coiera, E. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa131 · Cited by 112 · cited 9×
“from the early stages of the development process is crucial [66], especially as the particular context and non-linearity of primary care clinical consultations pose specific challenges for developing such systems [67].”
BackgroundMethodResultInfluentialSelf-citationProceedings of the 2nd Clinical Natural Language Processing Workshop
(2019) · doi:10.18653/v1/w19-19 · Cited by 107
Zheng, K., Ratwani, R. M., & Adler-Milstein, J. (2020) · Annals of Internal Medicine · doi:10.7326/m19-0871 · Cited by 105
Automation bias in electronic prescribing
Lyell, D., Magrabi, F., Raban, M. Z., Pont, L. G., Baysari, M. T., Day, R. O., & Coiera, E. (2017) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-017-0425-5 · Cited by 103 · cited 1×
“Aspects like explainability [3], automation bias [76], fairness [20], and other ethical challenges need to be addressed [129] in parallel with improving the precision and accuracy of AI.”
BackgroundHe, Y., Zhu, Z., Zhang, Y., Chen, Q., & Caverlee, J. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.372 · Cited by 103
Attitudes towards artificial intelligence within dermatology: an international online survey
Polesie, S., Gillstedt, M., Kittler, H., Lallas, A., Tschandl, P., Zalaudek, I., & Paoli, J. (2020) · British Journal of Dermatology · doi:10.1111/bjd.18875 · Cited by 101 · cited 2×
“More recently, attention has turned to exploring the recent developments related to the use of machine learning algorithms and more broadly AI [99].”
BackgroundBiomedical Named Entity Recognition with Multilingual BERT
Hakala, K., & Pyysalo, S. (2019) · Proceedings of The 5th Workshop on BioNLP Open Shared Tasks · doi:10.18653/v1/d19-5709 · Cited by 96 · cited 1×
“, trained on clinical text corpora [2], have allowed their application for complex NLP tasks including named entity recognition [49], summarization [110], or question answering [51], also in the medical domain and the health industry.”
BackgroundNatural Language Processing: A Human-Computer Interaction Perspective
Manaris, B. (1998) · Advances in computers · doi:10.1016/s0065-2458(08)60665-8 · Cited by 92 · cited 1×
“Although there is an increasing interest in the impacts of implementing advanced automation tools in the work environment, only a few studies have focused on the intersection between Human-Computer Interaction (HCI) and the use of NLP and text automation [80,127].”
BackgroundRealizing the full potential of electronic health records: the role of natural language processing
Ohno-Machado, L. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000501 · Cited by 89 · cited 11×
“This gradual implementation of technology has shown positive results previously as in the case of EHRs [113] where it helped overcome initial concerns that patients, clinicians and staff had regarding EHRs.”
MethodInfluentialThe age of surveillance capitalism: the fight for a human future at the new frontier of power
Donald, R. H. (2020) · Journal of Information Technology Case and Application Research · doi:10.1080/15228053.2020.1860404 · Cited by 88
Context-Based Electronic Health Record: Toward Patient Specific Healthcare
Hsu, W. W. Y., Taira, R. K., El-Saden, S. M., Kangarloo, H., & Bui, A. A. T. (2012) · IEEE Transactions on Information Technology in Biomedicine · doi:10.1109/titb.2012.2186149 · Cited by 86 · cited 1×
“In addition, the current episode-based EHR record, primarily inherited from paper-based records, may also be deemed obsolete, and moving beyond this type of record into a context-based [54] or semantically searchable record [30], could be a welcomed change.”
BackgroundThe Radicalization Risks of GPT-3 and Advanced Neural Language Models
McGuffie, K., & Newhouse, A. (2020) · arXiv · doi:10.48550/arxiv.2009.06807 · Cited by 86 · cited 1×
“It has been recently explored in NLP, where concerns about the widespread use and potential misuse of pretrained language models have been raised [8, 84, 96].”
BackgroundGradual Electronic Health Record Implementation: New Insights on Physician and Patient Adaptation
Shield, R. R., Goldman, R. E., Anthony, D. A., Wang, N., Doyle, R. J., & Borkan, J. (2010) · The Annals of Family Medicine · doi:10.1370/afm.1136 · Cited by 85 · cited 10×
“This gradual implementation of technology has shown positive results previously as in the case of EHRs [113] where it helped overcome initial concerns that patients, clinicians and staff had regarding EHRs.”
BackgroundInfluentialAttitudes Toward Artificial Intelligence Among Radiologists, IT Specialists, and Industry
Jungmann, F., Jorg, T., Hahn, F., Pinto dos Santos, D., Jungmann, S. M., Düber, C., Mildenberger, P., & Kloeckner, R. (2021) · Academic Radiology · doi:10.1016/j.acra.2020.04.011 · Cited by 85 · cited 1×
“Other medical areas show conflicting evidence, such as radiology, where there is a discrepancy between the future expectations of the technology and the confidence of clinicians in AI-based results [62], although other studies have reported a more positive view [134].”
ResultPractical Natural Language Processing: A Comprehensive Guide to Building Real-World NLP Systems
Vajjala, S., Majumder, B. P., Gupta, A., & Surana, H. (2020) · Cited by 77
Using text mining and natural language processing for health care claims processing
Popowich, F. (2005) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1089815.1089824 · Cited by 70 · cited 1×
“With the recent advancements in machine learning methods, NLP has been showing promising results in several fields and application domains [37] including healthcare [101].”
MethodArguing Machines: Human Supervision of Black Box AI Systems That Make Life-Critical Decisions
Fridman, L., Ding, L., Jenik, B., & Reimer, B. (2019) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2019.00173 · Cited by 68 · cited 1×
“Previous research has also shown that the use of a human supervisor as an arbitrator among several ML models might be a successful approach, especially when implementing AI in settings requiring life-critical decisions [40] or when deploying these systems in specific settings such as in rural context [133].”
BackgroundWhy we cannot trust artificial intelligence in medicine
DeCamp, M., & Tilburt, J. C. (2019) · The Lancet Digital Health · doi:10.1016/s2589-7500(19)30197-9 · Cited by 66 · cited 1×
“Trust in AI in a high-stake, high-risk area such as medicine [32] poses one of the main barriers to implementation.”
BackgroundBossen, C., & Pine, K. H. (2023) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3569892 · Cited by 63 · cited 1×
“An example of proposed approaches using NLP that led to implementation is described by Pine and Bossen [98] where clinical documentation integrity specialists use NLP in combination with document statistics for a computer-aided coding of the EHR.”
MethodPandya, R., Nadiadwala, S., Shah, R., & Shah, M. (2019) · Augmented Human Research · doi:10.1007/s41133-019-0021-6 · Cited by 58 · cited 2×
“General Practice, where the use of text records is pervasive, poses an area with both challenges and opportunities for automation [70], especially in the domain of NLP, as opposed to other specialities that rely more on imaging or other diagnostic technologies such as ECGs [7] or EEGs [97].”
BackgroundUser experience evaluation with a Wizard of Oz approach: Technical and methodological considerations
Weiss, A., Bernhaupt, R., Schwaiger, D., Altmaninger, M., Buchner, R., & Tscheligi, M. (2009) · IEEE-RAS International Conference on Humanoid Robots (Humanoids) · doi:10.1109/ichr.2009.5379559 · Cited by 58 · cited 1×
“Interviewing individuals to uncover their values and preferences, before further refinements such as live prototyping [53], or other more specific techniques to explore interactions, such as Wizard of Oz [136] can facilitate testing the more relevant ideas and design elements.”
BackgroundEW, J., & Grove D, H. (2016) · Journal of Health & Medical Informatics · doi:10.4172/2157-7420.1000240 · Cited by 55 · cited 2×
“Concerning the specific use of automation technologies in the EHR, a few studies have addressed the attitudes of clinicians towards those, and in particular, showing the influence of the systems lacking optimization [59] as well as its influence on clinicians’ burnout [119,130].”
BackgroundRethinking EHR interfaces to reduce click fatigue and physician burnout
Collier, R. (2018) · Canadian Medical Association Journal · doi:10.1503/cmaj.109-5644 · Cited by 55 · cited 1×
“Text automation solutions need to address the UX perspective not to create another click-fatigue scenario [28].”
BackgroundA Frame-Based NLP System for Cancer-Related Information Extraction.
Si, Y., & Roberts, K. E. (2018) · Europe PMC (PubMed Central) · Cited by 54
Pollack, A. H., & Pratt, W. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.19301 · Cited by 53 · cited 1×
“Similarly, previous literature has also shown the importance of improving EHR visualization and its effect on clinicians’ cognitive load [100].”
BackgroundRepresentations at work: a national standard for electronic health records
Bossen, C. (2006) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/1180875.1180887 · Cited by 46 · cited 1×
“The automation of medical documentation and EHR access, creation and maintenance has a major impact on the clinicians’ burnout [119] as well as on resources and costs, given that currently, clinicians spend a considerable amount of time producing and consulting EHRs [95] even as there have been earlier calls for EHR r…”
BackgroundBias and Fairness in Natural Language Processing
Chang, K.-W., Prabhakaran, V. M., & Ordóñez, V. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · Cited by 45
Electronic Health Record Optimization and Clinician Well-Being: A Potential Roadmap Toward Action
Shah, T., Borondy Kitts, A., Gold, J. A., Horvath, K., Ommaya, A., Opelka, F., Sato, L., Schwarze, G., Upton, M., & Sandy, L. (2020) · NAM Perspectives · doi:10.31478/202008a · Cited by 41 · cited 1×
“Ranging from decision support systems (DSS)[118] to clinical imaging [43] and clinical prediction models [117], it spans a wide range of optimizations surrounding the clinical pipeline [112].”
BackgroundChoudhury, M. D., Lee, M. K., Zhu, H., & Shamma, D. A. (2020) · Human-Computer Interaction · doi:10.1080/07370024.2020.1744146 · Cited by 34 · cited 1×
“Research into human-automation interaction [60] and specifically AI has gained more attention recently [25].”
BackgroundAutomating Clinical Documentation with Digital Scribes: Understanding the Impact on Physicians
Li, B., Crampton, N. H., Yeates, T., Xia, Y., Tian, X., & Truong, K. N. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445172 · Cited by 33 · cited 3×
“Several themes have been brought up, revolving around effective information extracting, deployment of automated Digital Scribes, the ability to create consultation summaries, as well as the ability to easily search through the records.”
BackgroundThe Electronic Health Record: A Comparison of Some European Countries
Bonomi, S. (2016) · Lecture notes in information systems and organisation · doi:10.1007/978-3-319-28907-6_3 · Cited by 32 · cited 1×
“Although several initiatives are underway (Bonomi, 2016; Mad-husoodanan, 2022), legal protection of health data still would limit the flow in various scenarios, making the need for systems that can extract and copy information from diverse sources, such as primary care, hospital records.”
BackgroundMedical Scribes: Salvation for Primary Care or Workaround for Poor EMR Usability?
Schiff, G. D., & Zucker, L. (2016) · Journal of General Internal Medicine · doi:10.1007/s11606-016-3788-x · Cited by 31 · cited 1×
“Considering the use of human medical scribes, there are important limitations to their applicability including costs [111], frequent changes to medical scribe workforce and a variable degree of satisfaction and availability [29].”
BackgroundCooperative documentation: the patient problem list as a nexus in electronic health records
Zhou, X., Zheng, K., Ackerman, M. S., & Hanauer, D. A. (2012) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2145204.2145340 · Cited by 30 · cited 1×
“Although doctors have previously made calls to reimagine EHR and design it with AI in mind [71] as well as considering certain elements of the EHR design and content structure such as the problem’s list [141], still very limited literature exists on the design and key features that an NLP text system needs to provide;…”
BackgroundWhat makes an EHR "open" or interoperable?
Sittig, D. F., & Wright, A. (2015) · J. Am. Medical Informatics Assoc. · doi:10.1093/jamia/ocv060 · Cited by 29 · cited 1×
“We recognise that there are other non-automated options available such as the use of human medical scribes or introducing greater interoperability between systems and standards [83,116].”
MethodArtificial intelligence in clinical imaging: a health system approach
Gilbert, F. J., Smye, S., & Schönlieb, C. (2019) · Clinical Radiology · doi:10.1016/j.crad.2019.09.122 · Cited by 24 · cited 1×
“Ranging from decision support systems (DSS)[118] to clinical imaging [43] and clinical prediction models [117], it spans a wide range of optimizations surrounding the clinical pipeline [112].”
BackgroundA network model of activities in primary care consultations
Kocaballi, A. B., Coiera, E., Tong, H. L., White, S. J., Quiroz, J. C., Rezazadegan, F., Willcock, S., & Laranjo, L. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz046 · Cited by 23 · cited 4×
“This would allow GPs to connect better with patients and decrease their cognitive load [27,67]”
BackgroundInfluentialSelf-citationEdwards, C. (2021) · Communications of the ACM · doi:10.1145/3449049 · Cited by 23 · cited 1×
“With the recent advancements in machine learning methods, NLP has been showing promising results in several fields and application domains [37] including healthcare [101].”
MethodDavid Oliver: Lessons from the Babylon Health saga
Oliver, D. (2019) · BMJ · doi:10.1136/bmj.l2387 · Cited by 22 · cited 1×
“Applications of AI in medicine recently have focused on high-stake decisions, such as ruling out diseases and providing diagnosis [124] or question-answering systems and conversational agents [94].”
BackgroundModernizing health information technology: lessons from healthcare delivery systems
Amlung, J., Huth, H. B., Cullen, T., & Sequist, T. D. (2020) · JAMIA Open · doi:10.1093/jamiaopen/ooaa027 · Cited by 21 · cited 1×
“The GP interviews showed a growing desire to modernize the EHR interface and interaction with health information and EHR software, which have been highlighted previously [4,86].”
BackgroundThe pattern of silent time in the clinical consultation: an observational multichannel video study
Gibbings-Isaac, D., Iqbal, M., Tahir, M. I., Kumarapeli, P., & de Lusignan, S. (2012) · Family Practice · doi:10.1093/fampra/cms001 · Cited by 19 · cited 1×
“Conversely, there have been numerous examples of poor implementation of technologies, and in particular, EHRs where it had a disruptive effect on clinical workflow, delaying care delivery [64] and increasing the time that clinicians spent in silence during the consultation with patients [42] as well as on decreasing s…”
BackgroundStudies of Interaction Between Human Traders and Algorithmic Trading Systems
Luca, M. D., Szostek, C., Cartlidge, J., & Cliff, D. (2011) · Bristol Research (University of Bristol) · Cited by 14
Corby, S., Whittaker, K., Ash, J. S., Mohan, V., Becton, J., Solberg, N., Bergstrom, R., Orwoll, B., Hoekstra, C., & Gold, J. A. (2021) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-021-01560-4 · Cited by 13 · cited 3×
“Considering the use of human medical scribes, there are important limitations to their applicability including costs [111], frequent changes to medical scribe workforce and a variable degree of satisfaction and availability [29].”
BackgroundA Light-Weight Text Summarization System for Fast Access to Medical Evidence
Sarker, A., Yang, Y., Al-Garadi, M. A., & Abbas, A. (2020) · Frontiers in Digital Health · doi:10.3389/fdgth.2020.585559 · Cited by 11 · cited 1×
“, trained on clinical text corpora [2], have allowed their application for complex NLP tasks including named entity recognition [49], summarization [110], or question answering [51], also in the medical domain and the health industry.”
BackgroundExploring the perceptions and use of electronic medical record systems by non-clinicians
Murphy, A. R., Reddy, M. C., & McNeese, N. J. (2014) · ACM Conference on Designing Interactive Systems (DIS) · doi:10.1145/2598510.2600884 · Cited by 9 · cited 1×
“Preliminary work has been carried out, in areas such as generating an automated textual summary of clinical encounters [27], practical challenges faced by its development and translation [105], as well as clinicians’ [66] and patients’ [90] perceptions of such a system.”
BackgroundPromise or precarity? South African attitudes towards the automation revolution
Roberts, B. J., Gordon, S. L., Struwig, J., Bohler‐Muller, N., & Gastrow, M. (2021) · Development Southern Africa · doi:10.1080/0376835x.2021.1978932 · Cited by 8 · cited 1×
“A wealth of literature has explored the attitudes toward different forms of automation [56,108] in diverse, highly specialized and high-risk environments such as aviation [55] and banking [89].”
BackgroundTIDA: A Spanish EHR Semantic Search Engine
Costumero, R., Gonzalo‐Martín, C., & Menasalvas, E. (2014) · Advances in intelligent systems and computing · doi:10.1007/978-3-319-07581-5_28 · Cited by 7 · cited 1×
“In addition, the current episode-based EHR record, primarily inherited from paper-based records, may also be deemed obsolete, and moving beyond this type of record into a context-based [54] or semantically searchable record [30], could be a welcomed change.”
BackgroundLindgaard, G., & Narasimhan, S. (2009) · International Journal of Mobile Human Computer Interaction · doi:10.4018/jmhci.2009070105 · Cited by 6 · cited 1×
“Our findings also confirm an increasing HCI research interest in newer forms of interaction beyond the Screen-Keyboard-Mouse paradigm [72].”
ResultTowards Natural Language Processing: An Accounting Case Study.
Mayer, J., Stritzel, O., Eßwein, M., & Quick, R. (2020) · Journal of the Association for Information Systems · Cited by 6
Horst, W., & Matthews, B. (2016) · Designing Interactive Systems Conference · doi:10.1145/2901790.2901807 · Cited by 4 · cited 1×
“Interviewing individuals to uncover their values and preferences, before further refinements such as live prototyping [53], or other more specific techniques to explore interactions, such as Wizard of Oz [136] can facilitate testing the more relevant ideas and design elements.”
BackgroundUnlocking the potential of health data to help research and treatments
Madhusoodanan, J. (2022) · Nature · doi:10.1038/d41586-022-01205-0 · Cited by 4
NATURAL LANGUAGE PROCESSING and Human–Computer Interaction
Valencia‐García, R., & García‐Sánchez, F. (2013) · Computer Standards & Interfaces · doi:10.1016/j.csi.2013.03.001 · Cited by 3
Donovan, G. E. (1956) · BMJ · doi:10.1136/bmj.2.4983.44-d · Cited by 3
Variations in the Use of Electronic Medical Records
Tandon, A. (2019) · ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW) · doi:10.1145/3311957.3361848 · Cited by 3
Siddharthan, A. (2001) · Natural Language Engineering · doi:10.1017/s1351324901222700 · Cited by 1
EHR market consolidation and the impact on physicians
Pratt, M. K. (2019) · Cited by 1
Research works printed but not linked: 14
- Åsa Cajander and Christiane Grünloh. 2019. Electronic health records are more than a work tool: Conflicting needs of direct and indirect stakeholders. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–13.
- Hao-Fei Cheng, Ruotong Wang, Zheng Zhang, Fiona O'Connell, Terrance Gray, F. Maxwell Harper, and Haiyi Zhu. 2019. Explaining decision-making algorithms through UI: Strategies to help non-expert stakeholders. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–12.
- Enrico Coiera, Baki Kocaballi, John Halamka, and Liliana Laranjo. 2018. The digital scribe. NPJ Digit. Med. 1, 1 (2018), 1–5.
- Christine Dearnley. 2005. A reflection on the use of semi-structured interviews. Nurse Res. 13, 1 (2005).
- Sarah Freymann Fontenot. 2013. The Affordable Care Act and electronic health care records. Physician Exec. 39, 6 (2013), 72–76.
- Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press.
- W. Gossman, V. Lew, and S. Ghassemzadeh. 2020. SOAP Notes. StatPearls [Internet]. https://www.ncbi.nlm.nih.gov/books/NBK482263/.
- Bruce Hanington and Bella Martin. 2019. Universal Methods of Design Expanded and Revised: 125 Ways to Research Complex Problems, Develop Innovative Ideas, and Design Effective Solutions. Rockport Publishers.
- Anssi Lensu. 2002. Computationally intelligent methods for qualitative data analysis. Jyväskylä Studies in Computing 23 (2002). Retrieved November 22, 2022 from https://jyx.jyu.fi/handle/123456789/13284.
- Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space BT-1st international conference on learning representations. In International Conference on Learning Representations (ICLR'13, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings). Retrieved from http://arxiv.org/abs/1301.3781. · arXiv
- James Proudman. 2018. Cyborg supervision–the application of advanced analytics in prudential supervision. In Proceedings of the Speech, Workshop on Research on Bank Supervision, London.
- Dean F. Sittig. 2013. Electronic Health Records: Challenges in Design and Implementation. CRC Press.
- Neeraj Varshney Swaroop Mishra and Chitta Baral. 2020. It's better to say “I can't answer” than answering incorrectly: Towards safety critical NLP systems. arXiv:2008.09371. Retrieved from https://arxiv.org/abs/2008.09371. · arXiv
- Dakuo Wang, Elizabeth Churchill, Pattie Maes, Xiangmin Fan, Ben Shneiderman, Yuanchun Shi, and Qianying Wang. 2020. From human-human collaboration to human-ai collaboration: Designing ai systems that can work together with people. In Proceedings of the Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems. 1–6.
Cited by
Yang, Z., Xu, X., Yao, B., Rogers, E., Zhang, S., Intille, S., Shara, N., Gao, G. G., & Wang, D. (2024) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3659625 · Cited by 120
AI and Primary Care: Scoping Review
Katonai, G., Árvai, N., & Meskó, B. (2025) · Journal of Medical Internet Research · doi:10.2196/65950 · Cited by 63
“GPs expressed support for doctor-AI collaboration but raised concerns about usability and workflow integration [92].”
Expert evaluation of large language models for clinical dialogue summarization
Navarro, D. F., Coiera, E. W., Hambly, T. W., Triplett, Z., Asif, N., Susanto, A. P., Chowdhury, A., Azcoaga-Lorenzo, A., Dras, M., & Berkovsky, S. (2025) · Scientific Reports · doi:10.1038/s41598-024-84850-x · Cited by 77
Self-citationAndersen, T. O., Nunes, F., Wilcox, L., Coiera, E., & Rogers, Y. (2023) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3589961 · Cited by 40
Martínez-Martínez, H., Alfonso, J. M., Sánchez-Rojo-Huertas, B., Reynolds-Cortez, V., Turégano-Chumillas, A., Meseguer-Ruiz, V. A., Cekrezi, S., & Martínez‐Vizcaíno, V. (2025) · Journal of Medical Internet Research · doi:10.2196/71186 · Cited by 40
“…clinical decisions and create an environment of distrust and pressure that could affect both their medical practice and personal well-being [28]: If everything was completely recorded and stored...the entire conversation, if that was recorded and stored, and the doctor missed something,…”
Influential
Metadata
- Authors
- Fraile Navarro, D., Kocaballi, A. B., Dras, M., & Berkovsky, S.
- Affiliations
- Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, NSW, Australia
- University of Technology Sydney (AU)
- Macquarie University (AU)
- Venue
- ACM Transactions on Computer-Human Interaction
- Published
- 2023-04-13
- Volume
- 30
- Issue
- 2
- Pages
- 1-34
- ISSN
- 1073-0516 (ISSN-L)
- 1557-7325
- OpenAlex
- W4307444957
- Cited by
- 28
- References
- 125
- Citations by year
- 2026: 7 · 2025: 12 · 2024: 7 · 2023: 2
- FWCI
- 3.94
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM Transactions on Computer-Human Interaction (publishedVersion)
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Technology and Human Factors in Education and Health
- Keywords
- general practitioners, natural language processing, electronic health records, clinical record keeping, clinician attitudes, clinical decision support, human-AI interaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.