Impact of a Digital Scribe System on Clinical Documentation Time and Quality: Usability Study
Open accessThe DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.
Abstract
Background: Physicians spend approximately half of their time on administrative tasks, which is one of the leading causes of physician burnout and decreased work satisfaction. The implementation of natural language processing–assisted clinical documentation tools may provide a solution. Objective: This study investigates the impact of a commercially available Dutch digital scribe system on clinical documentation efficiency and quality. Methods: Medical students with experience in clinical practice and documentation (n=22) created a total of 430 summaries of mock consultations and recorded the time they spent on this task. The consultations were summarized using 3 methods: manual summaries, fully automated summaries, and automated summaries with manual editing. We then randomly reassigned the summaries and evaluated their quality using a modified version of the Physician Documentation Quality Instrument (PDQI-9). We compared the differences between the 3 methods in descriptive statistics, quantitative text metrics (word count and lexical diversity), the PDQI-9, Recall-Oriented Understudy for Gisting Evaluation scores, and BERTScore. Results: The median time for manual summarization was 202 seconds against 186 seconds for editing an automatic summary. Without editing, the automatic summaries attained a poorer PDQI-9 score than manual summaries (median PDQI-9 score 25 vs 31, P<.001, ANOVA test). Automatic summaries were found to have higher word counts but lower lexical diversity than manual summaries (P<.001, independent t test). The study revealed variable impacts on PDQI-9 scores and summarization time across individuals. Generally, students viewed the digital scribe system as a potentially useful tool, noting its ease of use and time-saving potential, though some criticized the summaries for their greater length and rigid structure. Conclusions: This study highlights the potential of digital scribes in improving clinical documentation processes by offering a first summary draft for physicians to edit, thereby reducing documentation time without compromising the quality of patient records. Furthermore, digital scribes may be more beneficial to some physicians than to others and could play a role in improving the reusability of clinical documentation. Future studies should focus on the impact and quality of such a system when used by physicians in clinical practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Other authors have utilized similar simulation-based approaches for ADS performance testing, with inconsistent methods for evaluating the resultant output.13,35”
From Evaluating the Quality and Safety of Ambient Digital Scribe Platforms Using Simulated Ambulatory Encounters (2025) · cites this paper as [35]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944 · cited 1×
“The introduction of large language models has disrupted this field, with many papers describing their potential value in clinical note generation and multiple companies now offering digital scribe systems [13-15].”
BackgroundAllocation of Physician Time in Ambulatory Practice: A Time and Motion Study in 4 Specialties
Sinsky, C., Colligan, L., Li, L., Prgomet, M., Reynolds, S., Goeders, L., Westbrook, J., Tutty, M., & Blike, G. (2016) · Annals of Internal Medicine · doi:10.7326/m16-0961 · Cited by 1,494
Arndt, B. G., Beasley, J. W., Watkinson, M. D., Temte, J. L., Tuan, W.-J., Sinsky, C. A., & Gilchrist, V. J. (2017) · The Annals of Family Medicine · doi:10.1370/afm.2121 · Cited by 972 · cited 1×
“The introduction of the electronic health record (EHR) has led to physicians spending up to half of their working hours on administrative duties [3-5].”
BackgroundShanafelt, T. D., West, C. P., Sinsky, C., Trockel, M., Tutty, M., Satele, D. V., Carlasare, L. E., & Dyrbye, L. N. (2019) · Mayo Clinic Proceedings · doi:10.1016/j.mayocp.2018.10.023 · Cited by 914 · cited 1×
“A survey conducted in 2017 involving 5000 physicians in the United States found that 44% exhibited at least 1 sign of burnout [1].”
BackgroundPhysician stress and burnout: the impact of health information technology
Gardner, R. L., Cooper, E., Haskell, J., Harris, D. A., Poplau, S., Kroth, P. J., & Linzer, M. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy145 · Cited by 566 · cited 1×
“Additionally, research linking the use of EHR to burnout indicates that physicians spending more time on EHR, particularly outside of regular hours, face a greater risk of experiencing burnout [8,9].”
BackgroundTowards Generalist Biomedical AI
Tu, T., Azizi, S., Driess, D., Schaekermann, M., Amin, M., Chang, P.-C., Carroll, A., Lau, C., Tanno, R., Ktena, I., et al. (2024) · NEJM AI · doi:10.1056/aioa2300138 · Cited by 483
Towards Generalist Biomedical AI
Tu, T., Azizi, S., Driess, D., Schaekermann, M., Amin, M., Chang, P.-C., Carroll, A., Lau, C., Tanno, R., Ktena, I., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.14334 · Cited by 483
Tai-Seale, M., Olson, C. W., Li, J., Chan, A. S., Morikawa, C., Durbin, M., Wang, W., & Luft, H. S. (2017) · Health Affairs · doi:10.1377/hlthaff.2016.0811 · Cited by 379 · cited 1×
“The introduction of the electronic health record (EHR) has led to physicians spending up to half of their working hours on administrative duties [3-5].”
BackgroundHow Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection
Guo, B., Zhang, X., Wang, Z., Jiang, M., Nie, J., Ding, Y., Yue, J., & Wu, Y. (2023) · arXiv · doi:10.48550/arxiv.2301.07597 · Cited by 318 · cited 1×
“Two previous studies compared human and ChatGPT-written medical texts and reported similar results [22,23].”
ResultRosenbloom, L. (2019) · The Charleston Advisor · doi:10.5260/chara.21.2.8 · Cited by 284
Research works printed but not linked: 3
- 2019, Taking Action Against Clinician Burnout: A Systems Approach to Professional Well-Being
- LinCYROUGE: a package for automatic evaluation of summariesText Summarization Branches Out Barcelona2004Text Summarization Branches Out Barcelona2004-07-25BarcelonaSpainAssociation for Computational Linguistics7481
- Kittinger, BJ, 2016, Perspect Health Inf Manag, 13, 1f
Cited by
Ng, J. J. W., Wang, E., Zhou, X., Zhou, K. X., Goh, C. X. L., Sim, G.-H., Tan, H. K., Goh, S. S. N., & Ng, Q. X. (2025) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-025-03061-0 · Cited by 132
“High WER or omissions, as highlighted by Kodish-Wachs et al. [17], remain a threat to real-time decision-making, and this problem has not disappeared with the advent of LLM-based scribes, as seen in Bundy et al., van Buchem et al. and Biro et al. [23, 31, 32].”
InfluentialThe Impact of AI Scribes on Streamlining Clinical Documentation: A Systematic Review
Sasseville, M., Yousefi, F., Ouellet, S., Naye, F., Stefan, T., Carnovale, V., Bergeron, F., Ling, L., Gheorghiu, B., Hagens, S., et al. (2025) · Healthcare · doi:10.3390/healthcare13121447 · Cited by 122
Accuracy and Safety of AI-Enabled Scribe Technology: Instrument Validation Study
Biro, J., Handley, J. L., Cobb, N. K., Kottamasu, V., Collins, J., Krevat, S. A., & Ratwani, R. M. (2024) · Journal of Medical Internet Research · doi:10.2196/64993 · Cited by 47
“Generative artificial intelligence (AI)–enabled ambient digital scribe (ADS) technology uses the patient-clinician conversation to generate clinical documentation; it has the potential to improve patient engagement and reduce clinician burden [1,2].”
Ha, E., Choon-Kon-Yune, I., Murray, L., Luan, S., Montague, E. N. H., Bhattacharyya, O. K., & Agarwal, P. (2025) · JMIR Human Factors · doi:10.2196/71434 · Cited by 50
“In addition, a study on a commercially available Dutch AI scribe found that fully automated notes, without clinician editing, had poorer PDQI-9 scores, higher word counts, and reduced lexical diversity compared to those reviewed by clinicians [44].”
Evans, K. A., Papinniemi, A., Ploderer, B., Nicholson, V., Hindhaugh, T., Vuvan, V., Cowley, N., Tariq, A., & Thomson, H. (2025) · Musculoskeletal Science and Practice · doi:10.1016/j.msksp.2025.103333 · Cited by 47
Metadata
- Authors
- van Buchem, M. M., Kant, I. M. J., King, L., Kazmaier, J., Steyerberg, E. W., & Bauer, M. P.
- Affiliations
- University Medical Center Utrecht (NL)
- Leiden University Medical Center (NL)
- ORCID
- Venue
- JMIR AI
- Publisher
- JMIR Publications Inc.
- Published
- 2024-09-23
- Volume
- 3
- Pages
- e60020
- ISSN
- 2817-1705 (ISSN-L)
- PMC
- PMC11459111
- OpenAlex
- W4402732546
- Cited by
- 70
- References
- 27
- Citations by year
- 2026: 42 · 2025: 26 · 2024: 2
- FWCI
- 21.19
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- JMIR AI (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Leiden Repository (Leiden University) (submittedVersion, cc-by)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Nursing › Research and Theory › Nursing Diagnosis and Documentation
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- clinical documentation, natural language processing, clinical summarization, documentation time, documentation quality, physician burnout, usability, medical students, lexical diversity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Impact of a Digital Scribe System on Clinical Documentation Time and Quality: Usability Study. https://app.cochat.ai/papers/10.2196/60020
- The paper
- van Buchem, M. M., Kant, I. M. J., King, L., Kazmaier, J., Steyerberg, E. W., & Bauer, M. P. (2024). Impact of a Digital Scribe System on Clinical Documentation Time and Quality: Usability Study. JMIR AI. https://doi.org/10.2196/60020
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
