Cited by 64Electronic Health Records SystemsArtificial Intelligence in Healthcare and EducationMachine Learning in Healthcare

Development and Validation of an Artificial Intelligence System to Optimize Clinician Review of Patient Records

Open access
Chi, Ethan Andrew, Chi, Gordon, Tsui, Cheuk To, Jiang, Yan, Jarr, Karolin, Kulkarni, Chiraag V., Zhang, Michael, Long, Jin +3 more·2021·JAMA Network Open

Abstract

Importance: Physicians are required to work with rapidly growing amounts of medical data. Approximately 62% of time per patient is devoted to reviewing electronic health records (EHRs), with clinical data review being the most time-consuming portion. Objective: To determine whether an artificial intelligence (AI) system developed to organize and display new patient referral records would improve a clinician's ability to extract patient information compared with the current standard of care. Design, Setting, and Participants: In this prognostic study, an AI system was created to organize patient records and improve data retrieval. To evaluate the system on time and accuracy, a nonblinded, prospective study was conducted at a single academic medical center. Recruitment emails were sent to all physicians in the gastroenterology division, and 12 clinicians agreed to participate. Each of the clinicians participating in the study received 2 referral records: 1 AI-optimized patient record and 1 standard (non-AI-optimized) patient record. For each record, clinicians were asked 22 questions requiring them to search the assigned record for clinically relevant information. Clinicians reviewed records from June 1 to August 30, 2020. Main Outcomes and Measures: The time required to answer each question, along with accuracy, was measured for both records, with and without AI optimization. Participants were asked to assess overall satisfaction with the AI system, their preferred review method (AI-optimized vs standard), and other topics to assess clinical utility. Results: Twelve gastroenterology physicians/fellows completed the study. Compared with standard (non-AI-optimized) patient record review, the AI system saved first-time physician users 18% of the time used to answer the clinical questions (10.5 [95% CI, 8.5-12.6] vs 12.8 [95% CI, 9.4-16.2] minutes; P = .02). There was no significant decrease in accuracy when physicians retrieved important patient information (83.7% [95% CI, 79.3%-88.2%] with the AI-optimized vs 86.0% [95% CI, 81.8%-90.2%] without the AI-optimized record; P = .81). Survey responses from physicians were generally positive across all questions. Eleven of 12 physicians (92%) preferred the AI-optimized record review to standard review. Despite a learning curve pointed out by respondents, 11 of 12 physicians believed that the technology would save them time to assess new patient records and were interested in using this technology in their clinic. Conclusions and Relevance: In this prognostic study, an AI system helped physicians extract relevant patient information in a shorter time while maintaining high accuracy. This finding is particularly germane to the ever-increasing amounts of medical data and increased stressors on clinicians. Increased user familiarity with the AI system, along with further enhancements in the system itself, hold promise to further improve physician data extraction from large quantities of patient health records.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1001/jamanetworkopen.2021.17391

Cites

24 of 24 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • In an earlier study, twelve GI physicians utilized an institutionally developed AI tool to help with CDM (AI-Optimized) compared to standard processes.

    “In an earlier study, twelve GI physicians utilized an institutionally developed AI tool to help with CDM (AI-Optimized) compared to standard processes. 16”

    From The use of an artificial intelligence platform OpenEvidence to augment clinical decision-making for primary care physicians (2025) · cites this paper as [16]

    The Methods (Study Design; page not available in our copy) report that 12 gastroenterology clinicians each reviewed an AI-optimized record and a standard record using the study’s AI system. They answered questions designed to test extraction of information from referral records; the study did not test clinical decision-making (CDM) itself. (LLM, conf=0.84)

    Experimental — accuracy not yet measured
  • The AI-optimized tool saved physicians 18% of the time (P < .02) required to answer clinical questions with no significant decrease in accuracy in CDM compared to standard processes.

    “The AI-optimized tool saved physicians 18% of the time (P < .02) required to answer clinical questions with no significant decrease in accuracy in CDM compared to standard processes. 16”

    From The use of an artificial intelligence platform OpenEvidence to augment clinical decision-making for primary care physicians (2025) · cites this paper as [16]

    The Results (Association of AI Optimization With Information Extraction Time) report 18% less time to answer 22 clinical questions, but give P = .02, not P < .02. The Results (Association of AI Optimization With Clinician Accuracy) report comparable accuracy on those questions (P = .81); they do not directly measure accuracy in clinical decision-making. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Authors
Chi, E. A., Chi, G., Tsui, C. T., Jiang, Y., Jarr, K., Kulkarni, C. V., Zhang, M., Long, J., Ng, A. Y., Rajpurkar, P., & Sinha, S. R.
Affiliations
  • Stanford University (US)
  • Stanford Medicine (US)
Corresponding author
  • Rajpurkar, Pranav
  • Sinha, Sidhartha R.
Venue
JAMA Network Open
Published
2021-07-23
Volume
4
Issue
7
Pages
e2117391
ISSN
2574-3805 (ISSN-L)
OpenAlex
W3184771626
Cited by
64
References
24
Citations by year
2026: 10 · 2025: 22 · 2024: 17 · 2023: 8 · 2022: 5 · 2021: 2
FWCI
6.97
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • JAMA Network Open (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Topics
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords
artificial intelligence, electronic health records, clinical data retrieval, clinical decision support, gastroenterology, information accuracy, learning curve, physician satisfaction, natural language processing, clinical workflow efficiency
MeSH terms
Academic Medical Centers, User-Centered Design, Adult, Artificial Intelligence, Female, Humans, Job Satisfaction, Male, Medical Records, Middle Aged, Physicians, Prospective Studies, Referral and Consultation, Task Performance and Analysis, Time Factors, Information Storage and Retrieval, Workload

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review