The DOI matches its Crossref record; 43 of the 46 references it cites were found in a registry or an index; none of the 27 sources its claims cite is checked for support yet.
Timely Follow-up of Abnormal Diagnostic Imaging Test Results in an Outpatient Setting: Are Electronic Medical Records Achieving Their Potential?
Open accessAbstract
BACKGROUND: Given the fragmentation of outpatient care, timely follow-up of abnormal diagnostic imaging results remains a challenge. We hypothesized that an electronic medical record (EMR) that facilitates the transmission and availability of critical imaging results through either automated notification (alerting) or direct access to the primary report would eliminate this problem. METHODS: We studied critical imaging alert notifications in the outpatient setting of a tertiary care Department of Veterans Affairs facility from November 2007 to June 2008. Tracking software determined whether the alert was acknowledged (ie, health care practitioner/provider [HCP] opened the message for viewing) within 2 weeks of transmission; acknowledged alerts were considered read. We reviewed medical records and contacted HCPs to determine timely follow-up actions (eg, ordering a follow-up test or consultation) within 4 weeks of transmission. Multivariable logistic regression models accounting for clustering effect by HCPs analyzed predictors for 2 outcomes: lack of acknowledgment and lack of timely follow-up. RESULTS: Of 123 638 studies (including radiographs, computed tomographic scans, ultrasonograms, magnetic resonance images, and mammograms), 1196 images (0.97%) generated alerts; 217 (18.1%) of these were unacknowledged. Alerts had a higher risk of being unacknowledged when the ordering HCPs were trainees (odds ratio [OR], 5.58; 95% confidence interval [CI], 2.86-10.89) and when dual-alert (>1 HCP alerted) as opposed to single-alert communication was used (OR, 2.02; 95% CI, 1.22-3.36). Timely follow-up was lacking in 92 (7.7% of all alerts) and was similar for acknowledged and unacknowledged alerts (7.3% vs 9.7%; P = .22). Risk for lack of timely follow-up was higher with dual-alert communication (OR, 1.99; 95% CI, 1.06-3.48) but lower when additional verbal communication was used by the radiologist (OR, 0.12; 95% CI, 0.04-0.38). Nearly all abnormal results lacking timely follow-up at 4 weeks were eventually found to have measurable clinical impact in terms of further diagnostic testing or treatment. CONCLUSIONS: Critical imaging results may not receive timely follow-up actions even when HCPs receive and read results in an advanced, integrated electronic medical record system. A multidisciplinary approach is needed to improve patient safety in this area.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 of 27 cited sources checked · 0 supported · 0 flagged
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For 206 (17.2%) alerts, the radiologist also communicated verbally with a provider; in 172 (14.4%) the patient was subsequently admitted after alert transmission; and in 73 (6.1%) there was evidence of both.”
§ Timely Follow-Up of Abnormal Diagnostic Imaging Test Results in an Outpatient Setting: Are Electronic Medical Records Achieving Their Potential? > RESULTS
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Additional verbal communication by the radiologist and hospitalization of the patient within 2-weeks of alert transmission was not significantly associated with acknowledgement.”
§ Timely Follow-Up of Abnormal Diagnostic Imaging Test Results in an Outpatient Setting: Are Electronic Medical Records Achieving Their Potential? > RESULTS > Lack of Acknowledgement
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Half (50.0%) of the tests lacking timely follow-up were abnormal chest x-rays.”
§ Timely Follow-Up of Abnormal Diagnostic Imaging Test Results in an Outpatient Setting: Are Electronic Medical Records Achieving Their Potential? > RESULTS > Lack of Timely Follow-up
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Cancer was determined to be a new diagnosis in 11 of these cases.”
§ Timely Follow-Up of Abnormal Diagnostic Imaging Test Results in an Outpatient Setting: Are Electronic Medical Records Achieving Their Potential? > RESULTS > Lack of Timely Follow-up
Strength:plain· not yet calibratedn = 11Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Between November 2007 and June 2008, 123,638 outpatient studies (including X-rays, CT scans, ultrasounds, MRI and mammography) were performed, of which 1196 (0.97%) images generated alerts.”
§ Timely Follow-Up of Abnormal Diagnostic Imaging Test Results in an Outpatient Setting: Are Electronic Medical Records Achieving Their Potential? > RESULTS
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although in 5 cases we could not determine outcomes because the patient did not return to the institution”
§ Timely Follow-Up of Abnormal Diagnostic Imaging Test Results in an Outpatient Setting: Are Electronic Medical Records Achieving Their Potential? > RESULTS > Lack of Timely Follow-up
Strength:plain· not yet calibratedn = 5Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that in 46 (50%) cases providers had executed the appropriate action.”
§ Timely Follow-Up of Abnormal Diagnostic Imaging Test Results in an Outpatient Setting: Are Electronic Medical Records Achieving Their Potential? > RESULTS
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For 206 (17.2%) alerts, the radiologist also communicated verbally with a provider; in 172 (14.4%) the patient was subsequently admitted after alert transmission; and in 73 (6.1%) there was evidence of both.”
§ Timely Follow-Up of Abnormal Diagnostic Imaging Test Results in an Outpatient Setting: Are Electronic Medical Records Achieving Their Potential? > RESULTS
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In these 92 cases, more than a fourth (n=26; 28%) led to a new diagnosis being established.”
§ Timely Follow-Up of Abnormal Diagnostic Imaging Test Results in an Outpatient Setting: Are Electronic Medical Records Achieving Their Potential? > RESULTS > Lack of Timely Follow-up
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For 206 (17.2%) alerts, the radiologist also communicated verbally with a provider; in 172 (14.4%) the patient was subsequently admitted after alert transmission; and in 73 (6.1%) there was evidence of both.”
§ Timely Follow-Up of Abnormal Diagnostic Imaging Test Results in an Outpatient Setting: Are Electronic Medical Records Achieving Their Potential? > RESULTS
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
- It may state two separate claims as one. 11
- Probably not a substantial claim. 10
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Types and specialties of ordering providers and types of imaging tests were significantly different across the two groups.”
§ Timely Follow-Up of Abnormal Diagnostic Imaging Test Results in an Outpatient Setting: Are Electronic Medical Records Achieving Their Potential? > RESULTS > Lack of Acknowledgement
Strength:plain· not yet calibratedCausal:correlational· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Of 74 alerts unacknowledged by trainees that were followed by a documented response in the medical record at the time of review, the ordering trainee directly responded to the test result (without opening the alert) in 34 cases (9 of which were prompted by a direct notification by the radiologist).”
§ Timely Follow-Up of Abnormal Diagnostic Imaging Test Results in an Outpatient Setting: Are Electronic Medical Records Achieving Their Potential? > RESULTS > Lack of Acknowledgement
Strength:plain· not yet calibratedn = 34Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In the remaining 40 cases, another provider documented a response to the report (4 of which were prompted by a direct notification by the radiologist).”
§ Timely Follow-Up of Abnormal Diagnostic Imaging Test Results in an Outpatient Setting: Are Electronic Medical Records Achieving Their Potential? > RESULTS > Lack of Acknowledgement
Strength:plain· not yet calibratedn = 40Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Previous studies raise concern that providers may not read all their alerts.14, 16-19”
§ Timely Follow-Up of Abnormal Diagnostic Imaging Test Results in an Outpatient Setting: Are Electronic Medical Records Achieving Their Potential? > INTRODUCTION
What the sources say
- Not checked yetcites [14] Singh H Arora HS Vij MS Rao R Khan M Petersen LA Communication outcomes of critical imaging results in a computerized notification system. J Am Med Inform Assoc 2007 14 4 459 66 17460135 PMC2244901
- Not checked yetcites [16] Choksi VR Marn CS Bell Y Carlos R Efficiency of a semiautomated coding and review process for notification of critical findings in diagnostic imaging. AJR Am J Roentgenol April 2006 186 4 933 6 16554559
- Not checked yetcites [17] Wahls TL Haugen T Cram P Diagnostic errors associated with failure to follow up on abnormal results. Joint Commission Journal on Quality and Patient Safety/Joint Commission Resources 2007
- Not checked yetcites [18] Wahls TL Cram PM The frequency of missed test results and associated treatment delays in a highly computerized health system. BMC Fam Pract 2007 8 32 17519017 PMC1891295
- Not checked yetcites [19] Kuperman GJ Teich JM Bates DW Detecting alerts, notifying the physician, and offering action items: a comprehensive alerting system. Proc AMIA Annu Fall Symp 1996 704 8 8947756 PMC2233134
Load-bearingStrength:hedged· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
In the paper
“Electronic communication using alerts (computerized notifications of critical information such as abnormal diagnostic test results) can facilitate transmission and potentially, a response and follow-up action by an ordering practitioner – an advantage over paper based reporting.14”
§ Timely Follow-Up of Abnormal Diagnostic Imaging Test Results in an Outpatient Setting: Are Electronic Medical Records Achieving Their Potential? > INTRODUCTION
What the sources say
- Not checked yetcites [14] Singh H Arora HS Vij MS Rao R Khan M Petersen LA Communication outcomes of critical imaging results in a computerized notification system. J Am Med Inform Assoc 2007 14 4 459 66 17460135 PMC2244901
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
In the paper
“However, effective communication involves more than just information transfer; it requires a response from the recipient, such as taking a follow-up action and acknowledging receipt of the information to the sender.15”
§ Timely Follow-Up of Abnormal Diagnostic Imaging Test Results in an Outpatient Setting: Are Electronic Medical Records Achieving Their Potential? > INTRODUCTION
What the sources say
- Not checked yetcites [15] Ash JS Berg M Coiera E Some unintended consequences of information technology in health care: the nature of patient care information system-related errors. J Am Med Inform Assoc March 2004 11 2 104 12 14633936 PMC353015
Tests✓ Asserted: passes✗ Checkable: fails✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may be too vague to check against evidence.
None of the sources cited for this claim have been checked yet.
In the paper
“Communication breakdown is consistently identified as a preventable factor in studies of adverse events1-6 and a significant contributor to outpatient diagnostic errors from lack of follow-up of abnormal test results.7-12”
§ Timely Follow-Up of Abnormal Diagnostic Imaging Test Results in an Outpatient Setting: Are Electronic Medical Records Achieving Their Potential? > INTRODUCTION
What the sources say
- Not checked yetcites [1] Joint Commission on Accreditation of Healthcare Organizations Sentinel event statistics, March 31, 2006. Joint Commission on Accreditation of Healthcare Organizations 2005 Available at: URL: http://www.jointcommission.org/NR/rdonlyres/FA465646-5F5F-4543-AC8F-E8AF6571E372/0/root_cause_se.jpg 19343995
- Not checked yetcites [2] Arora V Johnson J Lovinger D Humphrey HJ Meltzer DO Communication failures in patient sign-out and suggestions for improvement: a critical incident analysis. Qual Saf Health Care 1 December 2005 14 6 401 7 16326783 PMC1744089
- Not checked yetcites [3] Graber ML Franklin N Gordon R Diagnostic error in internal medicine. Arch Intern Med 11 July 2005 165 13 1493 9 16009864
- Not checked yetcites [4] Greenberg CC Regenbogen SE Studdert DM Patterns of Communication Breakdowns Resulting in Injury to Surgical Patients. Journal of the American College of Surgeons April 2007 204 4 533 40 17382211
- Not checked yetcites [5] Kripalani S LeFevre F Phillips CO Williams MV Basaviah P Baker DW Deficits in Communication and Information Transfer Between Hospital-Based and Primary Care Physicians: Implications for Patient Safety and Continuity of Care. JAMA 28 February 2007 297 8 831 41 17327525
- Not checked yetcites [6] Singh H Thomas EJ Petersen LA Studdert DM Medical Errors Involving Trainees: A Study of Closed Malpractice Claims From 5 Insurers. Arch Intern Med 22 October 2007 167 19 2030 6 17954795
- Not checked yetcites [7] Gandhi TK Fumbled handoffs: one dropped ball after another. Ann Intern Med 1 March 2005 142 5 352 8 15738454
- Not checked yetcites [8] Poon EG Gandhi TK Sequist TD Murff HJ Karson AS Bates DW “I wish I had seen this test result earlier!”: Dissatisfaction with test result management systems in primary care. Arch Intern Med 8 November 2004 164 20 2223 8 15534158
- Not checked yetcites [9] Poon EG Haas JS Louise PA Communication factors in the follow-up of abnormal mammograms. J Gen Intern Med April 2004 19 4 316 23 15061740 PMC1492194
- Not checked yetcites [10] Schiff GD Introduction: Communicating critical test results. Jt Comm J Qual Patient Saf February 2005 31 2 63 5 15791764
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
In the paper
“the electronic medical record (EMR) used by the Department of Veterans Affairs (VA) mostly relies on a notification system (the “View Alert” window) to alert clinicians about critical test results, whereas only in selected life-threatening cases does the radiologist verbally communicate the abnormality to a provider.14”
§ Timely Follow-Up of Abnormal Diagnostic Imaging Test Results in an Outpatient Setting: Are Electronic Medical Records Achieving Their Potential? > INTRODUCTION
What the sources say
- Not checked yetcites [14] Singh H Arora HS Vij MS Rao R Khan M Petersen LA Communication outcomes of critical imaging results in a computerized notification system. J Am Med Inform Assoc 2007 14 4 459 66 17460135 PMC2244901
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Unexpectedly, our current study shows substantial lack of timely follow-up even in alerts read by providers; hence the increase from our previous estimate.”
§ Timely Follow-Up of Abnormal Diagnostic Imaging Test Results in an Outpatient Setting: Are Electronic Medical Records Achieving Their Potential? > DISCUSSION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“If no response was documented, a second study investigator confirmed the findings in the medical record and then contacted the ordering provider by telephone.”
§ Timely Follow-Up of Abnormal Diagnostic Imaging Test Results in an Outpatient Setting: Are Electronic Medical Records Achieving Their Potential? > METHODS > Data Collection
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Multivariable logistic regression models accounting for clustering effect by providers were used to identify factors associated with the two outcome variables (i.e. lack of acknowledgment, and lack of timely follow-up).”
§ Timely Follow-Up of Abnormal Diagnostic Imaging Test Results in an Outpatient Setting: Are Electronic Medical Records Achieving Their Potential? > METHODS > Data analysis
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
In the paper
“We allowed 4 weeks as the maximum time to follow-up on non-life threatening findings based on our previous work.14”
§ Timely Follow-Up of Abnormal Diagnostic Imaging Test Results in an Outpatient Setting: Are Electronic Medical Records Achieving Their Potential? > METHODS > Data Collection
What the sources say
- Not checked yetcites [14] Singh H Arora HS Vij MS Rao R Khan M Petersen LA Communication outcomes of critical imaging results in a computerized notification system. J Am Med Inform Assoc 2007 14 4 459 66 17460135 PMC2244901
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
In the paper
“The Computerized Patient Record System (CPRS) is the EMR used at all VA facilities. It is maintained within the Veterans Health Information Systems and Technology Architecture (VISTA), an automated information system used to support ambulatory and inpatient care.25”
§ Timely Follow-Up of Abnormal Diagnostic Imaging Test Results in an Outpatient Setting: Are Electronic Medical Records Achieving Their Potential? > METHODS > Data Collection
What the sources say
- Not checked yetcites [25] VA Information Resource Center Veterans Health Information Systems and Technology Architecture (VistA). Veterans Health Information Systems and Technology Architecture (VistA) - Description 2007 Available at: URL: http://www.virec.research.va.gov/DataSourcesName/VISTA/VISTA.htm. January 25, 2007
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To ensure the model had stable and precise estimates of odds ratios, variables with small numbers of patients with a given characteristic (such as mammography) were removed, and a final model was run on the remaining predictor variables.”
§ Timely Follow-Up of Abnormal Diagnostic Imaging Test Results in an Outpatient Setting: Are Electronic Medical Records Achieving Their Potential? > METHODS > Data analysis
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
None of the sources cited for this claim have been checked yet.
In the paper
“Data on outpatient drug related alerts has shown providers ignore alerts because of information overload, 20-23 often resulting in failure to act upon important as well as clinically unimportant alerts.20”
§ Timely Follow-Up of Abnormal Diagnostic Imaging Test Results in an Outpatient Setting: Are Electronic Medical Records Achieving Their Potential? > INTRODUCTION
What the sources say
- Not checked yetcites [20] van der Sijs H Aarts J Vulto A Berg M Overriding of Drug Safety Alerts in Computerized Physician Order Entry. J Am Med Inform Assoc 1 March 2006 13 2 138 47 16357358 PMC1447540
- Not checked yetcites [21] Taylor LK Tamblyn R Reasons for physician non-adherence to electronic drug alerts. Stud Health Technol Inform 2004 107 Pt 2 1101 5 15360983
- Not checked yetcites [22] Magnus D Rodgers S Avery AJ GPs’ views on computerized drug interaction alerts: questionnaire survey. J Clin Pharm Ther October 2002 27 5 377 82 12383140
- Not checked yetcites [23] Feldstein A Simon SR Schneider J How to design computerized alerts to safe prescribing practices. Jt Comm J Qual Saf November 2004 30 11 602 13 15565759
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“A reviewer blinded to acknowledgement status evaluated the medical records of patients to determine if there was any documented response to the alert.”
§ Timely Follow-Up of Abnormal Diagnostic Imaging Test Results in an Outpatient Setting: Are Electronic Medical Records Achieving Their Potential? > METHODS > Data Collection
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The local Institutional Review Boards approved the study.”
§ Timely Follow-Up of Abnormal Diagnostic Imaging Test Results in an Outpatient Setting: Are Electronic Medical Records Achieving Their Potential? > METHODS
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Conversely, many factors including a large sample size, involvement of several clinics, large number of patients (over 120,000 veterans in southeast Texas) and providers (over 500 from different specialties), rigorous methods to determine timely follow-up and outcomes at 9 months of alerts without timely follow-up, various types of diagnostic imaging tests and an advanced integrated EMR used in VA”
§ Timely Follow-Up of Abnormal Diagnostic Imaging Test Results in an Outpatient Setting: Are Electronic Medical Records Achieving Their Potential? > DISCUSSION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim. It may state two separate claims as one.
The paper relies on this without support.
No data exists to substantiate a similar problem for diagnostic test result alerts.
“While no data exists to substantiate a similar problem for diagnostic test result alerts”
The paper relies on this without support.
Because of computerized provider order entry, the ordering clinician is always known and notified.
“Because of computerized provider order entry, the ordering clinician is always known and notified.”
The paper relies on this without support.
Institutional protocol requires documentation of verbal communication in the imaging report.
“Institutional protocol requires that documentation of verbal communication is always included in the imaging report.”
The paper relies on this without support.
In our system, residents spend only ½ to 1 day in a two-week period in an outpatient setting usually in their own continuity clinics.
“residents spend only ½ to 1 day in a two-week period in an outpatient setting usually in their own continuity clinics.”
The paper relies on this without support.
Duty hour rules impact resident clinic schedule.
“In addition, duty hour rules also impact their clinic schedule.”
The paper relies on this without support.
Providers in the VA are highly dependent on the View Alert system for their abnormal test results.
“Being their only “inbox” for all types of notifications, providers in the VA are highly dependent on the View Alert system for their abnormal test results.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The result from the baseline EHR group is in line with findings from real clinical settings that indicated physicians did not acknowledge abnormal test results in approximately 4% of cases.19,20”
From Association of the Usability of Electronic Health Records With Cognitive Workload and Performance Levels Among Physicians (2019) · cites this paper as [20]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Even in the EHR environment, with alerts and reminders in place, physicians continue to often inappropriately manage abnormal test results.12,13,14,15,16,17,18,19,20,21”
From Association of the Usability of Electronic Health Records With Cognitive Workload and Performance Levels Among Physicians (2019) · cites this paper as [20]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“On the basis of the anticipated rate of appropriately identified abnormal test results in the literature12,13,14,15,16,17,18,19,20,21 and the anticipated magnitude of the association of the enhanced EHR, we required a sample size of 30 participants, each reviewing 35 test results, to achieve 80% power to detect a statistically significant difference in cognitive workload and performance.”
From Association of the Usability of Electronic Health Records With Cognitive Workload and Performance Levels Among Physicians (2019) · cites this paper as [20]
In the citing paper
“For instance, in our previous work, we found this sociotechnical model applicable in specific clinical contexts (eg, test results and referral communication), 43 -48”
From Exploring the sociotechnical intersection of patient safety and electronic health record implementation (2014) · cites this paper as [46]
In the citing paper
“These findings are similar to our previous findings of follow-up on abnormal imaging alerts in the same electronic medical record suggesting that this phenomenon may exist for all alerts of abnormal diagnostic test results.”
From Notification of Abnormal Lab Test Results in an Electronic Medical Record: Do Any Safety Concerns Remain? (2010) · cites this paper as [19]
In the citing paper
“In previous work, we found abnormal diagnostic imaging alerts may not always be reviewed by ordering practitioners, and practitioners who review them may not always act upon the transmitted results in a timely manner.”
From Notification of Abnormal Lab Test Results in an Electronic Medical Record: Do Any Safety Concerns Remain? (2010) · cites this paper as [19]
Citations
Cites
Kawamoto, K., Houlihan, C. A., Balas, E. A., & Lobach, D. F. (2005) · BMJ · doi:10.1136/bmj.38398.500764.8f · Cited by 2,703
Kripalani, S., LeFevre, F., Phillips, C. O., Williams, M. V., Basaviah, P., & Baker, D. W. (2007) · JAMA · doi:10.1001/jama.297.8.831 · Cited by 2,103
Work system design for patient safety: the SEIPS model
Carayon, P., Schoofs Hundt, A., Karsh, B.-T., Gurses, A. P., Alvarado, C. J., Smith, M., & Flatley Brennan, P. (2006) · Quality in Health Care · doi:10.1136/qshc.2005.015842 · Cited by 1,795 · cited 1×
“The impact of several factors such as time pressures, organizational characteristics, factors related to the task of processing alerts, technological factors, and HCP factors should be considered in the design of interventions to improve timely follow-up.(29,30) Future studies should address process-of-care issues lea…”
BackgroundAsh, J. S. (2003) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1471 · Cited by 1,789 · cited 1×
“just information transfer—it requires a response from the recipient, such as taking follow-up action and acknowledging receipt of the information to the sender.(15) Previous studies raise concern that HCPs may not read all their alerts.”
BackgroundDiagnostic error in internal medicine
Graber, M. L., Franklin, N., & Gordon, R. (2005) · Archives of Internal Medicine · doi:10.1001/archinte.165.13.1493 · Cited by 1,549
Overriding of Drug Safety Alerts in Computerized Physician Order Entry
van der Sijs, H., Aarts, J., Vulto, A., & Berg, M. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1809 · Cited by 1,117 · cited 1×
“Data on outpatient drug-related alerts have shown that HCPs ignore alerts because of information overload,(20-23) often resulting in failure to act on important as well as clinically unimportant alerts.(20) While no data exist to substantiate a similar problem for diagnostic test result alerts, HCPs often receive seve…”
BackgroundPatterns of Communication Breakdowns Resulting in Injury to Surgical Patients
Greenberg, C. C., Regenbogen, S. E., Studdert, D. M., Lipsitz, S. R., Rogers, S. O., Zinner, M. J., & Gawande, A. A. (2007) · Journal of the American College of Surgeons · doi:10.1016/j.jamcollsurg.2007.01.010 · Cited by 856
Arora, V. M. (2005) · BMJ Quality & Safety · doi:10.1136/qshc.2005.015107 · Cited by 607
Institute of Medicine (2004) · National Academies Press eBooks · doi:10.17226/10863 · Cited by 528
Expanding access to cancer screening and clinical follow-up among the medically underserved.
Freeman, H. P., Muth, B., & Kerner, J. F. (1995) · PubMed · Cited by 449
Medical Errors Involving Trainees: A Study of Closed Malpractice Claims From 5 Insurers
Singh, H., Thomas, E. J., Petersen, L. A., & Studdert, D. M. (2007) · Archives of Internal Medicine · doi:10.1001/archinte.167.19.2030 · Cited by 407
Self-citationPoon, E. G., Gandhi, T. K., Sequist, T. D., Murff, H. J., Karson, A. S., & Bates, D. W. (2004) · Archives of Internal Medicine · doi:10.1001/archinte.164.20.2223 · Cited by 301
Improving follow-up to abnormal breast cancer screening in an urban population
Battaglia, T. A., Roloff, K. S., Posner, M. A., & Freund, K. M. (2006) · Cancer · doi:10.1002/cncr.22354 · Cited by 263
Fumbled handoffs: one dropped ball after another
Gandhi, T. K. (2005) · Annals of Internal Medicine · doi:10.7326/0003-4819-142-5-200503010-00010 · Cited by 246
Patient navigation and case management following an abnormal mammogram: A randomized clinical trial
Ell, K., Vourlekis, B. S., Lee, P.-J., & Xie, B. (2006) · Preventive Medicine · doi:10.1016/j.ypmed.2006.08.001 · Cited by 220
Patient Navigation: A Community Centered Approach to Reducing Cancer Mortality
Freeman, H. P. (2006) · Journal of Cancer Education · doi:10.1207/s15430154jce2101s_4 · Cited by 199
Singh, H., Naik, A. D., Rao, R., & Petersen, L. A. (2008) · Journal of General Internal Medicine · doi:10.1007/s11606-007-0393-z · Cited by 162 · cited 2×
“The volume of outpatient care and nature of high-risk transitions between health care practitioners/ providers (HCPs), settings, and systems of care makes timely communication particularly challenging.(13) For example, a patient referred for diagnostic workup for respiratory symptoms by a primary care provider (PCP) m…”
BackgroundSelf-citationHickner, J., Graham, D. G., Elder, N. C., Brandt, E., Emsermann, C. B., Dovey, S., & Phillips, R. (2008) · Quality and Safety in Health Care · doi:10.1136/qshc.2006.021915 · Cited by 158
GPs' views on computerized drug interaction alerts: questionnaire survey
Magnus, D., Rodgers, S., & Avery, A. J. (2002) · Journal of Clinical Pharmacy and Therapeutics · doi:10.1046/j.1365-2710.2002.00434.x · Cited by 157
Understanding diagnostic errors in medicine: a lesson from aviation
Singh, H. (2006) · Quality and Safety in Health Care · doi:10.1136/qshc.2005.016444 · Cited by 156
Self-citationCommunication outcomes of critical imaging results in a computerized notification system
Singh, H., Arora, H. S., Vij, M. S., Rao, R., Khan, M. M., & Petersen, L. A. (2007) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2280 · Cited by 140 · cited 7×
“We allowed 4 weeks as the maximum time to follow up on non–life-threatening findings based on our previous work.(14) When we called HCPs and lack of timely follow-up was confirmed, we encouraged them to initiate appropriate follow-up action and rereviewed their records a week later to check for action.”
BackgroundMethodResultInfluentialSelf-citationTo err is not entirely human: Complex technology and user cognition
Horsky, J., Zhang, J., & Patel, V. L. (2005) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2005.05.002 · Cited by 138
Communication factors in the follow-up of abnormal mammograms
Poon, E. G., Haas, J. S., Puopolo, A. L., Gandhi, T. K., Burdick, E., Bates, D. W., & Brennan, T. A. (2004) · Journal of General Internal Medicine · doi:10.1111/j.1525-1497.2004.30357.x · Cited by 138 · cited 1×
“However, we believe that our estimates of lack of timely follow-up are likely better than in systems that do not use computerized notification, although accurate and comparable data from such systems is lacking.(9,31) Trainees had a higher likelihood of not reading their alerts but were not associated with a lack of t…”
BackgroundErrors in cancer diagnosis: current understanding and future directions
Singh, H., Sethi, S., Raber, M., & Petersen, L. A. (2007) · Journal of Clinical Oncology · doi:10.1200/jco.2007.13.2142 · Cited by 121 · cited 1×
“Policies and procedures regarding the use of verbal communication for such results may need to be reevaluated.(39,40) While the next generation of EMR-based notification systems are being designed, we propose several potential interventions based on our findings that can be used immediately to improve timely follow-up…”
BackgroundSelf-citationWahls, T. L., & Cram, P. M. (2007) · BMC Family Practice · doi:10.1186/1471-2296-8-32 · Cited by 121
Design and implementation of a comprehensive outpatient Results Manager
Poon, E. G., Wang, S. J., Gandhi, T. K., Bates, D. W., & Kuperman, G. J. (2003) · Journal of Biomedical Informatics · doi:10.1016/s1532-0464(03)00061-3 · Cited by 111 · cited 1×
“While no data exist to substantiate a similar problem for diagnostic test result alerts, HCPs often receive several hundred normal diagnostic test results and many abnormal alerts every week, suggesting that a similar problem could exist.(24) Given that research in this area has thus far focused on critical results th…”
BackgroundKuperman, G. J., Teich, J. M., Bates, D. W., Hiltz, F. L., Hurley, J. M., Lee, R. Y., & Paterno, M. D. (1996) · Europe PMC (PubMed Central) · Cited by 87
Reasons for Physician Non-Adherence to Electronic Drug Alerts
Taylor, L. K., & Tamblyn, R. M. (2004) · Studies in health technology and informatics · doi:10.3233/978-1-60750-949-3-1101 · Cited by 86
Murff, H. J., Gandhi, T. K., Karson, A. K., Mort, E. A., Poon, E. G., Wang, S. J., Fairchild, D. G., & Bates, D. W. (2003) · International Journal of Medical Informatics · doi:10.1016/s1386-5056(03)00133-3 · Cited by 83
How to Design Computerized Alerts to Ensure Safe Prescribing Practices
Feldstein, A. C., Simon, S. R., Schneider, J. P., Krall, M., Laferriere, D. S., Smith, D. H., Sittig, D. F., & Soumerai, S. B. (2004) · The Joint Commission Journal on Quality and Safety · doi:10.1016/s1549-3741(04)30071-7 · Cited by 81
Self-citationThe Patterns of Communication Breakdowns Resulting in Injury to Surgical Patients
Greenberg, C. C., Regenbogen, S. E., & Studdert, D. N. (2007) · Journal of Vascular Surgery · doi:10.1016/j.jvs.2007.06.019 · Cited by 79
Issues and Initiatives in the Testing Process in Primary Care Physician Offices
Hickner, J. M., Fernald, D. H., Harris, D. M., Poon, E. G., Elder, N. C., & Mold, J. W. (2005) · The Joint Commission Journal on Quality and Patient Safety · doi:10.1016/s1553-7250(05)31012-9 · Cited by 74 · cited 1×
“While institutional practices may determine whether the PCP or another ordering HCP (for instance the subspecialist) follows up an abnormality, this problem may be improved by better tracking systems,(14,32-34) using reminders built into the EMR, and care-coordination programs.(27,35-38) Second, direct (verbal) commun…”
BackgroundKushniruk, A. W., & Patel, V. L. (1998) · International Journal of Medical Informatics · doi:10.1016/s1386-5056(98)00106-3 · Cited by 70 · cited 1×
“The impact of several factors such as time pressures, organizational characteristics, factors related to the task of processing alerts, technological factors, and HCP factors should be considered in the design of interventions to improve timely follow-up.(29,30) Future studies should address process-of-care issues lea…”
BackgroundChoksi, V. R., Marn, C. S., Bell, Y., & Carlos, R. C. (2006) · American Journal of Roentgenology · doi:10.2214/ajr.04.1913 · Cited by 53
Failure to Recognize and Act on Abnormal Test Results: The Case of Screening Bone Densitometry
Cram, P. M., Rosenthal, G. E., Ohsfeldt, R. L., Wallace, R. B., Schlechte, J., & Schiff, G. D. (2005) · The Joint Commission Journal on Quality and Patient Safety · doi:10.1016/s1553-7250(05)31013-0 · Cited by 51 · cited 1×
“However, we believe that our estimates of lack of timely follow-up are likely better than in systems that do not use computerized notification, although accurate and comparable data from such systems is lacking.(9,31) Trainees had a higher likelihood of not reading their alerts but were not associated with a lack of t…”
BackgroundWahls, T. L., Haugen, T. H., & Cram, P. M. (2007) · The Joint Commission Journal on Quality and Patient Safety · doi:10.1016/s1553-7250(07)33052-3 · Cited by 46
Improving Colorectal Cancer Screening by Targeting Office Systems in Primary Care Practices
Wei, E. K., Ryan, C. T., Dietrich, A. J., & Colditz, G. A. (2005) · Archives of Internal Medicine · doi:10.1001/archinte.165.6.661 · Cited by 44
Gandhi, T. K., Sequist, T. D., Poon, E. G., et al. (2003) · PubMed · Cited by 35
Editorial: Doing Better with Critical Test Results
Bates, D. W., & Leape, L. L. (2005) · The Joint Commission Journal on Quality and Patient Safety · doi:10.1016/s1553-7250(05)31010-5 · Cited by 25
Introduction: Communicating Critical Test Results
Schiff, G. D. (2005) · The Joint Commission Journal on Quality and Patient Safety · doi:10.1016/s1553-7250(05)31009-9 · Cited by 19
Advanced alerting features: displaying new relevant data and retracting alerts.
Kuperman, G. J., Hiltz, F. L., & Teich, J. M. (1997) · Europe PMC (PubMed Central) · Cited by 12
Update: sentinel event statistics available online.
(2009) · PubMed · Cited by 5
Updated sentinel event statistics for midyear 2010.
(2010) · PubMed · Cited by 3
Flagged first, then by citations
3 of the 46 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- [1]Joint Commission on Accreditation of Healthcare Organizations Sentinel event statistics, March 31, 2006. Joint Commission on Accreditation of Healthcare Organizations 2005 Available at: URL: http://www.jointcommission.org/NR/rdonlyres/FA465646-5F5F-4543-AC8F-E8AF6571E372/0/root_cause_se.jpg 19343995
- [17]Wahls TL Haugen T Cram P Diagnostic errors associated with failure to follow up on abnormal results. Joint Commission Journal on Quality and Patient Safety/Joint Commission Resources 2007
Cited by
Sorted by citations
Holden, R. J., Carayon, P., Gurses, A. P., Hoonakker, P., Hundt, A. S., Ozok, A. A., & Rivera-Rodriguez, A. J. (2013) · Ergonomics · doi:10.1080/00140139.2013.838643 · Cited by 1,221
“…2008a, 2008b) Analysis of critical care work systems (Catchpole & McCulloch, 2010) Timeliness of follow-up of abnormal tests in outpatient settings (Singh et al., 2009) Workload and performance obstacles among ICU nurses (Carayon & Gurses, 2008; Gürses & Carayon, 2007; Holden, Scanlon, et al.,…”
BackgroundSittig, D. F., & Singh, H. (2010) · Quality and Safety in Health Care · doi:10.1136/qshc.2010.042085 · Cited by 897
“Safe and effective use of an EHR-based notification system involves many factors that are addressed by almost all dimensions of our model.”
BackgroundSelf-citationHuman factors systems approach to healthcare quality and patient safety
Carayon, P., Wetterneck, T. B., Rivera-Rodriguez, A. J., Hundt, A. S., Hoonakker, P., Holden, R., & Gürses, A. P. (2013) · Applied Ergonomics · doi:10.1016/j.apergo.2013.04.023 · Cited by 814
“Researchers have used the SEIPSmodel to study timeliness of follow-up of abnormal test results in outpatient settings (Singh et al., 2009), to examine the safety of EHR (Electronic Health Record) technology (Sittig and Singh, 2009), to evaluate ways of improving electronic communication and alerts (Hysong et al.”
MethodSingh, H., Meyer, A. N. D., & Thomas, E. J. (2014) · BMJ Quality & Safety · doi:10.1136/bmjqs-2013-002627 · Cited by 677
Self-citationThe incidence of diagnostic error in medicine
Graber, M. L. (2013) · BMJ Quality & Safety · doi:10.1136/bmjqs-2012-001615 · Cited by 626
Metadata
- Authors
- Singh, H., Thomas, E. J., Mani, S., Sittig, D., Arora, H., Espadas, D., Khan, M. M., & Petersen, L. A.
- Affiliations
- Michael E. DeBakey VA Medical Center (US)
- Memorial Hermann (US)
- Baylor College of Medicine (US)
- Venue
- Archives of Internal Medicine
- Publisher
- American Medical Association (AMA)
- Published
- 2009-09-28
- Volume
- 169
- Issue
- 17
- Pages
- 1578-86
- ISSN
- 0003-9926 (ISSN-L)
- PMC
- PMC2919821
- OpenAlex
- W2143111289
- Semantic Scholar
- e10c1783993bfba9a725ad6763f791cfa11a7c91
- Cited by
- 212
- References
- 46
- Citations by year
- 2026: 5 · 2025: 3 · 2024: 2 · 2023: 9 · 2022: 7 · 2021: 10 · 2020: 11 · 2019: 10 · 2018: 18 · 2017: 12 · 2016: 15 · 2015: 21 · 2014: 18 · 2013: 19 · 2012: 13
- FWCI
- 13.03
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Archives of Internal Medicine (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Digital Commons-TMC (Texas Medical Center) (submittedVersion)
- Funders
- National Institutes of Health
- National Cancer Institute (K23CA125585)
- Topics
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiology practices and education
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Health Professions › Medical Laboratory Technology › Clinical Laboratory Practices and Quality Control
- Keywords
- electronic health records, clinical decision support alerts, veterans' health care, trainee physicians, health care costs, quality of care, health care utilization
- MeSH terms
- Communication, Continuity of Patient Care, Diagnostic Imaging, Hospitals, Veterans, Humans, Outpatient Clinics, Hospital, Practice Patterns, Physicians', Radiology, Retrospective Studies, Time Factors, Medical Records Systems, Computerized, Reminder Systems, Process Assessment, Health Care
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
