Effectiveness of an Electronic Health Record-based Intervention to Improve Follow-up of Abnormal Pathology Results: A Retrospective Record Analysis
Open accessAbstract
BACKGROUND AND OBJECTIVE: On March 11, 2009, the Veterans Health Administration (VA) implemented an electronic health record (EHR)-based intervention that required all pathology results to be transmitted to ordering providers by mandatory automated notifications. We examined the impact of this intervention on improving follow-up of abnormal outpatient pathology results. RESEARCH DESIGN AND SUBJECTS: We extracted pathology reports from the EHR of 2 VA sites. From 16,738 preintervention and 17,305 postintervention reports between 09/01/2008 and 09/30/2009, we randomly selected about 5% and evaluated follow-up outcomes using a standardized chart review instrument. Documented responses to the alerted report (eg, ordering follow-up tests or referrals, notifying patients, and prescribing/changing treatment) were recorded. MEASURES: Primary outcome measures included proportion of timely follow-up responses (within 30 d) and median time to direct response for abnormal reports. RESULTS: Of 816 preintervention and 798 postintervention reports reviewed, 666 (81.6%) and 688 (86.2%) were abnormal. Overall, there was no apparent intervention effect on timely follow-up (69% vs. 67.1%; P=0.4) or median time to direct response (8 vs. 8 d; P=0.7). However, logistic regression uncovered a significant intervention effect (preintervention odds ratio, 0.7; 95% confidence interval, 0.5-1.0) after accounting for site-specific differences in follow-up, with a lower likelihood of timely follow-up at one site (odds ratio, 0.4; 95% confidence interval, 0.2-0.7). CONCLUSIONS: An electronic intervention to improve test result follow-up at 2 VA institutions using the same EHR was found effective only after accounting for certain local contextual factors. Aggregating the effect of EHR interventions across different institutions and EHRs without controlling for contextual factors might underestimate their potential benefits.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A sample size of 30 participants, each reviewing 35 test results, was required to achieve 80% power to detect a statistically significant difference in cognitive workload and performance.
“On the basis of the anticipated rate of appropriately identified abnormal test results in the literature12,13,14,15,16,17,18,19,20,21 and the anticipated magnitude of the association of the enhanced EHR, we required a sample size of 30 participants, each reviewing 35 test results, to achieve 80% power to detect a statistically significant difference in cognitive workload and performance.”
From Association of the Usability of Electronic Health Records With Cognitive Workload and Performance Levels Among Physicians (2019) · cites this paper as [16]
The Methods, ‘Study population and sample’ (page unavailable), describes an 80% power calculation, but it calls for 284 abnormal tests per group at each site to detect a difference in time to follow-up. It does not support the claimed 30 participants reviewing 35 results each or a power calculation for cognitive workload and performance. (LLM, conf=0.86)
Experimental — accuracy not yet measuredEven in the EHR environment, with alerts and reminders in place, physicians continue to often inappropriately manage abnormal test results.
“Even in the EHR environment, with alerts and reminders in place, physicians continue to often inappropriately manage abnormal test results.12,13,14,15,16,17,18,19,20,21”
From Association of the Usability of Electronic Health Records With Cognitive Workload and Performance Levels Among Physicians (2019) · cites this paper as [16]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Generalized Estimating Equations
Hardin, J. W., & Hilbe, J. M. (2002) · doi:10.1201/9781420035285 · Cited by 1,766 · see citing works
Sittig, D. F., & Singh, H. (2010) · Quality and Safety in Health Care · doi:10.1136/qshc.2010.042085 · Cited by 897 · see citing works
Self-citationMissed and delayed diagnoses in the ambulatory setting: a study of closed malpractice claims
Gandhi, T. K., Kachalia, A., Thomas, E. J., Puopolo, A. L., Yoon, C., Brennan, T. A., & Studdert, D. M. (2006) · Annals of Internal Medicine · doi:10.7326/0003-4819-145-7-200610030-00006 · Cited by 662 · see citing works
Electronic Health Record Use and the Quality of Ambulatory Care in the United States
Linder, J. A., Ma, J., Bates, D. W., Middleton, B., & Stafford, R. S. (2007) · Archives of Internal Medicine · doi:10.1001/archinte.167.13.1400 · Cited by 438 · see citing works
Electronic Health Records and Clinical Decision Support Systems
Romano, M. J., & Stafford, R. S. (2011) · Archives of Internal Medicine · doi:10.1001/archinternmed.2010.527 · Cited by 296 · see citing works
Sequist, T. D., Gandhi, T. K., Karson, A. S., Fiskio, J. M., Bugbee, D., Sperling, M., Cook, E. F., Orav, E. J., Fairchild, D. G., & Bates, D. W. (2005) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1788 · Cited by 250 · see citing works
Advancing the Science of Patient Safety
Shekelle, P., Pronovost, P. J., Wachter, R. M., Taylor, S. L., Dy, S. M., Foy, R., Hempel, S., McDonald, K. M., Øvretveit, J., Rubenstein, L. V., et al. (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-154-10-201105170-00011 · Cited by 213 · see citing works
Singh, H., Thomas, E. J., Mani, S., Sittig, D., Arora, H., Espadas, D., Khan, M. M., & Petersen, L. A. (2009) · Archives of Internal Medicine · doi:10.1001/archinternmed.2009.263 · Cited by 212 · see citing works
Self-citationThe Relationship between Electronic Health Record Use and Quality of Care over Time
Zhou, L., Soran, C. S., Jenter, C. A., Volk, L. A., Orav, E. J., Bates, D. W., & Simon, S. R. (2009) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m3128 · Cited by 194 · see citing works
Taylor, S. L., Dy, S. M., Foy, R., Hempel, S., McDonald, K. M., Øvretveit, J., Pronovost, P. J., Rubenstein, L. V., Wachter, R. M., & Shekelle, P. (2011) · BMJ Quality & Safety · doi:10.1136/bmjqs.2010.049379 · Cited by 177 · see citing works
Mixed Results In The Safety Performance Of Computerized Physician Order Entry
Metzger, J. B., Welebob, E. M., Bates, D. W., Lipsitz, S. R., & Classen, D. C. (2010) · Health Affairs · doi:10.1377/hlthaff.2010.0160 · Cited by 166 · see citing works
Singh, H., Thomas, E. J., Sittig, D. F., Wilson, L., Espadas, D., Khan, M. M., & Petersen, L. A. (2010) · The American Journal of Medicine · doi:10.1016/j.amjmed.2009.07.027 · Cited by 160 · see citing works
Self-citationFrequency of Failure to Inform Patients of Clinically Significant Outpatient Test Results
Casalino, L. P., Dunham, D., Chin, M. H., Bielang, R., Kistner, E. O., Karrison, T. G., Ong, M. K., Sarkar, U., McLaughlin, M. A., & Meltzer, D. O. (2009) · Archives of Internal Medicine · doi:10.1001/archinternmed.2009.130 · Cited by 159 · see citing works
Murphy, D. R., Reis, B., Sittig, D. F., & Singh, H. (2012) · The American Journal of Medicine · doi:10.1016/j.amjmed.2011.07.029 · Cited by 142 · see citing works
Self-citationCommunication outcomes of critical imaging results in a computerized notification system
Singh, H., Arora, H. S., Vij, M. S., Rao, R., Khan, M. M., & Petersen, L. A. (2007) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2280 · Cited by 140 · see citing works
Self-citationElectronic Health Record Components and the Quality of Care
Keyhani, S., Hebert, P. L., Ross, J. S., Federman, A. D., Zhu, C. W., & Siu, A. L. (2008) · Medical Care · doi:10.1097/mlr.0b013e31817e18ae · Cited by 92 · see citing works
Eight Recommendations for Policies for Communicating Abnormal Test Results
Singh, H., & Vij, M. S. (2010) · The Joint Commission Journal on Quality and Patient Safety · doi:10.1016/s1553-7250(10)36037-5 · Cited by 83 · see citing works
Self-citationUnderstanding the management of electronic test result notifications in the outpatient setting
Hysong, S. J., Sawhney, M. K., Wilson, L., Sittig, D. F., Esquivel, A., Singh, S., & Singh, H. (2011) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-11-22 · Cited by 68 · see citing works
Self-citationProvider management strategies of abnormal test result alerts: a cognitive task analysis
Hysong, S. J., Sawhney, M. K., Wilson, L., Sittig, D. F., Espadas, D., Davis, T., & Singh, H. (2010) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m3200 · Cited by 65 · cited 9× · see citing works
“For instance, certain specialists and trainees who rotate within the VA might not remotely access the EHR.”
BackgroundInfluentialSelf-citationFollow-up Actions on Electronic Referral Communication in a Multispecialty Outpatient Setting
Singh, H., Esquivel, A., Sittig, D. F., Murphy, D., Kadiyala, H., Schiesser, R., Espadas, D., & Petersen, L. A. (2011) · Journal of General Internal Medicine · doi:10.1007/s11606-010-1501-z · Cited by 57 · see citing works
Self-citationMurphy, D. R. (2012) · Archives of Internal Medicine · doi:10.1001/archinternmed.2011.740 · Cited by 52 · see citing works
Wahls, T. L., Haugen, T. H., & Cram, P. M. (2007) · The Joint Commission Journal on Quality and Patient Safety · doi:10.1016/s1553-7250(07)33052-3 · Cited by 46 · see citing works
(2010) · PubMed · Cited by 38 · see citing works
DesRoches, C. M., Agarwal, R., Angst, C. M., & Fischer, M. A. (2010) · Health Affairs · doi:10.1377/hlthaff.2010.0557 · Cited by 34 · see citing works
Improving outpatient safety through effective electronic communication: a study protocol
Hysong, S. J., Sawhney, M. K., Wilson, L., Sittig, D. F., Esquivel, A., Watford, M., Giardina, T. D., Espadas, D., & Singh, H. (2009) · Implementation Science · doi:10.1186/1748-5908-4-62 · Cited by 34 · see citing works
Self-citationMeasuring Quality in Anatomic Pathology
Raab, S. S., & Grzybicki, D. M. (2008) · Clinics in Laboratory Medicine · doi:10.1016/j.cll.2007.12.004 · Cited by 23 · see citing works
Clinical Decision Support and Rich Clinical Repositories: A Symbiotic Relationship
McDonald, C. J., & Abhyankar, S. (2011) · Archives of Internal Medicine · doi:10.1001/archinternmed.2010.518 · Cited by 8 · see citing works
Selvan, M. S., Sittig, D. F., Thomas, E. J., Arnold, C. C., MURPHY, R. E., & Shabot, M. M. (2011) · Journal of Patient Safety · doi:10.1097/pts.0b013e318223cb54 · Cited by 5 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Advancing the science of measurement of diagnostic errors in healthcare: the Safer Dx framework
Singh, H., & Sittig, D. F. (2015) · BMJ Quality & Safety · doi:10.1136/bmjqs-2014-003675 · Cited by 201 · see citing works
Self-citationPatient portals and health apps: Pitfalls, promises, and what one might learn from the other
Baldwin, J. L., Singh, H., Sittig, D. F., & Giardina, T. D. (2016) · Healthcare · doi:10.1016/j.hjdsi.2016.08.004 · Cited by 179 · see citing works
Self-citationMazur, L. M., Mosaly, P. R., Moore, C., & Marks, L. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.1709 · Cited by 127 · see citing works
“Perceived and Physiological Workload No statistically significant difference was noted in perceived workload between the baseline EHR and enhanced EHR groups (mean [SD] NASA-TLX score, 53 [14] vs 49 [16]; composite score, 4 [95% CI, –5 to 13]; P = .”
BackgroundIssaka, R. B., Singh, M. H., Oshima, S. M., Laleau, V., Rachocki, C., Chen, E. H., Day, L. W., Sarkar, U., & Somsouk, M. (2017) · The American Journal of Gastroenterology · doi:10.1038/ajg.2016.555 · Cited by 93 · see citing works
Singh, H., & Sittig, D. F. (2020) · Annals of Internal Medicine · doi:10.7326/m19-0879 · Cited by 77 · see citing works
Self-citationHow context affects electronic health record-based test result follow-up: a mixed-methods evaluation
Menon, S., Smith, M. W., Sittig, D. F., Petersen, N. J., Hysong, S. J., Espadas, D., Modi, V., & Singh, H. (2014) · BMJ Open · doi:10.1136/bmjopen-2014-005985 · Cited by 45 · see citing works
Self-citationGeorgiou, A., Li, J., Thomas, J., Dahm, M. R., & Westbrook, J. I. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz032 · Cited by 40 · see citing works
Jung, S. Y., Hwang, H. Y., Lee, K., Lee, D., Yoo, S., Lim, K., Lee, H., & Kim, E. (2021) · JMIR Formative Research · doi:10.2196/18764 · Cited by 35 · see citing works
Laboratory testing in primary care: A systematic review of health IT impacts
Maillet, É., Paré, G., Currie, L. M., Raymond, L., de Guinea, A. O., Trudel, M.-C., & Marsan, J. (2018) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2018.05.009 · Cited by 30 · see citing works
Patient Preferences for Test Result Notification
Shultz, S. K., Wu, R. C., Matelski, J. J., Lu, X., & Cram, P. M. (2015) · Journal of General Internal Medicine · doi:10.1007/s11606-015-3344-0 · Cited by 28 · see citing works
Using Natural Language Processing to Extract Abnormal Results From Cancer Screening Reports
Moore, C., Farrag, A., & Ashkin, E. (2014) · Journal of Patient Safety · doi:10.1097/pts.0000000000000127 · Cited by 27 · see citing works
Patient facing decision support system for interpretation of laboratory test results
Kopanitsa, G., & Semenov, I. (2018) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-018-0648-0 · Cited by 11 · see citing works
Meyer, A. N. D., Scott, T. M. T., & Singh, H. (2022) · JAMA Network Open · doi:10.1001/jamanetworkopen.2022.8568 · Cited by 9 · see citing works
Self-citationBuilding a Quality Management System for a Thyroid Nodule Clinic
Migdal, A. L., Sternberg, S. B., Oshin, A., Aronson, M. D., & Hennessey, J. V. (2016) · Thyroid · doi:10.1089/thy.2015.0674 · Cited by 8 · see citing works
Disparities in brain health comorbidity management in intracerebral hemorrhage
Mayerhofer, E., Zaba, N. O., Parodi, L., Ganbold, A., Biffi, A., Rosand, J. M., Yechoor, N., & Anderson, C. D. (2023) · Frontiers in Neurology · doi:10.3389/fneur.2023.1194810 · Cited by 7 · see citing works
Impact of Immediate Access to the Electronic Medical Record on Anatomic Pathology Performance
Renshaw, A. A., & Gould, E. W. (2013) · American Journal of Clinical Pathology · doi:10.1309/ajcp1zqa0ndmvdpe · Cited by 7 · see citing works
The Consequences of Electronic Health Record Adoption for Physician Productivity and Birth Outcomes
Meyerhoefer, C. D., Deily, M. E., Sherer, S. A., Chou, S., Peng, L., Sheinberg, M., & Levick, D. (2016) · Industrial and Labor Relations Review · doi:10.1177/0019793916642758 · Cited by 7 · see citing works
Sittig, D. F., & Singh, H. (2014) · Journal of Comparative Effectiveness Research · doi:10.2217/cer.14.46 · Cited by 6 · see citing works
Self-citationThomas, J., Dahm, M. R., Li, J., Westbrook, J. I., & Georgiou, A. (2019) · Health Informatics Journal · doi:10.1177/1460458219889223 · Cited by 4 · see citing works
Petroll, A. E., Phelps, J. K., & Fletcher, K. E. (2014) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2013.12.005 · Cited by 2 · see citing works
Wright, A. T., Maloney, F. L., Wien, M. F., Samal, L., Emani, S., & Zuccotti, G. (2015) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv041 · Cited by 2 · see citing works
Technology-Based Closed-Loop Tracking for Improving Communication and Follow-up of Pathology Results
Rajan, S. S., Baldwin, J. L., Giardina, T. D., & Singh, H. (2020) · Journal of Patient Safety · doi:10.1097/pts.0000000000000759 · Cited by 1 · see citing works
Self-citationMeyerhoefer, C. D. (2014) · Health & Healthcare in America: From Economics to Policy
Spiewak, T. A., Yan, S. L., & Tejaswi, S. (2020) · The Joint Commission Journal on Quality and Patient Safety · doi:10.1016/j.jcjq.2020.03.004
Assessing the impact of health information technology on community health administration
Alotaibi, B. N., Alshamri, A. S., Alanazi, M. A., & Alharbi, M. S. S. (2019) · International Journal of Health Sciences · doi:10.53730/ijhs.v3ns1.15277
Liao, J., Wu, C., Liu, Y., & Zhan, Y. (2026) · Healthcare · doi:10.3390/healthcare14213586
- Authors
- Laxmisan, A., Sittig, D. F., Pietz, K., Espadas, D., Krishnan, B., & Singh, H.
- Affiliations
- Michael E. DeBakey VA Medical Center (US)
- Memorial Hermann (US)
- Baylor College of Medicine (US)
- Corresponding author
- Laxmisan, Archana
- Pietz, Kenneth
- Espadas, Donna
- Singh, Hardeep
- Venue
- Medical Care
- Published
- 2012-10
- Volume
- 50
- Issue
- 10
- Pages
- 898-904
- ISSN
- 0025-7079 (ISSN-L)
- PMC
- PMC3444625
- OpenAlex
- W2035136345
- Cited by
- 26
- References
- 28
- Citations by year
- 2026: 1 · 2023: 1 · 2022: 1 · 2021: 1 · 2020: 3 · 2019: 4 · 2018: 2 · 2017: 1 · 2016: 3 · 2015: 3 · 2014: 5 · 2013: 1
- FWCI
- 4.68
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Medical Care (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Cancer Institute (K23CA125585)
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Medicine › Physiology › Clinical Laboratory Practices and Quality Control
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Keywords
- electronic health records, Veterans Health Administration, contextual factors
- MeSH terms
- Continuity of Patient Care, Follow-Up Studies, Humans, Pathology, Retrospective Studies, United States, United States Department of Veterans Affairs, Reminder Systems, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
