The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index; none of the 7 sources its claims cite is checked for support yet.
Does integrating nonurgent, clinically significant radiology alerts within the electronic health record impact closed-loop communication and follow-up?
Open accessAbstract
Objective To assess whether integrating critical result management software—Alert Notification of Critical Results (ANCR)—with an electronic health record (EHR)-based results management application impacts closed-loop communication and follow-up of nonurgent, clinically significant radiology results by primary care providers (PCPs). Materials and Methods This institutional review board-approved study was conducted at a large academic medical center. Postintervention, PCPs could acknowledge nonurgent, clinically significant ANCR-generated alerts (“alerts”) within ANCR or the EHR. Primary outcome was the proportion of alerts acknowledged via EHR over a 24-month postintervention. Chart abstractions for a random sample of alerts 12 months preintervention and 24 months postintervention were reviewed, and the follow-up rate of actionable alerts (eg, performing follow-up imaging, administering antibiotics) was estimated. Pre- and postintervention rates were compared using the Fisher exact test. Postintervention follow-up rate was compared for EHR-acknowledged alerts vs ANCR. Results Five thousand nine hundred and thirty-one alerts were acknowledged by 171 PCPs, with 100% acknowledgement (consistent with expected ANCR functionality). PCPs acknowledged 16% (688 of 4428) of postintervention alerts in the EHR, with the remaining in ANCR. Follow-up was documented for 85 of 90 (94%; 95% CI, 88%-98%) preintervention and 79 of 84 (94%; 95% CI, 87%-97%) postintervention alerts ( P > .99). Postintervention, 11 of 14 (79%; 95% CI, 52%-92%) alerts were acknowledged via EHR and 68 of 70 (97%; 95% CI, 90%-99%) in ANCR had follow-up ( P = .03). Conclusions Integrating ANCR and EHR provides an additional workflow for acknowledging nonurgent, clinically significant results without significant change in rates of closed-loop communication or follow-up of alerts.
Overview
CoChat Summary
TL;DR
Integrating ANCR-generated nonurgent, clinically significant radiology alerts into an EHR-based results management application gave primary care providers an additional acknowledgement workflow, but most alerts continued to be acknowledged in ANCR. The integration did not significantly change closed-loop communication or follow-up rates overall, although postintervention EHR-acknowledged alerts had lower follow-up than ANCR-acknowledged alerts.
Key findings
- Of 5931 Level 3 alerts acknowledged by 171 PCPs, postintervention PCPs acknowledged 16% (688 of 4428) in the EHR and 84% (3740 of 4428) in ANCR; preintervention, 100% (1503 of 1503) were acknowledged in ANCR and 0% in the EHR. Overall acknowledgement was 100%.
- Follow-up was documented for 94% of actionable alerts preintervention (85 of 90; 95% CI, 88%-98%) and 94% postintervention (79 of 84; 95% CI, 87%-97%; P > .99). The proportion of alerts judged actionable was 90% preintervention vs 84% postintervention (P = .29).
- Postintervention, follow-up was 79% for actionable alerts acknowledged via EHR (11 of 14; 95% CI, 52%-92%) vs 97% for those acknowledged in ANCR (68 of 70; 95% CI, 90%-99%; P = .03).
- Median time to acknowledgement was 0.6 hours preintervention and 0.6 hours postintervention (P = .003); postintervention, median time was 7 hours in the EHR vs 0.3 hours in ANCR (P < .001). Month-to-month variation in EHR acknowledgement was not statistically significant; median per-PCP EHR acknowledgement was 4% (IQR, 0-15).
Methods
- IRB-approved pre-post study at a 793-bed tertiary academic medical center with a radiology department performing over 600 000 examinations annually. Included nonurgent, clinically significant Level 3 ANCR alerts for 171 PCPs at 13 affiliated outpatient practices during a 12-month preintervention period (May 2011-April 2012) and 24-month postintervention period (June 2012-May 2014); excluded Level 1/2 alerts, inpatient/ED alerts, and non-PCP providers.
- Primary outcome was the proportion of Level 3 ANCR-generated alerts acknowledged in the EHR. Secondary outcome was follow-up rate of actionable alerts, defined as the proportion acted upon appropriately; a random sample of 200 acknowledged alerts (100 preintervention, 100 postintervention) was chart-abstracted and categorized by physicians with consensus review.
- Pre/post follow-up rates were compared using the Fisher exact test; the Cochran-Armitage trend test evaluated trends across 3 years; the Wilcoxon rank sum test compared time to acknowledgement; a P-type statistical process control chart analyzed system adoption over time.
Limitations
- Stated: the study was performed at 1 academic center with a single combination of critical radiology results software and EHR-based results management application.
- Stated: there was wide variation in the proportion of ANCR-generated alerts each PCP acknowledged in the EHR; because clustering was minimal, PCP was not included as a covariate. Some variation may have been due to lack of training, and adoption may have improved with additional training.
- Stated: the pre-postintervention design, rather than a randomized trial, cannot control for changes in follow-up due to factors other than the intervention; the authors note no other interventions targeting follow-up of nonurgent clinically significant radiology findings were introduced during the study period. Evident from the text: the postintervention EHR-acknowledged actionable alert subgroup was small (14 alerts), producing a wide confidence interval (52%-92%) and limited precision.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 0 Not valid · 5 Not checked · 25 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“PCPs acknowledged 16% (688 of 4428) of postintervention alerts in the EHR, with the remaining in ANCR.”
- Also stated in: § System Adoption · plain
“The EHR was used to acknowledge 15.5% (688 of 4428) of alerts generated in ANCR in the postintervention period (table 3).”
§ Does integrating nonurgent, clinically significant radiology alerts within the electronic health record impact closed-loop communication and follow-up? > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § System Adoption · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The 5931 Level 3 (nonurgent, clinically significant) alerts, comprising 92% of alerts generated for PCPs on outpatient examinations, formed the study cohort (figure 1).”
§ Does integrating nonurgent, clinically significant radiology alerts within the electronic health record impact closed-loop communication and follow-up? > Study Cohort
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The median time to acknowledgement was 0.6 hours (IQR 0.03-20.5 h) preintervention and 0.6 hours (IQR 0.03-6.9h) postintervention (P ¼ .003).”
§ Does integrating nonurgent, clinically significant radiology alerts within the electronic health record impact closed-loop communication and follow-up? > TIME TO ACKNOWLEDGEMENT
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Five thousand nine hundred and thirty-one alerts were acknowledged by 171 PCPs, with 100% acknowledgement (consistent with expected ANCR functionality).”
- Also stated in: § Study Cohort · plain
“The 5931 Level 3 (nonurgent, clinically significant) alerts, comprising 92% of alerts generated for PCPs on outpatient examinations, formed the study cohort (figure 1).”
§ Does integrating nonurgent, clinically significant radiology alerts within the electronic health record impact closed-loop communication and follow-up? > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Study Cohort · plain
In the paper
“A statistical process control chart (figure 3) demonstrated that the month-to-month variation in this proportion was not statistically significant.14”
§ Does integrating nonurgent, clinically significant radiology alerts within the electronic health record impact closed-loop communication and follow-up? > System Adoption
What the sources say
- Not checked yetcites [14] Benneyan, 2003, Statistical process control as a tool for research and healthcare improvement, Qual Saf Health Care source
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
10 more claims
Create a free account to see all 30 claims and to verify the 7 references still open.
- It may state two separate claims as one. 9
- Probably not a substantial claim. 6
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Does integrating nonurgent, clinically significant radiology alerts within the electronic health record impact closed-loop communication and follow-up? > DISCUSSION
“Nevertheless, the follow-up rate for actionable alerts acknowledged via the EHR and ANCR were still similar.”
§ Does integrating nonurgent, clinically significant radiology alerts within the electronic health record impact closed-loop communication and follow-up? > Alert Follow-up
“In the postintervention period, PCPs acted on 79% (11 of 14; 95% CI, 52%-92%) of actionable alerts acknowledged in the EHR and 97% (68 of 70; 95% CI, 90%-99%) of actionable alerts acknowledged in ANCR (P ¼ .03).”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“In a review of 50 random alerts in the preintervention cohort by 1 radiologist (SDO), 47 (94%) acknowledged alerts were actionable and 44 (89%) actionable alerts were actioned.”
§ Does integrating nonurgent, clinically significant radiology alerts within the electronic health record impact closed-loop communication and follow-up? > Alert Follow-up (Actionable and Actioned Alerts)
Reported 89 · recomputed 93.617021
Citations
Cites
Statistical process control as a tool for research and healthcare improvement
Benneyan, J. C. (2003) · Quality and Safety in Health Care · doi:10.1136/qhc.12.6.458 · Cited by 1,045
Blumenthal, D. (2010) · New England Journal of Medicine · doi:10.1056/nejmp0912825 · Cited by 714
Hsiao, C., & Hing, E. (2014) · PubMed · Cited by 285
Singh, H., Thomas, E. J., Mani, S., Sittig, D., Arora, H., Espadas, D., Khan, M. M., & Petersen, L. A. (2009) · Archives of Internal Medicine · doi:10.1001/archinternmed.2009.263 · Cited by 212 · cited 1×
“The followup rate we observed is higher than in other studies of abnormal findings(15,16) and follow-up recommendations(17,18) in radiology reports, but it is consistent with other critical result alert systems.(8,7) Additionally, our findings confirm that results management applications work best when designed to sup…”
ResultCommunication factors in the follow-up of abnormal mammograms
Poon, E. G., Haas, J. S., Puopolo, A. L., Gandhi, T. K., Burdick, E., Bates, D. W., & Brennan, T. A. (2004) · Journal of General Internal Medicine · doi:10.1111/j.1525-1497.2004.30357.x · Cited by 138
Hsiao, C., & Hing, E. (2012) · PubMed · Cited by 135
Design and implementation of a comprehensive outpatient Results Manager
Poon, E. G., Wang, S. J., Gandhi, T. K., Bates, D. W., & Kuperman, G. J. (2003) · Journal of Biomedical Informatics · doi:10.1016/s1532-0464(03)00061-3 · Cited by 111 · cited 2×
“Results Manager, Electronic Health Record The results management application developed by our enterprise information systems group to facilitate laboratory test results tracking is accessible within our web-based EHR and has also been previously described.(11) Providers are only required to acknowledge results with th…”
BackgroundFollow-up of Incidental Pulmonary Nodules and the Radiology Report
Blagev, D. P., Lloyd, J. F., Conner, K., Dickerson, J., Adams, D., Stevens, S. M., Woller, S. C., Evans, R. S., & Elliott, C. G. (2014) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2013.08.003 · Cited by 100
Lacson, R. C., Prevedello, L. M. S., Andriole, K. P., O’Connor, S. D., Roy, C. L., Gandhi, T. K., Dalal, A. K., Sato, L., & Khorasani, R. (2014) · American Journal of Roentgenology · doi:10.2214/ajr.14.13064 · Cited by 66
Self-citationChoksi, V. R., Marn, C. S., Bell, Y., & Carlos, R. C. (2006) · American Journal of Roentgenology · doi:10.2214/ajr.04.1913 · Cited by 53
Research works printed but not linked: 1
- Charles, Adoption of Electronic Health Record Systems Among U.S. Non-federal Acute Care Hospitals: 2008-2013
Cited by
Derevianko, A., Pizzoli, S. F. M., Pesapane, F., Rotili, A., Monzani, D., Grasso, R., Cassano, E., & Pravettoni, G. (2023) · Cancers · doi:10.3390/cancers15020470 · Cited by 81
Workflow Applications of Artificial Intelligence in Radiology and an Overview of Available Tools
Kapoor, N., Lacson, R. C., & Khorasani, R. (2020) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2020.08.016 · Cited by 78
Self-citationDetermining Adherence to Follow-up Imaging Recommendations
Mabotuwana, T., Hombal, V., Dalal, S., Hall, C. S., & Gunn, M. L. D. (2018) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2017.11.022 · Cited by 68
Artificial intelligence in cardiac computed tomography
Aromiwura, A. A., Settle, T., Umer, M., Joshi, J. K., Shotwell, M., Mattumpuram, J., Vorla, M., Sztukowska, M. N., Contractor, S. G., Amini, A. A., et al. (2023) · Progress in Cardiovascular Diseases · doi:10.1016/j.pcad.2023.09.001 · Cited by 33
Hossain, R., Wu, C. C., de Groot, P. M., Carter, B. W., Gilman, M. D., & Abbott, G. F. (2018) · Radiologic Clinics of North America · doi:10.1016/j.rcl.2018.01.004 · Cited by 32
Cook, T. S., Lalevic, D., Sloan, C. E., Chadalavada, S. C., Langlotz, C. P., Schnall, M. D., & Zafar, H. M. (2017) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2017.01.024 · Cited by 29
Assigning responsibility to close the loop on radiology test results
Kwan, J. L., & Singh, H. (2017) · Diagnosis · doi:10.1515/dx-2017-0019 · Cited by 26
Amroze, A., Field, T. S., Fouayzi, H., Sundaresan, D., Burns, L., Garber, L., Sadasivam, R. S., Mazor, K. M., Gurwitz, J. H., & Cutrona, S. L. (2019) · JMIR Medical Informatics · doi:10.2196/12650 · Cited by 25
Kadom, N., Venkatesh, A. K., Shugarman, S. A., Burleson, J., Moore, C. L., & Seidenwurm, D. J. (2022) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2022.03.017 · Cited by 23
Jabin, M. S. R., Schultz, T., Mandel, C., Bessen, T., Hibbert, P. D., Wiles, L., & Runciman, W. B. (2020) · Journal of Patient Safety · doi:10.1097/pts.0000000000000709 · Cited by 20
Sahni, V. A., & Khorasani, R. (2016) · Abdominal Radiology · doi:10.1007/s00261-016-0679-x · Cited by 16
Self-citationBužančić, I., Koh, H. J. W., Trin, C., Nash, C., Hadžiabdić, M. O., Belec, D., Zoungas, S., Zomer, E., Dalli, L. L., Ademi, Z., et al. (2024) · American Journal of Preventive Cardiology · doi:10.1016/j.ajpc.2024.100855 · Cited by 15
Guenette, J. P., Kapoor, N., Lacson, R. C., Lynch, E. A., Abbasi, N., Desai, S. P., Eappen, S., & Khorasani, R. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.6178 · Cited by 12
Self-citationThe need for closed-loop systems for management of abnormal test results
Zuccotti, G., Samal, L., Maloney, F. L., Ai, A., & Wright, A. (2018) · Annals of Internal Medicine · doi:10.7326/m17-2425 · Cited by 10
Santo, E. C., Dunbar, P. J., Sloan, C. E., Cook, T. S., Lalevic, D., & Zafar, H. M. (2016) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2016.06.016 · Cited by 10
Dibble, E. H., Swenson, D. W., Cobb, C., Paul, T. J., Karn, A. E., Portelli, D. C., & Movson, J. S. (2016) · Emergency Radiology · doi:10.1007/s10140-016-1452-8 · Cited by 9
Ahluwalia, V. S., Schapira, M. M., Weissman, G. E., & Parikh, R. B. (2025) · MDM Policy & Practice · doi:10.1177/23814683251329007 · Cited by 7
Semiautomated System for Nonurgent, Clinically Significant Pathology Results
O’Connor, S. D., Khorasani, R., Pochebit, S. M., Lacson, R. C., Andriole, K. P., & Dalal, A. K. (2018) · Applied Clinical Informatics · doi:10.1055/s-0038-1654700 · Cited by 7
Self-citationLicurse, M. Y., Lalevic, D., Zafar, H. M., Schnall, M. D., & Cook, T. S. (2016) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-016-9912-y · Cited by 5
Roth, B. J., Kampalath, R., Nakashima, K., Shieh, S., Bui, T., & Houshyar, R. (2023) · Current Problems in Diagnostic Radiology · doi:10.1067/j.cpradiol.2023.05.008 · Cited by 4
O’Connor, S. D., Alkasab, T. K., Samuel, J. K. R., & Sippo, D. A. (2025) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2025.10.019 · Cited by 1
Self-citationUrgent and unexpected findings in oncology and hematology patients: A practical approach to imaging
Viteri, A., Fernández, S., Persona, E. P., & de Celis, R. P. (2022) · Radiología (English Edition) · doi:10.1016/j.rxeng.2022.09.001 · Cited by 1
Quality Metrics: Definition, Creation, Presentation, and Use
Laroya, R., & Khorasani, R. (2018) · Medical radiology · doi:10.1007/174_2017_163
Self-citationViteri, A., Fernández, S., Persona, E. P., & de Celis, R. P. (2021) · Radiología · doi:10.1016/j.rx.2021.03.007
Oliveira, M., Gurgel, L., de Medeiros, M. C. S., Magalhães, A. G., Rodrigues, A. G. C. D. R., & Micussi, M. T. A. B. C. (2025) · doi:10.2196/preprints.78594
Metadata
- Authors
- O’Connor, S. D., Dalal, A. K., Sahni, V. A., Lacson, R., & Khorasani, R.
- Affiliations
- Brigham and Women's Hospital (US)
- Center for Evidence Based Imaging, Brookline, Massachusetts, USA
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2015-09-02
- Volume
- 23
- Issue
- 2
- Pages
- 333-338
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC5009922
- OpenAlex
- W1920681141
- Cited by
- 25
- References
- 21
- Citations by year
- 2025: 3 · 2024: 1 · 2023: 4 · 2022: 2 · 2021: 1 · 2020: 2 · 2019: 1 · 2018: 5 · 2017: 2 · 2016: 4
- FWCI
- 4.52
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Journal of the American Medical Informatics Association (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion)
- HighWire Press Open Archive (submittedVersion)
- Funders
- U.S. National Library of Medicine (T15LM007092)
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiology practices and education
- Health Sciences › Health Professions › Medical Laboratory Technology › Clinical Laboratory Practices and Quality Control
- Keywords
- electronic health record integration, closed-loop communication, primary care providers, clinical workflow
- MeSH terms
- Academic Medical Centers, Aftercare, Continuity of Patient Care, Humans, Monitoring, Physiologic, Radiology, Software, Reminder Systems, Systems Integration, Electronic Health Records, Physicians, Primary Care, Teach-Back Communication
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-17, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-17 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.