Measuring Harm in Health Care: Optimizing Adverse Event Review
Open accessThe DOI matches its Crossref record; 20 of the 21 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Objective: The objective of this study was to identify modifiable factors that improve the reliability of ratings of severity of health care–associated harm in clinical practice improvement and research. Methods: A diverse group of clinicians rated 8 types of adverse events: blood product, device or medical/surgical supply, fall, health care–associated infection, medication, perinatal, pressure ulcer, surgery. We used a generalizability theory framework to estimate the impact of number of raters, rater experience, and rater provider type on reliability. Results: Pharmacists were slightly more precise and consistent in their ratings than either physicians or nurses. For example, to achieve high reliability of 0.83, 3 physicians could be replaced by 2 pharmacists without loss in precision of measurement. If only 1 rater was available for rating, ∼5% of the reviews for severe harm would have been incorrectly categorized. Reliability was greatly improved with 2 reviewers. Conclusions: We identified factors that influence the reliability of clinician reviews of health care–associated harm. Our novel use of generalizability analyses improved our understanding of how differences affect reliability. This approach was useful in optimizing resource utilization when selecting raters to assess harm and may have similar applications in other settings in health care.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“inspired by the AHRQ harm scale43.”
From Adapted large language models can outperform medical experts in clinical text summarization (2024) · cites this paper as [43]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Harris, P. A., Taylor, R., Thielke, R., Payne, J., Gonzalez, N., & Conde, J. G. (2009) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2008.08.010 · Cited by 51,384
Incidence of Adverse Drug Events and Potential Adverse Drug Events: Implications for Prevention
Bates, D. W. (1995) · JAMA · doi:10.1001/jama.1995.03530010043033 · Cited by 2,127
Incidence and Preventability of Adverse Drug Events Among Older Persons in the Ambulatory Setting
Gurwitz, J. H., Field, T. S., Harrold, L. R., Rothschild, J., Debellis, K., Seger, A. C., Cadoret, C., Fish, L. S., Garber, L., Kelleher, M., et al. (2003) · JAMA · doi:10.1001/jama.289.9.1107 · Cited by 1,676
Interprofessional teamwork: Professional cultures as barriers
Hall, P. (2005) · Journal of Interprofessional Care · doi:10.1080/13561820500081745 · Cited by 1,392
Temporal Trends in Rates of Patient Harm Resulting from Medical Care
Landrigan, C. P., Parry, G. J., Bones, C. B., Hackbarth, A. D., Goldmann, D. A., & Sharek, P. J. (2010) · New England Journal of Medicine · doi:10.1056/nejmsa1004404 · Cited by 1,144
Incidence and preventability of adverse drug events in nursing homes
Gurwitz, J. H., Field, T. S., Avorn, J., McCormick, D., Jain, S., Eckler, M., Benser, M., Edmondson, A. C., & Bates, D. W. (2000) · The American Journal of Medicine · doi:10.1016/s0002-9343(00)00451-4 · Cited by 640
Incidence of adverse drug events and potential adverse drug events
Bates, D. W., Cullen, D. J., & Laird, N. (1995) · International Journal of Trauma Nursing · doi:10.1016/s1075-4210(05)80011-2 · Cited by 534
The incidence of adverse drug events in two large academic long-term care facilities
Gurwitz, J. H., Field, T. S., Judge, J., Rochon, P., Harrold, L. R., Cadoret, C., Lee, M., White, K., LaPrino, J., Erramuspe-Mainard, J., et al. (2005) · The American Journal of Medicine · doi:10.1016/j.amjmed.2004.09.018 · Cited by 404
Reducing barriers to interprofessional training: Promoting interprofessional cultural competence
Pecukonis, E., Doyle, O., & Bliss, D. L. (2008) · Journal of Interprofessional Care · doi:10.1080/13561820802190442 · Cited by 257
A validated, reliable method of scoring the severity of medication errors
Dean, B. S., & Barber, N. D. (1999) · American Journal of Health-System Pharmacy · doi:10.1093/ajhp/56.1.57 · Cited by 183
Research works printed but not linked: 1
- Griffin, 2009, IHI Global Trigger Tool for measuring adverse events, IHI Innovation Series White Paper
Cited by
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Adapted large language models can outperform medical experts in clinical text summarization
Van Veen, D., Van Uden, C., Blankemeier, L., Delbrouck, J.-B., Aali, A., Bluethgen, C., Pareek, A., Polacin, M., Reis, E. P., Seehofnerová, A., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02855-5 · Cited by 891
“We then conducted a supplemental reader study connecting summarization errors to medical harm, inspired by the Agency for Healthcare Research and Quality (AHRQ)’s harm scale 43 .”
InfluentialPrevalence of Medication Errors Among Paediatric Inpatients: Systematic Review and Meta-Analysis
Gates, P., Baysari, M. T., Gazarian, M., Raban, M. Z., Meyerson, S. A., & Westbrook, J. I. (2019) · Drug Safety · doi:10.1007/s40264-019-00850-1 · Cited by 89
Gates, P., Baysari, M. T., Mumford, V., Raban, M. Z., & Westbrook, J. I. (2019) · Drug Safety · doi:10.1007/s40264-019-00823-4 · Cited by 88
Metadata
- Authors
- Walsh, K. E., Harik, P., Mazor, K. M., Perfetto, D., Anatchkova, M., Biggins, C., Wagner, J., Schoettker, P. J., Firneno, C., Klugman, R., & Tjia, J.
- Affiliations
- Cincinnati Children's Hospital Medical Center (US)
- National Board of Medical Examiners (US)
- Meyers Primary Care Institute, Worcester, MA
- Agency for Healthcare Research and Quality (US)
- University of Massachusetts Chan Medical School (US)
- James M. Anderson Center
- Kindred Healthcare, Wellesley, MA
- Corresponding author
- Walsh, Kathleen E.
- Schoettker, Pamela J.
- ORCID
- Venue
- Medical Care
- Published
- 2017-04
- Volume
- 55
- Issue
- 4
- Pages
- 436-441
- ISSN
- 0025-7079 (ISSN-L)
- PMC
- PMC5352561
- OpenAlex
- W2558323601
- Cited by
- 20
- References
- 21
- Citations by year
- 2026: 2 · 2025: 2 · 2024: 7 · 2023: 2 · 2021: 1 · 2020: 3 · 2019: 2 · 2018: 1
- FWCI
- 1.72
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Medical Care (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- eScholarship@UMassChan (submittedVersion)
- Funders
- Agency for Healthcare Research and Quality (HHSA290201000022I)
- Topics
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Patient Safety and Medication Errors
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Healthcare Quality and Management
- Keywords
- adverse events, severity rating, inter-rater reliability, generalizability theory, rater experience, clinical practice improvement, nosocomial infections, pressure ulcers, falls, medication errors
- MeSH terms
- Attitude of Health Personnel, Humans, Iatrogenic Disease, Peer Review, Prospective Studies, United States, Reproducibility of Results, Medical Errors, Harm Reduction, Drug-Related Side Effects and Adverse Reactions
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Measuring Harm in Health Care: Optimizing Adverse Event Review. https://app.cochat.ai/papers/10.1097/mlr.0000000000000679
- The paper
- Walsh, K. E., Harik, P., Mazor, K. M., Perfetto, D., Anatchkova, M., Biggins, C., Wagner, J., Schoettker, P. J., Firneno, C., Klugman, R., & Tjia, J. (2017). Measuring Harm in Health Care: Optimizing Adverse Event Review. Medical Care. https://doi.org/10.1097/mlr.0000000000000679
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.