Methods to Achieve High Interrater Reliability in Data Collection From Primary Care Medical Records
Open accessThe DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Abstract
PURPOSE: We assessed interrater reliability (IRR) of chart abstractors within a randomized trial of cardiovascular care in primary care. We report our findings, and outline issues and provide recommendations related to determining sample size, frequency of verification, and minimum thresholds for 2 measures of IRR: the κ statistic and percent agreement. METHODS: We designed a data quality monitoring procedure having 4 parts: use of standardized protocols and forms, extensive training, continuous monitoring of IRR, and a quality improvement feedback mechanism. Four abstractors checked a 5% sample of charts at 3 time points for a predefined set of indicators of the quality of care. We set our quality threshold for IRR at a κ of 0.75, a percent agreement of 95%, or both. RESULTS: Abstractors reabstracted a sample of charts in 16 of 27 primary care practices, checking a total of 132 charts with 38 indicators per chart. The overall κ across all items was 0.91 (95% confidence interval, 0.90-0.92) and the overall percent agreement was 94.3%, signifying excellent agreement between abstractors. We gave feedback to the abstractors to highlight items that had a κ of less than 0.70 or a percent agreement less than 95%. No practice had to have its charts abstracted again because of poor quality. CONCLUSIONS: A 5% sampling of charts for quality control using IRR analysis yielded κ and agreement levels that met or exceeded our quality thresholds. Using 3 time points during the chart audit phase allows for early quality control as well as ongoing quality monitoring. Our results can be used as a guide and benchmark for other medical chart review studies in primary care.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The kappa statistic in reliability studies: use, interpretation, and sample size requirements
Sim, J., & Wright, C. C. (2005) · Physical Therapy · doi:10.1093/ptj/85.3.257 · Cited by 4,093
High agreement but low Kappa: I. the problems of two paradoxes
Feinstein, A. R., & Cicchetti, D. V. (1990) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(90)90158-l · Cited by 3,034
Byrt, T., Bishop, J., & Carlin, J. B. (1993) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(93)90018-v · Cited by 2,179
Chart reviews in emergency medicine research: Where are the methods?
Gilbert, E. H., Lowenstein, S. R., Koziol-McLain, J., Barta, D. C., & Steiner, J. (1996) · Annals of Emergency Medicine · doi:10.1016/s0196-0644(96)70264-0 · Cited by 1,165
The stepped wedge trial design: a systematic review
Brown, C. A., & Lilford, R. J. (2006) · BMC Medical Research Methodology · doi:10.1186/1471-2288-6-54 · Cited by 907
Cited by
Looking Through the Retrospectoscope: Reducing Bias in Emergency Medicine Chart Review Studies
Kaji, A. H., Schriger, D., & Green, S. (2014) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2014.03.025 · Cited by 703
Polnaszek, B., Gilmore-Bykovskyi, A., Hovanes, M., Roiland, R., Ferguson, P., Brown, R., & Kind, A. J. H. (2016) · Medical Care · doi:10.1097/mlr.0000000000000108 · Cited by 63
Concordance between medical records and interview data in correctional facilities
Bai, J. R., Mukherjee, D. V., Befus, M., Apa, Z., Lowy, F. D., & Larson, E. L. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-50 · Cited by 42
Uyeda, A. M., Curtis, J. R., Engelberg, R. A., Brumback, L. C., Guo, Y., Sibley, J., Lober, W. B., Cohen, T., Torrence, J., Heywood, J., et al. (2022) · Journal of Pain and Symptom Management · doi:10.1016/j.jpainsymman.2022.02.006 · Cited by 19
Metadata
- Authors
- Liddy, C., Wiens, M., & Hogg, W.
- Venue
- The Annals of Family Medicine
- Publisher
- Annals of Family Medicine
- Published
- 2011-01-01
- Volume
- 9
- Issue
- 1
- Pages
- 57-62
- ISSN
- 1544-1709 (ISSN-L)
- 1544-1717
- OpenAlex
- W2107006322
- Cited by
- 85
- References
- 5
- FWCI
- 6.08
- Type
- journal article
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.