The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Development and use of reporting guidelines for assessing the quality of validation studies of health administrative data
Abstract
Background and objectives Validation of health administrative data for identifying patients with different health states (diseases and conditions) is a research priority, but no guidelines exist for ensuring quality. We created reporting guidelines for studies validating administrative data identification algorithms and used them to assess the quality of reporting of validation studies in the literature.Methods Using Standards for Reporting of Diagnostic accuracy (STARD) criteria as a guide, we created a 40-item checklist of items with which identification accuracy studies should be reported. A systematic review identified studies that validated identification algorithms using administrative data. We used the checklist to assess the quality of reporting.Results In 271 included articles, goals and data sources were well reported but few reported four or more statistical estimates of accuracy (36.9%). In 65.9% of studies reporting positive predictive value (PPV)/negative predictive value (NPV), the prevalence of disease in the validation cohort was higher than in the administrative data, potentially falsely elevating predictive values. Subgroup accuracy (53.1%) and 95% confidence intervals for accuracy measures (35.8%) were also underreported.Conclusions The quality of studies validating health states in the administrative data varies, with significant deficits in reporting of markers of diagnostic accuracy, including the appropriate estimation of PPV and NPV. These omissions could lead to misclassification bias and incorrect estimation of incidence and health services utilization rates. Use of a reporting checklist, such as the one created for this study by modifying the STARD criteria, could improve the quality of reporting of validation studies, allowing for accurate application of algorithms, and interpretation of research using health administrative data.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Two recent systematic reviews also document poor reporting of studies undertaken to validate data from routine data sources [5,6], which can obscure sources of bias, hamper efforts to undertake meta-analyses, and lead to erroneous conclusions.”
From The Reporting of Studies Conducted Using Observational Routinely-Collected Health Data (RECORD) statement. (2015) · cites this paper as [5]
In the citing paper
“The most common reference standards are medical records, surveys of patients or practitioners, and clinical registries [5,34].”
From The Reporting of Studies Conducted Using Observational Routinely-Collected Health Data (RECORD) statement. (2015) · cites this paper as [5]
In the citing paper
“Measures of accuracy are similar to those reported in diagnostic test studies, including sensitivity, specificity, positive and negative predictive values, or kappa coefficients [5,34].”
From The Reporting of Studies Conducted Using Observational Routinely-Collected Health Data (RECORD) statement. (2015) · cites this paper as [5]
Citations
Cites
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040296 · Cited by 23,020
Towards complete and accurate reporting of studies of diagnostic accuracy: The STARD Initiative.
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2003) · Annals of Internal Medicine · doi:10.7326/0003-4819-138-1-200301070-00010 · Cited by 3,219
Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., & de Vet, H. C. W. (2003) · BMJ · doi:10.1136/bmj.326.7379.41 · Cited by 3,215
Improving the Quality of Reporting of Randomized Controlled Trials: The CONSORT Statement
Begg, C. (1996) · JAMA · doi:10.1001/jama.1996.03540080059030 · Cited by 3,127
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2007) · Preventive Medicine · doi:10.1016/j.ypmed.2007.08.012 · Cited by 2,698
Statistics Notes: Diagnostic tests 2: predictive values
Altman, D. G., & Bland, J. M. (1994) · BMJ · doi:10.1136/bmj.309.6947.102 · Cited by 1,711
Hux, J. E., Ivis, F. J., Flintoft, V. F., & Bica, A. (2002) · Diabetes Care · doi:10.2337/diacare.25.3.512 · Cited by 1,290
Cha, M., Eastwaad, S., Hartan, R., Maher, D., Olkin, I., Pitkin, R., Rennie, D., Schulz, K. F., Simel, D. L., & Straup, D. F. (1996) · Cited by 736
Bernstein, C. N., Blanchard, J., Rawsthorne, P. E., & Wajda, A. (1999) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a009735 · Cited by 661
ROGERS, F. B. (1965) · Nature · doi:10.1038/205236a0 · Cited by 571
Cited by
Sorted by citations
Items 301–369 of 369
Oyekan, B. M. (2023) · Sustainable development goals series · doi:10.1007/978-981-99-2411-0_12 · Cited by 1
Li, A. W. (2016) · Scholarship@Western (Western University) · Cited by 1
Escribà, J. M., Banqué, M., Macià, F., Galvez, J. A., Esteban, L., Pareja, L., Clèries, R., Sanz, X., Castells, X., Borràs, J. M., et al. (2019) · Clinical & Translational Oncology · doi:10.1007/s12094-019-02219-3 · Cited by 1
Catalá-López, F., Driver, J. A., Page, M. J., Hutton, B., Ridao-López, M., Berrozpe‐Villabona, C., Alonso‐Arroyo, A., Medín, C. A. F., Bernal‐Delgado, E., Valencia, A., et al. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-058738 · Cited by 1
Sun, Y., Dreier, J. W., Wu, C., Ehrenstein, V., & Christensen, J. (2022) · Clinical Epidemiology · doi:10.2147/clep.s353215 · Cited by 1
Tijssen, J. A., Richard, L., Shariff, S. Z., To, T. M., & Parshuram, C. S. (2020) · Canadian Journal of Respiratory Critical Care and Sleep Medicine · doi:10.1080/24745332.2020.1802627 · Cited by 1
Self-citationComment On: Use of Patient Registries and Administrative Datasets for the Study of Pediatric Cancer
Gupta, S. (2015) · Pediatric Blood & Cancer · doi:10.1002/pbc.25788 · Cited by 1
Administrative databases for observational studies in multiple sclerosis: power or peril?
Marrie, R. A. (2014) · Multiple Sclerosis Journal · doi:10.1177/1352458514521310 · Cited by 1
ROC Analysis in Business Decision-makings
Hu, N., & Zhou, X. (2014) · IGI Global eBooks · doi:10.4018/978-1-4666-5202-6.ch189 · Cited by 1
Hu, N. (2017) · IGI Global eBooks · doi:10.4018/978-1-5225-2255-3.ch192 · Cited by 1
Tshima, G. K., & Mulumba, P. M. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/524462 · Cited by 1
A population-based evaluation of long-term outcomes following major burn injury
Mason, S. A. (2018) · TSpace (University of Toronto) · Cited by 1
Pragmatic Trial End Point Capture: Making Sure Makes the Difference
Sundbøll, J., & Adelborg, K. (2021) · Circulation Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.121.008615 · Cited by 1
Kinugawa, A., Tamada, Y., Kusama, T., Hoshi‐Harada, M., Ono, S., Oda, F., Maeda, M., Yoda, N., Osaka, K., Fukuda, H., et al. (2025) · medRxiv · doi:10.1101/2025.10.17.25338212 · Cited by 1
Manzanares-Laya, S., Burón, A., Murta‐Nascimento, C., Servitja, S., Castells, X., & Macià, F. (2014) · Revista de Calidad Asistencial · doi:10.1016/j.cali.2014.05.004
McChesney, J. (2012) · Libraries and Cultural Resources (University of Calgary) · doi:10.11575/prism/25648
Myocardial Infarction among HIV-infected patients enrolled in the North Carolina Medicaid program
Brouwer, E. (2019) · Carolina Digital Repository (University of North Carolina at Chapel Hill) · doi:10.17615/zaz0-ee04
Chan, J. (2015) · uO Research (University of Ottawa) · doi:10.20381/ruor-4914
Physician Payment: Data Integrity and Policies for Improvement
Cunningham, C. (2016) · Libraries and Cultural Resources (University of Calgary) · doi:10.11575/prism/27353
Metformin and Prostate Cancer among Diabetic Men
Margel, D. (2014) · TSpace (University of Toronto)
Cancer risk among women living with HIV: Implications for care in the modern cART era
Salters, K. A. (2017) · Summit (Simon Fraser University)
Apuntes sobre el contrato por persona a nombrar
Posada, G. F. P. (2000) · Advocatus · doi:10.26439/advocatus2000.n003.2265
Momper, J. D., Mulberg, A. E., Mehrotra, N., Turner, D., Faubion, W. A., Conklin, L. S., Azer, K., & Dubinsky, M. C. (2019) · doi:10.1002/9781119289234.ch14
Brehaut, J., Guèvremont, A., Arım, R. G., Garner, R., Miller, A. R., McGrail, K. M., Brownell, M. D., Lach, L., Rosenbaum, P., & Kohen, D. (2019) · International Journal for Population Data Science · doi:10.23889/ijpds.v4i1.584
Population-based studies in sarcoma research
Bozzo, A. (2019) · MacSphere (McMaster University) · doi:10.71548/18155
Boyd, S., Chen, T., Blackmore, J., Becker, M., Kendall, C., Singer, A. G., Ireland, L., & Asghari, S. (2020) · doi:10.21203/rs.3.rs-18251/v1
Anan, K., Kataoka, Y., Ichikado, K., Kawamura, K., Johkoh, T., Fujimoto, K., Tobino, K., Tachikawa, R., Ito, H., Nakamura, T., et al. (2021) · medRxiv · doi:10.1101/2021.07.04.21259968
(2013) · New South Wales Public Health Bulletin · doi:10.1071/nbv24n2
Ahmed, S. S., Pollack, A., Havard, A., Pearson, S., & Chidwick, K. (2023) · International Journal for Population Data Science · doi:10.23889/ijpds.v8i1.2118
Quality Indicators in Paediatric and Perinatal Care in Japan: Development and Feasibility TestingPreprint
Shinjo, D., Ozawa, N., Nakadate, N., Kanamori, Y., Matsumoto, K., Noguchi, T., Ohtera, S., & Kato, H. (2023) · Research Square · doi:10.21203/rs.3.rs-2528400/v1
McGill, R., Scholes, C. J., Torbey, S., & Calabro, L. (2025) · The Journal of Knee Surgery · doi:10.1055/s-0044-1801750
Saka‐Herrán, C., Bennett, J. C., Alkabti, Y., Fatir, M., Clyne, B., McCarthy, C., Tynan, G., Dunne, N., Flood, M., McCarthy, E. M., et al. (2025) · medRxiv · doi:10.1101/2025.06.30.25330552
Guolla, L., Lau, C., Chan, K., Nathan, P. C., & Gupta, S. (2025) · Cancer Epidemiology · doi:10.1016/j.canep.2025.102902
κ=0.97: A Practical Framework Any Hospital Can Implement for Research-Grade Data QualityPreprint
Martínez-Ríos, D., Moreno-Rojas, J. C., Martínez-Ríos, A., Terán-Aguilera, G. D., Lulé-Martínez, C. E., & Hernández-López, D. (2025) · Research Square · doi:10.21203/rs.3.rs-8038431/v1
Paget, L.-M., Lorton, F., Forgeot, C., Beltzer, N., & Gallay, A. (2025) · Clinical Epidemiology · doi:10.2147/clep.s541052
Wang, W., Jin, Y.-H., Liu, M., He, Q., Xu, J.-Y., Wang, M.-Q., Li, G.-W., Fu, B., Yan, S.-Y., Zou, K., et al. (2024) · Figshare · doi:10.6084/m9.figshare.26507250
Shi, Y., Schulte, P. J., Hanson, A. C., Zaccariello, M. J., Hu, D., Crow, S. S., Flick, R. P., & Warner, D. O. (2020) · Figshare · doi:10.6084/m9.figshare.13203495
Additional file 2 of Validity of administrative data in recording sepsis: a systematic review
Jolley, R., Sawka, K. J., Yergens, D., Quan, H., Jetté, N., & Doig, C. J. (2021) · Figshare · doi:10.6084/m9.figshare.15094861
Shi, Y., Schulte, P. J., Hanson, A. C., Zaccariello, M. J., Hu, D., Crow, S. S., Flick, R. P., & Warner, D. O. (2020) · Figshare · doi:10.6084/m9.figshare.13203498
Zhao, R., Zhang, W., Zhang, Z., He, C., Xu, R., Tang, X., & Wang, B. (2024) · Figshare · doi:10.6084/m9.figshare.26600689
Additional file 1 of Validity of administrative data in recording sepsis: a systematic review
Jolley, R. J., Sawka, K. J., Yergens, D., Quan, H., Jetté, N., & Doig, C. J. (2021) · Figshare · doi:10.6084/m9.figshare.15094858
Zhao, R., Zhang, W., Zhang, Z., He, C., Xu, R., Tang, X., & Wang, B. (2024) · Figshare · doi:10.6084/m9.figshare.26600680
Ii, Y., Hiro, S., & Nakazuru, Y. (2022) · Figshare · doi:10.6084/m9.figshare.18864644
Zhao, R., Zhang, W., Zhang, Z., He, C., Xu, R., Tang, X., & Wang, B. (2024) · Figshare · doi:10.6084/m9.figshare.26600683
Catho, G., Fortchantre, L., Teixeira, D., Galas-Haddad, M., Boroli, F., Chraïti, M.-N., Abbas, M., Harbarth, S., & Buetti, N. (2024) · Figshare · doi:10.6084/m9.figshare.25583841
Havard, A., Manski-Nankervis, J.-A., Thistlethwaite, J. E., Daniels, B. J., Myton, R., Tu, K., & Chidwick, K. (2021) · Figshare · doi:10.6084/m9.figshare.14738523
Zhao, R., Zhang, W., Zhang, Z., He, C., Xu, R., Tang, X., & Wang, B. (2024) · Figshare · doi:10.6084/m9.figshare.26600686
Wang, W., Jin, Y.-H., Liu, M., He, Q., Xu, J.-Y., Wang, M.-Q., Li, G.-W., Fu, B., Yan, S.-Y., Zou, K., et al. (2024) · Figshare · doi:10.6084/m9.figshare.26507266
Knold, J., Bork, C., Snoer, M., Heitmann, M., Bang, L., Kanstrup, H., Hansen, K., Hedegaard, B. S., Thomsen, K. K., Lauridsen, B. K., et al. (2026) · Clinical Epidemiology · doi:10.2147/clep.s556764
Tassi, M., Lemaignen, A., Stéfic, K., & Grammatico‐Guillon, L. (2026) · BMC Medical Research Methodology · doi:10.1186/s12874-026-02820-5
Marhamah, U., Syukur, F., Panuntun, S., Choirudin, M., & Hakim, A. (2026) · Islamic Guidance and Counseling Journal · doi:10.25217/0020269740800
Dermott, J. A., To, T. M., Jaakkimainen, L., Bouchard, M. L., Howard, A., & Lebel, D. (2026) · Clinical Epidemiology · doi:10.2147/clep.s578144
Self-citationKwon, E., Yoon, W. S., Moon, J.-Y., Hwang, Y. I., Yoo, K.-H., Kim, Y.-Y., & Kim, Y. (2026) · Respiratory Research · doi:10.1186/s12931-026-03677-4
Zhao, H. J., Wu, V., Gandelman, M., Ushcatz, I., Muntyanu, A., & Drucker, A. M. (2026) · Journal of Cutaneous Medicine and Surgery · doi:10.1177/12034754261442933
Catho, G., Toko, J., Fortchantre, L., Priore, E. L., Balmelli, C., Damonti, L., Jent, P., Moulin, E., Grandbastien, B., Schreiber, P. W., et al. (2026) · Eurosurveillance · doi:10.2807/1560-7917.es.2026.31.18.2500650
Okazaki, Y., Kataoka, Y., & Ichiba, T. (2026) · Cureus · doi:10.7759/cureus.108794
Hand Trauma Case-Identification Using Administrative Data: A Systematic Review
Wong, C. R., Rosebush, C. E., & Baltzer, H. L. (2026) · Journal of Wrist Surgery · doi:10.1055/a-2868-4092
Validity of Denture Usage Definitions Based on Claims Data in Japanese Older Adults
Kinugawa, A., Tamada, Y., Kusama, T., Hoshi‐Harada, M., Ono, S., Oda, F., Maeda, M., Yoda, N., Osaka, K., Fukuda, H., et al. (2026) · JDR Clinical & Translational Research · doi:10.1177/23800844261444341
Quality measures in the design, conduct, and reporting of endoscopic research
Elmunzer, B. J., Madhoun, M., Coté, G. A., Cummings, L. C., Shahidi, N. C., Srivastava, N., Essex, E. A., Scheiman, J. M., May, F. P., Yen, T., et al. (2026) · Gastrointestinal Endoscopy · doi:10.1016/j.gie.2026.04.026
Johnston, A., Petrcich, W., Smith, G. N., Tanuseputro, P., Lee, D. S., Coutinho, T. A., & Edwards, J. D. (2026) · Journal of the American Heart Association · doi:10.1161/jaha.125.046265
Proper Phenotyping of Pancreatic Cancer in Database Research—Reply
Mavromatis, L. A., & Grams, M. E. (2026) · JAMA Oncology · doi:10.1001/jamaoncol.2026.2612
Gersekowski, K., Codde, J., Cohen, P. A., GRANT, P., Jordan, S. J., & Webb, P. M. (2026) · Public Health Research & Practice · doi:10.1071/pu25097
Saito, K., Ishiguro, C., Sato, K., Nakatochi, M., & Fukuda, H. (2026) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.70488
Sickness Absence and Record Linkage Using Primary Healthcare, Hospital and Occupational Databases
Gili‐Miner, M., Cabanillas-Moruno, J. L., & Ramírez-Ramírez, G. (2015) · Studies in big data · doi:10.1007/978-3-319-11056-1_10
Metadata
- Authors
- Benchimol, E. I., Manuel, D. G., To, T., Griffiths, A. M., Rabeneck, L., & Guttmann, A.
- Affiliations
- University of Ottawa (CA)
- Statistics Canada (CA)
- University of Toronto (CA)
- Corresponding author
- Benchimol, Eric I.
- ORCID
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2011-08
- Volume
- 64
- Issue
- 8
- Pages
- 821-829
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W1979334594
- Cited by
- 368
- References
- 32
- Citations by year
- 2026: 20 · 2025: 13 · 2024: 25 · 2023: 29 · 2022: 21 · 2021: 30 · 2020: 28 · 2019: 40 · 2018: 27 · 2017: 27 · 2016: 27 · 2015: 21 · 2014: 25 · 2013: 16 · 2012: 14
- FWCI
- 2.92
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Clinical Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- Funders
- American College of Gastroenterology
- Child and Family Research Institute
- Public Health Agency of Canada
- Canadian Child Health Clinician Scientist Program
- Hospital for Sick Children
- Institut pour la Recherche en Santé Publique
- Canadian Institutes of Health Research
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Health Professions › Health Information Management › Medical Coding and Health Information
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- administrative health data, data validation, identification algorithm, reporting guidelines, reporting quality, diagnostic accuracy, positive predictive value, negative predictive value, misclassification bias, disease prevalence, incidence estimation, health care utilization
- MeSH terms
- Algorithms, Forms and Records Control, Health Services Research, Health Status, Hospital Records, Humans, Quality Control, Guidelines as Topic, International Classification of Diseases, Validation Studies as Topic, Translational Research, Biomedical, Checklist
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review