The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Development and use of reporting guidelines for assessing the quality of validation studies of health administrative data
Abstract
Background and objectives Validation of health administrative data for identifying patients with different health states (diseases and conditions) is a research priority, but no guidelines exist for ensuring quality. We created reporting guidelines for studies validating administrative data identification algorithms and used them to assess the quality of reporting of validation studies in the literature.Methods Using Standards for Reporting of Diagnostic accuracy (STARD) criteria as a guide, we created a 40-item checklist of items with which identification accuracy studies should be reported. A systematic review identified studies that validated identification algorithms using administrative data. We used the checklist to assess the quality of reporting.Results In 271 included articles, goals and data sources were well reported but few reported four or more statistical estimates of accuracy (36.9%). In 65.9% of studies reporting positive predictive value (PPV)/negative predictive value (NPV), the prevalence of disease in the validation cohort was higher than in the administrative data, potentially falsely elevating predictive values. Subgroup accuracy (53.1%) and 95% confidence intervals for accuracy measures (35.8%) were also underreported.Conclusions The quality of studies validating health states in the administrative data varies, with significant deficits in reporting of markers of diagnostic accuracy, including the appropriate estimation of PPV and NPV. These omissions could lead to misclassification bias and incorrect estimation of incidence and health services utilization rates. Use of a reporting checklist, such as the one created for this study by modifying the STARD criteria, could improve the quality of reporting of validation studies, allowing for accurate application of algorithms, and interpretation of research using health administrative data.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Two recent systematic reviews also document poor reporting of studies undertaken to validate data from routine data sources [5,6], which can obscure sources of bias, hamper efforts to undertake meta-analyses, and lead to erroneous conclusions.”
From The Reporting of Studies Conducted Using Observational Routinely-Collected Health Data (RECORD) statement. (2015) · cites this paper as [5]
In the citing paper
“The most common reference standards are medical records, surveys of patients or practitioners, and clinical registries [5,34].”
From The Reporting of Studies Conducted Using Observational Routinely-Collected Health Data (RECORD) statement. (2015) · cites this paper as [5]
In the citing paper
“Measures of accuracy are similar to those reported in diagnostic test studies, including sensitivity, specificity, positive and negative predictive values, or kappa coefficients [5,34].”
From The Reporting of Studies Conducted Using Observational Routinely-Collected Health Data (RECORD) statement. (2015) · cites this paper as [5]
Citations
Cites
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040296 · Cited by 23,020
Towards complete and accurate reporting of studies of diagnostic accuracy: The STARD Initiative.
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2003) · Annals of Internal Medicine · doi:10.7326/0003-4819-138-1-200301070-00010 · Cited by 3,219
Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., & de Vet, H. C. W. (2003) · BMJ · doi:10.1136/bmj.326.7379.41 · Cited by 3,215
Improving the Quality of Reporting of Randomized Controlled Trials: The CONSORT Statement
Begg, C. (1996) · JAMA · doi:10.1001/jama.1996.03540080059030 · Cited by 3,127
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2007) · Preventive Medicine · doi:10.1016/j.ypmed.2007.08.012 · Cited by 2,698
Statistics Notes: Diagnostic tests 2: predictive values
Altman, D. G., & Bland, J. M. (1994) · BMJ · doi:10.1136/bmj.309.6947.102 · Cited by 1,711
Hux, J. E., Ivis, F. J., Flintoft, V. F., & Bica, A. (2002) · Diabetes Care · doi:10.2337/diacare.25.3.512 · Cited by 1,290
Cha, M., Eastwaad, S., Hartan, R., Maher, D., Olkin, I., Pitkin, R., Rennie, D., Schulz, K. F., Simel, D. L., & Straup, D. F. (1996) · Cited by 736
Bernstein, C. N., Blanchard, J., Rawsthorne, P. E., & Wajda, A. (1999) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a009735 · Cited by 661
ROGERS, F. B. (1965) · Nature · doi:10.1038/205236a0 · Cited by 571
Benchimol, E. I., Guttmann, A., Griffiths, A., Rabeneck, L., Mack, D. R., Brill, H., Howard, J., Guan, J., & To, T. M. (2009) · Gut · doi:10.1136/gut.2009.188383 · Cited by 412
Self-citationEstimating the Burden of Disease
Robinson, J. R., Young, T. K., Roos, L. L., & Gelskey, D. E. (1997) · Medical Care · doi:10.1097/00005650-199709000-00006 · Cited by 327
Defining asthma in epidemiological studies
Pekkanen, J., & Pearce, N. E. (1999) · European Respiratory Journal · doi:10.1034/j.1399-3003.1999.14d37.x · Cited by 278
An Algorithm for the Use of Medicare Claims Data to Identify Women with Incident Breast Cancer
Nattinger, A. B., Laud, P. W., Bajorunaite, R., Sparapani, R., & Freeman, J. L. (2004) · Health Services Research · doi:10.1111/j.1475-6773.2004.00315.x · Cited by 195
Chen, G., Faris, P., Hemmelgarn, B., Walker, R. L., & Quan, H. (2009) · BMC Medical Research Methodology · doi:10.1186/1471-2288-9-5 · Cited by 183
The Accuracy of Reporting of the Hypertensive Disorders of Pregnancy in Population Health Data
Roberts, C. L., Bell, J. C., Ford, J. B., Hadfield, R. M., Algert, C. S., & Morris, J. M. (2008) · Hypertension in Pregnancy · doi:10.1080/10641950701826695 · Cited by 176
De Coster, C., Quan, H., Finlayson, A., Gao, M., Halfon, P., Humphries, K. H., Johansen, H., Lix, L. M., Luthi, J.-C., Ma, J., et al. (2006) · BMC Health Services Research · doi:10.1186/1472-6963-6-77 · Cited by 167
Case verification of children with asthma in Ontario
To, T. M., Dell, S., Dick, P. T., Cicutto, L. C., Harris, J. K., MacLusky, I., & Tassoudji, M. (2005) · Pediatric Allergy and Immunology · doi:10.1111/j.1399-3038.2005.00346.x · Cited by 155
Self-citationGuttmann, A., Nakhla, M. M., Henderson, M., To, T. M., Daneman, D., Cauch‐Dudek, K., Wang, X., Lam, K., & Hux, J. E. (2009) · Pediatric Diabetes · doi:10.1111/j.1399-5448.2009.00539.x · Cited by 138
Self-citationBrenner, H., & Gefeller, O. (1993) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a116805 · Cited by 105
Scleroderma prevalence: Demographic variations in a population‐based sample
Bernatsky, S., Joseph, L., Pineau, C. A., Bélisle, P., Hudson, M., & Clarke, A. E. (2009) · Arthritis Care & Research · doi:10.1002/art.24339 · Cited by 104
Epidemiologic Methods for Health Policy
Cohen, J. E., & Spasoff, R. A. (2000) · Journal of Public Health Policy · doi:10.2307/3343346 · Cited by 94
Health services research: reporting on studies using secondary data sources.
Huston, P., & Naylor, C. D. (1996) · Europe PMC (PubMed Central) · Cited by 89
Green, M. S. (1983) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a113521 · Cited by 56
Liu, L., Allison, J. E., & Herrinton, L. J. (2009) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.1824 · Cited by 52
Measuring adherence to mammography screening recommendations among low-income women
Armstrong, K., Long, J. A., & Shea, J. A. (2004) · Preventive Medicine · doi:10.1016/j.ypmed.2003.12.023 · Cited by 51
Identifying Persons with Treated Asthma Using Administrative Data via Latent Class Modelling
Prosser, R. J., Carleton, B., & Smith, M. A. (2007) · Health Services Research · doi:10.1111/j.1475-6773.2007.00775.x · Cited by 44
Predictive Value of ICD-9-CM Codes Used in Vaccine Safety Research
Donahue, J. G., Destefano, F. R., Baggs, J. M., Eriksen, E., Mullooly, J. P., & for the VSD Data Quality Working Group (2008) · Methods of Information in Medicine · doi:10.3414/me0500 · Cited by 38
Manuel, D., Lim, J., Tanuseputro, P., & Stukel, T. (2007) · BMC Public Health · doi:10.1186/1471-2458-7-174 · Cited by 36
Self-citationAHMED, F., JANES, G., BARON, R., & Latts, L. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.11.020 · Cited by 23
Assessment of anthrax vaccination data in the Defense Medical Surveillance System, 1998–2004
Payne, D. C., Rose, C. E., Aranas, A. E., Zhang, Y., Tolentino, H., Weston, E. J., McNeil, M. M., & Ruscio, B. A. (2007) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.1395 · Cited by 14
van Walraven, C., Bennett, C., & Forster, A. J. (2010) · Health Services Research · doi:10.1111/j.1475-6773.2010.01159.x · Cited by 5
Hosmer (1997) · doi:10.1002/(sici)1097-0258(19970515)16:9
Cited by
Sorted by citations
Benchimol, E. I., Smeeth, L., Guttmann, A., Harron, K., Moher, D., Petersen, I., Sørensen, H. T., von Elm, E., Langan, S. M., & RECORD Working Committee (2015) · PLOS Medicine · doi:10.1371/journal.pmed.1001885 · Cited by 5,749
Self-citationA catalogue of reporting guidelines for health research
Simera, I., Moher, D., Hoey, J., Schulz, K. F., & Altman, D. G. (2010) · European Journal of Clinical Investigation · doi:10.1111/j.1365-2362.2009.02234.x · Cited by 1,047
Langan, S., Schmidt, S. A. J., Wing, K., Ehrenstein, V., Nicholls, S. G., Filion, K. B., Klungel, O. H., Petersen, I., Sørensen, H. T., Dixon, W. G., et al. (2018) · BMJ · doi:10.1136/bmj.k3532 · Cited by 673
Self-citationNg, S. C., Kaplan, G. G., Tang, W., Banerjee, R., Adigopula, B., Underwood, F. E., Tanyingoh, D., Wei, S.-C., Lin, W.-C., Lin, H.-H., et al. (2019) · American Journal of Gastroenterology · doi:10.1038/s41395-018-0233-2 · Cited by 261
Davis, K. A. S., Sudlow, C. L. M., & Hotopf, M. H. (2016) · BMC Psychiatry · doi:10.1186/s12888-016-0963-x · Cited by 264
Metadata
- Authors
- Benchimol, E. I., Manuel, D. G., To, T., Griffiths, A. M., Rabeneck, L., & Guttmann, A.
- Affiliations
- University of Ottawa (CA)
- Statistics Canada (CA)
- University of Toronto (CA)
- Corresponding author
- Benchimol, Eric I.
- ORCID
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2011-08
- Volume
- 64
- Issue
- 8
- Pages
- 821-829
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W1979334594
- Cited by
- 368
- References
- 32
- Citations by year
- 2026: 20 · 2025: 13 · 2024: 25 · 2023: 29 · 2022: 21 · 2021: 30 · 2020: 28 · 2019: 40 · 2018: 27 · 2017: 27 · 2016: 27 · 2015: 21 · 2014: 25 · 2013: 16 · 2012: 14
- FWCI
- 2.92
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Clinical Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- Funders
- American College of Gastroenterology
- Child and Family Research Institute
- Public Health Agency of Canada
- Canadian Child Health Clinician Scientist Program
- Hospital for Sick Children
- Institut pour la Recherche en Santé Publique
- Canadian Institutes of Health Research
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Health Professions › Health Information Management › Medical Coding and Health Information
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- administrative health data, data validation, identification algorithm, reporting guidelines, reporting quality, diagnostic accuracy, positive predictive value, negative predictive value, misclassification bias, disease prevalence, incidence estimation, health care utilization
- MeSH terms
- Algorithms, Forms and Records Control, Health Services Research, Health Status, Hospital Records, Humans, Quality Control, Guidelines as Topic, International Classification of Diseases, Validation Studies as Topic, Translational Research, Biomedical, Checklist
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
