Development and use of reporting guidelines for assessing the quality of validation studies of health administrative data
Abstract
Background and objectives Validation of health administrative data for identifying patients with different health states (diseases and conditions) is a research priority, but no guidelines exist for ensuring quality. We created reporting guidelines for studies validating administrative data identification algorithms and used them to assess the quality of reporting of validation studies in the literature.Methods Using Standards for Reporting of Diagnostic accuracy (STARD) criteria as a guide, we created a 40-item checklist of items with which identification accuracy studies should be reported. A systematic review identified studies that validated identification algorithms using administrative data. We used the checklist to assess the quality of reporting.Results In 271 included articles, goals and data sources were well reported but few reported four or more statistical estimates of accuracy (36.9%). In 65.9% of studies reporting positive predictive value (PPV)/negative predictive value (NPV), the prevalence of disease in the validation cohort was higher than in the administrative data, potentially falsely elevating predictive values. Subgroup accuracy (53.1%) and 95% confidence intervals for accuracy measures (35.8%) were also underreported.Conclusions The quality of studies validating health states in the administrative data varies, with significant deficits in reporting of markers of diagnostic accuracy, including the appropriate estimation of PPV and NPV. These omissions could lead to misclassification bias and incorrect estimation of incidence and health services utilization rates. Use of a reporting checklist, such as the one created for this study by modifying the STARD criteria, could improve the quality of reporting of validation studies, allowing for accurate application of algorithms, and interpretation of research using health administrative data.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Two recent systematic reviews document poor reporting of studies undertaken to validate data from routine data sources.
“Two recent systematic reviews also document poor reporting of studies undertaken to validate data from routine data sources [5,6], which can obscure sources of bias, hamper efforts to undertake meta-analyses, and lead to erroneous conclusions.”
From The Reporting of Studies Conducted Using Observational Routinely-Collected Health Data (RECORD) statement. (2015) · cites this paper as [5]
The most common reference standards are medical records, surveys of patients or practitioners, and clinical registries.
“The most common reference standards are medical records, surveys of patients or practitioners, and clinical registries [5,34].”
From The Reporting of Studies Conducted Using Observational Routinely-Collected Health Data (RECORD) statement. (2015) · cites this paper as [5]
Measures of accuracy are similar to those reported in diagnostic test studies, including sensitivity, specificity, positive and negative predictive values, or kappa coefficients.
“Measures of accuracy are similar to those reported in diagnostic test studies, including sensitivity, specificity, positive and negative predictive values, or kappa coefficients [5,34].”
From The Reporting of Studies Conducted Using Observational Routinely-Collected Health Data (RECORD) statement. (2015) · cites this paper as [5]
Cites
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040296 · Cited by 23,020 · see citing works
Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., & de Vet, H. C. W. (2003) · BMJ · doi:10.1136/bmj.326.7379.41 · Cited by 3,215
Towards Complete and Accurate Reporting of Studies of Diagnostic Accuracy: The STARD Initiative
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P., Irwig, L., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2003) · Clinical Biochemistry · doi:10.1016/s0009-9120(02)00443-5 · Cited by 3,182 · see citing works
Improving the Quality of Reporting of Randomized Controlled Trials: The CONSORT Statement
Begg, C. (1996) · JAMA · doi:10.1001/jama.1996.03540080059030 · Cited by 3,127 · see citing works
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2007) · Preventive Medicine · doi:10.1016/j.ypmed.2007.08.012 · Cited by 2,698 · see citing works
Statistics Notes: Diagnostic tests 2: predictive values
Altman, D. G., & Bland, J. M. (1994) · BMJ · doi:10.1136/bmj.309.6947.102 · Cited by 1,711 · see citing works
Hux, J. E., Ivis, F. J., Flintoft, V. F., & Bica, A. (2002) · Diabetes Care · doi:10.2337/diacare.25.3.512 · Cited by 1,290 · see citing works
Cha, M., Eastwaad, S., Hartan, R., Maher, D., Olkin, I., Pitkin, R., Rennie, D., Schulz, K. F., Simel, D. L., & Straup, D. F. (1996) · Cited by 736 · see citing works
Bernstein, C. N., Blanchard, J., Rawsthorne, P. E., & Wajda, A. (1999) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a009735 · Cited by 661 · see citing works
ROGERS, F. B. (1965) · Nature · doi:10.1038/205236a0 · Cited by 565 · see citing works
Benchimol, E. I., Guttmann, A., Griffiths, A., Rabeneck, L., Mack, D. R., Brill, H., Howard, J., Guan, J., & To, T. M. (2009) · Gut · doi:10.1136/gut.2009.188383 · Cited by 412 · see citing works
Self-citationEstimating the Burden of Disease
Robinson, J. R., Young, T. K., Roos, L. L., & Gelskey, D. E. (1997) · Medical Care · doi:10.1097/00005650-199709000-00006 · Cited by 327 · see citing works
Defining asthma in epidemiological studies
Pekkanen, J., & Pearce, N. (1999) · European Respiratory Journal · doi:10.1034/j.1399-3003.1999.14d37.x · Cited by 286 · see citing works
An Algorithm for the Use of Medicare Claims Data to Identify Women with Incident Breast Cancer
Nattinger, A. B., Laud, P. W., Bajorunaite, R., Sparapani, R., & Freeman, J. L. (2004) · Health Services Research · doi:10.1111/j.1475-6773.2004.00315.x · Cited by 195 · see citing works
Chen, G., Faris, P., Hemmelgarn, B., Walker, R. L., & Quan, H. (2009) · BMC Medical Research Methodology · doi:10.1186/1471-2288-9-5 · Cited by 183 · see citing works
The Accuracy of Reporting of the Hypertensive Disorders of Pregnancy in Population Health Data
Roberts, C. L., Bell, J. C., Ford, J. B., Hadfield, R. M., Algert, C. S., & Morris, J. M. (2008) · Hypertension in Pregnancy · doi:10.1080/10641950701826695 · Cited by 176 · see citing works
De Coster, C., Quan, H., Finlayson, A., Gao, M., Halfon, P., Humphries, K. H., Johansen, H., Lix, L. M., Luthi, J.-C., Ma, J., et al. (2006) · BMC Health Services Research · doi:10.1186/1472-6963-6-77 · Cited by 167 · see citing works
Case verification of children with asthma in Ontario
To, T. M., Dell, S., Dick, P. T., Cicutto, L. C., Harris, J. K., MacLusky, I., & Tassoudji, M. (2005) · Pediatric Allergy and Immunology · doi:10.1111/j.1399-3038.2005.00346.x · Cited by 155 · see citing works
Self-citationGuttmann, A., Nakhla, M. M., Henderson, M., To, T. M., Daneman, D., Cauch‐Dudek, K., Wang, X., Lam, K., & Hux, J. E. (2009) · Pediatric Diabetes · doi:10.1111/j.1399-5448.2009.00539.x · Cited by 138 · see citing works
Self-citationBrenner, H., & Gefeller, O. (1993) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a116805 · Cited by 108 · see citing works
Scleroderma prevalence: Demographic variations in a population‐based sample
Bernatsky, S., Joseph, L., Pineau, C. A., Bélisle, P., Hudson, M., & Clarke, A. E. (2009) · Arthritis Care & Research · doi:10.1002/art.24339 · Cited by 104 · see citing works
Epidemiologic Methods for Health Policy
Cohen, J. E., & Spasoff, R. A. (2000) · Journal of Public Health Policy · doi:10.2307/3343346 · Cited by 98 · see citing works
Health services research: reporting on studies using secondary data sources.
Huston, P., & Naylor, C. D. (1996) · Europe PMC (PubMed Central) · Cited by 89 · see citing works
Green, M. S. (1983) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a113521 · Cited by 56 · see citing works
Liu, L., Allison, J. E., & Herrinton, L. J. (2009) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.1824 · Cited by 52 · see citing works
Measuring adherence to mammography screening recommendations among low-income women
Armstrong, K., Long, J. A., & Shea, J. A. (2004) · Preventive Medicine · doi:10.1016/j.ypmed.2003.12.023 · Cited by 51 · see citing works
Identifying Persons with Treated Asthma Using Administrative Data via Latent Class Modelling
Prosser, R. J., Carleton, B., & Smith, M. A. (2007) · Health Services Research · doi:10.1111/j.1475-6773.2007.00775.x · Cited by 44 · see citing works
Predictive Value of ICD-9-CM Codes Used in Vaccine Safety Research
Donahue, J. G., Destefano, F. R., Baggs, J. M., Eriksen, E., Mullooly, J. P., & for the VSD Data Quality Working Group (2008) · Methods of Information in Medicine · doi:10.3414/me0500 · Cited by 38 · see citing works
Manuel, D., Lim, J., Tanuseputro, P., & Stukel, T. (2007) · BMC Public Health · doi:10.1186/1471-2458-7-174 · Cited by 36 · see citing works
Self-citationAHMED, F., JANES, G., BARON, R., & Latts, L. (2005) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2004.11.020 · Cited by 23 · see citing works
Assessment of anthrax vaccination data in the Defense Medical Surveillance System, 1998–2004
Payne, D. C., Rose, C. E., Aranas, A. E., Zhang, Y., Tolentino, H., Weston, E. J., McNeil, M. M., & Ruscio, B. A. (2007) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.1395 · Cited by 14 · see citing works
van Walraven, C., Bennett, C., & Forster, A. J. (2010) · Health Services Research · doi:10.1111/j.1475-6773.2010.01159.x · Cited by 5 · see citing works
Hosmer (1997) · doi:10.1002/(sici)1097-0258(19970515)16:9
Flagged first, then by citations
Cited by
Sorted by citations
Benchimol, E. I., Smeeth, L., Guttmann, A., Harron, K., Moher, D., Petersen, I., Sørensen, H. T., von Elm, E., Langan, S. M., & RECORD Working Committee (2015) · PLOS Medicine · doi:10.1371/journal.pmed.1001885 · Cited by 5,749 · see citing works
Self-citationA catalogue of reporting guidelines for health research
Simera, I., Moher, D., Hoey, J., Schulz, K. F., & Altman, D. G. (2010) · European Journal of Clinical Investigation · doi:10.1111/j.1365-2362.2009.02234.x · Cited by 1,047 · see citing works
Langan, S., Schmidt, S. A. J., Wing, K., Ehrenstein, V., Nicholls, S. G., Filion, K. B., Klungel, O. H., Petersen, I., Sørensen, H. T., Dixon, W. G., et al. (2018) · BMJ · doi:10.1136/bmj.k3532 · Cited by 644 · see citing works
Self-citationNg, S. C., Kaplan, G. G., Tang, W., Banerjee, R., Adigopula, B., Underwood, F. E., Tanyingoh, D., Wei, S.-C., Lin, W.-C., Lin, H.-H., et al. (2019) · American Journal of Gastroenterology · doi:10.1038/s41395-018-0233-2 · Cited by 261 · see citing works
Davis, K. A. S., Sudlow, C. L. M., & Hotopf, M. H. (2016) · BMC Psychiatry · doi:10.1186/s12888-016-0963-x · Cited by 261 · see citing works
Systematic review and assessment of validated case definitions for depression in administrative data
Fiest, K. M., Jetté, N., Quan, H., Smith, C., Metcalfe, A., Patten, S. B., & Beck, C. A. (2014) · BMC Psychiatry · doi:10.1186/s12888-014-0289-5 · Cited by 248 · see citing works
Stevens, A., Shamseer, L., Weinstein, E., Yazdi, F., Turner, L., Thielman, J., Altman, D. G., Hirst, A., Hoey, J., Palepu, A., et al. (2014) · BMJ · doi:10.1136/bmj.g3804 · Cited by 242 · see citing works
Muggah, E., Graves, E., Bennett, C. L., & Manuel, D. G. (2013) · BMC Public Health · doi:10.1186/1471-2458-13-16 · Cited by 229 · see citing works
Self-citationBenchimol, E. I., Guttmann, A., Mack, D. R., Nguyen, G. C., Marshall, J. K., Gregor, J. C., Wong, J., Forster, A. J., & Manuel, D. G. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.02.019 · Cited by 224 · see citing works
Self-citationNicholls, S. G., Quach, P., von Elm, E., Guttmann, A., Moher, D., Petersen, I., Sørensen, H. T., Smeeth, L., Langan, S. M., & Benchimol, E. I. (2015) · PLOS ONE · doi:10.1371/journal.pone.0125620 · Cited by 217 · see citing works
Self-citationTradeoffs between accuracy measures for electronic health care data algorithms
Chubak, J., Pocobelli, G. S., & Weiss, N. S. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.09.002 · Cited by 221 · see citing works
Comorbidity in Multiple Sclerosis
Magyari, M., & Sørensen, P. S. (2020) · Frontiers in Neurology · doi:10.3389/fneur.2020.00851 · Cited by 211 · see citing works
Recommendations for optimal ICD codes to study neurologic conditions: A systematic review
St. Germaine-Smith, C., Metcalfe, A., Pringsheim, T., Roberts, J. I., Beck, C. A., Hemmelgarn, B. R., McChesney, J., Quan, H., & Jette, N. (2012) · Neurology · doi:10.1212/wnl.0b013e3182684707 · Cited by 188 · see citing works
Validity of administrative data in recording sepsis: a systematic review
Jolley, R., Sawka, K. J., Yergens, D., Quan, H., Jetté, N., & Doig, C. J. (2015) · Critical Care · doi:10.1186/s13054-015-0847-3 · Cited by 196 · see citing works
Wang, S. V., Schneeweiß, S., Berger, M. L., Brown, J. S., de Vries, F., Douglas, I., Gagne, J. J., Gini, R., Klungel, O. H., Mullins, C. D., et al. (2017) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.4295 · Cited by 182 · see citing works
Antoniou, T., Zagorski, B., Loutfy, M. R., Strıke, C., & Glazier, R. H. (2011) · PLoS ONE · doi:10.1371/journal.pone.0021748 · Cited by 179 · see citing works
Quebec Integrated Chronic Disease Surveillance System (QICDSS), an innovative approach
Blais, C., Jean, S., Sirois, C., Rochette, L., Plante, C., Larocque, I., Doucet, M., Ruel, G., Simard, M., Gamache, P., et al. (2014) · Chronic diseases and injuries in Canada · doi:10.24095/hpcdp.34.4.06 · Cited by 161 · see citing works
Widdifield, J., Bombardier, C., Bernatsky, S., Paterson, J. M., Green, D., Young, J., Ivers, N. M., Butt, D. A., Jaakkimainen, R. L., Thorne, C., et al. (2014) · BMC Musculoskeletal Disorders · doi:10.1186/1471-2474-15-216 · Cited by 159 · see citing works
Widdifield, J., Bernatsky, S., Paterson, J. M., Tu, K., Ng, R., Thorne, C., Pope, J. E., & Bombardier, C. (2013) · Arthritis Care & Research · doi:10.1002/acr.22031 · Cited by 153 · see citing works
van Mourik, M. S. M., van Duijn, P. J., Moons, K. G. M., Bonten, M. J. M., & Lee, G. M. (2015) · BMJ Open · doi:10.1136/bmjopen-2015-008424 · Cited by 136 · see citing works
Chubak, J., Yu, O., Pocobelli, G. S., Lamerato, L. E., Webster, J., Prout, M. N., Yood, M. U., Barlow, W. E., & Buist, D. S. M. (2012) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/djs233 · Cited by 117 · see citing works
Kao, W., Hong, J., See, L., Yu, H., Hsu, J., Chou, I., Chou, W., Chiou, M., Wang, C., & Kuo, C. (2017) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.4267 · Cited by 116 · see citing works
Metcalfe, A., Neudam, A., Forde, S., Liu, M., Drösler, S. E., Quan, H., & Jetté, N. (2012) · Health Services Research · doi:10.1111/j.1475-6773.2012.01440.x · Cited by 114 · see citing works
Accuracy of Administrative Data for the Coding of Acute Stroke and TIAs
Hall, R., Mondor, L., Porter, J., Fang, J., & Kapral, M. K. (2016) · Canadian Journal of Neurological Sciences / Journal Canadien des Sciences Neurologiques · doi:10.1017/cjn.2016.278 · Cited by 113 · see citing works
Risk Factors for Periprosthetic Joint Infection Following Primary Total Hip Arthroplasty
a, T. M. A. C. (. (2019) · Journal of Bone and Joint Surgery · doi:10.2106/jbjs.19.00537 · Cited by 113 · see citing works
Visvanathan, K., Levit, L. A., Raghavan, D., Hudis, C. A., Wong, S. L., Dueck, A. C., & Lyman, G. H. (2017) · Journal of Clinical Oncology · doi:10.1200/jco.2017.72.6414 · Cited by 112 · see citing works
Bates, B. A., Akhabue, E., Nahass, M. M., Mukherjee, A., Hiltner, E. P., Rock, J., Wilton, B., Mittal, G., Visaria, A., Rua, M., et al. (2023) · Circulation Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.122.009078 · Cited by 113 · see citing works
Wang, S. V., Schneeweiß, S., Berger, M. L., Brown, J. S., de Vries, F., Douglas, I., Gagne, J. J., Gini, R., Klungel, O. H., Mullins, C. D., et al. (2017) · Value in Health · doi:10.1016/j.jval.2017.08.3018 · Cited by 113 · see citing works
Validity of Hospital Discharge Diagnosis Codes for Stroke
Jones, S., Gottesman, R. F., Shahar, E., Wruck, L. M., & Rosamond, W. D. (2014) · Stroke · doi:10.1161/strokeaha.114.006316 · Cited by 99 · see citing works
Abuabara, K., Magyari, A., Hoffstad, O., Jabbar‐Lopez, Z. K., Smeeth, L. L., Williams, H. C., Gelfand, J. M., Margolis, D. J., & Langan, S. (2017) · Journal of Investigative Dermatology · doi:10.1016/j.jid.2017.03.029 · Cited by 94 · see citing works
Benchimol, E. I., Smeeth, L. L., Guttmann, A., Harron, K. L., Hemkens, L. G., Moher, D., Petersen, I., Sørensen, H. T., von Elm, E. B., & Langan, S. (2016) · Zeitschrift für Evidenz Fortbildung und Qualität im Gesundheitswesen · doi:10.1016/j.zefq.2016.07.010 · Cited by 97 · see citing works
Self-citationPisesky, A., Benchimol, E. I., Wong, C. A., Hui, C. P., Crowe, M. L., Bélair, M., Pojsupap, S., Karnauchow, T., O’Hearn, K., Yasseen, A. S., et al. (2016) · PLoS ONE · doi:10.1371/journal.pone.0150416 · Cited by 91 · see citing works
Self-citationFoley, H. E., KNIGHT, J. C., Ploughman, M., Asghari, S., & Audas, R. P. (2021) · Pain · doi:10.1097/j.pain.0000000000002264 · Cited by 88 · see citing works
Konrad, R., Zhang, W., Bjarndóttir, M. V., & Proaño, R. A. (2019) · Health Systems · doi:10.1080/20476965.2019.1581433 · Cited by 86 · see citing works
Butt, D. A., Tu, K., Young, J., Green, D., Wang, M., Ivers, N. M., Jaakkimainen, L., Lam, R., & Guttman, M. (2014) · Neuroepidemiology · doi:10.1159/000365590 · Cited by 82 · see citing works
Sinha, S., Peach, G., Poloniecki, J. D., THOMPSON, M. M., & Holt, P. (2012) · European Journal of Public Health · doi:10.1093/eurpub/cks046 · Cited by 81 · see citing works
Helping everyone do better: a call for validation studies of routinely recorded health data
Ehrenstein, V., Petersen, I., Smeeth, L. L., Jick, S. S., Benchimol, E. I., Ludvigsson, J. F., & Sørensen, H. T. (2016) · Clinical Epidemiology · doi:10.2147/clep.s104448 · Cited by 79 · see citing works
Self-citationFriedman, M. Y., Leventer‐Roberts, M., Rosenblum, J., Zigman, N., Goren, I., Mourad, V., Lederman, N., Cohen, N., Matz, E., Dushnitzky, D., et al. (2018) · Clinical Epidemiology · doi:10.2147/clep.s151339 · Cited by 76 · see citing works
Self-citationBarber, C. E. H., Lacaille, D. V., & Fortin, P. R. (2013) · Arthritis Care & Research · doi:10.1002/acr.21959 · Cited by 72 · see citing works
Recommendations for respiratory syncytial virus surveillance at the national level
Teirlinck, A. C., Broberg, E., Berg, A. S., Campbell, H. J., Reeves, R. M., Carnahan, A., Lina, B., Pakarna, G., Bøås, H., Nohynek, H., et al. (2021) · European Respiratory Journal · doi:10.1183/13993003.03766-2020 · Cited by 70 · see citing works
Widdifield, J., Labrecque, J. A., Lix, L. M., Paterson, J. M., Bernatsky, S., Tu, K., Ivers, N. M., & Bombardier, C. (2013) · Arthritis Care & Research · doi:10.1002/acr.21993 · Cited by 66 · see citing works
Richard, L., Hwang, S. W., Forchuk, C., Nisenbaum, R. A., Clemens, K. K., Wiens, K., Booth, R. G., Azimaee, M., & Shariff, S. Z. (2019) · BMJ Open · doi:10.1136/bmjopen-2019-030221 · Cited by 65 · see citing works
Quality of administrative health databases in Canada: A scoping review
Hinds, A. M., Lix, L. M., Smith, M. J., Quan, H., & Sanmartin, C. (2016) · Canadian Journal of Public Health · doi:10.17269/cjph.107.5244 · Cited by 63 · see citing works
Maetens, A., Schreye, R. D., Faes, K., Houttekier, D., Deliëns, L., Gielen, B., Gendt, C. D., Lusyne, P., Annemans, L., & Cohen, J. (2016) · BMC Palliative Care · doi:10.1186/s12904-016-0159-7 · Cited by 59 · see citing works
Big Data Research in Neurosurgery: A Critical Look at this Popular New Study Design
Oravec, C. S., Motiwala, M., Reed, K., Kondziolka, D. S., Barker, F. G., Michael, L. M., & Klimo, P. (2017) · Neurosurgery · doi:10.1093/neuros/nyx328 · Cited by 59 · see citing works
Accuracy of administrative data for identification of patients with infective endocarditis
Tan, C., Hansen, M., Cohen, G. N., Boyle, K., Daneman, N., & Adhikari, N. K. J. (2016) · International Journal of Cardiology · doi:10.1016/j.ijcard.2016.09.030 · Cited by 58 · see citing works
Davis, K. A. S., Bashford, O., Jewell, A., Shetty, H., Stewart, R. J., Sudlow, C. L. M., & Hotopf, M. H. (2018) · PLoS ONE · doi:10.1371/journal.pone.0195002 · Cited by 58 · see citing works
van Walraven, C. (2017) · International Journal of Epidemiology · doi:10.1093/ije/dyx253 · Cited by 57 · see citing works
Nosyk, B., Colley, G., Yip, B., Chan, K., Heath, K., Lima, V. D., Gilbert, M., Hogg, R. S., Harrigan, P. R., Montaner, J. S. G., et al. (2013) · PLoS ONE · doi:10.1371/journal.pone.0054416 · Cited by 55 · see citing works
Mouratidou, N., Malmborg, P., Järås, J., Sigurdsson, G. V., Sandström, O., Fagerberg, U. L., Bröms, G., Ludvigsson, J. F., & Olén, O. (2022) · Clinical Epidemiology · doi:10.2147/clep.s358031 · Cited by 57 · see citing works
Sørensen, H. T., Langan, S., Benchimol, E. I., Guttmann, Moher, D., Petersen, I., Smeeth, L. L., Stanley, & von Elm, E. B. (2013) · Clinical Epidemiology · doi:10.2147/clep.s36885 · Cited by 53 · see citing works
Self-citationRoutinely collected data: the importance of high-quality diagnostic coding to research
Nicholls, S. G., Langan, S., & Benchimol, E. I. (2017) · Canadian Medical Association Journal · doi:10.1503/cmaj.170807 · Cited by 53 · see citing works
Self-citationMcGuinness, L. A., Warren‐Gash, C., Moorhouse, L. R., & Thomas, S. L. (2019) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.4669 · Cited by 52 · see citing works
Weisman, A., Tu, K., Young, J., Kumar, M., Austin, P. C., Jaakkimainen, L., Lipscombe, L., Aronson, R., & Booth, G. L. (2020) · BMJ Open Diabetes Research & Care · doi:10.1136/bmjdrc-2020-001224 · Cited by 62 · see citing works
Grosse, S. D., Green, N., & Reeves, S. L. (2020) · Pediatric Blood & Cancer · doi:10.1002/pbc.28703 · Cited by 51 · see citing works
Weinstein, E. J., Ritchey, M. E., & Re, V. L. (2022) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.5537 · Cited by 49 · see citing works
Aplenc, R., Fisher, B. T., Huang, Y. S., Li, Y., Alonzo, T. A., Gerbing, R. B., Hall, M., Bertoch, D., Keren, R., Seif, A. E., et al. (2012) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.3241 · Cited by 49 · see citing works
Validation of healthcare administrative data for the diagnosis of epilepsy
Franchi, C., Giussani, G., Messina, P., Montesano, M., Romi, S., Nobili, A., Fortino, I., Bortolotti, A., Merlino, L. G., Beghi, E., et al. (2013) · Journal of Epidemiology & Community Health · doi:10.1136/jech-2013-202528 · Cited by 49 · see citing works
Schreye, R. D., Smets, T., Annemans, L., Deliëns, L., Gielen, B., Gendt, C. D., & Cohen, J. (2017) · Health Affairs · doi:10.1377/hlthaff.2017.0199 · Cited by 49 · see citing works
Crane, H. M., Heckbert, S. R., Drozd, D. R., Budoff, M. J., Delaney, J. A., Rodriguez, C. V., Paramsothy, P., Lober, W. B., Burkholder, G., Willig, J. H., et al. (2014) · American Journal of Epidemiology · doi:10.1093/aje/kwu010 · Cited by 48 · see citing works
Sharp, M. K., Tokalić, R., Melis, G. G., Wager, E., Altman, D. G., & Hren, D. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.11.006 · Cited by 46 · see citing works
Jay, N., Nuemi, G., Gadreau, M., & Quantin, C. (2013) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-13-130 · Cited by 47 · see citing works
Validation of algorithms in studies based on routinely collected health data: general principles
Ehrenstein, V., Hellfritzsch, M., Kahlert, J. A., Langan, S., Urushihara, H., Marinac‐Dabic, D., Lund, J. L., Sørensen, H. T., & Benchimol, E. I. (2024) · American Journal of Epidemiology · doi:10.1093/aje/kwae071 · Cited by 47 · see citing works
Self-citationFisher, B. T., Harris, T., Torp, K., Seif, A. E., Shah, A., Huang, Y.-S. V., Bailey, L. C., Kersun, L. S., Reilly, A. F., Rheingold, S. R., et al. (2012) · Medical Care · doi:10.1097/mlr.0b013e31824deff9 · Cited by 46 · see citing works
Xu, Y., Kong, S., Cheung, W. Y., Bouchard‐Fortier, A., Dort, J. C., Quan, H., Buie, E. M., McKinnon, G., & Quan, M. L. (2019) · BMC Cancer · doi:10.1186/s12885-019-5432-8 · Cited by 46 · see citing works
Shrestha, S., Dave, A. J., Losina, E., & Katz, J. N. (2016) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-016-0319-y · Cited by 45 · see citing works
The Growing Need to Understand Very Early Onset Inflammatory Bowel Disease
Zheng, H. B., de la Morena, M. T., & Suskind, D. L. (2021) · Frontiers in Immunology · doi:10.3389/fimmu.2021.675186 · Cited by 44 · see citing works
Validation of the Hospital Episode Statistics Outpatient Dataset in England
Thorn, J. C., Turner, E. L., Hounsome, L. S., Walsh, E. I., Donovan, J., Verne, J., Neal, D. E., Hamdy, F. C., Martin, R. M., & Noble, S. (2015) · PharmacoEconomics · doi:10.1007/s40273-015-0326-3 · Cited by 43 · see citing works
Carrara, G., Scirè, C. A., Zambon, A., Cimmino, M. A., Cerra, C., Caprioli, M., Cagnotto, G., Nicotra, F., Arfè, A., Migliazza, S., et al. (2015) · BMJ Open · doi:10.1136/bmjopen-2014-006029 · Cited by 42 · see citing works
Validation of multiple sclerosis diagnoses in the Swedish National Patient Register
Murley, C., Friberg, E., Hillert, J. A., Alexanderson, K. A. E., & Yang, F. (2019) · European Journal of Epidemiology · doi:10.1007/s10654-019-00558-7 · Cited by 41 · see citing works
In Reply: Big Data Research in Neurosurgery: A Critical Look at this Popular New Study Design
Oravec, C. S., Motiwala, M., Reed, K., Kondziolka, D. S., Barker, F. G., Michael, L. M., & Klimo, P. (2018) · Neurosurgery · doi:10.1093/neuros/nyy102 · Cited by 41 · see citing works
Unicompartmental Knee Arthroplasty: Survivorship and Risk Factors for Revision
Ekhtiari, S., Bozzo, A., Madden, K., Winemaker, M., Adili, A. F., & Wood, T. J. (2021) · Journal of Bone and Joint Surgery · doi:10.2106/jbjs.21.00346 · Cited by 40 · see citing works
Call to RECORD: the need for complete reporting of research using routinely collected health data
Benchimol, E. I., Langan, S., & Guttmann, A. (2012) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.09.006 · Cited by 39 · see citing works
Self-citationAccuracy of Algorithms to Identify Pulmonary Arterial Hypertension in Administrative Data
Gillmeyer, K. R., Lee, M.-M., Link, A. P., Klings, E. S., Rinne, S. T., & Wiener, R. S. (2018) · CHEST Journal · doi:10.1016/j.chest.2018.11.004 · Cited by 40 · see citing works
England, B. R., Roul, P., Mahajan, T., Singh, N., Yu, F., Sayles, H. R., Cannon, G. W., Sauer, B. C., Baker, J. F., Curtis, J. R., et al. (2019) · Arthritis Care & Research · doi:10.1002/acr.24043 · Cited by 39 · see citing works
Havard, A., Manski-Nankervis, J.-A. E., Thistlethwaite, J. E., Daniels, B. J., Myton, R., Tu, K., & Chidwick, K. (2021) · BMC Health Services Research · doi:10.1186/s12913-021-06593-z · Cited by 39 · see citing works
Accuracy of diagnosis codes to identify febrile young infants using administrative data
Aronson, P. L., Williams, D. J., Thurm, C. W., Tieder, J. S., Alpern, E. R., Nigrovic, L. E., Schondelmeyer, A. C., Balamuth, F., Myers, A., McCulloh, R. J., et al. (2015) · Journal of Hospital Medicine · doi:10.1002/jhm.2441 · Cited by 38 · see citing works
Lacasse, A., Ware, M. A., Dorais, M., Lanctôt, H., & Choinière, M. (2015) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.3820 · Cited by 37 · see citing works
Ohlmeier, C., Langner, I., Hillebrand, K., Schmedt, N., Mikolajczyk, R., Riedel, O., & Garbe, E. (2015) · BMC Public Health · doi:10.1186/s12889-015-1943-7 · Cited by 36 · see citing works
Schwarzkopf, D., Rose, N., Fleischmann-Struzek, C., Boden, B., Dorow, H., Edel, A., Friedrich, M. E., Gonnert, F. A., Götz, J., Gründling, M., et al. (2023) · Infection · doi:10.1007/s15010-023-02091-y · Cited by 35 · see citing works
Abraha, I., Montedori, A., Serraino, D., Orso, M., Giovannini, G., Scotti, V., Granata, A., Cozzolino, F., Fusco, M., & Bidoli, E. (2018) · BMJ Open · doi:10.1136/bmjopen-2017-019264 · Cited by 35 · see citing works
Kavčič, M., Fisher, B. T., Torp, K., Li, Y., Huang, Y.-S. V., Seif, A. E., Vujković, M., & Aplenc, R. (2012) · Pediatric Blood & Cancer · doi:10.1002/pbc.24402 · Cited by 35 · see citing works
Earles, A., Liu, L., Bustamante, R., Coke, P., Lynch, J. A., Messer, K. S., Martı́nez, M. E., Murphy, J. D., Williams, C. D., Fisher, D. A., et al. (2018) · JCO Clinical Cancer Informatics · doi:10.1200/cci.17.00072 · Cited by 35 · see citing works
Validated administrative data based ICD-10 algorithms for chronic conditions: A systematic review
Kuang, A., Xu, C., Southern, D. A., Sandhu, N., & Quan, H. (2024) · Journal of Epidemiology and Population Health · doi:10.1016/j.jeph.2024.202744 · Cited by 36 · see citing works
Bernatsky, S., Lix, L. M., O’Donnell, S. C., Lacaille, D. V., & CANRAD Network (2012) · The Journal of Rheumatology · doi:10.3899/jrheum.120835 · Cited by 34 · see citing works
Validation of administrative health data for the pediatric population: a scoping review
Shiff, N. J., Jama, S., Boden, C., & Lix, L. M. (2014) · BMC Health Services Research · doi:10.1186/1472-6963-14-236 · Cited by 34 · see citing works
Goueslard, K., Cottenet, J., Benzenine, É., Tubert‐Bitter, P., & Quantin, C. (2020) · BMJ Open · doi:10.1136/bmjopen-2019-035218 · Cited by 32 · see citing works
Feasibility of using administrative data to compare hospital performance in the EU
Groene, O., Kristensen, S. R., Arah, O. A., Thompson, C. A., Bartels, P. D., Suñol, R., Klazinga, N., Klazinga, N., Kringos, D. S., Lombarts, K. M. J. M. H., et al. (2014) · International Journal for Quality in Health Care · doi:10.1093/intqhc/mzu015 · Cited by 32 · see citing works
Shiff, N. J., Oen, K., Rabbani, R., & Lix, L. M. (2017) · Rheumatology International · doi:10.1007/s00296-017-3734-1 · Cited by 31 · see citing works
Newcomer, S. R., Xu, S., Kulldorff, M., Daley, M. F., FIREMAN, B. H., & Glanz, J. M. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz094 · Cited by 31 · see citing works
Validity of Diagnostic Algorithms for Cardiovascular Diseases in Japanese Health Insurance Claims
Kanaoka, K., Iwanaga, Y., Okada, K., Terasaki, S., Nishioka, Y., Nakai, M., Kamon, D., Myojin, T., Soeda, T., Noda, T., et al. (2023) · Circulation Journal · doi:10.1253/circj.cj-22-0566 · Cited by 31 · see citing works
Pandolfi, F., Guillemot, D., Watier, L., & Brun‐Buisson, C. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-058205 · Cited by 30 · see citing works
Nicholls, S. G., Langan, S., Sørensen, H. T., Petersen, I., & Benchimol, E. I. (2016) · Clinical Epidemiology · doi:10.2147/clep.s110528 · Cited by 27 · see citing works
Self-citationPrevalence of multiple sclerosis in Sardinia: A systematic cross-sectional multi-source survey
Urru, S. A. M., Antonelli, A., Sechi, G. M., & MS Working Group (2019) · Multiple Sclerosis Journal · doi:10.1177/1352458519828600 · Cited by 29 · see citing works
Distributed data networks: a blueprint for Big Data sharing and healthcare analytics
Popovic, J. R. (2016) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.13287 · Cited by 29 · see citing works
Identifying multiple myeloma patients using data from the French health insurance databases
Palmaro, A., Gauthier, M., Conte, C., Grosclaude, P., Despas, F., & Lapeyre‐Mestre, M. (2017) · Medicine · doi:10.1097/md.0000000000006189 · Cited by 28 · see citing works
Boesen, M. S., Magyari, M., Born, A. P., & Thygesen, L. C. (2018) · Clinical Epidemiology · doi:10.2147/clep.s156997 · Cited by 28 · see citing works
The validity of heart failure diagnoses in the Finnish Hospital Discharge Register
Vuori, M. A., Laukkanen, J. A., Pietilä, A., Havulinna, A. S., Kähönen, M. A. P., Salomaa, V. V., Niiranen, T., & for the FinnGen investigators (2019) · Scandinavian Journal of Public Health · doi:10.1177/1403494819847051 · Cited by 27 · see citing works
Schwartz, K. L., Chen, C., Langford, B. J., Brown, K. A., Daneman, N., Johnstone, J., Wu, J. H., Leung, V., & Garber, G. E. (2019) · PLoS ONE · doi:10.1371/journal.pone.0223097 · Cited by 27 · see citing works
A strategy for validation of variables derived from large-scale electronic health record data
Liu, L., Bustamante, R., Earles, A., Demb, J., Messer, K. S., & Gupta, S. N. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103879 · Cited by 27 · see citing works
- Authors
- Benchimol, E. I., Manuel, D. G., To, T., Griffiths, A. M., Rabeneck, L., & Guttmann, A.
- Affiliations
- University of Ottawa (CA)
- Statistics Canada (CA)
- University of Toronto (CA)
- Corresponding author
- Benchimol, Eric I.
- ORCID
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2011-08
- Volume
- 64
- Issue
- 8
- Pages
- 821-829
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W1979334594
- Cited by
- 368
- References
- 32
- Citations by year
- 2026: 20 · 2025: 13 · 2024: 25 · 2023: 29 · 2022: 21 · 2021: 30 · 2020: 28 · 2019: 40 · 2018: 27 · 2017: 27 · 2016: 27 · 2015: 21 · 2014: 25 · 2013: 16 · 2012: 14
- FWCI
- 2.92
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Clinical Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- Funders
- American College of Gastroenterology
- Child and Family Research Institute
- Public Health Agency of Canada
- Canadian Child Health Clinician Scientist Program
- Hospital for Sick Children
- Institut pour la Recherche en Santé Publique
- Canadian Institutes of Health Research
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Health Professions › Health Information Management › Medical Coding and Health Information
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- administrative health data, data validation, identification algorithm, reporting guidelines, reporting quality, diagnostic accuracy, positive predictive value, negative predictive value, misclassification bias, disease prevalence, incidence estimation, health care utilization
- MeSH terms
- Algorithms, Forms and Records Control, Health Services Research, Health Status, Hospital Records, Humans, Quality Control, Guidelines as Topic, International Classification of Diseases, Validation Studies as Topic, Translational Research, Biomedical, Checklist
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
