A framework for evaluation of secondary data sources for epidemiological research
Abstract
BACKGROUND: As part of the development in information technology, increasing amounts of health care data are available for epidemiological research. METHODS: In this review, we discuss the following factors affecting the value of secondary data in research: 1) completeness of registration of individuals, 2) the accuracy and degree of completeness of the registered data, 3) the size of the data source, 4) the registration period, 5) data accessibility, availability and cost, 6) data format, and 7) possibilities of linkage with other data sources (record linkage). RESULTS AND CONCLUSION: The importance of these issues depends on the use of the data and on the problems they have to address. If the evaluation is satisfactory with respect to the above-mentioned factors relevant to the particular study, the data source could be a very cost-effective way of solving the research problem.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The most common reference standards are medical records, surveys of patients or practitioners, and clinical registries.
“The most common reference standards are medical records, surveys of patients or practitioners, and clinical registries [5,34].”
From The Reporting of Studies Conducted Using Observational Routinely-Collected Health Data (RECORD) statement. (2015) · cites this paper as [34]
Measures of accuracy are similar to those reported in diagnostic test studies, including sensitivity, specificity, positive and negative predictive values, or kappa coefficients.
“Measures of accuracy are similar to those reported in diagnostic test studies, including sensitivity, specificity, positive and negative predictive values, or kappa coefficients [5,34].”
From The Reporting of Studies Conducted Using Observational Routinely-Collected Health Data (RECORD) statement. (2015) · cites this paper as [34]
Cites
Discrete Multivariate Analysis: Theory and Practice.
Beniger, J. R., Bishop, Y. M. M., Feinberg, S. E., Holland, P. W., Light, R. J., & Mosteller, F. C. (1975) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2063625 · Cited by 5,119 · see citing works
Limitations of the Application of Fourfold Table Analysis to Hospital Data
Berkson, J. (1946) · Biometrics Bulletin · doi:10.2307/3002000 · Cited by 1,810 · see citing works
Hulley, S. B. (1997) · Cited by 1,753 · see citing works
Automatic Linkage of Vital Records
Newcombe, H. B., Kennedy, J. M., Axford, S., et al. (1959) · Science · doi:10.1126/science.130.3381.954 · Cited by 782 · see citing works
Clinical Epidemiology and Biostatistics
Kramer, M. S. (1988) · Springer eBooks · doi:10.1007/978-3-642-61372-2 · Cited by 332 · see citing works
USE OF MEDICAID DATA FOR PHARMACOEPIDEMIOLOGY
Ray, W. A., & Griffin, M. R. (1989) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a115198 · Cited by 316 · see citing works
Registries and Administrative Data
Roos, L. L., Mustard, C., Nicol, J. P., Comm, B. R. P., McLerran, D., Malenka, D. J., Young, T. K., & Cohen, M. M. (1993) · Medical Care · doi:10.1097/00005650-199303000-00002 · Cited by 298 · see citing works
A comparison of administrative versus clinical data: coronary artery bypass surgery as an example
Romano, P. S. (1994) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(94)90006-x · Cited by 198 · see citing works
A COMPARISON OF PREGNANCY HISTORY RECALL AND MEDICAL RECORDS
Tilley, B. C., BARNES, A. B., Bergstralh, E. J., et al. (1985) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a113997 · Cited by 189 · see citing works
Medicaid data as a resource for epidemiologic studies: Strengths and limitations
Bright, R. A., Avorn, J., & Everitt, D. E. (1989) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(89)90158-3 · Cited by 188 · see citing works
Using administrative data for longitudinal research: Comparisons with primary data collection
Roos, L. L., Nicol, J. P., & Cageorge, S. M. (1987) · Journal of Chronic Diseases · doi:10.1016/0021-9681(87)90095-6 · Cited by 177 · see citing works
Roos, L. L., Roos, N. P., Cageorge, S. M., & Nicol, J. P. (1982) · Medical Care · doi:10.1097/00005650-198203000-00003 · Cited by 162 · see citing works
Overcoming potential pitfalls in the use of Medicare data for epidemiologic research.
Fisher, E. S., Baron, J. A., Malenka, D. J., BARRETT, J. A., & Bubolz, T. A. (1990) · American Journal of Public Health · doi:10.2105/ajph.80.12.1487 · Cited by 146 · see citing works
Completeness of Cancer Registration in Denmark 1943–1966 and Efficacy of Record Linkage Procedures
Storm, H. H. (1988) · International Journal of Epidemiology · doi:10.1093/ije/17.1.44 · Cited by 141 · see citing works
REGISTRY EVALUATION METHODS: A REVIEW AND CASE STUDY1
Goldberg, J., Gelfand, H. M., & Levy, P. S. (1980) · Epidemiologic Reviews · doi:10.1093/oxfordjournals.epirev.a036224 · Cited by 133 · see citing works
Almdal, T., & Sørensen, T. I. A. (1991) · PubMed · Cited by 130 · see citing works
Comparing clinical information with claims data: Some similarities and differences
Roos, L. L. (1991) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(91)90050-j · Cited by 100 · see citing works
Assessing the reliability of epidemiologic data obtained from medical records
Horwitz, R. I., & Yu, E. (1984) · Journal of Chronic Diseases · doi:10.1016/0021-9681(84)90015-8 · Cited by 100 · see citing works
The Use of Large Data Bases in Health Care Studies
Connell, F. A., Diehr, P. H., & Hart, L. G. (1987) · Annual Review of Public Health · doi:10.1146/annurev.pu.08.050187.000411 · Cited by 96 · see citing works
Assessing data quality: A computerized approach
Roos, L. L., Sharp, S. M., & Wajda, A. (1989) · Social Science & Medicine · doi:10.1016/0277-9536(89)90145-7 · Cited by 93 · see citing works
Rodríguez, L. A. G., Walker, A. M., & Perez‐Gutthann, S. (1992) · Epidemiology · doi:10.1097/00001648-199207000-00008 · Cited by 91 · see citing works
Pryor, D. B., Califf, R. M., Harrell, F. E., et al. (1985) · Medical Care · doi:10.1097/00005650-198505000-00020 · Cited by 86 · see citing works
Epidemiologic Uses of Medicare Data
Lauderdale, D. S., Furner, S. E., Miles, T. P., et al. (1993) · Epidemiologic Reviews · doi:10.1093/oxfordjournals.epirev.a036123 · Cited by 85 · see citing works
Benign Anal Lesions and the Risk of Anal Cancer
Frisch, M., Olsen, J. H., Bautz, A., et al. (1994) · New England Journal of Medicine · doi:10.1056/nejm199408043310504 · Cited by 84 · see citing works
Self-citationSpeltz, M. L., Morton, K., Goodell, E. W., & Clarren, S. K. (1993) · The Cleft Palate-Craniofacial Journal · doi:10.1597/1545-1569(1993)030 · Cited by 84 · see citing works
Interobserver variability in medical record review: An epidemiological study of asthma
Beard, C. M., Yunginger, J. W., Reed, C. E., O’Connell, E. J., & Silverstein, M. D. (1992) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(92)90117-6 · Cited by 74 · see citing works
Tennis, P. S. (1993) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(93)90049-7 · Cited by 73 · see citing works
Anaphylactic shock occurring outside hospitals
Sørensen, H. T., Nielsen, B., & Nielsen, J. Ø. (1989) · Allergy · doi:10.1111/j.1398-9995.1989.tb01071.x · Cited by 72 · see citing works
Self-citationThe art and science of record linkage: Methods that work with few identifiers
Roos, L. L., Wajda, A., & Nicol, J. P. (1986) · Computers in Biology and Medicine · doi:10.1016/0010-4825(86)90061-2 · Cited by 70 · see citing works
Headache classification and epidemiology
Olesen, J. (1994) · Raven Press eBooks · Cited by 69 · see citing works
Frontiers in Headache Research
Marcus, D. A. (1995) · Clinical Journal of Pain · doi:10.1097/00002508-199506000-00019 · Cited by 62 · see citing works
Impact of Claims Data Research on Clinical Practice
Steinberg, E. P., Whittle, J., & Anderson, G. F. (1990) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462300000805 · Cited by 60 · see citing works
Efficient Study Designs to Assess the Accuracy of Screening Tests
Irwig, L., Glasziou, P., Berry, G., et al. (1994) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a117323 · Cited by 55 · see citing works
Confidential medical records and epidemiological research.
Knox, E. G. (1992) · BMJ · doi:10.1136/bmj.304.6829.727 · Cited by 53 · see citing works
Validity of rheumatoid arthritis diagnoses listed in the saskatchewan hospital separations database
Tennis, P. S. (1993) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(93)90048-6 · Cited by 49 · see citing works
Sørensen, H. T., & Larsen, B. O. (1994) · Journal of Medical Systems · doi:10.1007/bf00999322 · Cited by 46 · see citing works
Self-citationIncreasing the accessibility of data
Smith, G. D. (1994) · BMJ · doi:10.1136/bmj.308.6943.1519 · Cited by 43 · see citing works
Observer variation in the classification of information from medical records
Boyd, N. F., Pater, J. L., Ginsburg, A. D., et al. (1979) · Journal of Chronic Diseases · doi:10.1016/0021-9681(79)90088-2 · Cited by 40 · see citing works
Brownson, R. C., Davis, J. R., Chang, J. C., et al. (1989) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a115174 · Cited by 39 · see citing works
Aspects of confounding and effect modification in the assessment of occupational cancer risk
Axelson, O. (1980) · Journal of Toxicology and Environmental Health · doi:10.1080/15287398009529933 · Cited by 38 · see citing works
Early treatment of meningococcal disease.
Sørensen, H. T., Møller-Petersen, J., Krarup, H. B., Pedersen, H., Hansen, H., & Hamburger, H. J. (1992) · BMJ · doi:10.1136/bmj.305.6856.774-a · Cited by 34 · see citing works
Self-citationEuropean directive on confidential data: a threat to epidemiology
Lynge, E. (1994) · BMJ · doi:10.1136/bmj.308.6927.490 · Cited by 34 · see citing works
Diagnostic problems with meningococcal disease in general practice
Sørensen, H. T. (1992) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(92)90169-n · Cited by 26 · see citing works
Self-citationHealth care data, the epidemiologist's sand: Comments on the quantity and quality of data
Hierholzer, W. J. (1991) · The American Journal of Medicine · doi:10.1016/0002-9343(91)90339-y · Cited by 26 · see citing works
Ht, S., Rasmussen, H. H., Jf, M.-P., et al. (1992) · PubMed · Cited by 25 · see citing works
Increase in incidence of disease due to diagnostic drift: primary liver cancer in Denmark, 1943-85.
Anderson, I. B., Sorensen, T. I., & PRENER, A. (1991) · BMJ · doi:10.1136/bmj.302.6774.437 · Cited by 24 · see citing works
A COMPENDIUM OF PUBLIC HEALTH DATA SOURCES
Gable, C. B. (1990) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a115513 · Cited by 24 · see citing works
Confidentiality in the cancer registry
Coleman, M. P., Muir, C. S., & Ménégoz, F. (1992) · British Journal of Cancer · doi:10.1038/bjc.1992.424 · Cited by 22 · see citing works
A method for the validation of data in a register
Stone, D. H. (1986) · Public Health · doi:10.1016/s0033-3506(86)80054-3 · Cited by 21 · see citing works
rd, W. H., & Hughes, B. (1985) · PubMed · Cited by 20 · see citing works
Medical innovation at the crossroads.
Laubach, G. D., & Ac, G. (1995) · PubMed · Cited by 19 · see citing works
Interobserver and intraobserver reliability in the collection of emergency medical services data.
Herrmann, N., Cayten, C. G., Senior, J., Staroscik, R. N., Walsh, S., & Woll, M. (1980) · PubMed Central · Cited by 14 · see citing works
Data Bases, Data Banks and Data Dredging: The Agony and the Ecstasy
Feinleib, M. (1984) · Journal of Chronic Diseases · doi:10.1016/0021-9681(84)90048-1 · Cited by 14 · see citing works
New draft on European directive on confidential data
Lynge, E. (1995) · BMJ · doi:10.1136/bmj.310.6986.1024 · Cited by 13 · see citing works
Strengths and Weaknesses of Health Insurance Data Systems for Assessing Outcomes
Gelijns, A. C. (1990) · Cited by 12 · see citing works
RE: “A COMPENDIUM OF PUBLIC HEALTH DATA SOURCES”
Sørensen, H. T. (1992) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a116287 · Cited by 9 · see citing works
Self-citationOne-year follow-up after the first prescription of strong analgesics outside hospital
Sørensen, H. T., Ejlersen, E., Dahlerup, J. F., Rasmussen, H. H., Hamburger, H. J., & Olesen, F. (1994) · International Journal of Risk & Safety in Medicine · doi:10.3233/jrs-1994-4304 · Cited by 4 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Benchimol, E. I., Smeeth, L., Guttmann, A., Harron, K., Moher, D., Petersen, I., Sørensen, H. T., von Elm, E., Langan, S. M., & RECORD Working Committee (2015) · PLOS Medicine · doi:10.1371/journal.pmed.1001885 · Cited by 5,749 · see citing works
Self-citationExternal review and validation of the Swedish national inpatient register
Ludvigsson, J. F., Andersson, E., Ekbom, A., Feychting, M., Kim, J.-L., Reuterwall, C., Heurgren, M., & Olausson, P. O. (2011) · BMC Public Health · doi:10.1186/1471-2458-11-450 · Cited by 5,055 · see citing works
Moher, D., Schulz, K. F., & Altman, D. G. (2001) · BMC Medical Research Methodology · doi:10.1186/1471-2288-1-2 · Cited by 2,755 · see citing works
STARD 2015 guidelines for reporting diagnostic accuracy studies: explanation and elaboration
Cohen, J. F., Korevaar, D. A., Altman, D. G., Bruns, D. E., Gatsonis, C. A., Hooft, L., Irwig, L., Levine, D., Reitsma, J. B., de Vet, H. C. W., et al. (2016) · BMJ Open · doi:10.1136/bmjopen-2016-012799 · Cited by 2,570 · see citing works
Past and Future Burden of Inflammatory Bowel Diseases Based on Modeling of Population-Based Data
Coward, S., Clement, F., Benchimol, E. I., Bernstein, C. N., Avina-Zubieta, J. A., Bitton, A., Carroll, M. W., Hazlewood, G., Jacobson, K., Jelinski, S., et al. (2019) · Gastroenterology · doi:10.1053/j.gastro.2019.01.002 · Cited by 449 · see citing works
Data linkage: a powerful research tool with potential problems
Bohensky, M. A., Jolley, D., Sundararajan, V., Evans, S., Pilcher, D. V., Scott, I., & Brand, C. A. (2010) · BMC Health Services Research · doi:10.1186/1472-6963-10-346 · Cited by 234 · see citing works
Gedeborg, R., Furebring, M., & Michaëlsson, K. (2007) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.05.013 · Cited by 80 · see citing works
Catalog of 199 register-based definitions of chronic conditions
Hvidberg, M. F., Johnsen, S. P., Glümer, C., Petersen, K. D., Olesen, A. V., & Ehlers, L. (2016) · Scandinavian Journal of Public Health · doi:10.1177/1403494816641553 · Cited by 63 · see citing works
Identification of incident injuries in hospital discharge registers
Gedeborg, R., Engquist, H., Berglund, L., & Michaëlsson, K. (2008) · Epidemiology · doi:10.1097/ede.0b013e318181319e · Cited by 61 · see citing works
- Authors
- SØRENSEN, H. T., SABROE, S., & OLSEN, J.
- Affiliations
- Aarhus University (DK)
- Corresponding author
- SØRENSEN, HENRIK TOFT
- Venue
- International Journal of Epidemiology
- Publisher
- Oxford University Press (OUP)
- Published
- 1996
- Volume
- 25
- Issue
- 2
- Pages
- 435-442
- ISSN
- 0300-5771 (ISSN-L)
- 1464-3685
- OpenAlex
- W2153097236
- Cited by
- 455
- References
- 57
- Citations by year
- 2026: 6 · 2025: 8 · 2024: 15 · 2023: 10 · 2022: 17 · 2021: 9 · 2020: 17 · 2019: 20 · 2018: 27 · 2017: 20 · 2016: 29 · 2015: 31 · 2014: 32 · 2013: 32 · 2012: 33
- FWCI
- 14.72
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- International Journal of Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Health Professions › Health Information Management › Medical Coding and Health Information
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Keywords
- secondary data sources, health data, data source evaluation, data quality, completeness, accuracy, data linkage, data accessibility, data formats, cost-effectiveness, epidemiological studies
- MeSH terms
- Confidentiality, Cost-Benefit Analysis, Data Interpretation, Statistical, Epidemiologic Methods, Humans, Medical Record Linkage, Registries, Research, Sensitivity and Specificity, Reproducibility of Results, Bias, Databases, Factual
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
