Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography
The DOI matches its Crossref record; 73 of the 73 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Assessing quality and susceptibility to bias is essential when interpreting primary research and conducting systematic reviews and meta-analyses. Tools for assessing quality in clinical trials are well-described but much less attention has been given to similar tools for observational epidemiological studies. METHODS: Tools were identified from a search of three electronic databases, bibliographies and an Internet search using Google. Two reviewers extracted data using a pre-piloted extraction form and strict inclusion criteria. Tool content was evaluated for domains potentially related to bias and was informed by the STROBE guidelines for reporting observational epidemiological studies. RESULTS: A total of 86 tools were reviewed, comprising 41 simple checklists, 12 checklists with additional summary judgements and 33 scales. The number of items ranged from 3 to 36 (mean 13.7). One-third of tools were designed for single use in a specific review and one-third for critical appraisal. Half of the tools provided development details, although most were proposed for future use in other contexts. Most tools included items for selection methods (92%), measurement of study variables (86%), design-specific sources of bias (86%), control of confounding (78%) and use of statistics (78%); only 4% addressed conflict of interest. The distribution and weighting of domains across tools was variable and inconsistent. CONCLUSION: A number of useful assessment tools have been identified by this report. Tools should be rigorously developed, evidence-based, valid, reliable and easy to use. There is a need to agree on critical elements for assessing susceptibility to bias in observational epidemiology and to develop appropriate evaluation tools.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Such instruments have been developed by other groups and were the subject of a recent systematic review [28].”
From The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: Guidelines for Reporting Observational Studies (2007) · cites this paper as [28]
In the citing paper
“It is not clear how weights should be assigned to each item of the scale [18].”
From Tools used to assess the quality of peer review reports: A methodological systematic review (2019) · cites this paper as [18]
In the citing paper
“The reviewers concluded that there is no ‘single obvious candidate tool for assessing quality of observational epidemiological studies’.”
From COSMOS-E: Guidance on conducting systematic reviews and meta-analyses of observational studies of etiology (2019) · cites this paper as [45]
In the citing paper
“A review identified more than 80 tools for assessing risk of bias in nonrandomised studies [45].”
From COSMOS-E: Guidance on conducting systematic reviews and meta-analyses of observational studies of etiology (2019) · cites this paper as [45]
In the citing paper
“The need for complete reporting of all relevant methodological aspects as a prerequisite to assessing the risk of bias of studies was highlighted in 10 articles, [16, 33, 35, 47, 50, 56, 86, 87, 90, 91] and was the focus of recommendations in a publication from the Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group [12].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [90]
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Meta-analysis of Observational Studies in Epidemiology<SUBTITLE>A Proposal for Reporting</SUBTITLE>
Stroup, D. F. (2000) · JAMA · doi:10.1001/jama.283.15.2008 · Cited by 20,623
Downs, S. H., & Black, N. (1998) · Journal of Epidemiology and Community Health · doi:10.1136/jech.52.6.377 · Cited by 8,252
Moher, D., Cook, D. J., Eastwood, S., Olkin, I., Rennie, D., & Stroup, D. F. (1999) · The Lancet · doi:10.1016/s0140-6736(99)04149-5 · Cited by 4,826
Whiting, P., Rutjes, A. W., Reitsma, J. B., Bossuyt, P. M., & Kleijnen, J. (2003) · BMC Medical Research Methodology · doi:10.1186/1471-2288-3-25 · Cited by 3,592
Assessing the quality of controlled clinical trials
Jüni, P., Altman, D. G., & Egger, M. (2001) · BMJ · doi:10.1136/bmj.323.7303.42 · Cited by 3,309 · cited 5×
“a (5) (5) (5) ( 5 ) N N LongneckerCoh”
InfluentialUsers' guides to the medical literature
Guyatt, G. H. (1993) · JAMA · doi:10.1001/jama.270.17.2096 · Cited by 2,970
Evaluating non-randomised intervention studies
Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D. (2003) · Health Technology Assessment · doi:10.3310/hta7270 · Cited by 2,907
Cognitive and Behavioral Outcomes of School-Aged Children Who Were Born Preterm: A Meta-analysis
Bhutta, A. T., Cleves, M. A., Casey, P. H., Cradock, M. M., & Anand, K. J. S. (2002) · JAMA · doi:10.1001/jama.288.6.728 · Cited by 2,392 · cited 1×
“Y N N 9 (22) 2 (17) 5 (15) 16 (19) Y Y N 15 (36) 6 ( 50 ) 7 (21) 28 (32)”
The Hazards of Scoring the Quality of Clinical Trials for Meta-analysis
Jüni, P. (1999) · JAMA · doi:10.1001/jama.282.11.1054 · Cited by 1,981
Low grade inflammation and coronary heart disease: prospective study and updated meta-analyses
Danesh, J. (2000) · BMJ · doi:10.1136/bmj.321.7255.199 · Cited by 1,692
A meta-analysis of physical activity in the prevention of coronary heart disease.
BERLIN, J. A., & COLDITZ, G. A. (1990) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a115704 · Cited by 1,382
Moher, D., Jadad, A. R., Nichol, G., Penman, M., Tugwell, P., & Walsh, S. (1995) · Controlled Clinical Trials · doi:10.1016/0197-2456(94)00031-w · Cited by 1,308
Occupational risk factors for shoulder pain: a systematic review
van der Windt, D. A. W. M., Thomas, E., Pope, D. P., de Winter, A. F., Macfarlane, G. J., Bouter, L. M., & Silman, A. J. (2000) · Occupational and Environmental Medicine · doi:10.1136/oem.57.7.433 · Cited by 576
Critical appraisal of the health research literature: prevalence or incidence of a health problem.
Loney, P., Chambers, L. W., BENNETT, K. J., Roberts, J. G., & Stratford, P. W. (1998) · PubMed · Cited by 550
The clinical course and prognostic factors of non-specific neck pain: a systematic review
Borghouts, J. A. J., Koes, B. W., & Bouter, L. M. (1998) · Pain · doi:10.1016/s0304-3959(98)00058-x · Cited by 533
Physical risk factors for neck pain
Ariëns, G. A., van Mechelen, W., Bongers, P. M., Bouter, L. M., & van der Wal, G. (2000) · Scandinavian Journal of Work, Environment & Health · doi:10.5271/sjweh.504 · Cited by 525
Zaza, S., Wright-De Agüero, L. K., Briss, P. A., Truman, B. I., Hopkins, D. P., Hennessy, M. H., Sosin, D. M., Anderson, L., Carande-Kulis, V. G., Teutsch, S. M., et al. (2000) · American Journal of Preventive Medicine · doi:10.1016/s0749-3797(99)00122-1 · Cited by 464
Quality of Reporting of Randomized Trials as a Measure of Methodologic Quality
Huwiler-Müntener, K. (2002) · JAMA · doi:10.1001/jama.287.21.2801 · Cited by 397 · cited 7×
“Conflict of interest n; % (range) 1; 2% ( 1 ) 1; 8% (1) 1; 3% (1) 3 (4%)”
InfluentialA systematic review of the content of critical appraisal tools
Katrak, P., Bialocerkowski, A. E., Massy-Westropp, N., Kumar, V. S., & Grimmer, K. A. (2004) · BMC Medical Research Methodology · doi:10.1186/1471-2288-4-22 · Cited by 391
Corrao, G., Bagnardi, V., Zambon, A., & Arico, S. (1999) · Addiction · doi:10.1046/j.1360-0443.1999.9410155111.x · Cited by 389
Prevention of coronary heart disease.
(1976) · BMJ · doi:10.1136/bmj.1.6014.881 · Cited by 383
A Meta-analysis of Alcohol Consumption in Relation to Risk of Breast Cancer
Longnecker, M. P. (1988) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1988.03410050072032 · Cited by 368
Critical appraisal of published research: introductory guidelines.
Fowkes, F. G., & Fulton, P. M. (1991) · BMJ · doi:10.1136/bmj.302.6785.1136 · Cited by 360
Greenland, S. (2001) · Biostatistics · doi:10.1093/biostatistics/2.4.463 · Cited by 359
Rating the quality of evidence for clinical practice guidelines
Hadorn, D. C., Baker, D., Hodges, J. S., & Hicks, N. (1996) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(96)00019-4 · Cited by 332
Manuscript Quality before and after Peer Review and Editing at Annals of Internal Medicine
Goodman, S. N., Berlin, J., Fletcher, S. W., & Fletcher, R. H. (1994) · Annals of Internal Medicine · doi:10.7326/0003-4819-121-1-199407010-00003 · Cited by 324
Stock, S. R. (1991) · American Journal of Industrial Medicine · doi:10.1002/ajim.4700190111 · Cited by 319
Users' Guides to the Medical Literature: IV. How to Use an Article About Harm
Levine, M. (1994) · JAMA · doi:10.1001/jama.1994.03510440075037 · Cited by 311
Levine, M. (1994) · JAMA · doi:10.1001/jama.271.20.1615 · Cited by 233
A systematic review of the relationship between overjet size and traumatic dental injuries
Nguyen, Q. (1999) · The European Journal of Orthodontics · doi:10.1093/ejo/21.5.503 · Cited by 224
Systematic review of population-based epidemiological studies of oro-facial pain
Macfarlane, T. V., Glenny, A.-M., & Worthington, H. V. (2001) · Journal of Dentistry · doi:10.1016/s0300-5712(01)00041-0 · Cited by 201
Instruments for Assessing the Quality of Drug Studies Published in the Medical Literature
Cho, M. K. (1994) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1994.03520020027007 · Cited by 180
Adult respiratory distress syndrome
Garber, B. G., Hébert, P. C., Yelle, J.-D., Hodder, R. V., & McGowan, J. M. (1996) · Critical Care Medicine · doi:10.1097/00003246-199604000-00023 · Cited by 154
Methodologic standards and contradictory results in case-control research
Horwitz, R. I., & Feinstein, A. R. (1979) · The American Journal of Medicine · doi:10.1016/0002-9343(79)91164-1 · Cited by 153
The scandal of poor epidemiological research: ***
von Elm, E. B., & Egger, M. (2004) · BMJ · doi:10.1136/bmj.329.7471.868 · Cited by 144
Use of check lists in assessing the statistical content of medical studies.
Gardner, M. J., Machin, D., & Campbell, M. J. (1986) · BMJ · doi:10.1136/bmj.292.6523.810 · Cited by 143
Geriatric rehabilitation following fractures in older people: a systematic review.
Cameron, Crotty, M., Currie, Finnegan, Gillespie, Gillespie, Handoll, H. H., Kurrle, Madhok, Murray, G. D., et al. (2000) · Health Technology Assessment · doi:10.3310/hta4020 · Cited by 139
Closed Fractures of the Tibial Shaft
Littenberg, B., Weinstein, L. P., McCarren, M., MEAD, T., Swiontkowski, M. F., Rudicel, S. A., & Heck, D. A. (1998) · Journal of Bone and Joint Surgery · doi:10.2106/00004623-199802000-00004 · Cited by 137
Spitzer, W. O., Lawrence, Dales, R., Hill, G., Archer, M. C., Clark, P., Abenhaim, L. L., Hardy, J., Sampalis, J. S., & Pinfold, S. P. (1990) · PubMed · Cited by 132
Campos-Outcalt, D. E., Senf, J. H., Watkins, A. J., & Bastacky, S. (1995) · Academic Medicine · doi:10.1097/00001888-199507000-00012 · Cited by 128
Interpretation of genetic association studies in complex disease
Campbell, H., & Rudan, I. (2002) · The Pharmacogenomics Journal · doi:10.1038/sj.tpj.6500132 · Cited by 125
Aid to the Evaluation of Therapeutic Studies
REISCH, J. S., Tyson, J. E., & Mize, S. G. (1989) · PEDIATRICS · doi:10.1542/peds.84.5.815 · Cited by 120
Bollini, P. (1992) · Archives of Internal Medicine · doi:10.1001/archinte.1992.00400180139023 · Cited by 119
Solomon, D. H., Bates, D. W., Panush, R. S., & Katz, J. N. (1997) · Annals of Internal Medicine · doi:10.7326/0003-4819-127-1-199707010-00009 · Cited by 118
Medial Branch Neurotomy in Management of Chronic Spinal Pain: Systematic Review of the Evidence
Manchikanti, L. (2002) · Pain Physician · doi:10.36076/ppj.2002/5/405 · Cited by 117
Secondary failure rates of measles vaccines: a metaanalysis of published studies
Anders, J., Jacobson, R. M., Poland, G. A., Jacobsen, S. J., & Wollan, P. C. (1996) · The Pediatric Infectious Disease Journal · doi:10.1097/00006454-199601000-00014 · Cited by 111
Guidelines for reading case-control studies
Lichtenstein, M. J., Mulrow, C. D., & Elwood, P. C. (1987) · Journal of Chronic Diseases · doi:10.1016/0021-9681(87)90190-1 · Cited by 102 · cited 1×
“Y N N 9 (22) 2 (17) 5 (15) 16 (19) Y Y N 15 (36) 6 (50) 7 (21) 28 ( 32 )”
Life Support Courses: Are They Effective?
Jabbour, M., Osmond, M. H., & Klassen, T. P. (1996) · Annals of Emergency Medicine · doi:10.1016/s0196-0644(96)70095-1 · Cited by 100
How to Use an Article About harm.
Kim, Y.-K. (2000) · Gajeong yihag hoeji · Cited by 94
Development of a quality assessment scale for retrospective clinical studies in pediatric surgery
Rangel, S. J., Kelsey, J. L., Colby, C. E., Anderson, J. M., & Moss, R. L. (2003) · Journal of Pediatric Surgery · doi:10.1053/jpsu.2003.50114 · Cited by 93
An addition to the controversy on sunlight exposure and melanoma risk: A meta-analytical approach
Nelemans, P. J., Rampen, F. H. J., Ruiter, D. J., & Verbeek, A. L. M. (1995) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(95)00032-1 · Cited by 79
Steinberg, E. P., Eknoyan, G., Levin, N. W., Eschbach, J. W., Golper, T. A., Owen, W. F., & Schwab, S. (2000) · American Journal of Kidney Diseases · doi:10.1053/ajkd.2000.8233 · Cited by 76
Meta-analysis of anterior veneer restorations in clinical studies
Kreulen, C. M., Creugers, N. H. J., & Meijering, A. C. (1998) · Journal of Dentistry · doi:10.1016/s0300-5712(97)00023-7 · Cited by 74
Pladevall-Vila, M., Delclos, G. L., Varas, C., Guyer, H., BruguEs-Tarradellas, J., & Anglada-Arisa, A. (1996) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a008846 · Cited by 74
(1994) · PubMed · Cited by 69
Checklist for the evaluation of research articles
DuRant, R. H. (1994) · Journal of Adolescent Health · doi:10.1016/1054-139x(94)90381-6 · Cited by 61 · cited 1×
“Case-control Cohort Cross-sectional Simple checklists n (%) Simple checklists with a judgement n (%) Scales n (%) Total n (%) Y N N 9 ( 22 ) 2 (17) 5 (15) 16 (19)”
BackgroundMeijer, R., Dettling-Ihnenfeldt, D. S., van Limbeek, J., Vermeulen, M., & de Haan, R. J. (2003) · Clinical Rehabilitation · doi:10.1191/0269215503cr644oa · Cited by 60
Prostheses for Primary Total Hip Replacement:A Critical Appraisal of the Literature
Cowley, D. E. (1995) · International Journal of Technology Assessment in Health Care · doi:10.1017/s026646230000920x · Cited by 57
Critical Appraisal of the Literature
Miser, W. F. (1999) · The Journal of the American Board of Family Medicine · doi:10.3122/jabfm.12.4.315 · Cited by 53
Ciliska, D. K., Hayward, S., Thomas, H., Mitchell, A., Dobbins, M., Underwood, J. M., Rafael, A., & Martin, E. (1996) · PubMed · Cited by 50
Margetts, B., Thompson, R. L., Key, T. J., Duffy, S. W., Nelson, M., Bingham, S. A., & Wiseman, M. J. (1995) · Nutrition and Cancer · doi:10.1080/01635589509514412 · Cited by 45
A checklist system for critical review of medical literature
KROGH, C. L. (1985) · Medical Education · doi:10.1111/j.1365-2923.1985.tb01343.x · Cited by 35
Reading research critically. II. An introduction to appraisal: assessing the evidence
Avis, M. (1994) · Journal of Clinical Nursing · doi:10.1111/j.1365-2702.1994.tb00400.x · Cited by 30
Esdaile, J. M., & Horwitz, R. I. (1986) · Journal of Chronic Diseases · doi:10.1016/0021-9681(86)90086-x · Cited by 27
Zola, P., Volpe, T., Castelli, G. L., Sismondi, P., Nicolucci, A., Parazzini, F., & Liberati, A. M. (1989) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/0360-3016(89)90498-7 · Cited by 26
Lu, K. H., & Burke, T. W. (2000) · Current Treatment Options in Oncology · doi:10.1007/s11864-000-0059-0 · Cited by 25
Evidence-based nutrition - review of nutritional epidemiological studies
Margetts, B. M., Vorster, H., & Venter, C. S. (2002) · South African Journal of Clinical Nutrition · Cited by 24
Quality of published reports of the prognosis of community-acquired pneumonia
Carson, C. A., Fine, M. J., Smith, M. A., Weissfeld, L. A., Huber, J. T., & Kapoor, W. N. (1994) · Journal of General Internal Medicine · doi:10.1007/bf02599136 · Cited by 15
Forward projection—using critical appraisal in the design of studies
Elwood, M. (2002) · International Journal of Epidemiology · doi:10.1093/ije/31.5.1071 · Cited by 14
Beel-Bates, C. (1991) · PubMed · Cited by 14
Critical appraisal of prognostic evidence: practical rules.
Carneiro, A. V. (2002) · PubMed · Cited by 11
Blood glucose and diabetic retinopathy: A critical appraisal of new evidence
Mulrow, C. D., & Lichtenstein, M. J. (1986) · Journal of General Internal Medicine · doi:10.1007/bf02599803 · Cited by 8
Bertelsmann, A. (1993) · Springer eBooks · doi:10.1007/978-3-642-78713-3_8 · Cited by 3
Research works printed but not linked: 7
- Carneiro, 2002, Revista portuguesa de cardiologia : org o oficial da Sociedade Portuguesa de Cardiologia = Portuguese journal of cardiology : an official journal of the Portuguese Society of Cardiology, 21, 891
- 1981, Canadian Medical Association Journal, 124, 985
- 2002, SOUTH AFRICAN J CLIN NUTR, 15, 68
- 2000, Chronic diseases in Canada, 19, 170
- Ciliska, 1996, Canadian journal of public health. Revue canadienne de sant publique, 87, 193
- 1994, Canadian journal of public health. Revue canadienne de sant publique, 85, S8
- Spitzer, 1990, Clinical and investigative medicine. M decine clinique et experimentale, 13, 17
Cited by
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000100 · Cited by 28,164
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040296 · Cited by 23,020
“Such instruments have been developed by other groups and were the subject of a recent systematic review [28].”
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2007) · The Lancet · doi:10.1016/s0140-6736(07)61602-x · Cited by 21,320
“Such instruments have been developed by other groups and were the subject of a recent systematic review (28).”
ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions
Sterne, J. A., Hernán, M. A., Reeves, B. C., Savović, J., Berkman, N. D., Viswanathan, M., Henry, D., Altman, D. G., Ansari, M. T., Boutron, I., et al. (2016) · BMJ · doi:10.1136/bmj.i4919 · Cited by 20,329
Self-citationvon Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.11.008 · Cited by 17,893
“Such instruments have been developed by other groups and were the subject of a recent systematic review [ 28 ].”
Metadata
- Authors
- Sanderson, S., Tatt, I. D., & Higgins, J. P.
- Affiliations
- University of Cambridge (GB)
- F. Hoffmann-La Roche AG (CH)
- MRC Biostatistics Unit (GB)
- Corresponding author
- Sanderson, S.
- ORCID
- Higgins, J. P.
- Venue
- International Journal of Epidemiology
- Publisher
- Oxford University Press (OUP)
- Published
- 2007-04-30
- Volume
- 36
- Issue
- 3
- Pages
- 666-676
- ISSN
- 0300-5771 (ISSN-L)
- 1464-3685
- OpenAlex
- W2171251365
- Cited by
- 1,531
- References
- 73
- Citations by year
- 2026: 22 · 2025: 38 · 2024: 42 · 2023: 59 · 2022: 80 · 2021: 96 · 2020: 101 · 2019: 105 · 2018: 125 · 2017: 104 · 2016: 150 · 2015: 118 · 2014: 122 · 2013: 111 · 2012: 96
- FWCI
- 7.09
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- International Journal of Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- quality assessment tools, bias assessment, observational studies, critical appraisal, checklists, rating scales, STROBE, confounding adjustment, selection bias, measurement bias, conflicts of interest
- MeSH terms
- Humans, Research Design, Bias, Epidemiologic Studies, Evidence-Based Medicine, Observation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.