Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography
The DOI matches its Crossref record; 73 of the 73 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Assessing quality and susceptibility to bias is essential when interpreting primary research and conducting systematic reviews and meta-analyses. Tools for assessing quality in clinical trials are well-described but much less attention has been given to similar tools for observational epidemiological studies. METHODS: Tools were identified from a search of three electronic databases, bibliographies and an Internet search using Google. Two reviewers extracted data using a pre-piloted extraction form and strict inclusion criteria. Tool content was evaluated for domains potentially related to bias and was informed by the STROBE guidelines for reporting observational epidemiological studies. RESULTS: A total of 86 tools were reviewed, comprising 41 simple checklists, 12 checklists with additional summary judgements and 33 scales. The number of items ranged from 3 to 36 (mean 13.7). One-third of tools were designed for single use in a specific review and one-third for critical appraisal. Half of the tools provided development details, although most were proposed for future use in other contexts. Most tools included items for selection methods (92%), measurement of study variables (86%), design-specific sources of bias (86%), control of confounding (78%) and use of statistics (78%); only 4% addressed conflict of interest. The distribution and weighting of domains across tools was variable and inconsistent. CONCLUSION: A number of useful assessment tools have been identified by this report. Tools should be rigorously developed, evidence-based, valid, reliable and easy to use. There is a need to agree on critical elements for assessing susceptibility to bias in observational epidemiology and to develop appropriate evaluation tools.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Such instruments have been developed by other groups and were the subject of a recent systematic review [28].”
From The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: Guidelines for Reporting Observational Studies (2007) · cites this paper as [28]
In the citing paper
“It is not clear how weights should be assigned to each item of the scale [18].”
From Tools used to assess the quality of peer review reports: A methodological systematic review (2019) · cites this paper as [18]
In the citing paper
“The reviewers concluded that there is no ‘single obvious candidate tool for assessing quality of observational epidemiological studies’.”
From COSMOS-E: Guidance on conducting systematic reviews and meta-analyses of observational studies of etiology (2019) · cites this paper as [45]
In the citing paper
“A review identified more than 80 tools for assessing risk of bias in nonrandomised studies [45].”
From COSMOS-E: Guidance on conducting systematic reviews and meta-analyses of observational studies of etiology (2019) · cites this paper as [45]
In the citing paper
“The need for complete reporting of all relevant methodological aspects as a prerequisite to assessing the risk of bias of studies was highlighted in 10 articles, [16, 33, 35, 47, 50, 56, 86, 87, 90, 91] and was the focus of recommendations in a publication from the Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group [12].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [90]
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Meta-analysis of Observational Studies in Epidemiology<SUBTITLE>A Proposal for Reporting</SUBTITLE>
Stroup, D. F. (2000) · JAMA · doi:10.1001/jama.283.15.2008 · Cited by 20,623
Downs, S. H., & Black, N. (1998) · Journal of Epidemiology and Community Health · doi:10.1136/jech.52.6.377 · Cited by 8,252
Moher, D., Cook, D. J., Eastwood, S., Olkin, I., Rennie, D., & Stroup, D. F. (1999) · The Lancet · doi:10.1016/s0140-6736(99)04149-5 · Cited by 4,826
Whiting, P., Rutjes, A. W., Reitsma, J. B., Bossuyt, P. M., & Kleijnen, J. (2003) · BMC Medical Research Methodology · doi:10.1186/1471-2288-3-25 · Cited by 3,592
Assessing the quality of controlled clinical trials
Jüni, P., Altman, D. G., & Egger, M. (2001) · BMJ · doi:10.1136/bmj.323.7303.42 · Cited by 3,309 · cited 5×
“a (5) (5) (5) ( 5 ) N N LongneckerCoh”
InfluentialUsers' guides to the medical literature
Guyatt, G. H. (1993) · JAMA · doi:10.1001/jama.270.17.2096 · Cited by 2,970
Evaluating non-randomised intervention studies
Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D. (2003) · Health Technology Assessment · doi:10.3310/hta7270 · Cited by 2,907
Cognitive and Behavioral Outcomes of School-Aged Children Who Were Born Preterm: A Meta-analysis
Bhutta, A. T., Cleves, M. A., Casey, P. H., Cradock, M. M., & Anand, K. J. S. (2002) · JAMA · doi:10.1001/jama.288.6.728 · Cited by 2,392 · cited 1×
“Y N N 9 (22) 2 (17) 5 (15) 16 (19) Y Y N 15 (36) 6 ( 50 ) 7 (21) 28 (32)”
The Hazards of Scoring the Quality of Clinical Trials for Meta-analysis
Jüni, P. (1999) · JAMA · doi:10.1001/jama.282.11.1054 · Cited by 1,981
Low grade inflammation and coronary heart disease: prospective study and updated meta-analyses
Danesh, J. (2000) · BMJ · doi:10.1136/bmj.321.7255.199 · Cited by 1,692
Research works printed but not linked: 7
- Carneiro, 2002, Revista portuguesa de cardiologia : org o oficial da Sociedade Portuguesa de Cardiologia = Portuguese journal of cardiology : an official journal of the Portuguese Society of Cardiology, 21, 891
- 1981, Canadian Medical Association Journal, 124, 985
- 2002, SOUTH AFRICAN J CLIN NUTR, 15, 68
- 2000, Chronic diseases in Canada, 19, 170
- Ciliska, 1996, Canadian journal of public health. Revue canadienne de sant publique, 87, 193
- 1994, Canadian journal of public health. Revue canadienne de sant publique, 85, S8
- Spitzer, 1990, Clinical and investigative medicine. M decine clinique et experimentale, 13, 17
Cited by
Items 601–700 of 1,006
Treanor, C., & Donnelly, M. (2012) · BMC Health Services Research · doi:10.1186/1472-6963-12-316 · Cited by 63
Fuglkjær, S., Dissing, K. B., & Hestbæk, L. (2017) · BMC Musculoskeletal Disorders · doi:10.1186/s12891-017-1771-2 · Cited by 64
“This is supported by ‘Study Participation’ being the most frequently used parameter across different tools of evaluation of quality and susceptibility to bias in observational studies [26].”
MethodA systematic review of viral exposures as a risk for rheumatoid arthritis
Kudaeva, F. M., Speechley, M. R., & Pope, J. E. (2018) · Seminars in Arthritis and Rheumatism · doi:10.1016/j.semarthrit.2018.03.011 · Cited by 63
Manchikanti, L., Hirsch, J. A., Heavner, J. E., Cohen, S. P., Benyamin, R. M., Sehgal, N., Falco, F. J. E., Vallejo, R., Onyewu, O., Zhu, J., et al. (2014) · Pain Physician · doi:10.36076/ppj.2014/17/e291 · Cited by 63
Methods used to estimate the size of the owned cat and dog population: a systematic review
Downes, M., Dean, R. S., Stavisky, J., Adams, V. J., Grindlay, D. J. C., & Brennan, M. L. (2013) · BMC Veterinary Research · doi:10.1186/1746-6148-9-121 · Cited by 63
“There is a lack of critical appraisal tools available specifically aimed at cross sectional studies [81,82].”
Adverse Events During Intrahospital Transport of Critically Ill Children: A Systematic Review
Haydar, B., Baetzel, A. E., Elliott, A. B., MacEachern, M., Kamal, A., & Christensen, R. E. (2019) · Anesthesia & Analgesia · doi:10.1213/ane.0000000000004585 · Cited by 62
Impact of Cerebral Palsy on Health-Related Physical Fitness in Adults: Systematic Review
Hombergen, S. P., Huisstede, B. M. A., Streur-Kranenburg, M., Stam, H. J., Slaman, J., Bussmann, J. B. J., & van den Berg-Emons, R. J. G. (2012) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2011.11.032 · Cited by 62
Vargas‐Uricoechea, H., Meza-Cabrera, I. A., & Chaparro, J. A. H. (2017) · Thyroid Research · doi:10.1186/s13044-017-0037-2 · Cited by 62
“Those factors were evaluated with the survey administered to the study subjects that had been previously referred for tests to rule out malignancies, so these participants were more likely to recall past exposures (accurate or vague) potentially leading to a “recall bias” [30, 33].”
Neuropsychological Predictors of Outcome Following Traumatic Brain Injury in Adults: a Meta-Analysis
Allanson, F., Pestell, C., Gignac, G. E., Yeo, Y. X., & Weinborn, M. (2017) · Neuropsychology Review · doi:10.1007/s11065-017-9353-5 · Cited by 62
The Effects of Moderate Physical Exercise on Adult Cognition: A Systematic Review
Fernandes, R. M., Correa, M. G., dos Santos, M. A. R., Almeida, A. P. C. P. S. C., Fagundes, N. C. F., Maia, L. C., & Lima, R. R. (2018) · Frontiers in Physiology · doi:10.3389/fphys.2018.00667 · Cited by 63
“, 1997), in addition to evaluating the quality of the information and the risk of bias in the articles (Sanderson et al., 2007).”
Background(2017) · Europe PMC (PubMed Central) · Cited by 61
Hogan, C., Corbett, J.-A., Ashton, S., Perraton, L. G., Frame, R. A., & Dakic, J. G. (2020) · The American Journal of Sports Medicine · doi:10.1177/0363546520968508 · Cited by 62
“The Downs and Black checklist was selected on the basis of its reliability, validity, and suitability for the appraisal of observational studies.(14,47) The checklist was modified to include only the criteria relevant for observational studies, and item 27 was removed for its ambiguity.”
MethodSensory Processing Impairments in Children with Developmental Coordination Disorder
Tran, H.-T., Li, Y., Lin, H., Lee, S., & Wang, P.-J. (2022) · Children · doi:10.3390/children9101443 · Cited by 61
Refractive Errors and Age-related Macular Degeneration: A Systematic Review and Meta-Analysis
Pan, C. W., Ikram, M. K., Cheung, C. Y.-L., Choi, H.-W., Cheung, C.-M. G., Jonas, J. B., Saw, S., & Wong, T.-Y. (2013) · Ophthalmology · doi:10.1016/j.ophtha.2013.03.028 · Cited by 61
The Role of Observational Investigations in Comparative Effectiveness Research
Marko, N. F., & Weil, R. J. (2010) · Value in Health · doi:10.1111/j.1524-4733.2010.00786.x · Cited by 61
Rocks, T., Pelly, F. E., & Wilkinson, P. (2014) · Journal of the Academy of Nutrition and Dietetics · doi:10.1016/j.jand.2013.11.022 · Cited by 61
Tanvetyanon, T., Finley, D. J., Fabian, T., Riquet, M., Voltolini, L., Kocaturk, C. I., Fulp, W. J., Cerfolio, R. J., Park, B. J., & Robinson, L. A. (2012) · Annals of Oncology · doi:10.1093/annonc/mds495 · Cited by 61
“Although no uniform agreement exists regarding the instrument to assess the quality of study reporting, for an observational study, it is generally recommended that relevant details regarding participant selection and measurement of key variables be assessed [14, 20].”
BackgroundAcquavella, J., Garabrant, D. H., Marsh, G. M., Sorahan, T., & Weed, D. L. (2016) · Critical Reviews in Toxicology · doi:10.1080/10408444.2016.1214681 · Cited by 61
“exposure–response), whether the temporal relationship between exposure and effect is considered appropriate, and whether the results are statistically robust enough to rule out chance as an explanation (Hill 1965; Bhopal 2002; Aschengrau & Seage 2003a, 2003b; Sanderson et al. 2007).”
BackgroundMethodAlhaboby, Z. A., Barnes, J., Evans, H., & Short, E. (2017) · Trauma Violence & Abuse · doi:10.1177/1524838017717743 · Cited by 61
Armijo-Olivo, S., Fuentes, J., Ospina, M., Saltaji, H., & Hartling, L. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-116 · Cited by 60
Applying the ROBINS-I tool to natural experiments: an example from public health
Thomson, H., Craig, P., Hilton-Boon, M., Campbell, M., & Katikireddi, S. V. (2018) · Systematic Reviews · doi:10.1186/s13643-017-0659-4 · Cited by 60
“There are many tools available to assess study quality, [4] many of which focus on methodological quality and potentially conflate issues of internal and external validity which is problematic.”
BackgroundBrance, K., Chatzimpyros, V., & Bentall, R. P. (2022) · Clinical Psychology Review · doi:10.1016/j.cpr.2022.102216 · Cited by 60
Brück, C. C., Mooldijk, S. S., Kuiper, L. M., Sambou, M. L., Licher, S., Mattace-Raso, F. U. S., & Wolters, F. J. (2025) · BMJ · doi:10.1136/bmj-2024-080636 · Cited by 60
Biswas, S., Moghaddam, N. G., & Tickle, A. (2014) · International Journal of Developmental Disabilities · doi:10.1179/2047387714y.0000000043 · Cited by 60
“A numerical scoring system for assessing bias was not used as the accuracy and reliability of weighting individual components remains unclear (Sanderson et al. 2007).”
BackgroundMethodInfluentialInjury During U.S. Army Basic Combat Training
Bulzacchelli, M. T., Sulsky, S. I., Rodríguez-Monguió, R., Karlsson, L. H., & Hill, M. O. T. (2014) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2014.08.008 · Cited by 60
Hope, H. F., Bluett, J., Barton, A. C., Hyrich, K. L., Cordingley, L., & Verstappen, S. M. M. (2016) · RMD Open · doi:10.1136/rmdopen-2015-000171 · Cited by 60
“Second, the QA tool used was bespoke to the current review and not a validated measure; however, the domains of the tool were based on a systematic review of quality measures for observational studies.(19) Further, the QA tool is presented to guide the reader, and not to exclude articles deemed low quality.”
MethodQiu, D., Li, Y., Li, L., He, J., Ouyang, F., & Xiao, S. (2021) · European Psychiatry · doi:10.1192/j.eurpsy.2021.24 · Cited by 60
“Two independent reviewers (J.H. and F.Y.O.Y.) used the established guidelines, the Loney criteria, to evaluate the methodological quality of the included studies, which has been widely used to evaluate observational studies [23,24].”
Fischer, L. S., Asher, A., Stein, R., Becasen, J., Doreson, A., Mermin, J., Meltzer, M. I., & Edlin, B. R. (2025) · BMC Public Health · doi:10.1186/s12889-025-22210-8 · Cited by 59
“We report results separately for each question because assigning a value to the relative importance of each item would have been arbitrary or subjective [24].”
Larvin, H., Gao, C., Kang, J. X., Aggarwal, V. R., Pavitt, S. H., & Wu, J. (2023) · Age and Ageing · doi:10.1093/ageing/afad015 · Cited by 59
“Quality assessment tools for observational studies can be contentious [19]; therefore, this review employed the Risk of Bias in Non-Randomised Studies of Interventions (ROBINS-I) recommended by Cochrane to determine the risk of bias in cohort and longitudinal observational studies [20].”
MethodSex Differences in Childhood Sexual Abuse and Suicide‐Related Behaviors
Rhodes, A. E., Boyle, M. H., Tonmyr, L., Wekerle, C., Goodman, D., Leslie, B., Mironova, P., Bethell, J., & Manion, I. (2011) · Suicide and Life-Threatening Behavior · doi:10.1111/j.1943-278x.2011.00025.x · Cited by 45
Vadiveloo, M. K., Dixon, L. B., & Parekh, N. (2013) · British Journal Of Nutrition · doi:10.1017/s0007114512006150 · Cited by 59
“in the International Journal of Epidemiology ((17)).”
Umscheid, C. A., Agarwal, R. K., & Brennan, P. J. (2010) · American Journal of Infection Control · doi:10.1016/j.ajic.2009.12.005 · Cited by 59
Carter, G. L., Clover, K. A., Britton, B., Mitchell, A. J., White, M., McLeod, N., Denham, J. W., & Lambert, S. (2014) · Cancer Treatment Reviews · doi:10.1016/j.ctrv.2014.11.001 · Cited by 59
“There is no standard method or single tool for assessing quality and susceptibility to bias in observational studies [17], ‘‘Data collection forms may need to be customised to the research question being investigated.”
MethodOlder adults with difficulty swallowing oral medicines: a systematic review of the literature
Gillicuddy, A. M., Crean, A. M., & Sahm, L. J. (2015) · European Journal of Clinical Pharmacology · doi:10.1007/s00228-015-1979-8 · Cited by 59
“This scale was developed to critically appraise studies that investigated the prevalence or incidence of a health problem and is particularly appropriate for evaluating cross-sectional studies [38].”
Visual Impairment Following Stroke - The Impact on Quality of Life: A Systematic Review
Hepworth, L. R., & Rowe, F. J. (2016) · Ophthalmology Research An International Journal · doi:10.9734/or/2016/23272 · Cited by 59
“is currently no ‘gold standard’ quality assessment tool for observational studies [11].”
BackgroundThe relation between dietary intake and glaucoma: a systematic review
Ramdas, W. D. (2018) · Acta Ophthalmologica · doi:10.1111/aos.13662 · Cited by 59
“Data extraction and quality assessment The study quality was assessed with a tool described elsewhere (Sanderson et al. 2007).”
MethodBrain and spinal stimulation therapies for phantom limb pain: a systematic review
Corbett, M. S., South, E., Harden, M., Eldabe, S. S., Pereira, E. A. C., Sedki, I., Hall, N., & Woolacott, N. F. (2018) · Health Technology Assessment · doi:10.3310/hta22620 · Cited by 59
The neuroscience of spirituality, religion, and mental health: A systematic review and synthesis
Rosmarin, D. H., Kaufman, C. C., Ford, S. F., Keshava, P., Drury, M. J., Minns, S. A., Marmarosh, C. L., Chowdhury, A., & Sacchet, M. D. (2022) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2022.10.003 · Cited by 58
“(2007) (Sanderson et al., 2007), studies were categorized into “low quality,” “moderate quality,” or “high quality” based on three criteria: (1) post-hoc statistical power, (2) use of valid and reliable measures (of spirituality and/or mental health), and (3) clinical relevance of the study findings (e.”
BackgroundVacca, M., Ballesio, A., & Lombardo, C. (2020) · European Eating Disorders Review · doi:10.1002/erv.2793 · Cited by 58
Faillie, J., Ferrer, P., Gouverneur, A., Driot, D., Berkemeyer, S., Vidal, X., Martínez‐Zapata, M. J., Huerta, C., Cervelló, X. C., Rottenkolber, M., et al. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.04.023 · Cited by 57
Completed suicide among nursing home residents: a systematic review
Murphy, B., Bugeja, L. C., Schumann, J., & Ibrahim, J. (2015) · International Journal of Geriatric Psychiatry · doi:10.1002/gps.4299 · Cited by 57
“Scoring the quality of studies is a contentious area, but does provide an indication of the overall quality of research and reporting (Sanderson et al., 2007).”
BackgroundPalliative pelvic radiotherapy of symptomatic incurable prostate cancer – A systematic review
Cameron, M. G., Kersten, C., Guren, M. G., Fosså, S. D., & Vistad, I. (2013) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2013.08.008 · Cited by 57
“There is no standard reliable and validated tool for assessing the ‘‘quality’’ of observational and other nonrandomized studies [9].”
MethodTricco, A. C., Ng, C., Gîlca, V., Anonychuk, A. M., Pham, B., & Berliner, S. (2011) · BMC Infectious Diseases · doi:10.1186/1471-2334-11-235 · Cited by 57
“A commonly used and validated quality tool for observational studies does not exist [33].”
Fang, X. L., Liang, C., Li, M., Montgomery, S., Fall, K., Aaseth, J., & Cao, Y. (2016) · Journal of Trace Elements in Medicine and Biology · doi:10.1016/j.jtemb.2016.03.014 · Cited by 57
Lamontagne-Godwin, F., Burgess, C. C., Clement, S., Gasston-Hales, M., Greene, C., Manyande, A., Taylor, D., Walters, P., & Barley, E. A. (2018) · BMJ Open · doi:10.1136/bmjopen-2017-019412 · Cited by 57
“No similar ‘gold standard’ tool exists which could be used across the other study designs, so we assessed each study informed by a simple checklist based on the STrengthening the Reporting of OBservational studies in Epidemiology(50) statement and a recent review of tools to assess bias in observational studies.(51) E…”
MethodRizzo, S. M., Douglas, J. W., & Lawrence, J. C. (2018) · Nutrition in Clinical Practice · doi:10.1002/ncp.10187 · Cited by 57
Jarde, A., Losilla, J.-M., & Vives, J. (2012) · Anales de Psicología · doi:10.6018/analesps.28.2.148911 · Cited by 42
“This can be explained by the fact that lots of tools are developed for specific systematic reviews, which makes its identification using a database search difficult (Sanderson et al., 2007).”
BackgroundInfluentialPranger, I. G., Joustra, M. L., Corpeleijn, E., Muskiet, F. A. J., Kema, I. P., Elferink, S. J. W. H. O., Singh‐Povel, C. M., & Bakker, S. J. L. (2018) · Nutrition Reviews · doi:10.1093/nutrit/nuy048 · Cited by 57
“Previous studies have emphasized the need for a uniform methodological assessment tool for observational studies.(44,45) The STROBE (Strengthening the Reporting of Observational studies in Epidemiology) recommendations are frequently used in an inappropriate way, ie, as a methodological guideline (to assess the method…”
MethodHarder, T. (2014) · International Journal of Health Policy and Management · doi:10.15171/ijhpm.2014.99 · Cited by 46
“In general, it is important to distinguish between methodological quality and the transparency and quality of reporting (9) since it has been shown that studies with similar reporting quality may be different regarding methodological quality (14).”
BackgroundSTROBE: new standards for reporting observational epidemiology, a chance to improve
Ebrahim, S. B. J., & Clarke, M. J. (2007) · International Journal of Epidemiology · doi:10.1093/ije/dym185 · Cited by 57
Hoffmann, F., Eggers, D., Pieper, D., Zeeb, H., & Allers, K. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.12.003 · Cited by 56
Wright, B., Barry, M., Hughes, E., Trépel, D., Ali, S., Allgar, V. L., Cottrill, L., Duffy, S., Fell, J., Glanville, J., et al. (2015) · Health Technology Assessment · doi:10.3310/hta19520 · Cited by 59
Global Action to reduce HIV stigma and discrimination
Stangl, A. L., Grossman, C. I., Sidibé, M., Goosby, E., Grossman, C. I., Stangl, A. L., Katz, I. T., Ryu, A. E., Onuegbu, A., Psaros, C., et al. (2013) · Journal of the International AIDS Society · doi:10.7448/ias.16.3.18934 · Cited by 56
Glanville, J., King, S., Guarner, F., Hill, C., & Sanders, M. E. (2015) · Nutrition Journal · doi:10.1186/s12937-015-0004-5 · Cited by 51
“There are a number of quality assessment and risk of bias tools available [22,24,34-38] and they are”
BackgroundEffect of periodontal treatment on preterm birth rate: a systematic review of meta‐analyses
López, N., Uribe, S. E., & Martinez, B. (2014) · Periodontology 2000 · doi:10.1111/prd.12073 · Cited by 56
Bardach, A., Cafferata, M. L., Klein, K., Cormick, G., Gibbons, L., & Ruvinsky, S. (2012) · The Pediatric Infectious Disease Journal · doi:10.1097/inf.0b013e31826ff3a5 · Cited by 56
The experience of electroconvulsive therapy and its impact on associated stigma: A meta-analysis
Aoki, Y. Y., Yamaguchi, S., Ando, S., Sasaki, N., Bernick, P. J., & Akiyama, T. (2016) · International Journal of Social Psychiatry · doi:10.1177/0020764016675379 · Cited by 56
Callebaut, L., Molyneux, P., & Alexander, T. (2016) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.2061 · Cited by 56
Joustra, M. L., Minović, I., Janssens, K. A. M., Bakker, S. J. L., & Rosmalen, J. G. M. (2017) · PLoS ONE · doi:10.1371/journal.pone.0176631 · Cited by 56
“For observational studies, literature indicates lack of a single methodological assessment tool [21,22].”
BackgroundHovdestad, W., Campeau, A., Potter, D. A., & Tonmyr, L. (2015) · PLoS ONE · doi:10.1371/journal.pone.0123366 · Cited by 56
“In keeping with earlier recommendations [25], our assessment checklist tools were carefully developed, but their reliability and validity have not been tested.”
MethodResultChanges in sedentary behaviours across the retirement transition: a systematic review
Sprod, J. A., Ferrar, K., Olds, T., & Maher, C. A. (2015) · Age and Ageing · doi:10.1093/ageing/afv140 · Cited by 56
Correlates of Physical Activity in Adults With Rheumatoid Arthritis: A Systematic Review
Larkin, L., & Kennedy, N. (2014) · Journal of Physical Activity and Health · doi:10.1123/jpah.2012-0194 · Cited by 56
Review: The effectiveness of Theraplay for children under 12 – a systematic literature review
Money, R., Wilde, S., & Dawson, D. L. (2020) · Child and Adolescent Mental Health · doi:10.1111/camh.12416 · Cited by 55
Ho, E. T. L., Wong, G., Craig, J. C., & Chapman, J. R. (2013) · Transplantation · doi:10.1097/tp.0b013e318284c15b · Cited by 55
“recent review of quality assessment for observational studies (38, 39).”
BackgroundPosadzki, P., Alotaibi, A., & Ernst, E. (2012) · Clinical Medicine · doi:10.7861/clinmedicine.12-6-505 · Cited by 55
Verdoodt, F., Friis, S., Dehlendorff, C., Albieri, V., & Kjær, S. K. (2015) · Gynecologic Oncology · doi:10.1016/j.ygyno.2015.12.009 · Cited by 55
Empirical evaluation of the Q-Genie tool: a protocol for assessment of effectiveness
Sohani, Z. N., Sarma, S., Alyass, A., de Souza, R. J., Robiou-du-Pont, S., Li, A., Mayhew, A., Yazdi, F., Reddon, H., Lamri, A., et al. (2016) · BMJ Open · doi:10.1136/bmjopen-2015-010403 · Cited by 55
Campbell, L., DiLorenzo, M. G., Atkinson, N. H., & Riddell, R. P. (2017) · Journal of Pediatric Psychology · doi:10.1093/jpepsy/jsx054 · Cited by 55
Schrammen, E., Roesmann, K., Rosenbaum, D., Redlich, R., Harenbrock, J., Dannlowski, U., & Leehr, E. J. (2022) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2022.104895 · Cited by 54
The Global Impact of COVID-19 on Childhood Cancer Outcomes and Care Delivery - A Systematic Review
Majeed, A., Wright, T., Guo, B., Arora, R. S., Lam, C. G., & Martiniuk, A. L. C. (2022) · Frontiers in Oncology · doi:10.3389/fonc.2022.869752 · Cited by 54
Sangaré, L. R., van Eijk, A. M., ter Kuile, F. O., Walson, J. L., & Stergachis, A. S. (2014) · PLoS ONE · doi:10.1371/journal.pone.0087743 · Cited by 54
“The assessment of non-randomized study designs were based on source population, participant selection, completeness of exposure and outcome data, appropriate tests, sample size, and measures to control confounding [25,26].”
MethodMarks, M., Schoones, J. W., Kolling, C., Herren, D. B., Goldhahn, J., & Vlieland, T. P. M. V. (2013) · Journal of Hand Surgery (European Volume) · doi:10.1177/1753193413488301 · Cited by 54
“For observational studies, there are various checklists and scores available, but none of these can be recommended to be used as a gold-standard (Sanderson et al., 2007).”
BackgroundDunn, K., Cairns, D., Jahoda, A. J., & McConnachie, A. (2019) · BJPsych Open · doi:10.1192/bjo.2019.75 · Cited by 54
Badovinac, S. D., Riddell, R. P., Deneault, A., Martin, J., Bureau, J., & O’Neill, M. C. (2021) · Marriage & Family Review · doi:10.1080/01494929.2021.1879984 · Cited by 54
“Article quality and risk of bias Given the lack of a gold-standard measure for quality assessment of observational studies (Sanderson et al., 2007), the present study adopted the NIH National Heart, Blood, and Lungs Institute Quality Assessment Tool for Observational Cohort and Cross-Sectional…”
MethodHammad, T. A., Pinheiro, S. P., & Neyarapally, G. A. (2011) · Clinical Trials · doi:10.1177/1740774511419165 · Cited by 46
“Furthermore, there is a noticeable lack of standardized tools to assess the quality of drug safety evidence gleaned from observational studies [93,94].”
BackgroundPapageorgiou, S. N., Papadopoulos, M. A., & Athanasiou, A. E. (2014) · The European Journal of Orthodontics · doi:10.1093/ejo/cjt008 · Cited by 51
“A wide variety of checklists and scales exist (Moher et al., 1995; Sanderson et al., 2007; Higgins et al., 2011), although the use of scales has been cautioned (Jüni et al., 1999; Greenland and O’Rourke, 2001).”
BackgroundChallenges in the Use of Literature-based Meta-Analysis to Examine Gene-Environment Interactions
Palla, L., Higgins, J. P. T., Wareham, N. J., & Sharp, S. J. (2010) · American Journal of Epidemiology · doi:10.1093/aje/kwq051 · Cited by 52
Self-citationYoung, I., Waddell, L. A., Sánchez, J., Wilhelm, B., McEwen, S. A., & Rajić, A. (2013) · Preventive Veterinary Medicine · doi:10.1016/j.prevetmed.2013.11.009 · Cited by 53
Beliefs About Sexual Activity During Pregnancy: A Systematic Review of the Literature
Ribeiro, M. C., Scanavino, M. D. T., do Amaral, M. L. S., Horta, A. L. D. M., & Torloni, M. R. (2017) · Journal of Sex & Marital Therapy · doi:10.1080/0092623x.2017.1305031 · Cited by 53
“Studies with 3 or 4 points were classified as having a “low risk of bias”; those with a score of 0, 1, or 2 were categorized as having a “high risk of bias” (Sanderson et al., 2007).”
BackgroundOehr, L. E., & Anderson, J. F. I. (2017) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2017.03.019 · Cited by 53
“The quality of individual studies was assessed according to the recommendations of Viswanathan et al(29) and The Cochrane Collaboration(30) for the assessment of bias within intervention studies, as no such standards exist for observational epidemiologic studies.(31) Each study was examined for bias relating to (1) se…”
MethodDouble Burden of Malnutrition among Bangladeshi Women: A Literature Review
Hasan, M., Sutradhar, I., Shahabuddin, A., & Sarker, M. (2017) · Cureus · doi:10.7759/cureus.1986 · Cited by 53
“These criteria were developed based on the standards described in previous studies for quality assessment of published research [9-11].”
MethodTadesse, G., Tessema, T. S., Beyene, G., & Aseffa, A. (2018) · PLoS ONE · doi:10.1371/journal.pone.0192575 · Cited by 53
Moore, D. C., Keegan, T., Dunleavy, L., & Froggatt, K. A. (2019) · Systematic Reviews · doi:10.1186/s13643-019-0973-0 · Cited by 53
Task Shifting for the Delivery of Pediatric Antiretroviral Treatment: A Systematic Review
Penazzato, M., Davies, M., Apollo, T., Negussie, E. K., & Ford, N. P. (2013) · JAIDS Journal of Acquired Immune Deficiency Syndromes · doi:10.1097/qai.0000000000000024 · Cited by 52
“There is no single agreed tool for assessing the quality of observational studies.(11) For this review, we assessed study quality independently and in duplicated (M.”
MethodPrevalence and correlates of individuals with sexual interest in children: A systematic review
Savoie, V., Quayle, E., & Flynn, E. (2021) · Child Abuse & Neglect · doi:10.1016/j.chiabu.2021.105005 · Cited by 53
Vesterling, C., Wilke, J., Bäker, N., Bolz, T., Eilts, J., Koglin, U., Rademacher, A., & Goagoses, N. (2023) · Health & Social Care in the Community · doi:10.1155/2023/6242678 · Cited by 52
Boccia, S., Feo, E. D., Gallì, P., Gianfagna, F., Amore, R., & Ricciardi, G. (2010) · European Journal of Epidemiology · doi:10.1007/s10654-010-9503-z · Cited by 52
“So far, 86 tools for grading the quality of epidemiological studies have been published [ 25 ].”
BackgroundRisk Behaviours among Female Sex Workers in China: A Systematic Review and Data Synthesis
Chow, E. P. F., Muessig, K. E., Yuan, L., Wang, Y., Zhang, X., Zhao, R., Sun, P., Sun, X., Tucker, J. D., Jing, J., et al. (2015) · PLoS ONE · doi:10.1371/journal.pone.0120595 · Cited by 52
“Each study was assessed for risk of bias [29] by two independent investigators (EPFC, XZ).”
Smoking, Binge Drinking, and Drug Use Among Childhood Cancer Survivors: A Meta‐Analysis
Marjerrison, S., Hendershot, E., Empringham, B., & Nathan, P. C. (2016) · Pediatric Blood & Cancer · doi:10.1002/pbc.25943 · Cited by 52
“Risk of bias was assessed using the Health Evidence Bulletins—Wales: Critical Appraisal of Observational Studies tool,[35] as it was deemed the most appropriate tool for evaluating both cross-sectional and cohort studies.[36] Final appraisal was determined by consensus between the two reviewers who assessed the studie…”
MethodHasan, H., Shaikh, O. M., Rassekh, S. R., Howard, A. F., & Goddard, K. (2016) · Pediatric Blood & Cancer · doi:10.1002/pbc.26200 · Cited by 52
Migraine as a risk factor for primary open angle glaucoma: A systematic review and meta-analysis
Xu, C., Li, J., Li, Z., & Mao, X. (2018) · Medicine · doi:10.1097/md.0000000000011377 · Cited by 53
Effects of Arthroscopic Partial Meniscectomy on Quadriceps Strength: A Systematic Review
McLeod, M. M., Gribble, P. A., Pfile, K. R., & Pietrosimone, B. G. (2012) · Journal of Sport Rehabilitation · doi:10.1123/jsr.21.3.285 · Cited by 51
Cooper, S. L., Pelly, F. E., & Lowe, J. B. (2016) · Appetite · doi:10.1016/j.appet.2016.02.001 · Cited by 51
Anti-retroviral Therapy and Pregnancy Outcomes in Developing Countries: A Systematic Review
Alemu, F. M., Yalew, A. W., Fantahun, M., & Ashu, E. E. (2014) · International Journal of Maternal and Child Health and AIDS · doi:10.21106/ijma.36 · Cited by 51
“Methodological quality was assessed by a predefined set of criteria designed to grade each paper according to Sanderson et al.[15] The quality criteria were as follows: whether the papers included a sample size calculation; an explicit description for testing both HIV-infected and uninfected women, initiation of ART,…”
BackgroundMethodParenting behaviors and the well-being of children with a chronic physical condition.
Crandell, J., Sandelowski, M., Leeman, J., Havill, N. L., & Knafl, K. A. (2017) · Families Systems & Health · doi:10.1037/fsh0000305 · Cited by 51
“…for the following internal and external validity threats: reporting bias (selective reporting of results), nonrepresentative sampling (low response rates, not representative of target population), and approach to measurement (validity, reliability; Sanderson et al., 2007; von Elm et al., 2007).”
Lifetime prevalence of suicidal ideation among men who have sex with men: a meta-analysis
Luo, Z., Feng, T., Fu, H., & Yang, T. (2017) · BMC Psychiatry · doi:10.1186/s12888-017-1575-9 · Cited by 51
“Additionally, levels of quality of observational studies may differ by different types of assessment scales [43].”
BackgroundBraga, L. H., Pemberton, J., DeMaria, J., & Lorenzo, A. J. (2011) · Journal of Urology · doi:10.1016/j.juro.2011.03.044 · Cited by 50
Ciapponi, A., Alison, L., Mazzoni, A., Demián, G., Cesaroni, S., & Sobrino, E. M. (2013) · COPD Journal of Chronic Obstructive Pulmonary Disease · doi:10.3109/15412555.2013.836479 · Cited by 50
Wang, B., Liao, C., Zhou, B., Cao, W., Lv, J., Yu, C., Gao, W., & Li, L. (2015) · Twin Research and Human Genetics · doi:10.1017/thg.2015.8 · Cited by 50
“Three assessments (items 4, 5, and 11) were removed; the remainder included items to judge the selection bias (indicate source of information, inclusion and exclusion criteria, time period used for identifying twins, explain how missing data were handled and summarize twin response rate), measurement bias (assess for…”
An updated dose–response meta-analysis of coffee consumption and liver cancer risk
Yu, C., Cao, Q., Chen, P., Yang, S., Deng, M., Wang, Y., & Li, L. (2016) · Scientific Reports · doi:10.1038/srep37488 · Cited by 50
Metadata
- Authors
- Sanderson, S., Tatt, I. D., & Higgins, J. P.
- Affiliations
- University of Cambridge (GB)
- F. Hoffmann-La Roche AG (CH)
- MRC Biostatistics Unit (GB)
- Corresponding author
- Sanderson, S.
- ORCID
- Higgins, J. P.
- Venue
- International Journal of Epidemiology
- Publisher
- Oxford University Press (OUP)
- Published
- 2007-04-30
- Volume
- 36
- Issue
- 3
- Pages
- 666-676
- ISSN
- 0300-5771 (ISSN-L)
- 1464-3685
- OpenAlex
- W2171251365
- Cited by
- 1,531
- References
- 73
- Citations by year
- 2026: 22 · 2025: 38 · 2024: 42 · 2023: 59 · 2022: 80 · 2021: 96 · 2020: 101 · 2019: 105 · 2018: 125 · 2017: 104 · 2016: 150 · 2015: 118 · 2014: 122 · 2013: 111 · 2012: 96
- FWCI
- 7.09
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- International Journal of Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- quality assessment tools, bias assessment, observational studies, critical appraisal, checklists, rating scales, STROBE, confounding adjustment, selection bias, measurement bias, conflicts of interest
- MeSH terms
- Humans, Research Design, Bias, Epidemiologic Studies, Evidence-Based Medicine, Observation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.