Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography
The DOI matches its Crossref record; 73 of the 73 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Assessing quality and susceptibility to bias is essential when interpreting primary research and conducting systematic reviews and meta-analyses. Tools for assessing quality in clinical trials are well-described but much less attention has been given to similar tools for observational epidemiological studies. METHODS: Tools were identified from a search of three electronic databases, bibliographies and an Internet search using Google. Two reviewers extracted data using a pre-piloted extraction form and strict inclusion criteria. Tool content was evaluated for domains potentially related to bias and was informed by the STROBE guidelines for reporting observational epidemiological studies. RESULTS: A total of 86 tools were reviewed, comprising 41 simple checklists, 12 checklists with additional summary judgements and 33 scales. The number of items ranged from 3 to 36 (mean 13.7). One-third of tools were designed for single use in a specific review and one-third for critical appraisal. Half of the tools provided development details, although most were proposed for future use in other contexts. Most tools included items for selection methods (92%), measurement of study variables (86%), design-specific sources of bias (86%), control of confounding (78%) and use of statistics (78%); only 4% addressed conflict of interest. The distribution and weighting of domains across tools was variable and inconsistent. CONCLUSION: A number of useful assessment tools have been identified by this report. Tools should be rigorously developed, evidence-based, valid, reliable and easy to use. There is a need to agree on critical elements for assessing susceptibility to bias in observational epidemiology and to develop appropriate evaluation tools.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Such instruments have been developed by other groups and were the subject of a recent systematic review [28].”
From The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: Guidelines for Reporting Observational Studies (2007) · cites this paper as [28]
In the citing paper
“It is not clear how weights should be assigned to each item of the scale [18].”
From Tools used to assess the quality of peer review reports: A methodological systematic review (2019) · cites this paper as [18]
In the citing paper
“The reviewers concluded that there is no ‘single obvious candidate tool for assessing quality of observational epidemiological studies’.”
From COSMOS-E: Guidance on conducting systematic reviews and meta-analyses of observational studies of etiology (2019) · cites this paper as [45]
In the citing paper
“A review identified more than 80 tools for assessing risk of bias in nonrandomised studies [45].”
From COSMOS-E: Guidance on conducting systematic reviews and meta-analyses of observational studies of etiology (2019) · cites this paper as [45]
In the citing paper
“The need for complete reporting of all relevant methodological aspects as a prerequisite to assessing the risk of bias of studies was highlighted in 10 articles, [16, 33, 35, 47, 50, 56, 86, 87, 90, 91] and was the focus of recommendations in a publication from the Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group [12].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [90]
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Meta-analysis of Observational Studies in Epidemiology<SUBTITLE>A Proposal for Reporting</SUBTITLE>
Stroup, D. F. (2000) · JAMA · doi:10.1001/jama.283.15.2008 · Cited by 20,623
Downs, S. H., & Black, N. (1998) · Journal of Epidemiology and Community Health · doi:10.1136/jech.52.6.377 · Cited by 8,252
Moher, D., Cook, D. J., Eastwood, S., Olkin, I., Rennie, D., & Stroup, D. F. (1999) · The Lancet · doi:10.1016/s0140-6736(99)04149-5 · Cited by 4,826
Whiting, P., Rutjes, A. W., Reitsma, J. B., Bossuyt, P. M., & Kleijnen, J. (2003) · BMC Medical Research Methodology · doi:10.1186/1471-2288-3-25 · Cited by 3,592
Assessing the quality of controlled clinical trials
Jüni, P., Altman, D. G., & Egger, M. (2001) · BMJ · doi:10.1136/bmj.323.7303.42 · Cited by 3,309 · cited 5×
“a (5) (5) (5) ( 5 ) N N LongneckerCoh”
InfluentialUsers' guides to the medical literature
Guyatt, G. H. (1993) · JAMA · doi:10.1001/jama.270.17.2096 · Cited by 2,970
Evaluating non-randomised intervention studies
Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D. (2003) · Health Technology Assessment · doi:10.3310/hta7270 · Cited by 2,907
Cognitive and Behavioral Outcomes of School-Aged Children Who Were Born Preterm: A Meta-analysis
Bhutta, A. T., Cleves, M. A., Casey, P. H., Cradock, M. M., & Anand, K. J. S. (2002) · JAMA · doi:10.1001/jama.288.6.728 · Cited by 2,392 · cited 1×
“Y N N 9 (22) 2 (17) 5 (15) 16 (19) Y Y N 15 (36) 6 ( 50 ) 7 (21) 28 (32)”
The Hazards of Scoring the Quality of Clinical Trials for Meta-analysis
Jüni, P. (1999) · JAMA · doi:10.1001/jama.282.11.1054 · Cited by 1,981
Low grade inflammation and coronary heart disease: prospective study and updated meta-analyses
Danesh, J. (2000) · BMJ · doi:10.1136/bmj.321.7255.199 · Cited by 1,692
Research works printed but not linked: 7
- Carneiro, 2002, Revista portuguesa de cardiologia : org o oficial da Sociedade Portuguesa de Cardiologia = Portuguese journal of cardiology : an official journal of the Portuguese Society of Cardiology, 21, 891
- 1981, Canadian Medical Association Journal, 124, 985
- 2002, SOUTH AFRICAN J CLIN NUTR, 15, 68
- 2000, Chronic diseases in Canada, 19, 170
- Ciliska, 1996, Canadian journal of public health. Revue canadienne de sant publique, 87, 193
- 1994, Canadian journal of public health. Revue canadienne de sant publique, 85, S8
- Spitzer, 1990, Clinical and investigative medicine. M decine clinique et experimentale, 13, 17
Cited by
Items 701–800 of 1,006
A unified framework for bias assessment in clinical research
Stone, J. C., Glass, K., Clark, J., Munn, Z., Tugwell, P., & Doi, S. A. R. (2019) · International Journal of Evidence-Based Healthcare · doi:10.1097/xeb.0000000000000165 · Cited by 40
“In a review of tools that assess susceptibility to bias in observational studies, it was found that only half of the tools examined were transparent in their descriptions of the tool development process, including their design, applicability, and the empirical basis for item inclusion.(14) Yet, despite this absence of…”
BackgroundMethodInfluentialRod, J. E., Oviedo-Trespalacios, Ó., Senserrick, T. M., & King, M. (2021) · Accident Analysis & Prevention · doi:10.1016/j.aap.2021.105970 · Cited by 51
Minor Children of Palliative Patients: A Systematic Review of Psychosocial Family Interventions
Kühne, F., Krattenmacher, T., Beierlein, V., Grimm, J. C., Bergelt, C., Romer, G., & Möller, B. (2012) · Journal of Palliative Medicine · doi:10.1089/jpm.2011.0380 · Cited by 50
An exploration of the current knowledge on young people who kill: A systematic review
Gerard, F. J., Jackson, V. A., Chou, S., Whitfield, K. C., & Browne, K. D. (2014) · Aggression and Violent Behavior · doi:10.1016/j.avb.2014.07.002 · Cited by 50
Nissim, S., Idos, G., & Wu, B. U. (2012) · Pancreas · doi:10.1097/mpa.0b013e3182580fb4 · Cited by 50
Revision of the mixed methods appraisal tool (MMAT): A mixed methods study
Hong, Q. N. (2018) · eScholarship@McGill (McGill) · doi:10.82308/30348 · Cited by 50
Rydzewska, E., Dunn, K., Cooper, S., & Kinnear, D. (2021) · Journal of Intellectual Disability Research · doi:10.1111/jir.12827 · Cited by 49
“This covers all domains of quality assessment of observational studies (Sanderson et al. 2007), including clarity of the stated aims, methodology (including age/gender standardisation and consideration of group differences in disease prevalence rates), study design and size, participant selection,…”
BackgroundMethodVenous stenting after deep venous thrombosis and antithrombotic therapy: A systematic review
Eijgenraam, P., Cate, H. T., & ten Cate‐Hoek, A. J. (2014) · Reviews in vascular medicine · doi:10.1016/j.rvm.2014.03.001 · Cited by 49
“As far as we know no quality assessment tools designed for specific use in non-comparator, observational studies are available [22].”
BackgroundRevascularization of immature permanent teeth with apical periodontitis
Moreno-Hidalgo, M. C., Caleza-Jiménez, C., Mendoza-Mendoza, A., & Iglesias‐Linares, A. (2013) · International Endodontic Journal · doi:10.1111/iej.12154 · Cited by 49
Liu, N., Xu, Z., Li, D., Zhang, Q., Wang, H., & Duan, Z.-J. (2014) · Vaccine · doi:10.1016/j.vaccine.2014.06.018 · Cited by 49
Böck, C., Diehl, K., Schneider, S. O., Diehm, C., & Litaker, D. G. (2012) · Medical Care Research and Review · doi:10.1177/1077558712441084 · Cited by 49
Ayyaswamy, B., Saeed, B., Anand, A., Chan, L. K., & Shetty, V. (2019) · EFORT Open Reviews · doi:10.1302/2058-5241.4.190008 · Cited by 49
Murphy, J., McSharry, J., Hynes, L., Matthews, S., Rhoon, L. V., & Molloy, G. J. (2020) · Journal of Asthma · doi:10.1080/02770903.2020.1711916 · Cited by 49
“(26) for assessing quality and bias in observational studies (Figure 2).”
BackgroundWang, X., Cheng, B., Wang, S., Lu, F., Luo, Y., Long, X., & Kong, D. (2020) · Journal of Affective Disorders · doi:10.1016/j.jad.2020.11.057 · Cited by 50
Blinded versus unblinded assessments of risk of bias in studies included in a systematic review
Morissette, K., Tricco, A. C., Horsley, T., Chen, M. H., & Moher, D. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000025.pub2 · Cited by 49
Lower dental caries prevalence associated to chronic kidney disease: a systematic review
Andrade, M. R. T. C., Antunes, L. A. A., Soares, R. M. D. A., Leão, A. T. T., Maia, L. C., & Primo, L. G. (2013) · Pediatric Nephrology · doi:10.1007/s00467-013-2437-4 · Cited by 49
O’Neill, M. C., Badovinac, S. D., Riddell, R. P., Bureau, J., Rumeo, C., & Costa, S. (2021) · PLoS ONE · doi:10.1371/journal.pone.0245061 · Cited by 48
“quality assessment facilitated a hybrid approach to examining both a summary judgment checklist resulting in a total quantitative quality score, as well as the preferred method of a smaller checklist that focuses on the few main “potential sources of bias” [34] resulting in a more nuanced qualitative quality judgment.”
BackgroundMethodVitamin D status and bone mineral density in the Chinese population: a review
Man, P. W., van der Meer, I. M., Lips, P. T. A. M., & Middelkoop, B. J. (2016) · Archives of Osteoporosis · doi:10.1007/s11657-016-0265-4 · Cited by 48
“First, few studies were available and most of the 11 selected studies were crosssectional in design and susceptible to bias [33].”
BackgroundResultCassim, R., Koplin, J. J., Dharmage, S. C., Senaratna, C. V., Lodge, C., Lowe, A. J., & Russell, M. A. (2016) · Journal of Asthma · doi:10.1080/02770903.2016.1175474 · Cited by 48
The Development of a Checklist to Enhance Methodological Quality in Intervention Programs
Chacón-Moscoso, S., Sanduvete-Chaves, S., & Sánchez-Martín, M. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.01811 · Cited by 41
“Checklists may also propose a final global index (Effective Public Health Practice Project, 1998; Efficace et al., 2003; Sanderson et al., 2007; Pluye et al., 2009); just one individual component (Gilbody et al., 2007); or several components (Bossuyt et al., 2003; Taji et al., 2006; Schulz et al.,…”
Backgroundvan Ditshuizen, J. C., van den Driessche, C. R. L., Sewalt, C. A., Lieshout, E. M. M. V., Verhofstad, M. H. J., & Hartog, D. D. (2020) · The Journal of Trauma: Injury, Infection, and Critical Care · doi:10.1097/ta.0000000000002850 · Cited by 48
“These quality elements were based on existing literature.(10,15) Each included study was subjected to quality assessment by the reviewers.”
MethodSystematic review and meta-analysis of vitamin D deficiency in different pregnancy on preterm birth
Lian, R.-H., Qi, P.-A., Yuan, T., Yan, P., Qiu, W.-W., Wei, Y., Hu, Y., Yang, K., & Yi, B. (2021) · Medicine · doi:10.1097/md.0000000000026303 · Cited by 49
“The Agency for Healthcare Research and Quality (ARHQ) methodology checklist was used for cross-sectional studies, which included 11 items with a summary judgment.([23,24]) Two different authors (RHL and PAQ) independently appraised the risk of bias of the included studies, and disagreements were resolved by consensus…”
MethodPark, J. J., Narayanan, S., Tiefenbach, J., Lukšić, I., Ale, B. M., Adeloye, D., & Rudan, I. (2022) · Journal of Global Health · doi:10.7189/jogh.12.04014 · Cited by 48
“Furthermore, we adapted the GRADE and NHLBI approach to quality in the absence of a standardised tool for evaluating the quality of observational epidemiological studies [25,26].”
MethodStahl, S., Stahl, A. S., Meisner, C., Rahmanian‐Schwarz, A., Schaller, H., & Lotter, O. (2012) · BMC Musculoskeletal Disorders · doi:10.1186/1471-2474-13-225 · Cited by 48
“None of the previously described quality criteria to decrease susceptibility to bias as described by Sanderson et al. [17] have been met in the identified retrospective cohort studies (appropriate definition of inclusion or exclusion criteria for cohorts and controls to control selection bias;…”
TP53 Arg72Pro Polymorphism and Skin Cancer Risk: A Meta-Analysis
Jiang, D., Wang, W., Ren, W., Yao, L., Peng, B., & Yu, L. (2010) · Journal of Investigative Dermatology · doi:10.1038/jid.2010.270 · Cited by 48
“Therefore, in order to evaluate and minimize the potential for selection bias of cases and controls, inappropriate materials for assessment of genotype in cases, genotyping errors, andsor population stratification in controls, as well as insufficient statistical power, the quality of the individual studies in our meta…”
MethodIssues of reporting in observational studies in veterinary medicine
Sargeant, J. M., & O'Connor, A. M. C. (2013) · Preventive Veterinary Medicine · doi:10.1016/j.prevetmed.2013.09.004 · Cited by 48
Autopercepción de salud general y mortalidad en adultos mayores
Moreno, H., Huerta, M., & Cecilia Albala, B. (2013) · Gaceta Sanitaria · doi:10.1016/j.gaceta.2013.07.006 · Cited by 48
Russell, R. M., Chung, M., Balk, E. M., Atkinson, S. A., Giovannucci, E. L., Ip, S., Lichtenstein, A. H., Mayne, S. T., Raman, G., Ross, A. C., et al. (2009) · American Journal of Clinical Nutrition · doi:10.3945/ajcn.2008.27154 · Cited by 48
Dean, E., Bloom, A., Cirillo, M., Hong, Q., Jawl, B., Jukes, J., Nijjar, M., Sadovich, S., & Bruno, S. S. (2012) · Blood Pressure · doi:10.3109/08037051.2011.596320 · Cited by 39
Methadone Maintenance Treatment and Cognitive Function: A Systematic Review
Wang, G., Wouldes, T. A., & Russell, B. (2014) · Current Drug Abuse Reviews · doi:10.2174/18744737112059990020 · Cited by 47
“[26].”
Holma, I. A. K., Holma, K. M., Melartin, T. K., Rytsälä, H. J., & Isometsä, E. T. (2012) · Acta Psychiatrica Scandinavica · doi:10.1111/j.1600-0447.2011.01785.x · Cited by 46
“In addition, we did not investigate work motivation, work circumstances (38), work functioning (38, 39) or patient s cognitive capacity (36), which could be of importance.”
BackgroundMori, C., Park, J., Racine, N., Ganshorn, H., Hartwick, C., & Madigan, S. (2023) · Child Abuse & Neglect · doi:10.1016/j.chiabu.2023.106255 · Cited by 46
Nichols, J., Embleton, L., Mwangi, A., Morantz, G., Vreeman, R., Ayaya, S., Ayuku, D., & Braitstein, P. K. A. (2013) · Child Abuse & Neglect · doi:10.1016/j.chiabu.2013.09.012 · Cited by 46
Engel, J., Damen, N. L., van der Wulp, I., de Bruijne, M. C., & Wagner, C. (2016) · Current Cardiology Reviews · doi:10.2174/1573403x12666160504100025 · Cited by 46
“However, reliable and generally accepted tools to assess the quality of observational studies are lacking [85].”
Orthopaedic manifestations within the 22q11.2 Deletion syndrome: A systematic review
Homans, J. F., Tromp, I. N., Colo, D., Schlösser, T. P. C., Kruyt, M. C., Deeney, V. F. X., Crowley, T. B., McDonald‐McGinn, D. M., & Castelein, R. M. (2017) · American Journal of Medical Genetics Part A · doi:10.1002/ajmg.a.38545 · Cited by 46
Rhodes, R. E., Zhang, R., & Zhang, C. (2020) · Annals of Behavioral Medicine · doi:10.1093/abm/kaz068 · Cited by 46
Urbutė, A., Rasmussen, C. L., Belmonte, F., Obermueller, T., Prigge, E., Arbyn, M., Verdoodt, F., & Kjær, S. K. (2020) · Cancer Epidemiology Biomarkers & Prevention · doi:10.1158/1055-9965.epi-19-1259 · Cited by 46
“Quality assessment of individual studies In the absence of a standardized tool for assessing the quality of observational studies, we did not assign summary scores to individual studies (18).”
BackgroundSense of Coherence and Childbearing: A Scoping Review of the Literature
Ferguson, S. A., Davis, D. L., Browne, J., & Taylor, J. (2014) · International Journal of Childbirth · doi:10.1891/2156-5287.4.3.134 · Cited by 46
Risk factors for pressure injuries in critical care patients: an updated systematic review
Alderden, J., Brooks, K. R., Kennerly, S. M., Yap, T. L., Dworak, E., & Cox, J. (2025) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2025.105127 · Cited by 45
Quality of life in persons with partial foot or transtibial amputation: A systematic review
Quigley, M., & Dillon, M. P. (2014) · Prosthetics and Orthotics International · doi:10.1177/0309364614546526 · Cited by 45
Michelakos, T., Kousoulis, A. A., Katsiardanis, K., Dessypris, N., Anastasiou, A., Katsiardani, K.-P., Kanavidis, P., Stefanadis, C., Papadopoulos, F. C., & Petridou, E. T. (2013) · Journal of Aging and Health · doi:10.1177/0898264313482488 · Cited by 31
Methodological integrative review of the work sampling technique used in nursing workload research
Blay, N., Duffield, C. M., Gallagher, R. D., & Roche, M. A. (2014) · Journal of Advanced Nursing · doi:10.1111/jan.12466 · Cited by 45
Development of Cardiovascular Indices of Acute Pain Responding in Infants: A Systematic Review
Waxman, J. A., Riddell, R. P., Modica, P. T., Schmidt, L. A., & Pinhasov, A. (2016) · Pain Research and Management · doi:10.1155/2016/8458696 · Cited by 45
“Due to the fact that a gold-standard quality assessment measure was not available for observational studies [31], a modified checklist combining Downs and Black [32] and Crombie and McQuay [33] was utilized (see Supplementary File 2 for the checklist).”
Methodvan Roon, A. M., Mercat, M., Schaik, G. V., Nielen, M., Graham, D. A., More, S. J., Guelbenzu‐Gonzalo, M., Fourichon, C., Madouasse, A., & Santman-Berends, I. M. G. A. (2020) · Journal of Dairy Science · doi:10.3168/jds.2020-18193 · Cited by 46
“As no generic tool is available for appraisal of observational studies for meta-analysis (Sanderson et al., 2007), we created our own checklist with relevant checkpoints based on our own observations and in alignment with the methods used in previous studies (National Institutes of Health, 2014;…”
Iqbal, S., & Ali, I. (2021) · Heliyon · doi:10.1016/j.heliyon.2021.e07540 · Cited by 45
“While the tools for assessing quality and susceptibility to bias in observational studies were assessed by using the study protocols as describes in proposed by [19].”
MethodMeasuring Happiness in Adolescent Samples: A Systematic Review
Lukoševičiūtė, J., Argustaitė-Zailskienė, G., & Šmigelskas, K. (2022) · Children · doi:10.3390/children9020227 · Cited by 47
“However, the tools for observational epidemiological studies in general [28] and cross-sectional studies in particular are not well accepted [29], even though they comprise the largest part of the studies under this systematic review.”
BackgroundMartin, A., Ogilvie, D., & Suhrcke, M. (2014) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-014-0142-8 · Cited by 46
King, C., Alvarez, L. G., Holmes, A. H., Moore, L. S. P., Galletly, T., & Aylin, P. P. (2012) · Journal of Hospital Infection · doi:10.1016/j.jhin.2012.05.004 · Cited by 45
“pdf/view)(15) were employed for critical appraisal of the papers in this study as they were developed specifically for cohort and caseecontrol studies, and have been widely used in review articles.(16) For those papers describing prevalence and surveillance studies, an ‘adjusted CASP cohort tool’ was applied, which in…”
MethodInflammatory Bowel Disease in Latin America: A Systematic Review
Calderón, M., Minckas, N., Núñez-González, S., & Ciapponi, A. (2018) · Value in Health Regional Issues · doi:10.1016/j.vhri.2018.03.010 · Cited by 45
Hennegan, J. M., Henderson, J., & Redshaw, M. (2015) · BMJ Open · doi:10.1136/bmjopen-2015-008616 · Cited by 44
“While there are clear guidelines for assessing the risk of bias in RCTs,(29) risk of bias assessment for observational research remains controversial.(30) The risk of bias and quality of included studies were assessed for this review using a checklist developed based on the items of the STrengthening the Reporting of…”
MethodRas, J., Kengne, A. P., Smith, D. L., Soteriades, E. S., & Leach, L. L. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20042816 · Cited by 44
Menchini, F., Toneatto, G., Miele, A., Donati, S., Lanzetta, P., & Virgili, G. (2018) · Eye · doi:10.1038/s41433-018-0138-8 · Cited by 44
“We chose the new ROBINS-I tool to assess risk of bias of NRS because it addresses weaknesses not considered in previously available approaches [50], and includes key features such as specification of the target trial and effect of interest, and assessments of several signaling questions within seven bias domains.”
Agriculture and musculoskeletal disorders in low- and middle-income countries
Kumaraveloo, K. S., & Kolstrup, C. L. (2018) · Journal of Agromedicine · doi:10.1080/1059924x.2018.1458671 · Cited by 44
Sleep in parents of preterm infants: A systematic review
Haddad, S., Dennis, C., Shah, P. S., & Stremler, R. (2019) · Midwifery · doi:10.1016/j.midw.2019.01.009 · Cited by 44
Clinical effectiveness of orthodontic treatment on smile esthetics: a systematic review
Christou, T., Betlej, A., Aswad, N., Ogdon, D., & Kau, C. H. (2019) · Clinical Cosmetic and Investigational Dentistry · doi:10.2147/ccide.s189708 · Cited by 44
Suleiman‐Martos, N., Caballero-Vázquez, A., Gómez‐Urquiza, J. L., Albendín‐García, L., Romero‐Béjar, J. L., & la Fuente, G. A. C. (2021) · Journal of Personalized Medicine · doi:10.3390/jpm11050416 · Cited by 44
“In order to assess the quality of the studies included in the review, the levels of evidence and grades of recommendation of the Oxford Center for Evidence-based Medicine (OCEBM) were followed [22], as well as the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) guide [23].”
MethodD-dimer in preeclampsia: Systematic review and meta-analysis
Pinheiro, M. D. B., Junqueira, D. R., Coelho, F. F., Freitas, L. G., Ferriani, M. D. G. C., Gomes, K. B., & Dusse, L. M. S. (2012) · Clinica Chimica Acta · doi:10.1016/j.cca.2012.08.003 · Cited by 44
“This approach was designed to provide an overall quality assessment of the specific domains associated with potential source of bias in study findings and was not designed to provide a score to each individual study [18].”
MethodSeidel, D. H., Ditchen, D. M., Hoehne-Hückstädt, U. M., Rieger, M. A., & Steinhilber, B. (2019) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph16010130 · Cited by 43
“3; assessment according to [6], modified according to [19,20]; max.”
BackgroundMethodThe Effects of Type 2 Diabetes on Cognitive Performance: A Review of Reviews
Sola, T., Sola, F.-M., & Jehkonen, M. (2024) · International Journal of Behavioral Medicine · doi:10.1007/s12529-024-10274-6 · Cited by 43
Childhood physical abuse and suicide-related behavior: A systematic review
Mironova, P., Rhodes, A. E., Bethell, J., Tonmyr, L., Boyle, M. H., Wekerle, C., Goodman, D., & Leslie, B. (2011) · Vulnerable Children and Youth Studies · doi:10.1080/17450128.2010.542301 · Cited by 43
“Formal quality assessment rules were not applied, given the lack of consensus and evaluation tools to assess observational studies (Sanderson, Tatt, & Higgins, 2007 ), nor were results pooled meta-analytically, given the small number of studies and methodological heterogeneity.”
MethodModes of delivery in preventive intervention studies: a rapid review
Beall, R. F., Baskerville, N. B., Golfam, M., Saeed, S., & Little, J. (2014) · European Journal of Clinical Investigation · doi:10.1111/eci.12279 · Cited by 42
Children and adolescents adjustment to parental multiple sclerosis: a systematic review
Razaz, N., Nourian, R., Marrie, R. A., Boyce, W. T., & Tremlett, H. L. (2014) · BMC Neurology · doi:10.1186/1471-2377-14-107 · Cited by 43
“For observational and epidemiological study designs, there is currently a lack of consensus regarding the most appropriate methodology for assessing quality in the context of a systematic review [51].”
BackgroundBaltodano, P. A., Basdag, B., Bailey, C. R., Baez, M. J. L., Tong, A., Seal, S. M., Meléndez, M. M., Xie, L. H., Manahan, M. A., & Rosson, G. D. (2013) · Plastic & Reconstructive Surgery Global Open · doi:10.1097/gox.0b013e318299d02b · Cited by 43
Zeng, X.-X., Bian, Z., Wu, T.-X., Fu, S.-F., Ziea, E. T. C., & Woon, W. T. C. (2011) · The American Journal of Chinese Medicine · doi:10.1142/s0192415x11009408 · Cited by 43
“Since some scholars had pointed out that the weighing applied in scales to different study domains was inconsistent, we assessed the methodological quality of included studies using a checklist that covered the principles and potential sources of bias, as recommended (Sanderson et al., 2007), namely, study question, s…”
MethodThe management of secondary lower limb lymphoedema in cancer patients: A systematic review
Leung, E. Y. L., Tirlapur, S., & Meads, C. (2014) · Palliative Medicine · doi:10.1177/0269216314545803 · Cited by 43
A Primer on Performing Systematic Reviews and Meta-analyses
Umscheid, C. A. (2013) · Clinical Infectious Diseases · doi:10.1093/cid/cit333 · Cited by 44
“judge the risk of bias of non-RCTs [43, 44] and diagnostic studies [45].”
BackgroundConen, D., Arendacká, B., Röver, C., Bergau, L., Expósito, P. M., Wijers, S. C., Sticherling, C., Zabel, M., & Friede, T. (2016) · PLoS ONE · doi:10.1371/journal.pone.0162756 · Cited by 43
“ssessed the risk of bias regarding three domains (selection of participants, measurement of variables and outcomes, control of confounding) specific to observational studies as previously recommended [14].The study quality was assessed by two reviewers (BA, CR). Data extraction Hazard ratios quantifying the effect of…”
MethodNwolise, C. H., Carey, N., & Shawe, J. (2016) · Journal of Medical Internet Research · doi:10.2196/jmir.5615 · Cited by 43
“It contains summary judgments and an accompanying dictionary that increases standardization of the study quality assessment [51,52].”
Shame and guilt in the postnatal period: A systematic review
Caldwell, J., Meredith, P., Whittingham, K., & Ziviani, J. (2021) · Journal of Reproductive and Infant Psychology · doi:10.1080/02646838.2020.1754372 · Cited by 42
“The STROBE assessment is a comprehensive tool designed to assess the methodological rigour of studies, including cohort, case-control and cross-sectional designs (Sanderson et al., 2007).”
Periodontitis and risk of immune‐mediated systemic conditions: A systematic review and meta‐analysis
Larvin, H., Kang, J., Aggarwal, V. R., Pavitt, S., & Wu, J. (2023) · Community Dentistry and Oral Epidemiology · doi:10.1111/cdoe.12812 · Cited by 42
van Ditshuizen, J. C., Rojer, L. A., Lieshout, E. M. M. V., Bramer, W. M., Verhofstad, M. H. J., Sewalt, C. A., & Hartog, D. D. (2023) · The Journal of Trauma: Injury, Infection, and Critical Care · doi:10.1097/ta.0000000000003890 · Cited by 42
“In addition, a quality assessment tool, based on existing literature, was created to assess quality, generalizability, and risk of bias of the included studies.(18) For data synthesis J.”
MethodMoshel, M. L., Warburton, W., Batchelor, J., Bennett, J. M., & Ko, K. Y. (2023) · Neuropsychology Review · doi:10.1007/s11065-023-09612-4 · Cited by 44
“Given the modification of the scale for the purposes of this review and consistent with recommendations, we did not include an overall rating summary (O’Connor et al., 2015; Robinson et al., 2021; Sanderson et al., 2007).”
MethodBeyerlein, A., Ness, A., Streuling, I., Hadders‐Algra, M., & von Kries, R. (2010) · American Journal of Clinical Nutrition · doi:10.3945/ajcn.2009.29116 · Cited by 42
“To identify the relevant literature, we searched the databases MEDLINE (1950–2010; www.ncbi.nlm.nih.gov/pubmed), PSYCHInfo (1800–2010; www.apa.org/pubs/databases/psycinfo/index. aspx), and Web of Science (1900–2010; www.isiknowledge.com) by using the keywords (“Child” OR “Infant” OR “Childhood”) AND (“weight gain” OR…”
MethodInfluentialExtent and predictors of work-related distress in community correction officers: a systematic review
Page, J., & Robertson, N. (2021) · Psychiatry Psychology and Law · doi:10.1080/13218719.2021.1894259 · Cited by 42
“Whilst there are a range of validated tools for assessing the quality of other study designs, such as randomised-controlled trials, observational and case-control studies, the number of tools for appraising crosssectional studies is limited (Sanderson et al., 2007; Zeng et al., 2015).”
BackgroundStanhope, J., & Weinstein, P. (2022) · Research Synthesis Methods · doi:10.1002/jrsm.1609 · Cited by 42
Witteveen, E., Wieske, L., Verhamme, C., Schultz, M. J., van Schaik, I. N., & Horn, J. (2014) · Journal of the Neurological Sciences · doi:10.1016/j.jns.2014.07.014 · Cited by 42
“A formal quality assessment using a quality assessment tool was not performed, since there is no optimal tool for quality assessment of observational studies including case series [44].”
MethodChoo, P. L., Gallagher, H. L., Morris, J. H., Pomeroy, V. M., & van Wijck, F. (2015) · Brain and Behavior · doi:10.1002/brb3.411 · Cited by 42
“This tool was selected after considering the research questions and recommendations from four reviews (US Department of Health and Human Services, 2002; Deeks et al. 2003; Sanderson et al. 2007; Crowe and Sheppard 2011).”
MethodManderlier, A., de Fooz, M., Patris, S., & Berquin, A. D. (2022) · Annals of Physical and Rehabilitation Medicine · doi:10.1016/j.rehab.2022.101660 · Cited by 41
Acoustic Analysis of Voice in Singers: A Systematic Review
Gunjawate, D. R., Ravi, R., & Rajashekhar, B. (2018) · Journal of Speech Language and Hearing Research · doi:10.1044/2017_jslhr-s-17-0145 · Cited by 41
Prevalência de tuberculose na população privada de liberdade: revisão sistemática e metanálise
Moreira, T. R., Lemos, A. C., Colodette, R. M., Gomes, A. P., & Siqueira‐Batista, R. (2019) · Revista Panamericana de Salud Pública · doi:10.26633/rpsp.2019.16 · Cited by 41
Antibiotic use and the risk of breast cancer: A systematic review and dose-response meta-analysis
Simin, J., Tamimi, R. M., Engstrand, L. G., Callens, S. F. J., & Brusselaers, N. (2020) · Pharmacological Research · doi:10.1016/j.phrs.2020.105072 · Cited by 41
Glasgow, M. J., Edlin, R. P., & Harding, J. E. (2020) · BMJ Open · doi:10.1136/bmjopen-2020-037324 · Cited by 41
Instrumented Analysis of the Sit-to-Stand Movement for Geriatric Screening: A Systematic Review
Shukla, B. K., Bassement, J. N. C., Vijay, V., Yadav, S. K., & Hewson, D. J. (2020) · Bioengineering · doi:10.3390/bioengineering7040139 · Cited by 41
“The criteria adopted to appraise articles for this review was the method of Loney and colleagues [25] and Sanderson and colleagues [26], and modified by Payette and colleagues [27], which is suitable for observational studies.”
MethodBurton, A. J., Gilbert, R., Tilling, K. M., Langdon, R., Donovan, J., Holly, J. M. P., & Martin, R. M. (2021) · Scientific Reports · doi:10.1038/s41598-020-79345-4 · Cited by 41
Corrêa‐Faria, P., & Petti, S. (2015) · Dental Traumatology · doi:10.1111/edt.12172 · Cited by 41
“Nevertheless, meta-analysis is an alternative method to estimate the strength of the evidence, as it helps overcome the discrepancies between studies and offers a means to summarize the body of identified evidence quantitatively (14).”
MethodInjuries to Individuals Participating in Mountain and Wilderness Sports: A Review
Mort, A. J., & Godden, D. (2011) · Clinical Journal of Sport Medicine · doi:10.1097/jsm.0b013e318234be97 · Cited by 41
“A new matrix was developed, which included a total of 11 quality items; no gold standard matrix for appraising observational studies existed.(14) Each item was assigned to 1 of 3 categories; the categories were as follows: “A”: criteria met in full, “B”: criteria met in part, and “C”: criteria not met.”
MethodBekkering, W. P., Vlieland, T. P. M. V., Fiocco, M. F., Koopman, H. M., Schoones, J. W., Nelissen, R. G. H. H., & Taminiau, A. H. M. (2011) · Surgical Oncology · doi:10.1016/j.suronc.2011.09.002 · Cited by 41
“From a wide variation of tools and scales available to evaluate themethodological qualityof studies [37]weselected theSTROBEguidelines.”
BackgroundPijls, B. G. C. W., Dekkers, O. M., Middeldorp, S., Valstar, E. R., van der Heide, H. J. L., der Zwaag, H. M. J. V. D. L.-V., & Nelissen, R. G. H. H. (2011) · BMC Musculoskeletal Disorders · doi:10.1186/1471-2474-12-173 · Cited by 40
“org/ [18,19].”
Family correlates of childhood binge eating: A systematic review
Saltzman, J. A., & Liechty, J. M. (2016) · Eating Behaviors · doi:10.1016/j.eatbeh.2016.03.027 · Cited by 41
Socio‐economic differences in HIV / AIDS mortality in South Africa
Probst, C., Parry, C. D. H., & Rehm, J. T. (2016) · Tropical Medicine & International Health · doi:10.1111/tmi.12712 · Cited by 41
“A quality rating for the particular needs of this research question was developed focusing on core quality aspects of observational studies with respect to design, measurement and reporting [33].”
BackgroundRao, A., Brück, K., Methven, S., Evans, R., Stel, V. S., Jager, K. J., Hooft, L., Ben-Shlomo, Y., & Caskey, F. (2016) · PLOS ONE · doi:10.1371/journal.pone.0155078 · Cited by 35
“There are a number of assessment tools that have been developed to assess quality and susceptibility to bias in observational studies with only half of the identified tools have described their development or validity and reliability [26].”
BackgroundMethodInfluentialFawson, S., Moon, Z., Novogrudsky, K., Moxham, F., Forster, K., Tribe, I., Moss-Morris, R., Johnson, C., & Hughes, L. D. (2024) · Health Psychology Review · doi:10.1080/17437199.2023.2261518 · Cited by 40
“Risk of bias was assessed in all studies using a checklist adapted from ROBINS-E (Morgan et al., 2017) and a checklist used by Pasma et al. (2013) following guidelines on areas of bias (Dekkers et al., 2019; Sanderson et al., 2007).”
MethodHernández‐Bojorge, S., Campos, A., Parikh, J., Beckstead, J. W., Lajeunesse, M. J., & Wildman, D. E. (2023) · International Journal of Mental Health Nursing · doi:10.1111/inm.13257 · Cited by 40
Arcos‐Machancoses, J. V., Hernández, C., de Carpi, J. M., & Pisón, S. P. (2018) · Diseases of the Esophagus · doi:10.1093/dote/dox158 · Cited by 40
Terry, D. R., Peck, B., Carden, C., Perkins, A. J., & Smith, A. C. (2020) · European Journal of Investigation in Health Psychology and Education · doi:10.3390/ejihpe10030051 · Cited by 40
“[31], indicated that there are a plethora of instruments with various methods of examining risk of bias.”
MethodSłabuszewska-Jóźwiak, A., Szymański, J. K., Jóźwiak, Ł., & Sarecka‐Hujar, B. (2021) · Journal of Clinical Medicine · doi:10.3390/jcm10040675 · Cited by 41
“The studies included in the research were reviewed critically for bias analysis using the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) checklist [17].”
MethodZhang, X., Wu, Z., Peng, X., Wu, A., Yue, Y., Martin, J. E., & Cheng, D. (2010) · Journal of Cardiothoracic and Vascular Anesthesia · doi:10.1053/j.jvca.2010.09.021 · Cited by 42
Coles, E., Themessl-Huber, M., & Freeman, R. (2012) · Health Education Research · doi:10.1093/her/cys065 · Cited by 40
“Qualitative studies were not assessed for bias, given the lack of consensus regarding effective tools to assess bias within qualitative studies [33].”
BackgroundMay, S., Runge, N., & Aina, A. (2018) · Musculoskeletal Science and Practice · doi:10.1016/j.msksp.2018.09.006 · Cited by 40
Shah, A., Jones, M. P., & Holtmann, G. J. (2020) · Indian Journal of Gastroenterology · doi:10.1007/s12664-020-01107-x · Cited by 39
“Several tools are available that can be utilized to assess the quality of the studies that are to be included in the systematic reviews and meta-analysis [15].”
Method
Metadata
- Authors
- Sanderson, S., Tatt, I. D., & Higgins, J. P.
- Affiliations
- University of Cambridge (GB)
- F. Hoffmann-La Roche AG (CH)
- MRC Biostatistics Unit (GB)
- Corresponding author
- Sanderson, S.
- ORCID
- Higgins, J. P.
- Venue
- International Journal of Epidemiology
- Publisher
- Oxford University Press (OUP)
- Published
- 2007-04-30
- Volume
- 36
- Issue
- 3
- Pages
- 666-676
- ISSN
- 0300-5771 (ISSN-L)
- 1464-3685
- OpenAlex
- W2171251365
- Cited by
- 1,531
- References
- 73
- Citations by year
- 2026: 22 · 2025: 38 · 2024: 42 · 2023: 59 · 2022: 80 · 2021: 96 · 2020: 101 · 2019: 105 · 2018: 125 · 2017: 104 · 2016: 150 · 2015: 118 · 2014: 122 · 2013: 111 · 2012: 96
- FWCI
- 7.09
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- International Journal of Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- quality assessment tools, bias assessment, observational studies, critical appraisal, checklists, rating scales, STROBE, confounding adjustment, selection bias, measurement bias, conflicts of interest
- MeSH terms
- Humans, Research Design, Bias, Epidemiologic Studies, Evidence-Based Medicine, Observation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.