Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography
The DOI matches its Crossref record; 73 of the 73 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Assessing quality and susceptibility to bias is essential when interpreting primary research and conducting systematic reviews and meta-analyses. Tools for assessing quality in clinical trials are well-described but much less attention has been given to similar tools for observational epidemiological studies. METHODS: Tools were identified from a search of three electronic databases, bibliographies and an Internet search using Google. Two reviewers extracted data using a pre-piloted extraction form and strict inclusion criteria. Tool content was evaluated for domains potentially related to bias and was informed by the STROBE guidelines for reporting observational epidemiological studies. RESULTS: A total of 86 tools were reviewed, comprising 41 simple checklists, 12 checklists with additional summary judgements and 33 scales. The number of items ranged from 3 to 36 (mean 13.7). One-third of tools were designed for single use in a specific review and one-third for critical appraisal. Half of the tools provided development details, although most were proposed for future use in other contexts. Most tools included items for selection methods (92%), measurement of study variables (86%), design-specific sources of bias (86%), control of confounding (78%) and use of statistics (78%); only 4% addressed conflict of interest. The distribution and weighting of domains across tools was variable and inconsistent. CONCLUSION: A number of useful assessment tools have been identified by this report. Tools should be rigorously developed, evidence-based, valid, reliable and easy to use. There is a need to agree on critical elements for assessing susceptibility to bias in observational epidemiology and to develop appropriate evaluation tools.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Such instruments have been developed by other groups and were the subject of a recent systematic review [28].”
From The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: Guidelines for Reporting Observational Studies (2007) · cites this paper as [28]
In the citing paper
“It is not clear how weights should be assigned to each item of the scale [18].”
From Tools used to assess the quality of peer review reports: A methodological systematic review (2019) · cites this paper as [18]
In the citing paper
“The reviewers concluded that there is no ‘single obvious candidate tool for assessing quality of observational epidemiological studies’.”
From COSMOS-E: Guidance on conducting systematic reviews and meta-analyses of observational studies of etiology (2019) · cites this paper as [45]
In the citing paper
“A review identified more than 80 tools for assessing risk of bias in nonrandomised studies [45].”
From COSMOS-E: Guidance on conducting systematic reviews and meta-analyses of observational studies of etiology (2019) · cites this paper as [45]
In the citing paper
“The need for complete reporting of all relevant methodological aspects as a prerequisite to assessing the risk of bias of studies was highlighted in 10 articles, [16, 33, 35, 47, 50, 56, 86, 87, 90, 91] and was the focus of recommendations in a publication from the Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group [12].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [90]
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Meta-analysis of Observational Studies in Epidemiology<SUBTITLE>A Proposal for Reporting</SUBTITLE>
Stroup, D. F. (2000) · JAMA · doi:10.1001/jama.283.15.2008 · Cited by 20,623
Downs, S. H., & Black, N. (1998) · Journal of Epidemiology and Community Health · doi:10.1136/jech.52.6.377 · Cited by 8,252
Moher, D., Cook, D. J., Eastwood, S., Olkin, I., Rennie, D., & Stroup, D. F. (1999) · The Lancet · doi:10.1016/s0140-6736(99)04149-5 · Cited by 4,826
Whiting, P., Rutjes, A. W., Reitsma, J. B., Bossuyt, P. M., & Kleijnen, J. (2003) · BMC Medical Research Methodology · doi:10.1186/1471-2288-3-25 · Cited by 3,592
Assessing the quality of controlled clinical trials
Jüni, P., Altman, D. G., & Egger, M. (2001) · BMJ · doi:10.1136/bmj.323.7303.42 · Cited by 3,309 · cited 5×
“a (5) (5) (5) ( 5 ) N N LongneckerCoh”
InfluentialUsers' guides to the medical literature
Guyatt, G. H. (1993) · JAMA · doi:10.1001/jama.270.17.2096 · Cited by 2,970
Evaluating non-randomised intervention studies
Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D. (2003) · Health Technology Assessment · doi:10.3310/hta7270 · Cited by 2,907
Cognitive and Behavioral Outcomes of School-Aged Children Who Were Born Preterm: A Meta-analysis
Bhutta, A. T., Cleves, M. A., Casey, P. H., Cradock, M. M., & Anand, K. J. S. (2002) · JAMA · doi:10.1001/jama.288.6.728 · Cited by 2,392 · cited 1×
“Y N N 9 (22) 2 (17) 5 (15) 16 (19) Y Y N 15 (36) 6 ( 50 ) 7 (21) 28 (32)”
The Hazards of Scoring the Quality of Clinical Trials for Meta-analysis
Jüni, P. (1999) · JAMA · doi:10.1001/jama.282.11.1054 · Cited by 1,981
Low grade inflammation and coronary heart disease: prospective study and updated meta-analyses
Danesh, J. (2000) · BMJ · doi:10.1136/bmj.321.7255.199 · Cited by 1,692
Research works printed but not linked: 7
- Carneiro, 2002, Revista portuguesa de cardiologia : org o oficial da Sociedade Portuguesa de Cardiologia = Portuguese journal of cardiology : an official journal of the Portuguese Society of Cardiology, 21, 891
- 1981, Canadian Medical Association Journal, 124, 985
- 2002, SOUTH AFRICAN J CLIN NUTR, 15, 68
- 2000, Chronic diseases in Canada, 19, 170
- Ciliska, 1996, Canadian journal of public health. Revue canadienne de sant publique, 87, 193
- 1994, Canadian journal of public health. Revue canadienne de sant publique, 85, S8
- Spitzer, 1990, Clinical and investigative medicine. M decine clinique et experimentale, 13, 17
Cited by
Items 201–300 of 1,006
Barriuso, L., Miqueleiz, E., Albaladejo, R., Villanueva, R., Santos, J. M., & Regidor, E. (2015) · BMC Pediatrics · doi:10.1186/s12887-015-0443-3 · Cited by 191
“Since there are a variety of checklist and scales to evaluate the quality of observational studies and they differ by content, format, validity and applicability to different studies [14, 15], none of the proposed instruments was used.”
BackgroundDuration of Pertussis Immunity After DTaP Immunization: A Meta-analysis
McGirr, A., & Fisman, D. N. (2015) · Pediatrics · doi:10.1542/peds.2014-1729 · Cited by 191
Gu, J., Groot, G., Boden, C., Busch, A., Holtslander, L., & Lim, H. (2018) · Clinical Breast Cancer · doi:10.1016/j.clbc.2017.12.013 · Cited by 190
Methodological quality of COVID-19 clinical research
Jung, R. G., Di Santo, P., Clifford, C., Prosperi-Porta, G., Skanes, S., Hung, A., Parlow, S., Visintini, S., Ramirez, F. D., Simard, T., et al. (2021) · Nature Communications · doi:10.1038/s41467-021-21220-5 · Cited by 189
Müller, R., Peter, C., Cieza, A., & Geyh, S. (2012) · Spinal Cord · doi:10.1038/sc.2011.116 · Cited by 188
Ferguson, K. D., McCann, M., Katikireddi, S. V., Thomson, H., Green, M. J., Smith, D. J., & Lewsey, J. D. (2020) · International Journal of Epidemiology · doi:10.1093/ije/dyz150 · Cited by 188
“For example, integrating one of the many tools available for assessing risk of bias in observational studies could help systemise the level of confidence reviewers have in particular relationships.(35) Further, an I-DAG can be refined with algorithmic and data-driven techniques.”
Hempel, S., Shekelle, P. G., Liu, J. L., Sherwood Danz, M., Foy, R., Lim, Y.-W., Motala, A., & Rubenstein, L. V. (2015) · BMJ Quality & Safety · doi:10.1136/bmjqs-2014-003151 · Cited by 187
Methadone dose and neonatal abstinence syndrome—systematic review and meta‐analysis
Cleary, B. J., Donnelly, J., Strawbridge, J., Gallagher, P. J., Fahey, T., Clarke, M., & Murphy, D. J. (2010) · Addiction · doi:10.1111/j.1360-0443.2010.03120.x · Cited by 187
Padula, R. S., Comper, M. L. C., Sparer, E. H., & Dennerlein, J. T. (2017) · Applied Ergonomics · doi:10.1016/j.apergo.2016.07.018 · Cited by 150
“2.4 Methodological Quality Assessment of Studies Identified studies were assessed for their quality according to internal and external validity criteria (Sanderson et al., 2007).”
MethodInfluentialDo Couple-Based Interventions Make a Difference for Couples Affected by Cancer?: A Systematic Review
Regan, T. W., Lambert, S. D., Girgis, A., Kelly, B., Kayser, K., & Turner, J. (2012) · BMC Cancer · doi:10.1186/1471-2407-12-279 · Cited by 185
AL Wachami, N., Guennouni, M., Iderdar, Y., Boumendil, K., Arraji, M., Mourajid, Y., Bouchachi, F. Z., Barkaoui, M., Louerdi, M. L., Hilali, A., et al. (2024) · BMC Public Health · doi:10.1186/s12889-024-17686-9 · Cited by 185
The effects of safety checklists in medicine: a systematic review
THOMASSEN, Ø., STORESUND, A., SØFTELAND, E., & BRATTEBØ, G. (2014) · Acta Anaesthesiologica Scandinavica · doi:10.1111/aas.12207 · Cited by 185
“Currently, no quality assessment tool is regarded as a ‘gold standard’ for observational studies.(59) Although the studies included in this review vary widely in study design, settings, number of participants, and outcome measures, they have all been assessed using the same quality assessment tool.”
MethodAsylum Seekers, Violence and Health: A Systematic Review of Research in High-Income Host Countries
Kalt, A., Hossain, M., Kiss, L., & Zimmerman, C. (2013) · American Journal of Public Health · doi:10.2105/ajph.2012.301136 · Cited by 142
MacCannell, T., Umscheid, C. A., Agarwal, R. K., Lee, I., Kuntz, G., Stevenson, K. B., & Healthcare Infection Control Practices Advisory Committee (HICPAC) (2011) · Infection Control & Hospital Epidemiology · doi:10.1086/662025 · Cited by 183
The Resident Assessment Instrument-Minimum Data Set 2.0 quality indicators: a systematic review
Hutchinson, A. M., Milke, D. L., Maisey, S., Johnson, C., Squires, J. E., Teare, G., & Estabrooks, C. A. (2010) · BMC Health Services Research · doi:10.1186/1472-6963-10-166 · Cited by 182
“In accordance with the recommendations of Sanderson et al. [ 30 ], we selected this instrument because it covers a small number of key Table 2: RAI-MDS Quality Indicator Search”
MethodSTROBE (STrengthening the Reporting of Observational studies in Epidemiology)
Cevallos, M., & Egger, M. (2014) · Wiley eBooks · doi:10.1002/9781118715598.ch17 · Cited by 182
Belo, V. S., Werneck, G. L., Barbosa, D. S., Simões, T. C., Nascimento, B. W. L., da Silva, E. S., & Struchiner, C. J. (2013) · PLoS neglected tropical diseases · doi:10.1371/journal.pntd.0002182 · Cited by 181
“Although quality analysis performed in this review also has limitations arising from the fact that there is no single standardized tool recommended to evaluate observational studies [25], in particular regarding specific issues of ecological studies, several limitations and susceptibility to bias could be identified i…”
BackgroundMethodHoppen, T. H., & Morina, N. (2019) · European Journal of Psychotraumatology · doi:10.1080/20008198.2019.1578637 · Cited by 180
Bareeqa, S. B., Ahmed, S. I., Samar, S. S., Yasin, W., Zehra, S., Monese, G. M., & Gouthro, R. V. (2021) · The International Journal of Psychiatry in Medicine · doi:10.1177/0091217420978005 · Cited by 180
“Initially, relevant data and studies were extracted by two independent investigators in accordance with The Meta-analysis Of Observational Studies in Epidemiology standard (MOOSE)10 and Strengthening the reporting of observational studies in epidemiology (STROBE)11 22-item checklists, modified according to Sanderson e…”
MethodFraser, S. D. S., Roderick, P. J., Casey, M., Taal, M. W., Yuen, H. M., & Nutbeam, D. (2013) · Nephrology Dialysis Transplantation · doi:10.1093/ndt/gfs371 · Cited by 179
“Study quality was assessed using the following criteria (guided by a review of tools for assessing quality of observational studies) [35]: setting of sample (e.”
MethodPrevalence and Predictors of Burnout in Midwives: A Systematic Review and Meta-Analysis
Suleiman‐Martos, N., Albendín‐García, L., Gómez‐Urquiza, J. L., Vargas-Román, K., Ramírez‐Baena, L., Ortega‐Campos, E., & Fuente-Solana, E. I. D. L. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17020641 · Cited by 181
Larvin, H., Kang, J. X., Aggarwal, V. R., Pavitt, S. H., & Wu, J. (2020) · Clinical and Experimental Dental Research · doi:10.1002/cre2.336 · Cited by 179
Gärtner, F. R., Nieuwenhuijsen, K., van Dijk, F. J. H., & Sluiter, J. K. (2010) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2010.03.013 · Cited by 181
“the STROBE statement) and an example in previous studies (Downs and Black, 1998; Sanderson et al., 2007; Vandenbroucke et al., 2007; Von Elm et al., 2008; Wind et al., 2005).”
Noori, H. R., Cosa Linan, A., & Spanagel, R. (2016) · European Neuropsychopharmacology · doi:10.1016/j.euroneuro.2016.06.013 · Cited by 178
“The quality scores of the reviewed studies (Stroup et al., 2000; Sanderson et al., 2007; Poldrack et al., 2008) (Supplement eTable 1–3) also suggest that the quality of reporting the methodological details of the included experiments varied considerably.”
BackgroundTomasi, C., & Derks, J. (2012) · Journal of Clinical Periodontology · doi:10.1111/j.1600-051x.2011.01831.x · Cited by 149
Brusselaers, N., Mattsson, F., & Lagergren, J. (2014) · Gut · doi:10.1136/gutjnl-2013-306074 · Cited by 177
Evidence-Based Preconceptional Lifestyle Interventions
Temel, S., van Voorst, S. F., Jack, B. W., Denktaş, S., & Steegers, E. A. P. (2014) · Epidemiologic Reviews · doi:10.1093/epirev/mxt003 · Cited by 139
“A quality assessment checklist was constructed on the basis of the results of a systematic review evaluating tools for assessing quality and susceptibility to bias in observational studies and the Cochrane Handbook regarding quality assessment for randomized controlled trials (9, 10).”
MethodRisk Factors for Repetition of Self-Harm: A Systematic Review of Prospective Hospital-Based Studies
Larkin, C., Di Blasi, Z., & Arensman, E. (2014) · PLoS ONE · doi:10.1371/journal.pone.0084282 · Cited by 140
“In line with a systematic review of existing quality assessment tools [17], our original instrument incorporates items for five core quality concerns including selecting study participants, measuring outcomes, addressing design-specific sources of bias, control of confounding and analysing data [19].”
Malnutrition in childhood cancer patients: A review on its prevalence and possible causes
Brinksma, A., Huizinga, G., Sulkers, E., Kamps, W., Roodbol, P., & Tissing, W. (2012) · Critical Reviews in Oncology/Hematology · doi:10.1016/j.critrevonc.2011.12.003 · Cited by 175
Harkness, E. L., Mullan, B., & Blaszczynski, A. (2015) · Cyberpsychology, Behavior, and Social Networking · doi:10.1089/cyber.2014.0343 · Cited by 140
Embleton, L., Mwangi, A., Vreeman, R., Ayuku, D., & Braitstein, P. K. A. (2013) · Addiction · doi:10.1111/add.12252 · Cited by 175
Javidi, H., Vettore, M. V., & Benson, P. E. (2017) · American Journal of Orthodontics and Dentofacial Orthopedics · doi:10.1016/j.ajodo.2016.12.011 · Cited by 175
A principal component analysis is conducted for a case series quality appraisal checklist
Guo, B., Moga, C., Harstall, C., & Schopflocher, D. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.07.010 · Cited by 174
“However, no widely accepted and validated quality appraisal tool exists for assessing the methodologic quality of case-series studies [2,17].”
BackgroundThe Epidemiology of Injuries in Mixed Martial Arts: A Systematic Review and Meta-analysis
Lystad, R. P., Gregory, K., & Wilson, J. (2014) · Orthopaedic Journal of Sports Medicine · doi:10.1177/2325967113518492 · Cited by 174
“However, in the absence of a single obvious candidate tool for assessing quality of observational epidemiological studies, the STROBE Statement checklist should suffice as a good starting point for assessing the quality of observational studies.(31) Moreover, the STROBE Statement checklist has been adopted as a qualit…”
MethodZimmermann, H., Zimmermann, N., Hagenfeld, D., Veile, A., Kim, T., & Becher, H. (2014) · Community Dentistry And Oral Epidemiology · doi:10.1111/cdoe.12126 · Cited by 174
Karr, J. E., Graham, R. B., Hofer, S. M., & Muniz-Terrera, G. (2018) · Psychology and Aging · doi:10.1037/pag0000236 · Cited by 145
Hanna, N., Tamhankar, A. J., & Stålsby Lundborg, C. (2023) · The Lancet Planetary Health · doi:10.1016/s2542-5196(22)00254-6 · Cited by 172
Hootman, J. M., Driban, J. B., Sitler, M. R., Harris, K. P., & Cattano, N. M. (2011) · Research Synthesis Methods · doi:10.1002/jrsm.41 · Cited by 172
“, case-control, cohort) are more diverse, there are fewer quality rating instruments available, authors often create their own study-specific quality rating tool, and data on validity and reliability are sparse (Deeks et al., 2003; Sanderson et al., 2007; Mallen et al., 2006; Simunovic et al., 2009).”
BackgroundCuello-Garcia, C. A., Santesso, N., Morgan, R. L., Verbeek, J., Thayer, K., Ansari, M. T., Meerpohl, J., Schwingshackl, L., Katikireddi, S. V., Brozek, J. L., et al. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.11.026 · Cited by 172
Gane, E. M., Michaleff, Z. A., Cottrell, M. A., McPhail, S. M., Hatton, A. L., Panizza, B. J., & O'Leary, S. P. (2017) · European Journal of Surgical Oncology (EJSO) · doi:10.1016/j.ejso.2016.10.026 · Cited by 172
“However, there is no “gold stan d rd” assessment tool for epidemiological studies [64].”
The childbirth experience: A systematic review of predictors and outcomes
McKelvin, G., Thomson, G., & Downe, S. (2021) · Women and Birth · doi:10.1016/j.wombi.2020.09.021 · Cited by 171
“Sanderson et al (2007) in a review to assess tools available for observational studies despite making no suggestions of specific tools, recommended that ‘tools should (i) include a small number of key domains; (ii) be as specific as possible (with due consideration of the particular study design…”
BackgroundSpirituality and religion in older adults with dementia: a systematic review
Agli, O., Bailly, N., & Ferrand, C. (2015) · International Psychogeriatrics · doi:10.1017/s1041610214001665 · Cited by 170
“In accordance with the recommendations of Sanderson et al. (2007), articles were divided into two categories: “moderate to high quality” or “low quality.”
MethodMavaddat, N., Parker, R. A., Sanderson, S., Mant, J., & Kinmonth, A. L. (2014) · PLoS ONE · doi:10.1371/journal.pone.0103509 · Cited by 170
“Assessment of study quality was primarily concerned with identifying risk of bias, rather than merely examining the quality of article reporting [10] using the Newcastle-Ottawa scales.”
MethodSelf-citationEpidemiology of injuries in competition taekwondo: A meta-analysis of observational studies
Lystad, R. P., Pollard, H. P., & Graham, P. L. (2008) · Journal of science and medicine in sport · doi:10.1016/j.jsams.2008.09.013 · Cited by 171
Systematic review of the prevalence of faecal incontinence
Sharma, A., Yuan, L., Marshall, R. J., Merrie, A. E. H., & Bissett, I. P. (2016) · British journal of surgery · doi:10.1002/bjs.10298 · Cited by 170
Body mass index in Parkinson’s disease: A meta-analysis
van der Marck, M. A., Dicke, H. C., Uc, E. Y., Kentin, Z. H. A., Borm, G. F., Bloem, B. R., Overeem, S., & Munneke, M. (2012) · Parkinsonism & Related Disorders · doi:10.1016/j.parkreldis.2011.10.016 · Cited by 169
Serum 25-hydroxyvitamin D levels and the risk of depression: A systematic review and meta-analysis
Ju, S.-Y., Lee, Y.-J., & Jeong, S.-N. (2013) · The Journal of nutrition, health and aging · doi:10.1007/s12603-012-0418-0 · Cited by 147
“The methodological quality of the eligible nonrandomized studies reviewed in meta-analyses was assessed according to the 7 domains proposed as fundamental to the methodological quality of the included studies (17, 18).”
Harmsen, M. J., Wong, C. F. C., Mijatovic, V., Griffioen, A. W., Groenman, F. A., Hehenkamp, W. J. K., & Huirne, J. A. F. (2019) · Human Reproduction Update · doi:10.1093/humupd/dmz024 · Cited by 172
Manchikanti, L. (2009) · Pain Physician · doi:10.36076/ppj.2009/12/819 · Cited by 168
Pasipanodya, J. G., Ogbonna, D., Ferro, B. E., Magombedze, G., Srivastava, S., Deshpande, D., & Gumbo, T. (2017) · Antimicrobial Agents and Chemotherapy · doi:10.1128/aac.01206-17 · Cited by 167
“The ROBIN-I tool addresses several weaknesses identified in previous instruments used to measure study quality in observational studies, is easy to use, is easy to interpret, and is highly reproducible (39–41).”
Predictors of smoking cessation during pregnancy: a systematic review and meta‐analysis
Riaz, M., Lewis, S., Naughton, F., & Ussher, M. (2018) · Addiction · doi:10.1111/add.14135 · Cited by 166
“has known limitations, such as it often does not inform the evidence synthesis process, particularly for systematic reviews of non-randomised studies, including sensitivity analysis, narrative assessment and restricting the synthesis to studies at a lower risk of bias, which have been discussed elsewhere in some detai…”
BackgroundForshaw, J., Gerver, S. M., Gill, M., Cooper, E., Manikam, L., & Ward, H. (2017) · BMC Infectious Diseases · doi:10.1186/s12879-017-2890-y · Cited by 165
“All studies were kept in regardless of quality due to the small number of studies available and recognition of the limitations of the scoring systems [10, 12].”
BackgroundThe epidemiology and risk factors for recurrence after inguinal hernia surgery.
Burcharth, J. (2014) · PubMed · Cited by 165
Kopp, A., & Jekauc, D. (2018) · Sports · doi:10.3390/sports6040175 · Cited by 168
“It would also not be clear how weights for different items should be determined, and different scales might have reached different conclusions on the overall quality of an individual study [101].”
BackgroundThe prevalence of unwanted online sexual exposure and solicitation among youth: A meta-analysis
Madigan, S., Villani, V., Azzopardi, C., Laut, D., Smith, T., Temple, J. R., Browne, D., & Dimitropoulos, G. (2018) · Journal of Adolescent Health · doi:10.1016/j.jadohealth.2018.03.012 · Cited by 164
Eneriz-Wiemer, M., Sanders, L. M., Barr, D. A., & Mendoza, F. S. (2014) · Academic Pediatrics · doi:10.1016/j.acap.2013.10.003 · Cited by 164
Lau, A., Bradshaw, C. S., Lewis, D., Fairley, C. K., Chen, M. Y., Kong, F. Y. S., & Hocking, J. S. (2015) · Clinical Infectious Diseases · doi:10.1093/cid/civ644 · Cited by 164
“their systematic review of tools used to assess bias in observational studies [25].”
BackgroundGagnier, J. J., Morgenstern, H., & Chess, L. (2013) · The American Journal of Sports Medicine · doi:10.1177/0363546512458227 · Cited by 136
“Criteria were developed by referring to several resources that describe the accepted methodological aspects of such trials that might bias the estimates of treatment effect.(12,13,31,32) The criteria used to assess the risk of bias, together with their operational definitions, for randomized trials are listed in Appen…”
MethodAsthma inhaler adherence determinants in adults: systematic review of observational data
Dima, A. L., Hernández, G., Cunillera, O., Ferrer, M., de Bruin, M., & the ASTRO-LAB group (2014) · European Respiratory Journal · doi:10.1183/09031936.00172114 · Cited by 164
“Neither examined the quality of the methodology of included studies, which is important in interpreting empirical evidence [21 – 23].”
BackgroundResultMoon, Z., Moss-Morris, R., Hunter, M. S., Carlisle, S., & Hughes, L. D. (2017) · Patient Preference and Adherence · doi:10.2147/ppa.s126651 · Cited by 148
“The QA tool was adapted from Pasma et al 21 based on recommendations from Sanderson et al. 22 Studies were assessed on methods for selecting study participants and measuring study variables, appropriate statistical analyses, loss to follow-up and removal of nonpatient-initiated nonadherence (eg, due to contraindicatio…”
MethodMeta‐analysis: proton pump inhibitor use and the risk of community‐acquired pneumonia
JOHNSTONE, J., NERENBERG, K., & LOEB, M. (2010) · Alimentary Pharmacology & Therapeutics · doi:10.1111/j.1365-2036.2010.04284.x · Cited by 163
Prevalence, reasons, perceived effects, and correlates of medical marijuana use: A review
Park, J.-Y., & Wu, L.-T. (2017) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2017.03.009 · Cited by 161
“In terms of quality assessment, studies were excluded if they used a less rigorous sampling strategy (self-selected, convenience, or purposive sample), if they did not use appropriate measurement tools for exposure and outcome variables, or if they had a particular source of bias (Sanderson et al., 2007).”
Backgroundde Oliveira, L. H., Camacho, L. A. B., Coutinho, E. S. F., Martinez-Silveira, M. S., Carvalho, A. F., Ruiz‐Matus, C., & Toscano, C. M. (2016) · PLoS ONE · doi:10.1371/journal.pone.0166736 · Cited by 161
Associations between social jetlag and mental health in young people: A systematic review
Henderson, S. E. M., Brady, E. M., & Robertson, N. (2019) · Chronobiology International · doi:10.1080/07420528.2019.1636813 · Cited by 162
“A limited number of quality assessment tools are available for cross-sectional studies (Sanderson et al. 2007; Zeng et al. 2015).”
BackgroundManchikanti, L., Hirsch, J. A., & Smith, H. S. (2008) · Pain Physician · doi:10.36076/ppj.2008/11/717 · Cited by 161
Liu, A.-N., Wang, L.-L., Li, H.-P., Gong, J., & Liu, X.-H. (2017) · Journal of Nervous & Mental Disease · doi:10.1097/nmd.0000000000000605 · Cited by 160
“, 2007) combined with Sanderson's tools (Sanderson et al., 2007) in observational studies.”
Methodde Winter, C. F., Jansen, A. A. C., & Evenhuis, H. M. (2011) · Journal of Intellectual Disability Research · doi:10.1111/j.1365-2788.2011.01390.x · Cited by 128
“Assessment of the level of evidence At present, there is no consensus about a tool for the assessment of quality in observational studies (Sanderson et al. 2007).”
BackgroundA systematic review on the effect of work-related stressors on mental health of young workers
Law, P. C. F., Too, L. S., Butterworth, P., Witt, K., Reavley, N. J., & Milner, A. J. (2020) · International Archives of Occupational and Environmental Health · doi:10.1007/s00420-020-01516-7 · Cited by 161
Bloomfield, H. E., Koeller, E., Greer, N., MacDonald, R., Kane, R., & Wilt, T. J. (2016) · Annals of Internal Medicine · doi:10.7326/m16-0361 · Cited by 159
“Risk of bias for cohort studies was based on methods for selecting the population, identifying and measuring outcomes, length of follow-up, and controlling for confounding (11).”
MethodObesity in Pediatric ALL Survivors: A Meta-Analysis
Zhang, F. F., Kelly, M. J., Saltzman, E., Must, A., Roberts, S. B., & Parsons, S. K. (2014) · Pediatrics · doi:10.1542/peds.2013-3332 · Cited by 159
Pachêco-Pereira, C., Pereira, J. R., Dick, B. D., Perez, A., & Flores-Mir, C. (2015) · American Journal of Orthodontics and Dentofacial Orthopedics · doi:10.1016/j.ajodo.2015.04.039 · Cited by 159
Social support and glycemic control in type 2 diabetes: A systematic review of observational studies
Stopford, R., Winkley, K., & Ismail, K. (2013) · Patient Education and Counseling · doi:10.1016/j.pec.2013.08.016 · Cited by 159
“There is a lack of consensus regarding the optimal method of assessing quality in observational studies [21].”
Posadzki, P., Lee, M. S., Moon, T., Choi, T., Park, T. Y., & Ernst, E. (2013) · Maturitas · doi:10.1016/j.maturitas.2013.02.005 · Cited by 159
Thørrisen, M. M., Bonsaksen, T., Hashemi, N., Kjeken, I., van Mechelen, W., & Aas, R. W. (2019) · BMJ Open · doi:10.1136/bmjopen-2019-029184 · Cited by 128
“Low [1], [2], [3], [4], [5], [10], [12], [17], [19], [26], [39], [49], [51], [54], [55], [56], [58], [59], [60], [62], [64], [67], [69], [77], [78], [81], [82], [83], [84], [95], [96], [97], [118], [119], [120], [121], [124] and [125].”
Zia, J. K., Lenhart, A., Yang, P.-L., Heitkemper, M. M., Baker, J. R., Keefer, L., Saps, M., Cuff, C., Hungria, G., Videlock, E. J., et al. (2022) · Gastroenterology · doi:10.1053/j.gastro.2022.06.028 · Cited by 159
van Netten, J. J., Fortington, L. V., Hinchliffe, R. J., & Hijmans, J. M. (2015) · European Journal of Vascular and Endovascular Surgery · doi:10.1016/j.ejvs.2015.10.001 · Cited by 158
Systematic review: risk factors for musculoskeletal disorders in musicians
Baadjou, V., Roussel, N. A., Verbunt, J., Smeets, R., & de Bie, R. A. (2016) · Occupational Medicine · doi:10.1093/occmed/kqw052 · Cited by 159
Caini, S., Kroneman, M. W., Wiegers, T. A., Guerche‐Séblain, C. E., & Paget, J. (2018) · Influenza and Other Respiratory Viruses · doi:10.1111/irv.12575 · Cited by 158
“For observational studies, such as the studies included in our systematic review, several quality assessment tools or grids exist [13], many of which are however specifically”
BackgroundPyrethroid epidemiology: a quality-based review
Burns, C. J., & Pastoor, T. P. (2018) · Critical Reviews in Toxicology · doi:10.1080/10408444.2017.1423463 · Cited by 157
“There are multiple approaches to report and evaluate study quality and validity (Sanderson et al. 2007; Rhomberg et al. 2013).”
BackgroundDevelopmental coordination disorder and overweight and obesity in children: a systematic review
Hendrix, C. G., Prins, M. R., & Dekkers, H. (2014) · Obesity Reviews · doi:10.1111/obr.12137 · Cited by 156
“(40) identified a number of criteria, used in these tools, that represent the principal potential sources of bias and are generally considered to be important.”
BackgroundHope, H. F., Binkley, G. M., Fenton, S., Kitas, G. D., Verstappen, S. M. M., & Symmons, D. P. M. (2019) · PLOS ONE · doi:10.1371/journal.pone.0201196 · Cited by 114
“which state that observational studies should be evaluated on the use of appropriate methods to: 1) select participants, 2) measure exposure and outcome variables, 3) control for confounding, 4) reduce bias and 5) analyse data [29] (S2 Table).”
MethodPasma, A., van't Spijker, A., Hazes, J. M. W., Busschbach, J. J. V., & Luime, J. J. (2013) · Seminars in Arthritis and Rheumatism · doi:10.1016/j.semarthrit.2012.12.001 · Cited by 155
Qiu, D., Yu, Y., Li, R., Li, Y., & Xiao, S. (2019) · Sleep Medicine · doi:10.1016/j.sleep.2019.01.047 · Cited by 155
Methodological quality, guidance, and tools in scoping reviews: a scoping review protocol
Pollock, D., Tricco, A. C., Peters, M. D. J., Mclnerney, P. A., Khalil, H., Godfrey, C. M., Alexander, L. A., & Munn, Z. (2022) · JBI Evidence Synthesis · doi:10.11124/jbies-20-00570 · Cited by 154
Medication Adherence Among Pediatric Patients With Sickle Cell Disease: A Systematic Review
Walsh, K. E., Cutrona, S. L., Kavanagh, P. L., Crosby, L. E., Malone, C., Lobner, K. L., & Bundy, D. G. (2014) · PEDIATRICS · doi:10.1542/peds.2014-0177 · Cited by 153
Engeroff, T., Ingmann, T., & Banzer, W. E. (2018) · Sports Medicine · doi:10.1007/s40279-018-0920-6 · Cited by 154
ROCHE, A. M., Pidd, K., Fischer, J. A., Lee, N., Scarfe, A. J., & Kostadinov, V. (2016) · Safety and Health at Work · doi:10.1016/j.shaw.2016.04.005 · Cited by 152
“There is no standard tool for data extraction or for assessing study quality [78].”
BackgroundMethodPayette, M., Bélanger, C., Léveillé, V., & Grenier, S. (2016) · PLoS ONE · doi:10.1371/journal.pone.0152848 · Cited by 154
“The criteria used were inspired and adapted from Loney, Chambers, Benett, Roberts and Stratford and Sanderson, Tatt and Higgins’s criteria for evaluation of observational studies [30,31].”
MethodWilliams, G. M., Aardema, M. J., Acquavella, J., Berry, S. C., Brusick, D., Burns, M. M., de Camargo, J. L. V., Garabrant, D. H., Greim, H. A., Kier, L. D., et al. (2016) · Critical Reviews in Toxicology · doi:10.1080/10408444.2016.1214677 · Cited by 150
Incidence of penile cancer worldwide: systematic review and meta-analysis
Cardona, C. E. M., & García‐Perdomo, H. A. (2017) · Revista Panamericana de Salud Pública · doi:10.26633/rpsp.2017.117 · Cited by 151
“We two authors both independently reviewed the reporting quality of all the studies, based on the Strengthening the Reporting of Observational studies in Epidemiology (STROBE) checklist (10); the results from the evaluation tools of a systematic review assessing the quality and susceptibility of bias in observational…”
MethodSuicide in South Asia: a scoping review
Jordans, M. J., Kaufman, A., Brenman, N. F., Adhikari, R. P., Luitel, N. P., Tol, W. A., & Komproe, I. (2014) · BMC Psychiatry · doi:10.1186/s12888-014-0358-9 · Cited by 130
“Moreover, a review by Sanderson and colleagues [11] of instruments for appraising quality of observational studies recommends using tools that: include a small number of items on key domains; are as specific as possible with regard to aspects of quality that is evaluated; that are simple checklists rather than scales,…”
BackgroundTaioli, E., Lee, D., LESSER, M. L., & Flores, R. M. (2013) · European Journal of Cardio-Thoracic Surgery · doi:10.1093/ejcts/ezt051 · Cited by 151
“As previously observed, any meta-analysis of observational studies is affected by the same biases present in the original studies contained in it [56].”
ResultRothman, L. M., Buliung, R. N., Macarthur, C., To, T. M., & Howard, A. (2013) · Injury Prevention · doi:10.1136/injuryprev-2012-040701 · Cited by 148
Penoni, D. C., Fidalgo, T. K. S., Torres, S. R., Varela, V. M., Masterson, D., Leão, A. T. T., & Maia, L. C. (2017) · Journal of Dental Research · doi:10.1177/0022034516682017 · Cited by 126
“Assessing the quality of evidence from epidemiological studies requires tools that are designed and developed for this specific purpose (Sanderson et al. 2007).”
BackgroundAssociation between depression and resilience in older adults: a systematic review and meta-analysis
Wermelinger Ávila, M. P., Lucchetti, A. L. G., & Lucchetti, G. (2017) · International Journal of Geriatric Psychiatry · doi:10.1002/gps.4619 · Cited by 148
“Sanderson et al. (2007) identified 86 tools for assessing the quality and risk of bias of observational studies and highlighted the lack of a single tool for assessing the epidemiological quality of observational studies.”
Goemans, A., van Geel, M., van Beem, M., & Vedder, P. H. (2016) · Child Maltreatment · doi:10.1177/1077559516657637 · Cited by 149
Cognitive functioning in idiopathic generalised epilepsies: A systematic review and meta-analysis
Loughman, A., Bowden, S., & D'Souza, W. J. (2014) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2014.02.012 · Cited by 148
Grzybowski, A. E., Och, M., Kanclerz, P., Leffler, C. T., & Moraes, C. G. D. (2020) · Journal of Clinical Medicine · doi:10.3390/jcm9030761 · Cited by 147
“[27]; we evaluated the methods for selecting study participants, various study designs, and criteria for defining exposures and outcomes.”
MethodZhang, Y., Huang, L., Wang, D., Ren, P., Qi, H. Y., & Kang, D. (2021) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.12427 · Cited by 148
Barratt, H., Rojas-García, A., Clarke, K., Moore, A., Whittington, C., Stockton, S., Thomas, J., Pilling, S., & Raine, R. (2016) · PLOS ONE · doi:10.1371/journal.pone.0154449 · Cited by 117
“There is no consensus about the use of rating methods for the quality assessment of epidemiological studies, particularly those reporting cross-sectional observational data.[11] We therefore developed a quality assessment measure for the purposes of this review, which drew on the Newcastle-Ottawa Scale for assessing t…”
Method
Metadata
- Authors
- Sanderson, S., Tatt, I. D., & Higgins, J. P.
- Affiliations
- University of Cambridge (GB)
- F. Hoffmann-La Roche AG (CH)
- MRC Biostatistics Unit (GB)
- Corresponding author
- Sanderson, S.
- ORCID
- Higgins, J. P.
- Venue
- International Journal of Epidemiology
- Publisher
- Oxford University Press (OUP)
- Published
- 2007-04-30
- Volume
- 36
- Issue
- 3
- Pages
- 666-676
- ISSN
- 0300-5771 (ISSN-L)
- 1464-3685
- OpenAlex
- W2171251365
- Cited by
- 1,531
- References
- 73
- Citations by year
- 2026: 22 · 2025: 38 · 2024: 42 · 2023: 59 · 2022: 80 · 2021: 96 · 2020: 101 · 2019: 105 · 2018: 125 · 2017: 104 · 2016: 150 · 2015: 118 · 2014: 122 · 2013: 111 · 2012: 96
- FWCI
- 7.09
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- International Journal of Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- quality assessment tools, bias assessment, observational studies, critical appraisal, checklists, rating scales, STROBE, confounding adjustment, selection bias, measurement bias, conflicts of interest
- MeSH terms
- Humans, Research Design, Bias, Epidemiologic Studies, Evidence-Based Medicine, Observation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.