Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography
The DOI matches its Crossref record; 73 of the 73 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Assessing quality and susceptibility to bias is essential when interpreting primary research and conducting systematic reviews and meta-analyses. Tools for assessing quality in clinical trials are well-described but much less attention has been given to similar tools for observational epidemiological studies. METHODS: Tools were identified from a search of three electronic databases, bibliographies and an Internet search using Google. Two reviewers extracted data using a pre-piloted extraction form and strict inclusion criteria. Tool content was evaluated for domains potentially related to bias and was informed by the STROBE guidelines for reporting observational epidemiological studies. RESULTS: A total of 86 tools were reviewed, comprising 41 simple checklists, 12 checklists with additional summary judgements and 33 scales. The number of items ranged from 3 to 36 (mean 13.7). One-third of tools were designed for single use in a specific review and one-third for critical appraisal. Half of the tools provided development details, although most were proposed for future use in other contexts. Most tools included items for selection methods (92%), measurement of study variables (86%), design-specific sources of bias (86%), control of confounding (78%) and use of statistics (78%); only 4% addressed conflict of interest. The distribution and weighting of domains across tools was variable and inconsistent. CONCLUSION: A number of useful assessment tools have been identified by this report. Tools should be rigorously developed, evidence-based, valid, reliable and easy to use. There is a need to agree on critical elements for assessing susceptibility to bias in observational epidemiology and to develop appropriate evaluation tools.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Such instruments have been developed by other groups and were the subject of a recent systematic review [28].”
From The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: Guidelines for Reporting Observational Studies (2007) · cites this paper as [28]
In the citing paper
“It is not clear how weights should be assigned to each item of the scale [18].”
From Tools used to assess the quality of peer review reports: A methodological systematic review (2019) · cites this paper as [18]
In the citing paper
“The reviewers concluded that there is no ‘single obvious candidate tool for assessing quality of observational epidemiological studies’.”
From COSMOS-E: Guidance on conducting systematic reviews and meta-analyses of observational studies of etiology (2019) · cites this paper as [45]
In the citing paper
“A review identified more than 80 tools for assessing risk of bias in nonrandomised studies [45].”
From COSMOS-E: Guidance on conducting systematic reviews and meta-analyses of observational studies of etiology (2019) · cites this paper as [45]
In the citing paper
“The need for complete reporting of all relevant methodological aspects as a prerequisite to assessing the risk of bias of studies was highlighted in 10 articles, [16, 33, 35, 47, 50, 56, 86, 87, 90, 91] and was the focus of recommendations in a publication from the Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group [12].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [90]
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Meta-analysis of Observational Studies in Epidemiology<SUBTITLE>A Proposal for Reporting</SUBTITLE>
Stroup, D. F. (2000) · JAMA · doi:10.1001/jama.283.15.2008 · Cited by 20,623
Downs, S. H., & Black, N. (1998) · Journal of Epidemiology and Community Health · doi:10.1136/jech.52.6.377 · Cited by 8,252
Moher, D., Cook, D. J., Eastwood, S., Olkin, I., Rennie, D., & Stroup, D. F. (1999) · The Lancet · doi:10.1016/s0140-6736(99)04149-5 · Cited by 4,826
Whiting, P., Rutjes, A. W., Reitsma, J. B., Bossuyt, P. M., & Kleijnen, J. (2003) · BMC Medical Research Methodology · doi:10.1186/1471-2288-3-25 · Cited by 3,592
Assessing the quality of controlled clinical trials
Jüni, P., Altman, D. G., & Egger, M. (2001) · BMJ · doi:10.1136/bmj.323.7303.42 · Cited by 3,309 · cited 5×
“a (5) (5) (5) ( 5 ) N N LongneckerCoh”
InfluentialUsers' guides to the medical literature
Guyatt, G. H. (1993) · JAMA · doi:10.1001/jama.270.17.2096 · Cited by 2,970
Evaluating non-randomised intervention studies
Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D. (2003) · Health Technology Assessment · doi:10.3310/hta7270 · Cited by 2,907
Cognitive and Behavioral Outcomes of School-Aged Children Who Were Born Preterm: A Meta-analysis
Bhutta, A. T., Cleves, M. A., Casey, P. H., Cradock, M. M., & Anand, K. J. S. (2002) · JAMA · doi:10.1001/jama.288.6.728 · Cited by 2,392 · cited 1×
“Y N N 9 (22) 2 (17) 5 (15) 16 (19) Y Y N 15 (36) 6 ( 50 ) 7 (21) 28 (32)”
The Hazards of Scoring the Quality of Clinical Trials for Meta-analysis
Jüni, P. (1999) · JAMA · doi:10.1001/jama.282.11.1054 · Cited by 1,981
Low grade inflammation and coronary heart disease: prospective study and updated meta-analyses
Danesh, J. (2000) · BMJ · doi:10.1136/bmj.321.7255.199 · Cited by 1,692
Research works printed but not linked: 7
- Carneiro, 2002, Revista portuguesa de cardiologia : org o oficial da Sociedade Portuguesa de Cardiologia = Portuguese journal of cardiology : an official journal of the Portuguese Society of Cardiology, 21, 891
- 1981, Canadian Medical Association Journal, 124, 985
- 2002, SOUTH AFRICAN J CLIN NUTR, 15, 68
- 2000, Chronic diseases in Canada, 19, 170
- Ciliska, 1996, Canadian journal of public health. Revue canadienne de sant publique, 87, 193
- 1994, Canadian journal of public health. Revue canadienne de sant publique, 85, S8
- Spitzer, 1990, Clinical and investigative medicine. M decine clinique et experimentale, 13, 17
Cited by
Items 101–200 of 1,006
Hudelson, C. E., & Cluver, L. D. (2015) · AIDS Care · doi:10.1080/09540121.2015.1011073 · Cited by 322
Epidemiology of Spondyloarthritis
Stolwijk, C., Boonen, A. E. R. C. H., van Tubergen, A. M., & Reveille, J. D. (2012) · Rheumatic Disease Clinics of North America · doi:10.1016/j.rdc.2012.09.003 · Cited by 314
Martyr, A., Nelis, S. M., Quinn, C., Wu, Y.-T., Lamont, R. A., Henderson, C., Clarke, R., Hindle, J. V., Thom, J. M., Jones, I. R., et al. (2018) · Psychological Medicine · doi:10.1017/s0033291718000405 · Cited by 311
“Study quality was assessed by two independent reviewers using a checklist adapted from existing measures (DuRant, 1994; Downs & Black, 1998; Zaza et al. 2000; Sanderson et al. 2007); see online Supplementary Table S2.”
MethodAcute myocardial infarction and influenza: a meta-analysis of case–control studies
Barnes, M., Heywood, A. E., Mahimbo, A., Rahman, B., Newall, A. T., & Macintyre, C. R. (2015) · Heart · doi:10.1136/heartjnl-2015-307691 · Cited by 310
“The ORs from individual studies were pooled using the inverse-variance weighted random effects method.”
MethodDiabetes, Fasting Glucose, and the Risk of Glaucoma
Zhao, D., Cho, J., Kim, M. H., Friedman, D. S., & Guallar, E. (2015) · Ophthalmology · doi:10.1016/j.ophtha.2014.07.051 · Cited by 308
Mazzoni, A., Althabe, F., Liu, N., Bonotti, A., Gibbons, L., Sánchez, A., & Belizán, J. (2011) · BJOG: An International Journal of Obstetrics & Gynaecology · doi:10.1111/j.1471-0528.2010.02793.x · Cited by 306
Xu, B. Y., Yan, S., Low, L. L., Vasanwala, F. F., & Low, S. G. (2019) · BMC Musculoskeletal Disorders · doi:10.1186/s12891-019-2950-0 · Cited by 303
Compton, C. W. R., Heuer, C., Thomsen, P. T., Carpenter, T. E., Phyn, C. V. C., & McDougall, S. (2017) · Journal of Dairy Science · doi:10.3168/jds.2016-11302 · Cited by 303
“The appraisal tool used to assess each article for inclusion in the review was in the form of a checklist (Supplemental Table S1; https://doi.org/10.3168/jds.2016-11302) and was based on those of Sargeant and Del Rocio Amezcua (2005) and Sanderson et al. (2007).”
BackgroundMethodInfluentialObesity and dental caries in children: a systematic review and meta‐analysis
Hayden, C., Bowler, J. O., Chambers, S., Freeman, R., Humphris, G., Richards, D., & Cecil, J. E. (2013) · Community Dentistry and Oral Epidemiology · doi:10.1111/cdoe.12014 · Cited by 302
Mills, S., White, M., Brown, H., Wrieden, W., Kwasnicka, D., Halligan, J., Robalino, S., & Adams, J. (2017) · Appetite · doi:10.1016/j.appet.2016.12.022 · Cited by 300
Wolters, F. J., Segufa, R. A., Darweesh, S. K. L., Bos, D., Ikram, M. A., Sabayan, B., Hofman, A., & Sedaghat, S. (2018) · Alzheimer's & Dementia · doi:10.1016/j.jalz.2018.01.007 · Cited by 293
“We critically appraised all selected studies and formally assessed their quality by using a modification of the Newcastle-Ottawa-Scale [32], in line with prior recommendations for quality assessment of observational studies [33].”
MethodHakkarainen, K. M., Hedna, K., Petzold, M., & Hägg, S. (2012) · PLoS ONE · doi:10.1371/journal.pone.0033236 · Cited by 293
“the individual studies was applied, as no consensus on quality scoring of observational epidemiological studies exists [21].”
BackgroundHenry, B. M., Tomaszewski, K. A., Ramakrishnan, P. K., Roy, J., Vikse, J., Loukas, M., Tubbs, R. S., & Walocha, J. A. (2017) · Clinical Anatomy · doi:10.1002/ca.22799 · Cited by 288
“Considering the strengths and weaknesses of the tools available in anatomical research and observational studies (Sanderson et al., 2007; Smith et al., 2008; von Elm et al., 2008; Wilke et al., 2015), the AQUA Tool was designed to be applicable across different types of anatomical studies (gross,…”
BackgroundMethodInfluentialUses and misuses of the STROBE statement: bibliographic study
da Costa, B. R., Cevallos, M., Altman, D. G., Rutjes, A. W. S., & Egger, M. (2011) · BMJ Open · doi:10.1136/bmjopen-2010-000048 · Cited by 287
Alqenae, F. A., Steinke, D., & Keers, R. N. (2020) · Drug Safety · doi:10.1007/s40264-020-00918-3 · Cited by 287
Jackson, C. A., Clatworthy, J., Robinson, A., & Horne, R. (2010) · American Journal of Gastroenterology · doi:10.1038/ajg.2009.685 · Cited by 286
“Quality appraisal For the purposes of this review, “quality” was defined as the absence of susceptibility to bias (through selection of participants, measurement of variables, and control of confounding variables) (”
MethodPanagioti, M., Gooding, P. A., & Tarrier, N. (2012) · Comprehensive Psychiatry · doi:10.1016/j.comppsych.2012.02.009 · Cited by 280
“However, an increased probability for heterogeneity of results because of broad inclusion criteria should not be disregarded [78,79].”
BackgroundMurray, J., Farrington, D. P., & Eisner, M. P. (2009) · Journal of Experimental Criminology · doi:10.1007/s11292-008-9066-0 · Cited by 280
“Deeks et al. (2003) reviewed 193 quality scales and checklists that might be used to assess non-randomized intervention studies in medicine (see also Sanderson et al. 2007 ).”
BackgroundBetancourt, T. S., Borisova, I., Williams, T. P., Meyers‐Ohki, S. E., Rubin‐Smith, J. E., Annan, J., & Kohrt, B. A. (2013) · Journal of Child Psychology and Psychiatry · doi:10.1111/j.1469-7610.2012.02620.x · Cited by 279
English, C., Manns, P. J., Tucak, C., & Bernhardt, J. (2014) · Physical Therapy · doi:10.2522/ptj.20130175 · Cited by 278
Page, M. J., McKenzie, J. E., & Higgins, J. P. T. (2018) · BMJ Open · doi:10.1136/bmjopen-2017-019703 · Cited by 278
Self-citationdu Fossé, N. A., van der Hoorn, M.-L. P., van Lith, J. M. M., le Cessie, S., & Lashley, E. E. L. O. (2020) · Human Reproduction Update · doi:10.1093/humupd/dmaa010 · Cited by 276
Quality of life in adolescent and young adult cancer patients: a systematic review of the literature
Quinn, G., Goncalves, V., Sehovic, I., Bowman, M., & Reed, D. (2015) · Patient Related Outcome Measures · doi:10.2147/prom.s51658 · Cited by 276
“research.(26,27) Two reviewers assessed each paper for the fol-”
Social and academic functioning in adolescents with anxiety disorders: A systematic review
de Lijster, J. M., Dieleman, G. C., Utens, E. M. W. J., Dierckx, B., Wierenga, M., Verhulst, F. C., & Legerstee, J. S. (2018) · Journal of Affective Disorders · doi:10.1016/j.jad.2018.01.008 · Cited by 273
Albanese, E., Launer, L. J., Egger, M., Prince, M. J., Giannakopoulos, P., Wolters, F. J., & Egan, K. (2017) · Alzheimer's & Dementia: Diagnosis, Assessment & Disease Monitoring · doi:10.1016/j.dadm.2017.05.007 · Cited by 272
Oldenmenger, W. H., de Raaf, P. J., de Klerk, C., & van der Rijt, C. C. D. (2013) · Journal of Pain and Symptom Management · doi:10.1016/j.jpainsymman.2012.06.007 · Cited by 270
Probst, C., Roerecke, M., Behrendt, S., & Rehm, J. (2014) · International Journal of Epidemiology · doi:10.1093/ije/dyu043 · Cited by 270
Children’s Physical Activity and Depression: A Meta-analysis
Korczak, D. J., Madigan, S., & Colasanto, M. (2017) · Pediatrics · doi:10.1542/peds.2016-2266 · Cited by 269
Rao, G., Lopez-Jimenez, F., Boyd, J., D’Amico, F., Durant, N. H., Hlatky, M. A., Howard, G., Kirley, K., Masi, C., Powell-Wiley, T. M., et al. (2017) · Circulation · doi:10.1161/cir.0000000000000523 · Cited by 269
Depression and inflammation among children and adolescents: A meta-analysis
Colasanto, M., Madigan, S., & Korczak, D. J. (2020) · Journal of Affective Disorders · doi:10.1016/j.jad.2020.09.025 · Cited by 265
(2013) · doi:10.1016/j.jclinepi.2011.05.008 · Cited by 264
“[11] Sanderson S, Tatt ID, Higgins JP.”
Hill, J., Hoyt, J., van Eijk, A. M., D'Mello-Guyett, L., ter Kuile, F. O., Steketee, R., Smith, H., & Webster, J. (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001488 · Cited by 263
“analysis, and minimising bias through study design and analysis [23,24].”
A review of critical appraisal tools show they lack rigor: Alternative tool structure is proposed
Crowe, M., & Sheppard, L. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.02.008 · Cited by 261
Quintana, D. S. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.01549 · Cited by 260
“Over half of the 25 journals that Frontiers in Psychology | www.frontiersin.org 1 October 2015 | Volume 6 | Article 1549 FIGURE 1 | Meta-analysis in the psychological sciences.”
BackgroundInfluentialThe Newcastle–Ottawa Scale for Assessing the Quality of Studies in Systematic Reviews
Gualdi-Russo, E., & Zaccagni, L. (2026) · Publications · doi:10.3390/publications14010004 · Cited by 267
“The maximum attainable score in the NOS depends on the version used, ranging from 9 to 16 points, the latter applying to the extended version for cross-sectional studies (among others: Hillen et al., 2017; Sanderson et al., 2007; Nayebirad et al., 2023; Kien et al., 2022).”
Kröger, H., Pakpahan, E., & Hoffmann, R. (2015) · The European Journal of Public Health · doi:10.1093/eurpub/ckv111 · Cited by 259
Risk factors for pressure injuries among critical care patients: A systematic review
Alderden, J., Rondinelli, J., Pepper, G., Cummins, M., & Whitney, J. (2017) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2017.03.012 · Cited by 258
“The quality-appraisal tool developed for the current review (see Appendix B) includes the domains identi fi ed in Sanderson et al. (2007) review of quality appraisal among observational studies: methods for selecting participants, methods for measuring exposure and outcome variables, design-speci fi…”
BackgroundMethodWong, W. C., Cheung, C. S., & Hart, G. J. (2008) · Emerging Themes in Epidemiology · doi:10.1186/1742-7622-5-23 · Cited by 257
“However all these were aimed at authors of reports, not at those seeking to assess the validity of what they read [10].”
Gardner, T., Refshauge, K., Smith, L., McAuley, J., Hübscher, M., & Goodall, S. (2017) · Journal of Physiotherapy · doi:10.1016/j.jphys.2017.05.017 · Cited by 257
Milner, A., Page, A., & LaMontagne, A. D. (2014) · Psychological Medicine · doi:10.1017/s0033291713001621 · Cited by 255
“The quality of studies was assessed before inclusion in the meta-analysis based on published recommendations (Sanderson et al. 2007).”
MethodAfolalu, E. F., Ramlee, F., & Tang, N. K. Y. (2018) · Sleep Medicine Reviews · doi:10.1016/j.smrv.2017.08.001 · Cited by 255
The consequences of sickness presenteeism on health and wellbeing over time: A systematic review
Skagen, K., & Collins, A. M. (2016) · Social Science & Medicine · doi:10.1016/j.socscimed.2016.06.005 · Cited by 254
“We therefore assessed each paper for quality by identifying the strengths and limitations of the study design and methods that limit bias and increase internal validity (Crowe and Sheppard, 2011; Higgins and Green, 2011; Sanderson et al., 2007).”
MethodQuality assessment of prevalence studies: a systematic review
Migliavaca, C. B., Stein, C., Colpani, V., Munn, Z., & Falavigna, M. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.06.039 · Cited by 253
Edwards, J., Berube, M., Erlandson, K., Haug, S., Johnstone, H., Meagher, M., Sarkodee-Adoo, S., & Zwicker, J. G. (2011) · Journal of Developmental & Behavioral Pediatrics · doi:10.1097/dbp.0b013e31822a396a · Cited by 244
Meta-Analysis of APOE 4 Allele and Outcome after Traumatic Brain Injury
Zhou, W., Xu, D., Peng, X., Zhang, Q., Jia, J., & Crutcher, K. A. (2008) · Journal of Neurotrauma · doi:10.1089/neu.2007.0489 · Cited by 244
“Quality Assessment As quality scoring is controversial in meta-analyses of observational studies (Sanderson et al., 2007), we systematically assessed several key features of each study.”
Yank, V., Tuohy, C. V., Logan, A. C., Bravata, D. M., Staudenmayer, K., Eisenhut, R., Sundaram, V., McMahon, D., Olkin, I., McDonald, K. M., et al. (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-154-8-201104190-00004 · Cited by 240
Burden of severe neonatal jaundice: a systematic review and meta-analysis
Slusher, T. M., Zamora, T. G., Appiah, D., Stanke, J. U., Strand, M. A., Lee, B. W., Richardson, S. B., Keating, E. M., Siddappa, A. M., & Olusanya, B. O. (2017) · BMJ Paediatrics Open · doi:10.1136/bmjpo-2017-000105 · Cited by 239
Prevalence of Complementary Medicine Use in Pediatric Cancer: A Systematic Review
Bishop, F. L., Prescott, P., Chan, Y. K., Saville, J., von Elm, E., & Lewith, G. T. (2010) · Pediatrics · doi:10.1542/peds.2009-1775 · Cited by 235
Cao, B., Gupta, S., Wang, J., Hightow-Weidman, L. B., Muessig, K. E., Tang, W., Pan, S., Pendse, R., & Tucker, J. D. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.7997 · Cited by 232
“Two reviewers assessed the quality of the included studies using the checklist tool in Sanderson et al [24], and a third reviewer collated the results.”
MethodPredictors of the Onset of Cigarette Smoking
Wellman, R. J., Dugas, E. N., Dutczak, H., O’Loughlin, E. K., Datta, G. D., Lauzon, B., & O’Loughlin, J. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.04.003 · Cited by 230
Gladstone, J. R., & Cimpian, A. (2021) · International Journal of STEM Education · doi:10.1186/s40594-021-00315-x · Cited by 227
“…study quality has come under increasing scrutiny because of a general lack of agreement on which elements of a study speak to its quality (e.g., Sanderson et al., 2007) and—more fundamentally—on what is even meant by “quality” (e.g., Valentine & Cooper, 2005), as well as because of evidence…”
BackgroundFactors Influencing Unsafe Behaviors and Accidents on Construction Sites: A Review
Khosravi, Y., Asilian-Mahabadi, H., Hajizadeh, E., Hassanzadeh-Rangi, N., Bastani, H., & Behzadan, A. H. (2014) · International Journal of Occupational Safety and Ergonomics · doi:10.1080/10803548.2014.11077023 · Cited by 227
“We used three different checklists for rating selected studies based on whether they had a qualitative, qualitative, or mixed methodology [10, 11, 12, 13, 14].”
MethodVogel, M., Schmitz, R. P. H., Hagel, S., Pletz, M. W., Gagelmann, N., Scherag, A., Schlattmann, P., & Brunkhorst, F. M. (2016) · Journal of Infection · doi:10.1016/j.jinf.2015.09.037 · Cited by 227
“Selected studies were assessed for the quality evidence according to the guidelines of the Grading of Recommendations Assessment, Development and Evaluation (GRADE) working group (using the Cochrane GRADEpro 3.6 version software) 17,18 and summary of finding tables were generated by two authors.”
MethodInfluentialvon Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2008) · Revista Española de Salud Pública · doi:10.1590/s1135-57272008000300002 · Cited by 224
Hoppen, T. H., Priebe, S., Vetter, I., & Morina, N. (2021) · BMJ Global Health · doi:10.1136/bmjgh-2021-006303 · Cited by 223
Black, C., Sharma, P., Scotland, G., McCullough, K., McGurn, D., Robertson, L., Fluck, N., MacLeod, A., McNamee, P., Prescott, G., et al. (2010) · Health Technology Assessment · doi:10.3310/hta14210 · Cited by 222
Recommendations for assessing the risk of bias in systematic reviews of health-care interventions
Viswanathan, M., Patnode, C. D., Berkman, N. D., Bass, E. B., Chang, S., Hartling, L., Murad, M. H., Treadwell, J. R., & Kane, R. L. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.12.004 · Cited by 221
Social Media and HIV: A Systematic Review of Uses of Social Media in HIV Communication
Taggart, T., Grewe, M. E., Conserve, D. F., Gliwa, C., & Roman Isler, M. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4387 · Cited by 220
“Two members of the research team (TT, MG) conducted a quality assessment of the 35 included studies using a checklist tool for assessing quality in observational studies [40].”
Ferro, M. A., & Speechley, K. N. (2009) · Epilepsia · doi:10.1111/j.1528-1167.2009.02276.x · Cited by 220
“…making, the Quality Index has been shown to be valid and reliable for measuring the methodologic quality of epidemiologic and health research (The Standards of Reporting Trials Group, 1994, Downs & Black, 1998; Sanderson et al., 2007; Carville et al., 2008; Olivo et al., 2008; Wang et al., 2008).”
MethodA meta‐analysis of predictors of bullying and victimisation in adolescence
Kljakovic, M., & Hunt, C. J. (2016) · Journal of Adolescence · doi:10.1016/j.adolescence.2016.03.002 · Cited by 220
Beerens, H. C., Zwakhalen, S. M. G., Verbeek, H., Ruwaard, D., & Hamers, J. P. H. (2013) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2013.02.005 · Cited by 219
Masefield, S. C., Prady, S. L., Sheldon, T. A., Small, N., Jarvis, S., & Pickett, K. E. (2020) · Maternal and Child Health Journal · doi:10.1007/s10995-020-02896-5 · Cited by 217
Briggs, A. M., Smith, A. J., Straker, L. M., & Bragge, P. (2009) · BMC Musculoskeletal Disorders · doi:10.1186/1471-2474-10-77 · Cited by 215
Rupel, K., Ottaviani, G., Gobbo, M., Contardo, L., Tirelli, G., Vescovi, P., Di Lenarda, R., & Biasotto, M. (2014) · Oral Oncology · doi:10.1016/j.oraloncology.2014.08.016 · Cited by 214
“Since there is not a specific tool to be recommended among others [11], the Authors chose three parameters:”
MethodSTROBE-Initiative, F. D., von Elm, E., Altmann, D. G., Egger, M., Pocock, S. C., Gøtzsche, P. C., & Vandenbroucke, J. P. (2008) · Der Internist · doi:10.1007/s00108-008-2138-4 · Cited by 213
Westlake, S. L., Colebatch, A. N., Baird, J., Curzen, N., Kiely, P., Quinn, M., Choy, E., Ostor, A. J. K., & Edwards, C. J. (2011) · Rheumatology · doi:10.1093/rheumatology/keq316 · Cited by 213
Kennedy, C. E., Fonner, V. A., Armstrong, K. A., Denison, J. A., Yeh, P. T., O’Reilly, K. R., & Sweat, M. D. (2019) · Systematic Reviews · doi:10.1186/s13643-018-0925-0 · Cited by 212
Suicidal behaviour and psychosocial problems in veterinary surgeons: a systematic review
Platt, B., Hawton, K., Simkin, S., & Mellanby, R. J. (2012) · Social Psychiatry and Psychiatric Epidemiology · doi:10.1007/s00127-010-0328-6 · Cited by 212
“Guidelines for the conduct of systematic reviews highlight the importance of assessing study quality in order to maximize the reliability and validity of the review process [77, 87].”
BackgroundImpulsivity and self-harm in adolescence: a systematic review
Lockwood, J., Daley, D., Townsend, E., & Sayal, K. (2017) · European Child & Adolescent Psychiatry · doi:10.1007/s00787-016-0915-5 · Cited by 212
“Indicators of study quality A quality rating system was developed in line with criteria recommended for non-clinical study assessment [54].”
MethodThe Association of Blood Pressure and Primary Open-Angle Glaucoma: A Meta-analysis
Zhao, D., Cho, J., Kim, M. H., & Guallar, E. (2014) · American Journal of Ophthalmology · doi:10.1016/j.ajo.2014.05.029 · Cited by 212
Foley, L., Larkin, J., Lombard-Vance, R., Murphy, A. W., Hynes, L., Galvin, E., & Molloy, G. J. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-044987 · Cited by 208
The Effect of Giving Global Coronary Risk Information to Adults: A Systematic Review
Sheridan, S. L. (2010) · Archives of Internal Medicine · doi:10.1001/archinternmed.2009.516 · Cited by 208
Oosterwijk, A. M., Nieuwenhuis, M. K., van der Schans, C. P., & Mouton, L. J. (2018) · Physiotherapy Theory and Practice · doi:10.1080/09593985.2017.1422206 · Cited by 207
“To be able to include quality assessment of studies, the recommendations of Sanderson, Tatt, and Higgins (2007) were followed to assess the risk of bias (ROB) using a self-2 developed checklist covering the three most fundamental domains (i.e. participants, measurements of variables, and control of…”
MethodResultStress in fathers in the perinatal period: A systematic review
Philpott, L. F., Leahy-Warren, P., FitzGerald, S., & Savage, E. (2017) · Midwifery · doi:10.1016/j.midw.2017.09.016 · Cited by 206
“…observational studies (i.e. cross-sectional, longitudinal) were assessed for risk of bias using criteria based on guidelines for ‘STrengthening the Reporting of OBservational Epidemiological’ (STROBE) studies which Sanderson et al. (2007) reported as incorporating the key principal sources of bias.”
MethodOremus, M., Oremus, C., Hall, G. B. C., McKinnon, M. C., & ECT & Cognition Systematic Review Team (2012) · BMJ Open · doi:10.1136/bmjopen-2012-001368 · Cited by 206
Pega, F., Liu, S. Y., Walter, S., Pabayo, R., Saith, R., & Lhachimi, S. K. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011135.pub2 · Cited by 151
Moral foundations and political orientation: Systematic review and meta-analysis.
Kivikangas, J. M., Fernández-Castilla, B., Järvelä, S., Ravaja, N., & Lönnqvist, J.-E. (2021) · Psychological Bulletin · doi:10.1037/bul0000308 · Cited by 205
Oeseburg, B., Dijkstra, G. J., Groothoff, J. W., Reijneveld, S. A., & Jansen, D. E. M. C. (2011) · Intellectual and Developmental Disabilities · doi:10.1352/1934-9556-49.2.59 · Cited by 205
Mori, C., Temple, J. R., Browne, D., & Madigan, S. (2019) · JAMA Pediatrics · doi:10.1001/jamapediatrics.2019.1658 · Cited by 204
Hinkley, T., Brown, H., Carson, V., & Teychenne, M. (2018) · PLOS ONE · doi:10.1371/journal.pone.0193700 · Cited by 204
A Systematic Review of the Prevalence and Associations of Limited Health Literacy in CKD
Taylor, D. M., Fraser, S. D. S., Bradley, J. A., Bradley, C., Draper, H., Metcalfe, W., Oniscu, G. C., Tomson, C. R. V., Ravanan, R., Roderick, P. J., et al. (2017) · Clinical Journal of the American Society of Nephrology · doi:10.2215/cjn.12921216 · Cited by 204
Negro-Calduch, E., Azzopardi-Muscat, N., Krishnamurthy, R. S., & Novillo-Ortiz, D. (2021) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2021.104507 · Cited by 152
Epidemiology of ankle and foot overuse injuries in sports: A systematic review
Sobhani, S., Dekker, R., Postema, K., & Dijkstra, P. U. (2013) · Scandinavian Journal of Medicine & Science in Sports · doi:10.1111/j.1600-0838.2012.01509.x · Cited by 201
Shamliyan, T., Kane, R. L., & Dickinson, S. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.04.014 · Cited by 201
“Gold standards to evaluate internal and external validity of observational research do not exist [7].”
BackgroundMethodDahabreh, I. J. (2011) · JAMA · doi:10.1001/jama.2011.336 · Cited by 199
“To our knowledge, there are no standard criteria for the assessment of validity of case-crossover studies.(16) Instead, based on general epidemiological principles and the particular features of the case-crossover design,(10,15,17,18) we attempted to assess the validity of the eligible studies by reviewing the followi…”
Casetta, B., Videla, A. J., Bardach, A., Morello, P., Soto, N., Lee, K., Camacho, P. A., Hermoza Moquillaza, R. V., & Ciapponi, A. (2016) · Nicotine & Tobacco Research · doi:10.1093/ntr/ntw266 · Cited by 199
Luckett, T., Goldstein, D., Butow, P. N., Gebski, V., Aldridge, L. J., McGrane, J., Ng, W., & King, M. T. (2011) · The Lancet Oncology · doi:10.1016/s1470-2045(11)70212-1 · Cited by 199
What Predicts the Physical Activity Intention–Behavior Gap? A Systematic Review
Rhodes, R. E., Cox, A., & Sayar, R. (2022) · Annals of Behavioral Medicine · doi:10.1093/abm/kaab044 · Cited by 197
Childhood dietary patterns and later obesity: a review of the evidence
Ambrosini, G. L. (2014) · Proceedings of the Nutrition Society · doi:10.1017/s0029665113003765 · Cited by 197
“At present, there is no universally accepted tool for scoring the methodological quality of observational studies((18)).”
MethodKamper, S. J., Logan, G., Copsey, B., Thompson, J., Machado, G. C., Abdel-Shaheed, C., Williams, C. M., Maher, C. G., & Hall, A. M. (2020) · Pain · doi:10.1097/j.pain.0000000000001751 · Cited by 196
“We used items from key domains for assessing susceptibility to bias in observational studies as recommended in Sanderson et al.(46) This includes items in 4 domains: representativeness of the sample, potential misclassification of patients, potential misclassification of outcomes, and inconsistent data.”
MethodThe art and science of knowledge synthesis
Tricco, A. C., Tetzlaff, J., & Moher, D. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.11.007 · Cited by 195
“checklists, or components, and many tools are available for different study designs [54,55].”
BackgroundMorina, N., Stam, K., Pollet, T. V., & Priebe, S. (2018) · Journal of Affective Disorders · doi:10.1016/j.jad.2018.07.027 · Cited by 194
“, 2007) and based on previous literature (Sanderson et al., 2007; Wang et al., 2017).”
BackgroundProportion and Risk Indicators of Nonadherence to Statin Therapy: A Meta-analysis
Lemstra, M., Blackburn, D., Crawley, A., & Fung, R. (2012) · Canadian Journal of Cardiology · doi:10.1016/j.cjca.2012.05.007 · Cited by 194
Durand, H., Hayes, P., Morrissey, E. C., Newell, J., Casey, M., Murphy, A. W., & Molloy, G. J. (2017) · Journal of Hypertension · doi:10.1097/hjh.0000000000001502 · Cited by 194
“[11] in their review of quality and risk of bias assessment tools for observational studies.”
BackgroundPsychosocial stress during pregnancy and perinatal outcomes: a meta-analytic review
Littleton, H. L., Bye, K., Buck, K., & Amacker, A. (2010) · Journal of Psychosomatic Obstetrics & Gynecology · doi:10.3109/0167482x.2010.518776 · Cited by 150
“Unfortunately, there are no consensus guidelines for evaluating study quality in nonexperimental studies and thus there is no ‘gold standard’ quality rating tool available [15].”
BackgroundKnardahl, S., Johannessen, H. A., Sterud, T., Härmä, M., Rugulies, R., Seitsamo, J. T., & Borg, V. (2017) · BMC Public Health · doi:10.1186/s12889-017-4059-4 · Cited by 194
“Recommendations for reporting or evaluating observational studies (Sanderson et al, 2007; the STROBE statement) [35, 37] do not address evaluating psychometrics of variables [42].”
MethodThe global prevalence of sexual assault: A systematic review of international research since 2010.
Dworkin, E. R., Krahé, B., & Zinzow, H. M. (2021) · Psychology of Violence · doi:10.1037/vio0000374 · Cited by 193
Dietary patterns and the risk of depression in adults: a systematic review of observational studies
Rahe, C., Unrath, M., & Berger, K. (2014) · European Journal of Nutrition · doi:10.1007/s00394-014-0652-9 · Cited by 192
“The quality of a systematic review is depending on the quality of the included original studies [14, 15].”
BackgroundMethodElrod, M. G., & Hood, B. S. (2015) · Journal of Developmental & Behavioral Pediatrics · doi:10.1097/dbp.0000000000000140 · Cited by 191
Metadata
- Authors
- Sanderson, S., Tatt, I. D., & Higgins, J. P.
- Affiliations
- University of Cambridge (GB)
- F. Hoffmann-La Roche AG (CH)
- MRC Biostatistics Unit (GB)
- Corresponding author
- Sanderson, S.
- ORCID
- Higgins, J. P.
- Venue
- International Journal of Epidemiology
- Publisher
- Oxford University Press (OUP)
- Published
- 2007-04-30
- Volume
- 36
- Issue
- 3
- Pages
- 666-676
- ISSN
- 0300-5771 (ISSN-L)
- 1464-3685
- OpenAlex
- W2171251365
- Cited by
- 1,531
- References
- 73
- Citations by year
- 2026: 22 · 2025: 38 · 2024: 42 · 2023: 59 · 2022: 80 · 2021: 96 · 2020: 101 · 2019: 105 · 2018: 125 · 2017: 104 · 2016: 150 · 2015: 118 · 2014: 122 · 2013: 111 · 2012: 96
- FWCI
- 7.09
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- International Journal of Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- quality assessment tools, bias assessment, observational studies, critical appraisal, checklists, rating scales, STROBE, confounding adjustment, selection bias, measurement bias, conflicts of interest
- MeSH terms
- Humans, Research Design, Bias, Epidemiologic Studies, Evidence-Based Medicine, Observation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.