Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography
The DOI matches its Crossref record; 73 of the 73 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Assessing quality and susceptibility to bias is essential when interpreting primary research and conducting systematic reviews and meta-analyses. Tools for assessing quality in clinical trials are well-described but much less attention has been given to similar tools for observational epidemiological studies. METHODS: Tools were identified from a search of three electronic databases, bibliographies and an Internet search using Google. Two reviewers extracted data using a pre-piloted extraction form and strict inclusion criteria. Tool content was evaluated for domains potentially related to bias and was informed by the STROBE guidelines for reporting observational epidemiological studies. RESULTS: A total of 86 tools were reviewed, comprising 41 simple checklists, 12 checklists with additional summary judgements and 33 scales. The number of items ranged from 3 to 36 (mean 13.7). One-third of tools were designed for single use in a specific review and one-third for critical appraisal. Half of the tools provided development details, although most were proposed for future use in other contexts. Most tools included items for selection methods (92%), measurement of study variables (86%), design-specific sources of bias (86%), control of confounding (78%) and use of statistics (78%); only 4% addressed conflict of interest. The distribution and weighting of domains across tools was variable and inconsistent. CONCLUSION: A number of useful assessment tools have been identified by this report. Tools should be rigorously developed, evidence-based, valid, reliable and easy to use. There is a need to agree on critical elements for assessing susceptibility to bias in observational epidemiology and to develop appropriate evaluation tools.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Such instruments have been developed by other groups and were the subject of a recent systematic review [28].”
From The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: Guidelines for Reporting Observational Studies (2007) · cites this paper as [28]
In the citing paper
“It is not clear how weights should be assigned to each item of the scale [18].”
From Tools used to assess the quality of peer review reports: A methodological systematic review (2019) · cites this paper as [18]
In the citing paper
“The reviewers concluded that there is no ‘single obvious candidate tool for assessing quality of observational epidemiological studies’.”
From COSMOS-E: Guidance on conducting systematic reviews and meta-analyses of observational studies of etiology (2019) · cites this paper as [45]
In the citing paper
“A review identified more than 80 tools for assessing risk of bias in nonrandomised studies [45].”
From COSMOS-E: Guidance on conducting systematic reviews and meta-analyses of observational studies of etiology (2019) · cites this paper as [45]
In the citing paper
“The need for complete reporting of all relevant methodological aspects as a prerequisite to assessing the risk of bias of studies was highlighted in 10 articles, [16, 33, 35, 47, 50, 56, 86, 87, 90, 91] and was the focus of recommendations in a publication from the Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group [12].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [90]
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Meta-analysis of Observational Studies in Epidemiology<SUBTITLE>A Proposal for Reporting</SUBTITLE>
Stroup, D. F. (2000) · JAMA · doi:10.1001/jama.283.15.2008 · Cited by 20,623
Downs, S. H., & Black, N. (1998) · Journal of Epidemiology and Community Health · doi:10.1136/jech.52.6.377 · Cited by 8,252
Moher, D., Cook, D. J., Eastwood, S., Olkin, I., Rennie, D., & Stroup, D. F. (1999) · The Lancet · doi:10.1016/s0140-6736(99)04149-5 · Cited by 4,826
Whiting, P., Rutjes, A. W., Reitsma, J. B., Bossuyt, P. M., & Kleijnen, J. (2003) · BMC Medical Research Methodology · doi:10.1186/1471-2288-3-25 · Cited by 3,592
Assessing the quality of controlled clinical trials
Jüni, P., Altman, D. G., & Egger, M. (2001) · BMJ · doi:10.1136/bmj.323.7303.42 · Cited by 3,309 · cited 5×
“a (5) (5) (5) ( 5 ) N N LongneckerCoh”
InfluentialUsers' guides to the medical literature
Guyatt, G. H. (1993) · JAMA · doi:10.1001/jama.270.17.2096 · Cited by 2,970
Evaluating non-randomised intervention studies
Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D. (2003) · Health Technology Assessment · doi:10.3310/hta7270 · Cited by 2,907
Cognitive and Behavioral Outcomes of School-Aged Children Who Were Born Preterm: A Meta-analysis
Bhutta, A. T., Cleves, M. A., Casey, P. H., Cradock, M. M., & Anand, K. J. S. (2002) · JAMA · doi:10.1001/jama.288.6.728 · Cited by 2,392 · cited 1×
“Y N N 9 (22) 2 (17) 5 (15) 16 (19) Y Y N 15 (36) 6 ( 50 ) 7 (21) 28 (32)”
The Hazards of Scoring the Quality of Clinical Trials for Meta-analysis
Jüni, P. (1999) · JAMA · doi:10.1001/jama.282.11.1054 · Cited by 1,981
Low grade inflammation and coronary heart disease: prospective study and updated meta-analyses
Danesh, J. (2000) · BMJ · doi:10.1136/bmj.321.7255.199 · Cited by 1,692
Research works printed but not linked: 7
- Carneiro, 2002, Revista portuguesa de cardiologia : org o oficial da Sociedade Portuguesa de Cardiologia = Portuguese journal of cardiology : an official journal of the Portuguese Society of Cardiology, 21, 891
- 1981, Canadian Medical Association Journal, 124, 985
- 2002, SOUTH AFRICAN J CLIN NUTR, 15, 68
- 2000, Chronic diseases in Canada, 19, 170
- Ciliska, 1996, Canadian journal of public health. Revue canadienne de sant publique, 87, 193
- 1994, Canadian journal of public health. Revue canadienne de sant publique, 85, S8
- Spitzer, 1990, Clinical and investigative medicine. M decine clinique et experimentale, 13, 17
Cited by
Items 401–500 of 1,006
Influence of orthodontic forces on human dental pulp: A systematic review
Javed, F., Kheraif, A. A. A., Romanos, E. B., & Romanos, G. E. (2014) · Archives of Oral Biology · doi:10.1016/j.archoralbio.2014.11.011 · Cited by 107
Incidence and Relapse of Idiopathic Nephrotic Syndrome: Meta-analysis
Veltkamp, F., Rensma, L. R., & Bouts, A. H. M. (2021) · PEDIATRICS · doi:10.1542/peds.2020-029249 · Cited by 108
Is Weight-Bearing Asymmetry Associated with Postural Instability after Stroke? A Systematic Review
Kamphuis, J. F., de Kam, D., Geurts, A. C. H., & Weerdesteyn, V. (2013) · Stroke Research and Treatment · doi:10.1155/2013/692137 · Cited by 106
Niedzwiedz, C. L., Katikireddi, S. V., Pell, J. P., & Mitchell, R. (2012) · BMC Public Health · doi:10.1186/1471-2458-12-628 · Cited by 107
“Given the limitations associated with scoring criteria [30], the quality of articles was initially considered dur-”
BackgroundDas Strengthening the Reporting of Observational Studies in Epidemiology (STROBE-) Statement
von Elm, E. B., Altman, D. G., Egger, M., Pocock, S., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. D. S. (2008) · Notfall + Rettungsmedizin · doi:10.1007/s10049-008-1057-1 · Cited by 106
Pons, T., Shipton, E. A., Williman, J. A., & Mulder, R. (2015) · Anesthesiology Research and Practice · doi:10.1155/2015/956539 · Cited by 105
“Rating criteria for quality and relevance and weighted strength against bias were based on published recommendations [7, 62, 63].”
MethodAggarwal, A. K., Lewis, D. J., Mason, M. D., Sullivan, R. J., & van der Meulen, J. (2016) · Medical Care Research and Review · doi:10.1177/1077558716654631 · Cited by 105
“A review of published checklists and scoring scales for systematic reviews was undertaken to identify the appropriate tool to assess the selected studies (Higgins & Green, 2008; Sanderson, Tatt, & Higgins, 2007); however, no suitable tool was found.”
MethodKnowledge, attitudes and practice of physicians toward evidence‐based medicine: A systematic review
Barzkar, F., Baradaran, H. R., & Koohpayehzadeh, J. (2018) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.12325 · Cited by 105
Puntis, S., Rugkåsa, J., Forrest, A., Mitchell, A., & Burns, T. (2014) · Psychiatric Services · doi:10.1176/appi.ps.201400178 · Cited by 105
“To select a quality assessment instrument for this systematic review, the NOS was compared against selected checklists described in Sanderson and colleagues’ (19) review of quality assessment instruments.”
BackgroundMethodSoundy, A., Wampers, M., Probst, M., De Hert, M., Stubbs, B., Vancampfort, D., Attux, C., Leutwyler, H., & Ströhle, A. (2013) · International Journal of Therapy and Rehabilitation · doi:10.12968/ijtr.2013.20.12.588 · Cited by 92
Shoamanesh, A., Kwok, C. S., Lim, P. A., & Benavente, O. (2012) · International Journal of Stroke · doi:10.1111/j.1747-4949.2012.00869.x · Cited by 104
Kelly, P., Hegarty, J. M., Barry, J. M., Dyer, K. R., & Horgan, A. M. (2017) · Journal of Substance Abuse Treatment · doi:10.1016/j.jsat.2017.06.001 · Cited by 104
Anterior cranial-base time-related changes: A systematic review
Afrand, M., Ling, C. P., Khosrotehrani, S., Flores‐Mir, C., & Lagravère, M. O. (2014) · American Journal of Orthodontics and Dentofacial Orthopedics · doi:10.1016/j.ajodo.2014.03.019 · Cited by 104
Gross Motor Deficits in Children Prenatally Exposed to Alcohol: A Meta-analysis
Lucas, B. R., Latimer, J., Pinto, R. Z., Ferreira, M. L., Doney, R., Lau, M., Jones, T., Dries, D., & Elliott, E. J. (2014) · PEDIATRICS · doi:10.1542/peds.2013-3733 · Cited by 104
Lodge, C. J., Allen, K. J., Lowe, A. J., Hill, D. J., Hosking, C. S., Abramson, M. J., & Dharmage, S. C. (2012) · Clinical and Developmental Immunology · doi:10.1155/2012/176484 · Cited by 103
“When assessing the quality of observational studies, particularly for possible sources of bias, the importance of addressing the following essential areas has been highlighted [29]: appropriate selection of participants, appropriate measurement of variables, and appropriate control of confounding.”
BackgroundLancaster, H. S., Lien, K., Chow, J. C., Frey, J. R., Scherer, N. J., & Kaiser, A. P. (2019) · Journal of Speech Language and Hearing Research · doi:10.1044/2019_jslhr-19-00162 · Cited by 104
“Researchers are still working toward a conceptually similar study quality checklist for descriptive and observational studies (Page, Mckenzie, & Higgins, 2018; Sanderson, Tatt, & Higgins, 2007).”
Parental bonding and eating disorders: A systematic review
Tetley, A., Moghaddam, N. G., Dawson, D. L., & Rennoldson, M. (2013) · Eating Behaviors · doi:10.1016/j.eatbeh.2013.10.008 · Cited by 105
Leijendekkers, R. A., van Hinte, G., Frölke, J. P. M., van de Meent, H., der Sanden, M. W. G. N., & Staal, J. B. (2016) · Disability and Rehabilitation · doi:10.1080/09638288.2016.1186752 · Cited by 103
“[58] found that there was no obvious candidate tool for assessing the quality of 1054 R.”
MethodOkoye, J. O., Ofodile, C. A., Adeleke, O. K., & Obioma, O. (2021) · Epidemiology and Health · doi:10.4178/epih.e2021039 · Cited by 102
“Three authors assessed the risk of bias (as good, unclear, or poor) in non-randomized observational studies across 5 criteria: study population, imprecision, inconsistency, bias in study design, and disclosure of conflict of interest [27,28].”
BackgroundPlagiocephaly and Developmental Delay: A Systematic Review
Martiniuk, A. L. C., Vujovich‐Dunn, C., Park, M., Yu, W., & Lucas, B. R. (2016) · Journal of Developmental & Behavioral Pediatrics · doi:10.1097/dbp.0000000000000376 · Cited by 101
Simunovic, N., Sprague, S., & Bhandari, M. (2009) · Journal of Bone and Joint Surgery · doi:10.2106/jbjs.h.01576 · Cited by 100
Archer, S., Hull, L., Soukup, T., Mayer, E., Athanasiou, T., Sevdalis, N., & Darzi, A. (2017) · BMJ Open · doi:10.1136/bmjopen-2017-017155 · Cited by 100
“Third, it has been argued that it is important, but difficult, to distinguish between ‘quality of reporting’ and the ‘quality of a study’.(20) As such, articles were not excluded from the current review based on ‘quality’ nor was weight assigned to studies based on quality.”
BackgroundPilotto, A., Romagnolo, A., Tuazon, J. A., Vizcarra, J. A., Marsili, L., Zibetti, M., Rosso, M., Rodriguez‐Porcel, F. E., Borroni, B., Rizzetti, M. C., et al. (2019) · Journal of Neurology Neurosurgery & Psychiatry · doi:10.1136/jnnp-2019-320846 · Cited by 100
Guyatt, G., Wang, Y., Eachempati, P., Iorio, A., Murad, M., Hultcrantz, M., Chu, D. K., Flórez, I., Hemkens, L., Agoritsas, T., et al. (2025) · British medical journal · doi:10.1136/bmj-2024-083864 · Cited by 75
Li, L., Wong, H. M., Peng, S., & McGrath, C. P. J. (2015) · Advances in Nutrition · doi:10.3945/an.114.006395 · Cited by 100
Stapersma, L., van den Brink, G., Szigethy, E. M., Escher, J. C., & Utens, E. M. W. J. (2018) · Alimentary Pharmacology & Therapeutics · doi:10.1111/apt.14865 · Cited by 91
Murgatroyd, D. F., Casey, P. P., Cameron, I. D., & Harris, I. A. (2015) · PLOS ONE · doi:10.1371/journal.pone.0117597 · Cited by 91
The prevalence of onychomycosis in psoriatic patients: a systematic review
Klaassen, K. M. G., Dulak, M. G., van de Kerkhof, P. C. M., & Pasch, M. C. (2014) · Journal of the European Academy of Dermatology and Venereology · doi:10.1111/jdv.12239 · Cited by 90
Association of iron indices and type 2 diabetes: a meta‐analysis of observational studies
Orban, E., Schwab, S., Thorand, B., & Huth, C. (2013) · Diabetes/Metabolism Research and Reviews · doi:10.1002/dmrr.2506 · Cited by 98
“Currently, no gold standard for assessing risk of bias in observational studies exists, and there is a need for valid and reliable tools [50].”
BackgroundTherapeutic Dancing for Parkinson's Disease
Aguiar, L. P. C., Rocha, P., & Morris, M. E. (2016) · International journal of gerontology · doi:10.1016/j.ijge.2016.02.002 · Cited by 99
The Effect of Vitamins on Glaucoma: A Systematic Review and Meta-Analysis
Ramdas, W. D., Schouten, J., & Webers, C. A. B. (2018) · Nutrients · doi:10.3390/nu10030359 · Cited by 98
“The studies were described according to a method described elsewhere [11].”
MethodLambert, K., Coe, J., Niel, L., Dewey, C., & Sargeant, J. M. (2015) · Preventive Veterinary Medicine · doi:10.1016/j.prevetmed.2014.11.002 · Cited by 97
Does gender role explain a high risk of depression? A meta-analytic review of 40 years of evidence
Lin, J., Zou, L., Lin, W., Becker, B., Yeung, A., Cuijpers, P. J., & Li, H. (2021) · Journal of Affective Disorders · doi:10.1016/j.jad.2021.07.018 · Cited by 97
“…the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) checklist (Von Elm et al., 2014), and literature suggestions (Sanderson et al., 2007), in which four domains in each study were considered: (1) Appropriate design of the study (i.e. cross-sectional with/without…”
Methodde Gentile, A. S., Bardach, A., Ciapponi, A., Martí, S. G., Aruj, P. K., Glujovsky, D., Calcagno, J. I., Mazzoni, A., & Colindres, R. E. (2011) · International Journal of Infectious Diseases · doi:10.1016/j.ijid.2011.09.013 · Cited by 97
Palliative pelvic radiotherapy of symptomatic incurable rectal cancer – a systematic review
Cameron, M. G., Kersten, C., Vistad, I., Fosså, S. D., & Guren, M. G. (2013) · Acta Oncologica · doi:10.3109/0284186x.2013.837582 · Cited by 96
Systematic review of the link between tobacco and poverty.
Ciapponi, A. (2011) · Cited by 96
Tam, P. I., Bernstein, E., Ma, X., & Ferrieri, P. (2015) · Hospital Pediatrics · doi:10.1542/hpeds.2014-0138 · Cited by 96
Incidence of Rhegmatogenous Retinal Detachment in Europe – A Systematic Review and Meta-Analysis
Li, J. Q., Welchowski, T., Schmid, M. C., Holz, F. G., & Finger, R. P. (2019) · Ophthalmologica · doi:10.1159/000499489 · Cited by 96
Prevalence of depressive symptoms among nurses in China: A systematic review and meta-analysis
Xie, N., Qin, Y., Wang, T., Zeng, Y., Deng, X., & Guan, L. (2020) · PLoS ONE · doi:10.1371/journal.pone.0235448 · Cited by 95
Linhares, A. C., Stupka, J. A., Ciapponi, A., Bardach, A., Glujovsky, D., Aruj, P. K., Mazzoni, A., Buendía, J. A., Rearte, A., Lanzieri, T. M., et al. (2011) · Reviews in Medical Virology · doi:10.1002/rmv.682 · Cited by 95
“[27], and Fowkes et al.”
Bardach, A., García‐Perdomo, H. A., Alcaraz, A. O., Tapia, E., Gándara, R. A. R., Ruvinsky, S., & Ciapponi, A. (2019) · Tropical Medicine & International Health · doi:10.1111/tmi.13217 · Cited by 94
Li, L., Asemota, I., Liu, B., Gómez, J., Lin, L., Arif, A. W., Siddiqi, T. J., & Usman, M. (2022) · Systematic Reviews · doi:10.1186/s13643-022-02029-9 · Cited by 95
“Choosing the right RoB tool for RCTs and NRSIs remains a big challenge for authors of SRs because of the large number of tools available [47].”
MethodSinha, S., Peach, G., Poloniecki, J., THOMPSON, M. M., & Holt, P. (2012) · European Journal of Public Health · doi:10.1093/eurpub/cks046 · Cited by 94
de Raaf, P. J., de Klerk, C., & van der Rijt, C. C. D. (2013) · Psycho-Oncology · doi:10.1002/pon.3225 · Cited by 82
International travel and acquisition of multidrug-resistant Enterobacteriaceae: a systematic review
Hassing, R. J., Alsma, J., Arcilla, M. S., van Genderen, P. J. J., Stricker, B. H. C., & Verbon, A. (2015) · Eurosurveillance · doi:10.2807/1560-7917.es.2015.20.47.30074 · Cited by 94
Hart, L. M., Cornell, C., Damiano, S. R., & Paxton, S. J. (2015) · International Journal of Eating Disorders · doi:10.1002/eat.22284 · Cited by 93
“The original 11-item CASP tool was developed to critically appraise the quality of cohort studies and involves answering a series of questions about methodological quality.(32) The shortened, adapted version has previously been used in the eating disorder literature.”
MethodProbst, C., Roerecke, M., Behrendt, S., & Rehm, J. (2015) · Drug and Alcohol Review · doi:10.1111/dar.12184 · Cited by 81
Alcohol use among workers in male-dominated industries: A systematic review of risk factors
ROCHE, A. M., Lee, N., Battams, S., Fischer, J. A., Cameron, J., & McEntee, A. (2015) · Safety Science · doi:10.1016/j.ssci.2015.04.007 · Cited by 93
Negahban, H., Mazaheri, M., Kingma, I., & van Dieën, J. H. (2013) · Knee Surgery Sports Traumatology Arthroscopy · doi:10.1007/s00167-013-2501-4 · Cited by 92
“Different instruments have been used in the literature for quantifying methodological quality of observational studies [27, 39].”
MethodRare Diseases Epidemiology Research
de la Paz, M. P., Villaverde‐Hueso, A., Alonso‐Ferreira, V., Sándor, J., Zurriaga, Ó., Pollán, M., & Abaitua, I. (2010) · Advances in experimental medicine and biology · doi:10.1007/978-90-481-9485-8_2 · Cited by 92
“Hence, the importance of using good methods, quality data and bias control [54] (some discussion about type of bias is included in the chapter 6).”
MethodBurden of reduced work productivity among people with chronic knee pain: a systematic review
Agaliotis, M., Mackey, M., Jan, S., & Fransen, M. (2014) · Occupational and Environmental Medicine · doi:10.1136/oemed-2013-101997 · Cited by 92
Uncovering survivorship bias in longitudinal mental health surveys during the COVID-19 pandemic
Czeisler, M. É., Wiley, J. F., Czeisler, C. A., Rajaratnam, S. M. W., & Howard, M. E. (2021) · Epidemiology and Psychiatric Sciences · doi:10.1017/s204579602100038x · Cited by 90
“Survivorship bias assessment should therefore be among bias assessments (Sanderson et al., 2007; Mayeda et al., 2016; Griffith et al., 2020; Czeisler et al., 2021d) applied beforeconclusionsbasedonrepeatedassessments fromparticipants in a longitudinal study are generalised, and decisions regarding the allocation of me…”
BackgroundCook, J., Knight, E., Hume, I., & Qureshi, A. (2014) · ADHD Attention Deficit and Hyperactivity Disorders · doi:10.1007/s12402-014-0133-2 · Cited by 90
“Development of the quality assessment tool Sanderson et al. (2007) conducted a systematic review of quality assessment tools and provided recommendations regarding their development in observational or cohort studies.”
MethodInfluentialSport Injuries Sustained by Athletes with Disability: A Systematic Review
Weiler, R., van Mechelen, W., Fuller, C. W., & Verhagen, E. A. L. M. (2016) · Sports Medicine · doi:10.1007/s40279-016-0478-0 · Cited by 90
“Tools for assessing study quality are great in number, yet lack agreement on critical elements and validity for use with sports injury studies assessing injury rates [12, 13].”
BackgroundRedmon, J. H., Elledge, M. F., Womack, D. S., Wickremashinghe, R., Wanigasuriya, K. P., Peiris-John, R. J., Lunyera, J., Smith, K., Raymer, J. H., & Levine, K. E. (2014) · BMC Nephrology · doi:10.1186/1471-2369-15-125 · Cited by 90
Stone, J., Glass, K., Clark, J. M., Ritskes-Hoitinga, M. J., Munn, Z., Tugwell, P., & Doi, S. A. R. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.01.012 · Cited by 90
Kim, K., Ozegovic, D., & Voaklander, D. C. (2012) · Injury Prevention · doi:10.1136/injuryprev-2011-040306 · Cited by 89
Broekstra, D. C., Groen, H., Molenkamp, S., Werker, P. M. N., & van den Heuvel, E. R. (2018) · Plastic & Reconstructive Surgery · doi:10.1097/prs.0000000000004120 · Cited by 89
Prevalence of drug interactions in hospitalised elderly patients: a systematic review
de Oliveira, L. M., Diel, J. D. A. C., Nunes, A., & Pizzol, T. D. S. D. (2020) · European Journal of Hospital Pharmacy · doi:10.1136/ejhpharm-2019-002111 · Cited by 89
A Systematic Review of Topical Treatments to Control the Odor of Malignant Fungating Wounds
Santos, C. M. D. C., Pimenta, C. A. D. M., & Nobre, M. R. C. (2010) · Journal of Pain and Symptom Management · doi:10.1016/j.jpainsymman.2009.11.319 · Cited by 88
Bones of Contention: Bone Mineral Density Recovery in Celiac Disease—A Systematic Review
Grace-Farfaglia, P. (2015) · Nutrients · doi:10.3390/nu7053347 · Cited by 77
“Methodological quality was assessed using the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology Statement) recommendations using separate checklists for conference abstracts, case control studies, cohort studies, and cross-sectional studies [17].”
MethodDrug-Related Problems of Patients in Primary Health Care Institutions: A Systematic Review
Ni, X., Yang, C., Bai, Y.-M., Hu, Z.-X., & Zhang, L. (2021) · Frontiers in Pharmacology · doi:10.3389/fphar.2021.698907 · Cited by 87
“Quality Assessment Currently, there were no more accepted quality assessment instruments for cross-sectional studies (Sanderson et al., 2007).”
BackgroundGrowth patterns during and after treatment in patients with pediatric ALL: A meta‐analysis
Zhang, F. F., Liu, S., Chung, M., & KELLY, M. J. (2015) · Pediatric Blood & Cancer · doi:10.1002/pbc.25519 · Cited by 87
“There is no standard scale to assess quality in observational studies [13,14].”
BackgroundBeauregard, N., Marchand, A., & Blanc, M.-E. (2011) · BMC Public Health · doi:10.1186/1471-2458-11-439 · Cited by 56
“We partially addressed this limitation in our appraisal with the NOS instrument [ 139 ].”
MethodObserving the Counterfactual? The Search for Political Experiments in Nature
Robinson, G., McNulty, J. E., & Krasno, J. S. (2009) · Political Analysis · doi:10.1093/pan/mpp011 · Cited by 87
Early motor developmental milestones and schizophrenia: A systematic review and meta-analysis
Filatova, S., Koivumaa-Honkanen, H., Hirvonen, N., Freeman, A., Ivandic, I., Hurtig, T., Khandaker, G., Jones, P., Moilanen, K., & Miettunen, J. (2017) · Schizophrenia Research · doi:10.1016/j.schres.2017.01.029 · Cited by 71
“The scale has been shown to be a reliable instrument according to a recent systematic review (Sanderson et al. 2007).”
Hinkley, T., Timperio, A. F., Watson, A., Duckham, R. L., Okely, A. D., Cliff, D. P., Carver, A., & Hesketh, K. D. (2020) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-020-00935-6 · Cited by 86
Li, L., Wen, J., Wang, L., Li, Y., & Li, Y. (2010) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/j.1471-0528.2010.02802.x · Cited by 86
Reporting guidance for violence risk assessment predictive validity studies: The RAGEE Statement.
Singh, J. P., The RAGEE Group, Yang, S. M., & Mulvey, E. P. (2014) · Law and Human Behavior · doi:10.1037/lhb0000090 · Cited by 86
Chacko, L., Ford, N. P., Sbaiti, M., & Siddiqui, R. (2012) · Sexually Transmitted Infections · doi:10.1136/sextrans-2011-050371 · Cited by 86
“Quality and validity There is no agreed tool for the assessment of methodological quality for observational studies.(17) We assessed the following factors as determinants of study quality: proportion of refusals, method of adherence measure, reporting of side effects, specification of inclusion and exclusion criteria,…”
MethodFooter, K. H., Silberzahn, B. E., Tormohlen, K. N., & Sherman, S. G. (2016) · Journal of the International AIDS Society · doi:10.7448/ias.19.4.20883 · Cited by 81
Reddy, R. R., Reddy, S. G., Vaidhyanathan, A., Bergé, S. J., & Kuijpers‐Jagtman, A. M. (2017) · Journal of Cranio-Maxillofacial Surgery · doi:10.1016/j.jcms.2017.03.006 · Cited by 86
Omer, S., Jijon, A. M., & Leonard, H. C. (2018) · Journal of Child Psychology and Psychiatry · doi:10.1111/jcpp.13001 · Cited by 86
“The NOS satisfies relevant guidelines (Sanderson et al., 2007) and is recommended for systematic reviews of observational studies (Deeks et al.”
BackgroundVoss, P. H., & Rehfuess, E. A. (2013) · Journal of Epidemiology and Community Health · doi:10.1136/jech-2011-200940 · Cited by 72
“Most importantly, there is much variation in the full or partial coverage of key study quality domains as defined by Sanderson et al.7 Confounding is the only key domain addressed by all QATs; conflict of interest is not considered by any tool.”
BackgroundMedications & Polypharmacy Influence on Recurrent Fallers in Community: a Systematic Review
Ming, Y., & Zecevic, A. (2018) · Canadian Geriatrics Journal · doi:10.5770/cgj.21.268 · Cited by 85
“However, assessment of quality and susceptibility to bias for observational studies remains a methodological problem.(18) Hence, we used Strengthening the Reporting of Observational Studies in Epidemiology (STROBE)(19) checklist, to capture the key study design features.”
MethodChaw, P. S., Höpner, J., & Mikolajczyk, R. (2018) · Journal of Clinical Pharmacy and Therapeutics · doi:10.1111/jcpt.12730 · Cited by 85
Doenças infecciosas em crianças pré-escolares brasileiras assistidas em creches
Pedraza, D. F., de Queiroz, D., & Sales, M. C. (2014) · Ciência & Saúde Coletiva · doi:10.1590/1413-81232014192.09592012 · Cited by 84
Cafferata, M. L., Bardach, A., Rey‐Ares, L., Alcaraz, A. O., Cormick, G., Gibbons, L., Romano, M., Cesaroni, S., & Ruvinsky, S. (2013) · Value in Health Regional Issues · doi:10.1016/j.vhri.2013.10.002 · Cited by 84
Psychosocial predictors of depression and anxiety in patients with epilepsy: A systematic review
Gandy, M., Sharpe, L., & Perry, K. N. (2011) · Journal of Affective Disorders · doi:10.1016/j.jad.2011.11.039 · Cited by 84
Childhood socioeconomic position and adult leisure-time physical activity: a systematic review
Elhakeem, A., Cooper, R., Bann, D., & Hardy, R. (2015) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-015-0250-0 · Cited by 28
“Quality assessment Study quality was assessed using the Newcastle-Ottawa Scale [18] which was modified [19] for this review.”
MethodMaternal obesity increases the risk of metabolic disease and impacts renal health in offspring
Glastras, S. J., Chen, H., Pollock, C. A., & Saad, S. (2018) · Bioscience Reports · doi:10.1042/bsr20180050 · Cited by 83
Chisari, C., Monajemi, M. B., Scott, W., Moss‐Morris, R., & McCracken, L. M. (2021) · European Journal of Pain · doi:10.1002/ejp.1668 · Cited by 83
“Eligible full-text articles were assessed for methodological quality and risk of bias according to the Critical Appraisal Skills Programme (CASP) guidelines (Sanderson et al., 2007), selected by the specific methodological design of included studies.”
MethodReligion, faith, and spirituality influences on HIV prevention activities: A scoping review
Vigliotti, V. S., Taggart, T., Walker, M., Kusmastuti, S., & Ransome, Y. (2020) · PLoS ONE · doi:10.1371/journal.pone.0234720 · Cited by 83
“One reviewer (TT) conducted a quality assessment of the 29 included studies using a checklist for assessing quality in observational studies [38].”
MethodHemorrhagic Transformation in Acute Ischemic Stroke: A Quantitative Systematic Review
Honig, A., Percy, J., Sepehry, A. A., Gómez, A., Field, T. S., & Benavente, O. (2022) · Journal of Clinical Medicine · doi:10.3390/jcm11051162 · Cited by 83
“) independently assessed the risk of bias for individual studies according to widely accepted tools [15], noting methodology for participant selection, HT outcomes, blinding, loss to follow-up, methods for controlling confounding, and declaration of conflicts of interest.”
MethodImpact of Ambient Humidity on Child Health: A Systematic Review
Gao, J., Sun, Y., Lu, Y., & Li, L. (2014) · PLoS ONE · doi:10.1371/journal.pone.0112508 · Cited by 70
“Thus experts have suggested that the STROBE statement provides a suitable starting point for development of a quality assessment tool [20, 88].”
BackgroundGendered specialities during medical education: a literature review
Alers, M., van Leerdam, L., Dielissen, P. W., & Lagro-Janssen, A. L. M. (2014) · Perspectives on Medical Education · doi:10.1007/s40037-014-0132-1 · Cited by 83
“There are few tools in the literature available to assess quality in observational studies [16] and only one of them had some interface with the selected articles in our review [17].”
MethodSamuel, G. O., Hoffmann, S., Wright, R. A., Lalu, M. M., Patlewicz, G., Becker, R. A., DeGeorge, G. L., Fergusson, D., Hartung, T., Lewis, R. J., et al. (2016) · Environment International · doi:10.1016/j.envint.2016.03.010 · Cited by 82
Effects of blood glucose level on 18F-FDG uptake for PET/CT in normal organs: A systematic review
Sprinz, C., Altmayer, S., Zanon, M., Watte, G., Irion, K., Marchiori, E., & Hochhegger, B. (2018) · PLOS ONE · doi:10.1371/journal.pone.0193140 · Cited by 82
Pulpal Reactions to Orthodontic Force Application in Humans: A Systematic Review
von Böhl, M., Ren, Y., Fudalej, P. S., & Kuijpers‐Jagtman, A. M. (2012) · Journal of Endodontics · doi:10.1016/j.joen.2012.07.001 · Cited by 82
Prevalence and levels of burnout in nursing students: A systematic review with meta-analysis
Gómez‐Urquiza, J. L., Velando‐Soriano, A., Membrive‐Jiménez, M. J., Ramírez‐Baena, L., Aguayo‐Estremera, R., Ortega‐Campos, E., & la Fuente, G. A. C. (2023) · Nurse Education in Practice · doi:10.1016/j.nepr.2023.103753 · Cited by 82
Gut dysbiosis in patients with chronic pain: a systematic review and meta-analysis
Goudman, L., Demuyser, T., Pilitsis, J. G., Billot, M., Roulaud, M., Rigoard, P., & Moens, M. (2024) · Frontiers in Immunology · doi:10.3389/fimmu.2024.1342833 · Cited by 82
Song, E., Sun, H., Xu, Y., Ma, Y., Zhu, H. L., & Pan, C. (2014) · PLoS ONE · doi:10.1371/journal.pone.0112054 · Cited by 82
“The study quality were assessed using the tool described by Sanderson et al [17].”
MethodNeal, S., Stone, N., & Ingham, R. (2016) · BMC Public Health · doi:10.1186/s12889-016-2868-5 · Cited by 65
“A checklist tool for rating quality was developed based on recommended practice [13, 14] and studies were rated for quality based on such factors as sample selection and size, measurement methods for outcome, methodology, adjustment for other confounding factors, method of measuring exposure to conflict and descriptio…”
MethodHypertension in autosomal dominant polycystic kidney disease: a meta-analysis
Marlais, M., Cuthell, O., Langan, D., Dudley, J., Sinha, M. D., & Winyard, P. J. D. (2016) · Archives of Disease in Childhood · doi:10.1136/archdischild-2015-310221 · Cited by 82
“Risk of bias in included studies Although there are many tools which can be used to assess observational studies in epidemiology as detailed in the review by Sanderson et al,7 we are not aware of any widely used quality assessment tool for prevalence studies and as different study designs were included in this review,…”
MethodMannelli, G., Cecconi, L., & Gallo, O. (2016) · Critical Reviews in Oncology/Hematology · doi:10.1016/j.critrevonc.2016.07.004 · Cited by 82
Galloway, H., & Newman, E. (2016) · ADHD Attention Deficit and Hyperactivity Disorders · doi:10.1007/s12402-016-0210-9 · Cited by 82
“Although the intended purpose of STROBE was to act as a reporting guide to authors of observational studies, it has been endorsed by researchers as a starting point for the methodological appraisal of non-experimental studies (Sanderson et al. 2007).”
BackgroundMethodInfluentialAl-Omary, M. S., Davies, A. J., Evans, T., Bastian, B. C., Fletcher, P., Attia, J. R., & Boyle, A. (2018) · Heart Lung and Circulation · doi:10.1016/j.hlc.2018.01.009 · Cited by 82
A systematic review and meta-analysis of return to work after mild Traumatic brain injury
Bloom, B., Thomas, S. H., Ahrensberg, J. M., Weaver, R., Fowler, A. J., Bestwick, J. P., Harris, T., & Pearse, R. M. (2018) · Brain Injury · doi:10.1080/02699052.2018.1532111 · Cited by 82
“Multiple tools for assessing bias have been developed, with no single tool being globally accepted, but the most important domains of a tool are appropriate selection of participants, appropriate measurement of variables and appropriate control of confounding (30).”
MethodLaird, J. J., Klettke, B., Hall, K., Clancy, E., & Hallford, D. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.17682 · Cited by 81
Sutradhar, I., Gupta, R. D., Hasan, M., Wazib, A., & Sarker, M. (2019) · Cureus · doi:10.7759/cureus.3970 · Cited by 81
Metadata
- Authors
- Sanderson, S., Tatt, I. D., & Higgins, J. P.
- Affiliations
- University of Cambridge (GB)
- F. Hoffmann-La Roche AG (CH)
- MRC Biostatistics Unit (GB)
- Corresponding author
- Sanderson, S.
- ORCID
- Higgins, J. P.
- Venue
- International Journal of Epidemiology
- Publisher
- Oxford University Press (OUP)
- Published
- 2007-04-30
- Volume
- 36
- Issue
- 3
- Pages
- 666-676
- ISSN
- 0300-5771 (ISSN-L)
- 1464-3685
- OpenAlex
- W2171251365
- Cited by
- 1,531
- References
- 73
- Citations by year
- 2026: 22 · 2025: 38 · 2024: 42 · 2023: 59 · 2022: 80 · 2021: 96 · 2020: 101 · 2019: 105 · 2018: 125 · 2017: 104 · 2016: 150 · 2015: 118 · 2014: 122 · 2013: 111 · 2012: 96
- FWCI
- 7.09
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- International Journal of Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- quality assessment tools, bias assessment, observational studies, critical appraisal, checklists, rating scales, STROBE, confounding adjustment, selection bias, measurement bias, conflicts of interest
- MeSH terms
- Humans, Research Design, Bias, Epidemiologic Studies, Evidence-Based Medicine, Observation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.