Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography
The DOI matches its Crossref record; 73 of the 73 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Assessing quality and susceptibility to bias is essential when interpreting primary research and conducting systematic reviews and meta-analyses. Tools for assessing quality in clinical trials are well-described but much less attention has been given to similar tools for observational epidemiological studies. METHODS: Tools were identified from a search of three electronic databases, bibliographies and an Internet search using Google. Two reviewers extracted data using a pre-piloted extraction form and strict inclusion criteria. Tool content was evaluated for domains potentially related to bias and was informed by the STROBE guidelines for reporting observational epidemiological studies. RESULTS: A total of 86 tools were reviewed, comprising 41 simple checklists, 12 checklists with additional summary judgements and 33 scales. The number of items ranged from 3 to 36 (mean 13.7). One-third of tools were designed for single use in a specific review and one-third for critical appraisal. Half of the tools provided development details, although most were proposed for future use in other contexts. Most tools included items for selection methods (92%), measurement of study variables (86%), design-specific sources of bias (86%), control of confounding (78%) and use of statistics (78%); only 4% addressed conflict of interest. The distribution and weighting of domains across tools was variable and inconsistent. CONCLUSION: A number of useful assessment tools have been identified by this report. Tools should be rigorously developed, evidence-based, valid, reliable and easy to use. There is a need to agree on critical elements for assessing susceptibility to bias in observational epidemiology and to develop appropriate evaluation tools.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Such instruments have been developed by other groups and were the subject of a recent systematic review [28].”
From The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: Guidelines for Reporting Observational Studies (2007) · cites this paper as [28]
In the citing paper
“It is not clear how weights should be assigned to each item of the scale [18].”
From Tools used to assess the quality of peer review reports: A methodological systematic review (2019) · cites this paper as [18]
In the citing paper
“The reviewers concluded that there is no ‘single obvious candidate tool for assessing quality of observational epidemiological studies’.”
From COSMOS-E: Guidance on conducting systematic reviews and meta-analyses of observational studies of etiology (2019) · cites this paper as [45]
In the citing paper
“A review identified more than 80 tools for assessing risk of bias in nonrandomised studies [45].”
From COSMOS-E: Guidance on conducting systematic reviews and meta-analyses of observational studies of etiology (2019) · cites this paper as [45]
In the citing paper
“The need for complete reporting of all relevant methodological aspects as a prerequisite to assessing the risk of bias of studies was highlighted in 10 articles, [16, 33, 35, 47, 50, 56, 86, 87, 90, 91] and was the focus of recommendations in a publication from the Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group [12].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [90]
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Meta-analysis of Observational Studies in Epidemiology<SUBTITLE>A Proposal for Reporting</SUBTITLE>
Stroup, D. F. (2000) · JAMA · doi:10.1001/jama.283.15.2008 · Cited by 20,623
Downs, S. H., & Black, N. (1998) · Journal of Epidemiology and Community Health · doi:10.1136/jech.52.6.377 · Cited by 8,252
Moher, D., Cook, D. J., Eastwood, S., Olkin, I., Rennie, D., & Stroup, D. F. (1999) · The Lancet · doi:10.1016/s0140-6736(99)04149-5 · Cited by 4,826
Whiting, P., Rutjes, A. W., Reitsma, J. B., Bossuyt, P. M., & Kleijnen, J. (2003) · BMC Medical Research Methodology · doi:10.1186/1471-2288-3-25 · Cited by 3,592
Assessing the quality of controlled clinical trials
Jüni, P., Altman, D. G., & Egger, M. (2001) · BMJ · doi:10.1136/bmj.323.7303.42 · Cited by 3,309 · cited 5×
“a (5) (5) (5) ( 5 ) N N LongneckerCoh”
InfluentialUsers' guides to the medical literature
Guyatt, G. H. (1993) · JAMA · doi:10.1001/jama.270.17.2096 · Cited by 2,970
Evaluating non-randomised intervention studies
Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D. (2003) · Health Technology Assessment · doi:10.3310/hta7270 · Cited by 2,907
Cognitive and Behavioral Outcomes of School-Aged Children Who Were Born Preterm: A Meta-analysis
Bhutta, A. T., Cleves, M. A., Casey, P. H., Cradock, M. M., & Anand, K. J. S. (2002) · JAMA · doi:10.1001/jama.288.6.728 · Cited by 2,392 · cited 1×
“Y N N 9 (22) 2 (17) 5 (15) 16 (19) Y Y N 15 (36) 6 ( 50 ) 7 (21) 28 (32)”
The Hazards of Scoring the Quality of Clinical Trials for Meta-analysis
Jüni, P. (1999) · JAMA · doi:10.1001/jama.282.11.1054 · Cited by 1,981
Low grade inflammation and coronary heart disease: prospective study and updated meta-analyses
Danesh, J. (2000) · BMJ · doi:10.1136/bmj.321.7255.199 · Cited by 1,692
Research works printed but not linked: 7
- Carneiro, 2002, Revista portuguesa de cardiologia : org o oficial da Sociedade Portuguesa de Cardiologia = Portuguese journal of cardiology : an official journal of the Portuguese Society of Cardiology, 21, 891
- 1981, Canadian Medical Association Journal, 124, 985
- 2002, SOUTH AFRICAN J CLIN NUTR, 15, 68
- 2000, Chronic diseases in Canada, 19, 170
- Ciliska, 1996, Canadian journal of public health. Revue canadienne de sant publique, 87, 193
- 1994, Canadian journal of public health. Revue canadienne de sant publique, 85, S8
- Spitzer, 1990, Clinical and investigative medicine. M decine clinique et experimentale, 13, 17
Cited by
Items 501–600 of 1,006
Zhai, K., Gao, X., & Wang, G. (2019) · SAGE Open · doi:10.1177/2158244019850263 · Cited by 81
Schwingshackl, L., Schünemann, H. J., & Meerpohl, J. J. (2021) · European Journal of Nutrition · doi:10.1007/s00394-020-02464-1 · Cited by 71
“Focusing on NRSs, a SR identified 86 tools to assess study quality in NRSs showing high inconsistency in selection/inclusion and weighting of domains across tools [32].”
MethodSexual dysfunction related to psychiatric disorders: a systematic review
Herder, T., Spoelstra, S. K., Peters, A. W. M., & Knegtering, H. (2023) · The Journal of Sexual Medicine · doi:10.1093/jsxmed/qdad074 · Cited by 81
“38,39 We used the Newcastle–Ottawa Scale to estimate the risk of bias in individual studies.”
Television viewing associated with adverse dietary outcomes in children ages 2–6
Ford, C., Ward, D., & White, M. (2012) · Obesity Reviews · doi:10.1111/j.1467-789x.2012.01028.x · Cited by 63
“Potential quality evaluation instruments were identified from two recent literature reviews of tools for assessing study quality (37,38).”
MethodPartial Versus Radical Nephrectomy in Patients With Adverse Clinical or Pathologic Characteristics
Claudio Jeldres, A., Patard, J., Umberto Capitanio, A., Perrotte, P., Suardi, N. R., Crépel, M., Ficarra, V., Cindolo, L., TAILLE, A. D. L., Tostain, J., et al. (2009) · Urology · doi:10.1016/j.urology.2008.08.492 · Cited by 81
Nicoletti, D., Appel, L. D., Neto, P. S., Guimarães, G. W., & Zhang, L. (2014) · Cadernos de Saúde Pública · doi:10.1590/0102-311x00115813 · Cited by 80
Brons, S., van Beusichem, M. E., Bronkhorst, E. M., Draaisma, J. M. T., Bergé, S. J., Maal, T., & Kuijpers‐Jagtman, A. M. (2012) · PLoS ONE · doi:10.1371/journal.pone.0041898 · Cited by 80
“In literature, there is not one single tool that is an obvious candidate for assessment of methodological quality of non-randomized studies [82].”
BackgroundMeta-analysis of the efficacy of moxifloxacin in treating Mycoplasma genitalium infection
Li, Y., Le, W.-J., Li, S., Cao, Y.-P., & Su, X.-H. (2017) · International Journal of STD & AIDS · doi:10.1177/0956462416688562 · Cited by 78
“Within-study bias for observational studies was assessed according to the evaluation criteria adopted by Sanderson et al.18 in a systematic review of tools used to evaluate bias in observational studies.”
MethodRobotic versus thoracoscopic lung resection: A systematic review and meta-analysis
Emmert, A., Straube, C., Büntzel, J., & Röver, C. (2017) · Medicine · doi:10.1097/md.0000000000007633 · Cited by 80
Bodkin, C., Pivnick, L., Bondy, S. J., Ziegler, C. P., Martin, R. E., Jernigan, C. A., & Kouyoumdjian, F. G. (2019) · American Journal of Public Health · doi:10.2105/ajph.2018.304855 · Cited by 80
Influence of Obesity on Clinical Outcomes in Hospitalized Children: A Systematic Review
Bechard, L. J., Rothpletz‐Puglia, P., Touger-Decker, R. E., Duggan, C., & Mehta, N. M. (2013) · Archives of Pediatrics and Adolescent Medicine · doi:10.1001/jamapediatrics.2013.13 · Cited by 80
“The quality of the research articles included in this review was assessed using the Newcastle-Ottawa Scale, a tool developed and partially validated for the purposes of evaluating nonrandomized studies used in systematic reviews and meta-analyses.(7,15,16) One author (L.”
MethodSprague, S., Goslings, J. C., Hogentoren, C., de Milliano, S., Simunovic, N., Madden, K., & Bhandari, M. (2014) · Violence Against Women · doi:10.1177/1077801213520574 · Cited by 64
Workplace Social Capital and Employee Health
Oksanen, T. (2009) · UTUPub (University of Turku) · Cited by 78
Quilter, M., Hiraki, L., & Korczak, D. J. (2019) · Lupus · doi:10.1177/0961203319853621 · Cited by 77
Kirpalani, H. M., & Zupancic, J. A. F. (2012) · Seminars in Perinatology · doi:10.1053/j.semperi.2012.04.007 · Cited by 77
Miller, L., McFadyen, A. K., Lord, A. C., Hunter, R., Paul, L., Rafferty, D., Bowers, R. J., & Mattison, P. (2017) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2016.12.007 · Cited by 77
Limitations and Biases in Cohort Studies
Ramírez-Santana, M. (2018) · InTech eBooks · doi:10.5772/intechopen.74324 · Cited by 77
Systematic review with meta‐analysis: the risks of proton pump inhibitors during pregnancy
Li, C. M., Zhernakova, A., Engstrand, L. G., Wijmenga, C., & Brusselaers, N. (2020) · Alimentary Pharmacology & Therapeutics · doi:10.1111/apt.15610 · Cited by 76
Burden of influenza in Latin America and the Caribbean: a systematic review and meta‐analysis
Savy, V., Ciapponi, A., Bardach, A., Glujovsky, D., Aruj, P. K., Mazzoni, A., Gibbons, L., Ortega‐Barría, E., & Colindres, R. E. (2012) · Influenza and Other Respiratory Viruses · doi:10.1111/irv.12036 · Cited by 76
Paulsen, J., Solligård, E., Damås, J. K., DeWan, A. T., Åsvold, B. O., & Bracken, M. B. (2016) · Open Forum Infectious Diseases · doi:10.1093/ofid/ofw048 · Cited by 76
“In general, there is a need for improved instruments for assessing the quality of observational studies included in systematic reviews [39].”
BackgroundThe volume-outcome relationship in severely injured patients: A systematic review and meta-analysis
Sewalt, C. A., Wiegers, E., Venema, E., Lecky, F. E., Schuit, S. C. E., Hartog, D. D., & Lingsma, H. F. (2018) · The Journal of Trauma: Injury, Infection, and Critical Care · doi:10.1097/ta.0000000000002043 · Cited by 76
Drukker, M., Weltens, I., van Hooijdonk, C. F. M., Vandenberk, E., & Bak, M. (2021) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2021.675071 · Cited by 75
“Earlier, 80 observational study quality lists were found, of which none was identified as the single comprehensive quality assessment list (Sanderson et al., 2007; Dekkers et al., 2019).”
BackgroundPaulus, J. K., Dahabreh, I. J., Balk, E. M., Avendano, E. E., Lau, J., & Ip, S. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1101 · Cited by 62
“Several tools exist for the evaluation of risk of bias (or “quality”) in observational studies but none are specifically tailored or validated for use with single group studies (Deeks et al., 2003; Effective Practice and Organisation of Care, 2013; Ramsay et al., 2003; Sanderson et al., 2007).”
MethodBona, D. D., Plaia, A., Leto‐Barone, M. S., Piana, S. L., Macchia, L., & Lorenzo, G. D. (2016) · Allergy · doi:10.1111/all.13104 · Cited by 75
“Socioeconomic status, education, severity of pre-existing disease, and the presence of comorbidities are common confounding factors that have been shown to be particularly important when the choice (or refusal) of a treat-ment is left to the patients or children’s parents, as in this case (47).”
Duration of Breastfeeding and Early Growth: A Systematic Review of Current Evidence
Patro-Gołąb, B., Zalewski, B., Polaczek, A., & Szajewska, H. (2019) · Breastfeeding Medicine · doi:10.1089/bfm.2018.0187 · Cited by 74
“12 The original (unadapted) scale requires one to choose the most relevant confounding factor for further assessment in the comparability domain.”
Quality of Life After Coronary Artery Bypass Surgery: A Systematic Review and Meta-Analysis
Schmidt‐RioValle, J., Ejheisheh, M. A., Membrive‐Jiménez, M. J., Suleiman‐Martos, N., Albendín‐García, L., Correa‐Rodríguez, M., & Gómez‐Urquiza, J. L. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17228439 · Cited by 74
“st i s i cl i t r s rc r r i critic ll f r i s l sis, si t tr t i the Reporting of Observational Studies in Epidemiology (STROBE) checklist [12].”
BackgroundMethodMamikutty, R., Aly, A. S., & Marhazlinda, J. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18168623 · Cited by 75
“First, the ROB tool must be a simple checklist rather than a scale [8].”
BackgroundMethodInfluentialAntireflux Surgery and Risk of Esophageal Adenocarcinoma: A Systematic Review and Meta-analysis
Maret‐Ouda, J., Könings, P., Lagergren, J., & Brusselaers, N. (2015) · Annals of Surgery · doi:10.1097/sla.0000000000001438 · Cited by 74
“com exposure and outcome, sources of bias, methods for controlling for potential confounding, and conflicts of interests.(12)”
MethodTurner, S. L., Johnson, R. D., Weightman, A. L., Rodgers, S. E., Arthur, G., Bailey, R., & Lyons, R. A. (2017) · Injury Prevention · doi:10.1136/injuryprev-2016-042174 · Cited by 73
Jackson, V. A., Chou, S., & Browne, K. D. (2015) · Trauma Violence & Abuse · doi:10.1177/1524838015611675 · Cited by 73
“This centers on the Cochrane Collaboration’s bias assessment tool (Higgins & Altman, 2008), which meant quality assessment was based on key areas of bias instead of a points system (Sanderson et al., 2007; Stroup et al., 2000).”
MethodFood Insecurity and Cognitive Function in Middle to Older Adulthood: A Systematic Review
Na, M., Dou, N., Ji, N., Xie, D., Huang, J., Tucker, K. L., & Gao, X. (2019) · Advances in Nutrition · doi:10.1093/advances/nmz122 · Cited by 73
“Following the recommendation that a systematic review use tools for assessing data quality and bias in observational studies (23), we used a published checklist to qualitatively evaluate the study quality.”
MethodPrevalence of Depression in Coronary Artery Bypass Surgery: A Systematic Review and Meta-Analysis
Correa‐Rodríguez, M., Ejheisheh, M. A., Suleiman‐Martos, N., Membrive‐Jiménez, M. J., Velando‐Soriano, A., Schmidt‐RioValle, J., & Gómez‐Urquiza, J. L. (2020) · Journal of Clinical Medicine · doi:10.3390/jcm9040909 · Cited by 73
“For observational studies (cohort and cross-sectional), we followed the guidelines in “Strengthening the Reporting of Observational Studies in Epidemiology” (STROBE) [19].”
MethodSafety incidents associated with extended working hours. A systematic review and meta-analysis
Matre, D., Skogstad, M., Sterud, T., Nordby, K.-C., Knardahl, S., Christensen, J. O., & Lie, J.-A. S. (2021) · Scandinavian Journal of Work Environment & Health · doi:10.5271/sjweh.3958 · Cited by 73
“There is no consensus about how to assess certainty of evidence in observational epidemiological studies (19).”
BackgroundThe Use and Misuse of Biomedical Data: Is Bigger Really Better?
Hoffman, S., & Podgurski, A. (2013) · American Journal of Law & Medicine · doi:10.1177/009885881303900401 · Cited by 55
Winstanley, E. L., Mahoney, J. J., Castillo, F., & Comer, S. D. (2021) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2021.108838 · Cited by 72
“Quality assessment Existing tools to assess bias of studies included in systematic reviews are study design specific; few apply to multiple study designs and none can be used to assess case reports/series, cohort, and case-control studies (Sanderson et al., 2007; Viswanathan et al., 2012).”
MethodSargeant, J. M., O'Connor, A. M. C., Dohoo, I. R., Erb, H. N., Cevallos, M., Egger, M., Ersbøll, A. K., Martin, S. W., Nielsen, L. R., Pearl, D. L., et al. (2016) · Journal of Veterinary Internal Medicine · doi:10.1111/jvim.14574 · Cited by 73
Lyu, J. C., Huang, P., Jiang, N., & Ling, P. M. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19159263 · Cited by 72
“Therefore, we adapted a checklist from the Cochrane Collaboration’s tool for assessing risk of bias [32] and the six most commonly evaluated domains in quality and bias assessment identified by Sanderson and colleagues after reviewing 86 appraisal tools [33].”
MethodCementless total hip arthroplasty in rheumatoid arthritis: a systematic review of the literature
Zwartelé, R. E., Witjes, S., Doets, H. C., Stijnen, T., & Pöll, R. G. (2011) · Archives of Orthopaedic and Trauma Surgery · doi:10.1007/s00402-011-1432-0 · Cited by 72
Effects of product reformulation on sugar intake and health—a systematic review and meta-analysis
Hashem, K. M., He, F. J., & MacGregor, G. A. (2018) · Nutrition Reviews · doi:10.1093/nutrit/nuy015 · Cited by 72
HIV-related care for transgender people: A systematic review of studies from around the world
Fontanari, A. M. V., Zanella, G. I., Feijó, M., Churchill, S. M., Lobato, M. I. R., & Costa, A. B. (2019) · Social Science & Medicine · doi:10.1016/j.socscimed.2019.03.016 · Cited by 72
Adherence to Oral Maintenance Treatment in Adolescents with Inflammatory Bowel Disease
Spekhorst, L. M., Hummel, T. Z., Benninga, M. A., van Rheenen, P. F., & Kindermann, A. (2016) · Journal of Pediatric Gastroenterology and Nutrition · doi:10.1097/mpg.0000000000000924 · Cited by 71
“chos From the checklist, 6 of the best differentiating items were en (Table 1) (6).”
BackgroundDoes injury compensation lead to worse health after whiplash? A systematic review
Spearing, N., Connelly, L. B., Gargett, S., & Sterling, M. M. (2012) · Pain · doi:10.1016/j.pain.2012.03.007 · Cited by 71
“Following a review of the literature and existing quality checklists [1,10,21,27,31,42], a quality-assessment checklist comprising 16 items was developed to evaluate the potential for selection bias and measurement bias (Table 1).”
MethodLoudon, K., Santesso, N. A., Callaghan, M., Thornton, J., Harbour, J., Graham, K., Harbour, R., Kunnamo, I., Liira, H., McFarlane, E., et al. (2014) · BMC Health Services Research · doi:10.1186/1472-6963-14-321 · Cited by 71
“Two reviewers independently assessed the methodological quality of the included quantitative and qualitative studies using the relevant CASP tool [12,13].”
MethodPositive psychology in the investigation of psychedelics and entactogens: a critical review
Jungaberle, H., Thal, S., Zeuch, A., Rougemont-Bücking, A., von Heyden, M., Aicher, H., & Scheidegger, M. (2018) · Neuropharmacology · doi:10.1016/j.neuropharm.2018.06.034 · Cited by 70
Incidence rate of sudden death in epilepsy: A systematic review and meta-analysis
Sætre, E., & Abdelnoor, M. (2018) · Epilepsy & Behavior · doi:10.1016/j.yebeh.2018.06.037 · Cited by 70
Associations between lower limb strength and gait velocity following stroke: A systematic review
Mentiplay, B. F., Adair, B., Bower, K. J., Williams, G. P., Tolé, G., & Clark, R. A. (2014) · Brain Injury · doi:10.3109/02699052.2014.995231 · Cited by 69
“Currently, there is a lack of consensus on the most appropriate measure of methodological quality in observational research [28, 29].”
BackgroundChow, E. P. F., Tucker, J. D., Wong, F. Y., Nehl, E. J., Wang, Y., Zhuang, X., & Zhang, L. (2014) · PLoS ONE · doi:10.1371/journal.pone.0089959 · Cited by 70
“Quality Assessment The quality assessment of eligible studies was measured according to the checklist tools for assessing quality in observational studies [40].”
MethodWork above shoulder level and shoulder complaints: a systematic review
Wærsted, M., Koch, M., & Veiersted, K. B. (2020) · International Archives of Occupational and Environmental Health · doi:10.1007/s00420-020-01551-4 · Cited by 69
Systematic review: preoperative psychological factors and total hip arthroplasty outcomes
O’Connor, J., Holden, P., & Gagnier, J. J. (2022) · Journal of Orthopaedic Surgery and Research · doi:10.1186/s13018-022-03355-3 · Cited by 69
“Studies were divided into two groups based on the number of criteria fulfilled [27].”
BackgroundMindlin, M., Jenkins, R., & Law, C. (2009) · Journal of Epidemiology & Community Health · doi:10.1136/jech.2008.077073 · Cited by 69
Barr, P. J., Donnelly, M., Cardwell, C. R., Alam, S. S., Morris, K., Parker, M., & Bailie, K. (2011) · Transfusion Medicine Reviews · doi:10.1016/j.tmrv.2011.04.003 · Cited by 69
Targets for Health Interventions for Inflammatory Bowel Disease-fatigue
Artom, M., Czuber-Dochan, W., Sturt, J., & Norton, C. (2016) · Journal of Crohn's and Colitis · doi:10.1093/ecco-jcc/jjw029 · Cited by 64
“The quality of included papers (n = 43) was assessed using the Critical Appraisal Skills Programme (CASP) assessment tools specific to the methodological design of each study (27).”
A systematic review: Tools for assessing methodological quality of human observational studies
Wang, Z., Taylor, K., Allman-Farinelli, M., Armstrong, B., Askie, L., Ghersi, D., McKenzie, J., Norris, S. L., Page, M., Rooney, A. A., et al. (2019) · doi:10.31222/osf.io/pnqmy · Cited by 52
A Systematic Literature Review of Cerebrospinal Fluid Biomarkers in Delirium
Hall, R. J., Shenkin, S. D., & MacLullich, A. M. J. (2011) · Dementia and Geriatric Cognitive Disorders · doi:10.1159/000330757 · Cited by 68
“(2) that we could extract data on biomarker findings in CSF in delirium; (3) studies investigating CSF in neuropsychiatric systemic lupus erythematosus (SLE) were included only where delirium was thought to be secondary to a cause other than neuropsychiatric SLE. Exclusion criteria were: (1) case reports; (2) studies…”
Association Between Periodontitis and Cognitive Impairment in Adults: A Systematic Review
Nascimento, P. C., Castro, M. M. L., Magno, M. B., Almeida, A. P. C. P. S. C., Fagundes, N. C. F., Maia, L. C., & Lima, R. R. (2019) · Frontiers in Neurology · doi:10.3389/fneur.2019.00323 · Cited by 52
Richardson, E., Lewis, J. S., Gibson, J., Morgan, C., Halaki, M., Ginn, K. A., & Yeowell, G. (2020) · BMJ Open Sport & Exercise Medicine · doi:10.1136/bmjsem-2019-000683 · Cited by 68
In-work tax credits for families and their impact on health status in adults
Pega, F., Carter, K. N., Blakely, T. A., & Lucas, P. J. (2013) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd009963.pub2 · Cited by 67
Clopidogrel and hip fractures, is it safe? A systematic review and meta-analysis
Soo, C. G. K. M., Torre, P. K. D., Yolland, T. J., & Shatwell, M. A. (2016) · BMC Musculoskeletal Disorders · doi:10.1186/s12891-016-0988-9 · Cited by 68
Macri, E. M., Stefanik, J. J., Karim, K., & Crossley, K. M. (2016) · Arthritis Care & Research · doi:10.1002/acr.22842 · Cited by 68
“Our tool was adapted from Downs and Black (14) and influenced by previously published modifications (15,16), recent systematic reviews of quality assessment tools (17,18) and the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guidelines for reporting of observational studies (19).”
MethodCiapponi, A. (2018) · Evidencia actualizacion en la práctica ambulatoria · doi:10.51987/evidencia.v21i1.6834 · Cited by 68
Outcomes of endodontically treated cracked teeth: a systematic review and meta-analysis
Leong, D. J. X., de Souza, N. N., Sultana, R., & Yap, A. U. (2019) · Clinical Oral Investigations · doi:10.1007/s00784-019-03139-w · Cited by 67
Agabiirwe, C. N., Dambach, P., Methula, T. C., & Phalkey, R. K. (2022) · Environmental Health · doi:10.1186/s12940-022-00910-7 · Cited by 68
“Quality assessment of selected studies We assessed each of the selected study’s quality using the NIH tool for Observational Cohort and Cross-Sectional Studies [36].”
MethodInfluentialHändel, M. N., Heitmann, B. L., & Abrahamsen, B. (2015) · American Journal of Clinical Nutrition · doi:10.3945/ajcn.115.108456 · Cited by 67
Davidsen, K. A., Harder, S. K., MacBeth, A. M., Lundy, J.-M., & Gumley, A. I. (2015) · Social Psychiatry and Psychiatric Epidemiology · doi:10.1007/s00127-015-1127-x · Cited by 67
“Risk of bias We systematically assessed the risk of bias via a methodological evaluation of all studies (SH & AG) using methods developed for observational studies in epidemiology [22].”
MethodPreeclampsia and ABO blood groups: a systematic review and meta-analysis
Alpoim, P. N., Pinheiro, M. D. B., Junqueira, D. R., Freitas, L. G., Ferriani, M. D. G. C., Fernandes, A. P. S. M., Komatsuzaki, F., Gomes, K. B., & Dusse, L. M. S. (2012) · Molecular Biology Reports · doi:10.1007/s11033-012-2288-2 · Cited by 67
van Abbema, R., Lakke, S. E., Reneman, M. F., van der Schans, C. P., van Haastert, C. J. M., Geertzen, J. H. B., & Wittink, H. M. (2011) · Journal of Occupational Rehabilitation · doi:10.1007/s10926-011-9306-4 · Cited by 68
“However, no clearly defined tools for assessing quality and susceptibility to bias in cross-sectional studies are available [14, 15].”
BackgroundMethodPhysical activity in spondyloarthritis: a systematic review
O’Dwyer, T., O'Shea, F. D., & Wilson, F. (2014) · Rheumatology International · doi:10.1007/s00296-014-3141-9 · Cited by 67
“Numerous appraisal tools are described in the literature, yet no gold standard exists for evaluating risk of bias [38, 39].”
BackgroundMyopia and diabetic retinopathy: A systematic review and meta-analysis
Wang, X., Tang, L., Gao, L., Yang, Y., Cao, D., & Li, Y. (2015) · Diabetes Research and Clinical Practice · doi:10.1016/j.diabres.2015.10.020 · Cited by 67
“We assessed the study quality using the tool described by Sanderson and colleagues [6].”
MethodBilandzic, A., Fitzpatrick, T., Rosella, L., & Henry, D. (2016) · PLOS Medicine · doi:10.1371/journal.pmed.1001987 · Cited by 60
“In their review, Sanderson and colleagues identified 86 assessment tools for non-randomized studies, comprising 41 simple checklists, 12 checklists with additional summary judgments, and 33 scales [11].”
BackgroundMethodInfluentialTao, W., Agerholm, J., & Burström, B. (2016) · BMC Health Services Research · doi:10.1186/s12913-016-1805-8 · Cited by 67
“The checklists were selected after reviewing several tools for critical appraisal [26].”
MethodA systematic review of enteric dysbiosis in chronic fatigue syndrome/myalgic encephalomyelitis
Preez, S. D., Corbitt, M., Cabanas, H., Eaton-Fitch, N., Staines, D., & Marshall‐Gradisnik, S. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0909-0 · Cited by 67
Livingston, M. H., Shawyer, A. C., Rosenbaum, P., Jones, S., & Walton, J. M. (2015) · Journal of Pediatric Surgery · doi:10.1016/j.jpedsurg.2015.02.020 · Cited by 66
“A recent systematic review identified tools for evaluating observational studies but there was no evidence that one was superior [25].”
BackgroundCan physical activity prevent physical and cognitive decline in postmenopausal women?
Anderson, D. J., Seib, C. A., & Rasmussen, L. (2014) · Maturitas · doi:10.1016/j.maturitas.2014.06.010 · Cited by 66
Pavão, S. L., Silva, F. P. D. S., Savelsbergh, G. J. P., & Rocha, N. A. C. F. (2014) · Journal of Motor Behavior · doi:10.1080/00222895.2014.981498 · Cited by 66
Jarde, A., Losilla, J.-M., Vives, J., & F. Rodrigo, M. (2013) · International Journal of Clinical and Health Psychology · doi:10.1016/s1697-2600(13)70017-6 · Cited by 50
Population Estimation Methods for Free-Ranging Dogs: A Systematic Review
Belo, V. S., Werneck, G. L., da Silva, E. S., Barbosa, D. S., & Struchiner, C. J. (2015) · PLoS ONE · doi:10.1371/journal.pone.0144830 · Cited by 66
“medical field [31] should serve as an example and guidance leading to similar initiatives in the field of Ecology.”
MethodNexø, M. A., Meng, A., & Borg, V. (2016) · Occupational and Environmental Medicine · doi:10.1136/oemed-2016-103550 · Cited by 66
International Journal of Economics, Finance and Management Sciences
Mazzitelli, D., Fiorenza, E., Belgacem, I., & Arena, C. (2023) · International Journal of Economics Finance and Management Sciences · doi:10.11648/j.ijefm · Cited by 66
Antibiotic use and risk of colorectal cancer: a systematic review and dose–response meta-analysis
Simin, J., Fornes, R., Liu, Q., Olsen, R. S., Callens, S. F. J., Engstrand, L. G., & Brusselaers, N. (2020) · British Journal of Cancer · doi:10.1038/s41416-020-01082-2 · Cited by 67
Lenssinck, M.-L. B., Alex Burdorf, A., Boonen, A. E. R. C. H., Gignac, M. A. M., Hazes, J. M. W., & Luime, J. J. (2012) · Annals of the Rheumatic Diseases · doi:10.1136/annrheumdis-2012-201998 · Cited by 65
“Studies were evaluated for susceptibility to bias using a quality assessment list developed by the authors, based on recommendations of Sanderson et al.10 The constructed checklist covered 10 items related to selection methods, measurement of study variables, design-specific sources of bias, control for confounding an…”
MethodInfluentialCooper, Z. R., Scott, J. W., Rosenthal, R. A., & Mitchell, S. L. (2015) · Journal of the American Geriatrics Society · doi:10.1111/jgs.13818 · Cited by 65
Moradi, V., Memari, A. H., Shayestehfar, M., & Kordi, R. (2015) · Rehabilitation Research and Practice · doi:10.1155/2015/850184 · Cited by 65
“php?id=available-checklists) which has been illustrated to be a valid tool for observational studies [23, 24].”
MethodField reliability of competency and sanity opinions: A systematic review and meta-analysis.
Guarnera, L. A., & Murrie, D. C. (2017) · Psychological Assessment · doi:10.1037/pas0000388 · Cited by 65
Iijima, H., Shimoura, K., Aoyama, T., & Takahashi, M. (2018) · Gait & Posture · doi:10.1016/j.gaitpost.2018.03.002 · Cited by 66
“This scale is a useful tool for assessing the risk of bias in observational studies [13] and the methodological quality of both randomized controlled trial (RCT) and non-RCT of treatment [10].”
MethodCraciunoiu, O., & Holsti, L. (2016) · Physical & Occupational Therapy In Pediatrics · doi:10.1080/01942638.2016.1185501 · Cited by 64
“A challenge encountered when evaluating the methodological quality of neurodevelopmental assessments in preterm infants is the availability of numerous assessments juxtaposed with a lack of agreed upon “gold standard” for observational studies (e.g. Sanderson, 2007).”
BackgroundBardach, A., Rey‐Ares, L., Cafferata, M. L., Cormick, G., Romano, M., Ruvinsky, S., & Savy, V. (2013) · Reviews in Medical Virology · doi:10.1002/rmv.1775 · Cited by 64
Langham, S., Langham, J., Goertz, H., & Ratcliffe, M. (2011) · BMC Medical Research Methodology · doi:10.1186/1471-2288-11-32 · Cited by 64
“We devised our own quality assessment tool based on a number of papers including the Downs and Black score system [27], the STROBE statement [25] and a recent systematic review of measures for assessing quality and susceptibility to bias in observational studies [26].”
BackgroundMethodOverweight, Obesity, and Age‐Related Cataract: A Meta‐analysis
Pan, C., & Lin, Y. H. (2014) · Optometry and Vision Science · doi:10.1097/opx.0000000000000243 · Cited by 64
“We assessed the study quality using the tool described by Sanderson et al.10 The variables examined included the methods for selecting study participants, methods for measuring exposure (overweight/obesity) and outcome variable (age-related cataract), design-specific sources of bias (excluding confounding), methods fo…”
MethodNsengiyumva, V., Fernando, M. E., Moxon, J. V., Krishna, S. M., Pinchbeck, J. L., Omer, S. M., Morris, D., Jones, R. E., Moran, C. S., Seto, S. W., et al. (2015) · Atherosclerosis · doi:10.1016/j.atherosclerosis.2015.10.011 · Cited by 64
“Recognizing the limitations of generic quality assessment tools, a set of 21 content-specific questions that were relevant to PAD and vitamin D research were developed (Supplementary Table 1) [20].”
BackgroundDifferent Amounts of Alcohol Consumption and Cataract: A Meta‐analysis
Gong, Y., Feng, K., Yan, N., Xu, Y., & Pan, C. (2015) · Optometry and Vision Science · doi:10.1097/opx.0000000000000558 · Cited by 64
“We assessed the study quality using the tool described by Sanderson et al.20 The variables examined included the methods for selecting study participants, methods for measuring exposure (alcohol consumption) and outcome variable (age-related cataract), design-specific sources of bias (excluding confounding), methods f…”
MethodVeile, A., Zimmermann, H., Lorenz, E., & Becher, H. (2018) · BMJ Open · doi:10.1136/bmjopen-2017-016589 · Cited by 64
Kangaroo mother care and infant biopsychosocial outcomes in the first year: A meta-analysis
Akbari, E. M., Binnoon-Erez, N., Rodrigues, M., Ricci, A. L., Schneider, J., Madigan, S. L., & Jenkins, J. M. (2018) · Early Human Development · doi:10.1016/j.earlhumdev.2018.05.004 · Cited by 64
Management of the neurologically deceased organ donor: A Canadian clinical practice guideline
Ball, I. M., Hornby, L., Rochwerg, B. N., Weiss, M. J., Gillrie, C., Chassé, M., D’Aragon, F., Meade, M. O., Soliman, K., Ali, A., et al. (2020) · Canadian Medical Association Journal · doi:10.1503/cmaj.190631 · Cited by 64
Use of quality assessment tools in systematic reviews was varied and inconsistent
Seehra, J., Pandis, N., Koletsi, D., & Fleming, P. S. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.06.023 · Cited by 64
“11 without time limits undertaken in 2005 [5].”
BackgroundMethodInfluentialAssociations between renaming schizophrenia and stigma-related outcomes: a systematic review
Yamaguchi, S., Mizuno, M., Ojio, Y., Sawada, U., Matsunaga, A., Ando, S., & Koike, S. (2017) · Psychiatry and Clinical Neurosciences · doi:10.1111/pcn.12510 · Cited by 63
“We used this tool because it is the only appropriate tool for evaluating both cross-sectional and longitudinal studies.(14) The four domains consisted of: (i) proper assembly of cohort (longitudinal studies)/proper selection of study population (cross-sectional studies); (ii) control for confounders; (iii) soundness o…”
MethodWirsching, J., Graßmann, S., Eichelmann, F., Harms, L. M., Schenk, M., Barth, E., Berndzen, A., Olalekan, M., Sarmini, L., Zuberer, H., et al. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0583-x · Cited by 63
Wang, X., Cheng, B., Roberts, N., Wang, S., Luo, Y., Tian, F., & Yue, S. (2021) · Human Brain Mapping · doi:10.1002/hbm.25618 · Cited by 63
Probst, C., Lange, S., Kilian, C., Saul, C., & Rehm, J. T. (2021) · BMC Medicine · doi:10.1186/s12916-021-02132-z · Cited by 63
“Quality assessment was performed in line with the original systematic review [6] using the following criteria [19]: representativeness of the sample; measurement and definition of the independent and dependent variables; linkage of survey/census data; age-adjustment (for details of each criterion see Additional file 1…”
MethodBardach, A., Ciapponi, A., Martí, S. G., Glujovsky, D., Mazzoni, A., Fayad, A., Colindres, R. E., & de Gentile, A. S. (2011) · International Journal of Pediatric Otorhinolaryngology · doi:10.1016/j.ijporl.2011.05.014 · Cited by 63
“[22] and Fowkes and Fulton [23].”
van den Brink, A. M. A., Gerritsen, D. L., Voshaar, R. C. O., & Koopmans, R. T. C. M. (2013) · International Psychogeriatrics · doi:10.1017/s1041610212002025 · Cited by 51
Metadata
- Authors
- Sanderson, S., Tatt, I. D., & Higgins, J. P.
- Affiliations
- University of Cambridge (GB)
- F. Hoffmann-La Roche AG (CH)
- MRC Biostatistics Unit (GB)
- Corresponding author
- Sanderson, S.
- ORCID
- Higgins, J. P.
- Venue
- International Journal of Epidemiology
- Publisher
- Oxford University Press (OUP)
- Published
- 2007-04-30
- Volume
- 36
- Issue
- 3
- Pages
- 666-676
- ISSN
- 0300-5771 (ISSN-L)
- 1464-3685
- OpenAlex
- W2171251365
- Cited by
- 1,531
- References
- 73
- Citations by year
- 2026: 22 · 2025: 38 · 2024: 42 · 2023: 59 · 2022: 80 · 2021: 96 · 2020: 101 · 2019: 105 · 2018: 125 · 2017: 104 · 2016: 150 · 2015: 118 · 2014: 122 · 2013: 111 · 2012: 96
- FWCI
- 7.09
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- International Journal of Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- quality assessment tools, bias assessment, observational studies, critical appraisal, checklists, rating scales, STROBE, confounding adjustment, selection bias, measurement bias, conflicts of interest
- MeSH terms
- Humans, Research Design, Bias, Epidemiologic Studies, Evidence-Based Medicine, Observation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.