Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography
The DOI matches its Crossref record; 73 of the 73 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Assessing quality and susceptibility to bias is essential when interpreting primary research and conducting systematic reviews and meta-analyses. Tools for assessing quality in clinical trials are well-described but much less attention has been given to similar tools for observational epidemiological studies. METHODS: Tools were identified from a search of three electronic databases, bibliographies and an Internet search using Google. Two reviewers extracted data using a pre-piloted extraction form and strict inclusion criteria. Tool content was evaluated for domains potentially related to bias and was informed by the STROBE guidelines for reporting observational epidemiological studies. RESULTS: A total of 86 tools were reviewed, comprising 41 simple checklists, 12 checklists with additional summary judgements and 33 scales. The number of items ranged from 3 to 36 (mean 13.7). One-third of tools were designed for single use in a specific review and one-third for critical appraisal. Half of the tools provided development details, although most were proposed for future use in other contexts. Most tools included items for selection methods (92%), measurement of study variables (86%), design-specific sources of bias (86%), control of confounding (78%) and use of statistics (78%); only 4% addressed conflict of interest. The distribution and weighting of domains across tools was variable and inconsistent. CONCLUSION: A number of useful assessment tools have been identified by this report. Tools should be rigorously developed, evidence-based, valid, reliable and easy to use. There is a need to agree on critical elements for assessing susceptibility to bias in observational epidemiology and to develop appropriate evaluation tools.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Such instruments have been developed by other groups and were the subject of a recent systematic review [28].”
From The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: Guidelines for Reporting Observational Studies (2007) · cites this paper as [28]
In the citing paper
“It is not clear how weights should be assigned to each item of the scale [18].”
From Tools used to assess the quality of peer review reports: A methodological systematic review (2019) · cites this paper as [18]
In the citing paper
“The reviewers concluded that there is no ‘single obvious candidate tool for assessing quality of observational epidemiological studies’.”
From COSMOS-E: Guidance on conducting systematic reviews and meta-analyses of observational studies of etiology (2019) · cites this paper as [45]
In the citing paper
“A review identified more than 80 tools for assessing risk of bias in nonrandomised studies [45].”
From COSMOS-E: Guidance on conducting systematic reviews and meta-analyses of observational studies of etiology (2019) · cites this paper as [45]
In the citing paper
“The need for complete reporting of all relevant methodological aspects as a prerequisite to assessing the risk of bias of studies was highlighted in 10 articles, [16, 33, 35, 47, 50, 56, 86, 87, 90, 91] and was the focus of recommendations in a publication from the Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group [12].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [90]
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Meta-analysis of Observational Studies in Epidemiology<SUBTITLE>A Proposal for Reporting</SUBTITLE>
Stroup, D. F. (2000) · JAMA · doi:10.1001/jama.283.15.2008 · Cited by 20,623
Downs, S. H., & Black, N. (1998) · Journal of Epidemiology and Community Health · doi:10.1136/jech.52.6.377 · Cited by 8,252
Moher, D., Cook, D. J., Eastwood, S., Olkin, I., Rennie, D., & Stroup, D. F. (1999) · The Lancet · doi:10.1016/s0140-6736(99)04149-5 · Cited by 4,826
Whiting, P., Rutjes, A. W., Reitsma, J. B., Bossuyt, P. M., & Kleijnen, J. (2003) · BMC Medical Research Methodology · doi:10.1186/1471-2288-3-25 · Cited by 3,592
Assessing the quality of controlled clinical trials
Jüni, P., Altman, D. G., & Egger, M. (2001) · BMJ · doi:10.1136/bmj.323.7303.42 · Cited by 3,309 · cited 5×
“a (5) (5) (5) ( 5 ) N N LongneckerCoh”
InfluentialUsers' guides to the medical literature
Guyatt, G. H. (1993) · JAMA · doi:10.1001/jama.270.17.2096 · Cited by 2,970
Evaluating non-randomised intervention studies
Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D. (2003) · Health Technology Assessment · doi:10.3310/hta7270 · Cited by 2,907
Cognitive and Behavioral Outcomes of School-Aged Children Who Were Born Preterm: A Meta-analysis
Bhutta, A. T., Cleves, M. A., Casey, P. H., Cradock, M. M., & Anand, K. J. S. (2002) · JAMA · doi:10.1001/jama.288.6.728 · Cited by 2,392 · cited 1×
“Y N N 9 (22) 2 (17) 5 (15) 16 (19) Y Y N 15 (36) 6 ( 50 ) 7 (21) 28 (32)”
The Hazards of Scoring the Quality of Clinical Trials for Meta-analysis
Jüni, P. (1999) · JAMA · doi:10.1001/jama.282.11.1054 · Cited by 1,981
Low grade inflammation and coronary heart disease: prospective study and updated meta-analyses
Danesh, J. (2000) · BMJ · doi:10.1136/bmj.321.7255.199 · Cited by 1,692
Research works printed but not linked: 7
- Carneiro, 2002, Revista portuguesa de cardiologia : org o oficial da Sociedade Portuguesa de Cardiologia = Portuguese journal of cardiology : an official journal of the Portuguese Society of Cardiology, 21, 891
- 1981, Canadian Medical Association Journal, 124, 985
- 2002, SOUTH AFRICAN J CLIN NUTR, 15, 68
- 2000, Chronic diseases in Canada, 19, 170
- Ciliska, 1996, Canadian journal of public health. Revue canadienne de sant publique, 87, 193
- 1994, Canadian journal of public health. Revue canadienne de sant publique, 85, S8
- Spitzer, 1990, Clinical and investigative medicine. M decine clinique et experimentale, 13, 17
Cited by
Items 301–400 of 1,006
Laver, K., Prichard, I. J., Cations, M., Osenk, I., Govin, K., & Coveney, J. (2018) · BMJ Open · doi:10.1136/bmjopen-2017-020380 · Cited by 146
“The tools also rank well in systematic evaluations of quality assessment tools.(29) An amended version of the eight-item Checklist for Analytical Cross-Sectional Studies was used for observational and post-test only design studies, removing item four (‘Were objective, standard criteria used for measurement of the cond…”
MethodMortality in liver transplant recipients with portal vein thrombosis - an updated meta-analysis
Zanetto, A., Rodriguez–Castro, K. I., Germani, G., Ferrarese, A., Cillo, U., Burra, P., & Senzolo, M. (2018) · Transplant International · doi:10.1111/tri.13353 · Cited by 146
“In planning our systematic review, the Meta-analysis Of Observational Studies in Epidemiology (MOOSE) [12] guidelines were adopted for the meta-analysis, as they are more appropriate in the current setting than the Quality of Reporting of Meta-analyses (QUOROM), or Transparent Reporting of Evaluations with nonrandomiz…”
MethodGunasekara, F. I., Carter, K. N., & Blakely, T. A. (2010) · Social Science & Medicine · doi:10.1016/j.socscimed.2010.10.029 · Cited by 146
Benton, J. S., Anderson, J., Hunter, R. F., & French, D. P. (2016) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-016-0433-3 · Cited by 120
“Given that there has been little consistency in previous risk of bias tools that have weighted bias or quality domains [56], adjusting the principles of an established risk of bias tool by creating weighted bias domains would have been difficult to justify.”
BackgroundDewa, C. S., Loong, D., Bonato, S., Trojanowski, L., & Rea, M. (2017) · BMC Medical Education · doi:10.1186/s12909-017-1040-y · Cited by 126
“[22] recommend the use of a transparent checklist that concentrates on the “few, principal, and potential sources of bias in a study’s findings”.”
MethodBang, J. Y., Hawes, R. H., Bartolucci, A. A., & Varadarajulu, S. S. (2014) · Digestive Endoscopy · doi:10.1111/den.12418 · Cited by 144
Charlesworth, M., Verbeke, C. S., Falk, G. A., Walsh, M., Smith, A. M., & Morris‐Stiff, G. (2012) · Journal of Gastrointestinal Surgery · doi:10.1007/s11605-012-1847-0 · Cited by 144
Neal, S., Mahendra, S., Bose, K., Camacho, A. V., Mathai, M., Nove, A., de Santana, F. M., & Matthews, Z. (2016) · BMC Pregnancy and Childbirth · doi:10.1186/s12884-016-1120-8 · Cited by 144
“The authors made an assessment of the quality of each paper and risk of bias based on a checklist tool which drew on existing literature on the evaluation of crosssectional and observational studies [20, 21].”
MethodWondemagegn, A. T., Alebel, A., Leshargie, C. T., & Liyew, W. A. (2018) · Public health reviews · doi:10.1186/s40985-018-0110-y · Cited by 144
Ciapponi, A., Bardach, A., Glujovsky, D., Gibbons, L., & Picconi, M. A. (2011) · PLoS ONE · doi:10.1371/journal.pone.0025493 · Cited by 143
“stated in STROBE [19] (Strengthening the Reporting of Observational studies in Epidemiology) statement, two methodological papers [20,21] and the general guidelines of MOOSE [15].”
BackgroundDepression in the elderly with visual impairment and its association with quality of life
Renaud, J., & Bédard, E. (2013) · Clinical Interventions in Aging · doi:10.2147/cia.s27717 · Cited by 121
“Evaluation of methodological quality of studies There are numerous tools available to assess the methodological quality of studies.(33) Critical elements considered (eg, susceptibility to bias, confounding as well as the number and weighting of items) can vary widely among tools.”
Myopia and Age-Related Cataract: A Systematic Review and Meta-analysis
Pan, C., Cheng, C., Saw, S., Wang, J. J., & Wong, T. Y. (2013) · American Journal of Ophthalmology · doi:10.1016/j.ajo.2013.06.005 · Cited by 141
Wickens, S., Bowden, S. C., & D'Souza, W. J. (2017) · Epilepsia · doi:10.1111/epi.13865 · Cited by 141
“Assessment of methodological quality No gold standard assessment of methodological study quality for observational research currently exists.(46) The current study adopted the Newcastle–Ottawa Scale (NOS) as a tool to quantify the methodological quality of included studies.”
MethodBlöchl, M., Meissner, S., & Nestler, S. (2019) · Journal of Affective Disorders · doi:10.1016/j.jad.2018.12.082 · Cited by 117
Ogourtsova, T., Souza Silva, W., Archambault, P. S., & Lamontagne, A. (2017) · Neuropsychological Rehabilitation · doi:10.1080/09602011.2015.1113187 · Cited by 117
“Standards for evaluating the evidence of validity and responsiveness (McDowell, 1996; Salter et al., 2005; Sanderson et al., 2007): .”
BackgroundPark, L., Zeng, I. S. L., & Brainard, A. H. (2016) · Emergency Medicine Australasia · doi:10.1111/1742-6723.12704 · Cited by 140
Barley, E. A., Murray, J., Walters, P., & Tylee, A. (2011) · BMC Family Practice · doi:10.1186/1471-2296-12-47 · Cited by 110
“As there is no established instrument for quantitative observational studies,[20] we devised a simple checklist based on the STROBE statement [21] and a recent review of tools to assess bias in observational studies [ 22 ] (Appendix 2). Agreement was measured using Cohen’s Kappa weighted for closeness of scores.”
MethodPrevalence of Myofascial Trigger Points in Spinal Disorders: A Systematic Review and Meta-Analysis
Chiarotto, A., Clijsen, R., Fernández‐de‐las‐Peñas, C., & Barbero, M. (2015) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2015.09.021 · Cited by 139
Cannabis exposure and risk of testicular cancer: a systematic review and meta-analysis
Gurney, J., Shaw, C., Stanley, J., Signal, V., & Sarfati, D. (2015) · BMC Cancer · doi:10.1186/s12885-015-1905-6 · Cited by 137
“In the absence of such a gold standard, it has been recommended that any tools used to measure study quality should be as specific as possible to the given topic, and involve a simple checklist as opposed to a scale or score [14].”
MethodEmpirical research in clinical supervision: a systematic review and suggestions for future studies
Kühne, F., Maas, J., Wiesenthal, S., & Weck, F. (2019) · BMC Psychology · doi:10.1186/s40359-019-0327-7 · Cited by 137
Are youth sexting rates still on the rise? A meta-analytic update
Mori, C., Park, J., Temple, J. R., & Madigan, S. (2022) · Journal of Adolescent Health · doi:10.1016/j.jadohealth.2021.10.026 · Cited by 136
Tak, L. M., Riese, H., de Bock, G. H., Andiappan, M., Kok, I. C., & Rosmalen, J. G. M. (2009) · Biological Psychology · doi:10.1016/j.biopsycho.2009.05.002 · Cited by 137
“Recently, this lack of a single tool for assessing quality of observational epidemiological studies has been issued (Sanderson et al., 2007).”
BackgroundMethodKuijpers, M. A. R., Chiu, Y., Nada, R. M., Carels, C. E. L., & Fudalej, P. S. (2014) · PLoS ONE · doi:10.1371/journal.pone.0093442 · Cited by 136
“Yet, we decided to use this scoring system for the assessment of methodological quality of non-randomized studies [162] as there is no other obvious candidate for assessing these type of studies [162].”
Kastner, M., Harada, L., & Waserman, S. (2009) · Allergy · doi:10.1111/j.1398-9995.2009.02294.x · Cited by 137
“Epidemiological studies (cohort, case–control, and cross-sectional) were assessed using critical appraisal factors according to the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement (15), because there is no current reliable tool available to assess the quality of observational st…”
MethodKong, F. Y. S., Tabrizi, S. N., Fairley, C. K., Vodstrcil, L. A., Huston, W. M., Chen, M. Y., Bradshaw, C. S., & Hocking, J. S. (2015) · Journal of Antimicrobial Chemotherapy · doi:10.1093/jac/dku574 · Cited by 136
Hyperlipidemia, Blood Lipid Level, and the Risk of Glaucoma: A Meta-Analysis
Wang, S., & Bao, X. (2019) · Investigative Ophthalmology & Visual Science · doi:10.1167/iovs.18-25845 · Cited by 136
“The study quality was assessed by two reviewers (S.W. and X.B.) using the methods described by Sanderson et al. 12 and Viswanathan et al. 13 This quality assessment method contains 15 items involving criteria and evaluation of design and data analysis for observational studies.”
Burden of Heart Failure in Latin America: A Systematic Review and Meta-analysis
Ciapponi, A., Alcaraz, A. O., Calderón, M., Matta, M. G., Chaparro, M., Soto, N., & Bardach, A. (2016) · Revista Española de Cardiología (English Edition) · doi:10.1016/j.rec.2016.04.054 · Cited by 135
Ricci-Cabello, I., Ruiz-Pérez, I., De Labry-Lima, A. O., & Márquez-Calderón, S. (2010) · Health & Social Care in the Community · doi:10.1111/j.1365-2524.2010.00960.x · Cited by 120
Griffin, D. W., Harmon, D. C., & Kennedy, N. M. (2012) · Physiotherapy · doi:10.1016/j.physio.2011.04.350 · Cited by 101
Stark, L., & Landis, D. (2016) · Social Science & Medicine · doi:10.1016/j.socscimed.2016.01.052 · Cited by 133
“The authors used common quality metrics to assess the relative strength of included studies, based on similar approaches that have been used in other reviews (Loney et al., 1998; Mallen et al., 2006; Sanderson et al., 2007; Vu et al., 2014).”
MethodBranjerdporn, G., Meredith, P., Strong, J., & Garcia, J. (2017) · Maternal and Child Health Journal · doi:10.1007/s10995-016-2138-2 · Cited by 106
“The STROBE Statement is a comprehensive tool that is designed to critically appraise observational studies, including cohort, case– control, and cross-sectional designs (Sanderson et al. 2007).”
MethodExercise-induced biochemical changes and their potential influence on cancer: a scientific review
Thomas, R., Kenfield, S. A., & Jiménez, A. (2016) · British Journal of Sports Medicine · doi:10.1136/bjsports-2016-096343 · Cited by 134
“27 Three authors independently assessed their appropriateness for inclusion in this review according to guidelines suggested by Sanderson et al 28 and excluded studies with inappropriate selection of participants; inappropriate measurement of variables and controls or where not written in English.”
MethodGrenier, S., Payette, M., Gunther, B., Askari, S., Desjardins, F., Raymond, B., & Berbiche, D. (2018) · International Journal of Geriatric Psychiatry · doi:10.1002/gps.5035 · Cited by 135
“A discussion of quality is provided instead of numeric scores since there is no consensus on which tool to use and numeric scores may vary from one tool to another for the same study.(13) Quality was assessed independently by two researchers (MCP and BG).”
MethodRisk of Psychosis Among Refugees: A Systematic Review and Meta-analysis
Brandt, L., Henssler, J., Müller, M., Wall, S., Gabel, D., & Heinz, A. (2019) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2019.1937 · Cited by 111
Timmerman, L., Stronks, D. L., Groeneweg, J. G., & Huygen, F. J. (2016) · Acta Anaesthesiologica Scandinavica · doi:10.1111/aas.12697 · Cited by 132
“The methodological quality of the eligible studies was assessed at study level using an assessment list based on recommendations from Sanderson, Tatt, and Higgins.(17) This quality assessment checklist has been designed for use in observational adherence research and contains 11 items concerning selection methods, mea…”
MethodGandhi, K., Tofani, C. J., Sokach, C. E., Patel, D. B., Kastenberg, D. M., & Daskalakis, C. (2017) · Clinical Gastroenterology and Hepatology · doi:10.1016/j.cgh.2017.08.016 · Cited by 132
“There is currently no generally acceptable tool for the assessment of the quality of observational studies.(13,14) Empiric studies have shown that quality scoring scales do not correlate with study results,(15,16) and even can introduce bias.”
Lieber, M., Chin-Hong, P., Kelly, K., Dandu, M., & Weiser, S. D. (2022) · Global Public Health · doi:10.1080/17441692.2020.1860247 · Cited by 94
“We assessed the quality of each study using the NIH tool for Observational Cohort and Cross-Sectional Studies (Sanderson et al., 2007).”
MethodSegal, J. P., Ding, N. S., Worley, G., McLaughlin, S. D., Preston, S., Faiz, O. D., Clark, S. K., & Hart, A. L. (2016) · Alimentary Pharmacology & Therapeutics · doi:10.1111/apt.13905 · Cited by 131
“The Risk Of Bias In Non-randomised Studies – of Interventions (ROBINS-I) assessment tool was used to assess bias in nonrandomised controlled studies.(14) Assessment of bias was judged as low bias, moderate bias, serious bias, critical bias or no information.”
MethodPanagioti, M., Gooding, P. A., Triantafyllou, K., & Tarrier, N. (2014) · Social Psychiatry and Psychiatric Epidemiology · doi:10.1007/s00127-014-0978-x · Cited by 131
Grant, S., Bin, Y. S., Kiat, H., & Chang, D. H.-T. (2012) · BMC Public Health · doi:10.1186/1471-2458-12-299 · Cited by 131
“There is a paucity of methods for assessing the quality of observational studies [36,37].”
MethodSleep duration and metabolic syndrome in adult populations: a meta-analysis of observational studies
Ju, S.-Y., & Choi, W.-S. (2013) · Nutrition & Diabetes · doi:10.1038/nutd.2013.8 · Cited by 130
Ngamwong, Y., Tangamornsuksan, W., Lohitnavy, O., Chaiyakunapruk, N., Scholfield, C. N., Reisfeld, B., & Lohitnavy, M. (2015) · PLOS ONE · doi:10.1371/journal.pone.0135798 · Cited by 115
“The categories of NOS was based on selection of participants, comparability of study groups, and the exposure of interest (case-control studies) or outcome of interest (cohort studies) [36].”
MethodBeyond genomics: understanding exposotypes through metabolomics
Rattray, N. J. W., Deziel, N. C., Wallach, J. D., Khan, S. A., Vasiliou, V., Ioannidis, J. P. A., & Johnson, C. H. (2018) · Human Genomics · doi:10.1186/s40246-018-0134-x · Cited by 131
Greenwell, K., Gray, W. K., van Wersch, A., van Schaik, P., & Walker, R. W. (2014) · Parkinsonism & Related Disorders · doi:10.1016/j.parkreldis.2014.10.013 · Cited by 130
“Finally, although the quality assessment tool has not been validated for use in cross-sectional studies of predictors, it was chosen based onwidely-accepted quality criteria [64] and offered themost useful tool for assessing risk of bias as part of a narrative review.”
MethodMedical Respite Programs for Homeless Patients: A Systematic Review
Doran, K. M., Ragins, K. T., Gross, C. P., & Zerger, S. (2013) · Journal of Health Care for the Poor and Underserved · doi:10.1353/hpu.2013.0053 · Cited by 129
Taouk, Y., Spittal, M. J., LaMontagne, A. D., & Milner, A. J. (2019) · Scandinavian Journal of Work Environment & Health · doi:10.5271/sjweh.3854 · Cited by 128
“Assessment of study quality was conducted using a rigorously validated scoring tool recognized as a useful assessment tool for assessing quality and susceptibility to bias in observational studies (58, 85).”
MethodDo informal caregivers experience more burnout? A meta-analytic study
Gérain, P., & Zech, E. (2020) · Psychology Health & Medicine · doi:10.1080/13548506.2020.1803372 · Cited by 132
Edmonds, C. J., Foglia, E., Booth, P., Fu, C. H. Y., & Gardner, M. R. (2021) · Archives of Gerontology and Geriatrics · doi:10.1016/j.archger.2021.104380 · Cited by 128
Su, Y., Xu, S., Ma, Y., Yin, P., Hao, X., Zhou, J., Fung, W. K., Jiang, H., & Wang, P. (2024) · Statistics in Medicine · doi:10.1002/sim.10245 · Cited by 128
“However, one of the major challenges encountered in observational studies is unmeasured confounding [1, 2], which cannot be easily adjusted FIGURE 1 | Illustrative diagram shows the core IV assumptions for thegeneticvariant 𝑍 𝑗 (solidlines)withpotentialviolations(dashedlines).”
Fernando, M. E., Crowther, R. G., Pappas, E., Lazzarini, P. A., Cunningham, M., Sangla, K. S., Buttner, P. G., & Golledge, J. (2014) · PLoS ONE · doi:10.1371/journal.pone.0099050 · Cited by 127
Physical and psychosocial risk factors for musculoskeletal disorders in Brazilian and Italian nurses
Carugno, M., Pesatori, A. C., Ferrario, M., Ferrari, A. L., da Silva, F. J., Martins, A. C., Felli, V. E. A., Coggon, D., & Bonzini, M. (2012) · Cadernos de Saúde Pública · doi:10.1590/s0102-311x2012000900003 · Cited by 127
Katikireddi, S. V., Egan, M., & Petticrew, M. (2015) · Journal of Epidemiology and Community Health · doi:10.1136/jech-2014-204711 · Cited by 126
Subjective Cognitive Complaints after Stroke: A Systematic Review
van Rijsbergen, M. W. A., Mark, R. E., de Kort, P. L. M., & Sitskoorn, M. M. (2013) · Journal of Stroke and Cerebrovascular Diseases · doi:10.1016/j.jstrokecerebrovasdis.2013.05.003 · Cited by 126
“We devised our own tool for this review because an internationally accepted instrument for assessing the quality of observational epidemiological studies does not currently exist.(5) Disagreement between the raters about eligibility and/or quality was solved by discussion.”
MethodMiller, M. M., Meints, S. M., & Hirsh, A. T. (2018) · Pain · doi:10.1097/j.pain.0000000000001342 · Cited by 126
“Studies were assessed for common sources of bias in observational studies.(41,91) Many items on the typical quality rubrics used for meta-analyses are not relevant for correlational designs.”
Marušić, M. F., Fidahić, M., Cepeha, C. M., Farcaș, L. G., Tseke, A., & Puljak, L. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-00966-4 · Cited by 126
Hepworth, L. R., Rowe, F. J., Walker, M. F., Rockliffe, J., Noonan, C. P., Howard, C., & Currie, J. (2016) · Ophthalmology Research An International Journal · doi:10.9734/or/2016/21767 · Cited by 124
“There is currently no ‘gold standard’ quality assessment tool for observational studies [28].”
BackgroundThompson, E. L., Vamos, C. A., & Daley, E. M. (2015) · Journal of sport and health science/Journal of Sport and Health Science · doi:10.1016/j.jshs.2015.08.001 · Cited by 124
“However, given that this review integrates both qualitative and quantitative observational methodologies, this prohibits a single, standardized evaluation tool for quality assessment.(16,17)”
BackgroundBlank, L., Coster, J., O’Cathain, A., Knowles, E., Tosh, J., Turner, J., & Nicholl, J. (2012) · Journal of Advanced Nursing · doi:10.1111/j.1365-2648.2012.06052.x · Cited by 108
“Of 54 papers included in this review, 52 were observational studies and therefore, we assessed their quality using factors commonly considered in the appraisal of observational studies (Sanderson et al. 2007).”
MethodPrevalence of heart failure in Australia: a systematic review
Sahle, B. W., Owen, A. J., Mutowo, M. P., Krum, H., & Reid, C. M. (2016) · BMC Cardiovascular Disorders · doi:10.1186/s12872-016-0208-4 · Cited by 123
“[13].”
Tsai, A. C. (2015) · PLOS ONE · doi:10.1371/journal.pone.0123182 · Cited by 122
“To assess the quality of the included quantitative studies, I developed an assessment tool based on major conceptual domains identified by Sanderson and colleagues [28].”
MethodPotential links between health literacy and cervical cancer screening behaviors: a systematic review
Kim, K., & Han, H. (2015) · Psycho-Oncology · doi:10.1002/pon.3883 · Cited by 122
“While there is a lack of consensus regarding ways to critically appraise observational studies [34], the STrengthening the Reporting of OBservational studies in Epidemiology (STROBE) guidelines were used to evaluate the methodological rigor of each study included in the review [35].”
MethodFang, X., Han, H., Li, M., Liang, C., Fan, Z., Aaseth, J., He, J., Montgomery, S., & Cao, Y. (2016) · Nutrients · doi:10.3390/nu8110739 · Cited by 106
“The studies were evaluated and scored based on the guidelines adapted from the tools for assessing quality and susceptibility to bias in observational studies in epidemiology [31].”
MethodImpact of Specialized Pediatric Palliative Care: A Systematic Review
Marcus, K. L., Santos, G., Ciapponi, A., Comandé, D., Bilodeau, M., Wolfe, J., & Dussel, V. (2019) · Journal of Pain and Symptom Management · doi:10.1016/j.jpainsymman.2019.08.005 · Cited by 122
“(20) As such, we hope that these results will assist the design and conduct of future palliative care research in seriously ill children and their families.”
BackgroundFrampton, G. K., Whaley, P., Bennett, M. G., Bilotta, G. S., Dorne, J. L. C. M., Eales, J. F., James, K. L., Kohl, C., Land, M., Livoreil, B., et al. (2022) · Environmental Evidence · doi:10.1186/s13750-022-00264-0 · Cited by 122
“Risk of bias tools Many tools exist for critically appraising studies, covering a range of experimental and observational study designs [32, 34, 35, 56, 57].”
Does trauma in the primary dentition cause sequelae in permanent successors? A systematic review
Lenzi, M. M., Alexandria, A. K., Ferreira, D. M. T. P., & Maia, L. C. (2014) · Dental Traumatology · doi:10.1111/edt.12149 · Cited by 122
Sargeant, J. M., O'Connor, A. M. C., Dohoo, I. R., Erb, H. N., Cevallos, M., Egger, M., Ersbøll, A. K., Martin, S. W., Nielsen, L. R., Pearl, D. L., et al. (2016) · Preventive Veterinary Medicine · doi:10.1016/j.prevetmed.2016.09.005 · Cited by 121
Brborović, H., Daka, Q., Dakaj, K., & Brborović, O. (2017) · International Journal of Nursing Practice · doi:10.1111/ijn.12598 · Cited by 121
Prenatal Alcohol Exposure, FASD, and Child Behavior: A Meta-analysis
Tsang, T. W., Lucas, B. R., Carmichael Olson, H., Pinto, R. Z., & Elliott, E. J. (2016) · Pediatrics · doi:10.1542/peds.2015-2542 · Cited by 106
Workplace risk factors for anxiety and depression in male-dominated industries: a systematic review
Battams, S., Roche, A. M., Fischer, J. A., Lee, N. K., Cameron, J., & Kostadinov, V. (2014) · Health Psychology and Behavioral Medicine · doi:10.1080/21642850.2014.954579 · Cited by 92
Manchikanti, L., Singh, V., Smith, H. S., & Hirsch, J. A. (2009) · Pain Physician · doi:10.36076/ppj.2009/12/73 · Cited by 120
Treatment of Polymyalgia Rheumatica: A Systematic Review
Hernández-Rodríguez, J. (2009) · Archives of Internal Medicine · doi:10.1001/archinternmed.2009.352 · Cited by 118
Solmi, M., Veronese, N., Sergi, G., Luchini, C., Favaro, A., Santonastaso, P., Vancampfort, D., Correll, C. U., USSHER, M. H., Thapa‐Chhetri, N., et al. (2016) · Addiction · doi:10.1111/add.13457 · Cited by 118
Systematic review: interventions for abdominal pain management in inflammatory bowel disease
Norton, C. S., Czuber‐Dochan, W., Artom, M., Sweeney, L., & Hart, A. L. (2017) · Alimentary Pharmacology & Therapeutics · doi:10.1111/apt.14108 · Cited by 118
“ological design of each study (RCTs and cross-sectional studies) were utilised to assess the studies’ quality.(42) Two researchers (CN and MA or LS) appraised the studies independently and then agreed the final scores.”
Dunn, K., Hughes-McCormack, L. A., & Cooper, S. (2017) · Journal of Applied Research in Intellectual Disabilities · doi:10.1111/jar.12360 · Cited by 118
Childbearing attitudes and decisions of young breast cancer survivors: a systematic review
Gonçalves, V., Sehovic, I., & Quinn, G. P. (2013) · Human Reproduction Update · doi:10.1093/humupd/dmt039 · Cited by 116
“The criteria used to methodologically appraise the studies were based on those suggested in the literature on quantitative research (Sanderson et al., 2007; National Collaborating Centre for Methods and Tools, 2008; Jack et al., 2010).”
MethodKumar, S. P., Tadakamadla, J., Kroon, J., & Johnson, N. W. (2015) · Journal of Dentistry · doi:10.1016/j.jdent.2015.12.007 · Cited by 114
Inflammation and depression in young people: a systematic review and proposed inflammatory pathways
Toenders, Y. J., Laskaris, L., Davey, C. G., Berk, M., Milaneschi, Y., Lamers, F., Penninx, B. W. J. H., & Schmaal, L. (2021) · Molecular Psychiatry · doi:10.1038/s41380-021-01306-8 · Cited by 114
Breastfeeding Skills Training for Health Care Professionals: A Systematic Review
Mulcahy, H., Philpott, L. F., O’Driscoll, M., Bradley, R., & Leahy‐Warren, P. (2021) · Heliyon · doi:10.1016/j.heliyon.2022.e11747 · Cited by 114
“These criteria are selection bias; measurement bias; design specific bias; confounding bias; statistical method bias; and conflict of interest or funding sources (Sanderson et al., 2007).”
BackgroundEaster, G., Sharpe, L., & J. Hunt, C. (2015) · Journal of Pediatric Psychology · doi:10.1093/jpepsy/jsv012 · Cited by 105
Social inequality and infant health in the UK: systematic review and meta-analyses
Weightman, A. L., Morgan, H. E., Shepherd, M. A., Kitcher, H., Roberts, C., & Dunstan, F. D. (2012) · BMJ Open · doi:10.1136/bmjopen-2012-000964 · Cited by 113
Liperoti, R., Onder, G., Landi, F., Lapane, K. L., Mor, V., Bernabei, R., & Gambassi, G. (2009) · The Journal of Clinical Psychiatry · doi:10.4088/jcp.08m04597yel · Cited by 103
Morris, N., Moghaddam, N. G., Tickle, A., & Biswas, S. (2017) · Psycho-Oncology · doi:10.1002/pon.4509 · Cited by 114
“Measures should be specific and relevant to the study design and question.(51) Two authors (N.”
BackgroundAnxiety in fathers in the perinatal period: A systematic review
Philpott, L. F., Savage, E., FitzGerald, S., & Leahy-Warren, P. (2019) · Midwifery · doi:10.1016/j.midw.2019.05.013 · Cited by 112
Systematic review: psychosocial factors associated with pain in inflammatory bowel disease
Sweeney, L., Moss‐Morris, R., Czuber‐Dochan, W., Meade, L., Chumbley, G., & Norton, C. (2018) · Alimentary Pharmacology & Therapeutics · doi:10.1111/apt.14493 · Cited by 112
“Methodological quality of studies was assessed using the Critical Appraisal Skills Programme (CASP) guidelines, selected by the specific methodological design of included studies.(37) The same criteria have been applied in previous reviews on IBD populations.”
Ciapponi, A., Alcaraz, A. O., Calderón, M., Matta, M. G., Chaparro, M., Soto, N., & Bardach, A. (2016) · Revista Española de Cardiología · doi:10.1016/j.recesp.2016.04.045 · Cited by 112
Three risk of bias tools lead to opposite conclusions in observational research synthesis
Losilla, J.-M., Oliveras, I., Marin-Garcia, J. A., & Vives, J. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.05.021 · Cited by 111
Tools used to assess the quality of peer review reports: A methodological systematic review
Superchi, C., González, J. A., Solà, I., Cobo, E., Hren, D., & Boutron, I. (2019) · BMC Medical Research Methodology · doi:10.1186/s12874-019-0688-x · Cited by 128
Preschool predictors of later reading comprehension ability: a systematic review
Hjetland, H. N., Brinchmann, E. I., Scherer, R., & Melby‐Lervåg, M. (2017) · Campbell Systematic Reviews · doi:10.4073/csr.2017.14 · Cited by 112
Inferring pregnancy episodes and outcomes within a network of observational databases
Matcho, A., Ryan, P. B., Fife, D., Gifkins, D., Knoll, C., & Friedman, A. (2018) · PLoS ONE · doi:10.1371/journal.pone.0192033 · Cited by 110
McAllister, A., Fritzell, S. C., Almroth, M., Harber-Aschan, L., Larsson, S., & Burström, B. (2018) · International Journal for Equity in Health · doi:10.1186/s12939-018-0879-9 · Cited by 110
Correlates of Parental Support of Child and Youth Physical Activity: a Systematic Review
Rhodes, R. E., Perdew, M., & Malli, S. (2020) · International Journal of Behavioral Medicine · doi:10.1007/s12529-020-09909-1 · Cited by 111
Prevalence, Related Factors, and Levels of Burnout Among Midwives: A Systematic Review
Albendín‐García, L., Suleiman‐Martos, N., la Fuente, G. A. C., Ramírez‐Baena, L., Gómez‐Urquiza, J. L., & Fuente-Solana, E. I. D. L. (2021) · Journal of Midwifery & Women s Health · doi:10.1111/jmwh.13186 · Cited by 110
Sarcopenia as Prognostic Factor in Lung Cancer Patients: A Systematic Review and Meta-analysis
Büntzel, J., Heinz, J., Bleckmann, A., Bauer‐Büntzel, C., Röver, C., Bohnenberger, H., Saha, S., Hinterthaner, M., Baraki, H., Kutschka, I., et al. (2019) · Anticancer Research · doi:10.21873/anticanres.13640 · Cited by 109
Touboul-Lundgren, P., Jensen, S., Drai, J., & Lindbæk, M. (2015) · BMC Public Health · doi:10.1186/s12889-015-2254-8 · Cited by 108
“Thus, the following tools were used according to the defined criteria [16]: the NICE recommendation tools for both the correlation and qualitative studies (score 2+: All or most of the checklist criteria have been fulfilled, where they have not been fulfilled, the conclusions are very unlikely to alter/ + / -); the Ef…”
microRNAs with prognostic significance in pancreatic ductal adenocarcinoma: A meta-analysis
Frampton, A. E., Krell, J., Jamieson, N. B., Gall, T., Giovannetti, E., Funel, N., Prado, M. M., Krell, D., Habib, N., Castellano, L., et al. (2015) · European Journal of Cancer · doi:10.1016/j.ejca.2015.04.006 · Cited by 108
“There are various tools for the quality assessment and identification of bias in observational studies [83].”
MethodMaterial Needs of Emergency Department Patients: A Systematic Review
Malecha, P. W., Williams, J. H., Kunzler, N. M., Goldfrank, L. R., Alter, H., & Doran, K. M. (2017) · Academic Emergency Medicine · doi:10.1111/acem.13370 · Cited by 108
Lamerton, T. J., Torquati, L., & Brown, W. J. (2018) · Obesity Reviews · doi:10.1111/obr.12756 · Cited by 108
“An extensive systematic review of tools for assessing quality and susceptibility in observational studies indicated that the Newcastle-Ottawa scale was appropriate for future use (19).”
BackgroundThe use of commercial food purchase data for public health nutrition research: A systematic review
Bandy, L., Adhikari, V., Jebb, S. A., & Rayner, M. (2019) · PLoS ONE · doi:10.1371/journal.pone.0210192 · Cited by 109
Asperen, R. M. W., van Gestel, J. P. J., van Grotel, M., Tschiedel, E., Dohna‐Schwake, C., Valla, F. V., Willems, J., Nielsen, J. S. A., Krause, M., Potratz, J. C., et al. (2019) · Critical Reviews in Oncology/Hematology · doi:10.1016/j.critrevonc.2019.07.014 · Cited by 108
“(Viswanathan et al., 2019; Sanderson et al., 2007; Vandenbroucke et al., 2007) Each item was scored as high quality (1 point) or low quality (0 points).”
Dreyer, N. A., Velentgas, P., Westrich, K., & Dubois, R. (2014) · Journal of Managed Care Pharmacy · doi:10.18553/jmcp.2014.20.3.301 · Cited by 107
Metadata
- Authors
- Sanderson, S., Tatt, I. D., & Higgins, J. P.
- Affiliations
- University of Cambridge (GB)
- F. Hoffmann-La Roche AG (CH)
- MRC Biostatistics Unit (GB)
- Corresponding author
- Sanderson, S.
- ORCID
- Higgins, J. P.
- Venue
- International Journal of Epidemiology
- Publisher
- Oxford University Press (OUP)
- Published
- 2007-04-30
- Volume
- 36
- Issue
- 3
- Pages
- 666-676
- ISSN
- 0300-5771 (ISSN-L)
- 1464-3685
- OpenAlex
- W2171251365
- Cited by
- 1,531
- References
- 73
- Citations by year
- 2026: 22 · 2025: 38 · 2024: 42 · 2023: 59 · 2022: 80 · 2021: 96 · 2020: 101 · 2019: 105 · 2018: 125 · 2017: 104 · 2016: 150 · 2015: 118 · 2014: 122 · 2013: 111 · 2012: 96
- FWCI
- 7.09
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- International Journal of Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- quality assessment tools, bias assessment, observational studies, critical appraisal, checklists, rating scales, STROBE, confounding adjustment, selection bias, measurement bias, conflicts of interest
- MeSH terms
- Humans, Research Design, Bias, Epidemiologic Studies, Evidence-Based Medicine, Observation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.