Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography
The DOI matches its Crossref record; 73 of the 73 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Assessing quality and susceptibility to bias is essential when interpreting primary research and conducting systematic reviews and meta-analyses. Tools for assessing quality in clinical trials are well-described but much less attention has been given to similar tools for observational epidemiological studies. METHODS: Tools were identified from a search of three electronic databases, bibliographies and an Internet search using Google. Two reviewers extracted data using a pre-piloted extraction form and strict inclusion criteria. Tool content was evaluated for domains potentially related to bias and was informed by the STROBE guidelines for reporting observational epidemiological studies. RESULTS: A total of 86 tools were reviewed, comprising 41 simple checklists, 12 checklists with additional summary judgements and 33 scales. The number of items ranged from 3 to 36 (mean 13.7). One-third of tools were designed for single use in a specific review and one-third for critical appraisal. Half of the tools provided development details, although most were proposed for future use in other contexts. Most tools included items for selection methods (92%), measurement of study variables (86%), design-specific sources of bias (86%), control of confounding (78%) and use of statistics (78%); only 4% addressed conflict of interest. The distribution and weighting of domains across tools was variable and inconsistent. CONCLUSION: A number of useful assessment tools have been identified by this report. Tools should be rigorously developed, evidence-based, valid, reliable and easy to use. There is a need to agree on critical elements for assessing susceptibility to bias in observational epidemiology and to develop appropriate evaluation tools.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Such instruments have been developed by other groups and were the subject of a recent systematic review [28].”
From The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: Guidelines for Reporting Observational Studies (2007) · cites this paper as [28]
In the citing paper
“It is not clear how weights should be assigned to each item of the scale [18].”
From Tools used to assess the quality of peer review reports: A methodological systematic review (2019) · cites this paper as [18]
In the citing paper
“The reviewers concluded that there is no ‘single obvious candidate tool for assessing quality of observational epidemiological studies’.”
From COSMOS-E: Guidance on conducting systematic reviews and meta-analyses of observational studies of etiology (2019) · cites this paper as [45]
In the citing paper
“A review identified more than 80 tools for assessing risk of bias in nonrandomised studies [45].”
From COSMOS-E: Guidance on conducting systematic reviews and meta-analyses of observational studies of etiology (2019) · cites this paper as [45]
In the citing paper
“The need for complete reporting of all relevant methodological aspects as a prerequisite to assessing the risk of bias of studies was highlighted in 10 articles, [16, 33, 35, 47, 50, 56, 86, 87, 90, 91] and was the focus of recommendations in a publication from the Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group [12].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [90]
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Meta-analysis of Observational Studies in Epidemiology<SUBTITLE>A Proposal for Reporting</SUBTITLE>
Stroup, D. F. (2000) · JAMA · doi:10.1001/jama.283.15.2008 · Cited by 20,623
Downs, S. H., & Black, N. (1998) · Journal of Epidemiology and Community Health · doi:10.1136/jech.52.6.377 · Cited by 8,252
Moher, D., Cook, D. J., Eastwood, S., Olkin, I., Rennie, D., & Stroup, D. F. (1999) · The Lancet · doi:10.1016/s0140-6736(99)04149-5 · Cited by 4,826
Whiting, P., Rutjes, A. W., Reitsma, J. B., Bossuyt, P. M., & Kleijnen, J. (2003) · BMC Medical Research Methodology · doi:10.1186/1471-2288-3-25 · Cited by 3,592
Assessing the quality of controlled clinical trials
Jüni, P., Altman, D. G., & Egger, M. (2001) · BMJ · doi:10.1136/bmj.323.7303.42 · Cited by 3,309 · cited 5×
“a (5) (5) (5) ( 5 ) N N LongneckerCoh”
InfluentialUsers' guides to the medical literature
Guyatt, G. H. (1993) · JAMA · doi:10.1001/jama.270.17.2096 · Cited by 2,970
Evaluating non-randomised intervention studies
Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D. (2003) · Health Technology Assessment · doi:10.3310/hta7270 · Cited by 2,907
Cognitive and Behavioral Outcomes of School-Aged Children Who Were Born Preterm: A Meta-analysis
Bhutta, A. T., Cleves, M. A., Casey, P. H., Cradock, M. M., & Anand, K. J. S. (2002) · JAMA · doi:10.1001/jama.288.6.728 · Cited by 2,392 · cited 1×
“Y N N 9 (22) 2 (17) 5 (15) 16 (19) Y Y N 15 (36) 6 ( 50 ) 7 (21) 28 (32)”
The Hazards of Scoring the Quality of Clinical Trials for Meta-analysis
Jüni, P. (1999) · JAMA · doi:10.1001/jama.282.11.1054 · Cited by 1,981
Low grade inflammation and coronary heart disease: prospective study and updated meta-analyses
Danesh, J. (2000) · BMJ · doi:10.1136/bmj.321.7255.199 · Cited by 1,692
Research works printed but not linked: 7
- Carneiro, 2002, Revista portuguesa de cardiologia : org o oficial da Sociedade Portuguesa de Cardiologia = Portuguese journal of cardiology : an official journal of the Portuguese Society of Cardiology, 21, 891
- 1981, Canadian Medical Association Journal, 124, 985
- 2002, SOUTH AFRICAN J CLIN NUTR, 15, 68
- 2000, Chronic diseases in Canada, 19, 170
- Ciliska, 1996, Canadian journal of public health. Revue canadienne de sant publique, 87, 193
- 1994, Canadian journal of public health. Revue canadienne de sant publique, 85, S8
- Spitzer, 1990, Clinical and investigative medicine. M decine clinique et experimentale, 13, 17
Cited by
Items 801–900 of 1,006
Methodology of the Global and Regional Burden of Stroke Study
(2012) · Neuroepidemiology · doi:10.1159/000334744 · Cited by 39
Daniel, S., Koren, G., Lunenfeld, E., & Levy, A. (2014) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2014.09.028 · Cited by 39
Badovinac, S. D., Martin, J., Guérin‐Marion, C., O’Neill, M. C., Riddell, R. P., Bureau, J., & Spiegel, R. (2018) · PLoS ONE · doi:10.1371/journal.pone.0204374 · Cited by 39
Posadzki, P., Watson, L. K., Alotaibi, A., & Ernst, E. (2013) · Complementary Therapies in Medicine · doi:10.1016/j.ctim.2012.11.006 · Cited by 39
Funding Source and Research Report Quality in Nutrition Practice-Related Research
Myers, E. F., Parrott, J. S., Cummins, D. S., & Splett, P. L. (2011) · PLoS ONE · doi:10.1371/journal.pone.0028437 · Cited by 39
“A variety of instruments for appraising the quality of research reports have been developed; however, no gold standard has been identified [29–31].”
BackgroundSamargandi, O. A., Hasan, H., & Thoma, A. (2015) · Plastic & Reconstructive Surgery · doi:10.1097/prs.0000000000001898 · Cited by 38
Systematic reviews and meta-analysis in nutrition research
Kelley, G. A., & Kelley, K. S. (2019) · British Journal Of Nutrition · doi:10.1017/s0007114519002241 · Cited by 38
“There are at least eighty-six risk of bias/study quality assessment instruments(76).”
BackgroundMaurage, P., Bollen, Z., Masson, N., & D’Hondt, F. (2020) · Neuropsychology Review · doi:10.1007/s11065-020-09458-0 · Cited by 38
“As underlined in papers reviewing methodological quality assessment scales (Sanderson et al., 2007; Zeng et al., 2015), there is currently no gold-standard tool for assessing cross-sectional studies.”
BackgroundSearle, A., Spink, M. J., Ho, A., & Chuter, V. (2017) · Clinical Biomechanics · doi:10.1016/j.clinbiomech.2017.01.021 · Cited by 38
“This tool was selected as it allows use of one set of questions for all included studies, includes a small number of key domains, is a simple checklist rather than a scale and was developed using a variety of literature sources (Sanderson et al., 2007).”
MethodOsthoff, A.-K. R., Taeymans, J., Kool, J., Marcar, V. L., & van Gestel, A. J. R. (2013) · Journal of Cardiopulmonary Rehabilitation and Prevention · doi:10.1097/hcr.0000000000000022 · Cited by 37
Desmedt, M., Geest, S. D., Schubert, M., Schwendimann, R., & Ausserhofer, D. (2012) · Swiss Medical Weekly · doi:10.4414/smw.2012.13733 · Cited by 39
Psoriasis in Latin America and the Caribbean: a systematic review
Hernández‐Vásquez, A., Molinari, L. M., Larrea, N., & Ciapponi, A. (2017) · Journal of the European Academy of Dermatology and Venereology · doi:10.1111/jdv.14393 · Cited by 37
Cross, E. L. A., Jordan, H., Godfrey, R., Onakpoya, I., Shears, A., Fidler, K., Peto, T., Walker, A. S., & Llewelyn, M. (2020) · Journal of Infection · doi:10.1016/j.jinf.2020.07.030 · Cited by 37
Calibration of credibility of agnostic genome‐wide associations
Ioannidis, J. P. A. (2008) · American Journal of Medical Genetics Part B: Neuropsychiatric Genetics · doi:10.1002/ajmg.b.30721 · Cited by 32
“An extensive number of checklists exist for biases in observational epidemiology [Sanderson et al., 2007].”
BackgroundHypovitaminosis D and orthostatic hypotension: a systematic review and meta-analysis
Ometto, F., Stubbs, B., Annweiler, C., Duval, G. T., Jang, W., Kim, H.-T., McCarroll, K. G., Cunningham, C. J., Soysal, P., Işık, A. T., et al. (2016) · Journal of Hypertension · doi:10.1097/hjh.0000000000000907 · Cited by 37
“As suggested by Sanderson [13], we considered five items that could be of relevance, that are, clear description of participants (eligibility, setting, and methods of participant selection); reporting data on vitamin D supplementation; reporting data on serum PTH concentrations; reporting elaborations adjusted for pot…”
MethodFurtado, S., Errington, L., Godfrey, A., Rochester, L., & Gerrand, C. (2016) · European Journal of Surgical Oncology · doi:10.1016/j.ejso.2016.10.002 · Cited by 37
Analgesia use during pregnancy and risk of cryptorchidism: a systematic review and meta-analysis
Gurney, J., Richiardi, L., McGlynn, K. A., Signal, V., & Sarfati, D. (2017) · Human Reproduction · doi:10.1093/humrep/dex047 · Cited by 37
“…of risk of bias (individual and across studies) In the absence of a gold standard, it has been recommended that any tools used to measure study quality should be as specific as possible to the given topic, and involve a simple checklist as opposed to a scale or score (Sanderson et al., 2007).”
MethodMovement Disorders and Syndromic Autism: A Systematic Review
Bell, L., Wittkowski, A., & Hare, D. J. (2018) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-018-3658-y · Cited by 37
A systematic review of early life factors which adversely affect subsequent lung function
Kouzouna, A., Gilchrist, F. J., Ball, V., Kyriacou, T., Henderson, A. J., Pandyan, A. D., & Lenney, W. (2016) · Paediatric Respiratory Reviews · doi:10.1016/j.prrv.2016.03.003 · Cited by 36
Li, Z., Ying, X., Shan, F., & Ji, J. (2018) · Helicobacter · doi:10.1111/hel.12532 · Cited by 36
Botelho, R. V., Heringer, L. C., Botelho, P. B., Lopes, R. A., & Waisberg, J. (2020) · World Neurosurgery · doi:10.1016/j.wneu.2020.02.182 · Cited by 36
Wang, F., Xu, X., Zeng, Y., Qin, Y., Yu, X., & Ji, S. (2021) · Frontiers in Endocrinology · doi:10.3389/fendo.2021.691557 · Cited by 36
Incidence and severity of neck injury in Rugby Union: A systematic review
Swain, M. S., Lystad, R. P., Pollard, H. P., & Bonello, R. (2010) · Journal of science and medicine in sport · doi:10.1016/j.jsams.2010.10.460 · Cited by 36
Fuller, T., Pearson, M., Peters, J. L., & Anderson, R. (2012) · BMJ Open · doi:10.1136/bmjopen-2012-002073 · Cited by 33
“Second, given that there is no universally agreed definition of ‘quality’, there is no ‘gold standard’ measure of ‘quality’.(14) A range of methods have been used to assess reporting completeness and study quality, and the difficulties of doing so are well documented.”
BackgroundUser experience of transtibial prosthetic liners: A systematic review
Richardson, A., & Dillon, M. P. (2016) · Prosthetics and Orthotics International · doi:10.1177/0309364616631343 · Cited by 36
“28 5 (20) 56 13 20 80 Van de Weg and Van der Windt.”
Bucknall, T. K., Hutchinson, A. M., Botti, M. A., McTier, L., Rawson, H. A., Hewitt, N. A., McMurray, A. M., Marshall, A. P., Gillespie, B. M., & Chaboyer, W. P. (2016) · Journal of Advanced Nursing · doi:10.1111/jan.12953 · Cited by 36
“Evidence to support the validity and reliability of different methodological quality tools for assessing studies in mixed study integrative reviews is equivocal (Sanderson et al. 2007, Crowe & Sheppard 2011).”
BackgroundMinozzi, S., Cinquini, M., Gianola, S., Castellini, G., Gerardi, C., & Banzi, R. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.04.001 · Cited by 35
Child maltreatment and adult economic outcomes: A systematic review
Herbert, L., Comeau, J., Smith, C., Smith-Carrier, T. A., Saxby, C., & Botelho, N. (2022) · Children and Youth Services Review · doi:10.1016/j.childyouth.2022.106711 · Cited by 35
Common neural deficits across reward functions in major depression: a meta-analysis of fMRI studies
Zhao, X., Wu, S., Li, X., Liu, Z., Lu, W., Lin, K., & Shao, R. (2024) · Psychological Medicine · doi:10.1017/s0033291724001235 · Cited by 35
Charlesworth, M., Elliott, M., & Holmes, J. D. (2012) · Lung · doi:10.1007/s00408-012-9403-y · Cited by 35
“There is no gold-standard validated method for assessing bias and the quality of observational studies [19, 24], but there is consensus that the three most fundamental domains are (1) appropriate selection of participants, (2) appropriate measurement of variables, and (3) appropriate control of confounding, as well as…”
MethodA matrix for bridging the epidemiology and risk assessment gap
Burns, C. J., LaKind, J. S., Mattison, D. R., Alcala, C. S., Branch, F. M., Castillo, J., Clark, A., Clougherty, J. E., Darney, S. P., Erickson, H. S., et al. (2019) · Global Epidemiology · doi:10.1016/j.gloepi.2019.100005 · Cited by 35
“While guidance documents and frameworks related to observational epidemiology have been proposed and reviewed (e.g., [20–22]), these have been met with resistance for a variety of reasons including their eventual obsolescence, potential for dampening of creativity, and additional burden on…”
de Lima, C., Paiva, B. S. R., Neto, M. F. D. S., Hui, D., Perez‐Cruz, P. E., Zimmermann, C., Bruera, E. D., & Paiva, C. E. (2021) · Annals of Global Health · doi:10.5334/aogh.3158 · Cited by 35
Murray, A., Toussaint, A. C., Althaus, A., & Löwe, B. (2013) · Systematic Reviews · doi:10.1186/2046-4053-2-99 · Cited by 35
“In addition, the reviewers may identify additional sources of bias which are commonly observed in the literature [46,47].”
BackgroundBoisvert, I., Tourigny, M., Lanctôt, N., & Lemieux, S. (2017) · Revue de psychoéducation · doi:10.7202/1039163ar · Cited by 35
“…qualité méthodologique s’avère un concept complexe et relativement subjectif qui varie d’un auteur à l’autre (Deeks, Dinnes, D’Amico, Sowden, Sakarovitch, Song et al., 2003), il est recommandé d’utiliser un petit nombre de critères spécifiques au devis et au sujet d’étude (Sanderson et al., 2007).”
Lau, A., Kong, F. Y. S., Huston, W. M., Chow, E. P. F., Fairley, C. K., & Hocking, J. S. (2019) · Sexually Transmitted Infections · doi:10.1136/sextrans-2018-053950 · Cited by 35
“Assessment of within-study bias was undertaken using a combination of the evaluation criteria adopted by Sanderson et al 17 and the critical appraisal tool for cross-sectional studies, AXIS, 18 and assessed the following criteria: selection bias, measurement bias, confounding, statistical methods and ethical considera…”
Tang, B., Lv, J., Li, Y., Yuan, S., Wang, Z., & He, S. (2015) · Medicine · doi:10.1097/md.0000000000000177 · Cited by 35
Gonçalves, J. P. D. B., Lucchetti, G., Maraldi, E. D. O., Fernandez, P. E. L., Menezes, P. R., & Vallada, H. (2022) · Brazilian Journal of Psychiatry · doi:10.47626/1516-4446-2022-2832 · Cited by 34
“Since there is no gold standard for quality assessment of observational studies, 37 we used a critical appraisal tool 38 to assess the risk of bias (Supplementary Material S2, available online only).”
Roesel, R., Deantonio, L., Bernardi, L., Garo, M. L., Majno-Hurst, P. E., Vannelli, A., Cefalì, M., Palmarocchi, M. C., Valli, M. C., Pesola, G., et al. (2023) · Cancers · doi:10.3390/cancers15184627 · Cited by 34
“For non-RCT studies, the risk of bias was assessed using the Newcastle-Ottawa Quality Scale (NOS) [23].”
MethodWang, M. X., Koh, J., & Pang, J. (2019) · Nutrition Journal · doi:10.1186/s12937-019-0507-6 · Cited by 34
Vestibular and balance dysfunction in children with congenital CMV: a systematic review
Shears, A., Yan, G., Mortimer, H., Cross, E. L. A., Sapuan, S., Kadambari, S., Luck, S., Heath, P. T., Walter, S., & Fidler, K. (2022) · Archives of Disease in Childhood Fetal & Neonatal · doi:10.1136/archdischild-2021-323380 · Cited by 34
Asonganyi, E., Vaghasia, M., Rodrigues, C. G., Phadtare, A., Ford, A. C., Pietrobon, R. S., Atashili, J., & Lynch, C. A. (2013) · PLoS ONE · doi:10.1371/journal.pone.0072712 · Cited by 34
“We chose this checklist based on a systematic review previously published [18], which recommends to use checklists rather than scales, as well as to use a tool as specific as possible, considering study design and area.”
MethodDeficiência de micronutrientes e crescimento linear: revisão sistemática de estudos observacionais
Pedraza, D. F., Rocha, A. C. D., & Sales, M. C. (2013) · Ciência & Saúde Coletiva · doi:10.1590/s1413-81232013001100023 · Cited by 34
Chow, E. P. F., Tung, K. T. S., Tucker, J. D., Muessig, K. E., Su, S., Zhang, X., Jing, J., & Zhang, L. (2015) · AIDS Patient Care and STDs · doi:10.1089/apc.2015.0043 · Cited by 34
Pollock, K., Dorstyn, D., Butt, L., & Prentice, S. (2017) · Spinal Cord · doi:10.1038/sc.2017.45 · Cited by 34
van Netten, J. J., Baba, M., & Lazzarini, P. A. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0488-5 · Cited by 34
“In line with conclusions from a recent systematic review on tools for assessing the quality and susceptibility to bias in observational studies, a transparent checklist approach has been chosen, concentrating on the few—principal—potential sources of bias identified from this systematic review [19].”
BackgroundPrevalence of Head Injury and Associated Disability in Prison Populations: A Systematic Review
Moynan, C. R., & McMillan, T. M. (2017) · Journal of Head Trauma Rehabilitation · doi:10.1097/htr.0000000000000354 · Cited by 34
Kendrick, D., Drummond, A. E., Logan, P., Barnes, J., & Worthington, E. J. (2015) · Journal of Transport & Health · doi:10.1016/j.jth.2015.06.002 · Cited by 34
“3 Risk of bias in included studies A recent systematic review of tools for assessing quality and susceptibility to bias in observational studies in epidemiology identified a number of useful assessment tools [10].”
BackgroundSocial Cognition in Paediatric Traumatic Brain Injury: A Systematic Review and Meta-analysis
On, Z. X., Ryan, N. P., Konjarski, M., Catroppa, C., & Stargatt, R. (2021) · Neuropsychology Review · doi:10.1007/s11065-021-09488-2 · Cited by 34
“Validity Assessment There is currently no gold standard assessment of methodological study quality for observational research (Sanderson et al., 2007).”
BackgroundHouwen, T., de Munter, L., Lansink, K. W. W., & de Jongh, M. A. C. (2022) · Journal of Patient-Reported Outcomes · doi:10.1186/s41687-022-00440-3 · Cited by 33
Does stage-3 chronic kidney disease matter?: A systematic literature review
Sharma, P., McCullough, K., Scotland, G., McNamee, P., Prescott, G. J., MacLeod, A. M., Fluck, N., Smith, W. C., & Black, C. (2010) · British Journal of General Practice · doi:10.3399/bjgp10x502173 · Cited by 33
Petti, S., & Vitali, M. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-015374 · Cited by 33
“In addition, assessing quality and susceptibility to bias of observational studies is not easy, as there are almost 100 quality assessment tools, none of them rigorously developed, evidence based, valid, reliable and easy to use at the same time.(44) For these reasons, study quality was assessed but was considered onl…”
MethodHypothyroidism as a risk factor for open angle glaucoma: A systematic review and meta-analysis
Wang, S., Liu, Y., & Zheng, G. (2017) · PLoS ONE · doi:10.1371/journal.pone.0186634 · Cited by 33
Knorr, U., Simonsen, A. H., Zetterberg, H., Blennow, K., Hasselbalch, S. G., & Kessing, L. V. (2018) · European Neuropsychopharmacology · doi:10.1016/j.euroneuro.2018.04.002 · Cited by 34
“Tools for assessing quality in observational epidemiological studies has been given less attention compared to tools assessing quality in clinical trials (8).”
Intraocular pressure fluctuation and the risk of glaucomatous damage deterioration: a Meta-analysis
Guo, Z.-Z., Chang, K., & Wei, X. (2019) · International Journal of Ophthalmology · doi:10.18240/ijo.2019.01.19 · Cited by 33
Bowler, A., Arichi, T., Austerberry, C., Fearon, P., & Ronald, A. (2024) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2024.105825 · Cited by 32
Genes associated with persistent lumbar radicular pain; a systematic review
Bjorland, S., Moen, A., Schistad, E. I., Gjerstad, J., & Røe, C. (2016) · BMC Musculoskeletal Disorders · doi:10.1186/s12891-016-1356-5 · Cited by 32
“[34], QATSO (Quality Assessment Tool for Systematic Reviews of Observational studies) [35] and the STROBE statement guidelines (Strengthening the Reporting of Observational Studies in Epidemiology) [36] was used.”
MethodLeeyaphan, C., Ong, J. J., Chow, E. P. F., Kong, F. Y. S., Hocking, J. S., Bissessor, M., Fairley, C. K., & Chen, M. (2016) · Emerging infectious diseases · doi:10.3201/eid2210.160986 · Cited by 31
“) independently assessed within-study bias using evaluation criteria reported elsewhere (24).”
MethodGalvin, A., Delva, F., Helmer, C., Rainfray, M., Bellera, C. A., Rondeau, V., Soubeyran, P., Coureau, G., & Mathoulin‐Pélissier, S. (2017) · Journal of Geriatric Oncology · doi:10.1016/j.jgo.2017.07.007 · Cited by 31
Prevalence of Retinal Vein Occlusion in Europe: A Systematic Review and Meta-Analysis
Li, J. Q., Terheyden, J. H., Welchowski, T., Schmid, M. C., Letow, J., Wolpers, A. C., Holz, F. G., & Finger, R. P. (2018) · Ophthalmologica · doi:10.1159/000494224 · Cited by 31
Chuter, V., West, M., Hawke, F., & Searle, A. (2019) · Journal of Foot and Ankle Research · doi:10.1186/s13047-019-0326-1 · Cited by 31
García, Y. R., Guevara, J. N., Izurieta, P., Vojtek, I., Ortega‐Barría, E., & Guzmán-Holst, A. (2021) · Expert Review of Vaccines · doi:10.1080/14760584.2021.1873136 · Cited by 31
Wan, Z., & Han, B. (2021) · Therapeutic Advances in Hematology · doi:10.1177/20406207211032433 · Cited by 31
Smokeless tobacco use in pregnancy: an integrative review of the literature
Ratsch, A., & Bogossian, F. E. (2014) · International Journal of Public Health · doi:10.1007/s00038-014-0558-6 · Cited by 31
Harder, T., Takla, A., Rehfuess, E. A., Sanchez‐Vivar, A., Matysiak‐Klose, D., Eckmanns, T., Krause, G., Gomes, H. D. C., Jansen, A., Ellis, S., et al. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-69 · Cited by 31
The development of CHAMP: a checklist for the appraisal of moderators and predictors
van Hoorn, R., Tummers, M., Booth, A., Gerhardus, A., Rehfuess, E., Hind, D., Bossuyt, P. M., Welch, V., Debray, T. P. A., Underwood, M., et al. (2017) · BMC Medical Research Methodology · doi:10.1186/s12874-017-0451-0 · Cited by 28
“systematic reviews [22, 23]) to appraise the overall risk of bias for the main effect in any given study.”
BackgroundChoinière, R., Violette, P. D., Morin, M., Tu, L. M., Guyatt, G. H., Reed, C., Philie, C.-A., Legault, B., Beaudry, M.-M., Ahmed, M. M., et al. (2021) · European Urology Focus · doi:10.1016/j.euf.2021.09.007 · Cited by 31
Gin, K., Stewart, C. S., & Jolley, S. (2020) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.2493 · Cited by 33
Maternal and perinatal outcomes amongst haemoglobinopathy carriers: a systematic review
Jans, S. M. P. J., de Jonge, A., & Lagro-Janssen, A. L. M. (2010) · International Journal of Clinical Practice · doi:10.1111/j.1742-1241.2010.02451.x · Cited by 31
Are cancer survivors well‐performing workers? A systematic review
Soejima, T., & Kamibeppu, K. (2016) · Asia-Pacific Journal of Clinical Oncology · doi:10.1111/ajco.12515 · Cited by 31
Larkin, C., Blasi, Z. D., & Arensman, E. (2014) · PLoS ONE · doi:10.1371/annotation/b3dfd1b3-ada1-4f7e-89d4-a25ddf5d204a · Cited by 31
Sargeant, J. M., O'Connor, A. M. C., Dohoo, I. R., Erb, H. N., Cevallos, M., Egger, M., Ersbøll, A. K., Martin, S. W., Nielsen, L. R., Pearl, D. L., et al. (2016) · Journal of Food Protection · doi:10.4315/0362-028x.jfp-16-016 · Cited by 30
“There are several systematic reviews published on quality assessment tools for observational research (7, 16, 20).”
BackgroundHagedoorn, N. N., Zachariasse, J. M., & Moll, H. A. (2019) · Critical Care · doi:10.1186/s13054-019-2653-9 · Cited by 30
“Quality assessment No specific tool exists for quality assessment of observational studies [10].”
MethodJeyaraman, M. M., Al‐Yousif, N., Robson, R. C., Copstein, L., Balijepalli, C., Hofer, K., Fazeli, M. S., Ansari, M., Tricco, A. C., Rabbani, R., et al. (2020) · Systematic Reviews · doi:10.1186/s13643-020-01291-z · Cited by 31
Age at menopause in women living with HIV: a systematic review
Ommen, C. V. V., King, E. M., & Murray, M. C. M. (2021) · Menopause · doi:10.1097/gme.0000000000001871 · Cited by 24
“Currently, there is no consensus on the optimal assessment tool for observational studies.(29) We used a modified version of the Newcastle-Ottawa Scale to evaluate the included articles(30) (Supplementary Digital Content, http:// links.”
MethodTan, D. H. S., Murphy, K. E., Shah, P. S., & Walmsley, S. L. (2013) · BMC Infectious Diseases · doi:10.1186/1471-2334-13-502 · Cited by 30
“However, we systematically applied criteria based on the STROBE statement as has been advised elsewhere, because of the lack of a single obvious candidate tool for this purpose in systematic reviews of observational studies [11].”
MethodSheehan, M. C., & Lam, J. (2015) · Current Environmental Health Reports · doi:10.1007/s40572-015-0062-z · Cited by 31
van der Veek, S., Derkx, H. H. F., de Haan, E., Benninga, M. A., Plak, R. D., & Boer, F. (2011) · Journal of Health Psychology · doi:10.1177/1359105311410513 · Cited by 30
Ciapponi, A., Elorriaga, N., Rojas, J. I., Romano, M., Martí, S. G., Bardach, A., & Ruvinsky, S. (2014) · The Pediatric Infectious Disease Journal · doi:10.1097/inf.0000000000000363 · Cited by 30
“The risk of bias for observational studies was assessed by a modified checklist of essential items stated in the STROBE statement19 (Strengthening the Reporting of Observational studies in Epidemiology) and reported by Fowkes and Fulton20 and Sanderson et al.21 We used an algorithm to estimate a summary risk of bias c…”
MethodAdding life to years: quality of life of people with dementia receiving long-term care
Beerens, H. C. (2016) · Research Publications (Maastricht University) · doi:10.26481/dis.20160429hb · Cited by 30
Fitton, L., & Jones, D. R. (2018) · Journal of Intellectual Disabilities · doi:10.1177/1744629518778695 · Cited by 29
“…as it guides appraisal of the essential information that should be provided in cross-sectional studies through the use of a 22-item checklist assessing the study title, abstract, 270 Journal of Intellectual Disabilities 24(2) introduction, method, results and discussion (Sanderson et al., 2007).”
BackgroundLogan, G. S., Pike, A. E., Copsey, B. J., Parfrey, P. S., Etchegary, H., & Hall, A. M. (2019) · PLoS ONE · doi:10.1371/journal.pone.0225414 · Cited by 29
“which provides a list of specific domains to be considered.[22] RoB for these observational, non-randomised studies What we know about appropriateness of imaging in primary care: a systematic review and meta-analysis”
BackgroundRemington, R. (2020) · Springer eBooks · doi:10.1007/978-3-030-37504-1_4 · Cited by 29
Lyons, J. G., O’Dea, K., & Walker, K. Z. (2014) · BMC Public Health · doi:10.1186/1471-2458-14-545 · Cited by 29
“The quality of each study was assessed according to study-specific criteria (Table 1), based on previous general recommendations [30].”
MethodPedraza, D. F. (2014) · Revista Brasileira de Ciências da Saúde - USCS · doi:10.13037/rbcs.vol12n41.2237 · Cited by 29
Changes in sedentary behaviours across the retirement transition: a systematic review.
MURPHY, M. H. (2015) · PubMed · doi:10.1093/ageing/afv142 · Cited by 1
“We therefore opted to use the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) checklist [17] as it comprehensively examines numerous aspects of studies’ designs and reporting and has been used for risk of bias scoring in previous systematic reviews [18, 19].”
MethodWacker, K., Bourne, W. M., & Patel, S. V. (2015) · American Journal of Ophthalmology · doi:10.1016/j.ajo.2015.12.015 · Cited by 29
“addressing extreme individual outliers (leverage) and absence of discrepancy in number of patients between text, tables, and figures; control for confounding by appropriate design and analytic methods, and addressing design-specific sources of bias (Table 1).(12)”
MethodSargeant, J. M., O'Connor, A. M. C., Dohoo, I. R., Erb, H. N., Cevallos, M., Egger, M., Ersbøll, A. K., Martin, S. W., Nielsen, L. R., Pearl, D. L., et al. (2016) · Zoonoses and Public Health · doi:10.1111/zph.12314 · Cited by 29
Möhner, M., & Wendt, A. (2017) · Critical Reviews in Toxicology · doi:10.1080/10408444.2016.1266598 · Cited by 29
“There are many instruments for assessing the methodological quality of non-randomized studies (Stroup et al. 2000; Sanderson et al. 2007; Vandenbroucke et al. 2007; Moher et al. 2015).”
BackgroundSystematic review and meta‐analysis of studies on delivery preferences in Brazil
Reiter, M., Betrán, A. P., Marques, F. K., & Torloni, M. R. (2018) · International Journal of Gynecology & Obstetrics · doi:10.1002/ijgo.12570 · Cited by 29
Li, J., Wang, H., Li, D., Zhang, Q., & Liu, N. (2021) · Human Vaccines & Immunotherapeutics · doi:10.1080/21645515.2020.1849519 · Cited by 29
“This tool was based on a checklist of essential items in the STROBE statement(9) and two methodological studies.(10,11) A third investigator resolved disagreements by discussion.”
MethodPega, F., Walter, S., Liu, S. Y., Pabayo, R., Lhachimi, S. K., & Saith, R. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011135 · Cited by 29
Tseng, T. Y., Breau, R. H., Fesperman, S. F., Vieweg, J., & Dahm, P. (2008) · British Journal of Urology · doi:10.1111/j.1464-410x.2008.08155.x · Cited by 29
Not All Clinical AI Monitoring Systems Are Created Equal: Review and Recommendations
Feng, J., Xia, F., Singh, K., & Pirracchio, R. (2025) · NEJM AI · doi:10.1056/aira2400657 · Cited by 28
Tian, F., Diao, W., Yang, X., Wang, X., Roberts, N., Feng, C., & Jia, Z. (2019) · Psychological Medicine · doi:10.1017/s0033291719000473 · Cited by 28
Antenatal fear of childbirth and emergency cesarean section delivery: a systematic narrative review
Molgora, S., Fenaroli, V., Cracolici, E., & Saita, E. (2019) · Journal of Reproductive and Infant Psychology · doi:10.1080/02646838.2019.1636216 · Cited by 28
Kaur, A., Lewis, T., Lipkova, V., Fernando, S., Rayner, M., Harrington, R., Waterlander, W. E., & Scarborough, P. (2020) · Public Health Nutrition · doi:10.1017/s1368980019004129 · Cited by 28
Lević, M., Bogavac‐Stanojević, N., & Krajnović, D. (2021) · Frontiers in Public Health · doi:10.3389/fpubh.2021.747807 · Cited by 28
“At the time of our research, there were no accepted quality assessment instruments for cross-sectional studies (56), authors decided to choose a relatively widely used scale, the Agency for Healthcare Research and Quality scale (AHRQ scale) with 11 items, each of which was answered with “yes,” “no”…”
MethodKelly, S., Greene-Finestone, L. S., Yetley, E. A., Benkhedda, K., Brooks, S. P. J., Wells, G. A., & MacFarlane, A. J. (2021) · American Journal of Clinical Nutrition · doi:10.1093/ajcn/nqab335 · Cited by 28
Systematic Literature Review of Herpes Zoster Disease Burden in Southeast Asia
Martin, P. F. M. S., Aunhachoke, K., Batac, M. C. F. R., Lodroño, K., Kwanthitinan, C., Santoso, D., Fonseka, T., Nguyễn, M. T., & Guzmán-Holst, A. (2023) · Infectious Diseases and Therapy · doi:10.1007/s40121-023-00822-0 · Cited by 28
van Zwieten, A., Dai, J., Blyth, F., Wong, G., & Khalatbari-Soltani, S. (2023) · International Journal of Epidemiology · doi:10.1093/ije/dyad177 · Cited by 25
Metadata
- Authors
- Sanderson, S., Tatt, I. D., & Higgins, J. P.
- Affiliations
- University of Cambridge (GB)
- F. Hoffmann-La Roche AG (CH)
- MRC Biostatistics Unit (GB)
- Corresponding author
- Sanderson, S.
- ORCID
- Higgins, J. P.
- Venue
- International Journal of Epidemiology
- Publisher
- Oxford University Press (OUP)
- Published
- 2007-04-30
- Volume
- 36
- Issue
- 3
- Pages
- 666-676
- ISSN
- 0300-5771 (ISSN-L)
- 1464-3685
- OpenAlex
- W2171251365
- Cited by
- 1,531
- References
- 73
- Citations by year
- 2026: 22 · 2025: 38 · 2024: 42 · 2023: 59 · 2022: 80 · 2021: 96 · 2020: 101 · 2019: 105 · 2018: 125 · 2017: 104 · 2016: 150 · 2015: 118 · 2014: 122 · 2013: 111 · 2012: 96
- FWCI
- 7.09
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- International Journal of Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- quality assessment tools, bias assessment, observational studies, critical appraisal, checklists, rating scales, STROBE, confounding adjustment, selection bias, measurement bias, conflicts of interest
- MeSH terms
- Humans, Research Design, Bias, Epidemiologic Studies, Evidence-Based Medicine, Observation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.