Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography
The DOI matches its Crossref record; 73 of the 73 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Assessing quality and susceptibility to bias is essential when interpreting primary research and conducting systematic reviews and meta-analyses. Tools for assessing quality in clinical trials are well-described but much less attention has been given to similar tools for observational epidemiological studies. METHODS: Tools were identified from a search of three electronic databases, bibliographies and an Internet search using Google. Two reviewers extracted data using a pre-piloted extraction form and strict inclusion criteria. Tool content was evaluated for domains potentially related to bias and was informed by the STROBE guidelines for reporting observational epidemiological studies. RESULTS: A total of 86 tools were reviewed, comprising 41 simple checklists, 12 checklists with additional summary judgements and 33 scales. The number of items ranged from 3 to 36 (mean 13.7). One-third of tools were designed for single use in a specific review and one-third for critical appraisal. Half of the tools provided development details, although most were proposed for future use in other contexts. Most tools included items for selection methods (92%), measurement of study variables (86%), design-specific sources of bias (86%), control of confounding (78%) and use of statistics (78%); only 4% addressed conflict of interest. The distribution and weighting of domains across tools was variable and inconsistent. CONCLUSION: A number of useful assessment tools have been identified by this report. Tools should be rigorously developed, evidence-based, valid, reliable and easy to use. There is a need to agree on critical elements for assessing susceptibility to bias in observational epidemiology and to develop appropriate evaluation tools.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Such instruments have been developed by other groups and were the subject of a recent systematic review [28].”
From The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: Guidelines for Reporting Observational Studies (2007) · cites this paper as [28]
In the citing paper
“It is not clear how weights should be assigned to each item of the scale [18].”
From Tools used to assess the quality of peer review reports: A methodological systematic review (2019) · cites this paper as [18]
In the citing paper
“The reviewers concluded that there is no ‘single obvious candidate tool for assessing quality of observational epidemiological studies’.”
From COSMOS-E: Guidance on conducting systematic reviews and meta-analyses of observational studies of etiology (2019) · cites this paper as [45]
In the citing paper
“A review identified more than 80 tools for assessing risk of bias in nonrandomised studies [45].”
From COSMOS-E: Guidance on conducting systematic reviews and meta-analyses of observational studies of etiology (2019) · cites this paper as [45]
In the citing paper
“The need for complete reporting of all relevant methodological aspects as a prerequisite to assessing the risk of bias of studies was highlighted in 10 articles, [16, 33, 35, 47, 50, 56, 86, 87, 90, 91] and was the focus of recommendations in a publication from the Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group [12].”
From Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations (2018) · cites this paper as [90]
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Meta-analysis of Observational Studies in Epidemiology<SUBTITLE>A Proposal for Reporting</SUBTITLE>
Stroup, D. F. (2000) · JAMA · doi:10.1001/jama.283.15.2008 · Cited by 20,623
Downs, S. H., & Black, N. (1998) · Journal of Epidemiology and Community Health · doi:10.1136/jech.52.6.377 · Cited by 8,252
Moher, D., Cook, D. J., Eastwood, S., Olkin, I., Rennie, D., & Stroup, D. F. (1999) · The Lancet · doi:10.1016/s0140-6736(99)04149-5 · Cited by 4,826
Whiting, P., Rutjes, A. W., Reitsma, J. B., Bossuyt, P. M., & Kleijnen, J. (2003) · BMC Medical Research Methodology · doi:10.1186/1471-2288-3-25 · Cited by 3,592
Assessing the quality of controlled clinical trials
Jüni, P., Altman, D. G., & Egger, M. (2001) · BMJ · doi:10.1136/bmj.323.7303.42 · Cited by 3,309 · cited 5×
“a (5) (5) (5) ( 5 ) N N LongneckerCoh”
InfluentialUsers' guides to the medical literature
Guyatt, G. H. (1993) · JAMA · doi:10.1001/jama.270.17.2096 · Cited by 2,970
Evaluating non-randomised intervention studies
Deeks, J., Dinnes, J., D'Amico, R., Sowden, A., Sakarovitch, C., Song, F., Petticrew, M., & Altman, D. (2003) · Health Technology Assessment · doi:10.3310/hta7270 · Cited by 2,907
Cognitive and Behavioral Outcomes of School-Aged Children Who Were Born Preterm: A Meta-analysis
Bhutta, A. T., Cleves, M. A., Casey, P. H., Cradock, M. M., & Anand, K. J. S. (2002) · JAMA · doi:10.1001/jama.288.6.728 · Cited by 2,392 · cited 1×
“Y N N 9 (22) 2 (17) 5 (15) 16 (19) Y Y N 15 (36) 6 ( 50 ) 7 (21) 28 (32)”
The Hazards of Scoring the Quality of Clinical Trials for Meta-analysis
Jüni, P. (1999) · JAMA · doi:10.1001/jama.282.11.1054 · Cited by 1,981
Low grade inflammation and coronary heart disease: prospective study and updated meta-analyses
Danesh, J. (2000) · BMJ · doi:10.1136/bmj.321.7255.199 · Cited by 1,692
Research works printed but not linked: 7
- Carneiro, 2002, Revista portuguesa de cardiologia : org o oficial da Sociedade Portuguesa de Cardiologia = Portuguese journal of cardiology : an official journal of the Portuguese Society of Cardiology, 21, 891
- 1981, Canadian Medical Association Journal, 124, 985
- 2002, SOUTH AFRICAN J CLIN NUTR, 15, 68
- 2000, Chronic diseases in Canada, 19, 170
- Ciliska, 1996, Canadian journal of public health. Revue canadienne de sant publique, 87, 193
- 1994, Canadian journal of public health. Revue canadienne de sant publique, 85, S8
- Spitzer, 1990, Clinical and investigative medicine. M decine clinique et experimentale, 13, 17
Cited by
Items 901–1,000 of 1,006
Petersen, A. M., Halkjær, S. I., & Gluud, L. L. (2015) · Scandinavian Journal of Gastroenterology · doi:10.3109/00365521.2015.1028993 · Cited by 28
A systematic review of CD14 and toll-like receptors in relation to asthma in Caucasian children
Klaassen, E. M. M., Thönissen, B. E., van Eys, G. J. J. M., Dompeling, E., & Jöbsis, Q. (2013) · Allergy Asthma and Clinical Immunology · doi:10.1186/1710-1492-9-10 · Cited by 28
The prevalence of hyperprolactinemia in non-functioning pituitary macroadenomas.
Zhang, F., Huang, Y., Ding, C., Huang, G., & Wang, S. (2015) · PubMed · Cited by 28
Black Caribbean Emerging Adults: A Systematic Review of Religion and Health
Hope, M. O., Taggart, T., Galbraith‐Gyan, K. V., & Nyhan, K. (2019) · Journal of Religion and Health · doi:10.1007/s10943-019-00932-5 · Cited by 28
Inter-rater reliability and concurrent validity of ROBINS-I: protocol for a cross-sectional study
Jeyaraman, M. M., Rabbani, R., Al‐Yousif, N., Robson, R. C., Copstein, L., Xia, J., Pollock, M., Mansour, S., Ansari, M., Tricco, A. C., et al. (2020) · Systematic Reviews · doi:10.1186/s13643-020-1271-6 · Cited by 28
Leitão, S. M., Seabra‐Santos, M. J., & Gaspar, M. F. (2020) · Family Process · doi:10.1111/famp.12550 · Cited by 28
Konradsen, A. A., Lund, C. M., Vistisen, K. K., Albieri, V., Dalton, S. O., & Nielsen, D. L. (2020) · Acta Oncologica · doi:10.1080/0284186x.2020.1772501 · Cited by 27
More, S. J., Bambidis, V., Benford, D., Bragard, C., Hernández, A. F., Bennekou, S. H., Koutsoumanis, K., Machera, K. A., Naegeli, H., Nielsen, S. S., et al. (2020) · EFSA Journal · doi:10.2903/j.efsa.2020.6221 · Cited by 27
Brons, S., van Beusichem, M. E., Bronkhorst, E. M., Draaisma, J. M. T., Bergé, S. J., Schols, J. G. J. H., & Kuijpers‐Jagtman, A. M. (2014) · PLoS ONE · doi:10.1371/journal.pone.0089602 · Cited by 27
“In the literature, no single tool is an obvious candidate for assessment of methodological quality of nonrandomized studies [150].”
Knight, A., & Lindfield, R. (2015) · Public Health · doi:10.1016/j.puhe.2014.10.011 · Cited by 27
“This checklist had been originally developed as a guidance to authors writing observational research papers, and was not intended to be a quality measurement tool in itself.(17) Whilst it was adapted here to consider the suitability of each component in the studies, another tool may have offered a more reliable measur…”
MethodEffects of Tai Chi and Qigong in Children and Adolescents: A Systematic Review of Trials
Riskowski, J. L., & Almeheyawi, R. N. (2017) · Adolescent Research Review · doi:10.1007/s40894-017-0067-y · Cited by 27
“1 3 CASP tool was chosen as it is a widely used and well-developed critical appraisal tool to assess the methodological quality of studies (Sanderson et al. 2007).”
MethodThe effects of licit and illicit recreational drugs on prospective memory: a meta-analytic review
Platt, B., O’Driscoll, C., Curran, V. H., Rendell, P. G., & Kamboj, S. K. (2019) · Psychopharmacology · doi:10.1007/s00213-019-05245-9 · Cited by 27
Ecological Gait as a Fall Indicator in Older Adults: A Systematic Review
Job, M., Dottor, A., Viceconti, A., & Testa, M. (2019) · The Gerontologist · doi:10.1093/geront/gnz113 · Cited by 27
E-Synthesis: A Bayesian Framework for Causal Assessment in Pharmacosurveillance
Pretis, F. D., Landes, J., & Osimani, B. (2019) · Frontiers in Pharmacology · doi:10.3389/fphar.2019.01317 · Cited by 27
“…al., 2018), grading certainty of evidence of effects in studies (Alonso-Coello et al., 2016), grading observational studies based on study design (Sanderson et al., 2007; Sterne et al., 2016; Wells et al., 2018), thematic synthesis of qualitative research, decision making (Thomas and Harden,…”
BackgroundSocioeconomic differences in hyperuricemia and gout: a systematic review and meta-analysis
Pan, Z., Huang, M., Fang, M., Xie, X., & Huang, Z. (2020) · Endocrine · doi:10.1007/s12020-020-02281-w · Cited by 27
Perspective: Essential Study Quality Descriptors for Data from Nutritional Epidemiologic Research
Yang, C., Pinart, M., Kolsteren, P. W., Camp, J. V., Cock, N. D., Nimptsch, K., Pischon, T., Laird, E. J., Perozzi, G., Canali, R., et al. (2017) · Advances in Nutrition · doi:10.3945/an.117.015651 · Cited by 26
“Quality tools exist for epidemiologic research (13) but are designed to appraise the quality of the study findings, rather than the study data, and are influenced by the reporting quality of the study or judgement of external assessors.”
InfluentialBlaauw, J., Boxum, A. G., Jacobs, B., Groen, R. J. M., Peul, W. C., Jellema, K., Dammers, R., Gaag, N. A. V. D., Lingsma, H. F., den Hertog, H. M., et al. (2020) · Journal of Neurotrauma · doi:10.1089/neu.2020.7206 · Cited by 26
“21,22 The NOS form consists of eight items categorized into three groups, and describes the quality of selection, comparability, and outcome.”
Wu, D., Wu, X., Zhou, X., Huang, W., Luo, C., & Liu, Y. (2021) · Annals of Translational Medicine · doi:10.21037/atm-20-4708 · Cited by 26
“Each item of the checklist was scored with “Yes”, “No”, or “Unclear” (18,19).”
Effects of deployment on health behaviours in military forces: A review of longitudinal studies
Pietrzak, E. R., Pullman, S., Cotea, C., & Nasveld, P. (2013) · Journal of military and veterans' health/Journal of military and veterans' health. · Cited by 26
A Systematic Review of Self-Concept in Adolescents With Epilepsy
Ferro, M. A., Ferro, A. L., & Boyle, M. H. (2012) · Journal of Pediatric Psychology · doi:10.1093/jpepsy/jss076 · Cited by 26
Glasziou, P., Chalmers, I., Altman, D. G., Bastian, H., Boutron, I., Brice, A., Jamtvedt, G., Farmer, A. A., Ghersi, D., Groves, T., et al. (2010) · Cited by 26
Communicable diseases in rural and remote Australia: the need for improved understanding and action
Quinn, E., Massey, P., & Speare, R. (2015) · Rural and Remote Health · doi:10.22605/rrh3371 · Cited by 25
Direct observation of weight-related communication in primary care: a systematic review
McHale, C., Laidlaw, A. H., & Cecil, J. E. (2016) · Family Practice · doi:10.1093/fampra/cmw045 · Cited by 26
Muzembo, B. A., Akita, T., Matsuoka, T., & Tanaka, J. (2016) · Public Health · doi:10.1016/j.puhe.2016.06.017 · Cited by 26
“Studies were classified as high quality (score 8e10), moderate quality (score 5e7) or poor quality (score 4).(20,21)”
BackgroundThe effect of allergen immunotherapy in the onset of new sensitizations: a meta‐analysis
Lorenzo, G. D., Leto‐Barone, M. S., Piana, S. L., Plaia, A., & Bona, D. D. (2017) · International Forum of Allergy & Rhinology · doi:10.1002/alr.21946 · Cited by 26
Sanchez, Z. M., Valente, J. Y., Pereira, A. P. D., Cogo‐Moreira, H., Melo, M. H. D. S., Caetano, S. C., & Mari, J. D. J. (2019) · BMC Public Health · doi:10.1186/s12889-019-7090-9 · Cited by 26
Graham, S., Harrod, M. E., Iversen, J., & Hocking, J. S. (2016) · Hepatitis Monthly · doi:10.5812/hepatmon.38640 · Cited by 26
“An assessment of within study bias (selection, measurement and confounding) (17) was undertaken to examine the potential effects of bias for each individual study.”
MethodInternational changes in end-of-life practices over time: a systematic review
Chao, Y., Boivin, A., Marcoux, I., Garnon, G., Mays, N., Lehoux, P., Prémont, M., van Leeuwen, E., Pineault, R., Blackmer, J., et al. (2016) · BMC Health Services Research · doi:10.1186/s12913-016-1749-z · Cited by 26
Systematic reviews and meta-analyses in neurosurgery Part II: a guide to designing the protocol
Lee, K. S., & Prevedello, D. M. (2024) · Neurosurgical Review · doi:10.1007/s10143-024-02555-1 · Cited by 25
Price, G. J., Mackay, R. I., Aznar, M. C., McWilliam, A., Johnson‐Hart, C., van Herk, M., & Finn, C. F. (2021) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2021.09.030 · Cited by 25
Altered brain activation during reward anticipation in bipolar disorder
Long, X., Wang, X., Tian, F., Cao, Y., Xie, H., & Jia, Z. (2022) · Translational Psychiatry · doi:10.1038/s41398-022-02075-w · Cited by 25
Langerman, C., Forbes, A., & Robert, G. (2022) · Primary care diabetes · doi:10.1016/j.pcd.2022.08.008 · Cited by 25
Goodman, E., Reuschenbach, M., Kaminski, A., & Ronnebaum, S. M. (2022) · Vaccines · doi:10.3390/vaccines10091543 · Cited by 25
“The risk of bias in each included study was assessed using the Newcastle-Ottawa scale [20,21].”
MethodGarcía‐Lara, R. A., Suleiman‐Martos, N., Membrive‐Jiménez, M. J., García‐Morales, V., Quesada-Caballero, M., Guisado-Requena, I. M., & Gómez‐Urquiza, J. L. (2022) · Diagnostics · doi:10.3390/diagnostics12123094 · Cited by 25
“The degree of bias present in the observational studies considered was determined using the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) guidelines [19].”
MethodXiao, F., An, Z., Lv, J., Sun, X., Sun, H., Liu, Y., Liu, X., & Guo, H. (2023) · Frontiers in Public Health · doi:10.3389/fpubh.2023.1173101 · Cited by 26
Abbasi, E. (2025) · The Journal of Advances in Parasitology · doi:10.17582/journal.jap/2025/11.46.59 · Cited by 25
Michelet, M. (2012) · Evidence-Based Dentistry · doi:10.1038/sj.ebd.6400870 · Cited by 25
Increased risk for atypical fractures associated with bisphosphonate use
Lee, S., Yin, R. V., Hirpara, H., Lee, N. C., Lee, A., Llanos, S., & Phung, O. J. (2015) · Family Practice · doi:10.1093/fampra/cmu088 · Cited by 25
“Quality rating of studies was performed by two investigators using methodology recommended by the Evidence-based Practice Center Method Guide (15,16).”
MethodGiraud, R., Siegenthaler, N., Merlani, P. G., & Bendjelid, K. (2016) · Journal of Clinical Monitoring and Computing · doi:10.1007/s10877-016-9823-y · Cited by 25
“As there is no gold-standard validated method for assessing bias and the quality of observational studies (10, [12] but there is consensus that the three most fundamental domains are (1) appropriate selection of studies, (2) appropriate measurement of variables, and (3) appropriate control of confounding, as well as c…”
MethodSuphanchaimat, R., Cetthakrikul, N., Dalliston, A., & Putthasri, W. (2016) · Advances in Medical Education and Practice · doi:10.2147/amep.s116699 · Cited by 25
Gon, G., de Barra, M., Dansero, L., Nash, S., & Campbell, O. M. R. (2020) · BMC Health Services Research · doi:10.1186/s12913-020-05925-9 · Cited by 25
“For observational studies, we assessed quality using checklist we developed using eight items adapted from the STROBE guidelines’ [15] methods section (as recommended by Sanderson and colleagues) [16], to the specific context of observing hand hygiene in healthcare settings.”
MethodQiu, D., Li, Y., Li, L., He, J., Ouyang, F., & Xiao, S. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.668784 · Cited by 25
“Quality Assessment Two independent reviewers (JH and FYOY) used the established guidelines, the Loney criteria, to evaluate the methodological quality of the included studies, which has been widely used to evaluate observational studies (Loney et al., 1998; Sanderson et al., 2007).”
MethodDwek, M., Rixon, L., Hurt, C. S., Simon, A. E., & Newman, S. P. (2016) · Psycho-Oncology · doi:10.1002/pon.4331 · Cited by 25
“4 | Quality assessment As there is currently no agreed “gold standard” appraisal tool for observational studies, a quality‐scoring tool was developed on the basis of methodological quality assessment checklists from the NICE Methods for the development of NICE public health guidance (Appendix 1).(39,40) Valid criteria…”
MethodGebrye, T., Fatoye, F., Mbada, C. E., & Hakimi, Z. (2023) · Rheumatology International · doi:10.1007/s00296-023-05354-x · Cited by 24
“Some evaluation tools are, for example, used without recommendations for critical appraisal of evidence [45].”
MethodAlmeida, F. T., Pachêco‐Pereira, C., Porporatti, A. L., Flores‐Mir, C., Leite, A. F., Canto, G. D. L., & Guerra, E. N. S. (2015) · Journal of Gastroenterology and Hepatology · doi:10.1111/jgh.13149 · Cited by 24
Zhao, Z., Chen, M., Peng, Y., Chai, H., Liu, W., Li, Q., Ren, X., Wang, X., Luo, X., Zhang, C., et al. (2013) · Heart · doi:10.1136/heartjnl-2013-304138 · Cited by 24
Reporting quality and risk of bias in randomised trials in health professions education
Horsley, T., Galipeau, J., Petkovic, J., Zeiter, J., Hamstra, S. J., & Cook, D. A. (2016) · Medical Education · doi:10.1111/medu.13130 · Cited by 24
Ergin, E., & Akın, B. (2018) · Turkiye Klinikleri Journal of Nursing Sciences · doi:10.5336/nurses.2018-61334 · Cited by 24
Dual-task effects in children with neuromotor dysfunction: a systematic review
Pena, G. M., Pavão, S. L., Oliveira, M. F. P., de Campos, A. C., & Rocha, N. A. C. F. (2019) · European Journal of Physical and Rehabilitation Medicine · doi:10.23736/s1973-9087.19.05556-4 · Cited by 24
Gout and risk of diabetes mellitus: meta-analysis of observational studies
Zhao, R. Z., Wang, Y., Fu, T., Zhou, W., Ge, X., Sha, X., Guo, J., Dong, C., & Guo, G. (2019) · Psychology Health & Medicine · doi:10.1080/13548506.2019.1707241 · Cited by 24
Bardach, A., Ciapponi, A., Rey‐Ares, L., Rojas, J. I., Mazzoni, A., Glujovsky, D., Valanzasca, P., Romano, M., Elorriaga, N., Juri, M. J. D., et al. (2015) · Value in Health Regional Issues · doi:10.1016/j.vhri.2015.05.002 · Cited by 24
“The risk of bias of observational studies was assessed by a modified checklist of essential items stated in Strengthening the Reporting of Observational Studies in Epidemiology and in Fowkes and Sanderson [16–20].”
MethodCombined oral contraceptives: venous thrombosis
de Bastos, M., Stegeman, B. H., Rosendaal, F. R., Vlieg, A. V. H., Helmerhorst, F. M., Stijnen, T., & Dekkers, O. M. (2013) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd010813 · Cited by 24
Tak, L. M., Meijer, A., Manoharan, A., de Jonge, P., & Rosmalen, J. G. M. (2010) · Psychosomatic Medicine · doi:10.1097/psy.0b013e3181d714e1 · Cited by 23
Wiegers, E., Sewalt, C. A., Venema, E., Schep, N. W. L., Verhaar, J. A. N., Lingsma, H. F., & Hartog, D. D. (2019) · Acta Orthopaedica · doi:10.1080/17453674.2018.1545383 · Cited by 23
“Assessment of quality and generalizability of the studies was based on the key domains for observational studies (Sanderson et al. 2007).”
BackgroundMorche, J., Freitag, S., Hoffmann, F., Rissling, O., Langer, G., Nußbaumer-Streit, B., Toews, I., Gartlehner, G., & Meerpohl, J. J. (2020) · Zeitschrift für Evidenz Fortbildung und Qualität im Gesundheitswesen · doi:10.1016/j.zefq.2019.11.003 · Cited by 23
Indicators of Drug-Related Community Impacts of Open Drug Scenes: A Scoping Review
Zafarghandi, M. B. S., Eshrati, S., Rashedi, V., Vameghi, M., Arezoomandan, R., Clausen, T., & Waal, H. (2021) · European Addiction Research · doi:10.1159/000519886 · Cited by 23
Stylianou, M., Woodforde, J., Duncombe, S. L., Kolbe-Alexander, T. L., & Gomersall, S. R. (2021) · Health & Place · doi:10.1016/j.healthplace.2021.102705 · Cited by 23
Feifei, X., & Brodszky, V. (2024) · Discover Psychology · doi:10.1007/s44202-024-00204-8 · Cited by 23
“…review, we utilized three specific tools: the Appraisal Tool for Cross-Sectional Studies (AXIS) (N = 31) [48], Newcastle–Ottawa Scale (NOS) (N = 5) [49], and Consoli-dated Health Economic Evaluation Reporting Standards (CHEERS) (N = 1) [50] for cross-sectional studies, cohort studies, and…”
Chronic mercury exposure and blood pressure in children and adolescents: a systematic review
Gallego-Viñas, G., Ballester, F., & Llop, S. (2018) · Environmental Science and Pollution Research · doi:10.1007/s11356-018-3796-y · Cited by 23
Thompson, K., van Ophem, J., & Wagemakers, A. (2019) · Economics & Human Biology · doi:10.1016/j.ehb.2019.06.004 · Cited by 23
Martín-Cameán, A., Jos, A. M., Cameán, A. M., Solano, E., & Iglesias‐Linares, A. (2015) · Toxicology Mechanisms and Methods · doi:10.3109/15376516.2015.1062951 · Cited by 23
“After Sanderson et al. (2007) identified and analyzed the existing tools in the literature for assessing quality, they concluded that there was a need to agree on critical elements for assessing susceptibility to bias in observational epidemiology and to develop appropriate evaluation tools.”
MethodObesity studies in the circumpolar Inuit: a scoping review
Galloway, T., Blackett, H., Chatwood, S., Jeppesen, C., Kandola, K., Linton, J., & Bjerregaard, P. (2012) · International Journal of Circumpolar Health · doi:10.3402/ijch.v71i0.18698 · Cited by 23
“A review of 86 published QA instruments yielded no clear candidate for a generic QA tool as the majority of reviewers develop subject-specific instruments (26).”
BackgroundMethodGraham, S., Smith, L. W., Fairley, C. K., & Hocking, J. S. (2016) · Sexual Health · doi:10.1071/sh15171 · Cited by 23
GRADE-Leitlinien: 4. Bewertung der Qualität der Evidenz – Studienlimitationen (Risiko für Bias)
Meerpohl, J. J., Langer, G., Perleth, M., Gartlehner, G., Kaminski-Hartenthaler, A., & Schünemann, H. J. (2012) · Zeitschrift für Evidenz Fortbildung und Qualität im Gesundheitswesen · doi:10.1016/j.zefq.2012.06.014 · Cited by 23
Thomas, L. H., French, B., Sutton, C., Forshaw, D., Leathley, M. J., Burton, C. R., Roe, B. H., Cheater, F., Booth, J., McColl, E., et al. (2015) · Programme Grants for Applied Research · doi:10.3310/pgfar03010 · Cited by 23
Review of quality assessment tools for the evaluation of pharmacoepidemiological safety studies
Neyarapally, G. A., Hammad, T. A., Pinheiro, S. P., & Iyasu, S. (2012) · BMJ Open · doi:10.1136/bmjopen-2012-001362 · Cited by 20
Yu, Y., Liu, W., Jiang, H.-L., & Mao, B. (2021) · Respiration · doi:10.1159/000510615 · Cited by 22
Multigenerational transmission of obesity: A systematic review and meta‐analysis
Kanmiki, E. W., Fatima, Y., & Mamun, A. A. (2021) · Obesity Reviews · doi:10.1111/obr.13405 · Cited by 22
“The validity of our adopted quality assessment tool is comparable with other tools as we have considered important factors about the study design, context, data collection approach, information bias, confounding, statistical analysis, and reporting bias.(30) During the assessment, studies were assigned points for spec…”
MethodMekhael, A. A., Bent, J. E., Fawcett, J. M., Campbell, T. S., Aguirre‐Camacho, A., Farrell, A., & Rash, J. A. (2023) · Canadian Journal of Pain · doi:10.1080/24740527.2023.2191114 · Cited by 22
Xun, Y., Guo, Q., Ren, M., Liu, Y., Sun, Y., Wu, S., Lan, H., Zhang, J., Liu, H., Wang, J., et al. (2023) · Frontiers in Public Health · doi:10.3389/fpubh.2023.998588 · Cited by 22
Glaser, N., Jackson, V., Franco‐Cereceda, A., & Sartipy, U. (2018) · The Thoracic and Cardiovascular Surgeon · doi:10.1055/s-0038-1649513 · Cited by 22
D’Andrea, E., Vinals, L., Patorno, E., Franklin, J. M., Bennett, D., Largent, J. A., Moga, D. C., Yuan, H., Wen, X., Zullo, A. R., et al. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-043961 · Cited by 22
Yan, Q., Qiu, D., Liu, X., Guo, X., & Hu, Y. (2021) · Frontiers in Pediatrics · doi:10.3389/fped.2021.686600 · Cited by 22
Large Epidemiologic Studies of Gout: Challenges in Diagnosis and Diagnostic Criteria
Wijnands, J. M. A., Boonen, A. E. R. C. H., Arts, I. C. W., Dagnelie, P. C., Stehouwer, C. D. A., & van der Linden, S. (2010) · Current Rheumatology Reports · doi:10.1007/s11926-010-0157-3 · Cited by 22
“In addition to the above considerations that are critical in the appraisal of data on the occurrence of disease, several other issues merit consideration when appraising the results of such studies [32].”
Shamliyan, T., Kane, R. L., & Jansen, S. (2012) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.10.017 · Cited by 22
“Published epidemiologic analyses of the tools available to assess the quality of all types of observational studies have demonstrated inconsistent content and format [71].”
BackgroundKnoop, I., Picariello, F., Jenkinson, E., Gall, N., Chisari, C., & Moss-Morris, R. (2023) · Autonomic Neuroscience · doi:10.1016/j.autneu.2022.103052 · Cited by 21
“Eligible full-text articles were assessed for methodological quality according to the Downs & Black (Downs and Black, 1998) quality assessment, which was selected as it was adaptable to the broad range of methodological designs of included studies (Sanderson et al., 2007) (Table 3).”
Lunny, C., Veroniki, A. A., Higgins, J. P. T., Dias, S., Hutton, B., Wright, J. M., White, I. R., Whiting, P. F., & Tricco, A. C. (2024) · Systematic Reviews · doi:10.1186/s13643-023-02388-x · Cited by 21
Self-citationTotal laryngectomy vs. non-surgical organ preservation in advanced laryngeal cancer: a metanalysis
Pfuetzenreiter, E. G., Ferreron, G. F., Sadka, J. Z., de Souza, A. B. P., Matos, L. L., Kowalski, L. P., & Dedivitis, R. A. (2024) · Brazilian Journal of Otorhinolaryngology · doi:10.1016/j.bjorl.2024.101404 · Cited by 21
Li, M., Chen, M., Li, Y., Liu, Z., Li, X., Lang, X., Hu, B., & Li, W. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02549-6 · Cited by 21
“The Strengthening the Reporting of Observational Studies in Epidemiology checklist was used when assessing the eligibility of reports [13].”
MethodAnnemans, L., Boeckxstaens, P., Borgermans, L. A. D., Smedt, D. D., Duchesnes, C., Heyrman, J., Remmen, R., Sermeus, W., Broeke, C. V. D., Herck, P. V., et al. (2009) · doi:10.57598/r118c · Cited by 21
Hendijani, F., Azarpira, N., & Kaviani, M. (2018) · Clinical Transplantation · doi:10.1111/ctr.13306 · Cited by 21
Humanistic burden and economic impact of heart failure – a systematic review of the literature
Giles, L., Freeman, C. S., Field, P., Sörstadius, E., & Kartman, B. (2019) · F1000Research · doi:10.12688/f1000research.19365.1 · Cited by 21
Vaccination and the Risk of Childhood Cancer—A Systematic Review and Meta-Analysis
Marron, M., Brackmann, L. K., Kuhse, P., Christianson, L., Langner, I., Haug, U., & Ahrens, W. (2021) · Frontiers in Oncology · doi:10.3389/fonc.2020.610843 · Cited by 21
Determinants of neonatal jaundice in Ethiopia: a systematic review and meta-analysis
Belay, H. G., Arage, G., Ayele, A. D., Kassa, B. G., Mihretie, G. N., & Worke, M. D. (2022) · World Journal of Pediatrics · doi:10.1007/s12519-022-00597-3 · Cited by 21
“A Newcastle-Ottawa Scale quality assessment tool was used to assess the quality and strength of the included studies [17].”
MethodSpine surgery research: on and beyond current strategies
Jacobs, W. C. H., Kruyt, M. C., Verbout, A. J., & Öner, F. C. (2012) · The Spine Journal · doi:10.1016/j.spinee.2012.08.424 · Cited by 21
Mattos, M. G., Fernandez, C. C. A., Masterson, D., Maia, L. C., & Neves, A. D. A. (2019) · Clinical Oral Investigations · doi:10.1007/s00784-019-02814-2 · Cited by 21
Martins, A., Alves, S., Chaves, C., Canavarro, M. C., & Pereira, M. (2019) · Journal of the International AIDS Society · doi:10.1002/jia2.25241 · Cited by 21
“’s [38] systematic review and the US National Heart, Lung, and Blood Institute Quality Assessment Tool for Observational Cohort and CrossSectional Studies (individual criteria presented in Table 3) [39].”
MethodA Critical Review of Grading Systems: Implications for Public Health Policy
Irving, M., Eramudugolla, R., Cherbuin, N., & Anstey, K. J. (2017) · Evaluation & the Health Professions · doi:10.1177/0163278716645161 · Cited by 20
Produção Científica: um Guia Prático
Sampaio, M. I. C., Sabadini, A. A. Z. P., & Koller, S. H. (2022) · Universidade de São Paulo. Instituto de Psicologia eBooks · doi:10.11606/9786587596280 · Cited by 21
Bobrovitz, N., Noël, K. C., Li, Z., Cao, C., Deveaux, G., Selemon, A., Clifton, D. A., Yanes‐Lane, M., Yan, T., & Arora, R. K. (2023) · Research Synthesis Methods · doi:10.1002/jrsm.1620 · Cited by 20
“Some studies have trained deep learning algorithms to identify relevant text in publications of randomized controlled trials and predict RoB assessments, with reasonable accuracy compared to human reviewers.(33,34) However, these algorithms have largely been trained on small datasets and have limited interpretability…”
MethodYang, S., Park, S. W., Bae, S. J., Ahn, S. G., Jeong, J., & Park, K. (2023) · Journal of Breast Cancer · doi:10.4048/jbc.2023.26.e22 · Cited by 20
“[31] and was revised according to the following: 1) whether the adherence outcome of the study was clearly defined with measurable standards; 2) whether the measured adherence outcome was apparently reported; 3) whether the factors influencing the outcome were clearly identifiable; 4) whether the inclusion and exclusi…”
MethodOgourtsova, T., Archambault, P. S., & Lamontagne, A. (2015) · Topics in Stroke Rehabilitation · doi:10.1179/1074935714z.0000000046 · Cited by 20
Stallinga, H. A., ten Napel, H., Jansen, G. J., Geertzen, J. H. B., Robbé, P. F. D. V., & Roodbol, P. F. (2014) · Journal of Clinical Nursing · doi:10.1111/jocn.12624 · Cited by 20
Zoghbi, M. E., Salameh, P., Stücker, I., Brochard, P., Delva, F., & Lacourt, A. (2017) · BMC Public Health · doi:10.1186/s12889-017-4025-1 · Cited by 20
“Quality assessment and risk of bias The Newcastle-Ottawa quality assessment scale (NOS) was used to assess the quality of the design and the conduction of the included studies at the outcome level [20, 21].”
MethodPerelman, J., Rosado, R., Ferro, A., & Aguiar, P. M. V. (2017) · AIDS Care · doi:10.1080/09540121.2017.1417537 · Cited by 20
“We assessed the quality of quantitative studies using a tool based on three domains, inspired on Sanderson, Tatt, and Higgins (2007) and Elwood (2002).”
MethodOsteoarthritic knees have a highly variable patellofemoral alignment: a systematic review
Hochreiter, B., Moser, L. B., Hess, S., Hirschmann, M. T., Amsler, F., & Behrend, H. (2020) · Knee Surgery Sports Traumatology Arthroscopy · doi:10.1007/s00167-020-05928-3 · Cited by 20
McLeod, M., Signal, V., Gurney, J., & Sarfati, D. (2020) · JAMA Surgery · doi:10.1001/jamasurg.2020.0316 · Cited by 20
Schnatter, A. R., Rooseboom, M., Kocabaş, N. A., North, C. M., Dalzell, A. M., Twisk, J., Faulhammer, F., Rushton, E. K., Boogaard, P. J., Ostapenkaite, V., et al. (2020) · Toxicology Letters · doi:10.1016/j.toxlet.2020.05.036 · Cited by 20
Systematic reviews and meta‐analyses in coloproctology: interpretation and potential pitfalls
Wille‐Jørgensen, P., & Renehan, A. G. (2008) · Colorectal Disease · doi:10.1111/j.1463-1318.2007.01421.x · Cited by 19
Methodological quality is underrated in systematic reviews and meta-analyses in health psychology
Oliveras, I., Losilla, J.-M., & Vives, J. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.05.002 · Cited by 15
Metadata
- Authors
- Sanderson, S., Tatt, I. D., & Higgins, J. P.
- Affiliations
- University of Cambridge (GB)
- F. Hoffmann-La Roche AG (CH)
- MRC Biostatistics Unit (GB)
- Corresponding author
- Sanderson, S.
- ORCID
- Higgins, J. P.
- Venue
- International Journal of Epidemiology
- Publisher
- Oxford University Press (OUP)
- Published
- 2007-04-30
- Volume
- 36
- Issue
- 3
- Pages
- 666-676
- ISSN
- 0300-5771 (ISSN-L)
- 1464-3685
- OpenAlex
- W2171251365
- Cited by
- 1,531
- References
- 73
- Citations by year
- 2026: 22 · 2025: 38 · 2024: 42 · 2023: 59 · 2022: 80 · 2021: 96 · 2020: 101 · 2019: 105 · 2018: 125 · 2017: 104 · 2016: 150 · 2015: 118 · 2014: 122 · 2013: 111 · 2012: 96
- FWCI
- 7.09
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- International Journal of Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- quality assessment tools, bias assessment, observational studies, critical appraisal, checklists, rating scales, STROBE, confounding adjustment, selection bias, measurement bias, conflicts of interest
- MeSH terms
- Humans, Research Design, Bias, Epidemiologic Studies, Evidence-Based Medicine, Observation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.