How do systematic reviews incorporate risk of bias assessments into the synthesis of evidence? A methodological study
Open accessThe DOI matches its Crossref record; 42 of the 43 references it cites were found in a registry or an index.
Abstract
Background: Systematic reviews (SRs) are expected to critically appraise included studies and privilege those at lowest risk of bias (RoB) in the synthesis. This study examines if and how critical appraisals inform the synthesis and interpretation of evidence in SRs. Methods: All SRs published in March–May 2012 in 14 high-ranked medical journals and a sample from the Cochrane library were systematically assessed by two reviewers to determine if and how: critical appraisal was conducted; RoB was summarised at study, domain and review levels; and RoB appraisals informed the synthesis process. Results: Of the 59 SRs studied, all except six (90%) conducted a critical appraisal of the included studies, with most using or adapting existing tools. Almost half of the SRs reported critical appraisal in a manner that did not allow readers to determine which studies included in a review were most robust. RoB assessments were not incorporated into synthesis in one-third (20) of the SRs, with their consideration more likely when reviews focused on randomised controlled trials. Common methods for incorporating critical appraisals into the synthesis process were sensitivity analysis, narrative discussion and exclusion of studies at high RoB. Nearly half of the reviews which investigated multiple outcomes and carried out study-level RoB summaries did not consider the potential for RoB to vary across outcomes. Conclusions: The conclusions of the SRs, published in major journals, are frequently uninformed by the critical appraisal process, even when conducted. This may be particularly problematic for SRs of public health topics that often draw on diverse study designs.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“More guidance on incorporating riskof-bias assessments into meta-analyses and review conclusions is also needed. 4 5”
From RoB 2: a revised tool for assessing risk of bias in randomised trials (2019) · cites this paper as [5]
In the citing paper
“Two were narrative reviews [16,17], two focused on utility scores [15,17], and all had significant and important methodological weaknesses, such as failure to account for risk of bias in the evidence synthesis [18,19].”
From Evidence on the measurement properties of health-related quality of life instruments is largely missing in patients with low back pain: A systematic review (2018) · cites this paper as [19]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Dermyshi, E., Wang, Y., Yan, C., Hong, W., Qiu, G., Gong, X., & Zhang, T. (2017) · Neonatology · doi:10.1159/000454668 · Cited by 24,451
GRADE: an emerging consensus on rating quality of evidence and strength of recommendations
Guyatt, G. H., Oxman, A. D., Vist, G. E., Kunz, R., Falck-Ytter, Y., Alonso-Coello, P., & Schünemann, H. J. (2008) · BMJ · doi:10.1136/bmj.39489.470347.ad · Cited by 23,305
Assessing the quality of reports of randomized clinical trials: Is blinding necessary?
Jadad, A. R., Moore, R. A., Carroll, D., Jenkinson, C., Reynolds, D. J. M., Gavaghan, D. J., & McQuay, H. J. (1996) · Controlled Clinical Trials · doi:10.1016/0197-2456(95)00134-4 · Cited by 17,582
Zhang, Q., Wang, C., Zheng, M., Li, Y., Li, J., Zhang, L., Shang, X., & Yan, C. (2015) · Cerebrovascular Diseases · doi:10.1159/000369778 · Cited by 11,533
Elm, E. V., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2007) · BMJ · doi:10.1136/bmj.39335.541782.ad · Cited by 10,766
Schulz (1995) · doi:10.1001/jama.1995.03520290060030 · Cited by 5,634
Dong, Y., Hao, L., Shi, Z., Wang, G., Zhang, Z., & Han, C. (2013) · Urologia Internationalis · doi:10.1159/000351405 · Cited by 4,540
Evidence-Based Medicine: A New Approach to Teaching the Practice of Medicine
Guyatt, G. (1992) · JAMA · doi:10.1001/jama.1992.03490170092032 · Cited by 4,202
Systematic reviews in the social sciences: a practical guide
(2006) · Choice Reviews Online · doi:10.5860/choice.43-5664 · Cited by 4,089
Cited by
RoB 2: a revised tool for assessing risk of bias in randomised trials
Sterne, J. A. C., Savović, J., Page, M. J., Elbers, R. G., Blencowe, N. S., Boutron, I., Cates, C. J., Cheng, H.-Y., Corbett, M. S., Eldridge, S. M., et al. (2019) · BMJ · doi:10.1136/bmj.l4898 · Cited by 34,787
Tufanaru, C., Munn, Z., Stephenson, M., & Aromataris, E. (2015) · International Journal of Evidence-Based Healthcare · doi:10.1097/xeb.0000000000000065 · Cited by 1,057
“However, the issues of incorporating the risk of bias in the conduct of systematic reviews is complex.(10) In any case, the approach to be taken should be detailed a priori in the systematic review protocol.”
BackgroundSchünemann, H. J., Cuello, C., Akl, E. A., Mustafa, R. A., Meerpohl, J. J., Thayer, K., Morgan, R. L., Gartlehner, G., Kunz, R., Katikireddi, S. V., et al. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.01.012 · Cited by 911
Self-citationPetti, S., Glendor, U., & Andersson, L. (2018) · Dental Traumatology · doi:10.1111/edt.12389 · Cited by 698
Goh, S.-L., Persson, M. S. M., Stocks, J., Hou, Y., Lin, J., Hall, M. C., Doherty, M., & Zhang, W. (2019) · Annals of Physical and Rehabilitation Medicine · doi:10.1016/j.rehab.2019.04.006 · Cited by 451
“Because the risk of bias can differ across outcomes [25], the scoring was based on pain outcomes whenever possible.”
A Systematic Review of Neighborhood Disparities in Point-of-Sale Tobacco Marketing
Lee, J. G. L., Henriksen, L., Rose, S. W., Moreland-Russell, S., & Ribisl, K. M. (2015) · American Journal of Public Health · doi:10.2105/ajph.2015.302777 · Cited by 338
“Systematic reviewers are challenged by how to weight the evidence on the basis of studies’ risk of bias in narrative reviews.(36) To identify the strongest, most generalizable evidence for emphasis in the narrative review, we created an index of study characteristics, giving 1 point for”
Smoking as a risk factor for lung cancer in women and men: a systematic review and meta-analysis
O’Keeffe, L. M., Taylor, G., Huxley, R. R., Mitchell, P., Woodward, M., & Peters, S. A. E. (2018) · BMJ Open · doi:10.1136/bmjopen-2018-021611 · Cited by 332
Steuri, R., Sattelmayer, M., Elsig, S., Kolly, C., Tal, A., Taeymans, J., & Hilfiker, R. (2017) · British Journal of Sports Medicine · doi:10.1136/bjsports-2016-096515 · Cited by 312
“59 Therefore, we decided to include all studies and to perform a sensitivity analysis to evaluate the influence of the high risk of bias studies.”
MethodPredictors of the Onset of Cigarette Smoking
Wellman, R. J., Dugas, E. N., Dutczak, H., O’Loughlin, E. K., Datta, G. D., Lauzon, B., & O’Loughlin, J. (2016) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2016.04.003 · Cited by 230
Breast and Bottle Feeding as Risk Factors for Dental Caries: A Systematic Review and Meta-Analysis
Avila, W. M., Pordeus, I. A., Paiva, S. M., & Martins, C. C. (2015) · PLOS ONE · doi:10.1371/journal.pone.0142922 · Cited by 175
“The process was made complex [41] due to the heterogeneity of studies and differences in feeding habits and dental caries classification.”
BackgroundPredictors of smoking cessation during pregnancy: a systematic review and meta‐analysis
Riaz, M., Lewis, S., Naughton, F., & Ussher, M. (2018) · Addiction · doi:10.1111/add.14135 · Cited by 166
“has known limitations, such as it often does not inform the evidence synthesis process, particularly for systematic reviews of non-randomised studies, including sensitivity analysis, narrative assessment and restricting the synthesis to studies at a lower risk of bias, which have been discussed elsewhere in some detai…”
BackgroundEffectiveness of Mass Media Campaigns to Reduce Alcohol Consumption and Harm: A Systematic Review
Young, B., Lewis, S., Katikireddi, S. V., Bauld, L., Stead, M., Angus, K., Campbell, M., Hilton, S., Thomas, J., Hinds, K., et al. (2018) · Alcohol and Alcoholism · doi:10.1093/alcalc/agx094 · Cited by 164
“We privilege studies with high quality within the narrative synthesis (Katikireddi et al., 2015).”
Self-citationDufour, I., Chouinard, M.-C., Dubuc, N., Beaudin, J., Lafontaine, S., & Hudon, C. (2019) · BMC Geriatrics · doi:10.1186/s12877-019-1197-9 · Cited by 128
Ronzi, S., Orton, L., Pope, D., Valtorta, N. K., & Bruce, N. G. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0680-2 · Cited by 112
“RoB was discussed in each narrative summary [58].”
BackgroundMethodLi, M., & Katikireddi, S. V. (2019) · International Journal for Equity in Health · doi:10.1186/s12939-018-0881-2 · Cited by 104
“While we did assess risk of bias and consider this in the synthesis process [20], there remains nevertheless the possibility that some comparisons reflect methodological differences in study design or data sources, rather than genuine differences.”
BackgroundMethodSelf-citationBarnish, M. S., & Barran, S. M. (2020) · BMC Neurology · doi:10.1186/s12883-020-01938-3 · Cited by 123
Gebrekidan, T. K., Weldemariam, N. G., Hidru, H. D., Gebremedhin, G. G., & Weldemariam, A. K. (2024) · Science in One Health · doi:10.1016/j.soh.2024.100081 · Cited by 111
Methods for Developing Evidence Reviews in Short Periods of Time: A Scoping Review
Abou-Setta, A. M., Jeyaraman, M., Attia, A., Al-Inany, H. G., Ferri, M., Ansari, M. T., Garritty, C. M., Bond, K., & Norris, S. L. (2016) · PLOS ONE · doi:10.1371/journal.pone.0165903 · Cited by 83
Schwingshackl, L., Schünemann, H. J., & Meerpohl, J. J. (2021) · European Journal of Nutrition · doi:10.1007/s00394-020-02464-1 · Cited by 71
“A methodology study evaluating 59 SRs showed that only 50% of these SRs conducted sensitivity analyses for low RoB studies [49].”
MethodChiarotto, A., Terwee, C. B., Kamper, S. J., Boers, M., & Ostelo, R. W. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.05.006 · Cited by 69
“Two were narrative reviews [16,17], two focused on utility scores [15,17], and all had significant and important methodological weaknesses, such as failure to account for risk of bias in the evidence synthesis [18,19].”
BackgroundCampbell, J. M., Stephenson, M., Bateman, E. H., Peters, M. D. J., Keefe, D. M. K., & Bowen, J. M. (2016) · The Pharmacogenomics Journal · doi:10.1038/tpj.2016.58 · Cited by 68
Bilandzic, A., Fitzpatrick, T., Rosella, L., & Henry, D. (2016) · PLOS Medicine · doi:10.1371/journal.pmed.1001987 · Cited by 60
“found that the majority of 59 systematic reviews published between March and May 2012 included some form of critical appraisal of the included studies [62].”
BackgroundApplying the ROBINS-I tool to natural experiments: an example from public health
Thomson, H., Craig, P., Hilton-Boon, M., Campbell, M., & Katikireddi, S. V. (2018) · Systematic Reviews · doi:10.1186/s13643-017-0659-4 · Cited by 60
“All research is susceptible to bias, and any conclusions or lessons should be considered in light of any identified bias or limitations [3].”
BackgroundSelf-citationBüttner, F., Winters, M., Delahunt, E., Elbers, R., Lura, C. B., Khan, K. M., Weir, A., & Ardern, C. L. (2020) · British Journal of Sports Medicine · doi:10.1136/bjsports-2019-101675 · Cited by 55
“41 Kosik et al 16 reported patient-reported function in ADL and sporting activities.”
Abbott, R., Bethel, A., Rogers, M., Whear, R., Orr, N., Shaw, L., Stein, K., & Thompson Coon, J. (2022) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2021-111710 · Cited by 56
“26 27 We have in addition demonstrated the potential for increasing research waste”
Gates, A., Vandermeer, B., & Hartling, L. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.12.015 · Cited by 60
Goagoses, N., & Koglin, U. (2019) · Springer eBooks · doi:10.1007/978-3-658-27602-7_9 · Cited by 59
Movsisyan, A., Melendez-Torres, G. J., & Montgomery, P. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.03.014 · Cited by 50
“Meanwhile, those studies most frequently provide the ‘‘best possible’’ evidence for many complex interventions [75,79].”
BackgroundDíaz‐Sánchez, M., Soto‐Peñaloza, D., Peñarrocha‐Oltra, D., & Peñarrocha‐Diago, M. (2019) · Journal of Periodontal Research · doi:10.1111/jre.12663 · Cited by 55
Pollak, I., Mitić, M., Birchwood, J., Dörfler, S., Krammer, I., Rogers, J. C., Schek, E. J., Schrank, B., Stiehl, K. A. M., & Woodcock, K. A. (2022) · Adolescent Research Review · doi:10.1007/s40894-022-00195-4 · Cited by 56
Aramburú, A., Alvarado‐Gamarra, G., Cornejo, R., Curi-Quinto, K., Díaz-Parra, C. D. P., Rojas-Limache, G., & Lanata, C. F. (2024) · Frontiers in Nutrition · doi:10.3389/fnut.2024.1421728 · Cited by 48
“The results of the risk of bias assessment were considered in the analysis of evidence certainty, discussed narratively in the text, and considered when formulating the study’s overall conclusions (22).”
BackgroundA mapping review of take-home naloxone for people released from correctional settings
Horton, M., McDonald, R., Green, T. C., Nielsen, S., Strang, J., Degenhardt, L. J., & Larney, S. (2017) · International Journal of Drug Policy · doi:10.1016/j.drugpo.2017.05.015 · Cited by 47
Gates, N. J., & March, E. G. (2016) · Neuropsychology Review · doi:10.1007/s11065-016-9318-0 · Cited by 46
Sharma, S., & Oremus, M. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.03.012 · Cited by 44
Detweiler, B., Kollmorgen, L., Umberham, B. A., Hedin, R. J., & Vassar, M. B. (2016) · Anaesthesia · doi:10.1111/anae.13520 · Cited by 44
“Ideally, a systematic review would only include well-designed primary studies, but authors who choose to include low methodological quality/high risk of bias primary studies in systematic reviews should consider using meta-regression, subgroup or sensitivity analyses, if appropriate, to present the possible influence…”
BackgroundInfluentialLee, J. Y., Patel, M., & Scior, K. (2023) · Journal of Intellectual Disability Research · doi:10.1111/jir.13025 · Cited by 45
‘Bumping along’: a qualitative metasynthesis of challenges to interprofessional placements
O’Leary, N., Salmon, N., Clifford, A. M., O’Donoghue, M., & Reeves, S. (2019) · Medical Education · doi:10.1111/medu.13891 · Cited by 44
Stanhope, J., & Weinstein, P. (2022) · Research Synthesis Methods · doi:10.1002/jrsm.1609 · Cited by 42
Petersen, J. M., Barrett, M., Ahrens, K. A., Murray, E. J., Bryant, A., HOGUE, C. J. R., Mumford, S. L., Gadupudi, S. S., Fox, M. P., & Trinquart, L. (2021) · Research Synthesis Methods · doi:10.1002/jrsm.1544 · Cited by 36
“Moreover, it is insufficient to simply describe whether control was adequate; incorporation of such assessments into meta-analyses is necessary for drawing valid conclusions from the best available observational evidence.(7,8) Most systematic reviews of observational studies do not define the most important confounder…”
BackgroundFranklin, C., Guz, S., Zhang, A., Kim, J., Zheng, H., Hai, A. H., Cho, Y. J., & Shen, L. (2020) · Journal of the Society for Social Work and Research · doi:10.1086/712169 · Cited by 50
“We incorporated publication bias into the final synthesis using a sensitivity analysis framework (Katikireddi et al., 2015).”
MethodClare, A., & Camic, P. M. (2019) · Arts & Health · doi:10.1080/17533015.2019.1675732 · Cited by 35
Grant, S., Pedersen, E. R., Osilla, K. C., Kulesza, M., & D’Amico, E. J. (2015) · Addiction · doi:10.1111/add.13136 · Cited by 34
“Without details on decisions leading to qualityof-evidence ratings, we cannot assess clearly whether conclusions are imprecise or even misleading [36], as the”
BackgroundEmmerson, K. B., Harding, K. E., & Taylor, N. F. (2019) · Clinical Rehabilitation · doi:10.1177/0269215518819706 · Cited by 32
Lowe, C., Barrett, E., McCreesh, K. M., Búrca, N. D., & Lewis, J. S. (2019) · Journal of Rehabilitation Medicine · doi:10.2340/16501977-2578 · Cited by 32
A systematic review of the role of dialogic leadership: Characterization and impacts
Khaqan, S., & Redondo-Sama, G. (2024) · Educational Research Review · doi:10.1016/j.edurev.2024.100618 · Cited by 31
Sheehan, M. C., Lam, J., Navas‐Acién, A., & Chang, H. H. (2016) · Environment International · doi:10.1016/j.envint.2016.02.016 · Cited by 30
“Finally, a recent review of risk-of-bias assessments in epidemiological studies found assessment conclusions were often poorly integrated into study findings (Katikireddi et al., 2015).”
BackgroundElliott, A., & Gibson, S. (2022) · Journal of Human Nutrition and Dietetics · doi:10.1111/jhn.13063 · Cited by 31
Sheehan, M. C., & Lam, J. (2015) · Current Environmental Health Reports · doi:10.1007/s40572-015-0062-z · Cited by 31
Maeda, Y., Caskurlu, S., Kozan, K., & Kenney, R. H. (2022) · Quality & Quantity · doi:10.1007/s11135-022-01403-y · Cited by 31
“The use of critical appraisal tools like the one we have described here can differ by the type of qualitative synthesis techniques and synthesis studies applied (Gough et al. 2012; Katikireddi et al. 2015; Pawson et al. 2005).”
BackgroundMethodKranenburg, R., Tyer, R., Schmitt, M. A., Luijckx, G. J., van der Schans, C. P., Hutting, N., & Kerry, R. (2019) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.2019.8578 · Cited by 30
Area deprivation measures used in Brazil: a scoping review
Ichihara, M. Y., Ramos, D. D. O., Rebouças, P., Oliveira, F. J., Ferreira, A. J. F., Teixeira, C. S. S., Allik, M., Katikireddi, S. V., Barreto, M. L., Leyland, A. H., et al. (2018) · Revista de Saúde Pública · doi:10.11606/s1518-8787.2018052000933 · Cited by 27
Self-citationLiu, H., Muhunthan, J., Hayek, A., Hackett, M. L., Laba, T., Peiris, D., & Jan, S. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0314-5 · Cited by 27
“Instead, the quality of the studies is presented as a risk of bias graph (low, unclear and high risk) [41].”
BackgroundVeroniki, A., Jackson, D., Viechtbauer, W., Bender, R., Knapp, G., Kuss, O., & Langan, D. (2015) · Lirias · Cited by 34
Kim, S., Kim, H., An, J., Lee, Y., & Shin, Y. S. (2022) · Worldviews on Evidence-Based Nursing · doi:10.1111/wvn.12570 · Cited by 24
Morche, J., Freitag, S., Hoffmann, F., Rissling, O., Langer, G., Nußbaumer-Streit, B., Toews, I., Gartlehner, G., & Meerpohl, J. J. (2020) · Zeitschrift für Evidenz Fortbildung und Qualität im Gesundheitswesen · doi:10.1016/j.zefq.2019.11.003 · Cited by 23
Higgins, S., Katsipataki, M., Aguilera, A. B. V., Dobson, E., Gascoine, L., Rajab, T., Kalambouka, A., Reardon, J., Stafford, J., & Uwimpuhwe, G. (2022) · Review of Education · doi:10.1002/rev3.3327 · Cited by 27
Xun, Y., Guo, Q., Ren, M., Liu, Y., Sun, Y., Wu, S., Lan, H., Zhang, J., Liu, H., Wang, J., et al. (2023) · Frontiers in Public Health · doi:10.3389/fpubh.2023.998588 · Cited by 22
“Our finding that approximately 70% of SRs evaluated the quality of the included primary studies is lower than what was found by a previous survey of medical journals, reporting that 90% of SRs evaluated the quality of included studies (30).”
ResultTrends in Methodological Quality in Controlled Trials of Psychological and Social Interventions
Sundell, K., & Åhsberg, E. (2016) · Research on Social Work Practice · doi:10.1177/1049731516633297 · Cited by 18
Muchandifunga Trust Muchadeyi, Hernández-Villafuerte, K., & Schlander, M. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01784-6 · Cited by 16
“In these fields, the high prevalence is in part linked to the availability of standardised QA tools and the presence of generally accepted standards [15, 30].”
BackgroundMethodResultInfluentialMelani, M., Paiva, J. M., Silva, M. C., Mendlowicz, M. V., Figueira, I. L. D. V., Marques‐Portella, C., Luz, M. P., Ventura, P. R., & Berger, W. L. (2020) · Depression and Anxiety · doi:10.1002/da.23078 · Cited by 17
Setting Inclusion and Exclusion Criteria
Dekkers, R., Carey, L., & Langhorne, P. (2022) · Springer eBooks · doi:10.1007/978-3-030-90025-0_6 · Cited by 18
Canvas Framework for Performing Systematic Reviews Analysis
Almeida, F. L. (2018) · Multidisciplinary Journal for Education Social and Technological Sciences · doi:10.4995/muse.2018.9832 · Cited by 14
Cheurfa, C., Tsokani, S., Kontouli, K., Boutron, I., & Chaimani, A. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.02.008 · Cited by 13
Maden, M., & Kotas, E. (2016) · Evidence Based Library and Information Practice · doi:10.18438/b8f630 · Cited by 13
Bo, A., Hai, A. H., Chen, D., & Hammock, K. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.06.012 · Cited by 12
Quality Assessment Errors and Study Misclassification Threaten Systematic Review Validity
Orkin, A. M., Bingham, K., Buick, J. E., Klaiman, M., Leece, P., & Kouyoumdjian, F. G. (2015) · Journal of Addiction Medicine · doi:10.1097/adm.0000000000000161 · Cited by 12
Donohue, C., Carnaby, G., & Garand, K. L. (2021) · American Journal of Speech-Language Pathology · doi:10.1044/2021_ajslp-21-00004 · Cited by 11
Zeraatkar, D., Kohut, A., Bhasin, A. A., Morassut, R. E., Churchill, I. F., Gupta, A. A., Lawson, D. O., Miroshnychenko, A., Sirotich, E., Aryal, K., et al. (2021) · BMJ Nutrition Prevention & Health · doi:10.1136/bmjnph-2021-000248 · Cited by 17
Gebrekidan, T. K., Gebremariam, M. G., Teferi, M. K., & Yebyo, S. G. (2025) · Watershed Ecology and the Environment · doi:10.1016/j.wsee.2025.01.002 · Cited by 11
Maranda, M. J., Lee-Easton, M. J., & Magura, S. (2022) · Evaluation Review · doi:10.1177/0193841x221100356 · Cited by 10
Kangaroo Mother Care in Term and Late Preterm Neonates: A Systematic Review
Gajula, R., Kankanala, G. R., Mutukulla, R., & Kotha, R. (2024) · Cureus · doi:10.7759/cureus.60958 · Cited by 8
Jenkins, C., Lowe, C., & Barker, K. L. (2022) · Physiotherapy · doi:10.1016/j.physio.2022.05.003 · Cited by 8
Critical Appraisal: Assessing the Quality of Studies
Purssell, E., & McCrae, N. (2020) · Springer eBooks · doi:10.1007/978-3-030-49672-2_6 · Cited by 8
Evaluation of Spin in the Abstracts of Systematic Reviews and Meta-Analyses Focused on Tinnitus
Rucker, B., Umbarger, E. M., Ottwell, R., Arthur, W., Brame, L. S., Woodson, E., Wright, D., Hartwell, M. L., Khojasteh, J., & Vassar, M. B. (2021) · Otology & Neurotology · doi:10.1097/mao.0000000000003178 · Cited by 7
Shapland, C. Y., Bell, J. A., Borges, M. C., Soares, A. G., Smith, G. D., Hemani, Lawlor, D. A., McGuinness, L. A., Tilling, K. M., & Higgins, J. P. T. (2026) · European Journal of Epidemiology · doi:10.1101/2024.09.20.24314046 · Cited by 10
“Even though imposing prior distributions on bias is challenging, failure to acknowledge bias and assuming all effect estimates are unbiased is even more problematic and may lead to false conclusions, as seems to have been the case in some systematic reviews of RCTs 42 .”
ResultWong, L. Y. F., Sutcliffe, A. G., Ho, C. L. T., Lu, Y., Williams, C. L., Afzal, F., & Purkayastha, M. (2025) · Frontiers in Pharmacology · doi:10.3389/fphar.2025.1568320 · Cited by 6
Rahna, K., Shamim, M. A., Valappil, H. C., Subramanian, J., Sharma, G. A., & Padhi, B. K. (2024) · International Journal for Equity in Health · doi:10.1186/s12939-024-02275-4 · Cited by 8
“Prede-fined inclusion criteria minimize bias arising from the selective consideration of evidence [33].”
BackgroundMethodWang, Z., Wang, Y., Shang, W., Liu, W., Lu, C., Huang, J., Lei, C., Chen, Z., Yang, K., Yang, K., et al. (2023) · European Journal of Clinical Nutrition · doi:10.1038/s41430-023-01383-8 · Cited by 6
“…on these fi ndings, all stakeholders (investigators of original studies and SRs, editors, and peer reviewers) must make efforts to improve the quality and strengthen the robustness of future relevant SRs, further underpinning the evidence base for supporting clinical decisions and health policies.”
Reconsidering what makes syntheses of psychological intervention studies useful
Sakaluk, J. K., Santis, C. D., Kilshaw, R. E., Pittelkow, M.-M., Brandes, C. M., Boness, C. L., Botanov, Y., Williams, A. J., Wendt, D. C., Lorenzo‐Luaces, L., et al. (2023) · Nature Reviews Psychology · doi:10.1038/s44159-023-00213-9 · Cited by 5
Lensen, S. F., Wilkinson, J., Steeper, M., Melo, P., Kamath, M. S., Craciunas, L., Wang, R., Jordan, V., Showell, M. G., Attinger, S. A., et al. (2026) · The Lancet Obstetrics Gynaecology & Women s Health · doi:10.1016/s3050-5038(26)00054-3 · Cited by 4
Muchandifunga Trust Muchadeyi, Hernández-Villafuerte, K., Tanna, G. L. D., Eckford, R. D., Feng, Y., Meregaglia, M., Peasgood, T., Petrou, S., Ubels, J., & Schlander, M. (2024) · PharmacoEconomics · doi:10.1007/s40273-024-01365-z · Cited by 4
“…quality dimensions presented in the first-round questionnaire were crafted by the steering committee, drawing upon the prior literature [15–17 , To comprehensively capture the experts’ opinions and insights, a combination of a five-point Likert rating scale (with a range from strongly…”
Shereefdeen, H., Thaivalappil, A., Young, I., & MacKay, M. (2025) · INQUIRY The Journal of Health Care Organization Provision and Financing · doi:10.1177/00469580251399374 · Cited by 3
“35,37 Thus, there is a need to revisit the use of genAI to conduct quality appraisal and further explore its use and effectiveness in conducting reporting checklists for qualitative studies.”
Historical climate variability extremes and socio-environmental impacts in Ethiopia
Gebrekidan, T. K., & Kahsay, H. T. (2026) · Discover Environment · doi:10.1007/s44274-026-00749-6 · Cited by 3
“For grey literature, a modified critical appraisal checklist was used [54], assessing institutional credibility, methodological transparency, data quality, and clarity of reporting.”
Li, X., Tao, K., Yang, G., Zhou, Q.-H., & Ling, C. (2015) · Acupuncture in Medicine · doi:10.1136/acupmed-2015-010876 · Cited by 3
Menne, M. C., Su, N., & Faggion, C. M. (2023) · Irish Veterinary Journal · doi:10.1186/s13620-023-00261-w · Cited by 3
“It is therefore important use the results of the RoB assessment to critically appraise the results of primary studies and put them into context of each other [42].”
MethodComparative efficacy of flapped versus flapless dental implant procedures: A meta-analysis.
Jain, P., Jain, M., Sharma, C., Gaikwad, R. N., Porwal, A., Barman, D., Talukdar, R., & Rai, N. (2024) · PubMed Central · Cited by 3
Mei Ran Abellona, U., Shen, E. Y.-L., Cartlidge, C., Alkhatib, A., Thursz, M. R., Waked, I. A., Gomaa, A. I., Holmes, E. C., Sharma, R., & Taylor-Robinson, S. D. (2022) · Cancer Epidemiology Biomarkers & Prevention · doi:10.1158/1055-9965.epi-21-0687 · Cited by 3
Tonia, T., Buitrago‐García, D., Peter, N. L., Mesa-Vieira, C., Li, T., Furukawa, T. A., Cipriani, A., Leucht, S., Low, N. N., & Salanti, G. (2023) · medRxiv · doi:10.1101/2023.02.01.23285335 · Cited by 3
“3 the risk of bias (RoB) of the included studies (6,7) because the potential biases affect how 1 certain we are about the included evidence and its interpretation (8,9).”
BackgroundChandran, A., Veldhuizen, S., Mehra, K., Rodak, T., Vagharfard, D., Pham, M., Zawertailo, L. A., Rehm, J. T., Hendershot, C. S., Selby, P., et al. (2025) · Health Psychology and Behavioral Medicine · doi:10.1080/21642850.2025.2554182 · Cited by 2
“To enhance the validity of our synthesis, we prioritised findings from studies with robust methodological quality by including only those assessed as having a medium or low risk of bias, a practice endorsed by others (Katikireddi et al., 2015; Wood et al., 2008).”
Lee-Easton, M. J., & Magura, S. (2023) · INQUIRY The Journal of Health Care Organization Provision and Financing · doi:10.1177/00469580231186836 · Cited by 2
Kamioka, H., Kitayuguchi, J., Origasa, H., & Tsutani, K. (2024) · Nutrients · doi:10.3390/nu16162744 · Cited by 2
“For example, a previous study that evaluated 59 SRs reported that only 50% of the SRs performed sensitivity analyses for low RoB CTs [23].”
ResultBegum, R., Rushton, A., Chamaa, A. E., Walton, D. M., & Parikh, P. (2025) · PLoS ONE · doi:10.1371/journal.pone.0316827 · Cited by 2
Griffiths, P. G., Taylor, R. H., Henderson, L. M., & Barrett, B. T. (2017) · Journal of Optometry · doi:10.1016/j.optom.2016.11.004 · Cited by 2
Martinou, E., & Angelidi, A. M. (2022) · F1000Research · doi:10.12688/f1000research.127179.1 · Cited by 1
Critical Appraisal: Assessing the Quality of Evidence
Purssell, E., & McCrae, N. (2024) · Springer eBooks · doi:10.1007/978-3-031-71159-6_5 · Cited by 1
Inter-rater reliability and usability of CATHIS core for homeopathic intervention studies
Loef, M., Weiermayer, P., Gaertner, K., Wrzałko, D., Dutta, A., Saha, S., Lakshani, J. A. D. S., Baumgartner, S., & van Haselen, R. A. (2026) · Complementary Therapies in Medicine · doi:10.1016/j.ctim.2026.103338 · Cited by 1
Development of Reporting Guidelines for Systematic Review for Environmental Epidemiology Studies
Lee, K. J. (2021) · uO Research (University of Ottawa) · doi:10.20381/ruor-25825 · Cited by 2
Patil, S. R., & Bhide, S. S. (2021) · Perspectives in Clinical Research · doi:10.4103/picr.picr_19_21 · Cited by 1
“RoB is concerned with internal validity as it evaluates risk of methodological errors based on available information.[1] Data generated from clinical trials can be offset from the truth due to multiple factors.”
BackgroundSnider, K., Moore, T. M., Walters, C. G., Brachtenbach, T., Woods, W., Hartwell, M. L., Torgerson, T., Rauh, S. L., & Vassar, M. B. (2022) · Clinical Breast Cancer · doi:10.1016/j.clbc.2022.04.009 · Cited by 1
Metadata
- Authors
- Katikireddi, S. V., Egan, M., & Petticrew, M.
- Affiliations
- MRC/CSO Social and Public Health Sciences Unit (GB)
- London School of Hygiene & Tropical Medicine (GB)
- Corresponding author
- Katikireddi, Srinivasa Vittal
- Venue
- Journal of Epidemiology and Community Health
- Publisher
- BMJ
- Published
- 2014-12-06
- Volume
- 69
- Issue
- 2
- Pages
- 189-195
- ISSN
- 0141-7681 (ISSN-L)
- 0143-005X
- 1470-2738
- PMC
- PMC4316857
- OpenAlex
- W2098060885
- Cited by
- 126
- References
- 43
- Citations by year
- 2026: 11 · 2025: 9 · 2024: 17 · 2023: 11 · 2022: 13 · 2021: 9 · 2020: 6 · 2019: 12 · 2018: 11 · 2017: 6 · 2016: 12 · 2015: 9
- FWCI
- 2.48
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Journal of Epidemiology & Community Health (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- ENLIGHTEN (Jurnal Bimbingan dan Konseling Islam) (acceptedVersion)
- LSHTM Research Online (London School of Hygiene and Tropical Medicine) (acceptedVersion)
- Funders
- London School of Hygiene and Tropical Medicine
- Medical Research Council (MC_UU_12017/15)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- systematic reviews, risk of bias, critical appraisal, evidence synthesis, sensitivity analysis, randomized controlled trials, public health research, reporting quality
- MeSH terms
- Humans, Research Design, Review Literature as Topic, Bias, Risk Assessment, Evidence-Based Medicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
