Improving the trustworthiness of findings from nutrition evidence syntheses: assessing risk of bias and rating the certainty of evidence
Open accessThe DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Abstract
Suboptimal diet is recognized as a leading modifiable risk factor for non-communicable diseases. Non-randomized studies (NRSs) with patient relevant outcomes provide many insights into diet–disease relationships. Dietary guidelines are based predominantly on findings from systematic reviews of NRSs—mostly prospective observational studies, despite that these have been repeatedly criticized for yielding potentially less trustworthy results than randomized controlled trials (RCTs). It is assumed that these are a result of bias due to prevalent-user designs, inappropriate comparators, residual confounding, and measurement error. In this article, we aim to highlight the importance of applying risk of bias (RoB) assessments in nutritional studies to improve the credibility of evidence of systematic reviews. First, we discuss the importance and challenges of dietary RCTs and NRSs, and provide reasons for potentially less trustworthy results of dietary studies. We describe currently used tools for RoB assessment (Cochrane RoB, and ROBINS-I), describe the importance of rigorous RoB assessment in dietary studies and provide examples that further the understanding of the key issues to overcome in nutrition research. We then illustrate, by comparing the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) approach with current approaches used by United States Department of Agriculture Dietary Guidelines for Americans, and the World Cancer Research Fund, how to establish trust in dietary recommendations. Our overview shows that the GRADE approach provides more transparency about the single domains for grading the certainty of the evidence and the strength of recommendations. Despite not increasing the certainty of evidence itself, we expect that the rigorous application of the Cochrane RoB and the ROBINS-I tools within systematic reviews of both RCTs and NRSs and their integration within the GRADE approach will strengthen the credibility of dietary recommendations.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
RoB 2: a revised tool for assessing risk of bias in randomised trials
Sterne, J. A. C., Savović, J., Page, M. J., Elbers, R. G., Blencowe, N. S., Boutron, I., Cates, C. J., Cheng, H.-Y., Corbett, M. S., Eldridge, S. M., et al. (2019) · BMJ · doi:10.1136/bmj.l4898 · Cited by 34,787
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040296 · Cited by 23,020
ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions
Sterne, J. A., Hernán, M. A., Reeves, B. C., Savović, J., Berkman, N. D., Viswanathan, M., Henry, D., Altman, D. G., Ansari, M. T., Boutron, I., et al. (2016) · BMJ · doi:10.1136/bmj.i4919 · Cited by 20,329
Self-citationRoth, G. A., Abate, D., Abate, K. H., Abay, S. M., Abbafati, C., Abbasi, N., Abbastabar, H., Abd-Allah, F., Abdela, J., Abdelalim, A., et al. (2018) · The Lancet · doi:10.1016/s0140-6736(18)32203-7 · Cited by 8,600
Afshin, A., Sur, P. J., Fay, K. A., Cornaby, L., Ferrara, G., Salama, J. S., Mullany, E. C., Abate, K. H., Abbafati, C., Abebe, Z., et al. (2019) · The Lancet · doi:10.1016/s0140-6736(19)30041-8 · Cited by 5,811 · cited 1×
“According to the Global Burden of Disease studies, suboptimal diet is the leading risk factor for ~ 50% of disabilities from cardiovascular diseases [2].”
MethodNaghavi, M., Abajobir, A. A., Abbafati, C., Abbas, K. M., Abd-Allah, F., Abera, S. F., Aboyans, V., Adetokunboh, O., Afshin, A., Agrawal, A., et al. (2017) · The Lancet · doi:10.1016/s0140-6736(17)32152-9 · Cited by 4,821
Comparison of Weight-Loss Diets with Different Compositions of Fat, Protein, and Carbohydrates
Sacks, F. M., Bray, G. A., Carey, V. J., Smith, S. R., Ryan, D. H., Anton, S. D., McManus, K., Champagne, C. M., Bishop, L. M., Laranjo, N., et al. (2009) · New England Journal of Medicine · doi:10.1056/nejmoa0804748 · Cited by 1,966 · cited 1×
“Dietary adherence is probably the most important limitation of dietary RCTs, mainly in long-term RCTs [46].”
BackgroundVitamin E Supplementation and Cardiovascular Events in High-Risk Patients
(2000) · New England Journal of Medicine · doi:10.1056/nejm200001203420302 · Cited by 1,957
Dansinger, M. L., Gleason, J. A., Griffith, J. L., Selker, H. P., & Schaefer, E. J. (2005) · JAMA · doi:10.1001/jama.293.1.43 · Cited by 1,726 · cited 1×
“Difficulty of attrition (in free living populations, a 40–50% dropout rate is fairly common [47]) is mainly observed in longer-term dietary RCTs [48].”
BackgroundSanderson, S., Tatt, I. D., & Higgins, J. P. (2007) · International Journal of Epidemiology · doi:10.1093/ije/dym018 · Cited by 1,531 · cited 1×
“Focusing on NRSs, a SR identified 86 tools to assess study quality in NRSs showing high inconsistency in selection/inclusion and weighting of domains across tools [32].”
MethodThe Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta‐analyses
IOANNIDIS, J. P. A. (2016) · The Milbank Quarterly · doi:10.1111/1468-0009.12210 · Cited by 1,483 · cited 1×
“This can be done by exploring the nature of potentially less trustworthy results, a process that is often omitted, inconsistently applied across studies, or flawed [28].”
Backgroundde Souza, R. J., Mente, A., Maroleanu, A., Cozma, A. I., Ha, V., Kishibe, T., Uleryk, E., Budylowski, P., Schünemann, H., Beyene, J., et al. (2015) · BMJ · doi:10.1136/bmj.h3978 · Cited by 1,424 · cited 1×
“GRADE is also used in the field of nutrition research [53].”
MethodSelf-citationAlonso-Coello, P., Schünemann, H. J., Moberg, J., Brignardello-Petersen, R., Akl, E. A., Davoli, M., Treweek, S., Mustafa, R. A., Rada, G., Rosenbaum, S., et al. (2016) · BMJ · doi:10.1136/bmj.i2016 · Cited by 1,317
Self-citationBjelakovic, G., Nikolova, D., Gluud, L. L., Simonetti, R. G., & Gluud, C. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd007176.pub2 · Cited by 924
Schünemann, H. J., Cuello, C., Akl, E. A., Mustafa, R. A., Meerpohl, J. J., Thayer, K., Morgan, R. L., Gartlehner, G., Kunz, R., Katikireddi, S. V., et al. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.01.012 · Cited by 911
Self-citationThe GRADE Working Group clarifies the construct of certainty of evidence
Hultcrantz, M., Rind, D., Akl, E. A., Treweek, S., Mustafa, R. A., Iorio, A., Alper, B. S., Meerpohl, J. J., Murad, M. H., Ansari, M. T., et al. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.05.006 · Cited by 853 · cited 2×
“For example, high certainty of evidence is defined as: “High certainty that a true effect lies on one side of a specified threshold or within a chosen range” [5].”
BackgroundSelf-citationDiets with high or low protein content and glycemic index for weight-loss maintenance
Larsen, T. M., Dalskov, S.-M., van Baak, M., Jebb, S. A., Papadaki, A., Pfeiffer, A. F. H., Martinez, J. A., Handjieva-Darlenska, T., Kunešová, M., Pihlsgård, M., et al. (2010) · New England Journal of Medicine · doi:10.1056/nejmoa1007137 · Cited by 843
Dekkers, O. M., Vandenbroucke, J. P., Cevallos, M., Renehan, A. G., Altman, D. G., & Egger, M. (2019) · PLOS Medicine · doi:10.1371/journal.pmed.1002742 · Cited by 525 · cited 1×
“The COSMOS-E reporting guideline (Conducting Systematic Reviews and MetaAnalyses of Observational Studies of Etiology) has recently been published; COSMOS-E also recommends the use of the ROBINS tool to evaluate RoB in observational studies, and the GRADE approach to rate the certainty of evidence [43].”
MethodSpringmann, M., Spajic, L., Clark, M. A., Poore, J., Herforth, A., Webb, P., Rayner, M., & Scarborough, P. (2020) · BMJ · doi:10.1136/bmj.m2322 · Cited by 520
The GRADE Evidence to Decision (EtD) framework for health system and public health decisions
for the GRADE Working Group, Moberg, J., Oxman, A. D., Rosenbaum, S., Schünemann, H. J., Guyatt, G., Flottorp, S., Glenton, C., Lewin, S., Morelli, A., et al. (2018) · Health Research Policy and Systems · doi:10.1186/s12961-018-0320-2 · Cited by 376
Self-citationSchwingshackl, L., Knüppel, S., Schwedhelm, C., Hoffmann, G., Missbach, B., Stelmach-Mardas, M., Dietrich, S., Eichelmann, F., Kontopanteils, E., Iqbal, K., et al. (2016) · Advances in Nutrition · doi:10.3945/an.116.013052 · Cited by 360
Self-citationThe GRADE approach and Bradford Hill's criteria for causation
Schünemann, H., Hill, S., Guyatt, G., Akl, E. A., & Ahmed, F. (2011) · Journal of Epidemiology and Community Health · doi:10.1136/jech.2010.119933 · Cited by 262
Self-citationMorgan, R. L., Thayer, K. A., Santesso, N., Holloway, A. C., Blain, R., Eftim, S. E., Goldstone, A. E., Ross, P., Ansari, M., Akl, E. A., et al. (2019) · Environment International · doi:10.1016/j.envint.2018.11.004 · Cited by 248 · cited 1×
“In their user’s guide, the authors applied the draft ROBINS-(exposure) tool successfully, to a variety of study designs including prospective cohort studies and cross-sectional studies [40]; ROBINS-E was also recently used to evaluate RoB in case–control studies [41].”
MethodSelf-citationAkl, E. A., Sun, X., Busse, J. W., Johnston, B. C., Briel, M., Mulla, S., You, J. J., Bassler, D., Lamontagne, F., Vera, C., et al. (2012) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.04.015 · Cited by 179
The Effect of Keeping Food Records on Eating Patterns
REBRO, S. M., PATTERSON, R. E., KRISTAL, A. R., & CHENEY, C. L. (1998) · Journal of the American Dietetic Association · doi:10.1016/s0002-8223(98)00269-7 · Cited by 141 · cited 1×
“Across dietary NRSs, social desirability biases are prevalent: Participants may give perceived “healthy” responses, such as over-reporting fruit and vegetable intake or underestimating fat intake [25], whereas obese patients are more likely to underreport nutritional intake, particularly energy, which can lead to the…”
BackgroundKatikireddi, S. V., Egan, M., & Petticrew, M. (2015) · Journal of Epidemiology and Community Health · doi:10.1136/jech-2014-204711 · Cited by 126 · cited 1×
“A methodology study evaluating 59 SRs showed that only 50% of these SRs conducted sensitivity analyses for low RoB studies [49].”
MethodMorgan, R. L., Thayer, K. A., Santesso, N., Holloway, A. C., Blain, R., Eftim, S. E., Goldstone, A. E., Ross, P., Guyatt, G., & Schünemann, H. J. (2018) · Environment International · doi:10.1016/j.envint.2018.08.018 · Cited by 106
Self-citationFool’s gold? Why blinded trials are not always best
Anand, Norrie, Bradley, McAuley, & Clarke (2020) · BMJ · doi:10.1136/bmj.l6228 · Cited by 58 · cited 1×
“In this Cochrane Review, the authors took into account the difficulties of blinding participants (although, double blinded designs are not always ideal for providing a reliable answer to the trial’s research question [44]) in dietary interventions and rated this as unclear rather than high RoB [45].”
Background
Cited by
Dybvik, J. S., Svendsen, M., & Aune, D. (2023) · European Journal of Nutrition · doi:10.1007/s00394-022-02942-8 · Cited by 150
Ultra‐Processed Food and Obesity: What Is the Evidence?
Dicken, S. J., & Batterham, R. L. (2024) · Current Nutrition Reports · doi:10.1007/s13668-024-00517-z · Cited by 133
Selinger, E., Neuenschwander, M., Koller, A., Gojda, J., Kühn, T., Schwingshackl, L., Barbaresko, J., & Schlesinger, S. (2023) · Critical Reviews in Food Science and Nutrition · doi:10.1080/10408398.2022.2075311 · Cited by 114
“Overall using GRADE will strengthen the credibility of potential dietary recommendations (Schwingshackl, Schunemann, and Meerpohl 2021).”
MethodSelf-citationLee, J. J., Khan, T. A., McGlynn, N., Malik, V. S., Hill, J. O., Leiter, L. A., Jeppesen, P. B., Rahelić, D., Kahleová, H., Salas-Salvadó, J., et al. (2022) · Diabetes Care · doi:10.2337/dc21-2130 · Cited by 73
A Scoping Review of Current Guidelines on Dietary Fat and Fat Quality
Schwingshackl, L., Zähringer, J., Beyerbach, J., Werner, S. S., Nagavci, B., Heseker, H., Koletzko, B., Meerpohl, J. J., & on behalf of the International Union of Nutritional Sciences (IUNS) Task force on Dietary Fat Quality (2021) · Annals of Nutrition and Metabolism · doi:10.1159/000515671 · Cited by 53
Self-citation
Metadata
- Authors
- Schwingshackl, L., Schünemann, H. J., & Meerpohl, J. J.
- Venue
- European Journal of Nutrition
- Published
- 2020-12-30
- Volume
- 60
- Issue
- 6
- Pages
- 2893-2903
- ISSN
- 1436-6207 (ISSN-L)
- 1436-6215
- OpenAlex
- W3114003436
- Cited by
- 71
- References
- 28
- FWCI
- 6.63
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.