Why do we think we know what we know? A metaknowledge analysis of the salt controversy
The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Although several public health organizations have recommended population-wide reduction in salt intake, the evidence on the population benefits remains unclear. We conducted a metaknowledge analysis of the literature on salt intake and health outcomes. METHODS: We identified reports--primary studies, systematic reviews, guidelines and comments, letters or reviews--addressing the effect of sodium intake on cerebro-cardiovascular disease or mortality. We classified reports as supportive or contradictory of the hypothesis that salt reduction leads to population benefits, and constructed a network of citations connecting these reports. We tested for citation bias using an exponential random graph model. We also assessed the inclusion of primary studies in systematic reviews on the topic. RESULTS: We identified 269 reports (25% primary studies, 5% systematic reviews, 4% guidelines and 66% comments, letters, or reviews) from between 1978 and 2014. Of these, 54% were supportive of the hypothesis, 33% were contradictory and 13% were inconclusive. Reports were 1.51 [95% confidence interval (CI) 1.38 to 1.65] times more likely to cite reports that drew a similar conclusion, than to cite reports drawing a different conclusion. In all, 48 primary studies were selected for inclusion across 10 systematic reviews. If any given primary study was selected by a review, the probability that a further review would also have selected it was 27.0% (95% CI 20.3% to 33.7%). CONCLUSIONS: We documented a strong polarization of scientific reports on the link between sodium intake and health outcomes, and a pattern of uncertainty in systematic reviews about what should count as evidence.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For example, an analysis of the patterns of knowledge generation surrounding the controversy between proponents and opponents of a population-wide reduction in salt consumption showed that reports were more likely to cite studies that had conclusions similar to rather than different from those of the author doing the citing (43).”
From Misrepresentation and distortion of research in biomedical literature (2018) · cites this paper as [43]
Citations
Cites
(1987) · Cited by 8,955
ergm: A Package to Fit, Simulate and Diagnose Exponential-Family Models for Networks
Hunter, D. R., Handcock, M. S., Butts, C. T., Goodreau, S. M., & Morris, M. (2008) · Journal of Statistical Software · doi:10.18637/jss.v024.i03 · Cited by 1,283 · cited 1×
“primary study, systematic review, guideline and comment, letter or review), journal impact factor and network structure.(12,13) The model was used to formally test for homophily according to report classification (supportive, contradictory, inconclusive): our hypothesis was that a report would be more likely to cite a…”
MethodInformal Networks and Organizational Crises: An Experimental Simulation
Krackhardt, D., & Stern, R. N. (1988) · Social Psychology Quarterly · doi:10.2307/2786835 · Cited by 1,256 · cited 1×
“We estimated an homophily index as the number of co-authorships external to the groups minus the number of co-authorships internal to the groups divided by the total number of co-authorships.(15) The index ranges from 1 (homophily) toþ 1 (heterophily).”
MethodProjected effect of dietary salt reductions on future cardiovascular disease
Bibbins-Domingo, K., Chertow, G. M., Coxson, P. G., Moran, A., Lightwood, J. M., Pletcher, M. J., & Goldman, L. (2010) · New England Journal of Medicine · doi:10.1056/nejmoa0907355 · Cited by 1,253 · cited 1×
“Comment, letter, review 176 106 51 19 Publication year 2011 [1978–2014] 2011 [1992–2014] 2010 [1985–2014] 2012 [1978–2014] Number of citations received 2 [0–83] 2 [0–79] 1 [0–83] 2 [0–76]”
Global Sodium Consumption and Death from Cardiovascular Causes
Mozaffarian, D., Fahimi, S., Singh, G. M., Micha, R., Khatibzadeh, S., Engell, R. E., Lim, S., Danaei, G., Ezzati, M., & Powles, J. (2014) · New England Journal of Medicine · doi:10.1056/nejmoa1304127 · Cited by 1,218 · cited 1×
“Comment, letter, review 176 106 51 19 Publication year 2011 [1978–2014] 2011 [1992–2014] 2010 [1985–2014] 2012 [1978–2014] Number of citations received 2 [0–83] 2 [0–79] 1 [0–83] 2 [0–76]”
Arguing to Learn in Science: The Role of Collaborative, Critical Discourse
Osborne, J. (2010) · Science · doi:10.1126/science.1183944 · Cited by 948
Handcock, M. S., Hunter, D. R., Butts, C. T., Goodreau, S. M., & Morris, M. (2008) · Journal of Statistical Software · doi:10.18637/jss.v024.i01 · Cited by 926
Guideline: Potassium intake for adults and children
Hooper, L. (2012) · UEA Digital Repository (University of East Anglia) · Cited by 672
Conflict of Interest in the Debate over Calcium-Channel Antagonists
Stelfox, H. T., Chua, G., O'Rourke, K., & Detsky, A. S. (1998) · New England Journal of Medicine · doi:10.1056/nejm199801083380206 · Cited by 615
Urinary sodium and potassium excretion and risk of cardiovascular events
O'Donnell, M. J., Yusuf, S., Mente, A., Gao, P., Mann, J. F., Teo, K., McQueen, M., Sleight, P., Sharma, A. M., Dans, A., et al. (2011) · JAMA · doi:10.1001/jama.2011.1729 · Cited by 564
How citation distortions create unfounded authority: analysis of a citation network
Greenberg, S. A. (2009) · BMJ · doi:10.1136/bmj.b2680 · Cited by 499
Why review articles on the health effects of passive smoking reach different conclusions
Barnes, D. E. (1998) · JAMA · doi:10.1001/jama.279.19.1566 · Cited by 442 · cited 1×
“Comment, letter, review 176 106 51 19 Publication year 2011 [1978–2014] 2011 [1992–2014] 2010 [1985–2014] 2012 [1978–2014] Number of citations received 2 [0–83] 2 [0–79] 1 [0–83] 2 [0–76]”
Kontis, V., Mathers, C. D., Rehm, J., Stevens, G. A., Shield, K. D., Bonita, R., Riley, L. M., Poznyak, V., Beaglehole, R., & Ezzati, M. (2014) · The Lancet · doi:10.1016/s0140-6736(14)60616-4 · Cited by 418
Specification of Exponential-Family Random Graph Models: Terms and Computational Aspects
Morris, M., Handcock, M. S., & Hunter, D. R. (2008) · Journal of Statistical Software · doi:10.18637/jss.v024.i04 · Cited by 406 · cited 1×
“primary study, systematic review, guideline and comment, letter or review), journal impact factor and network structure.(12,13) The model was used to formally test for homophily according to report classification (supportive, contradictory, inconclusive): our hypothesis was that a report would be more likely to cite a…”
MethodCobb, L. K., Anderson, C. A. M., Elliott, P., Hu, F. B., Liu, K., Neaton, J. D., Whelton, P. K., Woodward, M., & Appel, L. J. (2014) · Circulation · doi:10.1161/cir.0000000000000015 · Cited by 300 · cited 1×
“Some have previously questioned the role of randomized trials as the source of definitive evidence, and a recent review of the methodological quality of observational studies relating sodium to cardiovascular outcomes found that methodological defects, considered to have the potential to alter the direction of associa…”
BackgroundSodium Intake in Populations: Assessment of Evidence
Strom, B. L., Yaktine, A. L., Oria, M. P., & Nutrition Board (2013) · Cited by 297
Evans, J. A., & Foster, J. G. (2011) · Science · doi:10.1126/science.1201765 · Cited by 252
Citation bias favoring statistically significant studies was present in medical research
Jannot, A.-S., Agoritsas, T., Gayet-Ageron, A., & Perneger, T. V. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.09.015 · Cited by 193 · cited 1×
“Comment, letter, review 176 106 51 19 Publication year 2011 [1978–2014] 2011 [1992–2014] 2010 [1985–2014] 2012 [1978–2014] Number of citations received 2 [0–83] 2 [0–79] 1 [0–83] 2 [0–76]”
The (Political) Science of Salt
Taubes, G. (1998) · Science · doi:10.1126/science.281.5379.898 · Cited by 176 · cited 1×
“Universal salt reduction has been the subject of considerable scientific controversy over several decades.(1,2) Proponents of population-wide reduction in salt consumption note that high sodium intakes increase blood pressure and thus the risk of cerebro-cardiovascular events, and assert that salt reduction policies w…”
BackgroundIoannidis, J. P. A. (2015) · Journal of Psychosomatic Research · doi:10.1016/j.jpsychores.2014.11.008 · Cited by 165 · cited 1×
“Part of the observed pattern may be explained by direct as well as coauthor and collaborative self-citations.(24) The data are consistent with a picture where a few authors and a few key papers dominate the literature.”
ResultScience, Policy, and the Transparency of Values
Elliott, K. C., & Resnik, D. B. (2014) · Environmental Health Perspectives · doi:10.1289/ehp.1408107 · Cited by 155 · cited 1×
“These findings have implications for the translation of conflicting, uncertain evidence into public health policy.(7,32,33) Both the misuse of uncertainty and the exaggeration of certainty can shape the outcomes of public health decision-making processes.”
BackgroundPolicy options to reduce population salt intake
Cappuccio, F. P., Capewell, S., Lincoln, P., & McPherson, K. (2011) · BMJ · doi:10.1136/bmj.d4995 · Cited by 142 · cited 1×
“Whereas several authors involved in the conduct of systematic reviews of the evidence on sodium have argued that the salt hypothesis is a distraction with little empirical grounding,(8) others have asserted that salt reduction in populations should be a central public health priority in the 21st century.(9) Assuming t…”
BackgroundStatistically significant papers in psychiatry were cited more often than others
Nieminen, P. A., Rücker, G., Miettunen, J., Carpenter, J. R., & Schumacher, M. C. (2007) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.11.014 · Cited by 89 · cited 1×
“Comment, letter, review 176 106 51 19 Publication year 2011 [1978–2014] 2011 [1992–2014] 2010 [1985–2014] 2012 [1978–2014] Number of citations received 2 [0–83] 2 [0–79] 1 [0–83] 2 [0–76]”
Citation bias of hepato-biliary randomized clinical trials
Gluud, L. L., & Gluud, C. N. (2002) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00513-3 · Cited by 84 · cited 1×
“Comment, letter, review 176 106 51 19 Publication year 2011 [1978–2014] 2011 [1992–2014] 2010 [1985–2014] 2012 [1978–2014] Number of citations received 2 [0–83] 2 [0–79] 1 [0–83] 2 [0–76]”
Making Sense of Non-Financial Competing Interests
Editors, T. P. M. (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050199 · Cited by 79
Screening nonrandomized studies for medical systematic reviews: a comparative study of classifiers
Bekhuis, T., & Demner-Fushman, D. (2012) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2012.05.002 · Cited by 70 · cited 1×
“As a consequence, the classification process was burdensome; natural language processing methods, especially supervised machine learning classification, may be used to reduce the amount of human review required to assess each article, so that the analysis method could be applied to a wider array of topics.(40,41) Thir…”
MethodDietary Sodium Intake and Cardiovascular Mortality: Controversy Resolved?
Alderman, M. H., & Cohen, H. W. (2012) · American Journal of Hypertension · doi:10.1038/ajh.2012.52 · Cited by 68
Unscientific Beliefs about Scientific Topics in Nutrition
Brown, A. W., Ioannidis, J. P. A., Cope, M. B., Bier, D. M., & Allison, D. B. (2014) · Advances in Nutrition · doi:10.3945/an.114.006577 · Cited by 54 · cited 1×
“It has previously been suggested that these conflicts of interest may be more powerful than financial consideration.(30,38) Our findings support a strong bias towards the status quo and the absence of a genuine scientific conversation where each side engages the other.”
BackgroundSalt And Public Health: Contested Science And The Challenge Of Evidence-Based Decision Making
Bayer, R., Johns, D. M., & Galea, S. (2012) · Health Affairs · doi:10.1377/hlthaff.2012.0554 · Cited by 46 · cited 2×
“Universal salt reduction has been the subject of considerable scientific controversy over several decades.(1,2) Proponents of population-wide reduction in salt consumption note that high sodium intakes increase blood pressure and thus the risk of cerebro-cardiovascular events, and assert that salt reduction policies w…”
BackgroundSelf-citationEtter, J.-F., & Stapleton, J. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.09.015 · Cited by 40
Understanding belief using citation networks
Greenberg, S. A. (2011) · Journal of Evaluation in Clinical Practice · doi:10.1111/j.1365-2753.2011.01646.x · Cited by 34 · cited 1×
“For example, reports may have cited previous content in order to criticize its validity or reframe its meaning to divert its implications.(42) Last but not least, we do not here intend to ‘take sides’ in the salt debate: we neither endorse nor oppose the central hypothesis at hand.”
BackgroundCONFLICT OF INTEREST IN THE DEBATE OVER CALCIUM-CHANNEL ANTAGONISTS
Telfox, T. H. S., Hua, G., & Etsky, A. L. S. D. (2013) · Cited by 31
Influence, originality and similarity in directed acyclic graphs
Gualdi, S., Medo, M., & Zhang, Y. (2011) · Europhysics Letters (EPL) · doi:10.1209/0295-5075/96/18004 · Cited by 25 · cited 1×
“Third, we identified the most influential reports in the citation network by using a random walk approach (see Supplementary data, available at IJE online).(14) We assessed the likelihood that a supportive, contradictory or inconclusive report was among the most influential reports.”
MethodKivimaki, M., Batty, G. D., Kawachi, I., Virtanen, M., Singh-Manoux, A., & Brunner, E. J. (2014) · American Journal of Epidemiology · doi:10.1093/aje/kwu164 · Cited by 23 · cited 1×
“Comment, letter, review 176 106 51 19 Publication year 2011 [1978–2014] 2011 [1992–2014] 2010 [1985–2014] 2012 [1978–2014] Number of citations received 2 [0–83] 2 [0–79] 1 [0–83] 2 [0–76]”
Perino, A. C., Hoang, D. D., Holmes, T. H., Santangeli, P., Heidenreich, P. A., Perez, M. V., Wang, P. J., & Turakhia, M. P. (2014) · Circulation Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.114.000912 · Cited by 20 · cited 1×
“Comment, letter, review 176 106 51 19 Publication year 2011 [1978–2014] 2011 [1992–2014] 2010 [1985–2014] 2012 [1978–2014] Number of citations received 2 [0–83] 2 [0–79] 1 [0–83] 2 [0–76]”
Influence, originality and similarity in directed acyclic graphs
Gualdi, S., Medo, M., & Zhang, Y.-C. (2012) · reroDoc Digital Library · Cited by 20
Consequences of Industry Relationships for Public Health and Medicine
Rothman, D. J. (2011) · American Journal of Public Health · doi:10.2105/ajph.2011.300507 · Cited by 18 · cited 1×
“Comment, letter, review 176 106 51 19 Publication year 2011 [1978–2014] 2011 [1992–2014] 2010 [1985–2014] 2012 [1978–2014] Number of citations received 2 [0–83] 2 [0–79] 1 [0–83] 2 [0–76]”
White hat bias: The need for authors to have the spin stop with them
Atkinson, R. L., & Macdonald, I. A. (2010) · International Journal of Obesity · doi:10.1038/ijo.2009.269 · Cited by 15 · cited 1×
“Comment, letter, review 176 106 51 19 Publication year 2011 [1978–2014] 2011 [1992–2014] 2010 [1985–2014] 2012 [1978–2014] Number of citations received 2 [0–83] 2 [0–79] 1 [0–83] 2 [0–76]”
Citations alone were enough to predict favorable conclusions in reviews of neuraminidase inhibitors
Zhou, X., Wang, Y., Tsafnat, G., Coiera, E. W., Bourgeois, F. T., & Dunn, A. G. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.09.014 · Cited by 13 · cited 1×
“As a consequence, the classification process was burdensome; natural language processing methods, especially supervised machine learning classification, may be used to reduce the amount of human review required to assess each article, so that the analysis method could be applied to a wider array of topics.(40,41) Thir…”
MethodTransparency of Values in Science: Elliott and Resnik Respond
Elliott, K. C., & Resnik, D. B. (2014) · Environmental Health Perspectives · doi:10.1289/ehp.1408936r · Cited by 13
Graudal, N. A., & Jűrgens, G. (2011) · BMJ · doi:10.1136/bmj.d6119 · Cited by 12 · cited 1×
“Comment, letter, review 176 106 51 19 Publication year 2011 [1978–2014] 2011 [1992–2014] 2010 [1985–2014] 2012 [1978–2014] Number of citations received 2 [0–83] 2 [0–79] 1 [0–83] 2 [0–76]”
Identifying and Managing Nonfinancial Conflicts of Interest for Systematic Reviews
Viswanathan, M., Carey, T. S., Belinson, S., Berliner, E., Chang, S. T., Graham, E., Guise, J., Ip, S., Maglione, M., McCrory, D. C., et al. (2013) · Agency for Healthcare Research and Quality (US) eBooks · Cited by 10
Controversial Salt Report Peppered with Uncertainty
Johns, D. M., Bayer, R., & Galea, S. (2013) · Science · doi:10.1126/science.341.6150.1063 · Cited by 6 · cited 2×
“These findings have implications for the translation of conflicting, uncertain evidence into public health policy.(7,32,33) Both the misuse of uncertainty and the exaggeration of certainty can shape the outcomes of public health decision-making processes.”
BackgroundSelf-citationRestricting Dietary Salt and Public Health: Is the Evidentiary Foundation Crumbling?
Bayer, R. (2014) · Milbank Quarterly · doi:10.1111/1468-0009.12088 · Cited by 2 · cited 1×
“These findings have implications for the translation of conflicting, uncertain evidence into public health policy.(7,32,33) Both the misuse of uncertainty and the exaggeration of certainty can shape the outcomes of public health decision-making processes.”
Background
Research works printed but not linked: 1
- West R Stenius K . To cite or not to cite? Use and abuse of citations. In: Babor T Stenius K Savva S O'Reilly J , (eds). Publishing Addiction Science: A Guide for the Perplexed. 2nd edn. Brentwood, UK: Multi-Science Publishing Company, 2008.
Cited by
Forouzanfar, M. H., Afshin, A., Alexander, L. T., Anderson, H. R., Bhutta, Z. A., Biryukov, S., Brauer, M., Burnett, R., Cercy, K., Charlson, F. J., et al. (2016) · The Lancet · doi:10.1016/s0140-6736(16)31679-8 · Cited by 7,698
Afshin, A., Sur, P. J., Fay, K. A., Cornaby, L., Ferrara, G., Salama, J. S., Mullany, E. C., Abate, K. H., Abbafati, C., Abebe, Z., et al. (2019) · The Lancet · doi:10.1016/s0140-6736(19)30041-8 · Cited by 5,811
Misrepresentation and distortion of research in biomedical literature
Boutron, I., & Ravaud, P. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1710755115 · Cited by 273
“For example, an analysis of the patterns of knowledge generation surrounding the controversy between proponents and opponents of a population-wide reduction in salt consumption showed that reports were more likely to cite studies that had conclusions similar to rather than different from those of the author doing the…”
ResultA Systematic Review of the Sources of Dietary Salt Around the World
Bhat, S., Marklund, M., Henry, M. E., Appel, L. J., Croft, K. D., Neal, B., & Wu, J. H. Y. (2020) · Advances in Nutrition · doi:10.1093/advances/nmz134 · Cited by 221
“effects of salt intake continue to be debated (4).”
BackgroundDietary Sodium and Cardiovascular Disease Risk — Measurement Matters
Cogswell, M. E., Mugavero, K., Bowman, B. A., & Frieden, T. R. (2016) · New England Journal of Medicine · doi:10.1056/nejmsb1607161 · Cited by 211
Metadata
- Authors
- Trinquart, L., Johns, D. M., & Galea, S.
- Affiliations
- Columbia University (US)
- Boston University (US)
- Corresponding author
- Trinquart, Ludovic
- Venue
- International Journal of Epidemiology
- Publisher
- Oxford University Press (OUP)
- Published
- 2016-02
- Volume
- 45
- Issue
- 1
- Pages
- 251-260
- ISSN
- 0300-5771 (ISSN-L)
- 1464-3685
- OpenAlex
- W2277886283
- Cited by
- 77
- References
- 44
- Citations by year
- 2025: 4 · 2024: 5 · 2023: 2 · 2022: 4 · 2021: 4 · 2020: 12 · 2019: 9 · 2018: 12 · 2017: 9 · 2016: 15
- FWCI
- 2.02
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- International Journal of Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- HighWire Press Open Archive (submittedVersion)
- Funders
- Mailman School of Public Health, Columbia University
- Topics
- Health Sciences › Nursing › Nutrition and Dietetics › Sodium Intake and Health
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Cardiology and Cardiovascular Medicine › Blood Pressure and Hypertension Studies
- Keywords
- salt intake, sodium intake, salt reduction, cardiovascular disease, mortality, meta-knowledge, citation bias, exponential random graph models, systematic reviews, evidence uncertainty, public health recommendations
- MeSH terms
- Humans, Bibliometrics, Bias, Sodium Chloride, Dietary
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.