Getting rigorous with scientific rigor
Open accessThe DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
Two of the cornerstones of science advancement are rigor in designing and performing scientific research and the ability to reproduce biomedical research findings. The application of rigor ensures robust and unbiased experimental design, methodology, analysis, interpretation, and reporting of results. When a result can be reproduced by multiple scientists, it validates the original results and readiness to progress to the next phase of research. This is especially important for clinical trials in humans, which are built on studies that have demonstrated a particular effect or outcome —Francis S. Collins, MD, PhD, NIH Director (www.nih.gov/research-training/rigor-reproducibility). If science isn’t rigorous, it’s reckless. With this, the NIH announced its implementation of rigor and transparency on 9 October 2015 (notice number: NOT-OD-16-011). Now, reviewers of NIH grants are instructed to consider scientific rigor (intertwined with the consideration of sex and other relevant biological variables) in their final evaluation. The aim is for robust and unbiased results to increase the likelihood: (i) for accuracy of results; (ii) that these accurate results can be independently repeated. Most strive for the former, and hope for the latter. We scientists are quite frankly, relieved when our results are repeated. And, for the most part, this is not because we do not believe our results are true. Scientists just inherently question things, including ourselves. Ultimately, the more both personal integrity and scientific rigor are addressed during the journey, the less we (or the public) will question ourselves. However, regardless of the extent of scientific rigor implemented, there is always a nagging question of accuracy of results. Scientific rigor can be visualized as a ship that has to be built with expert and sturdy craftsmanship. This will not only ensure it is kept afloat, but with ethical scientists to captain the ship, it will keep sailing in the appropriate direction (Figure 1A). The endpoint would be solid, repeatable, enduring data that can be confidently built upon to move a field forward with the full confidence of our colleagues and the public. The front end (the bow) involves proper project design before grant submission and initial project review by funding agencies (e.g. NIH Notice: NOT-OD-16-011). The middle of this ship is in the laboratory, performing experiments during testing of the hypothesis. For the most part, this is not scrutinized by peers, and is investigator-driven in that it is up to the investigator to be rigorous both scientifically and ethically. At the back end are peer and editor reviews during the publication process. Finally, at the far end are the published results and further scrutiny by peers and the public (Figure 1A). All these parts need to be there for the ship to sail in the right direction. (A) Community scientific rigor: scientific rigor policed by the scientific community. (B) Investigator-driven scientific rigor. Scientific rigor policed by the individual investigator(s). We are all charged and most of us strive to get it right. In the end, the onus of scientific rigor is on the investigator. If rigor is carried out appropriately, the investigator and the scientific community can build upon those results, and the cycle continues towards (accurate) progress. At the investigator level, the individual is charged with ‘getting it right’. There needs to be vigilance in our personal integrity and scientific rigor when carrying out experiments to test a hypothesis (Figure 1B). The NIH defines scientific rigor as ‘the strict application of the scientific method to ensure robust and unbiased experimental design, methodology, analysis, interpretation and reporting of results. This includes full transparency in reporting experimental details so that others may reproduce and extend the findings’. The rules of scientific rigor can differ from study-to-study, depending on methodology (e.g. in vitro studies vs. epidemiological studies), but generally include (where appropriate) proper negative and positive controls (in every experiment), appropriate replicates within experiments (as well as repeat experiments—preferably at least in triplicate), randomization, blinding, measures to control bias, controlling for inter-operator variability, robust and accurate statistical methods, appropriate and accurate experimental design (e.g. inclusion/exclusion criteria), suitable and authenticated models, consideration of sex and other relevant biological variables, and others as appropriate to the field. ‘Getting it right’ is the goal (Figure 1B). To address the complexity and intertwining nature of scientific rigor, it might help to place it into six general categories: insidious, creative, careless, selective, careful, and enduring rigor (Table 1). Unfortunately, placement is often only revealed post-publication or post-award (far end, Figure 1A), which is the premise for this commentary.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
1 retracted
Wakefield, A., Murch, S., Anthony, A., Linnell, J., Casson, D., Malik, M., Berelowitz, M., Dhillon, A., Thomson, M., Harvey, P., et al. (1998) · The Lancet · doi:10.1016/s0140-6736(97)11096-0 · Cited by 3,006
RetractedMisconduct accounts for the majority of retracted scientific publications
Fang, F. C., Steen, R. G., & Casadevall, A. (2012) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1212247109 · Cited by 1,232 · cited 2×
“(1), using the definition from the DHHS Office of Research Integrity, found the majority of retracted articles were due to some form of misconduct (fraud, duplicate publication, and plagiarism) (1).”
BackgroundMethodReproducibility crisis: Blame it on the antibodies
Baker, M. (2015) · Nature · doi:10.1038/521274a · Cited by 830 · cited 1×
“Recent initiatives aimed at enduring rigor spawning from a perceived ‘reproducibility crisis’ (16) will help steer the ship in the right direction.”
BackgroundAutism and measles, mumps, and rubella vaccine: no epidemiological evidence for a causal association
Taylor, B., Miller, E., Farrington, C., Petropoulos, M.-C., Favot-Mayaud, I., Li, J., & Waight, P. A. (1999) · The Lancet · doi:10.1016/s0140-6736(99)01239-8 · Cited by 753
Time trends in autism and in MMR immunization coverage in California
Dales, L. (2001) · JAMA · doi:10.1001/jama.285.9.1183 · Cited by 340
Why and how do journals retract articles? An analysis of Medline retractions 1988–2008
Wager, E., & Williams, P. (2011) · Journal of Medical Ethics · doi:10.1136/jme.2010.040964 · Cited by 263
Lancet retracts 12-year-old article linking autism to MMR vaccines
Eggertson, L. (2010) · Canadian Medical Association Journal · doi:10.1503/cmaj.109-3179 · Cited by 249
Nosek, B. A., & Errington, T. M. (2017) · eLife · doi:10.7554/elife.23383 · Cited by 242
Phenomena of Retraction: Reasons for Retraction and Citations to the Publications
Budd, J. M., Sievert, M., & Schultz, T. R. (1998) · JAMA · doi:10.1001/jama.280.3.296 · Cited by 230
Retractions in the research literature: misconduct or mistakes?
Nath, S. B., Marcus, S. C., & Druss, B. G. (2006) · Medical Journal of Australia · doi:10.5694/j.1326-5377.2006.tb00504.x · Cited by 161
Cited by
The ABCD study: understanding the development of risk for mental and physical health outcomes
Karcher, N. R., & Barch, D. M. (2021) · Neuropsychopharmacology · doi:10.1038/s41386-020-0736-6 · Cited by 351
“As the NIMH has recently advocated [70, 71], it will be critical to implement robust and rigorous practices for conducting exploratory analyses.”
BackgroundMethodThe strain on scientific publishing
Hanson, M. A., Barreiro, P. G., Crosetto, P., & Brockington, D. (2024) · Quantitative Science Studies · doi:10.1162/qss_a_00327 · Cited by 345
The strain on scientific publishing
Hanson, M. A., Barreiro, P. G., Crosetto, P., & Brockington, D. (2023) · arXiv · doi:10.48550/arxiv.2309.15884 · Cited by 345
Yi, N., Nemery, B., & Dierickx, K. (2019) · Accountability in Research · doi:10.1080/08989621.2019.1652096 · Cited by 34
Metadata
- Authors
- Hofseth, L. J.
- Venue
- Carcinogenesis
- Publisher
- Oxford University Press (OUP)
- Published
- 2017-08-31
- Volume
- 39
- Issue
- 1
- Pages
- 21-25
- ISSN
- 0143-3334 (ISSN-L)
- 1460-2180
- OpenAlex
- W2752895005
- Cited by
- 56
- References
- 11
- FWCI
- 5.25
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Getting rigorous with scientific rigor. https://app.cochat.ai/papers/10.1093/carcin/bgx085
- The paper
- Hofseth, L. J. (2018). Getting rigorous with scientific rigor. Carcinogenesis. https://doi.org/10.1093/carcin/bgx085
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.