The DOI matches its Crossref record; 1 of the 8 references it cites were found in a registry or an index, and 7 are not checked yet.
On the Rigour of Scientific Writing: Criteria, Analysis, and Insights
Open accessAbstract
Rigour is crucial for scientific research as it ensures the reproducibility and validity of results and findings.Despite its importance, little work exists on modelling rigour computationally, and there is a lack of analysis on whether these criteria can effectively signal or measure the rigour of scientific papers in practice.In this paper, we introduce a bottom-up, data-driven framework to automatically identify and define rigour criteria and assess their relevance in scientific writing.Our framework includes rigour keyword extraction, detailed rigour definition generation, and salient criteria identification.Furthermore, our framework is domain-agnostic and can be tailored to the evaluation of scientific rigour for different areas, accommodating the distinct salient criteria across fields.We conducted comprehensive experiments based on datasets collected from two high impact venues for Machine Learning and NLP (i.e., ICLR and ACL) to demonstrate the effectiveness of our framework in modelling rigour.In addition, we analyse linguistic patterns of rigour, revealing that framing certainty is crucial for enhancing the perception of scientific rigour, while suggestion certainty and probability uncertainty diminish it.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Scientific evaluation necessarily responds to how evidence and contributions are presented.”
From How Can Rhetoric Reward-Hack AI Reviewers? Dissecting Rhetorical Sensitivity in AI-Based Peer Review (2026) · cites this paper as James et al., 2024
The cited source says what the claim attributes to it.
In the citing paper
“Computational evidence likewise links linguistic certainty to perceived scientific rigour (James et al., 2024).”
From How Can Rhetoric Reward-Hack AI Reviewers? Dissecting Rhetorical Sensitivity in AI-Based Peer Review (2026) · cites this paper as James et al., 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 26
- Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. · arXiv
- Muhammad Afzal, Beom Joo Park, Maqbool Hussain, and Sungyoung Lee. 2020. Deep learning based biomedical literature classification using criteria of scientific rigor. Electronics, 9(8):1253.
- ARR. 2023. ACL Rolling Review -Responsible NLP Research Checklist. https://aclrollingreview .org/responsibleNLPresearch/. Accessed: 12 June 2024.
- Alina Beygelzimer, Yann N Dauphin, Percy Liang, and Jennifer Wortman Vaughan. 2023. Has the machine learning review process become more arbitrary as the field has grown? the neurips 2021 consistency experiment. arXiv preprint arXiv:2306.03262. · arXiv
- Corinna Cortes and Neil D Lawrence. 2021. In- consistency in conference peer review: revisit- ing the 2014 neurips experiment. arXiv preprint arXiv:2109.09774. Research England, the Scottish Funding Council, the Higher Education Funding Council for Wales, and Northern Ireland the Department for the Economy. 2021. Guidance on ref 2021 results -ref 2021. http s://2021.ref.ac.uk/guidance-on-results/g uidance-on-ref-2021-results/index.html#ou tputs. Accessed: 12 June 2024. · arXiv
- Andrew R Gilpin. 1993. Table for conversion of kendall's tau to spearman's rho within the con- text of measures of magnitude of effect for meta- analysis. Educational and psychological measure- ment, 53(1):87-92.
- Tomas Goldsack, Zhihao Zhang, Chenghua Lin, and Carolina Scarton. 2023. Domain-driven and discourse-guided scientific summarisation. In Eu- ropean Conference on Information Retrieval, pages 361-376. Springer.
- Rita González-Márquez and Dmitry Kobak. 2024. Learning representations of learning representations. In Data-centric Machine Learning Research (DMLR) workshop at ICLR 2024.
- Katarina Hamberg, Eva Johansson, Gerd Lindgren, and Göran Westman. 1994. Scientific rigour in qualitative research-examples from a study of women's health in family practice. Family practice, 11(2):176-181.
- Jessica L Johnson, Donna Adkins, and Sheila Chauvin. 2020. A review of the quality indicators of rigor in qualitative research. American journal of pharma- ceutical education, 84(1):7120.
- Alexander Kraskov, Harald Stögbauer, and Peter Grass- berger. 2004. Estimating mutual information. Physi- cal review E, 69(6):066138.
- Gordon J Lithgow, Monica Driscoll, and Patrick Phillips. 2017. A long journey to reproducible results. Nature, 548(7668):387-388.
- Zhuoran Luo, Wei Lu, Jiangen He, and Yuqi Wang. 2022. Combination of research questions and meth- ods: A new measurement of scientific novelty. Jour- nal of Informetrics, 16(2):101282.
- Nestor Maslej, Loredana Fattorini, Erik Brynjolfs- son, John Etchemendy, Katrina Ligett, Terah Lyons, James Manyika, Helen Ngo, Juan Carlos Niebles, Vanessa Parli, et al. 2023. Artificial intelligence in- dex report 2023. arXiv preprint arXiv:2310.03715. · arXiv
- Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. 2024. Generative representational in- struction tuning. arXiv preprint arXiv:2402.09906. · arXiv
- National Academies. 2017. Communicating science effectively: A research agenda. National Academies Press.
- NeurIPS. 2021. NeurIPS 2021 Paper Checklist Guide- lines. https://neurips.cc/Conferences/2021/ PaperInformation/PaperChecklist. Accessed: 12 June 2024.
- Wullianallur Raghupathi, Viju Raghupathi, and Jie Ren. 2022. Reproducibility in computing research: An empirical study. IEEE Access, 10:29207-29223.
- Hendrik Schuff, Lindsey Vanderlyn, Heike Adel, and Ngoc Thang Vu. 2023. How to do human evaluation: A brief introduction to user studies in nlp. Natural Language Engineering, 29(5):1199-1222.
- SciScore. 2024. The best methods review tool for scien- tific research. https://sciscore.com/. Accessed: 12 June 2024.
- Harald Semmelrock, Simone Kopeinik, Dieter Theiler, Tony Ross-Hellauer, and Dominik Kowald. 2023. Re- producibility in machine learning-driven research. arXiv preprint arXiv:2307.10320. · arXiv
- Kexuan Sun, Zhiqiang Qiu, Abel Salinas, Yuzhong Huang, Dong-Ho Lee, Daniel Benjamin, Fred Morstatter, Xiang Ren, Kristina Lerman, and Jay Pujara. 2022. Assessing scientific research papers with knowledge graphs. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2467-2472.
- Mike Thelwall. 2022. Can the quality of published academic journal articles be assessed with machine learning? Quantitative Science Studies, 3(1):208- 226.
- Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2023. Improving text embeddings with large language models. arXiv preprint arXiv:2401.00368. · arXiv
- Jennifer L Wilson and Crystal M Botham. 2021. Three questions to address rigour and reproducibility con- cerns in your grant proposal. Nature, 596(7873):609- 610.
- Chenghao Xiao, G Thomas Hudson, and Noura Al Moubayed. 2024. Rar-b: Reasoning as retrieval benchmark. arXiv preprint arXiv:2404.06347. · arXiv
Cited by
Sorted by citations
How Can Rhetoric Reward-Hack AI Reviewers? Dissecting Rhetorical Sensitivity in AI-Based Peer Review
Li, M., Wang, C., Li, X., Zeng, X., Li, D., Shi, P., Zhou, D., & Zhou, T. (2026) · arXiv · doi:10.48550/arxiv.2608.08975
Petrović, M., & Prosen, M. (2025) · Kontakt · doi:10.32725/kont.2025.042
Between clarity and mystery: Bridging language in empirical research and contemplation
Mozol, A. G. (2026) · Journal for the Study of Spirituality · doi:10.1080/20440243.2026.2630645
Metadata
- Authors
- James, J., Xiao, C., Li, Y., & Lin, C.
- Affiliations
- University of Sheffield (GB)
- Durham University (GB)
- University of Surrey (GB)
- University of Manchester (GB)
- Venue
- Findings of the Association for Computational Linguistics: EMNLP 2024
- Published
- 2024
- Pages
- 6523-6538
- OpenAlex
- W4404781581
- Cited by
- 2
- References
- 0
- Citations by year
- 2026: 1 · 2025: 1
- FWCI
- 8.24
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- UK Research and Innovation (EP/S023062, EP/S023062/1)
- Topics
- Social Sciences › Library and Information Sciences › Academic Writing and Publishing
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Social Sciences › Linguistics and Language › Discourse Analysis in Language Studies
- Keywords
- scientific rigor, scientific writing, reproducibility
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
