Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 66 of the 67 references it cites were found in a registry or an index; 20 of the 34 sources its claims cite were checked for support.

Text classification models for assessing the completeness of randomized controlled trial publications based on CONSORT reporting guidelines

Open access
Jiang, Lan, Lan, Mengfei, Menke, Joe D., Vorland, Colby J., Kilicoglu, Halil · 2024 · Scientific Reports

Published version of a preprint

Abstract

Complete and transparent reporting of randomized controlled trial publications (RCTs) is essential for assessing their credibility. We aimed to develop text classification models for determining whether RCT publications report CONSORT checklist items. Using a corpus annotated with 37 fine-grained CONSORT items, we trained sentence classification models (PubMedBERT fine-tuning, BioGPT fine-tuning, and in-context learning with GPT-4) and compared their performance. We assessed the impact of data augmentation methods (Easy Data Augmentation (EDA), UMLS-EDA, text generation and rephrasing with GPT-4) on model performance. We also fine-tuned section-specific PubMedBERT models (e.g., Methods) to evaluate whether they could improve performance compared to the single full model. We performed 5-fold cross-validation and report precision, recall, F 1 score, and area under curve (AUC). Fine-tuned PubMedBERT model that uses the sentence along with the surrounding sentences and section headers yielded the best overall performance (sentence level: 0.71 micro-F 1 , 0.67 macro-F 1 ; article-level: 0.90 micro-F 1 , 0.84 macro-F 1 ). Data augmentation had limited positive effect. BioGPT fine-tuning and GPT-4 in-context learning exhibited suboptimal results. Methods-specific model improved recognition of methodology items, other section-specific models did not have significant impact. Most CONSORT checklist items can be recognized reasonably well with the fine-tuned PubMedBERT model but there is room for improvement. Improved models can underpin the journal editorial workflows and CONSORT adherence checks.

Overview

CoChat Summary

TL;DR

This study develops and evaluates text classification models to recognize all CONSORT checklist items in randomized controlled trial publications. A fine-tuned PubMedBERT model using surrounding sentences and section headers performed best, while data augmentation and generative models gave limited benefit.

Key findings

  • The best PubMedBERT model achieved sentence-level 0.71 micro-F1 and 0.67 macro-F1, with balanced precision and recall (0.72 and 0.71, respectively).
  • At article level, the same model reached 0.92 micro-F1 and 0.87 macro-F1 under article (ANY) evaluation, and 0.90 micro-F1 and 0.84 macro-F1 under the stricter article (1+) evaluation.
  • Data augmentation had limited positive effect: for the best-performing model, F1 remained 0.71 across original and augmented settings, and differences were not statistically significant.
  • BioGPT fine-tuning and GPT-4 in-context learning underperformed the best PubMedBERT model; GPT-4 zero-shot scored 0.51 F1, one-shot 0.48 F1, and five-shot 0.46 F1. Methods-specific training improved recognition of methodology items (0.79 F1 vs. 0.75 F1 for the full model), but other section-specific models had limited impact.

Methods

  • Used the CONSORT-TM corpus of 50 RCT publications, containing 10,709 sentences, 4,845 annotated sentences, and 5,246 labels across 37 fine-grained CONSORT items; excluded Background (2a) and focused on 36 items, with items 1a and 1b handled by simple rules.
  • Trained sentence classification models via PubMedBERT fine-tuning, BioGPT fine-tuning, and GPT-4 in-context learning; tested input representations including section headers, surrounding sentences, and sentence position, plus data augmentation with EDA, UMLS-EDA, and GPT-4.
  • Evaluated with group 5-fold cross-validation, reporting precision, recall, F1, macro-F1, and AUC; GPT-4 in-context learning was evaluated on a randomly sampled test set of 10 articles.

Limitations

  • The paper states the dataset is small, consists of XML articles from PubMed Central that may not fully represent RCT literature, and contains limited data for infrequently reported CONSORT items; data augmentation had minimal effect.
  • The paper states generative model exploration was limited and calls for more systematic prompting strategies; it also notes the work only recognizes sentences reporting CONSORT items, not whether reporting is adequate or adherent.
  • Evident from the text: GPT-4 in-context learning was tested on only 10 articles due to cost, BioGPT fine-tuning used only the target sentence, and section-specific model comparisons were inconclusive, with label confusion between closely related items such as 17a/17b/18 and 12a/12b.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41598-024-72130-7

Cites

66 of 66 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 A high-level overview of our study, including NLP model training and evaluation.
Fig. 1 A high-level overview of our study, including NLP model training and evaluation.CC BY · PubMed Central
Fig. 2 Experimental flow including PubMedBERT fine-tuning, data augmentation strategies, few-shot prompting, and BioGPT fine-tuning. The PubMedBERT model takes sentences surrounding the target sentence into consideration. The prediction is made on the [CLS] token representation.
Fig. 2 Experimental flow including PubMedBERT fine-tuning, data augmentation strategies, few-shot prompting, and BioGPT fine-tuning. The PubMedBERT model takes sentences surrounding the target sentence into consideration. The prediction is made on the [CLS] token representation.CC BY · PubMed Central

Verification

Claims the paper makes

14 Valid · 2 Not valid · 9 Not checked · 81 Own results and interpretations

43 more claims

Create a free account to see all 106 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 31
  • It may state two separate claims as one. 32
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 67 references

Research works printed but not linked: 8

  1. Radford, A. et al. Language models are unsupervised multitask learners. OpenAI blog. 1(8), 9 (2019).
  2. Chowdhery A, Narang S, Devlin J, Bosma M, Mishra G, Roberts A, et al. PaLM: Scaling language modeling with pathways. arXiv preprint arXiv:220402311. 2022.
  3. Touvron H, Lavril T, Izacard G, Martinet X, Lachaux MA, Lacroix T, et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:230213971. 2023.
  4. Zhao WX, Zhou K, Li J, Tang T, Wang X, Hou Y, et al. A survey of large language models. arXiv preprint arXiv:230318223. 2023.
  5. Chen Q, Du J, Hu Y, Keloth VK, Peng X, Raja K, et al. Large language models in biomedical natural language processing: benchmarks, baselines, and recommendations. arXiv preprint arXiv:230516326. 2023.
  6. Dai H, Liu Z, Liao W, Huang X, Cao Y, Wu Z, et al. AugGPT: Leveraging ChatGPT for text data augmentation. arXiv preprint arXiv:230213007. 2023.
  7. Wu C, Zhang X, Zhang Y, Wang Y, Xie W. PMC-LLaMA: Further finetuning LLaMA on medical papers. arXiv preprint arXiv:230414454. 2023.
  8. Tang R, Han X, Jiang X, Hu X. Does synthetic data generation of LLMs help clinical text mining? arXiv preprint arXiv:230304360. 2023.

Cited by

See all 20

Metadata

Authors
Jiang, L., Lan, M., Menke, J. D., Vorland, C. J., & Kilicoglu, H.
Affiliations
  • University of Illinois Urbana-Champaign (US)
  • Indiana University Bloomington (US)
Venue
Scientific Reports
Published
2024-09-17
Volume
14
Issue
1
Pages
21721-21721
ISSN
2045-2322 (ISSN-L)
OpenAlex
W4402620497
Cited by
18
References
67
Citations by year
2026: 2 · 2025: 16
FWCI
5.01
Type
journal article
Language
en
Open access
Licence
Locations
  • Scientific Reports (publishedVersion, cc-by)
  • medRxiv (acceptedVersion, cc-by-nc)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • PubMed Central (submittedVersion, cc-by-nc)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Funders
National Institutes of Health (R01LM014079)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
Keywords
randomized controlled trials, text classification, reporting completeness, PubMedBERT, BioGPT, data augmentation, trial reporting quality, journal editorial workflow

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-02, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-02 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Text classification models for assessing the completeness of randomized controlled trial publications based on CONSORT reporting guidelines. https://app.cochat.ai/papers/10.1038/s41598-024-72130-7
The paper
Jiang, L., Lan, M., Menke, J. D., Vorland, C. J., & Kilicoglu, H. (2024). Text classification models for assessing the completeness of randomized controlled trial publications based on CONSORT reporting guidelines. Scientific Reports. https://doi.org/10.1038/s41598-024-72130-7

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.