CoChat

Citation indexes do not reflect methodological quality in lung cancer randomised trials.

Open access

The DOI matches its Crossref record; 21 of the 24 references it cites were found in a registry or an index; none of the 9 sources its claims cite is checked for support yet.

Abstract

Background Citation factors are applied to assess scientific work despite the fact that they were developed commercially in order to compare competing journals. The aim of the present study was to determine whether there is a relationship between citation factors and a trial's methodological quality using published randomised trials in lung cancer clinical research. Material and methods All of the randomised trials included in nine systematic reviews performed by the European Lung Cancer Working Party (ELCWP) were assessed using two quality scales (Chalmers and ELCWP). Results One hundred and eighty-one articles were eligible. The median overall ELCWP and Chalmers quality scores were 61.8% and 49.0%, respectively, with a correlation coefficient (r(s)) of 0.74 (P <0.001). A weak association was observed between citation factors and quality scores with the respective correlation coefficients ranging from 0.18 to 0.40 (ELCWP scale) and from 0.21 to 0.38 (Chalmers scale). American authors published trials significantly more often in journals with high citation factors than European or non-American authors (P <0.0001), despite no better methodological quality. Positive trials, which were significantly more likely to be published in journals with higher citation factors, were of no better quality than negative ones. Conclusion Journals with higher citation factors do not appear to publish clinical trials with higher levels of methodological quality, at least for trials in the field of lung cancer research.

Overview

CoChat Summary

TL;DR

In 181 lung cancer randomised trials, citation factors showed only weak correlations with methodological quality scores, despite statistically significant associations. The authors conclude that journals with higher citation factors do not appear to publish methodologically better clinical trials in this field.

Key findings

  • Median quality scores were 61.8% on the ELCWP scale and 49.0% on the Chalmers scale, with a correlation of \(r_s = 0.74\) (\(P < 0.001\)) between the two scales.
  • Citation factors correlated weakly with quality scores: \(r_s\) ranged from 0.18 to 0.40 for ELCWP and 0.21 to 0.38 for Chalmers.
  • American authors published significantly more often in journals with high citation factors than European or non-American authors (\(P < 0.0001\)), but had no better methodological quality; on the Chalmers scale there was a borderline difference favouring European authors.
  • Positive trials were significantly more likely to be published in journals with higher citation factors, but showed no quality difference compared with negative trials.

Methods

  • All randomised trials included in nine systematic reviews by the European Lung Cancer Working Party were assessed; 181 articles published in 31 journals and one book from 1977 to 2001 were eligible.
  • Methodological quality was scored using two published scales, ELCWP and Chalmers, with consensual agreement among medical doctors and a biostatistician.
  • Associations between citation factors (IF99, IF Pub, PF, EF) and quality scores were analysed with Spearman rank correlation, Mann-Whitney, Kruskal-Wallis, Student’s t-test, and chi-square tests; a \(P < 0.001\) threshold was used because of multiple testing.

Limitations

  • The paper states that results may not generalise outside lung cancer research, where mainly small to moderate randomised trials with equivocal results are available; extrapolation to large level I trials should be done with circumspection.
  • The paper states that only articles in English, French, or Dutch were eligible, and that citation factor values were not available for all articles (IF99 in 171, IF Pub in 150, PF in 175, EF in 39).
  • Evident from the text: the study is observational and correlational, so it cannot establish causality; EF analyses were based on only 39 articles, and EF was not considered for American authors because only two American trials were published in European journals.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1093/annonc/mdg203

Cites

21 of 21 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

0 Valid · 0 Not valid · 8 Not checked · 24 Own results and interpretations

9 more claims

Create a free account to see all 32 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 12
  • Probably not a substantial claim. 7
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 21 references

Cited by

See all 38

Metadata

Authors
Berghmans, T., Meert, A. P., Mascaux, C., Paesmans, M., Lafitte, J. J., & Sculier, J. P.
Affiliations
  • Institut Jules Bordet (BE)
  • Centre Hospitalier Universitaire de Lille (FR)
Corresponding author
Berghmans, T.
Venue
Annals of Oncology
Publisher
Elsevier BV
Published
2003-05
Volume
14
Issue
5
Pages
715-721
ISSN
0923-7534 (ISSN-L)
OpenAlex
W2162337243
Cited by
38
References
24
Citations by year
2026: 1 · 2022: 1 · 2021: 1 · 2020: 1 · 2018: 1 · 2017: 2 · 2016: 2 · 2014: 3 · 2013: 4 · 2012: 1
FWCI
4.13
Type
journal article
Language
en
Open access
Licence
Locations
  • Annals of Oncology (publishedVersion)
  • PubMed (publishedVersion)
  • Dépôt institutionnel de l'Université libre de Bruxelles (Université Libre de Bruxelles) (submittedVersion)
Topics
  • Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
Keywords
citation indexes, journal impact factor, methodological quality, randomised trials, lung cancer, publication bias, trial quality assessment
MeSH terms
Humans, Lung Neoplasms, Periodicals as Topic, Chi-Square Distribution, Randomized Controlled Trials as Topic, Peer Review, Research, Statistics, Nonparametric

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Citation indexes do not reflect methodological quality in lung cancer randomised trials. https://app.cochat.ai/papers/10.1093/annonc/mdg203
The paper
Berghmans, T., Meert, A. P., Mascaux, C., Paesmans, M., Lafitte, J. J., & Sculier, J. P. (2003). Citation indexes do not reflect methodological quality in lung cancer randomised trials. Annals of Oncology. https://doi.org/10.1093/annonc/mdg203

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.