Citation indexes do not reflect methodological quality in lung cancer randomised trials.
Open accessThe DOI matches its Crossref record; 21 of the 24 references it cites were found in a registry or an index; none of the 9 sources its claims cite is checked for support yet.
Abstract
Background Citation factors are applied to assess scientific work despite the fact that they were developed commercially in order to compare competing journals. The aim of the present study was to determine whether there is a relationship between citation factors and a trial's methodological quality using published randomised trials in lung cancer clinical research. Material and methods All of the randomised trials included in nine systematic reviews performed by the European Lung Cancer Working Party (ELCWP) were assessed using two quality scales (Chalmers and ELCWP). Results One hundred and eighty-one articles were eligible. The median overall ELCWP and Chalmers quality scores were 61.8% and 49.0%, respectively, with a correlation coefficient (r(s)) of 0.74 (P <0.001). A weak association was observed between citation factors and quality scores with the respective correlation coefficients ranging from 0.18 to 0.40 (ELCWP scale) and from 0.21 to 0.38 (Chalmers scale). American authors published trials significantly more often in journals with high citation factors than European or non-American authors (P <0.0001), despite no better methodological quality. Positive trials, which were significantly more likely to be published in journals with higher citation factors, were of no better quality than negative ones. Conclusion Journals with higher citation factors do not appear to publish clinical trials with higher levels of methodological quality, at least for trials in the field of lung cancer research.
Overview
CoChat Summary
TL;DR
In 181 lung cancer randomised trials, citation factors showed only weak correlations with methodological quality scores, despite statistically significant associations. The authors conclude that journals with higher citation factors do not appear to publish methodologically better clinical trials in this field.
Key findings
- Median quality scores were 61.8% on the ELCWP scale and 49.0% on the Chalmers scale, with a correlation of \(r_s = 0.74\) (\(P < 0.001\)) between the two scales.
- Citation factors correlated weakly with quality scores: \(r_s\) ranged from 0.18 to 0.40 for ELCWP and 0.21 to 0.38 for Chalmers.
- American authors published significantly more often in journals with high citation factors than European or non-American authors (\(P < 0.0001\)), but had no better methodological quality; on the Chalmers scale there was a borderline difference favouring European authors.
- Positive trials were significantly more likely to be published in journals with higher citation factors, but showed no quality difference compared with negative trials.
Methods
- All randomised trials included in nine systematic reviews by the European Lung Cancer Working Party were assessed; 181 articles published in 31 journals and one book from 1977 to 2001 were eligible.
- Methodological quality was scored using two published scales, ELCWP and Chalmers, with consensual agreement among medical doctors and a biostatistician.
- Associations between citation factors (IF99, IF Pub, PF, EF) and quality scores were analysed with Spearman rank correlation, Mann-Whitney, Kruskal-Wallis, Student’s t-test, and chi-square tests; a \(P < 0.001\) threshold was used because of multiple testing.
Limitations
- The paper states that results may not generalise outside lung cancer research, where mainly small to moderate randomised trials with equivocal results are available; extrapolation to large level I trials should be done with circumspection.
- The paper states that only articles in English, French, or Dutch were eligible, and that citation factor values were not available for all articles (IF99 in 171, IF Pub in 150, PF in 175, EF in 39).
- Evident from the text: the study is observational and correlational, so it cannot establish causality; EF analyses were based on only 39 articles, and EF was not considered for American authors because only two American trials were published in European journals.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 0 Not valid · 8 Not checked · 24 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Positive trials were published significantly more often in journals with higher citation factors (Table 7).”
§ Citation indexes do not reflect methodological quality in lung cancer randomised trials > Impact of trial significance on quality scores and citation factors
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A weak correlation between the scores and the number of eligible patients per article was found (r s = 0.19; P = 0.01).”
§ Citation indexes do not reflect methodological quality in lung cancer randomised trials > Quality scores
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that American academics published statistically significantly more often than Europeans in journals with high citation factors (P <0.0001), whichever type of citation factor was used for comparison.”
§ Citation indexes do not reflect methodological quality in lung cancer randomised trials > Relationship between citation factors and quality score according to the geographical distribution of the authors
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A weak association was observed between citation factors and quality scores with the respective correlation coefficients ranging from 0.18 to 0.40 (ELCWP scale) and from 0.21 to 0.38 (Chalmers scale).”
- Also stated in: § Discussion · plain
“we observe a weak, although statistically significant, relationship between citations factors (IF, PF or EF) and the methodological quality of articles, as assessed by two different quality scales.”
§ Citation indexes do not reflect methodological quality in lung cancer randomised trials > Abstract
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Also stated in: § Discussion · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A good correlation between the scores was observed with an r s of 0.74 (P <0.001).”
§ Citation indexes do not reflect methodological quality in lung cancer randomised trials > Quality scores
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
9 more claims
Create a free account to see all 32 claims and every source verdict.
- It may state two separate claims as one. 12
- Probably not a substantial claim. 7
- The quote only partly states this claim. 1
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Citation indexes do not reflect methodological quality in lung cancer randomised trials > Discussion
“55.4% (86) | 47.5% (46) | 0.009”
§ Citation indexes do not reflect methodological quality in lung cancer randomised trials > Discussion
“IF Pub (n) | 2.628 (86) | 5.473 (43) | 2.736 (13) | 2.095 (5) | 2.780 (3) | 0.004”
Citations
Cites
Why the impact factor of journals should not be used for evaluating research
Seglen, P. O. (1997) · BMJ · doi:10.1136/bmj.314.7079.497 · Cited by 2,447 · cited 5×
“Studies published in other languages are not included in this analysis, but the majority of articles published in non-English language journals are not included in the ISI database [ 5 ], so it seems unlikely that including them would change our results.”
BackgroundInfluentialA method for assessing the quality of a randomized control trial
Chalmers, T. C., Smith, H., Blackburn, B., Silverman, B., Schroeder, B., Reitman, D., & Ambroz, A. (1981) · Controlled Clinical Trials · doi:10.1016/0197-2456(81)90056-8 · Cited by 1,349
Language bias in randomised controlled trials published in English and German
Egger, M., Zellweger-Zähner, T., Schneider, M., Junker, C., Lengeler, C., & Antes, G. (1997) · The Lancet · doi:10.1016/s0140-6736(97)02419-7 · Cited by 1,028
College Libraries and Chemical Education
Gross, P. L. K., & Gross, E. M. (1927) · Science · doi:10.1126/science.66.1713.385 · Cited by 649
Fortnightly Review: How can impact factors be improved?
Garfield, E. (1996) · BMJ · doi:10.1136/bmj.313.7054.411 · Cited by 484
Meert, A.-P., Paesmans, M., Berghmans, T., Martin, B., Mascaux, C., Vallot, F., Verdebout, J.-M., Lafitte, J.-J., & Sculier, J.-P. (2001) · BMC Cancer · doi:10.1186/1471-2407-1-5 · Cited by 281
Self-citationMascaux, C., Paesmans, M., Berghmans, T., Branle, F., Lafitte, J. J., Lemaı̂tre, F., Meert, A. P., Vermylen, P., & Sculier, J. P. (2000) · Lung Cancer · doi:10.1016/s0169-5002(00)00127-6 · Cited by 270 · cited 1×
“Five reviews were dedicated to small-cell lung cancer [7, 8, 10 , 11, 14]; they assessed the role of prophylactic cranial irradiation, thoracic irradiation in limited disease, the respective role of cisplatin and etoposide in comparison with other drugs, the place of maintenance chemotherapy after induction treatment…”
BackgroundSelf-citationOpen peer review: a randomised controlled trial
Walsh, E., Rooney, M., Appleby, L., & Wilkinson, G. (2000) · British Journal of Psychiatry · doi:10.1192/bjp.176.1.47 · Cited by 261 · cited 1×
“For example, when the authors know the name of the reviewer, peer-review is of a higher quality than for unsigned forms [ 18 ].”
BackgroundCitation rates and journal impact factors are not suitable for evaluation of research
Seglen, P. O. (1998) · Acta Orthopaedica Scandinavica · doi:10.3109/17453679809000920 · Cited by 240
Invited Review Article: Bibliometrics and Evaluation of Research Performance
Luukkonen, T. (1990) · Annals of Medicine · doi:10.3109/07853899009147259 · Cited by 135
Cited by
Dong, P., Loh, M., & Mondry, A. (2005) · Biomedical Digital Libraries · doi:10.1186/1742-5581-2-7 · Cited by 228
Glänzel, W., Debackere, K., Thijs, B., & Schubert, A. (2006) · Scientometrics · doi:10.1007/s11192-006-0098-9 · Cited by 187
Allen, L., Jones, C., Dolby, K., Lynn, D., & Walport, M. (2009) · PLoS ONE · doi:10.1371/journal.pone.0005910 · Cited by 167
“There is also a wide-ranging critique of bibliometric indicators [1,2,3]; knowing”
BackgroundGoldkorn, T., Filosto, S., & Chung, S. (2014) · Antioxidants and Redox Signaling · doi:10.1089/ars.2013.5469 · Cited by 121
“Despite nearly 40 years of research involving a worldwide scientific community (13, 95), we are still far away from a sound understanding of the molecular mechanisms underlying smoking-induced chronic lung diseases, due to the diverse pathological respiratory conditions attributed to smoking (6, 79).”
A bibliometric study of highly cited reviews in the S cience C itation I ndex expanded™
Ho, Y., & Kahn, M. (2013) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.22974 · Cited by 117
“However, it is also reported that citation frequencies do not appear to reflect methodological quality in lung cancer randomized trials (Berghmans et al., 2003).”
Background
Metadata
- Authors
- Berghmans, T., Meert, A. P., Mascaux, C., Paesmans, M., Lafitte, J. J., & Sculier, J. P.
- Affiliations
- Institut Jules Bordet (BE)
- Centre Hospitalier Universitaire de Lille (FR)
- Corresponding author
- Berghmans, T.
- Venue
- Annals of Oncology
- Publisher
- Elsevier BV
- Published
- 2003-05
- Volume
- 14
- Issue
- 5
- Pages
- 715-721
- ISSN
- 0923-7534 (ISSN-L)
- OpenAlex
- W2162337243
- Cited by
- 38
- References
- 24
- Citations by year
- 2026: 1 · 2022: 1 · 2021: 1 · 2020: 1 · 2018: 1 · 2017: 2 · 2016: 2 · 2014: 3 · 2013: 4 · 2012: 1
- FWCI
- 4.13
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- www.elsevier.com/open-access/userlicense/1.0/ (vor, 6095 d)
- Locations
- Annals of Oncology (publishedVersion)
- PubMed (publishedVersion)
- Dépôt institutionnel de l'Université libre de Bruxelles (Université Libre de Bruxelles) (submittedVersion)
- Topics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- citation indexes, journal impact factor, methodological quality, randomised trials, lung cancer, publication bias, trial quality assessment
- MeSH terms
- Humans, Lung Neoplasms, Periodicals as Topic, Chi-Square Distribution, Randomized Controlled Trials as Topic, Peer Review, Research, Statistics, Nonparametric
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Citation indexes do not reflect methodological quality in lung cancer randomised trials. https://app.cochat.ai/papers/10.1093/annonc/mdg203
- The paper
- Berghmans, T., Meert, A. P., Mascaux, C., Paesmans, M., Lafitte, J. J., & Sculier, J. P. (2003). Citation indexes do not reflect methodological quality in lung cancer randomised trials. Annals of Oncology. https://doi.org/10.1093/annonc/mdg203
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.