CoChat
Sign Up
Cited by 5scientometrics and bibliometrics researchWeb visibility and informetricsPsychology Research and Bibliometrics

Implicit and explicit research quality score probabilities from ChatGPT

Open access
Thelwall, Mike, Yang, Yunhan·2025·Quantitative Science Studies
DOI 10.1162/qss.a.393
Open this paper in CoChat — free accountOpen PDF

Abstract

The large language model (LLM) ChatGPT’s quality scores for journal articles correlate more strongly with human judgments than some citation-based indicators in most fields. Averaging multiple ChatGPT scores improves the results, apparently exploiting its internal probability model. To leverage these probabilities, we test two novel strategies: requesting percentage likelihoods for scores and extracting the probabilities of alternative tokens in the responses. These probability estimates were used to calculate weighted average scores. Both strategies were evaluated with five iterations of ChatGPT 4o-mini on 96,800 articles submitted to the U.K. Research Excellence Framework (REF) 2021, using departmental average REF2021 quality scores as a proxy for article quality. The data were analyzed separately for each of the 34 field-based REF Units of Assessment. For the first strategy, explicit requests for tables of score percentage likelihoods substantially decreased the value of the scores (lower correlation with the proxy quality indicator). In contrast, weighed averages of score token probabilities slightly increased the correlation with the quality proxy indicator and these probabilities reasonably accurately reflected ChatGPT’s outputs. The token probability leveraging approach is therefore the most accurate method for ranking articles by research quality as well as being cheaper than comparable ChatGPT strategies.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1162/qss.a.393

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • A range of cloud-based and open weights LLMs have an ability to score published journal articles for research quality in the sense of giving scores that correlate positively and statistically significantly with expert review scores.

    “A range of cloud-based and open weights LLMs have an ability to score published journal articles for research quality in the sense of giving scores that correlate positively and statistically significantly with expert review scores (Thelwall, 2025; Thelwall & Mohammadi, 2026; Thelwall & Yang, 2025).”

    From Can Large Language Models Evaluate Grant Proposal Quality? Revisiting the Wennerås and Wold Peer Review Data (2026) · cites this paper as Thelwall & Yang, 2025

    Section 1.2 says ChatGPT quality scores correlate with a research-quality indicator and reports a similar, weaker ability for Gemini. Sections 3.1 and 3.6 clarify that this paper tests ChatGPT against departmental average expert scores as a proxy, not individual article review scores. The cited paper does not establish the claim for open-weight LLMs or state that the reported correlations are statistically significant. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • LLMs have shown a moderately good ability to estimate the research quality of published academic journal articles.

    “LLMs have shown a moderately good ability to estimate the research quality of published academic journal articles (Thelwall, 2025; Thelwall & Yang, 2025)”

    From Can Large Language Models Evaluate Grant Proposal Quality? Revisiting the Wennerås and Wold Peer Review Data (2026) · cites this paper as Thelwall & Yang, 2025

  • Previous research has shown that averaging multiple iterations increases correlations with human judgments.

    “Previous research has shown that averaging multiple iterations increases correlations with human judgments (e.g., Thelwall & Yang, 2025 )”

    From Can Large Language Models Evaluate Grant Proposal Quality? Revisiting the Wennerås and Wold Peer Review Data (2026) · cites this paper as Thelwall & Yang, 2025

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 42

  1. license Mike Thelwall: Methodology, Writing-original draft, Yunhan Yang: Writing-review & editing.
  2. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., & McGrew, B. (2023). Gpt-4 technical report. arXiv preprint arXiv:2303.08774. · arXiv
  3. Ahdritz, G., Qin, T., Vyas, N., Barak, B., & Edelman, B. L. (2024). Distinguishing the knowable from the unknowable with language models. arXiv preprint arXiv:2402.03563 · arXiv
  4. Beno, J. P. (2024). ELECTRA and GPT-4o: Cost-Effective Partners for Sentiment Analysis. arXiv preprint arXiv:2501.00062. · arXiv
  5. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., & Amodei, D. (2020). Language models are few-shot learners. Advances in neural information processing systems, 33, 1877-1901.
  6. Bucher, M. J. J., & Martini, M. (2024). Fine-Tuned 'Small' LLMs (Still) Significantly Outperform Zero-Shot Generative AI Models in Text Classification. arXiv preprint arXiv:2406.08660. · arXiv
  7. Chen, C. M. L., & Lin, W. Y. C. (2018). What indicators matter? The Analysis of Perception toward Research Assessment Indicators and Leiden Manifesto-The Case Study of Taiwan. arXiv preprint arXiv:1809.02953. · arXiv
  8. Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers) (pp. 4171- 4186).
  9. Duan, J., Cheng, H., Wang, S., Zavalny, A., Wang, C., Xu, R., & Xu, K. (2024). Shifting attention to relevance: Towards the uncertainty estimation of large language models. https://openreview.net/forum?id=yZJapMWdHZ
  10. Fu, T., Conde, J., Martínez, G., Grandury, M., & Reviriego, P. (2025). Multiple Choice Questions: Reasoning Makes Large Language Models (LLMs) More Self-Confident Even When They Are Wrong. arXiv preprint arXiv:2501.09775. · arXiv
  11. Gallifant, J., Fiske, A., Levites Strekalova, Y. A., Osorio-Valencia, J. S., Parke, R., Mwavu, R., & Pierce, R. (2024). Peer review of GPT-4 technical report and systems card. PLOS digital health, 3(1), e0000417.
  12. Gretz, S., Halfon, A., Shnayderman, I., Toledo-Ronen, O., Spector, A., Dankin, L., & Dor, L. E. (2023). Zero-shot topical text classification with llms-an experimental study. In Findings of the Association for Computational Linguistics: EMNLP 2023 (pp. 9647- 9676).
  13. Hicks, D., Wouters, P., Waltman, L., De Rijcke, S., & Rafols, I. (2015). Bibliometrics: the Leiden Manifesto for research metrics. Nature, 520(7548), 429-431.
  14. Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., & Kivlichan, I. (2024). Gpt-4o system card. arXiv preprint arXiv:2410.21276. · arXiv
  15. Langfeldt, L., Nedeva, M., Sörlin, S., & Thomas, D. A. (2020). Co-existing notions of research quality: A framework to study context-specific understandings of good research. Minerva, 58(1), 115-137.
  16. Lin, S., Hilton, J., & Evans, O. (2022). Teaching models to express their uncertainty in words. arXiv preprint arXiv:2205.14334. · arXiv
  17. McCain, K. W. (2011). Eponymy and obliteration by incorporation: The case of the "Nash Equilibrium". Journal of the American Society for Information Science and Technology, 62(7), 1412-1424.
  18. Minoura, A., Shimada, Y., Kuwahara, K., Kondo, M., Fukushima, H., & Sugiyama, T. (2024). Medical researchers' perceptions regarding research evaluation: a web-based survey in Japan. BMJ open, 14(5), e079269.
  19. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., & Lowe, R. (2022). Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35, 27730-27744.
  20. Pawitan, Y., & Holmes, C. (2024). Confidence in the reasoning of large language models. arXiv preprint arXiv:2412.15296. · arXiv
  21. Quevedo, E., Salazar, J. Y., Koerner, R., Rivas, P., & Cerny, T. (2024). Detecting hallucinations in large language model generation: A token probability approach. In World Congress in Computer Science, Computer Engineering & Applied Computing (pp. 154-173). Cham: Springer Nature Switzerland.
  22. Robinson-Garcia, N., Costas, R., Nane, G. F., & van Leeuwen, T. N. (2023). Valuation regimes in academia: Researchers' attitudes towards their diversity of activities and academic performance. Research Evaluation, 32(2), 496-514.
  23. Roumeliotis, K. I., Nikolaos, T. D., & Dimitrios, N. K. (2024). Leveraging LLMs in Tourism: A Comparative Study of the latest GPT Omni models and BERT NLP for Customer Review Classification and Sentiment Analysis.
  24. Rushforth, A., & Hammarfelt, B. (2023). The rise of responsible metrics as a professional reform movement: A collective action frames account. Quantitative Science Studies, 4(4), 879-897.
  25. Sakai, H., & Lam, S. S. (2025). Large language models for healthcare text classification: A systematic review. arXiv preprint arXiv:2503.01159. · arXiv
  26. Son, G., Baek, S., Nam, S., Jeong, I., & Kim, S. (2024). Multi-Task Inference: Can Large Language Models Follow Multiple Instructions at Once? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 5606-5627).
  27. Sun, F., Li, N., Wang, K., & Goette, L. (2025). Large Language Models are overconfident and amplify human bias. arXiv preprint arXiv:2505.02151. · arXiv
  28. Thelwall, M. (2025a). Evaluating research quality with Large Language Models: An analysis of ChatGPT's effectiveness with different settings and inputs. Journal of Data and Information Science, 10(1), 7-25. https://doi.org/10.2478/jdis-2025-0011 · DOI
  29. Thelwall, M. (2025b). Is Google Gemini better than ChatGPT at evaluating research quality? Journal of Data and Information Science, 10(2), 1-5. https://doi.org/10.2478/jdis- 2025-0014 with extended version here: https://doi.org/10.6084/m9.figshare.28089206.v1 · DOI
  30. Thelwall, M. (2025c). In which fields do ChatGPT 4o scores align better than citations with research quality? arXiv preprint arXiv:2504.04464. · arXiv
  31. Thelwall, M. (2025d). Responsible Uses of Large Language Models. In Proceedings of the 20th International Society of Scientometrics and Informetrics Conference (ISSI2025) Volume 1. Yerevan, Armenia (pp. 71-80).
  32. Thelwall, M., Jiang, X., & Bath, P. (2025). Estimating the quality of published medical research with ChatGPT. Information Processing & Management, 62(4), 104123. https://doi.org/10.1016/j.ipm.2025.104123 · DOI
  33. Thelwall, M., & Yaghi, A. (2025). In which fields can ChatGPT detect journal article quality? An evaluation of REF2021 results. Trends in Information Management, 13(1), 1-29. https://doi.org/10.48550/arXiv.2409.16695 · DOI
  34. Vajjala, S., & Shimangaud, S. (2025). Text Classification in the LLM Era--Where do we stand? arXiv preprint arXiv:2502.11830. · arXiv
  35. Wang, J. (2013). Citation time window choice for research impact evaluation. Scientometrics, 94(3), 851-872.
  36. Wang, X., Ma, B., Hu, C., Weber-Genzel, L., Röttger, P., Kreuter, F., & Plank, B. (2024). "My Answer is C": First-token probabilities do not match text answers in instruction-tuned language models. arXiv preprint arXiv:2402.14499. · arXiv
  37. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., & Zhou, D. (2022). Chain-of- thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35, 24824-24837.
  38. Wilsdon, J., Allen, L., Belfiore, E., Campbell, P., Curry, S., Hill, S., & Johnson, B. (2015). The metric tide. Report of the independent review of the role of metrics in research assessment and management.
  39. Wu, J., Gu, B., Zhou, R., Xie, K., Snyder, D., Jiang, Y., & Yang, J. (2025). BRIDGE: Benchmarking Large Language Models for Understanding Real-world Clinical Practice Text. arXiv preprint arXiv:2504.19467. · arXiv
  40. Xiong, M., Hu, Z., Lu, X., Li, Y., Fu, J., He, J., & Hooi, B. (2023). Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs. arXiv preprint arXiv:2306.13063. · arXiv
  41. Zhang, S., Dong, L., Li, X., Zhang, S., Sun, X., Wang, S., & Wang, G. (2023). Instruction tuning for large language models: A survey. arXiv preprint arXiv:2308.10792. · arXiv
  42. Zhao, H., Chen, H., Yang, F., Liu, N., Deng, H., Cai, H., & Du, M. (2024). Explainability for large language models: A survey. ACM Transactions on Intelligent Systems and Technology, 15(2), 1-38.

Cited by

Sorted by citations

Authors
Thelwall, M., & Yang, Y.
Affiliations
  • University of Wolverhampton (GB)
  • University of Hong Kong (HK)
Corresponding author
Thelwall, Mike
ORCID
Venue
Quantitative Science Studies
Publisher
MIT Press
Published
2025
Volume
6
Pages
1271-1293
ISSN
2641-3337 (ISSN-L)
DOI
10.1162/qss.a.393
OpenAlex
W4415157432
Cited by
5
References
6
Citations by year
2026: 5
FWCI
3.22
Type
journal article
Language
en
Open access
Licence
Locations
  • Quantitative Science Studies (publishedVersion, cc-by)
  • White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
Economic and Social Research Council (APP43146)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
  • Physical Sciences › Computer Science › Information Systems › Web visibility and informetrics
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
Keywords
ChatGPT, large language models, research quality assessment, journal article quality, Research Excellence Framework, citation metrics, research evaluation, AI-assisted peer review

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review