Estimating the quality of academic books from their descriptions with ChatGPT
Open accessThe claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLMs have shown a moderately good ability to estimate the research quality of other academic documents, such as research environment statements, conference papers and books.
“and other academic documents, such as research environment statements (Kousha et al., 2025), conference papers (Thelwall & Yaghi, 2025a) and books (Thelwall & Cox, 2025)”
From Can Large Language Models Evaluate Grant Proposal Quality? Revisiting the Wennerås and Wold Peer Review Data (2026) · cites this paper as Thelwall & Cox, 2025
The ‘Comparison with citations’ section reports a moderate positive correlation between ChatGPT book-quality scores and citation counts. It also says the evidence for either as a quality indicator is not strong and that both may be weak indicators. Thus, the source supports some ability to estimate book quality, but ‘moderately good’ overstates its conclusion. The other document types have separate citations and are not judged here. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482 · see citing works
Bibliometrics: The Leiden Manifesto for Research Metrics
Hicks, D., Wouters, P., Waltman, L., de Rijcke, S., & Rafols, I. (2015) · Nature · doi:10.1038/520429a · Cited by 2,828 · see citing works
Handbook of Quantitative Science and Technology Research
Moed, H. F., Glänzel, W., & Schmoch, U. (2005) · Springer eBooks · doi:10.1007/1-4020-2755-9 · Cited by 1,385 · see citing works
The Metric Tide: Independent Review of the Role of Metrics in Research Assessment and Management
Wilsdon, J. (2015) · doi:10.4135/9781473978782 · Cited by 610 · see citing works
Citation time window choice for research impact evaluation
Wang, J. (2012) · Scientometrics · doi:10.1007/s11192-012-0775-9 · Cited by 427 · see citing works
van Raan, A. F. J. (2004) · Springer eBooks · doi:10.1007/1-4020-2755-9_2 · Cited by 279 · see citing works
ChatGPT: A Case Study on Copyright Challenges for Generative Artificial Intelligence Systems
Lucchi, N. (2024) · European Journal of Risk Regulation · doi:10.1017/err.2023.59 · Cited by 173 · see citing works
Assessing the citation impact of books: The role of Google Books, Google Scholar, and Scopus
Kousha, K., Thelwall, M. A., & Rezaie, S. (2011) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.21608 · Cited by 161 · see citing works
Self-citationLangfeldt, L., Nedeva, M., Sörlin, S., & Thomas, D. A. (2019) · Minerva · doi:10.1007/s11024-019-09385-2 · Cited by 154 · see citing works
Three practical field normalised alternative indicator formulae for research evaluation
Thelwall, M. A. (2017) · Journal of Informetrics · doi:10.1016/j.joi.2016.12.002 · Cited by 104 · see citing works
Self-citationA Review of Citation Analysis Methodologies for Collection Management
Hoffmann, K., & Doucette, L. (2012) · College & Research Libraries · doi:10.5860/crl-254 · Cited by 103 · see citing works
Can ChatGPT evaluate research quality?
Thelwall, M. (2024) · Journal of Data and Information Science · doi:10.2478/jdis-2024-0013 · Cited by 68 · see citing works
Self-citationCuartas, G. V., Lucio‐Arias, D., & Loet Leydesdorff, A. (2016) · El Profesional de la Informacion · doi:10.3145/epi.2016.ene.05 · Cited by 66 · see citing works
Torres‐Salinas, D., Robinson‐García, N., Cabezas-Clavijo, Á., & Jiménez-Contreras, E. (2013) · Scientometrics · doi:10.1007/s11192-013-1168-4 · Cited by 56 · see citing works
Thelwall, M. A. (2024) · Journal of Data and Information Science · doi:10.2478/jdis-2025-0011 · Cited by 54 · see citing works
Self-citationAn automatic method for extracting citations from Google Books
Kousha, K., & Thelwall, M. A. (2014) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23170 · Cited by 51 · see citing works
Self-citationWord Association Thematic Analysis
Thelwall, M. A. (2021) · Synthesis lectures on information concepts, retrieval, and services · doi:10.2200/s01071ed1v01y202012icr072 · Cited by 37 · see citing works
Self-citationIs LLM a Reliable Reviewer? A Comprehensive Evaluation of LLM on Automatic Paper Reviewing Tasks
Zhou, R., Chen, L., & Yu, K. (2024) · Proceedings of the Language Resources and Evaluation Conference · doi:10.63317/48d359hjdvog · Cited by 28 · see citing works
Samuels, D. (2013) · PS Political Science & Politics · doi:10.1017/s1049096513001054 · Cited by 26 · see citing works
Zhu, Y., Yan, E., Peroni, S., & Che, C. (2019) · Scientometrics · doi:10.1007/s11192-019-03311-9 · Cited by 26 · see citing works
Yang, S., Xing, X., Qi, F., & Grácio, M. C. C. (2021) · Scientometrics · doi:10.1007/s11192-020-03808-8 · Cited by 18 · see citing works
Walker, K. W. (2021) · The Journal of Academic Librarianship · doi:10.1016/j.acalib.2021.102450 · Cited by 17 · see citing works
From Narratives to Intensification and Hyperbole: Promotional Uses of Book Blurbs
Cacchiani, S. (2007) · Cited by 13 · see citing works
In which fields can ChatGPT detect journal article quality? An evaluation of REF2021 results
Thelwall, M. A., & Yaghi, A. (2024) · Trends in Information Management · doi:10.48550/arxiv.2409.16695 · Cited by 10 · see citing works
Self-citationMei, Z. (2020) · The Journal of Academic Librarianship · doi:10.1016/j.acalib.2020.102252 · Cited by 7 · see citing works
Role of citation and non-citation metrics in predicting the educational impact of textbooks
Maleki, A., Abbaspour, J., Jowkar, A., & Sotudeh, H. (2023) · Library Hi Tech · doi:10.1108/lht-06-2022-0297 · Cited by 6 · see citing works
Tyler, D. C., & Boudreau, S. O. (2024) · The Journal of Academic Librarianship · doi:10.1016/j.acalib.2024.102919 · Cited by 6 · see citing works
THE SYNTACTIC FEATURES OF PROMOTIONAL LANGUAGE IN BOOK BLURBS
Bacić, M. (2021) · Nasledje Kragujevac · doi:10.46793/naskg2148.117b · Cited by 4 · see citing works
Evaluating the quality of published medical research with ChatGPT
Thelwall, M., Jiang, X., & Bath, P. A. (2024) · arXiv · doi:10.48550/arxiv.2411.01952
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Thelwall, M. A. (2025) · Scientometrics · doi:10.1007/s11192-025-05361-8 · Cited by 33 · see citing works
Self-citationKamińska, A. M. (2026) · The Journal of Academic Librarianship · doi:10.1016/j.acalib.2026.103218 · Cited by 10 · see citing works
Zhu, Y., Haunschild, R., Ye, J., & Xie, H. X. (2026) · Scientometrics · doi:10.1007/s11192-026-05570-9 · Cited by 5 · see citing works
Zhu, Y., Jia, Y., Zhu, Y., & Ye, J. (2026) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.70079 · Cited by 2 · see citing works
Sandström, U., & Thelwall, M. (2026) · Journal of Data and Information Science · doi:10.1515/jdis-2026-0048 · Cited by 1 · see citing works
Self-citationA KNN-Based Bilingual Book Recommendation System with Gamification and Learning Analytics
Kassenkhan, A. M. (2026) · Information · doi:10.3390/info17020120 · Cited by 1 · see citing works
Jiangbo, L., Li, J., Shihang, N., Wenhao, O., & Jiang, C. (2025) · Data Science and Informetrics · doi:10.1016/j.dsim.2025.11.002
Zhang, K., Jiang, B., Guo, Y., & Liu, X. (2026) · Research Square · doi:10.21203/rs.3.rs-9463160/v1
Kim, E., Min, S., & Chiu, J. L. (2026) · Journal of Librarianship and Information Science · doi:10.1177/09610006261468375
Assessing Outputs Other than Academic Journal Articles
Thelwall, M. A. (2026) · White Rose University Press eBooks · doi:10.22599/quantitativemethods.p
Self-citation
- Authors
- Thelwall, M., & Cox, A.
- Affiliations
- University of Sheffield (GB)
- Corresponding author
- Thelwall, Mike
- Venue
- The Journal of Academic Librarianship
- Publisher
- Elsevier BV
- Published
- 2025-03
- Volume
- 51
- Issue
- 2
- Pages
- 103023
- ISSN
- 0099-1333 (ISSN-L)
- OpenAlex
- W4407553542
- Cited by
- 9
- References
- 24
- Citations by year
- 2026: 7 · 2025: 2
- FWCI
- 5.79
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- The Journal of Academic Librarianship (publishedVersion)
- White Rose Research Online (University of Leeds, The University of Sheffield, University of York) (acceptedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Physical Sciences › Computer Science › Information Systems › Library Science and Information
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Publishing and Scholarly Communication
- Keywords
- ChatGPT, research quality assessment, academic books, citation counts, book acquisition, library collection evaluation, book descriptions, arts and humanities, edited collections, textbooks, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
