Classification and analysis of PubPeer comments: How a web journal club is used
Open accessThe DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Abstract
This study explores the use of PubPeer by the scholarly community, to understand the issues discussed in an online journal club, the disciplines most commented on, and the characteristics of the most prolific users. A sample of 39,985 posts about 24,779 publications were extracted from PubPeer in 2019 and 2020. These comments were divided into seven categories according to their degree of seriousness (Positive review, Critical review, Lack of information, Honest errors, Methodological flaws, Publishing fraud, and Manipulation). The results show that more than two‐thirds of comments are posted to report some type of misconduct, mainly about image manipulation. These comments generate most discussion and take longer to be posted. By discipline, Health Sciences and Life Sciences are the most discussed research areas. The results also reveal “super commenters,” users who access the platform to systematically review publications. The study ends by discussing how various disciplines use the site for different purposes.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Sentiment strength detection for the social web
Thelwall, M., Buckley, K., & Paltoglou, G. (2012) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.21662 · Cited by 1,062 · cited 1×
“However, more often, categorization through keywords has been used to classify posts on social networks (Carreño & Winbladh, 2013; Salehan & Kim, 2016; Thelwall et al., 2012) when the volume of entries is considerable and an automatic approach is necessary.”
BackgroundPeer Review: A Flawed Process at the Heart of Science and Journals
Smith, R. (2006) · Journal of the Royal Society of Medicine · doi:10.1177/014107680609900414 · Cited by 880
The publishing delay in scholarly peer-reviewed journals
Björk, B.-C., & Solomon, D. (2013) · Journal of Informetrics · doi:10.1016/j.joi.2013.09.001 · Cited by 432
Gender bias in scholarly peer review
Helmer, M., Schottdorf, M., Neef, A., & Battaglia, D. (2017) · eLife · doi:10.7554/elife.21718 · Cited by 380 · cited 1×
“In some cases, the subjectivity of the process has been reported when gender (Helmer et al., 2017), language (Herrera, 1999), and social biases (Walker et al., 2015) have been revealed.”
BackgroundThe Prevalence of Inappropriate Image Duplication in Biomedical Research Publications
Bik, E. M., Casadevall, A., & Fang, F. C. (2016) · mBio · doi:10.1128/mbio.00809-16 · Cited by 287 · cited 1×
“This result coincides with previous studies about the disciplinary distribution of retractions (Ortega, 2020; Tripathi et al., 2019) and highlights the great problem of image manipulation in Biomedicine today (Bik et al., 2016; Oksvold, 2015).”
ResultOpen peer review: a randomised controlled trial
Walsh, E., Rooney, M., Appleby, L., & Wilkinson, G. (2000) · British Journal of Psychiatry · doi:10.1192/bjp.176.1.47 · Cited by 261 · cited 1×
“Walsh et al. (2000) did find striking differences, observing that non-anonymous reviewers are more prone to being more polite and to recommending the publication.”
BackgroundWhat is open peer review? A systematic review
Ross-Hellauer, T. (2017) · F1000Research · doi:10.12688/f1000research.11369.1 · Cited by 232
Low agreement among reviewers evaluating the same NIH grant applications
Pier, E. L., Brauer, M., Filut, A., Kaatz, A., Raclaw, J., Nathan, M. J., Ford, C. E., & Carnes, M. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1714379115 · Cited by 210
The changing forms and expectations of peer review
Horbach, S. P. J. M., & Halffman, W. (. W. (2018) · Research Integrity and Peer Review · doi:10.1186/s41073-018-0051-5 · Cited by 179
Teixeira da Silva, J. A., & Dobránszki, J. (2015) · Accountability in Research · doi:10.1080/08989621.2014.899909 · Cited by 158
The ability of different peer review procedures to flag problematic publications
Horbach, S. P. J. M., & Halffman, W. (2019) · Scientometrics · doi:10.1007/s11192-018-2969-2 · Cited by 73
Bordignon, F. (2020) · Scientometrics · doi:10.1007/s11192-020-03536-z · Cited by 51 · cited 1×
“More recently, Bordignon (2020) used PubPeer comments to test the influence of negative comments in the retraction/correction of papers, concluding that PubPeer contributes more to the correction of science than negative citations.”
MethodAnalysis of single comments left for bioRxiv preprints till September 2019
Malički, M., Costello, J., Alperin, J. P., & Maggio, L. A. (2021) · Biochemia medica · doi:10.11613/bm.2021.020201 · Cited by 35
Malički, M., Costello, J., Alperin, J. P., & Maggio, L. A. (2020) · doi:10.1101/2020.10.14.340083 · Cited by 35
Incidence of Data Duplications in a Randomly Selected Pool of Life Science Publications
Oksvold, M. P. (2016) · Science and Engineering Ethics · doi:10.1007/s11948-015-9668-7 · Cited by 20
Research works printed but not linked: 39
- Bik E. M., 2019, PubPeeer—A website to comment on scientific papers
- 10.1104/pp.15.01443 · DOI
- 10.18352/lq.10347 · DOI
- 10.1080/08989621.2017.1327814 · DOI
- 10.4135/9781483348971.n24 · DOI
- 10.1109/ICSE.2013.6606604 · DOI · Paper page
- 10.1186/s13059-018-1584-0 · doi:10.1186/s13059-018-1584-0
- 10.1007/s11192-017-2489-5 · DOI
- 10.1038/17194 · DOI
- 10.3758/s13428-017-0955-x · DOI
- 10.1001/jama.1994.03520020022005 · DOI · Paper page
- Kelly J., 2014, Peer review in scientific publications: Benefits, critiques, & a survival guide, Ejifcc, 25, 227
- 10.1016/j.tig.2015.03.006 · DOI · Paper page
- 10.1007/s10805-008-9068-6 · DOI
- 10.1080/08989621.2018.1548283 · DOI
- 10.1007/978-3-030-12131-0 · doi:10.1007/978-3-030-12131-0
- 10.1016/j.oraloncology.2004.11.001 · DOI
- 10.1038/ncomms13626 · DOI
- 10.1007/s11192-017-2399-6 · DOI
- 10.1108/AJIM-06-2018-0153 · DOI
- 10.1002/leap.1339 · DOI
- 10.1177/0192623318754792 · DOI
- 10.1080/08989621.2012.650948 · DOI · Paper page
- 10.12688/f1000research.11369.2 · DOI · Paper page
- 10.1093/brain/123.9.1964 · DOI · Paper page
- 10.1016/j.dss.2015.10.006 · DOI · Paper page
- 10.3138/jsp.50.1.08 · DOI
- 10.1038/d41586-020-01363-z · DOI
- Teixeira da Silva J. A., 2018, The issue of comment ownership and copyright at PubPeer, Journal of Educational Media & Library Sciences, 55, 181
- 10.1108/OIR-04-2018-0139 · DOI
- 10.1104/pp.15.01939 · DOI
- 10.1177/0165551520938678 · DOI
- Torny D., 2018, PEERE international conference on peer review, PEERECOST network, Mar 2018, Rome, Italy
- 10.1080/17521742.2013.865333 · DOI
- 10.14429/djlit.39.2.14000 · DOI
- 10.1136/bmj.318.7175.23 · DOI · Paper page
- Wager E., 2017, Eighth international congress on peer review and scientific publication, September 2017, Chicago, USA
- 10.12688/f1000research.6012.2 · DOI
- 10.1038/503454a · DOI
Cited by
The entities enabling scientific fraud at scale are large, resilient, and growing rapidly
Richardson, R. A. K., Hong, S. S., Byrne, J. A., Stoeger, T., & Amaral, L. A. N. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2420092122 · Cited by 129
The PubPeer conundrum: administrative challenges in research misconduct proceedings
Caron, M. M., Lye, C. T., Bierer, B. E., & Barnes, M. (2025) · Accountability in Research · doi:10.1080/08989621.2024.2390007 · Cited by 14
“The study further found that publications relating to health sciences and life sciences received the vast majority of comments appearing on PubPeer, and that 77% and 79% of the comments relating to articles in these fields, respectively, were related to suspected fraud (Ortega 2021).”
InfluentialWhen AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research
Son, G., Hong, J., Fan, H., Nam, H., Ko, H., Lim, S., Song, J., Choi, J., Paulo, G., Yu, Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.11855 · Cited by 5
“Following [33], we query initial searches using alphabets, extract high-frequency keywords from the returned paper titles, re-query using those keywords, and scrape each paper’s metadata (title, authors, venue) alongside the entire comments.”
Prevalence of Problematic Papers in Non-Coding RNA Research
Lou, Y., Zhou, Z., Shen, Z., & Li, M. (2024) · doi:10.1101/2024.08.28.607530 · Cited by 3
Metadata
- Authors
- Ortega, J. L.
- Venue
- Journal of the Association for Information Science and Technology
- Publisher
- Wiley
- Published
- 2021-08-24
- Volume
- 73
- Issue
- 5
- Pages
- 655-670
- ISSN
- 2330-1635 (ISSN-L)
- 2330-1643
- OpenAlex
- W3194249382
- Cited by
- 37
- References
- 15
- FWCI
- 3.25
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.