Comparing the topological properties of real and artificially generated scientific manuscripts
Open accessThe DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.
Abstract
Recent years have witnessed the increase of competition in science. While promoting the quality of research in many cases, an intense competition among scientists can also trigger unethical scientific behaviors. To increase the total number of published papers, some authors even resort to software tools that are able to produce grammatical, but meaningless scientific manuscripts. Because automatically generated papers can be misunderstood as real papers, it becomes of paramount importance to develop means to identify these scientific frauds. In this paper, I devise a methodology to distinguish real manuscripts from those generated with SCIGen, an automatic paper generator. Upon modeling texts as complex networks (CN), it was possible to discriminate real from fake papers with at least 89 % of accuracy. A systematic analysis of features relevance revealed that the accessibility and betweenness were useful in particular cases, even though the relevance depended upon the dataset. The successful application of the methods described here show, as a proof of principle, that network features can be used to identify scientific gibberish papers. In addition, the CN-based approach can be combined in a straightforward fashion with traditional statistical language processing methods to improve the performance in identifying artificially generated papers.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Detecting h-index manipulation through self-citation analysis
Bartneck, C., & Kokkelmans, S. (2011) · Scientometrics · doi:10.1007/s11192-010-0306-5 · Cited by 301
A systematic comparison of supervised classifiers
Amancio, D. R., Comin, C. H., Casanova, D., Travieso, G., Bruno, O. M., Rodrigues, F. A., & da Fontoura Costa, L. (2014) · PLoS ONE · doi:10.1371/journal.pone.0094137 · Cited by 225
Self-citationThe complexity of Chinese syntactic dependency networks
Liu, H. (2008) · Physica A: Statistical Mechanics and its Applications · doi:10.1016/j.physa.2008.01.069 · Cited by 92
Cited by
Random Walks: A Review of Algorithms and Applications
Xia, F., Liu, J., Nie, H., Fu, Y., Wan, L., & Kong, X. (2020) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2019.2952908 · Cited by 295
“applications of random walks on graphs [45], [46], text analysis [47], science of science [48], and knowledge discovery [49].”
BackgroundSohail, S. S., Farhat, F., Himeur, Y., Nadeem, M., Madsen, D. Ø., Singh, Y., Atalla, S., & Mansoor, W. (2023) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2023.101675 · Cited by 192
“It is worth noting that the detection of artificially generated fake text and meaningless information is not a new challenge (Amancio, 2015; Jawahar et al., 2020).”
Cabanac, G., Labbé, C., & Magazinov, A. (2021) · arXiv · doi:10.48550/arxiv.2107.06751 · Cited by 224
“Automatic detection of computer generated text has already drawn attention in the past, see for example (Amancio, 2015; Cabanac & Labbé, in press; Dalkilic, Clark, Costello, & Radivo-jac, 2006; Labbé, Labbé, & Portet, 2016).”
BackgroundAssisting academics to identify computer generated writing
Abd-Elaal, E.-S., Gamage, S. H. P. W., & Mills, J. E. (2022) · European Journal of Engineering Education · doi:10.1080/03043797.2022.2046709 · Cited by 62
Byrne, J. A., & Labbé, C. (2017) · Scientometrics · doi:10.1007/s11192-016-2209-6 · Cited by 50
“Some tools aim to identify automatically-generated publications (Amancio 2015; Fahrenberg et al. 2014; Labbé and Labbé 2013), and have been adopted by publishers [Springer Nature with SciDetect(1) as well as the open archive ArXiv (Ginsparg 2014)].”
BackgroundMethod
Metadata
- Authors
- Amancio, D. R.
- Venue
- Scientometrics
- Published
- 2015-07-15
- Volume
- 105
- Issue
- 3
- Pages
- 1763-1779
- ISSN
- 0138-9130 (ISSN-L)
- 1588-2861
- OpenAlex
- W1880789213
- Cited by
- 65
- References
- 3
- FWCI
- 4.78
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Comparing the topological properties of real and artificially generated scientific manuscripts. https://app.cochat.ai/papers/10.1007/s11192-015-1637-z
- The paper
- Amancio, D. R. (2015). Comparing the topological properties of real and artificially generated scientific manuscripts. Scientometrics. https://doi.org/10.1007/s11192-015-1637-z
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.