Assessing citation integrity in biomedical publications: corpus annotation and NLP models
Open accessThe DOI matches its Crossref record; 38 of the 40 references it cites were found in a registry or an index; 3 of the 6 sources its claims cite were checked for support.
Abstract
MOTIVATION: Citations have a fundamental role in scholarly communication and assessment. Citation accuracy and transparency is crucial for the integrity of scientific evidence. In this work, we focus on quotation errors, errors in citation content that can distort the scientific evidence and that are hard to detect for humans. We construct a corpus and propose natural language processing (NLP) methods to identify such errors in biomedical publications. RESULTS: We manually annotated 100 highly-cited biomedical publications (reference articles) and citations to them. The annotation involved labeling citation context in the citing article, relevant evidence sentences in the reference article, and the accuracy of the citation. A total of 3063 citation instances were annotated (39.18% with accuracy errors). For NLP, we combined a sentence retriever with a fine-tuned claim verification model to label citations as ACCURATE, NOT_ACCURATE, or IRRELEVANT. We also explored few-shot in-context learning with generative large language models. The best performing model-which uses citation sentences as citation context, the BM25 model with MonoT5 reranker for retrieving top-20 sentences, and a fine-tuned MultiVerS model for accuracy label classification-yielded 0.59 micro-F1 and 0.52 macro-F1 score. GPT-4 in-context learning performed better in identifying accurate citations, but it lagged for erroneous citations (0.65 micro-F1, 0.45 macro-F1). Citation quotation errors are often subtle, and it is currently challenging for NLP models to identify erroneous citations. With further improvements, the models could serve to improve citation quality and accuracy. AVAILABILITY AND IMPLEMENTATION: We make the corpus and the best-performing NLP model publicly available at https://github.com/ScienceNLP-Lab/Citation-Integrity/.
Overview
CoChat Summary
TL;DR
The paper builds an annotated biomedical corpus for citation integrity, focusing on quotation/citation content errors that are hard for humans to detect, and releases it publicly. It then benchmarks retrieval and classification NLP models, finding that the best system reaches only moderate performance, especially on erroneous citations.
Key findings
- The authors manually annotated 100 highly-cited biomedical publications and their citations, producing 3063 citation instances; 39.18% were labeled with accuracy errors.
- The annotated corpus contains 3420 citation context sentences and 3791 evidence sentences; on average, there were 1.12 context and 1.24 evidence sentences per citation.
- For citation accuracy classification, the best-performing model used citation sentences as context, BM25 + MonoT5 to retrieve the top-20 evidence sentences, and a fine-tuned MultiVerS classifier; it achieved 0.59 micro-F1 and 0.52 macro-F1.
- GPT-4 in-context learning performed better on accurate citations but worse on errors, with 0.65 micro-F1 and 0.45 macro-F1; in the detailed results, GPT-4 had 0.80 F1 and 0.90 recall for ACCURATE citations, but very poor performance on NOT_ACCURATE.
Methods
- The study collected 100 highly-cited full-text biomedical research articles from PubMed Central Open Access, then sampled citing articles and annotated citation instances in three phases with five annotators.
- Annotation labeled three components: citation context in the citing article, evidence sentences in the reference article, and citation accuracy using fine-grained labels later collapsed to ACCURATE / NOT_ACCURATE / IRRELEVANT for modeling.
- Models included a PubMedBERT sentence classifier for citation context, BM25 + MonoT5 for evidence retrieval, MultiVerS for citation accuracy classification, and GPT-3.5-turbo / GPT-4 few-shot in-context learning; evaluation used precision, recall, F1, recall@k, MRR, and McNemar’s test.
Limitations
- The paper states that annotation agreement was only fair for evidence sentence annotation and citation accuracy classification; it suggests domain expertise might have improved agreement but was not feasible.
- The study did not consider whether a citation is significant/perfunctory for the citing article, and it excluded citations whose relevant evidence was in tables, figures, or Supplementary Material.
- The text makes evident that the modeling task is difficult because retrieval missed many relevant sentences: even with top-20 retrieval, almost half of annotated relevant sentences were missed, and model performance on citation errors was low.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
2 Valid · 0 Not valid · 2 Not checked · 53 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“However, citation inaccuracies and distortions are common (Wager and Middleton 2008, Jergas and Baethge 2015, Pavlovic et al. 2021).”
§ Assessing citation integrity in biomedical publications: corpus annotation and NLP models > 1 Introduction
What the sources say
- Not checked — no full text of the sourcecites Wager and Middleton 2008 source
- Supportedcites Jergas and Baethge 2015 source
The Discussion reports that all included studies found substantial quotation-error rates and estimates problems with about one fourth of references in medical articles. This supports the citing sentence’s characterization of citation inaccuracies and distortions as common. (LLM, conf=0.94)
- Supportedcites Pavlovic et al. 2021 source
The Abstract reports inaccurate citations in 11% and 15% of articles in two study sets and concludes that they are common in biomedical literature. The Results identify citations of nonexistent findings and inaccurate interpretations, supporting the reference to distortions. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Retrieving top-20 sentences with BM25 + MonoT5 reranker has a recall of 0.53, which indicates significant differences between the annotated and automatically retrieved sentences.”
§ Assessing citation integrity in biomedical publications: corpus annotation and NLP models > 4 Discussion > 4.2 NLP models
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We obtained high agreement on citation context annotation (κ = 0.96), as expected, as the agreement was measured at the sentence level and in most cases, citation context sentences are simply the citances.”
§ Assessing citation integrity in biomedical publications: corpus annotation and NLP models > 3 Results > 3.2 Inter-annotator agreement
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results show that even with top-20 sentence retrieval, almost half of the sentences annotated as relevant were missed.”
§ Assessing citation integrity in biomedical publications: corpus annotation and NLP models > 3 Results > 3.3 NLP results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A simplified formulation that used BM25 + MonoT5 reranker for sentence retrieval and MultiVerS for label classification only significantly improved results.”
§ Assessing citation integrity in biomedical publications: corpus annotation and NLP models > 4 Discussion > 4.2 NLP models
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
25 more claims
Create a free account to see all 57 claims and every source verdict.
- It may state two separate claims as one. 20
- Probably not a substantial claim. 11
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“When citations are missing, inaccurate, or fabricated, the resulting break in the evidence chain can obscure logical dependencies, weaken argumentation, and jeopardize academic integrity at the level of both individual papers and the broader literature [10, 22, 26].”
From CiteAudit: You Cited It, But Did You Read It? A Benchmark for Verifying Scientific References in the LLM Era (2026) · cites this paper as [22]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Given a report with citation content, tasks such as cited context identification, evidence sentence retrieval, and citation accuracy classification are commonly used to analyze citation quality Sarol et al. (2024).”
From ReportBench: Evaluating Deep Research Agents via Academic Survey Tasks (2025) · cites this paper as Sarol et al. (2024)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Research reports often include a substantial amount of citation-related content, and evaluating the precision and standardization of these citations plays a crucial role in assessing the overall quality of the report Sarol et al. (2024).”
From ReportBench: Evaluating Deep Research Agents via Academic Survey Tasks (2025) · cites this paper as Sarol et al. (2024)
Experimental — accuracy not yet measured
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · cited 1×
“Prompting the model to reason about the steps to arrive at a conclusion is shown to further improve performance for complex reasoning tasks (Wei et al. 2022).”
BackgroundThe probabilistic relevance framework: BM25 and beyond
Robertson, S., & Zaragoza, H. (2009) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000019 · Cited by 3,374 · cited 1×
“� BM25 þ MonoT5 reranker : To identify the most semantically relevant sentences, we leverage the simple BM25 retrieval model (Robertson and Zaragoza 2009) to obtain the top 60 sentences and the MonoT5 reranker (Nogueira et al. 2020) to rerank these sentences.”
MethodDomain-specific language model pretraining for biomedical natural language processing
Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H. (2022) · ACM Transactions on Computing for Healthcare · doi:10.1145/3458754 · Cited by 2,170 · cited 4×
“For this classification task, we fine-tuned the PubMedBERT model (Gu et al. 2021).”
MethodInfluentialWhat do citation counts measure? A review of studies on citing behavior
Bornmann, L., & Daniel, H. (2008) · Journal of Documentation · doi:10.1108/00220410810844150 · Cited by 1,607 · cited 2×
“While most work focuses on citation counts (Bornmann and Daniel 2008), there is increasing focus on describing and classifying relationships between the citing and reference papers (ie, citation content analysis ) (Zhang et al. 2013, Tahamtan and Bornmann 2019), which aims to provide a more…”
BackgroundA review of the literature on citation impact indicators
Waltman, L. (2016) · Journal of Informetrics · doi:10.1016/j.joi.2016.02.007 · Cited by 1,356 · cited 1×
“Citation-based metrics, such as h-index and journal impact factor, are the primary mechanisms for measuring the scholarly contributions of researchers and journals (Waltman 2016).”
BackgroundAddiction Rare in Patients Treated with Narcotics
(1980) · New England Journal of Medicine · doi:10.1056/nejm198001103020221 · Cited by 1,106 · cited 1×
“A recent bibliometric analysis suggested that inaccurate citations to a letter published in the New England Journal of Medicine in 1980 (Porter and Jick 1980) may have contributed to the opioid epidemic in the United States (Leung et al. 2017).”
BackgroundHow citation distortions create unfounded authority: analysis of a citation network
Greenberg, S. A. (2009) · BMJ · doi:10.1136/bmj.b2680 · Cited by 499 · cited 3×
“Greenberg (2009) has demonstrated how speculative claims have become accepted facts in Alzheimer’s disease research through citation distortions.”
BackgroundInfluentialTahamtan, I., & Bornmann, L. (2019) · Scientometrics · doi:10.1007/s11192-019-03243-4 · Cited by 236 · cited 2×
“…focus on describing and classifying relationships between the citing and reference papers (ie, citation content analysis ) (Zhang et al. 2013, Tahamtan and Bornmann 2019), which aims to provide a more fine-grained, qualitative perspective on how an article is cited, including citation…”
BackgroundMeasuring academic influence: Not all citations are equal
Zhu, X., Turney, P., Lemire, D., & Vellino, A. (2015) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23179 · Cited by 216 · cited 2×
“…qualitative perspective on how an article is cited, including citation function (Spiegel-Rosing 1977, Teufel et al. 2009, Agarwal et al. 2010), citation sentiment (or polarity) (Athar 2014, Xu et al. 2015, Kilicoglu et al. 2019), and citation influence (Valenzuela et al. 2015, Zhu et al. 2015).”
BackgroundAuthorship and citation manipulation in academic research
Fong, E. A., & Wilhite, A. W. (2017) · PLOS ONE · doi:10.1371/journal.pone.0187394 · Cited by 205 · cited 1×
“This often occurs when the citation is part of a multi-citation (e.g. [2–5] ), and may be a sign of “citation padding” (Fong and Wilhite 2017).”
BackgroundExplainable Automated Fact-Checking for Public Health Claims
Kotonya, N., & Toni, F. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.623 · Cited by 195
A 1980 Letter on the Risk of Opioid Addiction
Leung, P. T. M., Macdonald, E. M., Stanbrook, M. B., Dhalla, I. A., & Juurlink, D. N. (2017) · New England Journal of Medicine · doi:10.1056/nejmc1700150 · Cited by 193 · cited 1×
“A recent bibliometric analysis suggested that inaccurate citations to a letter published in the New England Journal of Medicine in 1980 (Porter and Jick 1980) may have contributed to the opioid epidemic in the United States (Leung et al. 2017).”
BackgroundZhang, G., Ding, Y., & Milojević, S. (2013) · Journal of the American Society for Information Science and Technology · doi:10.1002/asi.22850 · Cited by 169 · cited 2×
“…there is increasing focus on describing and classifying relationships between the citing and reference papers (ie, citation content analysis ) (Zhang et al. 2013, Tahamtan and Bornmann 2019), which aims to provide a more fine-grained, qualitative perspective on how an article is cited,…”
BackgroundHow accurate are quotations and references in medical journals?
de Lacey, G., Record, C., & Wade, J. (1985) · BMJ · doi:10.1136/bmj.291.6499.884 · Cited by 160 · cited 1×
“…errors are often categorized as major and minor; the former referring to errors that seriously misrepresent the claims of the reference paper and the latter referring to inconsistencies or factual errors that are not sufficiently serious to alter the meaning of the source (De Lacey et al. 1985).”
BackgroundCitation Analysis and Discourse Analysis
SWALES, J. (1986) · Applied Linguistics · doi:10.1093/applin/7.1.39 · Cited by 137 · cited 1×
“There is a long tradition of citation analysis studies, including qualitative, quantitative, and computational approaches (Smith 1981, Swales 1986, Bornmann and Daniel 2008, Zhang et al. 2013, Tahamtan and Bornmann 2019).”
BackgroundScience Studies: Bibliometric and Content Analysis
Spiegel-Rosing, I. (1977) · Social Studies of Science · doi:10.1177/030631277700700111 · Cited by 120 · cited 1×
“…and Bornmann 2019), which aims to provide a more fine-grained, qualitative perspective on how an article is cited, including citation function (Spiegel-Rosing 1977, Teufel et al. 2009, Agarwal et al. 2010), citation sentiment (or polarity) (Athar 2014, Xu et al. 2015, Kilicoglu et al. 2019),…”
BackgroundIqbal, S., Hassan, S., Aljohani, N. R., Alelyani, S., Nawaz, R., & Bornmann, L. (2021) · Scientometrics · doi:10.1007/s11192-021-04055-1 · Cited by 102 · cited 1×
“We refer to reader to Iqbal et al. (2021) for a relatively recent survey of citation analysis research based on NLP and machine learning.”
MethodHsiao, T.-K., & Schneider, J. (2021) · Quantitative Science Studies · doi:10.1162/qss_a_00155 · Cited by 98 · cited 1×
“More broadly, retracted articles continue to be cited (Van Der Vet and Nijveen 2016, Suelzer et al. 2019, Schneider et al. 2020, Hsiao and Schneider 2022), potentially leading to propagation of untrustworthy information.”
BackgroundSelf-citationQuotation accuracy in medical journal articles—a systematic review and meta-analysis
Jergas, H., & Baethge, C. (2015) · PeerJ · doi:10.7717/peerj.1364 · Cited by 88 · cited 5×
“A more recent meta-analysis showed that 25.4% of medical articles contained a quotation error, roughly half of them severe errors that distorted the claims from the cited articles (Jergas and Baethge 2015).”
BackgroundMethodInfluentialSchneider, J., Ye, D., Hill, A. M., & Whitehorn, A. S. (2020) · Scientometrics · doi:10.1007/s11192-020-03631-1 · Cited by 88 · cited 1×
“More broadly, retracted articles continue to be cited (Van Der Vet and Nijveen 2016, Suelzer et al. 2019, Schneider et al. 2020, Hsiao and Schneider 2022), potentially leading to propagation of untrustworthy information.”
BackgroundSelf-citationSuelzer, E. M., Deal, J., Hanus, K. L., Ruggeri, B., Sieracki, R., & Witkowski, E. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.15552 · Cited by 69 · cited 1×
“More broadly, retracted articles continue to be cited (Van Der Vet and Nijveen 2016, Suelzer et al. 2019, Schneider et al. 2020, Hsiao and Schneider 2022), potentially leading to propagation of untrustworthy information.”
BackgroundTechnical editing of research reports in biomedical journals
Wager, E., & Middleton, P. (2008) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000002.pub3 · Cited by 68
Biomedical text mining for research rigor and integrity: tasks, challenges, directions
Kilicoglu, H. (2017) · Briefings in Bioinformatics · doi:10.1093/bib/bbx057 · Cited by 61
Reference accuracy in peer-reviewed pediatric orthopaedic literature
Davids, J. R., Weigl, D. M., Edmonds, J. P., & Blackhurst, D. W. (2010) · The Journal of Bone and Joint Surgery-American Volume · doi:10.2106/jbjs.i.00063 · Cited by 55 · cited 2×
“Studies of quotation errors in various medical domains are mostly limited to small datasets in specialized domains that are not publicly available (Davids et al. 2010, Awrey et al. 2011, Jergas and Baethge 2015, Pavlovic et al. 2021).”
BackgroundAutomatically classifying the role of citations in biomedical articles.
Agarwal, S., Choubey, L., & Yu, H. Q. (2010) · PubMed · Cited by 51
van der Vet, P. E., & Nijveen, H. (2016) · Research Integrity and Peer Review · doi:10.1186/s41073-016-0008-5 · Cited by 50 · cited 1×
“More broadly, retracted articles continue to be cited (Van Der Vet and Nijveen 2016, Suelzer et al. 2019, Schneider et al. 2020, Hsiao and Schneider 2022), potentially leading to propagation of untrustworthy information.”
BackgroundCitation Sentiment Analysis in Clinical Trial Papers.
Xu, J., Zhang, Y., Wu, Y., Wang, J., Dong, X., & Xu, H. (2015) · PubMed · Cited by 49
How accurate are citations of frequently cited papers in biomedical literature?
Pavlovic, V., Weissgerber, T., Stanisavljevic, D., Pekmezovic, T., Milicevic, O., Lazovic, J. M., Cirkovic, A., Savic, M., Rajovic, N., Piperac, P., et al. (2021) · Clinical Science · doi:10.1042/cs20201573 · Cited by 46
Accuracy of citation and quotation in foot and ankle surgery journals
Luo, M., Li, C. C., Molina, D., Andersen, C. R., & Panchbhavi, V. K. (2013) · Foot & Ankle International · doi:10.1177/1071100713475354 · Cited by 41 · cited 1×
“Our citation accuracy classification is based on and extends prior quotation error classifications (Luo et al. 2013, Jergas and Baethge 2015).”
MethodReference accuracy in the general surgery literature
Awrey, J., Inaba, K., Barmparas, G., Recinos, G., Teixeira, P. G. R., Chan, L. S., Talving, P., & Demetriades, D. (2011) · World Journal of Surgery · doi:10.1007/s00268-010-0912-7 · Cited by 33 · cited 2×
“Studies of quotation errors in various medical domains are mostly limited to small datasets in specialized domains that are not publicly available (Davids et al. 2010, Awrey et al. 2011, Jergas and Baethge 2015, Pavlovic et al. 2021).”
BackgroundKilicoglu, H., Peng, Z., Tafreshi, S., Tran, T., Rosemblat, G., & Schneider, J. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103123 · Cited by 33 · cited 1×
“…qualitative perspective on how an article is cited, including citation function (Spiegel-Rosing 1977, Teufel et al. 2009, Agarwal et al. 2010), citation sentiment (or polarity) (Athar 2014, Xu et al. 2015, Kilicoglu et al. 2019), and citation influence (Valenzuela et al. 2015, Zhu et al. 2015).”
BackgroundSelf-citationContextualizing Citations for Scientific Summarization using Word Embeddings and Domain Knowledge
Cohan, A., & Goharian, N. (2017) · Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3077136.3080740 · Cited by 23
Literary runaway: Increasingly more references cited per academic research article from 1980 to 2019
Dai, C., Chen, Q., Wan, T., Liu, F., Gong, Y., & Wang, Q. (2021) · PLOS ONE · doi:10.1371/journal.pone.0255849 · Cited by 21 · cited 1×
“Given that an article on average cites about 45 articles (Dai et al. 2021) and the judgment requires expertise, this is a challenging task.”
BackgroundTechnical editing of research reports in biomedical journals
Wager, E., & Middleton, P. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000002.pub2 · Cited by 20 · cited 3×
“Two types of citation errors are often distinguished: citation accuracy and quotation accuracy errors (Wager and Middleton 2008).”
BackgroundMethodOpCitance: Citation contexts identified from the PubMed Central open access articles
Hsiao, T., & Torvik, V. I. (2023) · Scientific Data · doi:10.1038/s41597-023-02134-x · Cited by 16 · cited 3×
“We collected 100 highly-cited research articles available in full text from the PubMed Central Open Access Subset (PMC-OA) to form our reference article set.”
MethodInfluential
Research works printed but not linked: 2
- Athar, 2014, Technical report
- Smith, 1981, Citation analysis, Library Trends, 30, 83
Cited by
ReportBench: Evaluating Deep Research Agents via Academic Survey Tasks
Li, M., Zeng, Y., Cheng, Z., Ma, C., & Jia, K. (2025) · arXiv · doi:10.48550/arxiv.2508.15804 · Cited by 1
Zhang, T. M., & Abernethy, N. F. (2025) · arXiv · doi:10.48550/arxiv.2505.23824 · Cited by 1
Analysis of English Sentence Dependency Syntax Based on Python
Wang, Y. (2024) · International Conference on Power, Electrical Engineering, Electronics and Control (PEEEC) · doi:10.1109/peeec63877.2024.00222
“Word vector is a method of representing natural language as a mathematical vector matrix, which is an important method for natural language processing, modeling, feature extraction, etc. [9], laying the foundation for subsequent analysis.”
Writing Systems That Work: Dissertation Momentum, Coauthoring, and AI-Supported Revision
(2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-9340-7.ch012
Detection and interpretability analysis of quotation errors by large language models
Huang, B., Zhang, Y., Huang, S., & Zhang, C. (2026) · The Electronic Library · doi:10.1108/el-11-2025-0464
“…of quotation errors can be categorized into three types: similarity-based methods(Liu et al ., 2024), text classification methods based on small-scale pre-trained language models (PLMs) (Sarol et al ., 2024), and prompt-based approaches with large language models (LLMs)(Sarol et al ., 2024).”
Metadata
- Authors
- Sarol, M. J., Ming, S., Radhakrishna, S., Schneider, J., & Kilicoglu, H.
- Affiliations
- University of Illinois Urbana-Champaign (US)
- School of Information Sciences, University of Illinois Urbana-Champaign , Champaign, IL 61820, United States
- Venue
- Bioinformatics
- Publisher
- Oxford University Press (OUP)
- Published
- 2024-06-26
- Volume
- 40
- Issue
- 7
- ISSN
- 1367-4803 (ISSN-L)
- 1367-4811
- PMC
- PMC11231046
- OpenAlex
- W4400031865
- Cited by
- 12
- References
- 40
- Citations by year
- 2026: 10 · 2025: 1 · 2024: 1
- FWCI
- 4.72
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Bioinformatics (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- Office of Research Integrity
- US Department of Health and Human Services (ORIIR220073)
- Topics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Social Sciences › Education › Academic integrity and plagiarism
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- citation integrity, citation errors, quotation errors, biomedical literature, citation accuracy, corpus annotation, sentence retrieval, claim verification, large language models, in-context learning, GPT-4, scientific evidence
- MeSH terms
- Humans, Natural Language Processing, Publications, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
