The DOI matches its Crossref record; 23 of the 24 references it cites were found in a registry or an index, and 1 are not checked yet; none of the 3 sources its claims cite is checked for support yet.
Unraveling the Ai2 Asta Scholarly Research Assistant Citation System
Open accessAbstract
Despite the growing integration of Deep Research tools into academic workflows, empirical evidence on the operation, stability, and potential biases of their citation systems remains scarce. This study addresses this gap by evaluating the intensity, consistency, and bibliographic characteristics of references cited in the literature reports generated by Ai2 Asta, with the aim of understanding how its citation system operates and assessing its implications for scholarly communication. To this end, ten domain-specific queries were submitted to Asta’s Summarise Literature feature, and two independent rounds of data collection were conducted. From each report, in-text citations, cited references, as well as other metrics related to the response process were extracted and examined. The results reveal high citation intensity, with reports integrating numerous in-text citations grounded in retrieved evidence and a diverse yet concentrated set of venues. However, notable instability is observed in the composition of cited references across identical queries, alongside a lack of concordance between retrieved documents and those ultimately cited, suggesting additional opaque selection mechanisms during report generation. These findings indicate that, while Ai2 Asta produces well-structured and quality reports, its instability and opacity in the citation process pose challenges in quantitative science studies due to their lack of reproducibility and transparency. Despite the restricted number of queries and disciplinary scope, the results offer valuable insights for researchers, bibliometricians, developers, and research evaluators seeking to understand, use or regulate AI-based scholarly assistants responsibly.
Checked 2026-11-10 · 1 reference not checked yet
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
This study evaluates the citation system of Ai2 Asta’s “Summarise Literature” feature using ten domain-specific queries run in two independent rounds. It finds high citation intensity but substantial instability and opacity: identical queries produce different cited references, and retrieved documents do not reliably match those ultimately cited.
Key findings
- Citation intensity was high: reports cited a mean of 22.2 references in the first collection and 27.3 in the second, with mean in-text citations of 78.3 and 73.7, respectively; reports ranged from 1,500 to 4,000 words, and passages retrieved averaged 249, near the system maximum of 256.
- Citation composition was unstable across identical queries: overlap ranged from 34.2% in Query #6 to 95.5% in Query #10; for Query #1, only 4 of the 8 references from the first collection also appeared among the 9 references in the second.
- Venues were diverse but concentrated: Scientometrics, PLoS One, and arXiv.org together accounted for 29% of cited references in the first collection and 30.8% in the second; 56 of 262 references (21.4%) in the first collection had more than 100 citations, while 28 had none, with comparable figures in the second (57 and 29).
- There was no relationship between the number of relevant publications retrieved during deep search and the number ultimately cited; in 50% of queries, cited references exceeded retrieved relevant publications, suggesting additional opaque selection during report generation.
Methods
- Ten queries on topics in quantitative science studies were submitted to Asta’s “Summarise Literature” function, with two independent data collections on October 29 and November 2 using the same configuration, machine, and location.
- For each query, the authors extracted search duration, number of retrieved documents, field of study, report structure, in-text citations, and cited references in BIB format, including publication name, year, citations received, title, authors, and publication ID.
- Data were analyzed descriptively in a spreadsheet, with Venn diagrams for citation overlap and Scimago Graphica for figures; contextual data came from the Asta Summary Citation Counts parquet file downloaded October 28 and processed with an ad hoc Python script.
Limitations
- The paper states a restricted number of queries and disciplinary scope, and notes source-identification limitations: 23 cited references in the first collection and 19 in the second lacked source information, alongside normalization problems such as “Proceedings of the National Academy of Sciences of the United States of America” versus “Proc.Natl.Acad.Sci.USA.”
- The paper states that future work should expand queries and disciplines, assess report quality and reference reliability, and conduct longitudinal studies of Asta’s behavior.
- Evident from the text: the study examines only one tool, only ten queries, and only two rounds, and it does not analyze the separate “Find Papers” functionality; the assigned field of study also changed between collections in four cases, which may affect retrieval.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
55 Own result · 19 Own interpretation · 3 No basis given · 2 Own method · 1 Sources not found · 1 Check pending · 1 Not checked yet
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The ten queries generated a total of 262 cited references in the first data collection, drawn from 105 unique sources, with a small number of journals accounting for most citations (only 19% of sources were cited more than once).”
§ Unraveling the Ai2 Asta Scholarly Research Assistant Citation System > Additional information
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The total number of cited references differs between the first and second data collections, depending on the query.”
§ Unraveling the Ai2 Asta Scholarly Research Assistant Citation System > 4.2. Citation Stability
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“However, discrepancies were also observed in queries where the field of study remained identical across both data collections.”
§ Unraveling the Ai2 Asta Scholarly Research Assistant Citation System > Additional information
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition to the high proportion of recent publications (which tend to have lower accumulated citation counts), the reports also include a substantial number of highly cited works.”
§ Unraveling the Ai2 Asta Scholarly Research Assistant Citation System > Figure 2
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the mean number of cited references per report in the first data collection is 22.2, although the dispersion is considerable (ranging from 8 citations in Q1 to 42 in Q2).”
§ Unraveling the Ai2 Asta Scholarly Research Assistant Citation System > 4.1. Citation intensity
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Otherwise, abstracts were used less frequently, especially in cases where fewer passages were retrieved and fewer in-text citations appeared in the final reports.”
§ Unraveling the Ai2 Asta Scholarly Research Assistant Citation System > 4.1. Citation intensity
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Instead, it only includes those ultimately selected to generate the report, and these selections vary significantly depending on when the query is performed.”
§ Unraveling the Ai2 Asta Scholarly Research Assistant Citation System > 5. Discussion
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The literature reports generated by Ai2 Asta are representative of its citation system.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although the tool produces wellstructured reports enriched with numerous citation-backed claims-and can therefore serve as a valuable aid for researchers when used critically and appropriately-the underlying set of cited publications varies substantially even when identical queries are issued at different times.”
§ Unraveling the Ai2 Asta Scholarly Research Assistant Citation System > Additional information
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While the appearance of journals such as Scientometrics, JASIST, Journal of Informetrics, or QSS among the most cited sources is logical, given the nature of the questions, the results show an excessive presence of multidisciplinary journals (PLoS One, PNAS, IEEE Access) and the arXiv.orgrepository.”
§ Unraveling the Ai2 Asta Scholarly Research Assistant Citation System > 5. Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Indeed, in 50% of the queries, the number of cited references exceeded the number of relevant publications retrieved during deep search.”
§ Unraveling the Ai2 Asta Scholarly Research Assistant Citation System > 4.1. Citation intensity
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 16
- The quote only partly states this claim. 2
- It may state two separate claims as one. 12
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The reports are lengthy, ranging from 1,500 to 4,000 words (excluding abstracts and section titles), with longer reports generally -though not always -containing a greater number of cited references.”
§ Unraveling the Ai2 Asta Scholarly Research Assistant Citation System > 4.1. Citation intensity
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“with a preponderance of topics in medicine and computer science”
§ Unraveling the Ai2 Asta Scholarly Research Assistant Citation System > Technol." and "Journal of the Association for Information Science and Technology").
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Comparable figures were observed in the second data collection, with 57 references exceeding 100 citations and 29 receiving zero.”
§ Unraveling the Ai2 Asta Scholarly Research Assistant Citation System > Figure 2
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The proportion of recent publications (from 2023 onwards) among the cited references is noteworthy, although it fluctuates across queries and between data collections.”
§ Unraveling the Ai2 Asta Scholarly Research Assistant Citation System > 4.2. Citation Stability
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, 23 cited references in the first data collection and 19 in the second do not provide source information.”
§ Unraveling the Ai2 Asta Scholarly Research Assistant Citation System > 5. Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The reports are lengthy, ranging from 1,500 to 4,000 words (excluding abstracts and section titles), with longer reports generally -though not always -containing a greater number of cited references.”
§ Unraveling the Ai2 Asta Scholarly Research Assistant Citation System > 4.1. Citation intensity
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The ten queries generated a total of 262 cited references in the first data collection, drawn from 105 unique sources, with a small number of journals accounting for most citations (only 19% of sources were cited more than once).”
§ Unraveling the Ai2 Asta Scholarly Research Assistant Citation System > Additional information
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In four cases, the assigned field of study changed slightly between the two data collections (e.g., Query #8 shifted from”
§ Unraveling the Ai2 Asta Scholarly Research Assistant Citation System > Additional information
plainn = 4- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The number of in-text citations (or claims) per report is comparatively high (x̃ = 78.3 in the first data collection; x̃ = 73.7 in the second) in relation to the number of cited references.”
§ Unraveling the Ai2 Asta Scholarly Research Assistant Citation System > 4.1. Citation intensity
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Other frequently cited venues include JASIST (specialised in information science, including but not limited to bibliometrics), PNAS and IEEE Access (both multidisciplinary), Journal of Informetrics and Quantitative Science Studies (bibliometrics-focused), and Publications (specialised in scholarly publishing more broadly).”
§ Unraveling the Ai2 Asta Scholarly Research Assistant Citation System > Additional information
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“with a preponderance of topics in medicine and computer science”
§ Unraveling the Ai2 Asta Scholarly Research Assistant Citation System > Technol." and "Journal of the Association for Information Science and Technology").
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The figures are similar in the second data collection, which resulted in 273 cited references from 111 unique sources, 20.7% of which were cited more than once.”
§ Unraveling the Ai2 Asta Scholarly Research Assistant Citation System > Additional information
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results reveal high citation intensity, with reports integrating numerous in-text citations grounded in retrieved evidence and a diverse yet concentrated set of venues.”
- Also stated in: § 5. Discussion · plain
“Regarding the references cited in the reports (RQ3), the results show a significant concentration of journals.”
§ Unraveling the Ai2 Asta Scholarly Research Assistant Citation System > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 5. Discussion · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In the first data collection, 56 of the 262 cited references (21.4%) had received more than 100 citations, while 28 had received none.”
§ Unraveling the Ai2 Asta Scholarly Research Assistant Citation System > Figure 2
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A large number of passages was retrieved in both data collections (x̃ = 249), very close to the system maximum (256).”
§ Unraveling the Ai2 Asta Scholarly Research Assistant Citation System > 4.1. Citation intensity
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Some of these cited references are repeated quite frequently in the text, resulting in a large number of intext citations.”
§ Unraveling the Ai2 Asta Scholarly Research Assistant Citation System > 5. Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The ten queries generated a total of 262 cited references in the first data collection, drawn from 105 unique sources, with a small number of journals accounting for most citations (only 19% of sources were cited more than once).”
§ Unraveling the Ai2 Asta Scholarly Research Assistant Citation System > Additional information
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This procedure creates uncertainty within the system, since the same query can yield different results, both in terms of content and cited references.”
§ Unraveling the Ai2 Asta Scholarly Research Assistant Citation System > 5. Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The literature reports generated by Ai2 Asta are representative of its citation system.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, Query #1 has a time constraint (the year 2025) that could explain the low number of cited references found, making the clarity, precision, and conciseness of the prompt essential for evaluating the results.”
§ Unraveling the Ai2 Asta Scholarly Research Assistant Citation System > 5. Discussion
hedgedconditionalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Such changes may alter the deep search process and lead to the retrieval of different references.”
§ Unraveling the Ai2 Asta Scholarly Research Assistant Citation System > Additional information
hedgedconditionaln = 4- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“highlighting the risk that unstable citation patterns may inadvertently shape researchers' citation behaviour”
§ Unraveling the Ai2 Asta Scholarly Research Assistant Citation System > Table 5
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Otherwise, prompts with fewer cited references could be a consequence of the question's formulation rather than simply due to less coverage in Semantic Scholar.”
§ Unraveling the Ai2 Asta Scholarly Research Assistant Citation System > 5. Discussion
hedgedconditionalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To this end, ten domain-specific queries were submitted to Asta's Summarise Literature feature, and two independent rounds of data collection were conducted.”
§ Unraveling the Ai2 Asta Scholarly Research Assistant Citation System > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In any case, the purpose of this functionality is to provide the user with a generic report that answers the question posed in a well-supported manner, rather than offering a systematic and exhaustive review of the topic.”
§ Unraveling the Ai2 Asta Scholarly Research Assistant Citation System > 5. Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Regarding the citation intensity (RQ1), the reports tend to include approximately between 20 and 40 cited references, which is considered a high number given the length of the reports (around 2.3k-2.4kwords on average).”
§ Unraveling the Ai2 Asta Scholarly Research Assistant Citation System > 5. Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“which”
§ Unraveling the Ai2 Asta Scholarly Research Assistant Citation System > Technol." and "Journal of the Association for Information Science and Technology").
What the sources say
- Not checked yetcites Kinney et al., 2023 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These findings indicate that, while Ai2 Asta produces well-structured and quality reports, its instability and opacity in the citation process, pose challenges in quantitative science studies due to their lack of reproducibility and transparency.”
§ Unraveling the Ai2 Asta Scholarly Research Assistant Citation System > Abstract
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To address RQ3, two data collections were conducted.The first was performed on October 29th and the second on November 2nd, using the same configuration, machine and location in order to minimize external variables due to technical issues.”
§ Unraveling the Ai2 Asta Scholarly Research Assistant Citation System > 3.1. Data collection
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
The paper relies on this without support.
Empirical evidence on the operation, stability, and potential biases of citation systems of Deep Research tools remains scarce.
“Despite the growing integration of Deep Research tools into academic workflows, empirical evidence on the operation, stability, and potential biases of their citation systems remains scarce.”
The paper relies on this without support.
The ten queries are representative of topics in quantitative science studies and suitable for evaluating Ai2 Asta's citation system.
“The questions relate to topics of broad debate within the field of quantitative science studies and, in some cases, have generated controversy and opposing viewpoints, which may facilitate the creation of comparative tables and interpretive reports.”
The paper relies on this without support.
This is the first study to date in which the citation system of a deep research tool is analysed from an informetric point of view.
“constituting the first study to date in which the citation system of a deep research tool is analysed from an informetric point of view.”
The paper relies on this without support.
The literature reports generated by Ai2 Asta are representative of its citation system.
“This study addresses this gap by evaluating the intensity, consistency, and bibliographic characteristics of references cited in the literature reports generated by Ai2 Asta, with the aim of understanding how its citation system operates and assessing its implications for scholarly communication.”
The paper relies on this without support.
Ai2 Asta performs its deep search process through Semantic Scholar.
“Ai2 Asta performs the deep search process through Semantic Scholar.”
The paper relies on this without support.
The ten selected queries are representative of Ai2 Asta's citation system behavior more generally.
“The findings of this study indicate that Ai2 Asta's citation system displays a distinctive combination of high citation intensity, moderate bibliographic diversity, and considerable instability across repeated queries.”
The paper relies on this without support.
The two data collections were conducted under comparable conditions so that differences reflect system instability rather than external factors.
“The figures are similar in the second data collection, which resulted in 273 cited references from 111 unique sources, 20.7% of which were cited more than once.”
The paper relies on this without support.
The documents retrieved during deep search are a valid baseline for what the system should cite.
“Moreover, the disconnect between the documents retrieved during deep search and those ultimately cited suggests the presence of additional, opaque selection mechanisms during report generation.”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Unraveling the Ai2 Asta Scholarly Research Assistant Citation System > 4.1. Citation intensity
“ranging from 8 citations in Q1 to 42 in Q2”
§ Unraveling the Ai2 Asta Scholarly Research Assistant Citation System > Table 5
“| 2 | Sociology, Education | 3m 8s | 248 | 5 | 30 | 40 | 94 | 1931 | 5 | 1 | 1 |”
Citations
Cites
Conceptualizing AI literacy: An exploratory review
Ng, D. T. K., Leung, J. K. L., Chu, S. K. W., & Qiao, M. S. (2021) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2021.100041 · Cited by 2,205 · cited 2×
“This issue is even enhanced due to the potential user's lack of literacy in the use of AI tools (Ng et al.,2021).”
Retrieval-Augmented Generation for Large Language Models: A Survey
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., & Wang, H. (2023) · arXiv · doi:10.48550/arxiv.2312.10997 · Cited by 876
Fabrication and errors in the bibliographic citations generated by ChatGPT
Walters, W. H., & Wilder, E. I. (2023) · Scientific Reports · doi:10.1038/s41598-023-41032-5 · Cited by 499
AI and the transformation of social science research
Grossmann, I., Feinberg, M., Parker, D. C., Christakis, N. A., Tetlock, P. E., & Cunningham, W. A. (2023) · Science · doi:10.1126/science.adi1778 · Cited by 377 · cited 2×
“LLMs are being used in tasks such as writing support (Chen, 2023), literature synthesis (Zheng et al., 2023; Scherbakov, 2025; Silva and Wickramaarachchi, 2025), data analysis (Nejjar et al., 2025), assistance in designing research techniques such as interviews or surveys (Jansen et al., 2023), and…”
ChatGPT and other artificial intelligence applications speed up scientific writing
Chen, T.-J. (2023) · Journal of the Chinese Medical Association · doi:10.1097/jcma.0000000000000900 · Cited by 240
Large language models for scientific discovery in molecular property prediction
Zheng, Y., Koh, H. Y., Ju, J., Nguyen, A. T. N., May, L. T., Webb, G. I., & Pan, S. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-00994-z · Cited by 180
SCImago Graphica: a new tool for exploring and visually communicating data
Hassan‐Montero, Y., Anegón, F. D. M., & Guerrero‐Bote, V. P. (2022) · El Profesional de la Informacion · doi:10.3145/epi.2022.sep.02 · Cited by 163
Artificial intelligence to support publishing and peer review: A summary and review
Kousha, K., & Thelwall, M. (2023) · Learned Publishing · doi:10.1002/leap.1570 · Cited by 162
Scherbakov, D., Hubig, N., Jansari, V., Bakumenko, A., & Lenert, L. A. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf063 · Cited by 154
Employing large language models in survey research
Jansen, B. J. J., Jung, S., & Salminen, J. O. (2023) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100020 · Cited by 151
LLMs for science: Usage for code generation and data analysis
Nejjar, M., Zacharias, L., Stiehle, F., & Weber, I. M. (2024) · Journal of Software Evolution and Process · doi:10.1002/smr.2723 · Cited by 126
A Comprehensive Survey of Deep Research: Systems, Methodologies, and ApplicationsPreprint
Xu, R., & Peng, J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.12594 · Cited by 115
How should the advancement of large language models affect the practice of science?
Binz, M., Alaniz, S., Roskies, A., Aczel, B., Bergstrom, C. T., Allen, C., Schad, D., Wulff, D., West, J. D., Zhang, Q., et al. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2401227121 · Cited by 74
The Semantic Scholar Open Data Platform
Kinney, R., Anastasiades, C., Authur, R., Beltagy, I., Bragg, J., Buraczynski, A., Cachola, I., Candra, S., Chandrasekhar, Y., Cohan, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.10140 · Cited by 64
Large Language Models for Scientific Synthesis, Inference and ExplanationPreprint
Zheng, Y., Koh, H. Y., Ju, J.-X., Nguyen, A., May, L. T., Webb, G. I., & Pan, S.-R. (2023) · arXiv.org · doi:10.48550/arxiv.2310.07984 · Cited by 64
ResearcherBench: Evaluating Deep AI Research Systems on the Frontiers of Scientific InquiryPreprint
Xu, T., Lu, P., Ye, L., Hu, X., & Liu, P. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.16280 · Cited by 44
Vector Search with OpenAI Embeddings: Lucene Is All You Need
Xian, J., Teofili, T., Pradeep, R., & Lin, J. (2024) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3616855.3635691 · Cited by 35
The Problems of LLM-generated Data in Social Science Research
Rossi, L., Harrison, K., & Shklovski, I. (2024) · Sociologica · doi:10.6092/issn.1971-8853/19576 · Cited by 33
ChatGPT and the potential growing of ghost bibliographic references
Orduña-Malea, E., & Cabezas-Clavijo, Á. (2023) · Scientometrics · doi:10.1007/s11192-023-04804-4 · Cited by 31
Self-citationAi2 Scholar QA: Organized Literature Synthesis with Attribution
Singh, A., Chang, J. C., Haddad, D., Naik, A., Hwang, J. D., Kinney, R., Weld, D. S., Downey, D., & Feldman, S. (2025) · Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations) · doi:10.18653/v1/2025.acl-demo.49 · Cited by 19
Large language models in peer review: challenges and opportunities
Sun, Z. (2025) · Scientometrics · doi:10.1007/s11192-025-05440-w · Cited by 15
Silva, N., & Wickramaarachchi, D. (2025) · International Conference on Advanced Research in Computing (ICARC) · doi:10.1109/icarc64760.2025.10963273 · Cited by 15
Deep Research Agents: A Systematic Examination And Roadmap
Huang, Y., Chen, Y., Zhang, H., Li, K., Zhou, H., Fang, M., Yang, L., Li, X., Shang, L., Xu, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.18096 · Cited by 3
Research works printed but not linked: 2
- Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W-T., Rocktäschel, T., Riedel, T., & Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. In H. Larochelle and M. Ranzato and R. Hadsell and M.F. Balcan and H. Lin (Eds.). Advances in neural information processing systems (pp. 9459-9474). Curran Associates. https://proceedings.neurips.cc/paper_files/paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf
- Rane, N.L., Tawde, A., Choudhary, S.P., & Rane, J. (2023). Contribution and performance of ChatGPT and other Large Language Models (LLM) for scientific and research advancements: a double-edged sword. International Research Journal of Modernization in Engineering Technology and Science, 5(10), 875-899. https://doi.org/10.56726/IRJMETS45312; https://goo.su/gvScINu · DOI
Cited by
Still being checked.
Metadata
- Authors
- Orduña-Malea, E., & Lopezosa, C.
- Affiliations
- Universitat de Barcelona (ES)
- Venue
- Revista Panamericana de Comunicación
- Publisher
- Universidad Panamericana
- Published
- 2025-12-24
- Volume
- 7
- Issue
- 2
- ISSN
- 2683-2208 (ISSN-L)
- OpenAlex
- W4417078087
- Semantic Scholar
- f6019894431e306d1528be383598f493eb295a24
- Cited by
- 0
- References
- 23
- FWCI
- 0.00
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Revista Panamericana de Comunicación (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- RiuNet (Universitat Politècnica de València) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- European Regional Development Fund (MCIN/AEI/10)
- Agencia Estatal de Investigación (13039, MCIN / AEI/10, 10.13039, AEI/10)
- Topics
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Social Sciences › Library and Information Sciences › Library Science and Information
- Physical Sciences › Computer Science › Information Systems › Web visibility and informetrics
- Keywords
- AI research assistant, citation systems, reproducibility, transparency, in-text citations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
