Reflexion: Language Agents with Verbal Reinforcement Learning
Open accessThe DOI matches its DataCite record; 42 of the 42 references it cites were found in a registry or an index; 7 of the 8 sources its claims cite were checked for support.
Abstract
Large language models (LLMs) have been increasingly used to interact with external environments (e.g., games, compilers, APIs) as goal-driven agents. However, it remains challenging for these language agents to quickly and efficiently learn from trial-and-error as traditional reinforcement learning methods require extensive training samples and expensive model fine-tuning. We propose Reflexion, a novel framework to reinforce language agents not by updating weights, but instead through linguistic feedback. Concretely, Reflexion agents verbally reflect on task feedback signals, then maintain their own reflective text in an episodic memory buffer to induce better decision-making in subsequent trials. Reflexion is flexible enough to incorporate various types (scalar values or free-form language) and sources (external or internally simulated) of feedback signals, and obtains significant improvements over a baseline agent across diverse tasks (sequential decision-making, coding, language reasoning). For example, Reflexion achieves a 91% pass@1 accuracy on the HumanEval coding benchmark, surpassing the previous state-of-the-art GPT-4 that achieves 80%. We also conduct ablation and analysis studies using different feedback signals, feedback incorporation methods, and agent types, and provide insights into how they affect performance.
Overview
CoChat Summary
TL;DR
Reflexion reinforces language agents through verbal self-reflection stored in episodic memory rather than weight updates, improving performance across decision-making, reasoning, and coding tasks. It reports gains including 91% pass@1 on HumanEval, a 22% absolute improvement on AlfWorld, and 20% on HotPotQA.
Key findings
- On HumanEval Python, Reflexion achieves 91.0 pass@1, surpassing the previous state-of-the-art GPT-4 at 80.1; on HumanEval Rust it reaches 68.0 vs. 60.0, MBPP Rust 75.4 vs. 70.9, and Leetcode Hard 15.0 vs. 7.5.
- In AlfWorld, ReAct + Reflexion completes 130 out of 134 tasks using a simple heuristic and improves over strong baselines by an absolute 22% over 12 iterative learning steps; ReAct-only converges at a 22% hallucination rate with no long-term recovery.
- In HotPotQA, Reflexion improves over baselines by 20%; self-reflection gives an 8% absolute boost over episodic memory alone, and although CoT (GT) fails on 39% of questions, Reflexion improves accuracy by 14% without access to ground-truth answers.
- Ablations on the 50 hardest HumanEval Rust problems show full Reflexion at 0.68 vs. baseline 0.60; omitting test generation drops to 0.52, and omitting self-reflection stays at 0.60.
Methods
- Reflexion uses a modular loop with an Actor that generates text/actions, an Evaluator that scores outputs, and a Self-Reflection model that converts feedback into verbal summaries stored in memory; trials repeat until the Evaluator passes or a max trial count is reached, with memory usually bounded to 1–3 experiences.
- Experiments cover AlfWorld (134 environments, six tasks, ReAct + GPT-3), HotPotQA (100 questions, CoT/ReAct, exact-match grading, memory size 3), and programming (HumanEval, MBPP, LeetcodeHardGym; Python/Rust via MultiPL-E; self-generated unit tests up to 6).
- Ablations and analyses vary feedback signals, feedback incorporation methods, and agent types; compromised approaches are tested on HumanEval Rust, and a WebShop experiment is run across 100 environments.
Limitations
- The paper states Reflexion may fall into non-optimal local minima, limits long-term memory to a sliding window with maximum capacity, and notes code-generation test-driven development is limited by non-deterministic, impure, hardware-dependent, or concurrent functions; it also relies on LLM self-evaluation/heuristics and lacks a formal success guarantee.
- The paper reports WebShop failure: after only four trials the agent shows no improvement and does not generate helpful self-reflections, concluding Reflexion cannot solve tasks requiring significant diversity and exploration.
- Evident from the text: self-correction is an emergent quality of stronger, larger models—starchat-beta shows no improvement (baseline 0.26 vs. Reflexion 0.26)—and flaky or incorrect test suites can cause false positives/negatives, with false negatives preferred.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
4 Valid · 2 Not valid · 0 Not checked · 59 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Therefore, in the case in which the model generates a flaky test suite, it is possible that all tests pass on an incorrect solution and lead to a false positive label on a code completion [11].”
§ Reflexion: Language Agents with Verbal Reinforcement Learning > Analysis
What the sources say
- Partly supportedcites [11] 11. Lam, W., Winter, S., Wei, A., Xie, T., Marinov, D., and Bell, J. (2020). A large-scale longitudinal study of flaky tests. Proc. ACM Program. Lang., 4(OOPSLA). sourceExperimental — accuracy not yet measured
The Introduction (page not provided) says a flaky test can pass and fail on the same code and that changes may be merged when all tests pass. It does not address model-generated test suites, incorrect code completions, or false-positive labels, so those parts of the claim are not established by this paper. (LLM, conf=0.82)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“3 shows an inferior 52% vs 60% (baseline) accuracy, which suggests that the agent is unable to determine if the current implementation is correct without unit tests.”
§ Reflexion: Language Agents with Verbal Reinforcement Learning > Ablation study
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ReAct + Reflexion significantly outperforms ReAct by completing 130 out of 134 tasks using the simple heuristic to detect hallucinations and inefficient planning.”
§ Reflexion: Language Agents with Verbal Reinforcement Learning > Results
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“4 shows that self-reflection improves learning by an 8% absolute boost over the episodic memory learning advantage.”
§ Reflexion: Language Agents with Verbal Reinforcement Learning > Analysis
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“on reasoning questions in HotPotQA [28] by 20%”
- Also stated in: § 4 Experiments · plain
“Most notably, Reflexion improves performance over strong baselines by 22% in AlfWorld, 20% in HotPotQA, and 11% on HumanEval.”
§ Reflexion: Language Agents with Verbal Reinforcement Learning > 1 Introduction
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 4 Experiments · plain
30 more claims
Create a free account to see all 65 claims and every source verdict.
- It may state two separate claims as one. 18
- Probably not a substantial claim. 11
- The quote only partly states this claim. 3
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Memory and reflection systems such as MemAgent (Yu et al., 2025a), Reflexion (Shinn et al., 2023), and Generative Agents (Park et al., 2023) improve retrieval and episodic memory, but do not perform context shaping, state restructuring, or nextinstruction refinement.”
From PAACE: A Plan-Aware Automated Agent Context Engineering Framework (2025) · cites this paper as Shinn et al., 2023
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“However, Reflexion computes verbal reward over a full trajectory, yielding a single end-of-episode signal that conflates credit across reasoning steps: a failure due to a single intermediate error receives the same critique pressure as one due to a flawed strategy.”
From Process Supervision via Verbal Critique Improves Reasoning in Large Language Models (2026) · cites this paper as Shinn et al. [2023]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“Unlike Reflexion, which only reflects on unsuccessful trajectories, our approach separately reflects on both successful and unsuccessful ones.”
From An Efficient Context-Dependent Memory Framework for LLM-Centric Agents (2025) · cites this paper as Shinn et al., 2023
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“The dominant approach is to curate memory at write time.”
From Just-in-Time Memory: Learning to Curate Task-Adaptive Memory for LLM Agents (2026) · cites this paper as Shinn et al., 2023
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Interaction and code generation are increasingly used together, with code as the modality of choice for actions [48, 59], tool construction [13, 51, 69], and reasoning [39, 66, 67].”
From SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering (2024) · cites this paper as [39]
Experimental — accuracy not yet measured
107 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Reflexion: Language Agents with Verbal Reinforcement Learning > 3 Reflexion: reinforcement via verbal reflection
“while M e not pass or t < max trials do”
§ Reflexion: Language Agents with Verbal Reinforcement Learning > Self-reflection
“The Actor, Evaluator, and Self-Reflection models work together through trials in a loop until the Evaluator deems τ t to be correct.”
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Cited by
Items 401–500 of 3,537
Self-Evaluation Guided Beam Search for Reasoning
Xie, Y., Kawaguchi, K., Zhao, Y., Zhao, J. X., Kan, M.-Y., He, J., & Xie, M. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1802 · Cited by 27
Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
Snell, C., Lee, J., Xu, K., & Kumar, A. (2024) · arXiv · doi:10.48550/arxiv.2408.03314 · Cited by 27
Long, R., Sebo, J., Butlin, P., Finlinson, K., Fish, K., Harding, J., Pfau, J., Sims, T., Birch, J., & Chalmers, D. (2024) · arXiv · doi:10.48550/arxiv.2411.00986 · Cited by 27
PPTC Benchmark: Evaluating Large Language Models for PowerPoint Task Completion
Guo, Y., Zhang, Z., Liang, Y., Zhao, D., & Duan, N. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.514 · Cited by 38
Kim, J., Lee, S., Jeon, H., Lee, K., Bae, H., Kim, B., & Seo, J. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3456215 · Cited by 36
“To reduce hallucination and enable inspection, we adopted a self-reflection approach [46].”
MethodWhy Do LLM Agents Fail in Exploring New Environments? A World-Modeling Perspective
Chen, S., Zhu, T., Wang, Z., Zhang, J., Wang, K., Zhou, R., Gao, S., Xiao, T., Teh, Y. W., He, J., et al. (2026) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/2026.findings-emnlp.1023 · Cited by 31
On Verifiable Legal Reasoning: A Multi-Agent Framework with Formalized Knowledge Representations
Sadowski, A., & Chudziak, J. A. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761057 · Cited by 27
“Alternative approaches like Reflexion [43] or Multi-agent Debate [44], while potentially improving reasoning quality, would similarly confound general reasoning enhancements with our specific hypothesis about ontological structuring.”
Ashrafi, N., Bouktif, S., & Mediani, M. (2025) · IEEE International Conference on Application of Information and Communication Technologies (AICT) · doi:10.1109/aict67988.2025.11268754 · Cited by 32
Joint Prompt Optimization of Stacked LLMs using Variational Inference
Sordoni, A., Yuan, E., Côté, M.-A., Pereira, M., Trischler, A., Xiao, Z., Hosseini, A., Niedtner, F., & Roux, N. L. (2023) · neural information processing systems · doi:10.52202/075280-2534 · Cited by 64
Yue, L., Somasekharan, N., Zhang, T., Cao, Y., & Pan, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.18178 · Cited by 37
Pan, Q., Ji, W., Ding, Y., Li, J., Chen, S., Wang, J., Zhou, J., Chen, Q., Zhang, M., Wu, Y., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.02665 · Cited by 31
“…Llasa [141], HuatuoGPT-o1 [10] Self-Refine Strategy E.g., Self-Verification [114], ToT [138], AutoMathCritique [119], Gero et al. [25], Reflexion [88] (§5) Policy Network (§5.1) Training Data Acquisition E.g., T1 [35], SCoRe [45], LLaVA-O1 [129], ReST [29], SPIN [14], DeepseekMath [84], MiniLLM…”
InfluentialMeasuring General Intelligence with Generated GamesPreprint
Verma, V. K., Huang, D., Chen, W., Klein, D., & Tomlin, N. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.07215 · Cited by 30
“Tool-use agents (e.g. ReAct, Reflexion) further integrate search, calculators, or external APIs into the reasoning loop, enabling models to plan, act, and reflect iteratively [44, 36].”
Dai, D. W., Hua, Z., & Chen, G. (2025) · Annual Review of Applied Linguistics · doi:10.1017/s0267190525000054 · Cited by 35
Exchange-of-Thought: Enhancing Large Language Model Capabilities through Cross-Model Communication
Yin, Z., Sun, Q., Chang, C., Guo, Q., Dai, J., Huang, X., & Qiu, X. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.936 · Cited by 26
Bellini-Leite, S. C. (2023) · Adaptive Behavior · doi:10.1177/10597123231206604 · Cited by 34
“Following this perception, an increasing trend (Hsieh et al., 2023; Shinn et al., 2023; Si et al., 2022; Wang et al., 2022; Wei et al., 2022; Yao et al., 2023) has been emerging in AI to design prompting strategies to be coupled onto LLMs to organize prompting or re fl ect on responses, without the…”
Emergent language: a survey and taxonomy
Peters, J., de Puiseau, C. W., Tercan, H., Gopikrishnan, A., de Carvalho, G. A. L., Bitter, C., & Meisen, T. (2025) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-025-09691-y · Cited by 38
Self-Contrast: Better Reflection Through Inconsistent Solving Perspectives
Zhang, W., Shen, Y., Wu, L., Peng, Q., Wang, J., Zhuang, Y., & Lu, W. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.197 · Cited by 27
Multi-Agent Collaboration Mechanisms: A Survey of LLMs
Tran, K.-T., Dao, D., Nguyen, M.-D., Pham, Q.-V., O'Sullivan, B., & Nguyen, H. D. (2025) · arXiv · doi:10.48550/arxiv.2501.06322 · Cited by 680
“For instance, in [117], a feedback loop is carried out as the main collaboration channel, where the task is first handled by an LLM model (Actor), then an Evaluator and Self-Reflection model rates the output and results, producing verbal guidance for the Actor to improve.”
AgentBoard: An analytical evaluation board of multi-turn LLM agents
Ma, C., Zhang, J., Zhu, Z., Yang, C., Yang, Y., Jin, Y., Lan, Z., Kong, L., & He, J. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-2365 · Cited by 26
Survey on Evaluation of LLM-based Agents
Yehudai, A., Eden, L., Li, A., Uziel, G., Zhao, Y., Bar-Haim, R., Cohan, A., & Shmueli-Scheuer, M. (2025) · arXiv · doi:10.48550/arxiv.2503.16416 · Cited by 26
“For example, Reflexion (Shinn et al., 2023) tracks success rate on tasks like HotPotQA (Yang et al., 2018) and ALFWorld (Shridhar et al., 2021), while RAISE (Liu et al., 2024a) enhances the ReAct framework with a two-part memory system evaluated through human judgment on quality metrics and efficiency.”
InfluentialRe2LLM: Reflective Reinforcement Large Language Model for Session-based Recommendation
Wang, Z., Du, Y., Sun, Z., Chua, H., Feng, K., Wang, W., & Zhang, J. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i12.33399 · Cited by 38
Process-Centric Analysis of Agentic Software Systems
Liu, S., Chen, Y., Krishna, R., Sinha, S., Ganhotra, J., & Jabbarvand, R. (2026) · Proceedings of the ACM on Programming Languages · doi:10.1145/3798271 · Cited by 42
“We sample 15% of the studied Graphectory, equally from resolved and unresolved instances for each <agent,model> pair, accounting for project diversity to avoid sampling bias.”
Chen, Y., Lv, A., Lin, T.-E., Chen, C., Wu, Y., Huang, F., Li, Y., & Yan, R. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.601 · Cited by 42
Confidence in the Reasoning of Large Language Models
Pawitan, Y., & Holmes, C. C. (2025) · Harvard Data Science Review · doi:10.1162/99608f92.b033a087 · Cited by 42
AIOS Compiler: LLM as Interpreter for Natural Language Programming and Flow Programming of AI AgentsPreprint
Xu, S., Li, Z., Mei, K., & Zhang, Y. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.06907 · Cited by 35
The AI Cosmologist I: An Agentic System for Automated Data AnalysisPreprint
Moss, A. (2025) · Research Square · doi:10.21203/rs.3.rs-6514417/v1 · Cited by 28
Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from RewardsPreprint
Wu, X. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.02686 · Cited by 29
“Another popular direction re-fines the generated code to correct errors and bugs through the language model itself (Shinn et al., 2023; Zhang et al., 2023a; Chen et al., 2023) or external feedback (Xie et al., 2025b).”
Open-Ended Instructable Embodied Agents with Memory-Augmented Large Language Models
Sarch, G., Wu, Y., Tarr, M., & Fragkiadaki, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.226 · Cited by 25
Li, X., Xu, W., Zhao, R., Jiao, F., Joty, S., & Bing, L. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.1244 · Cited by 29
Chen, P., Bu, P., Wang, Y., Wang, X., Wang, Z., Guo, J., Zhao, Y., Zhu, Q., Song, J., Yang, S., et al. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.01016 · Cited by 37
“With the advancement of visual language models (VLMs) [32, 42], works like Cradle [33] and VARP [5] demonstrate significant potential in playing video games.”
CLOVA: A Closed-LOop Visual Assistant with Tool Usage and Update
Gao, Z. F., Du, Y., Zhang, X., Ma, X., Han, W., Zhu, S. C., & Li, Q. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01259 · Cited by 72
Is Programming by Example Solved by LLMs?
Ellis, K., & Ellis, K. (2024) · neural information processing systems · doi:10.52202/079017-1422 · Cited by 49
ACC-Collab: An Actor-Critic Approach to Multi-Agent LLM Collaboration
Estornell, A., Ton, J.-F., Yao, Y., & Yang, L. (2025) · International Conference on Learning Representations · doi:10.48550/arxiv.2411.00053 · Cited by 35
“To address these limitations, several techniques have been proposed, such as Chain-of-Thought (CoT) prompting (Wei et al., 2022), Self-Reflection (Ji et al., 2023; Shinn et al., 2023), and Multi-Agent Debate (MAD) (Du et al., 2023), to name a few.”
ToolFuzz -- Automated Agent Tool TestingPreprint
Milev, I., Balunović, M., Baader, M., & Vechev, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.04479 · Cited by 30
ArcMemo: Abstract Reasoning Composition with Lifelong LLM MemoryPreprint
Ho, M., Chen, S., Feng, Z., Yu, F., Yang, Y., Liu, Z., Hu, Z., & Qin, L. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.04439 · Cited by 29
“Methods such as Reflexion (Shinn et al., 2023), Self-Refine (Madaan et al., 2023), and Self-RAG (Asai et al., 2023), inter alia attempt to correct mistakes at test-time by way of self-reflection.”
Stronger-MAS: Multi-Agent Reinforcement Learning for Collaborative LLMs
Zhao, Y., Hu, L., Wang, Y., Hou, M., Zhang, H., Ding, K., & Zhao, J. (2026) · International Conference on Learning Representations · doi:10.48550/arxiv.2510.11062 · Cited by 33
“Large Language Model (LLM) agents are task-specific workflows (Yao et al., 2023; Xi et al., 2023; Wang et al., 2023b) that utilize LLMs as key components for decision making (Shinn et al., 2023), action taking (Wang et al., 2023a), and tool use (Qian et al., 2025; Schick et al., 2023).”
Multi-Agent Deep Research: Training Multi-Agent Systems with M-GRPOPreprint
Hong, H., Yin, J., Wang, Y., Liu, J., Chen, Z., Yu, A., Li, J., Ye, Z., Xiao, H., Chen, Y., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2511.13288 · Cited by 30
“To improve reliability, self-monitoring approaches introduce process-level checks and feedback, including Reflexion-style reflection and LLM-as-a-judge techniques [Shinn et al., 2023, Gu et al., 2025].”
SPADE: Self-Play in Adaptive Synthetic Executable EnvironmentsPreprint
Liu, B., Yu, S., Jiang, Y., Qu, A., Zhao, A., Liu, Z., Kim, J., Zhou, Z., Kim, S., Ren, T., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.19197 · Cited by 27
Pan, L., Saxon, M., Xu, W., Nathani, D., Wang, X., & Wang, W. Y. (2023) · arXiv · doi:10.48550/arxiv.2308.03188 · Cited by 25
From Promise to Peril: Rethinking Cybersecurity Red and Blue Teaming in the Age of LLMs
Abuadbba, A., Moore, K., Goel, D., Hicks, C., Mavroudis, V., Hasırcıoglu, B., & Jennings, P. (2026) · IEEE Security & Privacy · doi:10.1109/msec.2025.3633466 · Cited by 28
A Survey of Progress in LLM Alignment From the Perspective of Reward Design
Ji, M., Wu, Y., Wu, Z., Wang, S., Yang, J., Dras, M., & Naseem, U. (2026) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2026.3654921 · Cited by 30
“For example, Reflexion distills failure experiences into prompt fragments so that the agent avoids repeating similar mistakes [56].”
Failures Pave the Way: Enhancing Large Language Models through Tuning-free Rule Accumulation
Yang, Z., Li, P., & Liu, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.109 · Cited by 49
OpenHands: An Open Platform for AI Software Developers as Generalist Agents
Wang, X., Li, B., Song, Y., Xu, F. F., Tang, X., Zhuge, M., Pan, J., Song, Y., Li, B., Singh, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.16741 · Cited by 24
“For example, thanks to OpenDevin’s extensible architecture, Auto Eval & Refine [42], an agent retry-on-error strategy with Reflexion [54] prompts and task completion reward models, will be integrated as an optional component attached to our browsing agent.”
LLMs cannot find reasoning errors, but can correct them given the error location
Tyen, G., Mansoor, H., Carbune, V., Chen, P., & Mak, T. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.826 · Cited by 156
Mindstorms in Natural Language-Based Societies of Mind
Zhuge, M., Liu, H., Faccio, F., Ashley, D. R., Csordás, R., Gopalakrishnan, A., Hamdi, A., Hammoud, H. A. A. K., Herrmann, V., Irie, K., et al. (2025) · Computational Visual Media · doi:10.26599/cvm.2025.9450460 · Cited by 24
From Illusion to Insight: A Taxonomic Survey of Hallucination Mitigation Techniques in LLMs
Kazlaris, I., Antoniou, E. N., Diamantaras, K., & Bratsas, C. (2025) · AI · doi:10.3390/ai6100260 · Cited by 35
Plehn, T., Schiller, D., & Schmal, N. (2026) · SciPost Physics Codebases · doi:10.21468/scipostphyscodeb.86 · Cited by 24
Toward Automated Validation of Language Model Synthesized Test Cases Using Semantic Entropy
Taherkhani, H., Shin, J., Tahir, M. A., Misu, M. R. H., Gattani, V. S., & Hemmati, H. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3664287 · Cited by 25
“In-valid or hallucinated tests pose two major challenges in LLM-based code generation: (1) false positives, where defective code passes erroneous tests, and (2) false negatives, where correct code fails due to flawed tests [3].”
InfluentialDriving Style Alignment for LLM-powered Driver Agent
Yang, R., Zhang, X., Fernandez-Laaksonen, A., Ding, X., & Gong, J. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10802629 · Cited by 36
“Recently, some studies have leveraged AI to generate feed-back or reflections[19], [20], [21], [22], yet they fall short in aligning such reflections with human perspectives.”
Thought Cloning: Learning to Think while Acting by Imitating Human Thinking
Hu, S., & Clune, J. (2023) · neural information processing systems · doi:10.52202/075280-1924 · Cited by 52
Build the web for agents, not agents for the webPreprint
Lù, X. H., Kamath, G., Mosbach, M., & Reddy, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.10953 · Cited by 36
“Consequently, methods like Reflexion [Shinn et al., 2023], SkillWeaver [Zheng et al., 2025] and Tree Search [Koh et al., 2024b] can be scaled to achieve better test-time performance, whereas methods like ADaPT [Prasad et al., 2024] can be accelerated to enable real-time usage.”
TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon AgentsPreprint
Tao, L., Peng, B., Yao, W., Ge, T., Cheng, H., Wang, M. H., Gao, J., & Li, S. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.13988 · Cited by 26
“…tool use through self-supervised tool-call prediction, instruction tuning on tool-use traces, and agent-specific tuning corpora (Schick et al., 2023; Qin et al., 2023; Zeng et al., 2023), while related approaches use feedback across trials to revise plans, memory, or behavior (Shinn et al., 2023).”
Shi, Z., Xin, C., Huo, T., Jiang, Y., Wu, B., Chen, X., Qin, W. D., Ma, X., Huang, G., Wang, Z., et al. (2025) · Scientific Reports · doi:10.1038/s41598-025-92337-6 · Cited by 43
An empirical study on capability of Large Language Models in understanding code semantics
Nguyen, T.-T., Vu, T. T., Vo, H. D., & Nguyen, S. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107780 · Cited by 33
Plan, Verify and Switch: Integrated Reasoning with Diverse X-of-Thoughts
Liu, T., Guo, Q., Yang, Y., Hu, X., Zhang, Y., Qiu, X., & Zhang, Z. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.169 · Cited by 49
RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning
Gehring, J., Zheng, K., Copet, J., Mella, V., Carbonneaux, Q., Cohen, T., & Synnaeve, G. (2024) · arXiv · doi:10.48550/arxiv.2410.02089 · Cited by 23
“This setup is applicable to language models tuned for the use case of interacting with users in a chat format and follows previous work on self-repair for code generation (Shinn et al., 2023; Olausson et al., 2024).”
Liao, D., Pan, S., Sun, X., Ren, X., Huang, Q., Xing, Z., Jin, H., & Li, Q. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3486195 · Cited by 23
“Instance Attributes.”
Enhancing Automated Program Repair with Solution Design
Zhao, J., Yang, D., Zhang, L., Lian, X., Yang, Z., & Liu, F. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695537 · Cited by 41
“Fortunately, recent research has shown that LLMs can better handle complex problems through various types of feedback and self-correction mechanisms [16, 49].”
STRATUS: A Multi-agent System for Autonomous Reliability Engineering of Modern Clouds
Chen, Y., Pan, J., Clark, J., Su, Y., Zheutlin, N., Bhavya, B., Arora, R., Deng, Y., Jha, S., & Xu, T. (2025) · neural information processing systems · doi:10.52202/085713-1671 · Cited by 54
Reasoning and Planning with Large Language Models in Code Development
Ding, H., Fan, Z., Guehring, I., Gupta, G. P., Ha, W., Huan, J., Liu, L. Q., Omidvar-Tehrani, B., Wang, S., & Zhou, H. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671452 · Cited by 40
Chen, Y.-C., Hsu, P.-C., Hsu, C.-J., & Shiu, D.-S. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-industry.9 · Cited by 35
Semantically Aligned Task Decomposition in Multi-Agent Reinforcement LearningPreprint
Li, W., Qiao, D., Wang, B., Wang, X., Jin, B., & Zha, H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.10865 · Cited by 30
WebWISE: Web Interface Control and Sequential Exploration with Large Language ModelsPreprint
Tao, H., Sethuraman, T. V., Shlapentokh-Rothman, M., & Hoiem, D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.16042 · Cited by 23
Yu, Y., Zhang, Q., Li, J., Fu, Q., & Ye, D. (2024) · Transactions on machine learning research · doi:10.48550/arxiv.2402.02053 · Cited by 29
LaDiR: Latent Diffusion Enhances LLMs for Text Reasoning
Kang, H., Zhang, Y., Kuang, N. L., Majamaki, N., Jaitly, N., Ma, Y.-A., & Qin, L. (2025) · International Conference on Learning Representations · doi:10.48550/arxiv.2510.04573 · Cited by 25
“Beyond following specific algorithms, works that implement long chain-of-thought (combining extensive reasoning, exploration, and reflection) have also demonstrated improved reasoning performance (Shinn et al., 2023; Gandhi et al., 2025; Saha et al., 2025; Xie et al., 2025).”
ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat InvestigationPreprint
Wu, Y., Velazco, M., Zhao, A., Luján, M. R. M., Movva, S., Roy, Y. K., Nguyen, Q., Rodriguez, R., Wu, Q., Albada, M., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.14201 · Cited by 30
From Lossy to Verified: A Provenance-Aware Tiered Memory for AgentsPreprint
Zhu, Q., Chen, S., Yu, R., Wu, Z., & Wang, B. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.17913 · Cited by 24
“Related work studies iterative retrieval and reflection-based improvement, (Shinn et al., 2023) and emphasizes verifiable grounding via retrieve–revise pipelines and fine-grained citations (Gao et al., 2022; Cao & Wang, 2024).”
Dhanorkar, S., Passi, S., & Vorvoreanu, M. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812402 · Cited by 36
“The 2021 breakthrough demonstrating the ability of large language models (LLMs) to analyze and generate software code [21] sparked widespread interest, driving significant technical advancements [116] including agentic frameworks such as OpenHands CodeAct [137].”
Reducing Hallucinations in Large Language Models Through Contextual Position EncodingPreprint
Desrochers, S., Wilson, J., & Beauchesne, M. (2024) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/exjqb · Cited by 22
Jiang, Y., Ko-Wong, L., & Gutierrez, I. V. (2025) · Educational Researcher · doi:10.3102/0013189x251314821 · Cited by 29
Democratizing Reasoning Ability: Tailored Learning from Large Language Model
Wang, Z., Huang, S., Liu, Y., Wang, J., Song, M., Zhang, Z., Huang, H., Wei, F., Deng, W., Sun, F., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.120 · Cited by 41
Li, J., Zhang, Q., Yu, Y., Fu, Q., & Ye, D. (2024) · arXiv · doi:10.48550/arxiv.2402.05120 · Cited by 253
“We also select representative methods (Du et al., 2023; Shinn et al., 2023) to combine with our method, achieving further enhancements.”
MethodMultiple LLM Agents Debate for Equitable Cultural Alignment
Ki, D., Rudinger, R., Zhou, T., & Carpuat, M. J. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.1210 · Cited by 45
Uncertainty Propagation on LLM Agent
Zhao, Q. C., Li, D., Liu, Y., Cheng, W., Sun, Y., Oishi, M., Osaki, T., Matsuda, K., Yao, H., Chen, Z., et al. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.302 · Cited by 68
FoE: Forest of Errors Makes the First Solution the Best in Large Reasoning Models
Jiang, K., Dong, H., Kang, Z., Zhu, Z., & Song, G. (2026) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2026.acl-long.1128 · Cited by 25
Lin, Y., Li, C., Ding, M., Tomizuka, M., Zhan, W., & Althoff, M. (2024) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2024.3441493 · Cited by 30
“1.”
Recover: A Neuro-Symbolic Framework for Failure Detection and Recovery
Cornelio, C., & Diab, M. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10801853 · Cited by 34
“Reflexion [15] offers a unique approach to reinforcing LLMs-based agents through verbal reflection on task feedback.”
MockLLM: A Multi-Agent Behavior Collaboration Framework for Online Job Seeking and Recruiting
Sun, H., Lin, H., Yan, H., Song, Y., Gao, X., & Yan, R. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3737051 · Cited by 28
“With recent enhancements in model scales [6, 23], LLMs have shown impressive capabilities in various domains such as text generation [2, 3], ranking [7, 18], and evaluation/verification [19, 31].”
Where's the Liability in Harmful AI Speech?Preprint
Henderson, P., Hashimoto, T., & Lemley, M. A. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.04635 · Cited by 22
Towards Goal-oriented Prompt Engineering for Large Language Models: A SurveyPreprint
Li, H., Leung, J., & Shen, Z. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.14043 · Cited by 26
Natural Language Reinforcement LearningPreprint
Feng, X., Liu, B., Song, Y., Fu, H., Wan, Z., Koushik, G. A., Hu, Z., Yang, M., Wen, Y., & Wang, J. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2411.14251 · Cited by 24
“This analysis is mainly based on human’s correlation judgment between the language evaluation Q Lπ old ( s, a ) and task objective T L .”
IMPROVE: Iterative Model Pipeline Refinement and Optimization Leveraging LLM ExpertsPreprint
Xue, E., Chen, K., Huang, Z., Ji, Y., & Wang, H. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.18530 · Cited by 27
“Additionally, LLMs tend to repeat or overwrite past decisions without introducing truly novel configurations [Shinn et al., 2023], especially in long-horizon tasks.”
L-MARS: Legal Multi-Agent System with Agentic Search and Citation-Faithfulness AuditPreprint
Yuan, B., & Wang, Z. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.00761 · Cited by 22
“Multi-agent frameworks (Wu et al., 2023; LangChain, 2024) enable role specialization and iterative critique, and recent agentic-RAG variants combine planning, verification, and retrieval for long-horizon tasks (Singh et al., 2025; Shinn et al., 2023).”
Data-to-Dashboard: Multi-Agent LLM Framework for Insightful Visualization in Enterprise AnalyticsPreprint
Zhang, R., & Elhamod, M. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.23695 · Cited by 22
“Self-Reflector We adopt the Reflexion framework [24] to enhance reasoning after the evaluation stage.”
MagicGUI: A Foundational Mobile GUI Agent with Scalable Data Pipeline and Reinforcement Fine-tuningPreprint
Tang, L., Dong, S., Huang, Y., Xiang, M., Ruan, H., Wang, B., Li, S., Xi, Z., Cao, Z., Pang, H., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.03700 · Cited by 25
“Additionally, recent advances emphasize the importance of key reasoning components such as self-reflection and planning for efficient task execution (Shinn et al., 2023; Dagan et al., 2023; Shen & Yang, 2025).”
Mao, L., Long, R., Huang, H., Ma, L., & Zhou, S. (2026) · International Conference on Communications, Information System and Computer Engineering (CISCE) · doi:10.1109/cisce69494.2026.11504499 · Cited by 21
Measuring the Influence of Incorrect Code on Test Generation
Huang, D., Zhang, J. M., Harman, M., Du, M., & Cui, H. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3764532 · Cited by 26
Automating traffic model enhancement with AI research agent
Guo, X., Yang, X., Peng, M., Lu, H., Zhu, M., & Yang, H. (2025) · Transportation Research Part C Emerging Technologies · doi:10.1016/j.trc.2025.105187 · Cited by 30
“Using advanced prompting techniques 21,22,38 and following carefully designed prompts, the Idea Generator first comprehends the original traffic model, then identifies its deficiencies and ultimately proposes improvements grounded in mathematical modeling principles.”
Recursive Introspection: Teaching Language Model Agents How to Self-Improve
Qu, Y., Zhang, T., Garg, N., & Kumar, A. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-1754 · Cited by 21
SceneGenAgent: Precise Industrial Scene Generation with Coding Agent
Xia, X., Zhang, D., Liao, Z., Hou, Z., Sun, T. R., Li, J., Fu, L., & Dong, Y. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.873 · Cited by 29
Wang, Z., Yoon, J., Yu, S., Islam, M. M., Bertasius, G., & Bansal, M. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.1428 · Cited by 51
Blueprint First, Model Second: A Framework for Deterministic LLM Workflow
Qiu, L., Ye, Y., Gao, Z., Zou, X., Chen, J., Gui, Z., Huang, W., Xue, X., Qiu, W., & Zhao, K. (2026) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme67971.2026.00018 · Cited by 24
ELT-Bench: An End-to-End Benchmark for Evaluating AI Agents on ELT Pipelines
Jin, T., Zhu, Y., & Kang, D. D. (2025) · Proceedings of the VLDB Endowment · doi:10.14778/3773749.3773750 · Cited by 32
CorNav: Autonomous Agent with Self-Corrected Planning for Zero-Shot Vision-and-Language Navigation
Liang, X., Ma, L., Guo, S., Han, J., Xu, H., Ma, S., & Liang, X. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.745 · Cited by 24
Formally Specifying the High-Level Behavior of LLM-Based AgentsPreprint
Crouse, M., Abdelaziz, I., Astudillo, R., Basu, K., Dan, S., Kumaravel, S., Fokoue, A., Kapanipathi, P., Roukos, S., & Lastras, L. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.08535 · Cited by 22
Exposing Limitations of Language Model Agents in Sequential-Task Compositions on the Web
Furuta, H., Matsuo, Y., Faust, A., & Gür, İ. (2024) · Transactions on machine learning research · doi:10.48550/arxiv.2311.18751 · Cited by 33
“2022; Kojima et al., 2022), program synthesis (Chen et al., 2021) and self-improvement (Shinn et al., 2023; Madaan et al., 2023; To et al., 2023), language model agents (LMAs) have recently emerged to tackle various decision making problems, such as robotics (Huang et al., 2022a; Ahn et al., 2022),…”
The Art of Scaling Test-Time Compute for Large Language ModelsPreprint
Agarwal, A., Sengupta, A., & Chakraborty, T. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.02008 · Cited by 21
“Chain-of-Thought (CoT) prompting (Wei et al., 2023) is a fundamental idea, and subsequent work like STaR (Zelikman et al., 2022) and Reflexion (Shinn et al., 2023) explore revision through trial-and-error or verbal self-reflection.”
AI Agent Systems: Architectures, Applications, and EvaluationPreprint
Xu, B. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.01743 · Cited by 30
“Variants such as “verbal reinforcement” (reflection-based self-improvement) adapt the idea of learning from feedback to language-agent loops [53].”
InfluentialHarness-R1: Learning to Edit Executable Runtime Harnesses from Agent Failure TrajectoriesPreprint
Shao, S., Zhang, K., Li, Q., Wang, S., Wang, H., Jiao, W., Yuan, L., Guo, Y., Liu, W., & Zhang, W. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.02276 · Cited by 23
“Context construction, memory and skills, tool mediation, action validation, and control and recovery logic are all harness components (Shinn et al., 2023; Zhao et al., 2024; Karten et al., 2026).”
Prime Agent: A Self-Improving RLM HarnessPreprint
Karten, S., Zhang, A., Thomas, K., Müller, S., Bakouch, E., Auras, D., Senghaas, M., Obeid, F., Dunas, K., Hagemann, J., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.23552 · Cited by 28
“Memory and refinement methods retain selected observations, feedback, skills, or reasoning traces across turns and tasks [22, 24, 26, 31, 36, 43].”
Metadata
- Authors
- Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., & Yao, S.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 8634-8652
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2303.11366
- OpenAlex
- W7133222210
- Cited by
- 1,087
- References
- 0
- Citations by year
- 2026: 760 · 2025: 167 · 2024: 105 · 2023: 55
- FWCI
- 192.15
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- language agents, verbal behavior, language acquisition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.