Towards AI-Assisted Research Writing: Benchmarking LLMs for AI/ML Introduction Generation
Open accessAbstract
As researchers increasingly adopt LLMs as writing assistants, generating high-quality research paper introductions remains both challenging and essential. We introduce Scientific Introduction Generation (SciIG), a task that evaluates LLMs' ability to produce coherent introductions from titles, abstracts, and related works. Curating new datasets from NAACL 2025 and ICLR 2025 papers, we assess five state-of-the-art models, including both open-source (DeepSeek-v3, Gemma-3-12B, LLaMA 4-Maverick, MistralAI Small 3.1) and closed-source GPT-4o systems, across multiple dimensions: lexical overlap, semantic similarity, content coverage, faithfulness, consistency, citation correctness, and narrative quality. Our comprehensive framework combines automated metrics with LLM-as-a-judge evaluations. Results demonstrate LLaMA-4 Maverick's superior performance on most metrics, particularly in semantic similarity and faithfulness. Moreover, three-shot prompting consistently outperforms fewer-shot approaches. These findings provide practical insights into developing effective research writing assistants and set realistic expectations for LLM-assisted academic writing. To foster re- producibility and future research, we publicly release all code and datasets.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Several recent benchmarks, including AcademicEval [63], LongBench-Cite [64] and SciIG [13], evaluate long-form generation tasks that do not require search over the live web.
“Alternatively, several recent benchmarks, including AcademicEval [63], LongBench-Cite [64] and SciIG [13], evaluate long-form generation tasks that do not require search over the live web , which is a key component of generative research synthesis and our benchmark.”
From DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesis (2025) · cites this paper as [13]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1× · see citing works
“The rapid advancement of generative artificial intelligence, particularly large language models (LLMs), like OpenAI’s GPT series (Brown et al., 2020; Achiam et al., 2023), has sparked significant tasks (Lu et al., 2020; LIU et al., 2022).”
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · see citing works
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 16× · see citing works
“Citation Precision and Citation Recall operate over citation identifiers (“et al.” strings) rather than over titles or DOIs; identifiers are extracted from the generated introduction using LLaMA 4-Maverick with the prompt in Table 10.”
InfluentialTraining Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 1× · see citing works
“5 (Arshad et al., 2023), GPT-4 (Achiam et al., 2023; Ouyang et al., 2022), PaLM (Chowdhery et al., 2023), LLaMA (Touvron et al., 2023), Bloom (Workshop et al., 2022), and GLaM (Du et al., 2022) have demonstrated impressive performance across summarization benchmarks.”
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · cited 1× · see citing works
“5 (Arshad et al., 2023), GPT-4 (Achiam et al., 2023; Ouyang et al., 2022), PaLM (Chowdhery et al., 2023), LLaMA (Touvron et al., 2023), Bloom (Workshop et al., 2022), and GLaM (Du et al., 2022) have demonstrated impressive performance across summarization benchmarks.”
GLaM: Efficient Scaling of Language Models with Mixture-of-ExpertsPreprint
Du, N., Huang, Y.-P., Dai, A. M., Tong, S., Lepikhin, D., Xu, Y.-Z., Krikun, M., Zhou, Y.-Q., Yu, A. W., Firat, O., et al. (2021) · International Conference on Machine Learning · doi:10.48550/arxiv.2112.06905 · Cited by 1,379 · cited 1× · see citing works
“5 (Arshad et al., 2023), GPT-4 (Achiam et al., 2023; Ouyang et al., 2022), PaLM (Chowdhery et al., 2023), LLaMA (Touvron et al., 2023), Bloom (Workshop et al., 2022), and GLaM (Du et al., 2022) have demonstrated impressive performance across summarization benchmarks.”
Gemma Team, Kamath, A., Ferret, J., Pathak, S., Vieillard, N., Merhej, R., Perrin, S., Matejovicova, T., Ramé, A., Rivière, M., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.19786 · Cited by 1,156 · cited 1×
“1-24b 7 , Gemma-3-12b-it (Team et al., 2025), and GPT-4o (Hurst et al., 2024).”
BLEURT: Learning Robust Metrics for Text Generation
Sellam, T., Das, D., & Parikh, A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.704 · Cited by 1,005 · cited 2× · see citing works
“Semantic Similarity assesses meaning-level alignment using BERTScore (Zhang et al., 2019), BLEURT (Sellam et al., 2020) against the input context (Title and Abstract).”
G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.153 · Cited by 948 · cited 1× · see citing works
“By combining automated metrics with LLM-as-a-judge evaluations (Liu et al., 2023), we seek to provide a holistic understanding of the strengths and limitations of current state-of-the-art models.”
Maier, C. (2012) · doi:10.4324/9780203432860-37 · Cited by 726 · cited 1×
“Under Swales’ CARS model (Swales, 1990), introductions establish the research territory, identify a gap, and position the contribution.”
S2ORC: The Semantic Scholar Open Research Corpus
Lo, K., Wang, L. L., Neumann, M., Kinney, R., & Weld, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.447 · Cited by 457 · cited 2× · see citing works
“We process the PDFs using the grobid2json parsing tool provided by the S2ORC project (Lo et al., 2020).”
Best practices for the human evaluation of automatically generated text
van der Lee, C., Gatt, A., van Miltenburg, E., Wubben, S., & Krahmer, E. (2019) · Proceedings of the 12th International Conference on Natural Language Generation · doi:10.18653/v1/w19-8643 · Cited by 205 · see citing works
TLDR: Extreme Summarization of Scientific Documents
Cachola, I., Lo, K., Cohan, A., & Weld, D. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.428 · Cited by 169 · see citing works
Lu, Y., Dong, Y., & Charlin, L. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.648 · Cited by 97 · cited 1× · see citing works
“The rapid advancement of generative artificial intelligence, particularly large language models (LLMs), like OpenAI’s GPT series (Brown et al., 2020; Achiam et al., 2023), has sparked significant tasks (Lu et al., 2020; LIU et al., 2022).”
Can LLMs Generate Architectural Design Decisions? - An Exploratory Empirical Study
Dhar, R., Vaidhyanathan, K., & Varma, V. (2024) · IEEE International Conference on Software Architecture (ICSA) · doi:10.1109/icsa59870.2024.00016 · Cited by 62 · see citing works
Efficiently Summarizing Text and Graph Encodings of Multi-Document Clusters
Pasunuru, R., Liu, M., Bansal, M., Ravi, S., & Dreyer, M. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.380 · Cited by 55 · cited 1× · see citing works
“Meanwhile, graph-based methods (Liao et al., 2018; Pasunuru et al., 2021; Bandyopadhyay and Muppidi, 2026) leverage structured representations such as Abstract Meaning Representation (AMR) and discourse relations to encode inter-document connections.”
PaperRobot: Incremental Draft Generation of Scientific Ideas
Wang, Q., Huang, L., Jiang, Z., Knight, K. K., Ji, H., Bansal, M., & Luan, Y. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1191 · Cited by 48 · see citing works
Hase, P., & Bansal, M. (2022) · LNLS · doi:10.18653/v1/2022.lnls-1.4 · Cited by 40 · cited 1× · see citing works
“While these models excel in single-document generation tasks, their application to multi-input, constraint-driven generation—such as meta-review synthesis or domain-specific aggregation—remains under-investigated (Hase and Bansal, 2022).”
Generating a Structured Summary of Numerous Academic Papers: Dataset and Method
Liu, S., Cao, J.-N., Yang, R., & Wen, Z. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/591 · Cited by 32 · cited 1×
“The rapid advancement of generative artificial intelligence, particularly large language models (LLMs), like OpenAI’s GPT series (Brown et al., 2020; Achiam et al., 2023), has sparked significant tasks (Lu et al., 2020; LIU et al., 2022).”
Overview and Results: CL-SciSumm Shared Task 2019
Chandrasekaran, M. K., Yasunaga, M., Radev, D., Freitag, D., & Kan, M.-Y. (2019) · arXiv · doi:10.48550/arxiv.1907.09854 · Cited by 27 · see citing works
LDKP: A Dataset for Identifying Keyphrases from Long Scientific Documents
Mahata, D., Agarwal, N., Gautam, D., et al. (2022) · arXiv.org · doi:10.48550/arxiv.2203.15349 · Cited by 19 · cited 1×
“Our ICLR and NAACL datasets comprise 3,900 high-quality long-form NLP and ML papers, comparable to existing benchmarks (Costa-jussà et al., 2025; Chan-drasekaran et al., 2019; Mahata et al., 2022; Ca-chola et al., 2020).”
SurveyGen: Quality-Aware Scientific Survey Generation with Large Language Models
Bao, T., Nayeem, M. T., Rafiei, D., et al. (2025) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2508.17647 · Cited by 13
LCFO: Long Context and Long Form Output Dataset and Benchmarking
Costa-jussà, M., Andrews, P., Meglioli, M. C., Chen, J., Chuang, J., Dale, D., Ropers, C., Mourachko, A., Sánchez, E., Schwenk, H., et al. (2024) · arXiv.org · doi:10.48550/arxiv.2412.08268 · Cited by 5
Jaisankar, V., Bandyopadhyay, S., Vyas, K., et al. (2025) · AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i23.34598 · Cited by 3
Self-citationThe Performance of GPT-3.5 in Summarizing Scientific and News Articles
Arshad, S., Yaqoob, M., & Mehmood, T. (2024) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-981-97-0293-0_4 · Cited by 3 · cited 1× · see citing works
“5 (Arshad et al., 2023), GPT-4 (Achiam et al., 2023; Ouyang et al., 2022), PaLM (Chowdhery et al., 2023), LLaMA (Touvron et al., 2023), Bloom (Workshop et al., 2022), and GLaM (Du et al., 2022) have demonstrated impressive performance across summarization benchmarks.”
Nandy, A., & Bandyopadhyay, S. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i23.34676 · Cited by 1 · cited 1× · see citing works
“Nandy and Bandyopadhyay (2025) have proposed an approach to represent a collection of documents in the form of a hierarchical reference layout tree and generate a summary with attribution from this tree with selected in-context examples.”
Self-citationBandyopadhyay, S., & Muppidi, A. (2026) · arXiv.org · doi:10.48550/arxiv.2607.06527 · Cited by 1 · cited 1×
“Meanwhile, graph-based methods (Liao et al., 2018; Pasunuru et al., 2021; Bandyopadhyay and Muppidi, 2026) leverage structured representations such as Abstract Meaning Representation (AMR) and discourse relations to encode inter-document connections.”
Self-citationOpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774 · cited 15×
“In a detailed analysis of a NAACL 2025 dataset paper in Appendix H, both LLaMA-4 Maverick and GPT-4o produced fluent and faithful introductions with strong narrative flow and accurate citation usage.”
InfluentialBLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Workshop, B., :, Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A. S., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.05100 · cited 1×
“5 (Arshad et al., 2023), GPT-4 (Achiam et al., 2023; Ouyang et al., 2022), PaLM (Chowdhery et al., 2023), LLaMA (Touvron et al., 2023), Bloom (Workshop et al., 2022), and GLaM (Du et al., 2022) have demonstrated impressive performance across summarization benchmarks.”
Flagged first, then by citations
14 of the 43 references couldn’t be matched to a known paper.
Research works printed but not linked: 15
- Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization , pages 65-72, Ann Arbor, Michigan. Association for Computational Linguistics.
- Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. 2025. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems , 43(2):155.
- Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276 . · arXiv
- Rodney Michael Kinney, Chloe Anastasiades, Russell Authur, Iz Beltagy, Jonathan Bragg, Alexandra Buraczynski, Isabel Cachola, Stefan Candra, Yoganand Chandrasekhar, Arman Cohan, Miles Crawford, Doug Downey, Jason Dunkelberger, Oren Etzioni, Rob Evans, Sergey Feldman, Joseph Gorney, David W. Graham, F.Q. Hu, Regan Huff, Daniel King, Sebastian Kohlmeier, Bailey Kuehl, Michael Langan, Daniel Lin, Haokun Liu, Kyle Lo, Jaron Lochner, Kelsey MacMillan, Tyler C. Murray, Christopher Newell, Smita R Rao, Shaurya Rohatgi, Paul Sayre, Zejiang Shen, Amanpreet Singh, Luca Soldaini, Shivashankar Subramanian, A. Tanaka, Alex D Wade, Linda M. Wagner, Lucy Lu Wang, Christopher Wilhelm, Caroline Wu, Jiangjiang Yang, Angele Zamarron, Madeleine van Zuylen, and Daniel S. Weld. 2023. The semantic scholar open data platform. ArXiv , abs/2301.10140. · arXiv
- Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. 2024. Llms-as-judges: a comprehensive survey on llm-based evaluation methods. arXiv preprint arXiv:2412.05579 . · arXiv
- Xiangci Li and Jessica Ouyang. 2025. Explaining relationships among research papers. In Proceedings of the 31st International Conference on Computational Linguistics , pages 1080-1105, Abu Dhabi, UAE. Association for Computational Linguistics.
- Kexin Liao, Logan Lebanoff, and Fei Liu. 2018. Abstract Meaning Representation for multi-document summarization. In Proceedings of the 27th International Conference on Computational Linguistics , pages 1178-1190, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
- Daniel J. Liebling, Malcolm Kane, Madeleine GrundeMcLaughlin, Ian Lang, Subhashini Venugopalan, and Michael Brenner. 2025. Towards AI-assisted academic writing. In Proceedings of the 1st Workshop on AI and Scientific Discovery: Directions and Opportunities , pages 31-45, Albuquerque, New Mexico, USA. Association for Computational Linguistics.
- Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out , pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
- Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 . · arXiv
- Yao Lu, Yue Dong, and Laurent Charlin. 2020. Multixscience: A large-scale dataset for extreme multidocument summarization of scientific articles. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP) , pages 8068-8074.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9.
- Yidong Wang, Qi Guo, Wenjin Yao, Hongbo Zhang, Xin Zhang, Zhen Wu, Meishan Zhang, Xinyu Dai, Min Zhang, Qingsong Wen, et al. 2024. Autosurvey: Large language models can automatically write surveys. Advances in neural information processing systems , 37:115119-115145.
- Wen Xiao, Iz Beltagy, Giuseppe Carenini, and Arman Cohan. 2022. PRIMERA: Pyramid-based masked sentence pre-training for multi-document summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 5245-5263, Dublin, Ireland. Association for Computational Linguistics.
- Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 . · arXiv
Cited by
Sorted by citations
Eliminating Agentic Workflow for Introduction Generation with Parametric Stage Tokens
Zhang, M., Su, T., & He, G.-X. (2025) · arXiv.org · doi:10.48550/arxiv.2601.09728 · Cited by 1
“The absence of any single argumentative link disrupts textual coherence and precipitates the omission of critical evidentiary elements such as experimental outcomes (Garg et al., 2025).”
DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesis
Patel, L., Arabzadeh, N., Gupta, H., Sundar, A., Stoica, I., Zaharia, M., & Guestrin, C. (2025) · arXiv · doi:10.48550/arxiv.2508.20033
How Is Automated Research Evaluated? A Survey of Benchmarks and Evaluation Practices
Zhang, L.-L., Zhou, D.-J., Huang, C.-Y., et al. (2026) · doi:10.48550/arxiv.2610.11877
“At introduction level, SciIG separately evaluates reference-set recovery, content, and narrative quality from supplied metadata (Garg et al., 2026).”
Zhang, M., Su, T., Cheng, J.-H., et al. (2026) · doi:10.48550/arxiv.2608.03138
“We adapt ELABORATE prompting (Garg et al. 2025) to introduction generation by enforcing a four-part rhetorical structure covering context, research gap, proposed solution and contribution.”
DraftNEPABench: A Benchmark for Drafting NEPA Document Sections with Coding Agents
Acharya, A., Lakha, B., Meyur, R., Nuttall, R., Chaturvedi, S., Halappanavar, A., Hare, L., Zeng, L., Parker, M., Munikoti, S., et al. (2026) · ACM Conference on AI and Agentic Systems (CAIS) · doi:10.1145/3786335.3813132
“However, limitations in technical accuracy, citation contextualization, and hallucination make human oversight essential [4, 12].”
LECTOR: Joint Optimization of Scientific Reasoning Graphs and Introduction Generation
Xiao, J., Wang, Y., Tang, C., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2605.25964
“While SciIG’s setup prioritizes topical relevance and summary, our CCIG task focuses on logic grounding and writing quality .”
- Authors
- Garg, K., Shaik, F., Bandyopadhyay, S., & Caragea, C.
- Affiliations
- University of Oxford (GB)
- University of Illinois Chicago (US)
- ORCID
- Caragea, Cornelia
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- Pages
- 29341-29365
- arXiv
- 2508.14273
- OpenAlex
- W7077070083
- Semantic Scholar
- 6926057cee944726a4a8aa1194e1bc068d283db7
- Cited by
- 0
- References
- 43
- FWCI
- 0.00
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Academic Writing and Publishing
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- LLM-as-a-judge, LLM benchmarking, AI writing tools, scientific writing, semantic similarity, faithfulness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
