PreprintCited by 2Topic ModelingLibrary Science and Information SystemsText and Document Classification Technologies
The DOI matches its DataCite record.
SemEval-2025 Task 5: LLMs4Subjects - LLM-based Automated Subject Tagging for a National Technical Library's Open-Access Catalog
Open accessDOI 10.48550/arxiv.2504.07199arXiv 2504.07199
Abstract
We present SemEval-2025 Task 5: LLMs4Subjects, a shared task on automated subject tagging for scientific and technical records in English and German using the GND taxonomy. Participants developed LLM-based systems to recommend top-k subjects, evaluated through quantitative metrics (precision, recall, F1-score) and qualitative assessments by subject specialists. Results highlight the effectiveness of LLM ensembles, synthetic data generation, and multilingual processing, offering insights into applying LLMs for digital library classification.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
DataCite · 10.48550/arxiv.2504.07199
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“A key conclusion of the shared task was that 'the advantages of LLMs over traditional machine learning algorithms for subject indexing remain debatable' (D'Souza et al., 2025, p. 2), with smaller, well-engineered systems often rivaling large instruction-tuned LLMs.”
From A Skill-Based AI Agentic Pipeline for Library of Congress Subject Indexing (2026) · cites this paper as D'Souza et al., 2025
The cited source says what the claim attributes to it.
In the citing paper
“Yet the top-performing system in the allsubjects category was Annif (Suominen et al., 2025)-a traditional XMTC ensemble that used LLMs only for auxiliary preprocessing (translation and synthetic data generation), not for the core subject prediction task.”
From A Skill-Based AI Agentic Pipeline for Library of Congress Subject Indexing (2026) · cites this paper as D'Souza et al., 2025
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 41
- Mohammad Munzir Ahanger and M Arif Wani. 2022. Novel deep learning approach for scientific literature classification. In 2022 9th International Conference on Computing for Sustainable Global Development (INDIACom) , pages 249-254. IEEE.
- Hadi Bayrami Asl Tekanlou, Jafar Razmara, Mahsa Sanaei, Mostafa Rahgouy, and Hamed Babaei Giglou. 2025. Homa at semeval-2025 task 5: Aligning librarian records with ontoaligner for subject tagging. In Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025) , pages 23712377, Vienna, Austria. Association for Computational Linguistics.
- Levent Bolelli, ¸ Seyda Ertekin, and C Lee Giles. 2009. Topic and trend detection in text collections using latent dirichlet allocation. In European conference on information retrieval , pages 776-780. Springer.
- Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. M3embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through selfknowledge distillation. In Findings of the Association for Computational Linguistics ACL 2024 , pages 2318-2335.
- Aleksei Dorkin and Kairit Sirts. 2025. Tartunlp at semeval-2025 task 5: Subject tagging as two-stage information retrieval. In Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025) , pages 2414-2419, Vienna, Austria. Association for Computational Linguistics.
- Paul Gooding, Melissa Terras, Linda Berube, Mike Bennett, and Richard Hadden. 2019. Subjectifying library users to the macroscope using automatic classification matching. In Digital Humanities 2019: Annual conference of ADHO (Assoc of Digital Humanities Organisations) .
- Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AlDahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 . · arXiv
- Thomas L Griffiths and Mark Steyvers. 2004. Finding scientific topics. Proceedings of the National academy of Sciences , 101(suppl\_1):5228-5235.
- Jim Hahn. 2025. Jim at semeval-2025 task 5: Multilingual bert ensemble. In Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025) , pages 2378-2383, Vienna, Austria. Association for Computational Linguistics.
- Matthew Henderson, Rami Al-Rfou, Brian Strope, YunHsuan Sung, László Lukács, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and Ray Kurzweil. 2017. Efficient natural language response suggestion for smart reply. arXiv preprint arXiv:1705.00652 . · arXiv
- Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. Lora: Low-rank adaptation of large language models. ICLR , 1(2):3.
- Baharul Islam, Nasim Ahmad, Ferdous Barbhuiya, and Kuntal Dey. 2025. Nbf at semeval-2025 task 5: Light-burst attention enhanced system for multilingual subject recommendation. In Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025) , pages 956-961, Vienna, Austria. Association for Computational Linguistics.
- Sujay Khandagale, Han Xiao, and Rohit Babbar. 2020. Bonsai: diverse and shallow trees for extreme multi-label classification. Machine Learning , 109(11):2099-2119.
- Lisa Kluge and Maximilian Kähler. 2024. Few-shot prompting for subject indexing of German medical book titles. In Proceedings of the 20th Conference on Natural Language Processing (KONVENS 2024) , pages 141-148, Vienna, Austria. Association for Computational Linguistics.
- Lisa Kluge and Maximilian Kähler. 2025. Dnb-aiproject at semeval-2025 task 5: An llm-ensemble https://aclanthology.org/2025.semeval2025-1.147 approach for automated subject indexing. In Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025) , pages 1114-1124, Vienna, Austria. Association for Computational Linguistics.
- Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems , 33:9459-9474.
- Abdulhussain E Mahdi and Arash Joorabchi. 2011. Automatic subject classification of scientific literature using citation metadata. In International Conference on Digital Enterprise and Information Systems , pages 545-559. Springer.
- Yuheng Mao, Jin Wang, and Xuejie Zhang. 2025. Ynuhpcc at semeval-2025 task 7: Multilingual and crosslingual fact-checked claim retrieval. In Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025) , pages 238-244, Vienna, Austria. Association for Computational Linguistics.
- Olena Medelyan. 2009. Human-competitive automatic topic indexing . Ph.D. thesis, The University of Waikato.
- Zach Nussbaum, John X Morris, Brandon Duderstadt, and Andriy Mulyar. 2024. Nomic embed: Training a reproducible long context text embedder. arXiv preprint arXiv:2402.01613 . · arXiv
- Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems , 36:5372853741.
- Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 3982-3992.
- Michal Rˇ užiˇ cka and Petr Sojka. 2021. Towards math-aware automated classification and similarity search of scientific publications: Methods of mathematical content representations. arXiv preprint arXiv:2110.04040 . · arXiv
- Angelo A Salatino, Francesco Osborne, Thiviyan Thanapalasingam, and Enrico Motta. 2019. The cso classifier: Ontology-driven detection of research topics in scholarly articles. In Digital Libraries for Open Knowledge: 23rd International Conference on Theory and Practice of Digital Libraries, TPDL 2019, Oslo, Norway, September 9-12, 2019, Proceedings 23 , pages 296-311. Springer.
- Lauro Snidaro. 2025. La²i²f at semeval-2025 task 5: Reasoning in embedding space - fusing analogical and ontology-based reasoning for document subject tagging. In Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025) , pages 2384-2394, Vienna, Austria. Association for Computational Linguistics.
- Zahra Sarlak and Ebrahim Ansari. 2025. Last minute at semeval-2025 task 5: Rag system for subject tagging. In Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025) , pages 2403-2407, Vienna, Austria. Association for Computational Linguistics.
- Sumit Singh, Pankaj Kumar Goyal, and Uma Shanker Tiwary. 2025. silp\_nlp at semeval-2025 task 5: Subject recommendation with sentence transformer. In Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025) , pages 24202425, Vienna, Austria. Association for Computational Linguistics.
- Henry Small, Kevin W Boyack, and Richard Klavans. 2014. Identifying emerging topics in science and technology. Research policy , 43(8):1450-1467.
- Saba Sturua, Isabelle Mohr, Mohammad Kalim Akram, Michael Günther, Bo Wang, Markus Krimmel, Feng Wang, Georgios Mastrapas, Andreas Koukounas, Nan Wang, et al. 2024. jina-embeddings-v3: Multilingual embeddings with task lora. arXiv preprint arXiv:2409.10173 . · arXiv
- Ao Sun, Weilin Zhao, Xu Han, Cheng Yang, Zhiyuan Liu, Chuan Shi, and Maosong Sun. 2024. Burstattention: An efficient distributed attention framework for extremely long sequences. arXiv preprint arXiv:2403.09347 . · arXiv
- Osma Suominen. 2019. Annif: Diy automated subject indexing using multiple algorithms. LIBER Quarterly: The Journal of the Association of European Research Libraries , 29(1):1-25.
- Osma Suominen, Juho Inkinen, and Mona Lehtinen. 2022. Annif and finto ai: developing and implementing automated subject indexing. Bibliographic control in the digital ecosystem.-(Biblioteche & bibliotecari, 2612-7709; 5) , pages 265-282.
- Osma Suominen, Juho Inkinen, and Mona Lehtinen. 2025. Annif at semeval-2025 task 5: Traditional xmtc augmented by llms. In Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025) , pages 2395-2402, Vienna, Austria. Association for Computational Linguistics.
- Abhishek Thakur. 2024. Autotrain: No-code training for state-of-the-art models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 419-423.
- Hu Run Dong, Chen Tao, and Jia Jun Zhi. 2025a. RUC Team at SemEval-2025 Task 5: Fast Automated Subject Indexing: A Method Based on Similar Records Matching and Related Subject Ranking. In Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025) , Vienna, Austria. Association for Computational Linguistics.
- Yicen Tian, Erchen Yu, Yanan Wang, Dailin Li, jiaqi yao, Hongfei LIN, Linlin Zong, and Bo Xu. 2025b. Dutir831 at semeval-2025 task 5: A multi-stage llm approach to gnd subject assignment for tibkat records. In Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025) , pages 367376, Vienna, Austria. Association for Computational Linguistics.
- Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Multilingual e5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672 . · arXiv
- An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115 . · arXiv
- Hsiang-Fu Yu, Kai Zhong, Jiong Zhang, Wei-Cheng Chang, and Inderjit S Dhillon. 2022. Pecos: Prediction for enormous and correlated output spaces. Journal of Machine Learning Research , 23(98):1-32.
- Puxuan Yu, Luke Merrick, Gaurav Nuti, and Daniel Campos. 2024. Arctic-embed 2.0: Multilingual retrieval without compromise. arXiv preprint arXiv:2412.04506 . · arXiv
- Dun Zhang, Jiacheng Li, Ziyang Zeng, and Fulong Wang. 2024. Jasper and stella: distillation of sota embedding models. arXiv preprint arXiv:2412.19048 . · arXiv
Cited by
Sorted by citations
Research on Large Language Model-Based Bibliographic Cataloging Agent in the CNMARC Context
Tan, Z., Yang, X., Chen, Q., & Chen, T. (2026) · Publications · doi:10.3390/publications14010019 · Cited by 2
A Skill-Based AI Agentic Pipeline for Library of Congress Subject Indexing
Chow, E. H. C. (2026) · arXiv · doi:10.48550/arxiv.2605.03537 · Cited by 1
“A key conclusion of the shared task was that “the advantages of LLMs over traditional machine learning algorithms for subject indexing remain debat-able” (D’Souza et al., 2025, p. 2), with smaller, well-engineered systems often rivaling large instruction-tuned LLMs.”
Metadata
- Authors
- D'Souza, J., Sadruddin, S., Israel, H., Begoin, M., & Slawig, D.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2504.07199
- OpenAlex
- W4414792145
- Cited by
- 1
- References
- 0
- Citations by year
- 2025: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Library Science and Information Systems
- Physical Sciences › Computer Science › Artificial Intelligence › Text and Document Classification Technologies
- Keywords
- LLM ensembles, synthetic data generation, multilingual processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
