The DOI matches its DataCite record.
Annif at SemEval-2025 Task 5: Traditional XMTC augmented by LLMs
Open accessAbstract
This paper presents the Annif system in SemEval-2025 Task 5 (LLMs4Subjects), which focussed on subject indexing using large language models (LLMs). The task required creating subject predictions for bibliographic records from the bilingual TIBKAT database using the GND subject vocabulary. Our approach combines traditional natural language processing and machine learning techniques implemented in the Annif toolkit with innovative LLM-based methods for translation and synthetic data generation, and merging predictions from monolingual models. The system ranked first in the all-subjects category and second in the tib-core-subjects category in the quantitative evaluation, and fourth in qualitative evaluations. These findings demonstrate the potential of combining traditional XMTC algorithms with modern LLM techniques to improve the accuracy and efficiency of subject indexing in multilingual contexts.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The top-performing system in the all-subjects category of SemEval-2025 LLMs4Subjects was Annif, a traditional XMTC ensemble that used LLMs only for auxiliary preprocessing.
“Yet the top-performing system in the allsubjects category was Annif (Suominen et al., 2025)-a traditional XMTC ensemble that used LLMs only for auxiliary preprocessing (translation and synthetic data generation), not for the core subject prediction task.”
From A Skill-Based AI Agentic Pipeline for Library of Congress Subject Indexing (2026) · cites this paper as Suominen et al., 2025
Unlike XMTC approaches, the system does not require training data or corpus-level pattern learning.
“Unlike the XMTC approaches of Asula et al. (2021) and Suominen et al. (2025), the system does not require training data or corpus-level pattern learning.”
From A Skill-Based AI Agentic Pipeline for Library of Congress Subject Indexing (2026) · cites this paper as Suominen et al. (2025)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 9
- Jennifer D'Souza, Sameer Sadruddin, Holger Israel, Mathias Begoin, and Diana Slawig. 2025. SemEval- 2025 task 5: LLMs4Subjects -LLM-based auto- mated subject tagging for a national technical li- brary's open-access catalog. In Proceedings of the 19th International Workshop on Semantic Evaluation (SemEval-2025), pages 1082-1095, Vienna, Austria. Association for Computational Linguistics.
- T. J. Kao Eric H. C. Chow and Xiaoli Li. 2024. An experiment with the use of ChatGPT for LCSH sub- ject assignment on electronic theses and dissertations. Cataloging & Classification Quarterly, 62(5):574- 588. · DOI · Paper page
- Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The Llama 3 herd of models. Preprint, arXiv:2407.21783. · arXiv
- Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumu- lated gain-based evaluation of IR techniques. ACM Trans. Inf. Syst., 20(4):422-446. · DOI · Paper page
- Sujay Khandagale, Han Xiao, and Rohit Babbar. 2020. Bonsai: Diverse and shallow trees for ex- treme multi-label classification. Machine Learning, 109(11):2099-2119. · DOI
- Olena Medelyan. 2009. Human-competitive automatic topic indexing. Ph.D. thesis, The University of Waikato.
- Yashoteja Prabhu, Anil Kag, Shrutendra Harsola, Rahul Agrawal, and Manik Varma. 2018. Parabel: Par- titioned label trees for extreme classification with application to dynamic search advertising. WWW '18, page 993-1002, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee. · DOI
- Osma Suominen, Juho Inkinen, and Mona Lehtinen. 2022. Annif and Finto AI: Developing and im- plementing automated subject indexing. JLIS.it, 13(1):265-282. · DOI
- Hsiang-Fu Yu, Kai Zhong, Jiong Zhang, Wei-Cheng Chang, and Inderjit S. Dhillon. 2022. PECOS: Pre- diction for enormous and correlated output spaces. Preprint, arXiv:2010.05878. · arXiv
Cited by
Sorted by citations
A Skill-Based AI Agentic Pipeline for Library of Congress Subject Indexing
Chow, E. H. C. (2026) · arXiv · doi:10.48550/arxiv.2605.03537 · Cited by 1
“The National Library of Finland’s Annif toolkit represents a more mature approach, combining multiple XMTC algorithms— including Omikuji/Bonsai (partitioned label trees), MLLM (lexical matching against controlled vocabulary terms), and XTransformer (fine-tuned BERT-style models)—into configurable ensembles (Suominen e…”
Influential
Metadata
- Authors
- Suominen, O., Inkinen, J., & Lehtinen, M.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2025
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2504.19675
- OpenAlex
- W4416902886
- Cited by
- 1
- References
- 0
- Citations by year
- 2026: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- Työväentutkimus Vuosikirja (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Helsingin Yliopisto
- Leibniz-Gemeinschaft
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Library Science and Information Systems
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- subject indexing, Annif, large language models, synthetic data generation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
