Low Resource Question Answering: An Amharic Benchmarking Dataset
Open accessAbstract
Question Answering (QA) systems return concise answers or answer lists based on natural language text, which uses a given context document.Many resources go into curating QA datasets to advance the development of robust QA models.There is a surge in QA datasets for languages such as English; this is different for low-resource languages like Amharic.Indeed, there is no published or publicly available Amharic QA dataset.Hence, to foster further research in low-resource QA, we present the first publicly available benchmarking Amharic Question Answering Dataset (Amh-QuAD).We crowdsource 2,628 question-answer pairs from over 378 Amharic Wikipedia articles.Using the training set, we fine-tune an XLM-R-based language model and introduce a new reader model.Leveraging our newly fine-tuned reader run a baseline model to spark open-domain Amharic QA research interest.The bestperforming baseline QA achieves an F-score of 80.3 and 81.34 in retriever-reader and reading comprehension settings.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The paper introduces the first comprehensive Amharic Retrieval-Augmented Generation Benchmark (ARGB).
“we introduce the first comprehensive Amharic Retrieval-Augmented Generation Benchmark (ARGB)”
From Improving Amharic Information Retrieval with Translative and Multi-Agent Debate Retrieval Augmented Generation (2026) · cites this paper as Taffa et al., 2024
The Introduction (page not provided) says the paper introduces Amh-QuAD, a factoid extractive open-domain Amharic question-answering dataset—not an Amharic Retrieval-Augmented Generation Benchmark called ARGB. The Reader section (page not provided) further describes selecting answer spans rather than generating answers. (LLM, conf=0.91)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredCurrently, there is only one publicly available question-answering benchmark for Amharic, Amh-QuAD.
“Currently, there is only one publicly available question-answering benchmark for Amharic, Amh-QuAD (Taffa et al., 2024)”
From Improving Amharic Information Retrieval with Translative and Multi-Agent Debate Retrieval Augmented Generation (2026) · cites this paper as Taffa et al., 2024
The Introduction says earlier Amharic QA models provided no public dataset and identifies Amh-QuAD as the first publicly available dataset. That supports its uniqueness when the paper was published, but does not establish that it remains the only publicly available benchmark at the citing document’s unspecified ‘currently.’ (LLM, conf=0.84)
Experimental — accuracy not yet measuredThe contexts defined for extraction in Amh-QuAD are not large enough (only a few sentences) to represent a knowledge space with high coverage and accurate retrieval quality measures.
“the contexts defined for extraction are not large enough (only a few sentences) to represent a knowledge space with high coverage and accurate retrieval quality measures.”
From Improving Amharic Information Retrieval with Translative and Multi-Agent Debate Retrieval Augmented Generation (2026) · cites this paper as Taffa et al., 2024
Section 5.1 (page unavailable) says contexts are split into segments of at most 200 words, which supports the description of relatively short extraction contexts. It does not say they contain only a few sentences or conclude that their size prevents high-coverage knowledge representation or accurate retrieval-quality measurement. Section 6.3 (page unavailable) instead reports retriever-performance measures for the dataset. (LLM, conf=0.78)
Experimental — accuracy not yet measuredAmh-QuAD is the first publicly available Amharic question-answering benchmark.
“The foundational effort in this domain is Amh-QuAD, introduced by (Taffa et al., 2024), which constitutes the first publicly available Amharic question-answering benchmark.”
From Improving Amharic Information Retrieval with Translative and Multi-Agent Debate Retrieval Augmented Generation (2026) · cites this paper as Taffa et al., 2024
Amh-QuAD was developed using the Amharic Wikipedia dump dataset and manual annotation.
“which was developed using the Amharic Wikipedia dump dataset and manual annotation.”
From Improving Amharic Information Retrieval with Translative and Multi-Agent Debate Retrieval Augmented Generation (2026) · cites this paper as Taffa et al., 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 14
- Bibliographical References Tilahun Abedissa and Mulugeta Libsie. 2019. Amharic Question Answering for Biography, Def- inition, and Description Questions. In Fisseha Mekuria, Ethiopia Nigussie, and Tesfa Tegegne, editors, Information and Communication Tech- nology for Development for Africa, volume 1026, pages 301-310. Springer International Publish- ing, Cham. Series Title: Communications in Computer and Information Science. · DOI
- Getahun Amare. 2013. Amharic Grammar in Sim- ple Way. Addis Ababa, Addis Ababa University Press.
- Phong Nguyen-Thuan Do, Nhat Duy Nguyen, Tin Van Huynh, Kiet Van Nguyen, Anh Gia-Tuan Nguyen, and Ngan Luu-Thuy Nguyen. 2021. Sen- tence Extraction-Based Machine Reading Com- prehension for Vietnamese. In Knowledge Sci- ence, Engineering and Management: 14th Inter- national Conference, KSEM 2021, Tokyo, Japan, August 14-16, 2021, Proceedings, Part II, page 511-523, Berlin, Heidelberg. Springer-Verlag. · DOI
- Tekabe Legesse Feleke. 2017. The similarity and mutual intelligibility between Amharic and Tigrigna varieties. In Proceedings of the Fourth Workshop on NLP for Similar Languages, Vari- eties and Dialects (VarDial), pages 47-54, Va- lencia, Spain. Association for Computational Lin- guistics. · DOI
- Fitsum Gaim, Wonsuk Yang, Hancheol Park, and Jong Park. 2023. Question-answering in a low- resourced language: Benchmark dataset and models for Tigrinya. In Proceedings of the 61st Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 11857-11870, Toronto, Canada. Asso- ciation for Computational Linguistics. · DOI
- Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT, volume 1, page 2.
- Md Tahmid Rahman Laskar, Jimmy Xiangji Huang, and Enamul Hoque. 2020. Contextualized em- beddings based transformer encoder for sen- tence similarity modeling in answer selection task. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 5505-5514, Marseille, France. European Lan- guage Resources Association.
- Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020b. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20, Red Hook, NY, USA.
- Timo Möller, Julian Risch, and Malte Pietsch. 2021. GermanQuAD and GermanDPR: Improving non- English question answering and passage re- trieval. In Proceedings of the 3rd Workshop on Machine Reading for Question Answering, pages 42-50, Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics. · DOI
- Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Ex- ploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(1). · DOI
- Pranav Rajpurkar, Jian Zhang, Konstantin Lopy- rev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. arXiv:1606.05250 [cs]. ArXiv: 1606.05250. · arXiv
- Fandy Setyo Utomo, Nanna Suryana, and Mohd Sanusi Aami. 2017. Question Answer- ing System: A Review on Question Analysis, Document Processing, and Answer Extraction Techniques. Journal of Theoretical & Applied Information Technology, 95(14).
- Baye Yimam. 2009. Amharic Grammar. Addis Ababa, Addis Ababa University Press.
- Seid Muhie Yimam and Mulugeta Libsie. 2009. TETEYEQ: Amharic question answering for fac- toid questions. IE-IR-LRL, 3(4):17.
Cited by
Sorted by citations
Jotie, A. A., & Mitra, P. (2026) · Proceedings of the Language Resources and Evaluation Conference · doi:10.63317/3hbm9z4okcrx
Desalegn, E., Berhanu, N., Hailu, S., Feyisa, T., Legesse, T., Bekele, Y., Adigeh, Y., & Assefa, Z. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.116194
Jotie, A. A., & Mitra, P. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808532
- Authors
- Taffa, T. A., Usbeck, R., & Assabie, Y.
- Affiliations
- Leuphana University of Lüneburg (DE)
- Addis Ababa University (ET)
- Venue
- Proceedings of the Language Resources and Evaluation Conference
- Published
- 2024-05
- Pages
- 124-132
- ISSN
- 2522-2686
- OpenAlex
- W7131799150
- Cited by
- 2
- References
- 0
- Citations by year
- 2026: 2
- FWCI
- 0.59
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Language Resources and Evaluation Conference (LREC) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- Amharic language, benchmark dataset, multilingual question answering, low-resource languages
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
