CoChat
Sign Up
PreprintCited by 6Topic ModelingNatural Language Processing TechniquesInformation Retrieval and Search Behavior

The DOI matches its DataCite record.

A Benchmark Dataset with Larger Context for Non-Factoid Question Answering over Islamic Text

Open access
Qamar, Faiza, Latif, Seemab, Latif, Rabia·2024·arXiv
DOI 10.48550/arxiv.2409.09844arXiv 2409.09844
Open this paper in CoChat — free account

Abstract

Accessing and comprehending religious texts, particularly the Quran (the sacred scripture of Islam) and Ahadith (the corpus of the sayings or traditions of the Prophet Muhammad), in today's digital era necessitates efficient and accurate Question-Answering (QA) systems. Yet, the scarcity of QA systems tailored specifically to the detailed nature of inquiries about the Quranic Tafsir (explanation, interpretation, context of Quran for clarity) and Ahadith poses significant challenges. To address this gap, we introduce a comprehensive dataset meticulously crafted for QA purposes within the domain of Quranic Tafsir and Ahadith. This dataset comprises a robust collection of over 73,000 question-answer pairs, standing as the largest reported dataset in this specialized domain. Importantly, both questions and answers within the dataset are meticulously enriched with contextual information, serving as invaluable resources for training and evaluating tailored QA systems. However, while this paper highlights the dataset's contributions and establishes a benchmark for evaluating QA performance in the Quran and Ahadith domains, our subsequent human evaluation uncovered critical insights regarding the limitations of existing automatic evaluation techniques. The discrepancy between automatic evaluation metrics, such as ROUGE scores, and human assessments became apparent. The human evaluation indicated significant disparities: the model's verdict consistency with expert scholars ranged between 11% to 20%, while its contextual understanding spanned a broader spectrum of 50% to 90%. These findings underscore the necessity for evaluation techniques that capture the nuances and complexities inherent in understanding religious texts, surpassing the limitations of traditional automatic metrics.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2409.09844

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 44

  1. [1][1] Heba Abdelnasser, Maha Ragab, Reham Mohamed, Alaa Mohamed, Bassant Farouk, Nagwa M El-Makky, and Marwan Torki. 2014. Al-Bayan: an Arabic question answering system for the Holy Quran. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP) . 57-64. -
  2. [2][2] Alyssa Ayelet Aharon. 2022. Deep Torah Learning: A Deep Learning Approach to Identifying Anomalous Letters in Torah Scrolls . Ph.D. Dissertation. Yeshiva University. -
  3. [3][3] Basem Ahmed, Motaz Saad, and Eshrag A Refaee. 2022. QQATeam at Qur'an QA 2022: Fine-Tunning Arabic QA Models for Qur'an QA Task. In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection . 130-135. -
  4. [4][4] Yazeed Alkhurayyif and Abdul Rahaman Wahab Sait. 2023. A comprehensive survey of techniques for developing an Arabic question answering system. PeerJ Computer Science 9 (2023), e1413. -
  5. [5][5] Reinald Kim Amplayo, Kellie Webster, Michael Collins, Dipanjan Das, and Shashi Narayan. 2022. Query Refinement Prompts for Closed-Book LongForm Question Answering. arXiv preprint arXiv:2210.17525 (2022). - · arXiv
  6. [6][6] Negar Arabzadeh, Bhaskar Mitra, and Ebrahim Bagheri. 2021. Ms marco chameleons: challenging the ms marco leaderboard with extremely obstinate queries. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management . 4426-4435. -
  7. [7][7] Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268 (2016). - · arXiv
  8. [8][8] Muhammad Huzaifa Bashir, Aqil M Azmi, Haq Nawaz, Wajdi Zaghouani, Mona Diab, Ala Al-Fuqaha, and Junaid Qadir. 2023. Arabic natural language processing for Qur'anic research: A systematic review. Artificial Intelligence Review 56, 7 (2023), 6801-6854. -
  9. [9][9] Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150 (2020). - · arXiv
  10. [10][10] Meghana Moorthy Bhat, Rui Meng, Ye Liu, Yingbo Zhou, and Semih Yavuz. 2023. Investigating Answerability of LLMs for Long-Form Question Answering. arXiv preprint arXiv:2309.08210 (2023). - · arXiv
  11. [11][11] Eduardo Gabriel Cortes, Vinicius Woloszyn, Dante Barone, Sebastian M¨ oller, and Renata Vieira. 2022. A systematic review of question answering systems for non-factoid questions. Journal of Intelligent Information Systems (2022), 1-28. -
  12. [13][13] Aprameya Dash, Mohit Awachar, Anshul Patel, and Bhawana Rudra. 2023. Open-Domain Long-Form Question-Answering Using Transformer-Based Pipeline. SN Computer Science 4, 5 (2023), 595. -
  13. [14][14] Markus Eberts and Adrian Ulges. 2019. Span-based joint entity and relation extraction with transformer pre-training. arXiv preprint arXiv:1909.07755 (2019). - · arXiv
  14. [15][15] Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. ELI5: Long form question answering. arXiv preprint arXiv:1907.09190 (2019). - · arXiv
  15. [16][16] Mandy Guo, Joshua Ainslie, David Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang. 2021. LongT5: Efficient text-to-text transformer for long sequences. arXiv preprint arXiv:2112.07916 (2021). - · arXiv
  16. [17][17] Aimad Hakkoum and Said Raghay. 2016. Semantic Q&A System on the Qur'an. Arabian Journal for Science and Engineering 41, 12 (2016), 52055214. -
  17. [18][18] Suhaib Kh. Hamed and Mohd Juzaiddin Ab Aziz. 2016. A Question Answering System on Holy Quran Translation Based on Question Expansion Technique and Neural Network Classification. J. Comput. Sci. 12 (2016), 169-177. -
  18. [20][20] Zhifeng Hao, Junhao Chen, Wen Wen, Biao Wu, and Ruichu Cai. 2022. Motif-based memory networks for complex-factoid question answering. Neurocomputing 485 (2022), 12-21. -
  19. [21][21] Harold G Koenig and Saad Al Shohaib. 2014. Health and well-being in Islamic societies . Springer. -
  20. [22][22] Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. 2021. Hurdles to progress in long-form question answering. arXiv preprint arXiv:2103.06332 (2021). - · arXiv
  21. [23][23] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics 7 (2019), 453-466. -
  22. [24][24] Carlos Lassance and St´ ephane Clinchant. 2023. The tale of two MS MARCO-and their unfair comparisons. arXiv preprint arXiv:2304.12904 (2023). - · arXiv
  23. [25][25] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461 (2019). - · arXiv
  24. [27][27] Antoine Louis, Gijs van Dijck, and Gerasimos Spanakis. 2023. Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Models. arXiv preprint arXiv:2309.17050 (2023). - · arXiv
  25. [28][28] Rana Malhas, Watheq Mansour, and Tamer Elsayed. 2022. Qur'an QA 2022: Overview of the first shared task on question answering over the holy qur'an. In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and FineGrained Hate Speech Detection . 79-87. -
  26. [29][29] Hajer Maraoui, Kais Haddar, and Laurent Romary. 2021. Arabic factoid Question-Answering system for Islamic sciences using normalized corpora. Procedia Computer Science 192 (2021), 69-79. https://doi.org/10. 1016/j.procs.2021.08.008 Knowledge-Based and Intelligent Information & Engineering Systems: Proceedings of the 25th International Conference KES2021. -
  27. [30][30] Hajer Maraoui, Kais Haddar, and Laurent Romary. 2022. Development of a normalized hadith narrator encyclopedia with TEI. Computaci˝ on y Sistemas 26, 3 (2022), 1283-1292. -
  28. [31][31] Ensaf Hussein Mohamed and Eyad Mohamed Shokry. 2022. QSST: A Quranic Semantic Search Tool based on word embedding. Journal of King Saud University-Computer and Information Sciences 34, 3 (2022), 934945. -
  29. [32][32] Ali Mostafa and Omar Mohamed. 2022. Gof at qur'an qa 2022: Towards an efficient question answering for the holy qu'ran in the arabic language using deep learning-based approach. In Proceedinsg of the 5th Workshop on Open-Source Arabic Corpora and Processing Tools with Shared Tasks on Qur'an QA and Fine-Grained Hate Speech Detection . 104-111. -
  30. [33][33] V´ ıt Novotn` y. 2018. Implementation notes for the soft cosine measure. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management . 1639-1642. -
  31. [34][34] Aarthi Paramasivam and S Jaya Nirmala. 2022. A survey on textual entailment based question answering. Journal of King Saud University-Computer and Information Sciences 34, 10 (2022), 9644-9653. -
  32. [35][35] Syopiansyah Jaya Putra, Ria Hari Gusmita, Khodijah Hulliyah, and Husni Teja Sukmana. 2016. A semantic-based question answering system for Indonesian translation of quran. In Proceedings of the 18th International Conference on Information Integration and Web-based Applications and Services . 504-507. -
  33. [36][36] Yujia Qin, Zihan Cai, Dian Jin, Lan Yan, Shihao Liang, Kunlun Zhu, Yankai Lin, Xu Han, Ning Ding, Huadong Wang, et al. 2023. WebCPM: Interactive Web Search for Chinese Long-form Question Answering. arXiv preprint arXiv:2305.06849 (2023). - · arXiv
  34. [38][38] Jawad Sadek and Farid Meziane. 2016. A discourse-based approach for Arabic question answering. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP) 16, 2 (2016), 1-18. -
  35. [39][39] H Shmeisani, S Tartir, A Al-Na'ssaan, and M Naji. 2014. Semantically answering questions from the Holy Quran. In International Conference on Islamic Applications in Computer Science And Technology . 1-8. -
  36. [40][40] Dan Su, Xiaoguang Li, Jindi Zhang, Lifeng Shang, Xin Jiang, Qun Liu, and Pascale Fung. 2022. Read before generate! faithful long form question answering with machine reading. arXiv preprint arXiv:2203.00343 (2022). - · arXiv
  37. [42][42] FS Utomo, N Suryana, and MS Azmi. 2020. Question Answering Systems on Holy Quran: A Review of Existing Frameworks, Approaches, Algorithms and Research Issues. In Journal of Physics: Conference Series , Vol. 1501. IOP Publishing, 012022. -
  38. [43][43] Shufan Wang, Fangyuan Xu, Laure Thompson, Eunsol Choi, and Mohit Iyyer. 2022. Modeling exemplification in long-form question answering via retrieval. arXiv preprint arXiv:2205.09278 (2022). - · arXiv
  39. [44][44] Kemas Wiharja, Danang Murdiansyah, Muhammad Romdlony, Tiwa Ramdhani, and Muhammad Gandidi. 2022. A Questions Answering System on Hadith Knowledge Graph. Journal of ICT Research and Applications 16, 2 (2022), 184-196. -
  40. [45][45] Hua Yu and Jie Yang. 2001. A direct LDA algorithm for high-dimensional data-with application to face recognition. Pattern recognition 34, 10 (2001), 2067-2070. -
  41. [46][46] Li Yujian and Liu Bo. 2007. A normalized Levenshtein distance metric. IEEE transactions on pattern analysis and machine intelligence 29, 6 (2007), 1091-1095. -
  42. [47][47] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 (2019). - · arXiv
  43. [48][48] Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. 2024. Toolqa: A dataset for llm question answering with external tools. Advances in Neural Information Processing Systems 36 (2024). -
  44. [49][49] Zoya, Seemab Latif, Faisal Shafait, and Rabia Latif. 2021. Analyzing LDA and NMF Topic Models for Urdu Tweets via Automatic Labeling. , 127531127547 pages. https://doi.org/10.1109/ACCESS.2021.3112620 · DOI

Cited by

Sorted by citations

Metadata

Authors
Qamar, F., Latif, S., & Latif, R.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2096-7004 (ISSN-L)
DOI
10.48550/arxiv.2409.09844
arXiv
2409.09844
OpenAlex
W4413372247
Cited by
6
References
0
Citations by year
2026: 1 · 2025: 5
FWCI
2.43
Type
journal article
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • Data Intelligence (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
Prince Sultan University
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
Keywords
non-factoid question answering, Quranic exegesis, hadith, Quran, benchmark dataset, question-answer pairs, contextual information, human evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review