Sacred or Synthetic? Evaluating LLM Reliability and Abstention for Religious Questions

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Previous works often use some form of semantic distance for open-ended answers, and more modern approaches utilize LLM-as-a-judge to evaluate answers.

“For such cases, previous works often use some form of semantic distance (Zhong et al. 2022;Zhang et al. 2019;Yuan, Neubig, and Liu 2021), and more modern approaches utilize LLM-as-a-judge to evaluate answers (Kojima et al. 2022;Bai et al. 2024;Liu et al. 2023).”

• cites Zhong et al. 2022 sourcePartly supported

  The Related Work section (p. 2) describes similarity-based metrics that compare generated text with reference text, and says they are common in NLG evaluation. It does not specifically establish their use for evaluating open-ended answers. The LLM-as-a-judge clause cites other works, not Zhong et al. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites Zhang et al. 2019 sourceNot checked — no full text of the source
• cites Yuan, Neubig, and Liu 2021Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Abstention is an active area of LLM-related research.

“Abstention is an active area of LLM-related research (Kasai et al. 2023;Röttger et al. 2024;Ahdritz et al. 2024;Kim and Thorne 2024;Madhusudhan et al. 2024) and is central to Islamic tradition.”

• cites Kasai et al. 2023 sourceNot checked — no full text of the source
• cites Röttger et al. 2024 sourceSupported
• cites Ahdritz et al. 2024 sourceSupported

May combine two claims.

The FiqhQA dataset consists of 960 question-answer pairs, based on 120 unique questions, each available in both Arabic and English, with four answers corresponding to the four schools of thought.

“In total, the dataset consists of 960 question-answer pairs.These are based on 120 unique questions, each available in both Arabic and English, with four answers corresponding to the four schools of thought (120 questions x 2 languages x 4 schools of thought = 960).”

May combine two claims.

For English, GPT-4o stands out as the top overall performer with 46% of the answers being fully correct and is either ahead for all madhhabs or closely trailing for the top spot (Shafi'i).

“For English, GPT-4o stands out as the top overall performer with 46% of the answers being fully correct and is either ahead for all madhhabs or closely trailing for the top spot (Shafi'i).”

May combine two claims. The quote only partly states this claim.

Fanar displayed a slightly higher abstention rate (25%) than Gemini on English data, but it suffers from a significantly higher error rate of 48%.

“Fanar displayed a slightly higher abstention rate (25%) than Gemini, but it suffers from a sig-”

• Also stated in: § Variability in Question and Answer Interpretation · plain

  “nificantly higher error rate of 48%.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The choice of deterrent clauses (e.g., 'I will be imprisoned and fined') directly positively affects the abstention behavior.

“The choice of deterrent clauses (e.g., “I will be imprisoned and fined”) directly positively affects the abstention behavior.”

May combine two claims.

For English, GPT-4o exhibits no abstention, producing outputs for all inputs, of which approximately 56% are correct and 45% are incorrect.

“For English, GPT-4o exhibits no abstention, producing outputs for all inputs, of which approximately 56% are correct and 45% are incorrect.”

May combine two claims.

When English results are broken down by madhhab, GPT-4o achieves the highest percentage of correct answers for the Hanafi madhhab (56%), followed by the Hanbali madhhab (49%), and the results were much lower for the Shafi'i and Maliki madhhabs (41% and 37% respectively).

“When English results are broken down by madhhab (Table 3), we see that GPT-4o achieves the highest percentage of correct answers for the Hanafi madhhab (56%), followed by the Hanbali madhhab (49%), and the results were much lower for the Shafi'i and Maliki madhhabs (41% and 37% respectively).”

May combine two claims.

For Arabic, correlation is acceptable between annotators and GPT-4o with comparable levels of correlations between humans and the LLM.

“For Arabic, correlation is acceptable between annotators and GPT-4o with comparable levels of correlations between humans and the LLM.”

May combine two claims.

The results reveal a marked disparity in performance among the models and between Arabic and English data.

“The results reveal a marked disparity in performance among the models and between Arabic and English data.”

May combine two claims.

Models like Gemini, GPT-4o, and Fanar show strong promise, but their performance remains uneven across languages and religious traditions.

“While models like Gemini, GPT-4o, and Fanar show strong promise, their performance remains uneven across languages and religious traditions.”

May combine two claims.

Gemini abstained in only 20% of cases and showed a 40% error rate alongside a 45% accuracy rate on English data.

“Gemini abstained in only 20% of cases and showed a 40% error rate alongside a 45% accuracy rate.”

May combine two claims.

Under strict abstention on Arabic, GPT-4o abstained in only 38% of cases, produced a lower rate of correct answers, and had a relatively high error rate of 30%.

“GPT-4o, in comparison, abstained in only 38% of the cases, produced a lower rate of correct answers, and produced a relatively high error rate of 30%.”

The quote only partly states this claim.

GPT-4o had the lowest percentage of fully wrong answers in Arabic.

“GPT-4o had the lowest percentage of fully wrong answers.”

May combine two claims.

In zero-shot experiments, the Hanafi school has the highest correctness rate compared to other schools for GPT-4o, while other models performed more consistently across different schools of thought.

“First, the zero-shot experiments show that the Hanafi school has the highest correctness rate compared to other schools for GPT-4o, while other models performed more consistently across different schools of thought.”

May combine two claims.

Under strict abstention on English, GPT-4o slightly outperformed Gemini in correct answers, with both models yielding comparable error rates.

“GPT-4o outperformed Gemini slightly in terms of correct answers, with both models yielding comparable error rates.”

May combine two claims.

Models like Gemini and Fanar showed more conservative behavior, especially in Arabic, with higher abstention rates and lower error rates.

“In contrast, models like Gemini and Fanar showed more conservative behavior, especially in Arabic, with higher abstention rates and lower error rates.”

May combine two claims.

While Fanar and Gemini 2.0 Flash trailed GPT-4o in the percentage of fully correct answers, Gemini 2.0 Flash had the lowest percentage of completely wrong answers.

“While Fanar and Gemini 2.0 Flash trailed GPT-4o in the percentage of fully correct answers, Gemini 2.0 Flash had the lowest percentage of completely wrong answers.”

May combine two claims.

Gemini has an abstention rate of 90%, with around 9% of responses being correct and 1% being incorrect.

“Notably, Gemini has an abstention rate of 90%, with around 9% of responses being correct and 1% being incorrect”

May combine two claims.

Under strict abstention on Arabic, Fanar outperformed Gemini in answer quality, with a higher proportion of correct responses and a lower error rate.

“Fanar also outperformed Gemini in terms of answer quality, with a higher proportion of correct responses and a lower error rate.”

May combine two claims.

The drop in accuracy from English to Arabic is unlikely to be due to random variation and indicates a cross-language performance disparity.

“confirms that the drop in accuracy from English to Arabic is unlikely to be due to random variation and indicates a cross-language performance disparity.”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLMs often provide additional information beyond the scope of the question.

“Another observed behavior is that LLMs often provide additional information beyond the scope of the question.”

May combine two claims.

For GPT-4o, the fully correct rate falls from 46% in English down to 28% in Arabic, and error rates (Score 1) rise by 2 percentage points.

“For GPT-4o, the fully correct rate falls from 46% in English down to 28% in Arabic, and error rates (Score 1) rise by 2 percentage points.”

May combine two claims.

For Arabic, GPT-4o and Fanar are tied for the top spot (28%), followed closely by Gemini 2.0 Flash.

“For Arabic (Table 4), GPT-4o and Fanar are tied for the top spot (28%), followed closely by Gemini 2.0 Flash.”

May combine two claims.

Fanar performs the best for the Shafi'i madhhab (42%) and the results for the other madhhabs are very close.

“For example, Fanar performs the best for the Shafi'i madhhab (42%) and the results for the other madhhabs are very close.”

May combine two claims.

Under strict abstention on English, Fanar's abstention rate was 4%, with a high error rate of 54% and a correct answer rate of 42%.

“Fanar's abstention rate remained notably low at just 4%, coupled with a high error rate of 54% and a correct answer rate of 42%.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The inclusion criteria for the dataset were developed in consultation with a domain expert.

“The inclusion criteria were developed in consultation with a domain expert.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Many benchmarks use multiple-choice questions for the sake of simplicity.

“Many benchmarks use multiple-choice questions for the sake of simplicity (Koto et al. 2024;Rajpurkar et al. 2016).”

• cites Koto et al. 2024 sourceNot checked — no full text of the source
• cites Rajpurkar et al. 2016 sourceNot checked — no full text of the source

Classified as probably not a substantial claim.

Each LLM output is a sequence of tokens generated based on likelihood, not epistemic certainty.

“Each output is a sequence of tokens generated based on likelihood, not epistemic certainty.”

May combine two claims.

The Kuwaiti Fiqh Encyclopedia is a well-established and authoritative reference extensively covering the legal rulings across the four Sunni madhhabs.

“We chose Kuwaiti Fiqh Encyclopediafoot_1 as the primary source, as it is a well-established and is an authoritative reference extensively covering the legal rulings (ahkam) across the four Sunni madhhabs.”

• cites foot_1Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

The evaluation combined automated and human-driven procedures to evaluate LLM zero-shot QA capabilities and abstention.

“To ensure the rigor and reliability of our evaluation, we combined automated and human-driven procedures to evaluate LLM zero-shot QA capabilities and abstention.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

AI may be limited in replicating intention or exertion, which are key factors in the ethical evaluation of juristic reasoning in Sunni tradition.

“While AI can imitate outputs, it may be limited in replicating intention or exertion, which are key factors in the ethical evaluation of juristic reasoning in Sunni tradition.”

Classified as probably not a substantial claim. May combine two claims.

LLMs are probabilistic language models, not knowledge-grounded reasoners.

“This underscores a fundamental limitation of LLMs: they are probabilistic language models, not knowledgegrounded reasoners.”

May combine two claims.

Gemini's abstention strategy is more conservative and reliable.

“indicating a more conservative and reliable abstention strategy.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In some cases, the LLM's understanding of the question differs from the expected interpretation, though the response may remain plausible within the domain.

“In some cases, the LLM's understanding of the question differs from the expected interpretation, though the response may remain plausible within the domain.”

Probably not a substantial claim: may not pass S4 (based).

This ensured that the LLM-judge's assessments aligned with expert judgments.

“This ensured that the LLM-judge's assessments aligned with expert judgments.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The formulation of the answer itself impacts the rating.

“Moreover, the formulation of the answer itself impacts the rating.”

Classified as probably not a substantial claim. May combine two claims.

The encyclopedia is systematically organized by topic and terminology, and has a clear presentation of the differing positions among the schools, making it particularly well suited for dataset construction.

“The encyclopedia is systematically organized by topic and terminology, and has a clear presentation of the differing positions among the schools, making it particularly well suited for dataset construction.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Categories of sincerity and effort can apply to the engineer or researcher developing AI systems for religious use.

“Given the importance of sincerity and effort, these categories can apply to the engineer or researcher developing AI systems for religious use.”

Probably not a substantial claim: may not pass S3 (load-bearing).

According to prophetic narrations, a jurist who errs is still rewarded if a sincere effort was made.

“According to prophetic narrations, a jurist who errs is still rewarded if a sincere effort was madefoot_5 .”

• cites foot_5Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

This discrepancy can complicate the evaluation process.

“This discrepancy can complicate the evaluation process.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Translations of both questions and answers were manually checked and corrected where necessary to maintain clarity and precision across languages.

“Translations of both questions and answers were also manually checked and corrected, where necessary, to maintain clarity and precision across languages.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The complexity of terminology presents a challenge even for human scholars, and it was not explicitly accounted for in the evaluation prompts.

“This complexity presents a challenge even for human scholars, and it was not explicitly accounted for in the evaluation prompts.”

• AssumptionRests on an assumption: Each madhhab categorizes human actions using terms such as fard, wajib, sunnah, mandub, mustahab, makrooh, and others.
• AssumptionRests on an assumption: These terms often have both specific and general meanings, and the same term can signify varying degrees of obligation or recommendation across different schools.
• AssumptionRests on an assumption: Wajib in the Hanafi school has a specific juristic meaning, while being equivalent to fard in other schools.

Classified as probably not a substantial claim.

By introducing the FiqhQA dataset and incorporating both correctness and abstention into our evaluation, we offer a more nuanced view of model reliability in high-stakes religious contexts.

“By introducing the FiqhQA dataset and incorporating both correctness and abstention into our evaluation, we offer a more nuanced view of model reliability in high-stakes religious contexts.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The disparity in cross-madhhab accuracy may be the result of pre-training data construction, where Fanar seems to follow a more equitable distribution of material from different madhhabs, while GPT-4o's pre-training might be skewed towards the Hanafi madhhab, which appears to have the most data of all madhhabs available in the web.

“We think that the disparity in cross-madhhab accuracy may be the result of pre-training data construction, where Fanar seems to follow a more equitable distribution of material from different madhhabs, while GPT-4o's pre-training might be skewed towards the Hanafi madhhab, which appears to have the most data of all madhhabs available in the web.”

Classified as probably not a substantial claim.

All generated content for the dataset was reviewed and validated by native Arabic speakers.

“Since the original source is in Arabic, all generated content was reviewed and validated by native Arabic speakers.”

Classified as probably not a substantial claim. May combine two claims.

Krippendorff's alpha values ≥ 0.8 indicate highly reliable correlation and values between 0.8 and 0.667 are sufficient to draw tentative conclusions.

“Typically alpha values ≥ 0.8 indicate highly reliable correlation and values between 0.8 and 0.667 are sufficient to draw tentative conclusionsfoot_4 .”

• cites foot_4Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

When an AI system delivers an incorrect response, it is difficult to assign responsibility.

“When an AI system delivers an incorrect response, it is difficult to assign responsibility.”

• AssumptionRests on an assumption: Currently, most models offer little to no transparency about how responses are generated.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Providing extra information mirrors practical fatwa-giving behavior but introduces complexity in evaluation.

“While this mirrors practical fatwa-giving behavior, it introduces complexity in evaluation.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The research team identified and corrected several entries where inconsistencies or omissions were observed.

“The research team, being most familiar with the Hanafi school, identified and corrected several entries where inconsistencies or omissions were observed.”

May combine two claims.

The drop in accuracy from English to Arabic is unlikely to be due to random variation and indicates a cross-language performance gap.

“This confirms that the drop in accuracy from English to Arabic is unlikely to be due to random variation and indicates a cross-language performance”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

The dataset is not without imperfections.

“The dataset is not without imperfections.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Certain nuances were omitted in favor of a simplified evaluation scheme.

“Consequently, certain nuances were omitted in favor of a simplified evaluation scheme.”

• AssumptionRests on an assumption: Each madhhab categorizes human actions using terms such as fard, wajib, sunnah, mandub, mustahab, makrooh, and others.
• AssumptionRests on an assumption: These terms often have both specific and general meanings, and the same term can signify varying degrees of obligation or recommendation across different schools.
• AssumptionRests on an assumption: Wajib in the Hanafi school has a specific juristic meaning, while being equivalent to fard in other schools.

Probably not a substantial claim: may not pass S4 (based).

This study presents the first benchmark specifically evaluating LLM performance in answering Islamic questions in the four major Sunni schools of thought.

“This study presents the first benchmark specifically evaluating LLM performance in answering Islamic questions in the four major Sunni schools of thought.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

When using an LLM as a judge, it is important to iterate and systematically tune the prompts, as the phrasing will influence the judgment consistency.

“When using an LLM as a judge, it is important to iterate and systematically tune the prompts, as the phrasing will influence the judgment consistency.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

For the English version of the dataset, the Arabic question-answer pairs were translated using GPT-4o, with subsequent manual verification by human annotators to ensure accuracy and fidelity.

“For the English version of the dataset, the Arabic question-answer pairs were translated using GPT-4o, with subsequent manual verification by human annotators to ensure accuracy and fidelity.”

Classified as probably not a substantial claim.

We randomly sampled 10% of the zero-shot outputs in each thematic category (i.e. 12 questions per madhhab for both Arabic and English -96 in total) for manual verification by two human annotators for Arabic outputs and two human annotators for English.

“We randomly sampled 10% of the zero-shot outputs in each thematic category (i.e. 12 questions per madhhab for both Arabic and English -96 in total) for manual verification by two human annotators for Arabic outputs and two human annotators for English.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Identifying gaps in the query and requesting further clarification requires capabilities in dialog management, contextual reasoning, and even empathy.

“This requires capabilities in dialog management, contextual reasoning, and even empathy.”

• AssumptionRests on an assumption: Valid religious reasoning requires a deep understanding of the question and its context.
• AssumptionRests on an assumption: In complex areas such as mu'amalat, abstaining from answering is insufficient.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Realizing such a system may demand advances beyond current LLM capabilities.

“However, realizing such a system may demand advances beyond current LLM capabilities.”

May combine two claims.

GPT-4o offers strong accuracy but is less likely to abstain when uncertain, resulting in higher rates of confidently incorrect answers.

“Third, the abstention experiments demonstrated that while GPT-4o offers strong accuracy, it is less likely to abstain when uncertain, resulting in higher rates of confidently incorrect answers.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Extraction and transformation of the Kuwaiti Fiqh Encyclopedia's rulings into a structured QA format required editorial interventions by the research team.

“Although the Kuwaiti Fiqh Encyclopedia provides a comprehensive reference, the extraction and transformation of its rulings into a structured QA format required editorial interventions by the research team.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The study followed a systematic methodology in the construction and evaluation of the dataset.

“Despite following a systematic methodology in the construction and evaluation of the dataset, several limitations warrant consideration.”

Probably not a substantial claim: may not pass S2 (checkable).

Our findings support a cautious and ethically grounded approach to deploying LLMs in religious contexts.

“Our findings support a cautious and ethically grounded approach to deploying LLMs in religious contexts.”

• AssumptionRests on an assumption: In a religious consultation, the goal is to help the questioner attain benefit and avoid harm, which is the central aim of Islamic Law.
• AssumptionRests on an assumption: Valid religious reasoning requires a deep understanding of the question and its context.
• AssumptionRests on an assumption: In complex areas such as mu'amalat, abstaining from answering is insufficient.
• AssumptionRests on an assumption: Currently, most models offer little to no transparency about how responses are generated.
• AssumptionRests on an assumption: Techniques such as Retrieval-Augmented Generation (RAG) and chain-of-thought prompting provide partial visibility into the reasoning process and sources.
• AssumptionRests on an assumption: Full interpretability remains a largely unsolved problem.

Probably not a substantial claim: may not pass S3 (load-bearing).

The open-ended nature of the questions results in variability in how the LLM interprets and answers them.

“The open-ended nature of the questions results in variability in how the LLM interprets and answers them.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

The Arabic language and Islamic jurisprudence are structured and rule-governed, making them fitting for computational modeling.

“Given the structured and rule-governed nature of both the Arabic language and Islamic jurisprudence, these domains appear fitting for computational modeling.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Some original formulations were ambiguous or open to interpretation, requiring clarification before inclusion in the dataset.

“In some cases, the original formulations were ambiguous or open to interpretation, requiring clarification before inclusion in the dataset.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

Accurately representing the rulings of a single madhhab is already a complex task; extending this effort to cover the four Sunni madhhabs substantially increases the complexity.

“Accurately representing the rulings of a single madhhab is already a complex task; extending this effort to cover the four Sunni madhhabs substantially increases the complexity.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In our framework, providing extra relevant information may result in a rating of 2 (partial correctness).

“In our framework, providing extra relevant information may result in a rating of 2 (partial correctness), although it may be argued that such comprehensive answers should be rated as fully correct.”