CoChat
Sign Up
Cited by 10Multi-Criteria Decision MakingEthics and Social Impacts of AIExplainable Artificial Intelligence (XAI)

Trustworthiness Evaluation of Large Language Models Using Multi-Criteria Decision Making

Open access
Aksoy, Meltem, Adem, Aylin, Dağdeviren, Metin·2025·IEEE Access
DOI 10.1109/access.2025.3612568
Open this paper in CoChat — free account

Abstract

As Large language models (LLMs) become increasingly integrated into high-stakes applications, ensuring their trustworthiness has emerged as a critical research concern. This study proposes a novel evaluation framework that applies a multi-criteria decision making (MCDM) methodology, specifically the hesitant fuzzy analytic hierarchy process (AHP), to assess and rank LLMs based on five key trust dimensions: fairness, robustness, integrity, explainability, and safety. Drawing from expert evaluations, the framework systematically determines the relative importance of each criterion and applies a weighted scoring approach to compare seven leading LLMs, including both proprietary models such as GPT-3.5, GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 and open-source models such as Llama 3.1, Mistral Large 2 and DeepSeek V3. Results reveal GPT-4o as the most trustworthy model, significantly outperforming its peers, particularly in robustness and fairness. Open-source models showed lower scores, especially in safety and explainability, highlighting persistent gaps in their alignment with trust expectations. The findings demonstrate the effectiveness of MCDM in capturing expert uncertainty and prioritizing trust criteria, offering a robust and adaptable framework for evaluating LLMs in dynamic and sensitive domains.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper proposes a hesitant fuzzy AHP multi-criteria decision-making framework to evaluate LLM trustworthiness across fairness, robustness, integrity, explainability, and safety. Using seven experts and seven LLMs, it ranks GPT-4o as the most trustworthy model, with open-source models generally scoring lower, especially on safety and explainability.

Key findings

  • GPT-4o (A1) had the highest weighted trustworthiness score at 6.77, followed by GPT-3.5 (A2) at 5.82, Gemini 1.5 (A5) at 5.63, Llama 3.1 (A3) at 5.38, Claude 3.5 Sonnet (A4) at 5.30, DeepSeek V3 (A7) at 4.99, and Mistral Large 2 (A6) at 4.94.
  • Experts weighted fairness highest (0.25), then robustness (0.21), integrity (0.19), safety (0.18), and explainability lowest (0.16).
  • Safety scores were low across models, with DeepSeek V3 at 3.00 and Mistral Large 2 at 3.71; explainability scores ranged from 4.29 to 5.29, and robustness scores ranged from 5.57 to 8.14.
  • In sensitivity analysis across 11 scenarios, GPT-4o consistently ranked best, while Mistral Large 2 was generally among the lowest.

Methods

  • The study used a two-stage hesitant fuzzy AHP approach: first, seven experts independently evaluated five FRIES-based trust criteria using a hesitant fuzzy linguistic term set; second, the same experts scored seven LLM alternatives on a 1–10 scale.
  • Criteria weights were derived from pairwise comparisons via hesitant fuzzy AHP, while alternative evaluations were aggregated using arithmetic means and combined into weighted total scores.
  • Sensitivity analysis tested 11 scenarios, including dominant-criterion weights of 0.99 and 0.90 and equal weights.

Limitations

  • The paper states that the method was applied to a single decision problem and only seven LLMs, limiting generalizability; it also used only five FRIES-based dimensions, excluding privacy, transparency, and accountability as separate criteria, and did not tailor criteria or weights to domain-specific contexts such as healthcare or law.
  • The paper states that the analysis relied on a single expert group and expert-based judgments without objective task-based metrics or user-centered feedback, and future work could report inter-rater reliability and involve more diverse experts.
  • Evident from the text: despite mitigation through independent assessments and arithmetic averaging, the small expert panel and lack of reported inter-rater reliability leave potential subjectivity and limited perspective diversity.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/access.2025.3612568

Cites

51 of 51 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

9 of 34 cited sources checked · 0 supported · 1 flagged · accuracy not yet measured

  • Premisecitation
    § C. RESEARCH GAP · p. 3

    Large-scale benchmarks such as TRUSTLLM, HELM, DecodingTrust, and XTRUST primarily rely on performance-based task evaluations, focusing on metrics such as fairness, robustness, or toxicity in isolation.

    “Large-scale benchmarks such as TRUSTLLM [26], HELM [27], DecodingTrust [28], and XTRUST [30] have systematically assessed models across multiple tasks and dimensions. However, these frameworks primarily rely on performance-based task evaluations, focusing on metrics such as fairness, robustness, or toxicity in isolation.”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites [26] Sun, TrustLLM: Trustworthiness in large language models

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [27] Liang, 2022, Holistic evaluation of language models, arXiv:2211.09110 source

      The Multi-metric measurement section says HELM evaluates seven metrics for each of its 16 core scenarios, rather than treating fairness, robustness, and toxicity in isolation. The claim's characterization of HELM as focusing on those metrics in isolation is contrary to the paper's stated approach. (LLM, conf=0.88)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
    • cites [28] Wang, 2023, DecodingTrust: A comprehensive assessment of trustworthiness in GPT models, arXiv:2306.11698 source

      The Introduction describes DecodingTrust as a task- and metric-based evaluation across eight trustworthiness perspectives, and Table 57 (p. 102) reports separate scores for toxicity, robustness, and fairness. That supports the characterization of its evaluations, but the paper presents them as a comprehensive assessment across perspectives; it does not establish that the metrics are considered only in isolation. (LLM, conf=0.79)

      Experimental — accuracy not yet measured
  • Premisecitation
    § C. RESEARCH GAP · p. 3

    MCDM-based applications have not been applied with a primary focus on systematically assessing the trustworthiness of LLMs.

    “However, they have not been applied with a primary focus on systematically assessing the trustworthiness of LLMs.”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites [32] source
    • cites [33] Zheng, 2025, LLM-as-a-fuzzy-judge: Fine-tuning large language models as a clinical evaluation judge with fuzzy logic, arXiv:2506.11221 source

      The Introduction (p. 2) discusses fuzzy evaluation criteria for clinical interactions, and the Discussion (p. 10) reports how the authors' clinical-evaluation model performs. Neither passage addresses whether MCDM-based applications have primarily focused on systematically assessing LLM trustworthiness. Searches for MCDM and that specific trustworthiness focus found no passage establishing the claim. (LLM, conf=0.73)

      Experimental — accuracy not yet measured
    • cites [34] Svoboda, 2024, Enhancing multi-criteria decision analysis with AI: Integrating analytic hierarchy process and GPT-4 for automated decision support, arXiv:2402.07404 source

      The Introduction (page unavailable) identifies a different research gap: limited work integrating AHP with LLMs for decision support. The Abstract (page unavailable) describes a cybersecurity MCDA application, but the copy does not address whether MCDM-based applications have systematically assessed LLM trustworthiness. Searches for trustworthiness and LLM reliability assessment found no passage establishing that broader claim. (LLM, conf=0.73)

      Experimental — accuracy not yet measured
  • Premisecitation
    § V. DISCUSSION · p. 10

    Large-scale benchmarks such as TRUSTLLM, DecodingTrust, and XTRUST highlight fairness as one of the most critical and frequently evaluated dimensions in trustworthiness assessments of LLMs.

    “Moreover, large-scale benchmarks such as TRUSTLLM [26] and DecodingTrust [28], XTRUST [30] highlight fairness as one of the most critical and frequently evaluated dimensions in trustworthiness assessments of LLMs.”
    Quote in the paperNot a substantial claim2 of 3 sources checked
    • cites [26] Sun, TrustLLM: Trustworthiness in large language models

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [28] Wang, 2023, DecodingTrust: A comprehensive assessment of trustworthiness in GPT models, arXiv:2306.11698 source

      DecodingTrust includes fairness among its eight trustworthiness perspectives and devotes Section 10 (p. 53) to evaluating it. The paper supports fairness as an evaluated dimension, but does not establish that it is one of the most critical or most frequently evaluated dimensions across LLM trustworthiness benchmarks. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
    • cites [30] Li, 2024, XTRUST: On the multilingual trustworthiness of large language models, arXiv:2409.15762 source

      Sections 3.1 and 4.8 identify fairness and bias as an XTRUST trustworthiness category and describe its evaluation; §4.8 also says the evaluation follows DecodingTrust’s approach. The paper does not establish that fairness is among the most critical or most frequently evaluated dimensions across LLM trustworthiness benchmarks. Page numbers are unavailable in our copy. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
  • Premisecitation
    § V. DISCUSSION · p. 10

    User-centered studies empirically identified perceived credibility, neutrality, and bias minimization as key determinants of trust in LLMs.

    “Additionally, usercentered studies like [39] empirically identified perceived credibility, neutrality, and bias minimization as key determinants of trust in LLMs.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [39] source

      The Discussion (section 4; page unavailable) identifies perceived credibility as the highest-weighted trust determinant (0.049). The Research Methodology (section 2) includes neutrality and bias minimization among the factors evaluated by 15 office employees, but the ranked top-ten results do not identify it as a key determinant. The claim therefore overstates that part of the empirical finding and leaves out the study’s office-employee setting. (LLM, conf=0.90)

      Experimental — accuracy not yet measured
  • Findingown data
    § MELTEM AKSOY 1 , AYLIN ADEM 2 , AND METIN DAĞDEVIREN 2,3 · p. 1

    Among the seven evaluated LLMs, GPT-4o is the most trustworthy model.

    “Results reveal GPT-4o as the most trustworthy model, significantly outperforming its peers, particularly in robustness and fairness.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 39
  • May combine two claims. 24
  • The quote only partly states this claim. 2

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation
    § I. INTRODUCTION · p. 2

    MCDM allows for systematic aggregation of expert opinions through pairwise comparisons and assigns weights to criteria reflecting their relative importance.

    “MCDM allows for the systematic aggregation of expert opinions through pairwise comparisons and assigns weights to criteria, reflecting their relative importance.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [16] source

      The Introduction (page not provided) describes MCDM as involving the elicitation of individual preferences and the aggregation of multiple characteristics. It does not say that MCDM systematically aggregates expert opinions through pairwise comparisons or assigns criteria weights reflecting relative importance; searches for those terms found no supporting passage. (LLM, conf=0.85)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § V. DISCUSSION · p. 10

    Studies report that while explainability is recognized as an important trust dimension, it often receives less emphasis in practical evaluations due to the lack of standardized metrics and the difficulty in producing human-interpretable explanations from large-scale neural models.

    “Studies [26], [27] report that while explainability is recognized as an important trust dimension, it often receives less emphasis in practical evaluations due to the lack of standardized metrics and the difficulty in producing human-interpretable explanations from large-scale neural models.”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites [26] Sun, TrustLLM: Trustworthiness in large language models

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [27] Liang, 2022, Holistic evaluation of language models, arXiv:2211.09110 source

      Split into 2 atoms — 0 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 2 unverifiable. Expand the claim to see per-atom verdicts.

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data
    § C. EVALUATION OF LLMS BASED ON TRUSTWORTHINESS · p. 8

    In the criteria weighting phase, experts ranked fairness as the most important criterion and explainability as the least important.

    “The results of the criteria weighting phase in this study show that the experts ranked fairness as the most important criterion, while they considered explainability the least important.”
    • Also stated in: § V. DISCUSSION · p. 10 · plain
      “The results from the criteria weighting phase reveal that the experts prioritized fairness the most, while they placed the least emphasis on explainability.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § C. EVALUATION OF LLMS BASED ON TRUSTWORTHINESS · p. 8

    Llama 3.1 (A3) is the most trustworthy open-source model with a weighted trustworthiness score of 5.38.

    “In the open-source category, Llama 3.1 (A3) emerges as the most trustworthy with a score of 5.38.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § MELTEM AKSOY 1 , AYLIN ADEM 2 , AND METIN DAĞDEVIREN 2,3 · p. 1

    The proposed framework is robust and adaptable for evaluating LLMs in dynamic and sensitive domains.

    “The findings demonstrate the effectiveness of MCDM in capturing expert uncertainty and prioritizing trust criteria, offering a robust and adaptable framework for evaluating LLMs in dynamic and sensitive domains.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The relative importance of trust criteria is stable across high-stakes domains.

  • May combine two claims.

    Interpretationreasoning
    § C. RESEARCH GAP · p. 3

    The study addresses the gap by applying a hesitant fuzzy AHP approach that explicitly models expert hesitation, incorporates the relative importance of trust dimensions, and provides interpretable comparative rankings of leading LLMs.

    “Our study addresses this gap by applying a hesitant fuzzy AHP approach, which explicitly models expert hesitation, incorporates the relative importance of trust dimensions, and provides interpretable comparative rankings of leading LLMs, thereby combining technical and ethical considerations into a structured evaluation process.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductcitation
    § III. METHODOLOGY · p. 5

    The arithmetic mean method was chosen to aggregate experts' linguistic evaluations to prevent data loss.

    “To prevent data loss, we choose the arithmetic mean method to aggregate the experts' linguistic evaluations [53].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [53] source

  • May combine two claims.

    Interpretationreasoning
    § I. INTRODUCTION · p. 2

    Unlike FRIES, MCDM generates comparative rankings of alternatives, enabling clearer and more actionable evaluations of trustworthiness.

    “Unlike FRIES, MCDM generates comparative rankings of alternatives, enabling clearer and more actionable evaluations of trustworthiness.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation
    § I. INTRODUCTION · p. 2

    The FRIES trust score framework defines five trust dimensions (fairness, robustness, integrity, explainability, and safety) based on an extensive literature review.

    “This study is inspired by the FRIES trust score framework proposed by [15], which evaluates the trustworthiness of machine learning models and datasets. The framework defines five trust dimensions (fairness, robustness, integrity, explainability, and safety) based on an extensive literature review.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [15] source

  • May combine two claims.

    Premisecitation
    § I. INTRODUCTION · p. 2

    The hesitant fuzzy AHP approach offers greater flexibility by accommodating expert hesitation in comparative evaluations and utilizing linguistic expressions instead of rigid numerical scales.

    “This approach offers greater flexibility by accommodating expert hesitation in comparative evaluations and utilizing linguistic expressions instead of rigid numerical scales [17], [22], [23], [24].”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [17] source
    • cites [22] source
    • cites [23] source

  • May combine two claims.

    Premisecitation
    § B. CRITERIA · p. 7

    The five criteria (fairness, robustness, integrity, explainability, safety) reflect trustworthiness aspects of LLMs highlighted in recent studies, emphasizing their relevance and growing consensus.

    “These criteria reflect trustworthiness aspects of LLMs highlighted in recent studies [26], [63], emphasizing their relevance and the growing consensus on these standards within the field.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [26] Sun, TrustLLM: Trustworthiness in large language models

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [63] Liu, 2023, Trustworthy LLMs: A survey and guideline for evaluating large language models’ alignment, arXiv:2308.05374 source

  • May combine two claims.

    Premisereasoning
    § I. INTRODUCTION · p. 2

    The FRIES framework's methodology, inspired by FMEA, relies on averaging individual scores, which fails to capture nuanced expert perspectives or account for the relative importance of different criteria.

    “However, while the FRIES framework provides a solid foundation, its methodology, inspired by failure mode and effects analysis (FMEA), has notable limitations. It relies on averaging individual scores, which fails to capture the nuanced perspectives of experts or account for the relative importance of different criteria.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [15] source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning
    § III. METHODOLOGY · p. 4

    No formal group discussion or consensus-building method was employed to avoid groupthink and ensure independent judgments.

    “To avoid groupthink and ensure independent judgments, no formal group discussion or consensus-building method (e.g., Delphi technique) was employed. Instead, each expert independently evaluated the alternatives and criteria using the predetermined hesitant fuzzy linguistic term set.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning
    § III. METHODOLOGY · p. 4

    Each expert independently evaluated the alternatives and criteria using the predetermined hesitant fuzzy linguistic term set.

    “To avoid groupthink and ensure independent judgments, no formal group discussion or consensus-building method (e.g., Delphi technique) was employed. Instead, each expert independently evaluated the alternatives and criteria using the predetermined hesitant fuzzy linguistic term set.”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Interpretationown data
    § MELTEM AKSOY 1 , AYLIN ADEM 2 , AND METIN DAĞDEVIREN 2,3 · p. 1

    The MCDM methodology, specifically hesitant fuzzy AHP, is effective in capturing expert uncertainty and prioritizing trust criteria.

    “The findings demonstrate the effectiveness of MCDM in capturing expert uncertainty and prioritizing trust criteria, offering a robust and adaptable framework for evaluating LLMs in dynamic and sensitive domains.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The selected experts' evaluations are reliable and unbiased enough to derive valid criteria weights.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning
    § I. INTRODUCTION · p. 2

    The five FRIES trust dimensions (fairness, robustness, integrity, explainability, and safety) are highly relevant and applicable for assessing the trustworthiness of LLMs.

    “These dimensions are also highly relevant and applicable for assessing the trustworthiness of LLMs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning
    § V. DISCUSSION · p. 11

    The observed performance gap between proprietary and open-source models can be better understood in light of their underlying training methodologies and deployment strategies.

    “The observed performance gap between proprietary and open-source models can be better understood in light of their underlying training methodologies and deployment strategies.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Proprietary models such as GPT-4o, Claude 3.5, and Gemini 1.5 are typically developed with access to significantly larger and more diverse training corpora, extensive compute resources, and rigorous post-training alignment pipelines.
    • Rests on an assumption: Open-source models like Llama 3.1 and Mistral Large 2 are often constrained by more limited compute budgets and smaller-scale or less diverse alignment data.

  • May combine two claims.

    Study design and conductreasoning
    § VI. LIMITATIONS AND FUTURE STUDIES · p. 17

    The evaluation primarily relied on expert-based judgments, without integrating objective task-based metrics or user-centered feedback, potentially limiting the comprehensiveness of the assessment.

    “The evaluation primarily relied on expert-based judgments, without integrating objective task-based metrics or user-centered feedback, potentially limiting the comprehensiveness of the assessment.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. The quote only partly states this claim.

    Interpretationown data
    § TABLE 10. Enveloped matrices. · p. 13

    The persistently low safety and explainability scores signal an industry-wide challenge that has yet to be fully addressed.

    “This pattern signals an industry-wide challenge that has yet to be fully addressed.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Study design and conductnone given
    § VI. LIMITATIONS AND FUTURE STUDIES · p. 16

    The evaluation was conducted by experts with the necessary technical and contextual knowledge.

    “Thus, the evaluation was conducted by experts with the necessary technical and contextual knowledge.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning
    § VI. LIMITATIONS AND FUTURE STUDIES · p. 13

    The multi-criteria nature of the problem, the use of expert opinion-based data, and its grounding in a real-world scenario demonstrate that the method is an effective tool for such decision-making contexts.

    “The multi-criteria nature of the problem, the use of expert opinion-based data, and its grounding in a real-world scenario demonstrate that the method is an effective tool for such decision-making contexts.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given
    § VII. CONCLUSION · p. 17

    The integration of a technologically grounded method using hesitant fuzzy linguistic term sets based on word calculation for determining the importance of trust-related evaluation criteria, as judged by multiple decision-makers, has yet to be explored in the existing literature.

    “the integration of a technologically grounded method-utilizing hesitant fuzzy linguistic term sets based on word calculation-for determining the importance of trust-related evaluation criteria, particularly as judged by multiple decision-makers, has yet to be explored in the existing literature.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationnone given
    § VII. CONCLUSION · p. 17

    This study highlights the growing importance of trust in the evaluation of LLMs, especially as their presence expands across various domains.

    “This study highlights the growing importance of trust in the evaluation of LLMs, especially as their presence expands across various domains.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning
    § V. DISCUSSION · p. 11

    Proprietary models' investments in larger training corpora, compute resources, and alignment pipelines enable stronger safeguards in terms of safety, robustness, and fairness.

    “These investments enable stronger safeguards in terms of safety, robustness, and fairness.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Proprietary models such as GPT-4o, Claude 3.5, and Gemini 1.5 are typically developed with access to significantly larger and more diverse training corpora, extensive compute resources, and rigorous post-training alignment pipelines.

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given
    § VII. CONCLUSION · p. 17

    No prior research has employed MCDM techniques to evaluate LLMs specifically from the perspective of trustworthiness.

    “no prior research has employed MCDM techniques to evaluate these models specifically from the perspective of trustworthiness.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given
    § VI. LIMITATIONS AND FUTURE STUDIES · p. 13

    Applying the framework to a general trustworthiness evaluation of LLMs, without tailoring criteria or weights to domain-specific contexts, may reduce its applicability in highly specialized areas such as healthcare or law.

    “The framework was applied to a general trustworthiness evaluation of LLMs, without tailoring criteria or weights to domain-specific contexts, which may reduce its applicability in highly specialized areas such as healthcare or law.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given
    § V. DISCUSSION · p. 10

    The evaluation metrics were defined in a clear and measurable manner, and the evaluation process was standardized.

    “First, the evaluation metrics were defined in a clear and measurable manner, and the evaluation process was standardized.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data
    § V. DISCUSSION · p. 11

    Technical complexity alone is insufficient for fostering trust; comprehensive safeguards and maturity in deployment processes play a crucial role.

    “This outcome suggests that technical complexity alone is insufficient for fostering trust; rather, comprehensive safeguards and maturity in deployment processes play a crucial role.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given
    § V. DISCUSSION · p. 10

    Experts were asked to make independent and individual assessments, and joint decision-making processes were deliberately avoided.

    “Additionally, experts were asked to make independent and individual assessments, and joint decision-making processes were deliberately avoided.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning
    § TABLE 10. Enveloped matrices. · p. 13

    The MCDM-based framework enables informed selection among alternatives and supports the development of models that align with both technical excellence and societal values.

    “It enables informed selection among alternatives and supports the development of models that align with both technical excellence and societal values.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § C. EVALUATION OF LLMS BASED ON TRUSTWORTHINESS · p. 7

    Each expert independently evaluated the trustworthiness criteria using a hesitant fuzzy linguistic term set.

    “Each expert independently evaluated the trustworthiness criteria using a hesitant fuzzy linguistic term set.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationcitation
    § B. CRITERIA · p. 7

    Focusing on the established five-criteria set follows prior work and ensures that the evaluation framework remains tractable for expert judgment while still covering essential aspects of trustworthiness.

    “Focusing on this established five-criteria set not only follows prior work [15] but also ensures that the evaluation framework remains tractable for expert judgment while still covering the essential aspects of trustworthiness.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [15] source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data
    § C. EVALUATION OF LLMS BASED ON TRUSTWORTHINESS · p. 7

    The expert team included a balanced mix in terms of expertise domains, career stages, and gender to ensure diversity and minimize potential bias.

    “To ensure diversity and minimize potential bias, the team included a balanced mix in terms of expertise domains, career stages, and gender (four male and three female experts).”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given
    § VI. LIMITATIONS AND FUTURE STUDIES · p. 16

    To minimize subjectivity and bias, the evaluation metrics were clearly defined, the process was standardized, and experts provided independent assessments without group influence.

    “To minimize subjectivity and bias, the evaluation metrics were clearly defined, the process was standardized, and experts provided independent assessments without group influence.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationown data
    § VII. CONCLUSION · p. 17

    Cultivating user trust in an LLM is influenced not only by technical performance but also by the model's maturity and the amount of time it has spent within the product life cycle.

    “This outcome may suggest that cultivating user trust in an LLM is influenced not only by technical performance but also by the model's maturity and the amount of time it has spent within the product life cycle.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § V. DISCUSSION · p. 11

    Open-source models' limited compute budgets and smaller-scale or less diverse alignment data can impact their safety and explainability scores.

    “In contrast, open-source models like Llama 3.1 and Mistral Large 2, while technically competitive in architecture, are often constrained by more limited compute budgets and smaller-scale or less diverse alignment data, which can impact their safety and explainability scores.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Open-source models like Llama 3.1 and Mistral Large 2 are often constrained by more limited compute budgets and smaller-scale or less diverse alignment data.

  • May combine two claims.

    Interpretationown data
    § C. EVALUATION OF LLMS BASED ON TRUSTWORTHINESS · p. 8

    Current LLMs continue to struggle with mitigating risks, preventing harmful outputs, and ensuring responsible deployment.

    “Current LLMs continue to struggle with mitigating risks, preventing harmful outputs, and ensuring responsible deployment.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Interpretationreasoning
    § V. DISCUSSION · p. 10

    The relatively lower weight given to explainability may reflect either a perceived maturity of this dimension or an ongoing challenge in making LLMs truly interpretable.

    “On the other hand, the relatively lower weight given to explainability may reflect either a perceived maturity of this dimension or, conversely, an ongoing challenge in making LLMs truly interpretable, which might discourage experts from expecting high standards in this area.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § TABLE 10. Enveloped matrices. · p. 13

    Architectural sophistication alone is insufficient for LLM trustworthiness without equally mature training and alignment practices.

    “that architectural sophistication alone is insufficient without equally mature training and alignment practices.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning
    § V. DISCUSSION · p. 10

    The experts' independent evaluation results were aggregated using arithmetic averages to reduce the effect of individual differences.

    “The experts' independent evaluation results were then aggregated using arithmetic averages to reduce the effect of individual differences.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning
    § VI. LIMITATIONS AND FUTURE STUDIES · p. 13

    The MCDM method is generally applicable to different decision problems, based on its structural flexibility and the diversity of its applications in previous studies.

    “our statement that the method is generally applicable to different decision problems is based on its structural flexibility and the diversity of its applications in previous studies.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning
    § VI. LIMITATIONS AND FUTURE STUDIES · p. 13

    The analysis relied on a single expert group.

    “The analysis relied on a single expert group.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationreasoning
    § V. DISCUSSION · p. 10

    The proposed approach addresses key limitations in existing trust evaluation frameworks, such as the inability to systematically incorporate expert uncertainty and the lack of nuanced weighting for trust dimensions.

    “The proposed approach addresses the key limitations found in existing trust evaluation frameworks, such as the inability to systematically incorporate expert uncertainty and the lack of nuanced weighting for trust dimensions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § V. DISCUSSION · p. 10

    The proposed approach can consider multiple trust-related criteria simultaneously and consolidate them into a single overall score, allowing for a clear and interpretable ranking of alternatives.

    “A key strength of the proposed approach is its ability to consider multiple trust-related criteria simultaneously and consolidate them into a single overall score, allowing for a clear and interpretable ranking of alternatives.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § VI. LIMITATIONS AND FUTURE STUDIES · p. 13

    The evaluation was restricted to seven LLMs.

    “The evaluation was restricted to seven LLMs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § VI. LIMITATIONS AND FUTURE STUDIES · p. 13

    The study adopted only the five FRIES-based dimensions (fairness, robustness, integrity, explainability, and safety).

    “The study adopted only the five FRIES-based dimensions (fairness, robustness, integrity, explainability, and safety).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § VI. LIMITATIONS AND FUTURE STUDIES · p. 13

    The limited scope of seven evaluated LLMs may reduce the generalizability of the findings.

    “the limited scope mayreduce the generalizability of the findings.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation
    § V. DISCUSSION · p. 10

    Several studies have documented that unfairness in LLM outputs can propagate discrimination and harm marginalized groups, particularly in domains like healthcare, law, and education.

    “Several studies have documented that unfairness in LLM outputs can propagate discrimination and harm marginalized groups, particularly in domains like healthcare, law, and education [11], [26].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [11] Guo, 2024, Bias in large language models: Origin, evaluation, and mitigation, arXiv:2411.10915 source

      Checked before — its earlier verdict is held back until a new check runs.

    • cites [26] Sun, TrustLLM: Trustworthiness in large language models

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given
    § VI. LIMITATIONS AND FUTURE STUDIES · p. 13

    The MCDM technique used in this study has been validated in the literature and successfully applied in various fields.

    “The method used in this study is an MCDM technique that has been validated in the literature and successfully applied in various fields.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning
    § TABLE 10. Enveloped matrices. · p. 13

    The MCDM-based framework adopted in this study offers a structured and transparent means of navigating the complexity of LLM trustworthiness evaluation.

    “The MCDM-based framework adopted here offers a structured and transparent means of navigating these complexities.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § VI. LIMITATIONS AND FUTURE STUDIES · p. 16

    The experts' results were aggregated using arithmetic averages to reduce individual differences.

    “Their results were aggregated using arithmetic averages to reduce individual differences.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § VI. LIMITATIONS AND FUTURE STUDIES · p. 13

    The single expert group may not fully capture the diversity of perspectives across different application contexts.

    “it may not fully capture the diversity of perspectives across different application contexts.”
    Quote in the paperNot a substantial claim
52 of 52

  • The paper relies on this without support.

    The selected experts' evaluations are reliable and unbiased enough to derive valid criteria weights.

    “In step 3, i.e., expert selection and evaluation, experts were chosen based on their domain knowledge and familiarity with trustworthiness evaluation in AI systems.”

  • The paper relies on this without support.

    The scoring-based approach used to assess alternatives is a valid measure of LLM trustworthiness.

    “Finally, in step 5, the linguistic scoring system is used to evaluate the alternatives with respect to the criteria.”

  • The paper relies on this without support.

    The relative importance of trust criteria is stable across high-stakes domains.

    “This contribution is particularly valuable in high-stakes domains, where subjective expertise and contextual weighting of criteria are crucial for a comprehensive and actionable assessment of trustworthiness.”

  • The authors state this as an assumption.

    Newer iterations of LLMs enhance trustworthiness.

    “validating the assumption that newer iterations enhance trustworthiness.”

  • The paper relies on this without support.

    Proprietary models such as GPT-4o, Claude 3.5, and Gemini 1.5 are typically developed with access to significantly larger and more diverse training corpora, extensive compute resources, and rigorous post-training alignment pipelines.

    “Proprietary models such as GPT-4o, Claude 3.5, and Gemini 1.5 are typically developed with access to significantly larger and more diverse training corpora, extensive compute resources, and rigorous post-training alignment pipelines (e.g., multi-stage RLHF, large-scale human preference data, and continuous deployment feedback loops).”

  • The paper relies on this without support.

    Open-source models like Llama 3.1 and Mistral Large 2 are often constrained by more limited compute budgets and smaller-scale or less diverse alignment data.

    “In contrast, open-source models like Llama 3.1 and Mistral Large 2, while technically competitive in architecture, are often constrained by more limited compute budgets and smaller-scale or less diverse alignment data, which can impact their safety and explainability scores.”

  • The paper relies on this without support.

    The five FRIES-based dimensions (fairness, robustness, integrity, explainability, and safety) adequately capture the trustworthiness of LLMs for the purpose of this evaluation.

    “The study adopted only the five FRIES-based dimensions (fairness, robustness, integrity, explainability, and safety).”
7 of 7

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Healthcare Fuzzy AHP study is attributed to the wrong reference

    Minor

    The description attached to [33] does not match its reference entry, which identifies an LLM-as-a-fuzzy-judge study. The healthcare fuzzy-MCDM study appears instead as reference [36].

  • DeepSeek V3 is incorrectly called the lowest-scoring model

    Minor

    The reported scores put Mistral Large 2 at 4.94 and DeepSeek V3 at 4.99. DeepSeek is therefore second lowest, not lowest.

2 of 2

Cites

Flagged first, then by citations

Research works printed but not linked: 12

  1. Sun, TrustLLM: Trustworthiness in large language models, 105
  2. Song, Measuring and enhancing trustworthiness of LLMs in RAG through grounded attributions and learning to refuse, 1
  3. Eshlaghy, 2011, MCDM methodologies and applications: A literature review from 1999 to 2009, Res. J. Int. Stud., 21, 86
  4. Singh, 2016, Analytical hierarchical process (AHP) and fuzzy AHP applications—A review paper, Int. J. Pharm. Technol., 8, 4925
  5. 2024, Hello GPT-4o
  6. 2022, Introducing ChatGPT
  7. 2024, Introducing Llama 3.1: Our Most Capable Models to Date
  8. 2024, Claude 3.5 Sonnet
  9. Pichai, 2024, Our Next-Generation Model: Gemini 1.5
  10. 2024, Large Enough
  11. 2025, ChatBot Arena Leaderboard
  12. 2025, Open LLM Leaderboard

Cited by

Sorted by citations

Related

Authors
Aksoy, M., Adem, A., & Dağdeviren, M.
Affiliations
  • TU Dortmund University (DE)
  • Gazi University (TR)
ORCID
Venue
IEEE Access
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2025
Volume
13
Pages
168183-168201
ISSN
2169-3536 (ISSN-L)
DOI
10.1109/access.2025.3612568
OpenAlex
W4414404575
Cited by
10
References
38
Citations by year
2026: 10
FWCI
7.74
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Access (publishedVersion, cc-by)
  • Technische Universität Dortmund Eldorado (Technische Universität Dortmund) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Multi-Criteria Decision Making
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Keywords
large language models, trust evaluation, multi-criteria decision making, fairness, robustness, integrity, explainability, safety, expert evaluation, open-source vs closed-source models, GPT-4o, Llama 3.1, DeepSeek-V3

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-29.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-29.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review