An Integrated AHP–Fuzzy AHP Evaluation Framework for Large Language Models in Software Engineering Education

Probably not a substantial claim: may not pass S3 (load-bearing).

Recently introduced semi-pessimistic and semi-optimistic points of view have been proposed, corresponding to λ = 0.25 and λ = 0.75, respectively, providing a more nuanced interpretation of uncertainty.

“In addition to the pessimistic and optimistic perspectives, recently introduced semi-pessimistic and semi-optimistic points of view have been proposed, corresponding to λ = 0.25 and λ = 0.75, respectively, providing a more nuanced interpretation of uncertainty [51].”

• cites [51] Simjanović, D.J., Zdravković, N., and Vesić, N.O. (2022). On the Factors of Successful E-Commerce Platform Design during and after COVID-19 Pandemic Using Extended Fuzzy AHP Method. Axioms, 11. sourceSupported

May combine two claims.

Increased reliance on LLMs for critical thinking tasks such as code generation and debugging may negatively affect student performance, while their use for explanations appears more beneficial.

“Empirical evidence shows that increased reliance on LLMs, particularly for tasks requiring critical thinking such as code generation and debugging, may negatively affect student performance, while their use for explanations appears more beneficial, indicating a complex and non-trivial role in educational contexts [3].”

• cites [3] Jošt, G., Taneski, V., and Karakatič, S. (2024). The Impact of Large Language Models on Programming Education and Student Learning Outcomes. Appl. Sci., 14. sourceSupported

Classified as probably not a substantial claim. May combine two claims.

LLM-based evaluation can achieve a high correlation with human judgments and can be used to evaluate explanation quality, coherence, and overall usefulness of generated content.

“Studies have shown that LLM-based evaluation can achieve a high correlation with human judgments and can be used to evaluate explanation quality, coherence, and overall usefulness of generated content [34].”

• cites [34] Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., and Zhu, C. (2023). G-Eval: NLG Evaluation Using GPT-4 with Better Human Alignment. arXiv. sourcePartly supported

  The Introduction (page unavailable) reports that G-EVAL’s scores correlate strongly with human quality judgments, and the Evaluation Criteria section explicitly includes coherence. The paper evaluates summaries and dialogue responses, but the passages do not establish that it evaluates explanation quality or overall usefulness as distinct criteria. (LLM, conf=0.88)

  Experimental — accuracy not yet measured

Classified as probably not a substantial claim.

In software engineering education, the quality of educational content depends on several interrelated factors, such as content accuracy, depth of explanation, clarity of presentation, content organization, and practical applicability.

“In software engineering education, the quality of educational content depends on several interrelated factors, such as content accuracy, depth of explanation, clarity of presentation, content organization, and practical applicability [9,17].”

• cites [9] Washizaki, H. (2024). Guide to the Software Engineering Body of Knowledge, IEEE Computer Society.Not checked yet

  Not checked in the free run.

• cites [17] Garousi, 2020, Closing the Gap between Software Engineering Education and Industrial Needs, IEEE Softw. sourcePartly supported

  Section 7 (p. 7) says software engineering teaching should use real-world examples and include practical experience, supporting the practical-applicability aspect. Section 8 (p. 7) discusses aligning teaching materials with important topics, but neither passage presents content accuracy, depth of explanation, clarity, and organization as interrelated determinants of educational-content quality. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

Classified as probably not a substantial claim. May combine two claims.

AHP is used to determine criteria weights and rank alternatives based on expert judgment.

“One of the most commonly used multi-criteria decision-making methods is the Analytic Hierarchy Process (AHP), which is used to determine criteria weights and rank alternatives based on expert judgment [41].”

• cites [41] Saaty, 1987, The Analytic Hierarchy Process—What It Is and How It Is Used, Math. Model. sourceSupported

May combine two claims.

P3 is ranked thirteenth in the AHP, and one place higher in five FAHP cases.

“Understandable with increased effort from the reader (P3) is ranked thirteenth in the AHP, and one place higher in five FAHP cases.”

May combine two claims.

For T1, Model M2 has identical weights assigned by Experts 1 to 6, amounting to w_T1(M2) = 0.524942 in the AHP case, and 0.471839, 0.450208, and 0.440029 in the FAHP (λ = 0, λ = 0.5, and λ = 1) cases, indicating that it is the most important alternative within this group of evaluations.

“Comparing the following results, it can be observed that Model M2 has identical weights assigned by Experts 1 to 6, amounting to w T1 (M2) = 0.524942 in the AHP case, and 0.471839, 0.450208, and 0.440029 in the FAHP (λ = 0, λ = 0.5, and λ = 1) cases, indicating that it is the most important alternative within this group of evaluations.”

May combine two claims.

A comparative analysis of the rankings obtained using both AHP and FAHP approaches indicates a high level of agreement between the two methods, particularly with respect to the top-ranked alternatives, which remain unchanged across both models.

“A comparative analysis of the rankings obtained using both AHP and FAHP approaches indicates a high level of agreement between the two methods, particularly with respect to the top-ranked alternatives, which remain unchanged across both models.”

May combine two claims.

Six experts assigned the highest evaluations to M2, yielding weights of 0.524942 (AHP), and 0.471839, 0.450208, and 0.440029 in the FAHP cases (λ = 0; λ = 0.5; λ = 1), confirming a strong consensus regarding the importance of this model.

“Six experts assigned the highest evaluations to M2, yielding weights of 0.524942 (AHP), and 0.471839, 0.450208, and 0.440029 in the FAHP cases (λ = 0; λ = 0.5; λ = 1), confirming a strong consensus regarding the importance of this model.”

May combine two claims.

For topic T2, Model M2 is again identified as the highest-ranked alternative (consistent with T1), with an average weight of 0.465038 in the AHP case, while in the balanced FAHP scenario the weight amounts to 0.413013.

“For the topic T2-Requirements Analysis and Specification, Model M2 is again identified as the highest-ranked alternative (consistent with the results for T1), with an average weight of 0.465038 in the AHP case, while in the balanced FAHP scenario the weight amounts to 0.413013.”

May combine two claims.

The overall ranking remains stable under moderate variations of the leading criteria, with only minor changes observed among closely ranked alternatives.

“The results indicate that the overall ranking remains stable under moderate variations of the leading criteria, as presented in Figure 9 below, with only minor changes observed among closely ranked alternatives.”

May combine two claims.

In the AHP sensitivity analysis, the majority of criteria exhibit stable behavior, with no significant impact on the final ranking, indicating the robustness of the model.

“The majority of criteria exhibit stable behavior, with no significant impact on the final ranking, indicating the robustness of the model.”

May combine two claims.

The highest weight of the sub-criterion regarding minor inaccuracies (A2) is obtained in the optimistic FAHP case, w(A2) = 0.29, while the lowest value is reached in the AHP case, w(A2) = 0.26.

“The highest weights of the sub-criteria regarding to minor inaccuracies is obtained in the optimistic FAHP case, w(A2) = 0.29, while the lowest value is reached in the AHP case, w(A2) = 0.26.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The highest individual weight (0.524942) is assigned to Model M2 by Expert 2.

“The highest individual weight (0.524942) is assigned to Model M2 by Expert 2”

May combine two claims.

In the balanced FAHP case, the most significant criteria have weight w(A) = 0.319, having 1.53 times higher weight than criteria D, and 1.6 and 1.69 times more importance than criteria E and P.

“In the balanced FAHP case, the most significant criteria have weight w(A) = 0.319, having 1.53 times higher weight than criteria D, and 1.6 and 1.69 times more importance than criteria E and P.”

May combine two claims.

The leading sub-criterion from group A (A1) has weight w(A1) = 0.503 in the AHP case, and an averaged weight in all FAHP cases of w(A1) = 0.423.

“As can be seen in Table 3, the leading sub-criteria from group A deals with a completely accurate concept and terminology, with no conceptual or technical errors (A1) with a corresponding weight equal to w(A1) = 0.503 in the AHP case, and an averaged weight in all FAHP cases of w(A1) = 0.423.”

May combine two claims.

In the FAHP, E1, E2, and E3 are ranked fourth, seventh, and eleventh, while E4 ranks seventeenth in the pessimistic case and sixteenth in other cases.

“A similar situation is within the group E sub-criteria, ranking E1, E2, and E3 fourth, seventh, and eleventh in the FAHP, while E4 ranks seventeenth in the pessimistic case and sixteenth in other cases.”

May combine two claims.

P1 is placed fifth in all six cases, while P2 is ranked ninth in crisp and three FAHP cases (λ = 1, λ = 0.75, and λ = 0.5) and eighth in the rest.

“Exceptionally clear and logically structured presentation, the leading sub-criteria from the group P, is placed fifth in all six cases, while P2 is ranked ninth in crisp and three FAHP cases (λ = 1, λ = 0.75, and λ = 0.5) and eighth in the rest of the possibilities.”

May combine two claims.

E5 is ranked twenty-second for λ = 0, λ = 0.25, and λ = 0.5, while in the semi-optimistic and optimistic case it is ranked one place lower.

“E5 is ranked twenty-second for λ = 0, λ = 0.25, and λ = 0.5, while in the semi-optimistic and optimistic case is ranked one place lower.”

May combine two claims.

In the FAHP sensitivity analysis, the obtained results show a similar overall pattern as in the AHP case, with most criteria exhibiting stable behavior under weight variations.

“The obtained results show a similar overall pattern as in the AHP case, with most criteria exhibiting stable behavior under weight variations.”

The quote only partly states this claim.

The evaluated large language models differ in their performance across the evaluation criteria.

“The findings also indicate differences across models and a high level of agreement between AHP and FAHP rankings.”

• AssumptionRests on an assumption: Expert judgments from the eight-member panel are reliable and valid for determining criteria weights.
• AssumptionRests on an assumption: The five selected LLMs are representative of contemporary high-performance LLMs for educational content generation.
• AssumptionRests on an assumption: The adapted OER-based criteria adequately capture the quality of LLM-generated educational content in software engineering.

May combine two claims.

Experts 1 and 3 evaluated Model M2 the same across all of the FAHP cases, as did Experts 6 and 8.

“Experts 1 and 3 evaluated Model M2 the same across all of the FAHP cases, as well as Experts 6 and 8.”

May combine two claims.

Certain criteria (A2 and L1) show noticeable sensitivity, suggesting that they play a more influential role in the evaluation process.

“However, certain criteria (A2 and L1) show noticeable sensitivity, suggesting that they play a more influential role in the evaluation process.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The sub-criteria placed at the bottom are E5, P5, V3, and L3, related to non-existence of useful examples, low pedagogical quality, disorganized content and inappropriate style for academic use.

“The sub-criteria placed at the bottom of the ladder are related to the non-existence of useful examples, low pedagogical quality, disorganized content and inappropriate style for academic use, named E5, P5, V3, and L3.”

Classified as probably not a substantial claim.

Six main groups of evaluation criteria were defined: accuracy and professional correctness (A), depth and coverage of content (D), examples and practical application (E), pedagogical clarity and didactic quality (P), visual and structural presentation (V), and language style and academic tone (L).

“Based on the OER rubrics and the specific requirements of software engineering education, six main groups of evaluation criteria were defined: accuracy and professional correctness (A), depth and coverage of content (D), examples and practical application (E), pedagogical clarity and didactic quality (P), visual and structural presentation (V), and language style and academic tone (L).”

May combine two claims.

V1 and L1 are ranked tenth in all six cases, and L1 is ranked fourteenth in AHP and four places below in FAHP.

“Furthermore, the leading sub-criteria from groups V and L, V1 and L1, are ranked tenth (all six cases) and fourteenth (AHP) and four places below (FAHP).”

May combine two claims.

In the pessimistic FAHP case, the quotients of A's weight to D, E, and P are 1.6, 1.64, and 1.67, while for λ = 1 they are 1.5, 1.59, and 1.7.

“In the pessimistic FAHP case, the previous quotients are equal to 1.6, 1.64, and 1,67, while for λ = 1 they respectively are 1.5, 1.59, and 1.7.”

May combine two claims.

The FAHP-based results exhibit a more balanced distribution of criteria weights, reflecting the incorporation of uncertainty and imprecision in expert judgments.

“At the same time, the FAHP-based results exhibit a more balanced distribution of criteria weights, reflecting the incorporation of uncertainty and imprecision in expert judgments.”

May combine two claims.

In all five FAHP cases, the weights of V are 0.057, 0.06, 0.062, 0.063, and 0.064; the weights of L are 0.026, 0.026, 0.025, 0.025, and 0.025.

“Considering the weights of lastly ranked criteria, in all five cases of the FAHP they are equal: w(V) = 0.057, w(V) = 0.06, w(V) = 0.062, w(V) = 0.063, and w(V) = 0.064; w(L) = 0.026, w(L) = 0.026, w(L) = 0.025, w(L) = 0.025, and w(L) = 0.025.”

May combine two claims.

In the AHP case, the leading criteria has weight w(A) = 0.405, while w(D) = w(E) = w(P) = 0.1735.

“In the AHP case, the leading criteria has weight w(A) = 0.405, while w(D) = w(E) = w(P) = 0.1735.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The highest scores for Model M4 are assigned by Expert 7, amounting to 0.344618 and 0.342063 in the semi-pessimistic and semi-optimistic FAHP cases, respectively.

“The highest scores are assigned by Expert 7, amounting to 0.344618 and 0.342063 in the semi-pessimistic and semi-optimistic FAHP cases, respectively.”

May combine two claims.

Comparing the results in the semi-pessimistic and semi-optimistic points of view (FAHP), w(D3) = 0.109, and w(D3) = 0.119, while trivial content has weights w(D4) = 0.044 = w(D4), being 11.288 and 11.064 times less important than D1.

“Comparing the results in the semi-pessimistic and semi-optimistic points of view (FAHP), w(D3) = 0.109, and w(D3) = 0.119, while trivial content has weights w(D4) = 0.044 = w(D4), being 11.288 and 11.064 times less important than D1.”

May combine two claims.

With the same ranking order as presented in T2, model M2 remains highly graded, the leading one in topic T3, followed by M3, M1, M4, and at the bottom, M5.

“With the same ranking order, as it was presented in T2, model M2 remains highly graded, the leading one in topic T3, followed by M3, M1, M4, and at the bottom, M5”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The weights assigned to those experts in the balanced FAHP case were 0.393184 and 0.278096, respectively; in the λ = 0.25 case they were 0.397617 and 0.275044; and in the λ = 0.75 case they were 0.3902 and 0.280204.

“The weights assigned to those experts in the balanced FAHP case were 0.393184 and 0.278096, respectively, whereas their corresponding weights in the λ = 0.25 case were 0.397617 and 0.275044, and were 0.3902 and 0.280204 in the λ = 0.75 case.”

May combine two claims.

The weights obtained from Expert 4 for M4, the lowest of all, are 0.207347 (smallest weight, AHP), and 0.213914, 0.22054, and 0.223811 (FAHP: λ = 0., λ = 0.5, and λ = 1).

“The weights obtained from Expert 4, the lowest of all, determined by some differences in judgements, are 0.207347 (smallest weight, AHP), and 0.213914, 0.22054, and 0.223811 (FAHP: λ = 0., λ = 0.5, and λ = 1).”

May combine two claims.

The highest value of the sub-criteria D1 is obtained in the AHP case, w(D1) = 0.574, while the averaged weight in the FAHP cases is equal to w(D1) = 0.497.

“The highest value of the sub-criteria D1 is obtained in the AHP case, w(D1) = 0.574, while the averaged weight in the FAHP cases is equal to w(D1) = 0.497.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Expert 2 assigns the highest scores to Model M2 (AHP: 0.524942; FAHP, λ = 0.5: 0.450208).

“Expert 2 assigns the highest scores to Model M2 (AHP: 0.524942; FAHP, λ = 0.5: 0.450208)”

May combine two claims.

Model M5 lies at the bottom of the ladder, two places lower in rank than in T1.

“At the bottom of the ladder, two places lower in rank than in T1, lies model M5”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Those quotients in the FAHP cases (λ = 0.25, λ = 0.75) are equal to 0.964 and 1.047; and 0.963 and 1.048, respectively.

“Those quotients in the FAHP cases (λ = 0.25, λ = 0.75) are equal to 0.964 and 1.047; and 0.963 and 1.048, respectively.”

May combine two claims.

Model M1 lies in the middle of the ranking, with identical evaluations provided by Experts 1, 3, and 5.

“In the middle of the ranking lies Model M1, with identical evaluations provided by Experts 1, 3, and 5.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The tendency of similar experts' opinions can be seen in the results of Experts 2 and 7, whose determined weights are only 1.016 and 1.043 times less important than the leading ones in the AHP case.

“The tendency of similar experts' opinions can be seen in the results of Experts 2 and 7, whose determined weights are only 1.016 and 1.043 times less important than the leading ones in the AHP case.”

May combine two claims.

Expert 2 favors model M3, assigning it near-maximum evaluations with weight 0.424096 in the balanced FAHP case, and 0.430207 and 0.419841 when λ = 0.25 and λ = 0.75.

“Expert 2 favors model M3, assigning it near-maximum evaluations with weight 0.424096 in the balanced FAHP case, and 0.430207 and 0.419841 when λ = 0.25 and λ = 0.75.”

Probably not a substantial claim: may not pass S3 (load-bearing).

All experts gave Model M4 similar scores.

“and all experts gave it similar scores.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Expert 6 yields smaller weights for model M1: 0.166418, 0.16476, and 0.160969.

“while Expert 6, with some differences in judgements, yields smaller weights: 0.166418, 0.16476, and 0.160969.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Expert 7's lowest grades in the same cases correspond to the weights 0.148925, 0.1593, and 0.164251.

“especially Expert 7, whose lowest grades in the same cases correspond to the weights 0.148925, 0.1593, and 0.164251”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Expert 6 provides the lowest evaluation for Model M4 (0.225310), which is lower by a factor of 1.615 compared to the highest value.

“Expert 6 provides the lowest evaluation (0.225310), which is lower by a factor of 1.615 compared to the highest value”

The quote only partly states this claim.

The average weights for the different FAHP scenarios for Model M4 range from 0.295099 to 0.296535.

“The average weights for the different FAHP scenarios range from 0.295099 to 0.296535”

May combine two claims.

The consensus of the first three experts yields weights for M4 of 0.433814 (AHP) and 0.393814 for balanced FAHP, higher than the corresponding average values of 0.12 and 0.083, respectively.

“The consensus of the first three experts, as can be seen in Table 10, yields the following weights for M4, 0.433814 (AHP) and 0.393814 for balanced FAHP, higher than the corresponding average values of 0.12 and 0.083, respectively”

Classified as probably not a substantial claim. May combine two claims.

MCDM methods allow the evaluation of alternatives based on multiple criteria and the determination of the relative importance of each criterion.

“Multi-criteria decision-making methods allow the evaluation of alternatives based on multiple criteria and the determination of the relative importance of each criterion.”

Classified as probably not a substantial claim. May combine two claims.

Each LLM generated one response for each of the three selected topics using the same predefined system and user prompts, resulting in a total of 15 generated responses (3 topics × 5 models).

“Each LLM generated one response for each of the three selected topics using the same predefined system and user prompts, resulting in a total of 15 generated responses (3 topics × 5 models).”

Classified as probably not a substantial claim.

TFNs are particularly suitable when evaluations are expressed in linguistic terms, as they allow each assessment to be modeled as a triplet (l, m, u), representing the lower, most probable, and upper values.

“TFNs are particularly suitable when evaluations are expressed in linguistic terms, as they allow each assessment to be modeled as a triplet (l, m, u), representing the lower, most probable, and upper values.”

May combine two claims.

These findings provide a clear answer to RQ3, confirming that there are no significant changes in sub-criteria ranking when different optimism index values are used.

“Therefore, these findings provide a clear answer to RQ3, confirming that there are no significant changes in sub-criteria ranking when different optimism index values are used.”

Classified as probably not a substantial claim.

Based on expert agreement and analysis, the most relevant sub-criteria were selected and organized into six criteria groups to form a structured evaluation framework.

“Based on expert agreement and analysis, the most relevant sub-criteria were selected and organized into six criteria groups to form a structured evaluation framework.”

Classified as probably not a substantial claim.

Fuzzy set theory has been recognized as an effective framework for addressing the uncertainty and imprecision inherent in linguistic assessments, thereby providing substantial support for decision-making processes.

“For more than half a century, fuzzy set theory has been recognized as an effective framework for addressing the uncertainty and imprecision inherent in linguistic assessments, thereby providing substantial support for decision-making processes [47].”

• cites [47] Zadeh, 1975, The Concept of a Linguistic Variable and Its Application to Approximate Reasoning—II, Inf. Sci. sourceNot checked — no full text of the source

May combine two claims.

The results confirm RQ2, showing that highly influential sub-criteria do exist, with accuracy and professional correctness (A1) and depth and coverage of content (D1) identified as the most influential factors in the evaluation process.

“Therefore, the results confirm RQ2, showing that highly influential sub-criteria do exist, with accuracy and professional correctness (A1) and depth and coverage of content (D1) identified as the most influential factors in the evaluation process.”

May combine two claims.

Hesitant Fuzzy AHP allows for handling uncertainty in expert evaluations and for determining the relative importance of evaluation criteria.

“In particular, the use of hesitant Fuzzy AHP allows for handling uncertainty in expert evaluations and for determining the relative importance of evaluation criteria.”

Classified as probably not a substantial claim. May combine two claims.

The selection of topics ensured that the evaluation covered different cognitive levels, which increases the relevance and reliability of the research results.

“This structure ensures that the evaluation covers different cognitive levels, which increases the relevance and reliability of the research results.”

• AssumptionRests on an assumption: The three selected software engineering topics are representative of software engineering education and sufficiently cover relevant cognitive levels.

May combine two claims.

The same system prompt and the same user prompts were used for all evaluated LLM models, enabling comparability of the results.

“The same system prompt and the same user prompts were used for all evaluated LLM models.”

• AssumptionRests on an assumption: The five selected LLMs are representative of contemporary high-performance LLMs for educational content generation.

May combine two claims.

The findings verify that there are influential sub-criteria (A1 and D1) identified in the evaluation framework; these have consistently had larger weights than other sub-criteria across several experts and methods (AHP and FAHP), indicating they contribute more toward the overall decision-making process.

“The findings verify that there are some influential sub-criteria (A1 and D1) identified in the evaluation framework.These sub-criteria have consistently had larger weights than other sub-criteria across several experts and methods (AHP and FAHP), indicating they contribute more toward the overall decision-making process.”

Classified as probably not a substantial claim.

The evaluation of LLM-generated educational content is a multi-dimensional problem involving multiple criteria such as accuracy, depth of explanation, pedagogical clarity, structure, and practical applicability.

“The evaluation of LLM-generated educational content represents a multi-dimensional problem that involves multiple criteria such as accuracy, depth of explanation, pedagogical clarity, structure, and practical applicability.”

Classified as probably not a substantial claim.

The normalized weights of the leading criteria from all groups were varied within a predefined range (±10-20%), in all six cases.

“Specifically, the normalized weights of the leading criteria from all groups were varied within a predefined range (±10-20%), in all six cases.”

Classified as probably not a substantial claim.

Triangular fuzzy numbers are employed to represent expert judgments due to their simplicity and effectiveness in handling uncertainty in decision-making problems.

“Triangular fuzzy numbers are employed to represent expert judgments due to their simplicity and effectiveness in handling uncertainty in decision-making problems.”

May combine two claims.

The improved ranking position of Model M3 is primarily attributable to strong evaluation from an average of all three sets of evaluation criteria and to a number of experts who provided consistent evaluations over time.

“This change is primarily attributable to strong evaluation from an average of all three sets of evaluation criteria as well as by a number of experts who provided consistent evaluations over time.”

Classified as probably not a substantial claim. May combine two claims.

The definition of sub-criteria was based on the literature review presented in Section 2 and further refined through expert consultation.

“The definition of sub-criteria was based on the literature review presented in Section 2 and further refined through expert consultation.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The experts were selected to include different professional backgrounds and perspectives from academia, industry, and the public sector.

“The experts were selected to include different professional backgrounds and perspectives from academia, industry, and the public sector.”

Classified as probably not a substantial claim. May combine two claims.

The value CR < 0.1 verifies the matrix H's consistency; otherwise, the reason for inconsistency should be determined, and all calculations redone.

“The value CR < 0.1 verifies the matrix H's consistency; otherwise, the reason for inconsistency should be determined, and all calculations redone.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A minimum of five years of experience was considered as a baseline criteria.

“A minimum of five years of experience was considered as a baseline criteria.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Existing OER rubrics provide a solid basis for evaluating educational content but do not fully capture the specific characteristics and challenges of content generated by LLMs, especially in software engineering education.

“Existing OER rubrics provide a solid basis for evaluating educational content, but they do not fully capture the specific characteristics and challenges of content generated by LLMs, especially in software engineering education.”

May combine two claims.

Overall, these results demonstrate that the proposed framework provides reliable and interpretable rankings while maintaining resilience to minor variations in expert input.

“Overall, these results demonstrate that the proposed framework provides reliable and interpretable rankings while maintaining resilience to minor variations in expert input.”

May combine two claims.

The fuzzy matrices for the sub-criteria of groups A, E, and P are the same, as well as the matrices for the sub-criteria of groups V and L; all comparison matrices are consistent.

“According to the experts' opinions, the fuzzy matrices for the sub-criteria of groups A, E, and P are the same, as well as the matrices for the sub-criteria of groups V and L. All comparison matrices are consistent.”

May combine two claims.

The dominance of Model M2 remains consistent across all considered scenarios, supporting the robustness of the ranking, despite variability in individual judgments and criteria group sensitivities.

“However, the dominance of Model M2 remains consistent across all considered scenarios, supporting the robustness of the ranking, despite variability in individual judgments and criteria group sensitivities.”

Classified as probably not a substantial claim.

AHP provides a consistency verification mechanism (the consistency ratio) that is essential when working with subjective inputs.

“In addition, AHP provides a consistency verification mechanism (the consistency ratio), which is essential when working with subjective inputs.”

Classified as probably not a substantial claim. May combine two claims.

Classical AHP requires precise numerical judgments, which can be difficult when evaluation is based on subjective expert opinions.

“However, classical AHP requires precise numerical judgments, which can be difficult when evaluation is based on subjective expert opinions.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The panel was composed to ensure a balance between theory, pedagogical and practical perspectives.

“The panel was composed to ensure a balance between theory, pedagogical and practical perspectives.”

Classified as probably not a substantial claim. May combine two claims.

Existing OER rubrics were used as a starting point, and their criteria were systematically adapted and extended to reflect key aspects of LLM-generated content, including technical and professional correctness, control of hallucinations, variability in the quality of explanations, and alignment with curricula, academic and instructional requirements.

“In this paper, existing OER rubrics were used as a starting point, but their criteria were systematically adapted and extended to reflect key aspects of LLM-generated content, including technical and professional correctness, control of hallucinations, variability in the quality of explanations, and alignment with curricula, academic and instructional requirements.”

• AssumptionRests on an assumption: The adapted OER-based criteria adequately capture the quality of LLM-generated educational content in software engineering.

Classified as probably not a substantial claim.

Fuzzy AHP allows the modeling of uncertainty and vagueness inherent in human judgment.

“the AHP framework is extended using fuzzy logic (FAHP), which allows the modeling of uncertainty and vagueness inherent in human judgment.”

Probably not a substantial claim: may not pass S3 (load-bearing).

All participants declared any potential conflicts of interest before taking part in the study.

“All participants declared any potential conflicts of interest before taking part in the study.”

Probably not a substantial claim: may not pass S2 (checkable).

There is a need for a structured multi-criteria evaluation model that can consider multiple quality criteria and determine their relative importance when evaluating LLM-generated educational content.

“Therefore, there is a need for a structured multi-criteria evaluation model that can consider multiple quality criteria and determine their relative importance when evaluating LLM-generated educational content.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The proposed framework is standardized so that it can be applied not only to software engineering courses, but also to other educational domains where LLMs are used for generating educational content.

“The framework is standardized so that it can be applied not only to software engineering courses, but also to other educational domains where LLMs are used for generating educational content.”

• AssumptionRests on an assumption: The three selected software engineering topics are representative of software engineering education and sufficiently cover relevant cognitive levels.
• AssumptionRests on an assumption: The adapted OER-based criteria adequately capture the quality of LLM-generated educational content in software engineering.

Classified as probably not a substantial claim.

The selection of sub-criteria was conducted through several rounds of expert consultation.

“The selection of sub-criteria was conducted through several rounds of expert consultation.”

Classified as probably not a substantial claim.

Fuzzy AHP uses fuzzy numbers to model uncertainty and imprecision in expert judgments.

“For this reason, Fuzzy AHP was developed, which uses fuzzy numbers to model uncertainty and imprecision in expert judgments [44].”

• cites [44] Chang, 1996, Applications of the Extent Analysis Method on Fuzzy AHP, Eur. J. Oper. Res. sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

As a generalization of classical set theory, the primary objective of fuzzy sets is to enable a mathematical representation of linguistic variables, allowing decision-makers to model systems characterized by partially known or incomplete information.

“As a generalization of classical (crisp) set theory, the primary objective of fuzzy sets is to enable a mathematical representation of linguistic variables, allowing decision-makers to model systems characterized by partially known or incomplete information [48].”

• cites [48] Zadeh, 1975, The Concept of a Linguistic Variable and Its Application to Approximate Reasoning—I, Inf. Sci. sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The FAHP extension contributes to more balanced weight distributions and improves the interpretability and stability of the results without altering the overall ranking structure.

“This extension contributes to more balanced weight distributions and improves the interpretability and stability of the results without altering the overall ranking structure.”

Classified as probably not a substantial claim.

The use of TFNs enables the modeling of different sources of uncertainty inherent in expert evaluations, including imprecision, vagueness, and subjectivity.

“The use of TFNs enables the modeling of different sources of uncertainty inherent in expert evaluations, including imprecision, vagueness, and subjectivity.”

May combine two claims.

Despite local variations, the overall ranking remains largely unchanged, indicating the potential robustness and consistency of the proposed FAHP-based evaluation framework.

“Despite these local variations, the overall ranking remains largely unchanged, indicating the potential robustness and consistency of the proposed FAHP-based evaluation framework.”

May combine two claims.

Experts 7 and 8 adopt a more conservative assessment of Model M2 compared to the rest of the expert group, although the overall ranking of the model remains unchanged.

“This deviation suggests that Experts 7 and 8 adopt a more conservative assessment of Model M2 compared to the rest of the expert group, although the overall ranking of the model remains unchanged.”

Classified as probably not a substantial claim.

During the selection process, experts evaluated the importance of each sub-criterion using predefined factors such as its relevance to the educational process, measurability, and suitability for the educational context.

“During this process, experts evaluated the importance of each sub-criterion using predefined factors such as its relevance to the educational process, measurability, and suitability for the educational context.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Although a fuzzy representation using triangular fuzzy numbers (TFNs) is employed, the initial judgments are collected in a crisp form, which limits the extent to which uncertainty is genuinely captured.

“Second, although a fuzzy representation using triangular fuzzy numbers (TFNs) is employed, the initial judgments are collected in a crisp form, which limits the extent to which uncertainty is genuinely captured.”

May combine two claims.

Consistency within expert pairs suggests the presence of coherent evaluation patterns, making the reliability of the obtained results stronger despite variations in individual scoring levels.

“One can say that consistency within expert pairs suggests the presence of coherent evaluation patterns, making the reliability of the obtained results stronger despite variations in individual scoring levels.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Expert 6 reflects a more critical viewpoint.

“reflecting a more critical viewpoint.”

May combine two claims.

The framework allows the identification of variability among expert judgments while maintaining stable overall ranking results, which confirms its suitability for complex evaluation tasks such as the assessment of LLM-generated educational content.

“the framework allows the identification of variability among expert judgments while maintaining stable overall ranking results, which confirms its suitability for complex evaluation tasks such as the assessment of LLM-generated educational content.”

May combine two claims.

The proposed model is stable and not sensitive to variations in decision-maker attitudes or fuzzy parameters.

“indicating that the proposed model is stable and not sensitive to variations in decision-maker attitudes or fuzzy parameters.”

May combine two claims.

The improved ranking position of Model M3 is driven not only by stronger performance across key criteria groups but also by reduced variability in expert assessments, which contributes to a more stable and reliable evaluation outcome.

“The results highlight that the improved ranking position of Model M3 is driven not only by stronger performance across key criteria groups, but also by the reduced variability in expert assessments, which contributes to a more stable and reliable evaluation outcome.”

May combine two claims.

Experts mostly agree on Model M4, even though there are some differences in how they rate things.

“This distribution shows that experts mostly agree, even though there are some differences in how they rate things.”

May combine two claims.

This ranking order supports the ranking robustness and proposed framework justification.

“supporting the ranking robustness and proposed framework justification”

Probably not a substantial claim: may not pass S3 (load-bearing).

The structure of the criteria and sub-criteria may still involve a degree of overlap or dependency, which is not explicitly modeled within the current AHP-based framework.

“In addition, the structure of the criteria and sub-criteria, although carefully designed, may still involve a degree of overlap or dependency, which is not explicitly modeled within the current AHP-based framework.”

May combine two claims.

The integration of classical AHP and Fuzzy AHP enables both clear prioritization of the evaluation criteria and the incorporation of uncertainty, vagueness, and subjectivity present in expert-based evaluations.

“The integration of classical AHP and Fuzzy AHP enables both clear prioritization of the evaluation criteria and the incorporation of uncertainty, vagueness, and subjectivity present in expert-based evaluations.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The results are context-dependent and may vary with different expert groups, evaluation settings, or alternative methodological choices.

“Finally, the results are context-dependent and may vary with different expert groups, evaluation settings, or alternative methodological choices.”

Classified as probably not a substantial claim. May combine two claims.

The proposed framework relies on expert-based evaluations, which are inherently subjective and may introduce bias depending on the selection and experience of the experts involved.

“Additionally, the proposed framework relies on expert-based evaluations, which are inherently subjective and may introduce bias depending on the selection and experience of the experts involved.”

May combine two claims.

The framework enables systematic comparison of different models across different topics, criteria, and cognitive levels, confirming its practical applicability in real educational evaluation contexts.

“The framework enables systematic comparison of different models across different topics, criteria, and cognitive levels, confirming its practical applicability in real educational evaluation contexts.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Experts 6 and 8 reflect a more critical perspective.

“reflecting a more critical perspective.”