Comparing predictive, simulation-based and hybrid assessments in undergraduate process engineering

May combine two claims.

Across the implemented instructional sequence, higher mean marks and lower dispersion were observed under the hybrid assessment format than under the predictive and simulation-based formats in both modules.

“Across the implemented instructional sequence, higher mean marks and lower dispersion were observed under the hybrid assessment format relative to the predictive and simulation-based formats in both modules.”

• Also stated in: § 6 Conclusion · plain

  “Under the implemented sequence, higher mean marks and lower dispersion were observed for the hybrid format relative to the predictive and simulation-based formats”

May combine two claims.

Mark dispersion was lowest under the hybrid format, with the narrowest observed range.

“Mark dispersion was lowest under the hybrid format”

• Also stated in: § 4 Results · plain

  “Under the hybrid format, the observed range was narrowest”

May combine two claims.

Predictive assessments showed wider mark ranges than the simulation-based and hybrid formats, with simulation-based assessments showing intermediate dispersion.

“Predictive assessments also showed wider mark ranges than the simulation-based and hybrid formats, with simulation-based assessments showing intermediate dispersion.”

May combine two claims.

Descriptively, the hybrid format showed the highest mean marks in both modules (TPSC: M = 3.94; TPT: M = 4.02), compared with the predictive format (TPSC: M = 2.56; TPT: M = 2.92) and the simulation-based format (TPSC: M = 3.38; TPT: M = 3.57).

“Descriptively, the hybrid format showed the highest mean marks in both modules (TPSC: M = 3.94; TPT: M = 4.02), compared with the predictive format (TPSC: M = 2.56; TPT: M = 2.92) and the simulation-based format (TPSC: M = 3.38; TPT: M = 3.57).”

May combine two claims.

Repeated-measures analyses indicated differences in mean marks across assessment formats in both modules, with higher mean marks under the simulation-based and hybrid formats than under the predictive format, and higher hybrid marks than simulation-based marks.

“Repeated-measures analyses indicated differences in mean marks across assessment formats in both modules, with higher mean marks observed under the simulation-based and hybrid formats than under the predictive format; hybrid marks were also higher than simulation-based marks.”

May combine two claims.

Under the simulation-based format, the range was narrower and the central tendency was higher than under the predictive format.

“Under the simulation-based format, the range was narrower and the central tendency was higher than under the predictive format”

May combine two claims. The quote only partly states this claim.

Students' end-of-study questionnaire responses indicated generally positive perceptions of simulation-supported learning and of the integrated assessment workflow, including linking theoretical concepts with simulated process scenarios and increased confidence in applying course knowledge to problem-solving tasks.

“Responses indicated generally positive perceptions of simulation-supported learning and the integrated assessment workflow.”

• Also stated in: § 6 Conclusion · plain

  “students reported positive perceptions of linking theoretical concepts with simulated process scenarios and increased confidence in applying course knowledge to problem-solving tasks.”

May combine two claims.

Under the hybrid format, the lower bound of marks was higher than in earlier phases, reflecting fewer very low marks during the hybrid phase in this cohort.

“the lower bound of marks was higher than in the earlier phases, reflecting fewer very low marks during the hybrid phase in this cohort.”

May combine two claims.

Simulation-based learning enables structured 'what-if' exploration and interpretation of outputs, but its learning value depends on whether assessment prompts require students to articulate governing principles and justify decisions rather than only operating the tool.

“In classroom settings, SBL enables structured ‘what-if’ exploration and interpretation of outputs (e.g., trend plots and computed variables), but its learning value depends on whether assessment prompts require students to articulate governing principles and justify decisions rather than only operating the tool [21].”

• cites [21] Psycharis S. Examining the effect of the computational models on learning performance, scientific reasoning, epistemic beliefs and argumentation: an implication for the STEM agenda. Comput Educ. 2013;68:253–65. https://doi.org/10.1016/j.compedu.2013.05.015. sourceNot checked — no full text of the source

Classified as probably not a substantial claim. May combine two claims.

Cognitive engagement is conceptualized as students' investment in the cognitive work of learning rather than as a learning outcome score; performance outcomes should not be treated as direct measures of engagement.

“Cognitive engagement is commonly conceptualized as students’ investment in the cognitive work of learning (e.g., depth of processing, strategic effort, and self-monitoring/metacognitive regulation) rather than as a learning outcome score; accordingly, performance outcomes should not be treated as direct measures of engagement [25].”

• cites [25] Luo L, Yusuf A. Bridging AI and pedagogy: how AI-adaptive feedback shapes Chinese EFL students’ writing engagement, metacognitive writing strategies, and writing performance. Assess Eval High Educ. 2025;0:1–24. https://doi.org/10.1080/02602938.2025.2548919. sourceNot checked — no full text of the source

May combine two claims.

Constructive alignment theory holds that assessment tasks strongly shape what students attend to and practice, and that assessments requiring explanation, justification, and evidence-based decision making are more likely to elicit higher-quality disciplinary reasoning than tasks focused primarily on procedural completion.

“Constructive alignment theory emphasizes that assessment tasks strongly shape what students attend to and practice; therefore, assessments that require explanation, justification, and evidence-based decision making are more likely to elicit higher-quality disciplinary reasoning than tasks focused primarily on procedural completion [23, 24].”

• cites [23] Biggs J. What the student does: teaching for enhanced learning. High Educ Res Dev. 1999;18:57–75. https://doi.org/10.1080/0729436990180105. sourceNot checked — no full text of the source
• cites [24] McCune V, Entwistle N. The deep approach to learning: analytic abstraction and idiosyncratic development. In: Innovations in Higher Education Conference. University of Helsinki Helsinki. 2000:1–17.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Alternative explanations for the observed mark differences include cumulative learning, increasing familiarity with EnVision, and repeated exposure to similar scenario structures.

“Alternative explanations include cumulative learning, increasing familiarity with EnVision, and repeated exposure to similar scenario structures.”

May combine two claims.

Assessments were graded by a single instructor using a scenario-specific answer key; inter-rater reliability was not estimated and grading was not blinded to format.

“Fifth, assessments were graded by a single instructor using a scenario-specific answer key; inter-rater reliability was not estimated and grading was not blinded to format.”

Classified as probably not a substantial claim. May combine two claims.

The same learning-outcome domains were targeted across formats; the primary difference was whether students were allowed to run EnVision, required to reason without it, or required to do both in sequence.

“The same learning-outcome domains were targeted across formats; the primary difference was whether students were allowed to run EnVision (simulation-based), required to reason without it (predictive), or required to do both in sequence (hybrid).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Reduced variability under the hybrid format is reported descriptively and is not interpreted as evidence of equity or fairness, or of differential support for specific student subgroups, in the absence of dedicated subgroup analyses and appropriate modelling.

“These distributional patterns are reported as differences in score consistency and should not be interpreted as evidence of equity or fairness, nor as evidence of differential support for specific student subgroups, without dedicated subgroup analyses and appropriate modelling.”

• Also stated in: § 6 Conclusion · plain

  “Reduced variability is reported descriptively and is not interpreted as evidence of equity or fairness in the absence of dedicated subgroup analyses and appropriate modelling.”

May combine two claims.

Outcomes were assessed within one academic year, and long-term retention and transfer were not evaluated.

“Third, outcomes were assessed within one academic year; long-term retention and transfer were not evaluated.”

Classified as probably not a substantial claim. May combine two claims.

Because CGPA bands represent between-student groupings and the study was not designed or analysed to test a formal format × CGPA interaction, the CGPA-band patterns are treated as hypothesis-generating rather than confirmatory.

“Because CGPA bands represent between-student groupings and the study was not designed or analysed to test a formal format × CGPA interaction, these patterns are treated as hypothesis-generating rather than confirmatory.”

Probably not a substantial claim: may not pass S3 (load-bearing).

One plausible interpretation of the higher hybrid marks is that the pattern aligns with constructive alignment principles, because prompts requiring students to articulate theory-based predictions and then evaluate them with simulation evidence may elicit more explicit disciplinary reasoning than tasks focused on prediction or tool-based interpretation alone.

“One plausible interpretation is that this pattern aligns with constructive alignment principles [23]; assessment prompts that require students to articulate theory-based predictions and then evaluate those predictions using simulation evidence may elicit more explicit disciplinary reasoning than tasks focused mainly on either prediction or tool-based interpretation alone.”

• cites [23] Biggs J. What the student does: teaching for enhanced learning. High Educ Res Dev. 1999;18:57–75. https://doi.org/10.1080/0729436990180105. sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Institutional CGPA was used only to stratify descriptive plots and was not converted to, or mathematically mapped onto, the course mark scale.

“Institutional CGPA is recorded on a 0–10 scale, whereas course marks are recorded on a 0–5 scale; CGPA was used only to stratify descriptive plots and was not converted to, or mathematically mapped onto, the course mark scale.”

Classified as probably not a substantial claim.

Given the sequential implementation, the observed distributional patterns may reflect progression in course learning and students' increasing familiarity with simulation-supported tasks and expectations rather than differences attributable solely to assessment format.

“Given the sequential implementation, these patterns may reflect progression in course learning and students’ increasing familiarity with simulation-supported tasks and expectations rather than differences attributable solely to assessment format.”

May combine two claims.

Within the study's limitations, it provides classroom-based evidence that an integrated hybrid assessment workflow can be implemented in undergraduate process engineering modules and is perceived positively by students with respect to theory-practice connection.

“Within these limitations, the study provides classroom-based evidence that an integrated hybrid assessment workflow can be implemented in undergraduate process engineering modules and is perceived positively by students with respect to theory–practice connection.”

Probably not a substantial claim: may not pass S3 (load-bearing).

We do not infer differential benefits by CGPA band from these plots.

“Accordingly, we do not infer differential benefits by CGPA band from these plots”

May combine two claims.

Formal validation of the performance measures was not undertaken; findings should be interpreted as classroom-based evidence rather than validated measurement of latent constructs.

“Formal validation of the performance measures (e.g., external expert panel review) was not undertaken because the assessments were embedded in routine course delivery; therefore, findings should be interpreted as classroom-based evidence rather than validated measurement of latent constructs.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The study was conducted in accordance with the IMCO Ethics and Biosafety Guidelines and relevant institutional regulations, and the IMCO Ethics and Biosafety Committee confirmed that this minimal-risk, classroom-based educational study did not require formal prior ethics approval under institutional policy at the time of the research.

“This study was conducted in accordance with the IMCO Ethics and Biosafety Guidelines and relevant institutional regulations. The IMCO Ethics and Biosafety Committee confirmed that this minimal-risk, classroom-based educational study did not require formal prior ethics approval under institutional policy at the time of the research”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Because grading was conducted by a single instructor, inter-rater reliability was not estimated, and grading was not blinded to format.

“Because grading was conducted by a single instructor, inter-rater reliability was not estimated, and grading was not blinded to format as the formats were embedded in routine instruction.”

Classified as probably not a substantial claim. May combine two claims.

The study directly measured performance outcomes and collected retrospective student perceptions, so claims regarding cognitive engagement, deep learning, self-regulated learning and motivational change are not presented as directly measured outcomes.

“In addition, the study directly measured performance outcomes and collected retrospective student perceptions; accordingly, claims regarding cognitive engagement, deep learning, self-regulated learning, and motivational change are not presented as directly measured outcomes.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Blinding to assessment format was not feasible because the formats were implemented as part of routine course delivery, but analyses were conducted on de-identified data.

“Blinding to assessment format was not feasible because the formats were implemented as part of routine course delivery; however, analyses were conducted on de-identified data.”

May combine two claims.

The same students completed the predictive, simulation-based and hybrid assessments in a fixed sequence, so differences observed across formats cannot be separated from order effects such as cumulative learning, practice/familiarity, and maturation across the academic year.

“Second, the same students completed the predictive, simulation-based, and hybrid assessments in a fixed sequence; therefore, differences observed across formats cannot be separated from order effects such as cumulative learning, practice/familiarity, and maturation across the academic year.”

May combine two claims.

Deep learning, cognitive engagement, and the "black box" effect were not measured using validated multi-item psychometric instruments; they were used as a design rationale.

“In the present study, deep learning, cognitive engagement, and the “black box” effect were not measured using validated multi-item psychometric instruments. Instead, these constructs are used as a design rationale”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Because CGPA bands represent between-student groupings and the study was not designed to test differential effects by prior attainment, Fig. 3 is interpreted as exploratory.

“Because CGPA bands represent between-student groupings and the present study was not designed to test differential effects by prior attainment, Fig. 3 is interpreted as exploratory.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Participation in the research component was voluntary, and students were informed that choosing not to participate would not affect their grades, assessment outcomes or course standing.

“Participation in the research component (permission to use de-identified course performance records for analysis and completion of the end-of-study questionnaire) was voluntary. Students were informed that choosing not to participate would not affect their grades, assessment outcomes, or course standing.”

May combine two claims.

Because the same cohort completed all three formats in both modules, observations were not independent; a repeated-measures analytical framework was used to account for within-student dependence.

“Because the same cohort of students completed all three formats in both modules, observations were not independent. We therefore used a repeated-measures analytical framework to account for within-student dependence.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The answer key was mapped to the relevant module learning outcomes to ensure that marks reflected the intended technical competencies for each scenario.

“The answer key was mapped to the relevant module learning outcomes to ensure that marks reflected the intended technical competencies for each scenario.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

From an outcomes-based education perspective, an integrated assessment workflow may help instructors align assessment demands with intended learning outcomes that emphasize both conceptual understanding and application.

“From an outcomes-based education (OBE) perspective, an integrated assessment workflow may help instructors align assessment demands with intended learning outcomes that emphasize both conceptual understanding and application.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Informed consent to publish anonymized, aggregated results was obtained from participating students.

“Informed consent to publish anonymized, aggregated results was obtained from participating students.”

May combine two claims.

Because the questionnaire was administered once at the end of the study period, items referencing 'before/during/after' represent retrospective self-reports rather than a true pre/post design, and the results are therefore interpreted as perceptions rather than evidence of measured motivational or engagement change.

“Because the questionnaire was administered once at the end of the study period, items referencing ‘before/during/after’ represent retrospective self-reports rather than a true pre/post design; accordingly, these results are interpreted as perceptions rather than evidence of measured motivational or engagement change.”

May combine two claims.

Across formats, students responded to comparable scenario types; only the evidence base differed.

“Across formats, students responded to comparable scenario types; only the evidence base differed (theory-only vs. simulation output vs. theory + simulation evidence).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Because the questionnaire instrument was purpose-developed and administered at the end of the study period, results are reported descriptively at the item level and interpreted as retrospective perceptions rather than true pre/post change scores; no internal consistency reliability is claimed.

“Because the instrument was purpose-developed and administered at the end of the study period, results are reported descriptively at the item level and are interpreted as retrospective perceptions rather than true pre/post change scores; no internal consistency reliability is claimed.”

May combine two claims.

Collectively, the study's limitations restrict causal inference, and the findings should be interpreted as descriptive patterns observed under the implemented instructional sequence.

“Collectively, these limitations restrict causal inference, and findings should be interpreted as descriptive patterns observed under the implemented instructional sequence.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The distributional patterns are reported descriptively as differences in score dispersion and should not be interpreted as evidence of equity, fairness, or subgroup benefit.

“These distributional patterns are reported descriptively as differences in score dispersion across phases and should not be interpreted as evidence of equity, fairness, or subgroup benefit without dedicated subgroup analyses and appropriate modelling.”

Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing), S4 (based).

Simulation-only tasks may not consistently elicit explicit theory-based justification unless prompts and scoring criteria explicitly require students to articulate governing principles and justify decisions.

“One interpretation that warrants future testing is that simulation-only tasks may not consistently elicit explicit theory-based justification unless prompts and scoring criteria explicitly require students to articulate governing principles and justify decisions.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Active learning approaches in engineering education have been associated with improved student engagement and academic performance.

“In this context, active learning approaches in engineering education have been associated with improved student engagement and academic performance [30].”

• cites [30] Munna AS, Kalam MA. Impact of active learning strategy on the student engagement. GNOSI Interdiscip J Hum Theory Prax. 2021;4:96–114.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The Figure 5 responses are presented as perceived helpfulness by outcome and should not be interpreted as an overall preference for the hybrid approach or as validated evidence of skill development.

“These responses are presented as perceived helpfulness by outcome and should not be interpreted as an overall preference for the hybrid approach or as validated evidence of skill development.”