The limited value of machine learning approach to improving predictive performance: The Ministry of Justice Case Assessment Tool.

The variability ranged from no predictive differences (0.74 vs. 0.74 in Sheppard et al., 2024) to moderate improvements (0.75 vs. 0.67 in Mori et al., 2024; 0.73 vs. 0.64 in Miron et al., 2021; and 0.69 vs. 0.64 in Ting et al., 2018).

“This variability ranged from no predictive differences (0.74 vs. 0.74 in Sheppard et al., 2024) to moderate improvements (0.75 vs. 0.67 in Mori et al., 2024;0.73 vs. 0.64 in Miron et al., 2021;and 0.69 vs. 0.64 in Ting et al., 2018).”

When directly comparing computational performance against the score assigned by the human evaluator or against strictly linear comparators, moderate improvements in discriminative capacity are observed.

“Empirical results demonstrate that, when directly comparing computational performance against the score assigned by the human evaluator (Mori et al., 2024) or against strictly linear comparators (Ting et al., 2018), moderate improvements in discriminative capacity are observed.”

Mori et al. (2024) found that the ML model outperforms the MJCA score in terms of discrimination (ΔAUC = 0.08).

“The ML model outperforms the MJCA score in terms of discriminati on (ΔAUC = 0.08).”

The potential added value of AI/ML, when observed, may lie in capturing nonlinear interactions and relationships between variables, especially when the comparator is linear or when the human score does not explicitly incorporate such combinations; improvements are more likely when the comparator is strictly linear or when compared with unstructured judgment.

“These findings are consistent with the idea that the potential added value of AI/ML, when observed, may lie in capturing nonlinear interactions and relationships between variables, especially when the comparator is linear or when the human score does not explicitly incorporate such combinations (Mori et al., 2024;Steyerberg, 2019;Ting et al., 2018).”