Comparing effect sizes in follow-up studies: ROC Area, Cohen's d, and r.

“AUC also has a plain language interpretation because it is conceptually and mathematically similar to the common language effect size or CL ( McGraw & Wong, 1992; see also Delaney & Vargha, 2002).”

“The intuitive interpretation of AUCs and their resistance to base rate differences across groups led Rice and Harris (2005) to recommend the use of AUCs to compare risk instruments.”

“Our statistical tests (t-test [33], Cohen’s D [34]) also report that their performance is significantly lower.”

“The AUC is a common discrimination metric provided in RNA evaluation and estimates accuracy magnitude, where 0.55 indicates a small effect, 0.63 a moderate effect, and 0.71 a large effect (Rice & Harris, 2005).”

“Effect sizes (Cohen's d ) were calculated for all outcome measures, with values of 0.2, 0.5, and 0.8 indicating small, medium, and large effect sizes, respectively (Rice & Harris, 2005).”

“AUC ’ s between .56 and .64 were perceived as small effect, AUC ’ s between .64 and .71 were perceived as medium, and AUC ’ s above .71 were perceived as large (Rice & Harris, 2005).”

“Rice and Harris (2005) provided guidelines for interpreting AUC values, which facilitate comparison across studies applying different effect sizes.”

“Area Under the Curve (AUC) is an overall estimation of classification accuracy for a scale, across all potential scale scores observed and ranges in magnitude from 0 to 1.0, with effects at or exceeding .71 indicating large magnitude classifications (Rice & Harris, 2005).”

“With the Area Under the Curve (AUC) values, the effect size d (Rice & Harris, 2005) and the probability of occurrence OR were calculated (Borenstein et al., 2009; Salgado, 2018).”

“Values 0.556 were considered ‘small’, values from .639 to .713 were considered ‘moderate’, and values .714 indicated a ‘large’ effect size (Rice & Harris, 2005).”

“Rice and Harris (2005) interpret AUC values analogously to Cohen’s effect size d as small (AUC = 0.556–0.639; d = 0.20–0.50), moderate (AUC = 0.639–0.714; d = 0.50–0.80), and large (AUC ≥ 0.714; d ≥ 0.80).”

“The AUC statistic is derived from the area under the Receiver Operating Characteristic (ROC) curve, and represent the most widely used and recommended measure of predictive validity of risk assessments (e.g., Rice & Harris, 2005).”

“…such as to have, respectively, no, small, medium and large effects of X1, X3, X4 on Y and X2, where effect size = Cohen’s d = regression coefficient divided by square root of residual variance (d = 0, 0.2, 0.5 and 0.8 is no, small, medium and large, respectively) (Cohen 1969; Rice and Harris 2005).”

“Area under the ROC curve (AUC) values of .71 and greater correspond to excellent predictive validity, .64 to .71 indicate good accuracy, .55 to .63 indicate fair accuracy, and below .55 indicate poor accuracy (Desmarais & Singh, 2013; Rice & Harris, 2005).”

“For AUC values, 0.56 indicates a small effect, 0.64 indicates a moderate effect, and 0.71 indicates a large effect (Rice & Harris, 2005).”

“They indicate moderate links between non-abusive prepubescent sexual experiences and frequency of masturbation (during puberty) on the one hand and sexual recidivism on the other hand, whereas the AUC of sexual biographical index values indicates a strong effect (Rice & Harris, 2005).”

“71, indicating that CAT-IA had a large predictive effect (Rice and Harris, 2005).”

“As one would expect, the point-biserial correlations and ROC analyses converge on the same results, which indicates that these results have been successfully cross-validated (Rice & Harris, 2005).”

“We also employed Cohen’s d metric [24] to measure the effect size.”

“If primary studies reported other effect sizes in terms of Cohen’s d , the area under the curve (AUC), or the diagnostic odds ratio (DOR), a base-rate-cor-rected conversion (Salgado, 2018) was applied to account for the potential impact of base rates on the magnitude of r pb (Rice & Harris, 2005).”

“Cohen’s d indicates a small effect size of 0.2, a medium effect size of 0.5, and a large effect size of 0.8 (Rice and Harris, 2005).”

“Receiver operating characteristics (ROC) method has been a favored test of predictive accuracy in the risk assessment literature since it is robust against variation in reoffending base rates and provides clear numerical comparisons between studies (Rice & Harris, 2005).”

“Overall classification accuracy was evaluated, with values at or above .71 indicating large classification effects (Rice & Harris, 2005).”

“An AUC of .71 or greater is considered a large effect (Rice & Harris, 2005).”

“With C values of .50 representing chance level probability, we use Rice and Harris ’ (2005) AUC interpretation guidelines in which values of .56, .64, and .71 correspond to small, medium, and large effects, respectively.”

“According to the criteria of Rice and Harris (2005), the effects of the three domains were small (i.e. political ideology, safety personnel/measures, and campus life), while the effects of the six remaining domains were trivial.”

“…as r ¼ z p N , where N is the total number of participants and z is the z-value computed by SPSS, the groups had small to moderate differences in terms of pre-test conceptual, and procedural, and post-test conceptual understanding of differential calculus in both groups (Rice & Harris, 2005).”

“…interval (CI), there was a 70% probability that a randomly selected recidivist would obtain a higher total yLS/CMI score than a randomly selected nonrecidivist, 95% CI [0.67, 0.74], which is considered a medium effect (with the threshold for a large effect starting at .71; Rice & Harris, 2005).”

“…of the SOCCM social and psychological collateral consequences scales on recidivism was assessed by using Receiver Operating Characteristics (ROC) analyses, the preferred measure of predictive accuracy in forensic psychology research that is useful for dichotomous data (Rice & Harris, 2005).”

“For context, the results are interpreted according to the guidelines developed by Rice and Harris (2005) for determining small (AUC= .556), moderate (AUC= .639), and large (AUC= .714) effect sizes.”

“AUCs were interpreted using Rice and Harris’ (2005) criteria (0.56, 0.64, and 0.71 are respectively considered low, moderate, and high).”

“For reference, an AUC of 0.56 is considered small, 0.64 is medium, and large AUC is 0.71 (Rice & Harris, 2005).”

“The Cohen’s d value [52] indicates a large effect size (d = 2.”

“…an area under the curve (AUC) values that range from 0 to 1.0 representing the probability that a randomly selected feigner has a higher score than a randomly selected non-feigner; values of .56, .63, and .71 correspond to small, medium, and large effect sizes, respectively (Rice & Harris, 2005).”

“Following Rice and Harris (2005) [22], AUCs higher than 0.”

“…by the area under the curve (AUC), indicating the probability that a NoA-BPT score drawn at random from the first sample (i.e. patients unlikely having simultanagnosia) is higher than a score drawn at random from the second sample (i.e. patients likely having simultanagnosia) (Rice & Harris, 2005).”

“As such, applying the aforementioned Rice and Harris (2005) guidelines, extant static and dynamic measures have broadly moderate accuracy for sexual recidivism, and individual studies further tend to show dynamic measures to be incrementally predictive of sexual recidivism after controlling for…”

“AUC probabilities range from 58,4% (burglary) to 66.2% (weapons), suggesting overall poor to medium model performances (Rice & Harris, 2005)”

“It is possible, in fact, to establish a numerical correspondence between AUC values and the Cohen index of effect size, d (Cohen, 1988; Rice & Harris, 2005).”

“In addition, Rice and Harris (2005) provided further criteria for interpreting AUCs as follows: AUCs = .56 to .63 represent small effects, AUCs = .64 to 70 represent medium effects, and AUCs ⩾ .71 represent large effects.”

“…Harris (1997) Cross-validation and extension of the Violence Risk Appraisal Guide for child molesters and rapists Risk assessment; sex offenders 463 Rice and Harris (2005) Comparing effect sizes in follow-up studies: ROC area, Cohen’s d, and r General psych and law 458 Grisso et al.…”

“For context, these results are compared to minimum standards set forth by Rice and Harris (2005) for small (area under the curve [AUC] 5 .556; rpb 5 .100), medium (AUC 5 .639; rpb 5 .243), and large (AUC 5 .714; rpb 5 .371) effect sizes.”

“A general heuristic for interpreting effect sizes metrics when at least one of the variables is binary or categorical (due to a loss of variance), as in the case of Cramer’s V and Phi, indicates that .10 is a small effect, .24 is a medium effect, and .37 is a large effect (Rice & Harris, 2005).”

“AUC values of .56 are considered small, .64 medium, and .71 large effect sizes (Rice & Harris, 2005).”

“However, the AUC postmatch would ideally be 0.50, suggesting each offender had an equal chance of being in the participant or control group; no observable bias unduly influenced the results (Rice & Harris, 2005).”

“For example, the juvenile justice history domain scores were significantly associated with new arrests for youth in low crime counties (rpb ¼ .090, p .01), but this correlation did not even reach the level necessary to be classified as “weak” (Rice & Harris, 2005).”

“Several researchers recommend reporting area under the receiver operating characteristic curve (AUC) values as the preferred statistic of predictive or diagnostic accuracy (e.g., Mossman, 1994; Rice & Harris, 2005), and therefore, AUC values were also calculated in the present study.”

“correlations (rpb) were calculated for the different base rates in FSOs and FNSOs, based on a conversion formulae (after Rosental, 1991; Swets, 1986) provided by Rice and Harris [46].”

“The vast majority of the AUCs were in excess of .71, a value suggested by Rice and Harris (2005) as corresponding to “large” effect sizes.”

“The LSI-R’s predictive validity for general recidivism has a mean area under the curve (AUC) of .71 (r¼ .36), and for violent recidivism it has a mean AUC of .64 (r¼ .25; Andrews et al., 2006; AUCs calculated using the conversion table of Rice & Harris, 2005).”

“According to convention, a ROC AUC value of .56 indicates a small effect size, a value of .64 indicates a moderate effect size, and a value of .71 indicates a large effect size (Rice & Harris, 2005).”

“Second, the median AUCs of all four risk assessment instruments showed moderate discrimination (0.68–0.71), corresponding to a medium effect size [54].”

“…(thesis con fi rmed also by the completely non-overlapping to quantify the surgical treatment e ff ect, the Cohen’s d e ff ect size was Moreover, to quantify the surgical treatment effect, the Cohen’s d effect size was computed [69,70], resulting in 1.82 (>0.8), highlighting a large effect size.”

“The effect size (d) categories were 0.2, 0.5, and 0.8 for small, medium, and large effects, respectively [30].”

“Findings generally indicated the ML model could form groups consistent with predetermined credibility groups with clinically acceptable accuracy (Rice & Harris, 2005).”

“This corresponds to an effect size of d = 1.22 (Rice & Harris, 2005).”

“To verify whether the generated data are realistic, we computed the similarity be-tween the actual and generated measurements for each modality using Cohen’s d [21].”

“In short, larger AUC values correspond to more accurate model predictions (Mossman, 1994; Rice & Harris, 1995, 2005).”

“This argument is especially valid based on the effect size for expectancy in Table 2 ( r = 0.770, converted to Cohen's d = 2.41), which is large (Rice & Harris, 2005), and the fact that this test was planned as a manipulation check.”

“Effect sizes of .10 are considered small, .24 moderate, and .37 large (Rice & Harris, 2005).”

“To interpret the AUCs, we took as reference points: AUCs in the range 0.55–0.63 have low predictive value; those in the range 0.64–0.70 have moderate predictive value; and those above 0.71 have good predictive value, complemented by estimation of effect size using Cohen’s index [65,66].”