Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors

The probability of a sign error in the effects of enrollment to MIR is 0.00002 and 0.00003 for rearrests within one and four years, respectively.

“Following the procedure proposed by Gelman and Carlin (2014), we estimate the probability of a sign error in the effects of enrollment to MIR to be 0.00002 and 0.00003 for rearrests within one and four years, for example.”

The average potential exaggeration ratio is 1.2 in the effect of enrollment to MIR on rearrests within one and four years.

“Again, following the procedure proposed by Gelman and Carlin (2014), we estimate an average potential exaggeration ratio of 1.2 in the effect of enrollment to MIR on rearrests within one and four years.”

Significant findings from underpowered experiments are more likely to exaggerate or reverse the true effect (Type-M and Type-S errors).

“Moreover, significant findings from underpowered experiments are more likely to exaggerate or reverse the true effect - so-called Type-M (magnitude) and Type-S (sign) errors, respectively (Gelman and Carlin, 2014).”

Underpowered experiments increase the chances of exaggerated findings.

“Underpowered experiments make it more difficult to discern the difference between statistical noise and meaningful model improvements, and increase the chances of exaggerated findings.”

Studies with low power are more likely to produce false negatives and more likely to report exaggerated effect sizes when significant.

“Studies with low power are more likely to produce false negatives and more likely to report exaggerated effect sizes when significant (Gelman & Carlin, 2014).”

When samples are small, observed effects can misestimate both the magnitude and direction of a true effect.

“With small sample sizes, observed effects may misestimate both the size and the direction of any true underlying effect (Gelman & Carlin, 2014).”

Small samples, often under 100 participants, may contribute to heterogeneity.

“and from small sample sizes (N < 100) that are characteristic of the literature.”

Unreliable measurement can make estimates of relationships wrong in both magnitude and sign.

“unreliable measurement can yield estimates of relationships that are incorrect in both magnitude (e.g., an estimate is either larger or smaller than the true underlying relationship) and sign (e.g., a positive estimate is obtained when the true underlying relationship is negative) (Gelman & Carlin, 2014).”