Transformations Related to the Angular and the Square Root

Abstract

The use of transformations to stabilize the variance of binomial or Poisson data is familiar(Anscombe [1], Bartlett [2, 3], Curtiss [4], Eisenhart [5]). The comparison of transformed binomial or Poisson data with percentage points of the normal distribution to make approximate significance tests or to set approximate confidence intervals is less familiar. Mosteller and Tukey [6] have recently made a graphical application of a transformation related to the square-root transformation for such purposes, where the use of "binomial probability paper" avoids all computation. We report here on an empirical study of a number of approximations, some intended for significance and confidence work and others for variance stabilization. For significance testing and the setting of confidence limits, we should like to use the normal deviate $K$ exceeded with the same probability as the number of successes $x$ from $n$ in a binomial distribution with expectation $np$, which is defined by $\frac{1}{2\pi} \int^K_{-\infty} e^{-\frac{1}{2}t^2} dt = \operatorname{Prob} \{x \leq k |mid \operatorname{binomial}, n, p\}.$ The most useful approximations to $K$ that we can propose here are $N$ (very simple), $N^+$ (accurate near the usual percentage points), and $N^{\ast\ast}$ (quite accurate generally), where $N = 2 (\sqrt{(k + 1)q} - \sqrt{(n - k)p)}.$ (This is the approximation used with binomial probability paper.) $N^+ = N + \frac{N + 2p - 1}{12\sqrt{E}},\quad E = \text{lesser of} np \text{and} nq, N^\ast = N + \frac{(N - 2)(N + 2)}{12} \big(\frac{1}{\sqrt{np + 1}} - \frac{1}{\sqrt{nq + 1}}\big), N^{\ast\ast} = N^\ast + \frac{N^\ast + 2p - 1}{12 \sqrt{E}}\cdot\quad E = \text{lesser of} np \text{and} nq.$ For variance stabilization, the averaged angular transformation $\sin^{-1}\sqrt{\frac{x}{n + 1}} + \sin^{-1} \sqrt{\frac{x + 1}{n+1}}$ has variance within $\pm 6%$ of $\frac{1}{n + \frac{1}{2}} \text{(angles in radians)}, \frac{821}{n + \frac{1}{2}} \text{(angles in degrees)},$ for almost all cases where $np \geq 1$. In the Poisson case, this simplifies to using $\sqrt{x} + \sqrt{x + 1}$ as having variance 1.