Probabilities of Type I and Type II Errors (9709 PS2 §6.5)

Core Concept

The critical region and its role

The critical region (rejection region) of a test is determined before any data are collected. It consists of all values of the test statistic that would lead to rejection of $H_0$.

• P(Type I error) = P(test statistic falls in the critical region $\mid$ $H_0$ is true).
  For discrete distributions this equals the actual significance level of the test, which may be less than the nominal level.

• P(Type II error) = P(test statistic does not fall in the critical region $\mid$ $H_1$ is true at a specific alternative value).
  A specific alternative value of the parameter must be stated — without it, P(Type II error) cannot be computed.

Key point for discrete distributions

Because the binomial and Poisson distributions are discrete, the critical region is chosen so that P(Type I error) $\leq$ significance level (e.g. $\leq 0.05$). The actual P(Type I error) is usually strictly less than the nominal level and is found by direct probability evaluation using tables or the distribution formula.

Key point for the normal distribution

For a normal-based $z$-test the critical region is defined by a $z$-value (e.g. $z > 1.645$ for a one-tailed 5% test). P(Type I error) equals the nominal significance level exactly. P(Type II error) is found by standardising using the true population mean stated in $H_1$.