Confidence intervals: coverage and bias

A confidence interval moves when the sample changes; the population mean stays fixed. The 95% describes how often the procedure captures that mean across repeated samples under its assumptions. After one interval is observed, it either contains the mean or misses it. NIST explicitly distinguishes these two perspectives.

Here the model is deliberately simple: n independent observations, each normally distributed with mean μ and known standard deviation σ > 0. These are mathematical assumptions, not findings about a population of learners.

Write s = σ/√n for the mean’s standard error. The standardised error Z = (X̄ − μ)/s has a standard normal distribution. If Φ is its cumulative distribution function and z = Φ⁻¹(0.975) ≈ 1.95996, then P(−z ≤ Z ≤ z) = 0.95. Rearranging the same event gives μ ∈ [X̄ − zs, X̄ + zs]. Coverage comes from this distributional step, not from the interval’s appearance.

For an original sensitivity model, shift the centre by δ while retaining the same width: Iδ = [X̄ + δ − zs, X̄ + δ + zs]. Define standardised bias b = δ/s. Containment now requires −z − b ≤ Z ≤ z − b, so coverage = Φ(z − b) − Φ(−z − b).

At b = 0 the nominal coverage is recovered. Increasing |b| moves the acceptance window away from the normal curve’s centre and reduces the probability inside it. This is a mathematical consequence of the constructed model; a displayed curve is not an observed success rate.

A smaller standard error does not remove a fixed shift δ. As n grows, the interval narrows while |δ|/s grows. Precision around a displaced centre can therefore accompany poorer coverage of the intended target.

Read any interval together with its target, sampling assumptions and construction. The bias model isolates one failure mechanism; it does not quantify every consequence of dependence, an unknown σ or a different population distribution.