LIMIT THEOREMS

Chapter 8, Sampling Distributions and Limit Theorems , delves into the crucial theoretical concepts that underpin almost all statistical inference. This chapter explains what happens to sample statistics, like the mean ($\bar{X}$), when the sample size ($n$) becomes very large, focusing on two foundational results: the Weak Law of Large Numbers (WLLN) and the Central Limit Theorem (CLT).

8.1 Setting the Stage: Multi-dimensional Continuous Variables

To study sample statistics, we first need a framework for discussing several random variables simultaneously. Since a sample $\{X_1, X_2, \dots, X_n\}$ consists of $n$ random variables, understanding their relationships is essential.

Concepts of Joint Distributions

1. Joint Distribution Function ($F$): This function gives the probability that all variables fall below specified values: $F(x_1, x_2, \dots, x_n) = P(X_1 \leq x_1, X_2 \leq x_2, \dots, X_n \leq x_n)$

2. Joint Density ($f$): For continuous variables, probability is found by integrating the joint density function $f(x_1, \dots, x_n)$ over the desired region.

3. Independence is Key: If the variables are mutually independent (a fundamental assumption for most random samples), their joint density function is simply the product of their individual marginal densities: $f(x_1, x_2, \dots, x_n) = f_{X_1}(x_1) f_{X_2}(x_2) \cdots f_{X_n}(x_n)$

Order Statistics

When we observe a sample, arranging the values from smallest to largest gives us the order statistics. $X_{(1)} \le X_{(2)} \le \cdots \le X_{(n)}$.

8.2 The Weak Law of Large Numbers (WLLN)

The WLLN is the first major limit theorem, providing formal confirmation of the common intuition that as you gather more data, the sample average gets closer to the true population average.

Concept: Convergence in Probability

The WLLN Theorem

If $X_1, X_2, \dots$ are i.i.d. random variables with finite mean $\mu$ and finite variance $\sigma^2$, then the sample mean $\bar{X}_n$ converges in probability to $\mu$: $\lim_{n \to \infty} P(|\bar{X}_n - \mu| > \epsilon) = 0$

Application: Sample Proportion

The WLLN formally validates the intuitive link between theoretical probability and observed frequency. The sample proportion ($\hat{p}$) converges to the true probability ($p$).

8.3 Convergence in Distribution

Convergence in distribution ($\xrightarrow{d}$) describes how the shape of the distribution of a random variable sequence approaches a limiting shape.

The Power of Moment Generating Functions (MGFs)

The most practical way to prove convergence in distribution is often through Moment Generating Functions (MGFs).

MGF Convergence Theorem: If the MGFs $M_n(t)$ of a sequence $X_n$ exist and converge to $M(t)$, and $M(t)$ is the MGF of some random variable $X$, then $X_n$ converges in distribution to $X$.

Example: Proving that Binomial approaches Poisson when $n \to \infty$ and $p \to 0$ ($np = \lambda$) is done by showing the limit of the Binomial MGF equals the Poisson MGF $e^{\lambda(e^t - 1)}$.

8.4 The Central Limit Theorem (CLT)

The Central Limit Theorem is arguably the most fundamental result in statistics, explaining why the Normal distribution appears so frequently.

The CLT Theorem

Insight: The sampling distribution of the sample mean approaches a Normal distribution, even if the original population distribution is highly non-normal.

Application: Normal Approximation

Continuity Correction

When approximating discrete, integer-valued RVs (like Binomial), extend the interval by 0.5 units: $P(a < S_n \leq b) \approx P\left( \frac{a + 0.5 - n\mu}{\sigma \sqrt{n}} < Z \leq \frac{b + 0.5 - n\mu}{\sigma \sqrt{n}} \right)$