Recitation 4 Note

Yiming Gong

2021-07-27

Chapter 1 Method of Moments

1.1 Introduction

Method of moments estimation is based solely on the law of large numbers, which we repeat here:

Let $M_1, M_2,...$ be independent random variables having a common distribution possessing a mean $\mu_M$. Then the sample means converge to the distributional mean as the number of observations increase.

$$\bar{M}_n = \frac{1}{n}\sum_{i=1}^n M_i \rightarrow \mu_M \text{, as } n \rightarrow\infty$$

To show how the method of moments determines an estimator, we first consider the case of one parameter. We start with independent random variables $X_1, X_2,...$ chosen according to the probability density $f_X(x|\theta)$ associated to an unknown parameter value $\theta$. The common mean of the $X_i$, $\mu_X$, is a function $k(\theta)$ of $\theta$. For example, if the $X_i$ are continuous random variables, then

$$\mu_X = \int_{-\infty}^{\infty}xf_X(x|\theta) dx =k(\theta).$$

The law of large numbers states that

$$\bar{X}_n = \frac{1}{n}\sum_{i=1}^n X_i \rightarrow \mu_X \text{, as } n \rightarrow\infty$$

Thus, if the number of observations $n$ is large, the distributional mean, $\mu= k(\theta)$, should be well approximated by the sample mean, i.e.,

$$\bar{X} \approx k(\theta)$$

This can be turned into an estimator $\hat{\theta}$ by setting

$$\bar{X} \approx k(\hat{\theta})$$

and solving for $\hat{\theta}$.

1.2 The Procedure

More generally, for independent random variables $X_1, X_2,...$ chosen according to the probability distribution derived from the parameter value $\theta$ and $m$ a real valued function, if $k(\theta) = E_{\theta}m(X_1)$, then

$$\frac{1}{n}\sum_{i=1}^n m(X_i) \rightarrow k(\theta) \text{, as } n \rightarrow\infty$$

The method of moments results from the choices $m(x) = x^m$. Write

$$µ_m = EX^m = k_m(\theta).$$

for the $m$-th moment.

Our estimation procedure follows from these 4 steps to link the sample moments to parameter estimates.

• Step 1. If the model has $d$ parameters, we compute the functions $k_m$ for the first $d$ moments,

$$\mu_1 = k_1(\theta_1, \theta_2 ..., \theta_d), \mu_2 = k_2(\theta_1, \theta_2 ..., \theta_d), ..., \mu_d = k_d(\theta_1, \theta_2 ..., \theta_d),$$

obtaining $d$ equations in $d$ unknowns.

• Step 2. We then solve for the $d$ parameters as a function of the moments.

$$\theta_1 = g_1(\mu_1, \mu_2, ··· , \mu_d), \theta_2 = g_2(\mu_1, \mu_2, ··· , \mu_d), ..., \theta_d = g_d(\mu_1, \mu_2, ··· , \mu_d)$$

• Step 3. Now, based on the data $x = (x_1, x_2,...,x_n)$, we compute the first $d$ sample moments

$$\bar{x} = \frac{1}{n}\sum_{i=1}^n x_i, \bar{x^2} = \frac{1}{n}\sum_{i=1}^n x_i^2, \dots, \bar{x^d} = \frac{1}{n}\sum_{i=1}^n x_i^d,$$

• Step 4. We replace the distributional moments $\mu_m$ by the sample moments $x_m$, then the formulas for the method of moment estimators $(\hat{\theta}_1,\hat{\theta}_2,\dots,\hat{\theta}_d)$. For the data $x$, these estimates are

$$\hat{\theta}_1(x)=g_1(\bar{x},\bar{x^2},\dots,\bar{x^d}),\hat{\theta}_2(x)=g_1(\bar{x},\bar{x^2},\dots,\bar{x^d}),\dots,\hat{\theta}_d(x)=g_d(\bar{x},\bar{x^2},\dots,\bar{x^d}).$$

1.3 Example

Consider the uniform distribution on the interval $[a,b]$, $U(a,b)$. If $W\sim U(a,b)$ then we have

$$\mu_1 = \mathbb{E}[W] = \frac{1}{2}(a+b) \\ \mu_2 = \mathbb{E}[W^2] = \frac{1}{3}(a^2 + ab +b^2)$$

Solving these equations gives

$$\hat{a} = \mu_1 - \sqrt{3(\mu_2-\mu_1^2)} \\ \hat{b} = \mu_1 + \sqrt{3(\mu_2-\mu_1^2)}$$

Given a set of samples $w_1,w_2,\dots$ we can use the sample moments $\hat{\mu}_1$ and $\hat{\mu}_2$ in these formula in order to estimate $a$ and $b$.