Inferences from regression equations

If we are interested in \(\mathbf{b}^T\boldsymbol{\beta}\) (a linear function of the parameters), where \(\mathbf{b}\) is a given vector of constants, we will use the concept of pivotal functions that you have learnt in Statistical Inference.

Recall that a pivotal function is a function of observations and unobservable parameters such that the function’s probability distribution does not depend on the unknown parameters.

Pivotal function for a linear function of the parameters

\[\frac{(\mathbf{b}^T\boldsymbol{\hat{\beta}}-\mathbf{b}^T\boldsymbol{\beta})}{\sqrt{\frac{RSS}{n-p}\mathbf{b}^T(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{b}}} \]

is a pivotal function since

\[\frac{(\mathbf{b}^T\boldsymbol{\hat{\beta}}-\mathbf{b}^T\boldsymbol{\beta})}{\sqrt{\frac{RSS}{n-p}\mathbf{b}^T(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{b}}} \sim t(n-p),\]

where \(p\) is the number of parameters, \(n\) is the sample size and \(RSS\) is the residual sum-of-squares in a linear model.

The following result is stated without proof at the moment but we will revist this at the end of the course. For now, we will note that the estimated standard error of \(\mathbf{b}^T\boldsymbol{\beta}\) can be estimated by

\[\sqrt{\frac{RSS}{n-p}\mathbf{b}^T(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{b}}\]

The above result can be used to construct hypothesis tests and interval estimates for model parameters.

For information only

To see what is going on here, suppose you have random variables \(X_1, \ldots, X_n \sim N(\mu, \sigma^2)\), then you should be able to see that

\[\frac{1}{n}\sum{X_i}=\bar{X}\sim N\bigg(\mu,\frac{\sigma^2}{n} \bigg)\] and

\[=\frac{\bar{X}-\mu}{\sqrt{\frac{\sigma^2}{n}}}\sim N(0,1)\]

If we want to estimate \(\sigma^2\) with a small finite sample, then

\[=\frac{\bar{X}-\mu}{\sqrt{\frac{1}{n}\hat{\sigma}^2}}\sim t(n-1)\] essentially the t distribution is a modification of the normal distribution that occurs because of the finite sample size.