5.1 Past FYE Problems

Exercise 5.1 (By Robert, FYE 2020) Consider the general linear model \[\begin{equation} \mathbf{Y}=\mathbf{X}\boldsymbol{\beta}+\boldsymbol{\epsilon} \tag{5.1} \end{equation}\] where \(\mathbf{Y}\) is an \(n\)-dimensional vector of observations, \(\mathbf{X}\) is an \(n\times p\) matrix of full rank, \(\boldsymbol{\beta}\) is a \(p\)-dimensional unknown parameter vector, and \(\boldsymbol{\epsilon}\) are n-dimensional random errors that have the multivariate normal distribution \[\begin{equation} \boldsymbol{\epsilon}\sim N(\mathbf{0},\boldsymbol{\Gamma}) \tag{5.2} \end{equation}\] You may assume that \(\boldsymbol{\Gamma}\) is invertible.

  • (i). (25%) What is the distribution of \(\boldsymbol{\Gamma}^{-1/2}\mathbf{Y}\), where \(\boldsymbol{\Gamma}^{-1/2}\) is a matrix square root of \(\boldsymbol{\Gamma}^{-1}\)?

  • (ii). (25%) Derive an expression for the weighted least squares estimator of \(\boldsymbol{\beta}\), which is denoted by \(\hat{\boldsymbol{\beta}}\).

  • (iii). (25%) Derive the mean and variance of \(\hat{\boldsymbol{\beta}}\).

  • (iv). (25%) Explain how to test the null hypothesis that \(\beta_i=0\) against the alternative that \(\beta_i\neq 0\). Here, \(\beta_i\) denotes the \(i\)th component of \(\boldsymbol{\beta}\).

Proof. (i). Since \(\mathbf{Y}=\mathbf{X}\boldsymbol{\beta}+\boldsymbol{\epsilon}\) and \(\boldsymbol{\epsilon}\sim N(\mathbf{0},\boldsymbol{\Gamma})\), we have \(\mathbf{Y}\sim N(\mathbf{X}\boldsymbol{\beta},\boldsymbol{\Gamma})\). Thus, \(\boldsymbol{\Gamma}^{-1/2}\mathbf{Y}\) also has a multivariate normal distribution. We left with finding the mean and variance. \[\begin{equation} \begin{split} &E(\boldsymbol{\Gamma}^{-1/2}\mathbf{Y})=\boldsymbol{\Gamma}^{-1/2}E(\mathbf{Y})=\boldsymbol{\Gamma}^{-1/2}\mathbf{X}\boldsymbol{\beta}\\ &Var(\boldsymbol{\Gamma}^{-1/2}\mathbf{Y})=\boldsymbol{\Gamma}^{-1/2}Var(\mathbf{Y})(\boldsymbol{\Gamma}^{-1/2})^T=\mathbf{I} \end{split} \tag{5.3} \end{equation}\] Thus, \(\boldsymbol{\Gamma}^{-1/2}\mathbf{Y}\sim N(\boldsymbol{\Gamma}^{-1/2}\mathbf{X}\boldsymbol{\beta},\mathbf{I})\).

(ii). From the OLS model, that is, if \(\mathbf{Y}^*=\mathbf{X}^*\boldsymbol{\beta}+\boldsymbol{\epsilon}^*\) and \(\boldsymbol{\epsilon}^*\sim N(\mathbf{0},\mathbf{I})\), then \(\hat{\boldsymbol{\beta}}^{OLS}=((\mathbf{X}^*)^T\mathbf{X}^*)^{-1}(\mathbf{X}^*)^T\mathbf{Y}^*\).

Define \(\mathbf{Y}^*=\boldsymbol{\Gamma}^{-1/2}\mathbf{Y}\), \(\mathbf{X}^*=\boldsymbol{\Gamma}^{-1/2}\mathbf{X}\) and \(\boldsymbol{\epsilon}^*=\boldsymbol{\Gamma}^{-1/2}\boldsymbol{\epsilon}\), we can change the original weighted least square problem to the OLS problem as it is easy to check the newly defined model satisfies the OLS setting. Thus, plug in the form of \(\mathbf{X}^*,\mathbf{Y}^*\), we have \[\begin{equation} \hat{\boldsymbol{\beta}}^{WLS}=(\mathbf{X}^T\boldsymbol{\Gamma}^{-1}\mathbf{X})^{-1}\mathbf{X}^T\boldsymbol{\Gamma}^{-1}\mathbf{Y} \tag{5.4} \end{equation}\]

(iii). We have \[\begin{equation} \begin{split} &E(\hat{\boldsymbol{\beta}})=(\mathbf{X}^T\boldsymbol{\Gamma}^{-1}\mathbf{X})^{-1}\mathbf{X}^T\boldsymbol{\Gamma}^{-1}E(\mathbf{Y})=\boldsymbol{\beta}\\ &Var(\hat{\boldsymbol{\beta}})=(\mathbf{X}^T\boldsymbol{\Gamma}^{-1}\mathbf{X})^{-1}\mathbf{X}^T\boldsymbol{\Gamma}^{-1}Var(\mathbf{Y})\boldsymbol{\Gamma}^{-1}\mathbf{X}(\mathbf{X}^T\boldsymbol{\Gamma}^{-1}\mathbf{X})^{-1}=(\mathbf{X}^T\boldsymbol{\Gamma}^{-1}\mathbf{X})^{-1} \end{split} \tag{5.5} \end{equation}\]

(iv). Since \(\mathbf{Y}\) has a normal distribution, we have \(\hat{\boldsymbol{\beta}}\sim N(\boldsymbol{\beta},(\mathbf{X}^T\boldsymbol{\Gamma}^{-1}\mathbf{X})^{-1})\). Therefore, the \(j\)th component of \(\hat{\boldsymbol{\beta}}\), denoted as \(\hat{\beta}_j\), follows a normal distribution \(\hat{\beta}_j\sim N(\beta_j,\sigma^2_j)\), where \(\sigma^2_j\) is the \(j\)th diagnoal component of matrix \((\mathbf{X}^T\boldsymbol{\Gamma}^{-1}\mathbf{X})^{-1}\).

Thus, the test statistic is \(\mathbf{z}=\frac{\hat{\beta}_j}{sigma_j}\), it is compared with \(Z_{\frac{\alpha}{2}}\), where \(Z\) is standard normal distributed r.v. and \(\alpha\) is confidence level. We can also calculate the p-value of \(\frac{\hat{\beta}_j}{sigma_j}\) with respect to standard normal distribution and making decisions accordingly.

Exercise 5.2 (By Rajarshi, FYE 2018) 1. (50%) Consider the model \[\begin{equation} y_i=\alpha+\sin(x_i+\beta)+\epsilon_i,\quad \epsilon_i\stackrel{i.i.d.}{\sim}N(0,\sigma^2),\quad i=1,\cdots,n \tag{5.6} \end{equation}\] Here \(y_1,\cdots,y_n\) are the responses and \(x_1,\cdots,x_n\) are the predictors. Assume \(\sum_{i=1}^n\sin(x_i)=\sum_{i=1}^n\cos(x_i)=\sum_{i=1}^nsin(2x_i)=0\).

(a). (20%) Represent (5.6) in the form of a linear model with vector of regression coefficients \((\alpha,\cos(\beta),\sin(\beta))\). (Hint: \(\sin(A+B)=\sin(A)\cos(B)+\cos(A)\sin(B)\).)

(b). (30%) Provide the explicit form of the least squares estimators of the parameters in the linear model representation.

  1. (50%) Consider the two-way ANOVA model \(y_{ij}=\mu+\zeta_i+\lambda_j+\epsilon_{ij},\epsilon_{ij}\stackrel{i.i.d.}{\sim} N(0,\sigma^2)\), \(i=1,\cdots,3\) and \(j=1,\cdots,4\).

(a). (30%) Express the above equations in the form of \(\mathbf{y}=\mathbf{X}\boldsymbol{\gamma}+\boldsymbol{\epsilon}\), and specify \(\mathbf{y},\mathbf{X}\) and \(\boldsymbol{\gamma}\). What is the \(rank(\mathbf{X})\)? How many constraints are required to make the predictor matrix full column rank?

(b). (20%) Is \(\zeta_1-\zeta_3+\lambda_4-\lambda_1\) estimable?

Exercise 5.3 (By Juhee, FYE 2015) Consider a two-way ANOVA model \[\begin{equation} y_{i,j}=\alpha_i+\tau_j+\epsilon_{i,j}\quad i=1,2;\quad j=1,2 \tag{5.7} \end{equation}\] where the parameters \(\alpha_i\) and \(\tau_j\) are unknown. For the errors, assume that \(\epsilon_{i,j}\stackrel{i.i.d.}{\sim}N(0,\sigma^2)\).

  1. (10%) Express these observations into a general linear model form, \(\mathbf{y}=\mathbf{X}\boldsymbol{\beta}+\boldsymbol{\epsilon}\), where \(\boldsymbol{\beta}=[\alpha_1,\alpha_2,\tau_1,\tau_2]^T\). That is, specify \(\mathbf{y},\mathbf{X}\) and \(\boldsymbol{\epsilon}\). Identifythe rank of \(\mathbf{X}\).

  2. (15%) Show that for this design, if a function of \(\boldsymbol{\beta}\), \(c_1\alpha_1+c_2\alpha_2+d_1\tau_1+d_2\tau_2\) is estimatable, then \(c_1+c_2=d_1+d_2\).

  3. (40%) Consider the function of \(\boldsymbol{\beta}\) defined by \(\alpha_1-\alpha_2\). A g-inverse of the matrix \(\mathbf{X}^T\mathbf{X}\) for the model in (1) is given by \[\begin{equation} (\mathbf{X}^T\mathbf{X})^-=\frac{1}{4}\begin{pmatrix} 0 & 0 & 0 & 0 \\ 0 & 4 & -2 & -2 \\ 0 & -2 & 3 & 1 \\ 0 & -2 & 1 & 3 \end{pmatrix} \tag{5.8} \end{equation}\] Find the best linear unbiased estimator of the function. Also, find its probability distribution (assume that \(\sigma^2\) is known for this part).

  4. (15%) Is \(y_{11}\) the best linear unbiased estimator of \(\alpha_1+\tau_1\)? Explain why or why not.

  5. (20%) Assume that \(\sigma^2\) is unknown. Derive a test for \(H_0:\alpha_1=\alpha_2\) at significance level \(\gamma\). That is, describe a test statistic, its distribution under the null and the alternative, and a rejection region. Expressing SSE in the form of vectors is enough for an answer.

Proof. 1. \(\mathbf{y}=\mathbf{X}\boldsymbol{\beta}+\boldsymbol{\epsilon}\) where \[\begin{equation} \mathbf{y}=\begin{pmatrix} y_{11}\\ y_{12}\\ y_{21}\\ y_{22}\end{pmatrix}, \quad \mathbf{X}=\begin{pmatrix} 1 & 0 & 1 & 0 \\1 & 0 & 0 & 1\\ 0 & 1 & 1 & 0\\ 0 & 1 & 0 & 1 \end{pmatrix}, \quad \boldsymbol{\beta}=\begin{pmatrix} \alpha_{1}\\ \alpha_{2}\\ tau_{1}\\ tau_{2}\end{pmatrix},\quad \boldsymbol{\epsilon}=\begin{pmatrix} \epsilon_{11}\\ \epsilon_{12}\\ \epsilon_{21}\\ \epsilon_{22}\end{pmatrix} \tag{5.9} \end{equation}\]

  1. \(\boldsymbol{\lambda}^T\boldsymbol{\beta}\) is estimable iff \(\exists \alpha\) s.t. \(\boldsymbol{\lambda}=\mathbf{X}^T\mathbf{a}\). That is \[\begin{equation} \begin{split} &a_1+a_2=c_1\\ &a_3+a_4=c_2\\ &a_1+a_3=d_1\\ &a_2+a_4=d_2 \end{split} \tag{5.10} \end{equation}\] Thus, \(a_1+a_2+a_3+a_4=c_1+c_2=d_1+d_2\) for any estimable \(\boldsymbol{\lambda}^T\boldsymbol{\beta}\).

  2. \[\begin{equation} \boldsymbol{\lambda}^T\boldsymbol{\beta}=\begin{pmatrix} 1 & -1 & 0 & 0 \end{pmatrix} \begin{pmatrix} \alpha_{1}\\ \alpha_{2}\\ \tau_{1}\\ \tau_{2}\end{pmatrix}=\begin{pmatrix} \alpha_1-\alpha_2 \end{pmatrix} \end{equation}\] LSE of \(\boldsymbol{\beta}\) is \[\begin{equation} \hat{\boldsymbol{\beta}}=(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}=\frac{1}{4}\begin{pmatrix} 0\\ 2(y_{21}+y_{22})-2(y_{11}+y_{12})\\ 3y_{11}+y_{12}+y_{21}-y_{22} \\ y_{11}+3y_{12}-y_{21}+y_{22} \end{pmatrix} \tag{5.11} \end{equation}\] \(\boldsymbol{\lambda}^T\hat{\boldsymbol{\beta}}\) is the BLUE of \(\boldsymbol{\lambda}^T\boldsymbol{\beta}\) where \[\begin{equation} \boldsymbol{\lambda}^T\hat{\boldsymbol{\beta}}=\frac{1}{2}(y_{21}+y_{22})-2(y_{11}+y_{12}) \tag{5.12} \end{equation}\]
    From the lecture, \(\boldsymbol{\lambda}^T\hat{\boldsymbol{\beta}}\sim N(\boldsymbol{\lambda}^T\boldsymbol{\beta},\sigma^2\boldsymbol{\lambda}^T\boldsymbol{\lambda})\). We can easily verify \(\boldsymbol{\lambda}^T\boldsymbol{\lambda}=1\).

  3. \(\alpha_1+\tau_1=\boldsymbol{\lambda}^T\boldsymbol{\beta}\) with \(\boldsymbol{\lambda}^T=(1,0,1,0)^T\). From part 2, we know \(\alpha_1+\tau_1\) is estimable. The BLUE of \(\boldsymbol{\lambda}^T\boldsymbol{\beta}\) is \(\boldsymbol{\lambda}^T\hat{\boldsymbol{\beta}}\) and unique. From part 3, the BLUE of \(\boldsymbol{\lambda}^T\boldsymbol{\beta}\) is \[\begin{equation} \frac{3y_{11}+y_{12}+y_{21}-y_{22}}{4}\neq y_{11} \tag{5.13} \end{equation}\]

  4. Reject \(H_0\) if \[\begin{equation} F=\frac{(\boldsymbol{\lambda}^T\hat{\boldsymbol{\beta}}-0)^TH^{-1}(\boldsymbol{\lambda}^T\hat{\boldsymbol{\beta}}-0)}{SSE/(N-r)}>F_{s,N-r,\alpha} \tag{5.14} \end{equation}\] where \[\begin{equation} \boldsymbol{\lambda}^T\hat{\boldsymbol{\beta}}=\frac{y_{11}+y_{12}-y_{21}-y_{22}}{2},\quad H^{-1}=1,\quad SSE=\sum_{i,j}(y_{ij}-\bar{y}_{ij})^2 \tag{5.14} \end{equation}\] \(s=1,N=4\) and \(r=rank(\mathbf{X})=3\). The test statistic \(F\) follows the central F distribution under \(H_0\) with degree of freedom 1 and 1. If \(H_0\) is false, F follows the noncentral F distribution with the same degrees of freedom and the noncentrality parameter,
    \[\begin{equation} \delta=\frac{(\boldsymbol{\lambda}^T\boldsymbol{\beta})^2}{2\sigma^2}=\frac{(\alpha_1-\alpha_2)^2}{2\sigma^2} \tag{5.15} \end{equation}\]

Exercise 5.4 (By Juhee, FYE 2015 Retake) Two groups of \(n\) observations are fitted using the following fixed-effects model: \[\begin{equation} y_{ij}=\mu+\theta_i+\epsilon_{ij},\quad \epsilon_{ij}\stackrel{i.i.d.}{\sim} N(0,\sigma^2) \tag{5.16} \end{equation}\] where \(i=1,2\) and \(j=1,\cdots,n\). To avoid identifiability issues, we set \(\sum_{i=1}^2\theta_i=0\) and remove \(\theta_2\) from the above formulation, that is, the model parameters are \(\mu\) and \(\theta_1\).

  1. (20%) Express these observations into a general linear model form, \(\mathbf{y}=\mathbf{X}\boldsymbol{\beta}+\boldsymbol{\epsilon}\), where \(\boldsymbol{\beta}=[\mu,\theta_1]^T\). That is, specify \(\mathbf{y},\mathbf{X}\) and \(\boldsymbol{\epsilon}\). Identifythe rank of your design matrix.

  2. (20%) Derive the least squares estimates (LSE) of \(\mu\) and \(\theta_1\). Find their joint distribution.

  3. (15%) Derive the LSE of \(\sigma^2\) and find its distribution.

  4. (15%) Express \(\theta_1\) as a linear function of \(\boldsymbol{\beta}\). Is \(\theta_1\) estimatable? Justify your answer.

  5. (30%) Derive an \(F-\)test for \(H_0:\theta_1=0\) against \(H_a:\theta_1\neq 0\). Show the details. Be concise in your expression for the test statistic and specify its number of degrees of freedom under the null.

Proof. 1. \[\begin{equation} \begin{pmatrix} y_{11}\\ \vdots\\ y_{1n}\\ y_{21}\\ \vdots\\ y_{2n} \end{pmatrix}=\begin{pmatrix} 1 & 1\\ \vdots & \vdots \\ 1 & 1 \\ 1 & 0 \\ \vdots & \vdots \\ 1 & 0 \end{pmatrix} \begin{pmatrix} \mu \\ \theta_1 \end{pmatrix} + \begin{pmatrix} \epsilon_{11}\\ \vdots\\ \epsilon_{1n}\\ \epsilon_{21}\\ \vdots\\ \epsilon_{2n} \end{pmatrix} \tag{5.17} \end{equation}\] The rank of \(\mathbf{X}\) is 2 (full rank).

  1. \[\begin{equation} \hat{\boldsymbol{\beta}}=(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{Y} =\begin{pmatrix} \bar{y}_2 \\ \bar{y}_1-\bar{y}_2 \end{pmatrix} \end{equation}\] The distribution of \(\hat{\boldsymbol{\beta}}\) is \(\hat{\boldsymbol{\beta}}\sim N(\boldsymbol{\beta},\sigma^2(\mathbf{X}^T\mathbf{X})^{-1})\), where \[\begin{equation} (\mathbf{X}^T\mathbf{X})^{-1}=\frac{1}{n}\begin{pmatrix} 1 & -1 \\ -1 & 2 \end{pmatrix} \tag{5.18} \end{equation}\]

  2. \(SSE=(\mathbf{y}-\hat{\mathbf{y}})^T(\mathbf{y}-\hat{\mathbf{y}})=\sum_{i,j}(y_{ij}-\hat{y}_{ij})^2\) where \(\hat{y}_{1j}=\bar{y}_1\) and \(\hat{y}_{2j}=\bar{y}_2\). So, \[\begin{equation} \hat{\sigma}^2=\frac{SSE}{2n-2}=\frac{\sum_{i,j}(y_{ij}-\hat{y}_{ij})^2}{2n-2} \tag{5.19} \end{equation}\] and \(\frac{(2n-2)\hat{\sigma}^2}{\sigma^2}\sim\chi^2(2n-2)\).

  3. \(\mathbf{X}\) has the full rank so any linear function of \(\boldsymbol{\beta}\) is estimable. \(\theta_1=\begin{pmatrix} 0 & 1 \end{pmatrix}\boldsymbol{\beta}\).

  4. \(\hat{\theta}_1\sim N(\theta_1,\frac{2\sigma^2}{n})\). Thus, under \(H_0:\theta=0\), \[\begin{equation} F=\frac{(\hat{\theta}_1)^2}{2\hat{\sigma}^2}\sim F(1,2n-2) \tag{5.20} \end{equation}\]

Exercise 5.5 (By Abel, FYE 2014) Consider the random blocks model \(y_{i,j}=\mu+\alpha_i+\beta_j+\epsilon_{i,j}\), with \(\epsilon_{i,j}\sim N(0,\sigma^2)\), for \(i=1,2,3\) and \(j=1,2,3\).

  1. (15%) Write this model in matrix form as \(\mathbf{y}=\mathbf{X}\boldsymbol{\theta}+\boldsymbol{\epsilon}\), where \(\boldsymbol{\theta}=(\mu,\alpha_1,\alpha_2,\alpha_3,\beta_1,\beta_2,\beta_3)\).

  2. What is the rank of the design matrix \(\mathbf{X}\)? Justify your answer.

  3. List one possible set of constraints on \(\boldsymbol{\theta}\) that makes the corresponding constrained least squares estimator be unique, and provide the solution to the normal equations under those constraints. (Hint: Rather than solving the normal equations, try to propose a solution and show that it satisfies the normal equations.)

  4. Suppose that we are interested in testing the hypothesis \(H_0:\alpha_1=\alpha_2=\alpha_3=0\) against the alternative \(H_a:\) at least one \(\alpha_i\neq 0\).

  • (a). (10%) Write these hypotheses as a general linear hypotheses \(H_0: \mathbf{K}^T\boldsymbol{\theta}=\mathbf{m}\) and \(H_a: \mathbf{K}^T\boldsymbol{\theta}\neq\mathbf{m}\). Make sure to show that \(\mathbf{K}^T\boldsymbol{\theta}=\mathbf{m}\) is testable!

  • (b). (30%) Describe a test (i.e., a statistic, its distribution under the null and the alternative, and a rejection region) for the general linear hypotheses described before.

Proof. 1. \[\begin{equation} \mathbf{y}=\begin{pmatrix} y_{1,1}\\ y_{1,2}\\ y_{1,3}\\ y_{2,1}\\ y_{2,2}\\y_{2,3}\\y_{3,1}\\y_{3,2}\\y_{3,3}\end{pmatrix}\quad \mathbf{X}=\begin{pmatrix} 1 & 1 & 0 & 0 & 1 & 0 & 0\\1 & 1 & 0 & 0 & 0 & 1 & 0\\ 1 & 1 & 0 & 0 & 0 & 0 & 1\\ 1 & 0 & 1 & 0 & 1 & 0 & 0\\ 1 & 0 & 1 & 0 & 0 & 1 & 0\\ 1 & 0 & 1 & 0 & 0 & 0 & 1 \\ 1 & 0 & 0 & 1 & 1 & 0 & 0\\ 1 & 0 & 0 & 1 & 0 & 1 & 0 \\ 1 & 0 & 0 & 1 & 0 & 0 & 1 \end{pmatrix} \quad \boldsymbol{\epsilon}\sim N(\mathbf{0},\sigma^2\mathbf{I}) \tag{5.21} \end{equation}\]

  1. The rank is 5 (the fourth column of \(\mathbf{X}\) is linearly dependent of the first 3, while the last column is linearly dependent of the first, fifth and sixth).

  2. The simplest set of constraints is \(\alpha_1=0\) and \(\beta_1=0\). However, a simpler set of constraints to work with are \(\sum_i\alpha_i=0\) and \(\sum_j\beta_j=0\).

The normal equations in this case correspond to \(\mathbf{X}^T\mathbf{X}=\mathbf{X}^T\mathbf{Y}\) where \[\begin{equation} \mathbf{X}^T\mathbf{X}=\begin{pmatrix} 9 & 3 & 3 & 3 & 3 & 3 & 3\\3 & 3 & 0 & 0 & 1 & 1 & 1\\ 3 & 0 & 3 & 0 & 1 & 1 & 1\\ 3 & 0 & 0 & 3 & 1 & 1 & 1\\ 3 & 1 & 1 & 1 & 3 & 0 & 0\\ 3 & 1 & 1 & 1 & 0 & 3 & 0 \\ 3 & 1 & 1 & 1 & 0 & 0 & 3\end{pmatrix}\quad \mathbf{X}^T\mathbf{y}=\begin{pmatrix} y_{\cdot,\cdot}\\ y_{1,\cdot}\\ y_{2,\cdot}\\ y_{3,\cdot}\\ y_{\cdot,1}\\y_{\cdot,2}\\y_{\cdot,3}\end{pmatrix} \tag{5.22} \end{equation}\] Because of the interpretation of the coefficients, it is natural to assume that the solution is of the form \[\begin{equation} \begin{pmatrix} \hat{\mu}\\ \hat{\alpha}_1\\ \hat{\alpha}_2\\ \hat{\alpha}_3\\ \hat{\beta}_1\\ \hat{\beta}_2 \\ \hat{\beta}_3\end{pmatrix}=\begin{pmatrix} \bar{y}_{\cdot,\cdot}\\ \bar{y}_{1,\cdot}-\bar{y}_{\cdot,\cdot} \\ \bar{y}_{2,\cdot}-\bar{y}_{\cdot,\cdot} \\ \bar{y}_{3,\cdot}-\bar{y}_{\cdot,\cdot} \\ \bar{y}_{\cdot,1}-\bar{y}_{\cdot,\cdot} \\ \bar{y}_{\cdot,2}-\bar{y}_{\cdot,\cdot} \\ \bar{y}_{\cdot,3}-\bar{y}_{\cdot,\cdot} \end{pmatrix} \tag{5.23} \end{equation}\] which indeed satisfy the normal equations. For example, for the first row of the normal equations, \[\begin{equation} \begin{split} 9\bar{y}_{\cdot,\cdot}+&3(\bar{y}_{1,\cdot}-\bar{y}_{\cdot,\cdot})+3(\bar{y}_{2,\cdot}-\bar{y}_{\cdot,\cdot})+3(\bar{y}_{3,\cdot}-\bar{y}_{\cdot,\cdot})+3(\bar{y}_{\cdot,1}-\bar{y}_{\cdot,\cdot})+3(\bar{y}_{\cdot,2}-\bar{y}_{\cdot,\cdot})+3(\bar{y}_{\cdot,3}-\bar{y}_{\cdot,\cdot})\\ &=-y_{\cdot,\cdot}+y_{1,\cdot}+y_{2,\cdot}+y_{3,\cdot}+y_{\cdot,1}+y_{\cdot,2}+y_{\cdot,3}\\ &=-y_{\cdot,\cdot}+2y_{\cdot,\cdot}=y_{\cdot,\cdot} \end{split} \tag{5.24} \end{equation}\] while the second row
\[\begin{equation} \begin{split} 3\bar{y}_{\cdot,\cdot}+&3(\bar{y}_{1,\cdot}-\bar{y}_{\cdot,\cdot})+(\bar{y}_{\cdot,1}-\bar{y}_{\cdot,\cdot})+(\bar{y}_{\cdot,2}-\bar{y}_{\cdot,\cdot})+(\bar{y}_{\cdot,3}-\bar{y}_{\cdot,\cdot})\\ &=-3\bar{y}_{\cdot,\cdot}+3\bar{y}_{1,\cdot}+\bar{y}_{\cdot,1}+\bar{y}_{\cdot,2}+\bar{y}_{\cdot,3}\\ &=-\frac{1}{3}y_{\cdot,\cdot}+y_{1,\cdot}+\frac{1}{3}(y_{\cdot,1}+y_{\cdot,2}+y_{\cdot,3})=\bar{y}_{1,\cdot} \end{split} \tag{5.24} \end{equation}\] the remaining of the rows are analogous to the second.

  1. (a). Let \[\begin{equation} \mathbf{K}^T=\begin{pmatrix} 0 & 1 & -1 & 0 & 0 & 0 & 0 \\ 0 & 0 & 1 & -1 & 0 & 0 & 0 \end{pmatrix}\quad \mathbf{m}=\begin{pmatrix} 0 \\ 0 \end{pmatrix} \tag{5.25} \end{equation}\] Note that the columns of \(\mathbf{K}\) are linearly independent and that each of the rows of \(\mathbf{K}^T\boldsymbol{\theta}\) is estimable. This is not the only possible configuration, another option is \[\begin{equation} \mathbf{K}^T=\begin{pmatrix} 0 & 1 & -1 & 0 & 0 & 0 & 0 \\ 0 & 1 & 0 & -1 & 0 & 0 & 0 \end{pmatrix}\quad \mathbf{m}=\begin{pmatrix} 0 \\ 0 \end{pmatrix} \tag{5.26} \end{equation}\]
(b). First focus on the statistic. The general result is \(\mathbf{K}^T\hat{\boldsymbol{\theta}}\sim N(\mathbf{K}^T\boldsymbol{\theta},\sigma^2\mathbf{K}^T(\mathbf{X}^T\mathbf{X})^-\mathbf{K})\). However, working with the full form is cumbersome, as it involves a generalized inverse of a large matrix. Instead, note that \[\begin{equation} \mathbf{K}^T\hat{\boldsymbol{\theta}}=\begin{pmatrix} \bar{y}_{1,\cdot}-\bar{y}_{2,\cdot} \\ \bar{y}_{1,\cdot}-\bar{y}_{3,\cdot} \end{pmatrix} \tag{5.27} \end{equation}\]
Since the observations are independent we have that \[\begin{equation} \begin{pmatrix}\bar{y}_{1,\cdot} \\ \bar{y}_{2,\cdot} \\ \bar{y}_{3,\cdot} \end{pmatrix}\sim N(\begin{pmatrix}\mu+\alpha_1+\frac{1}{3}(\beta_1+\beta_2+\beta_3) \\ \mu+\alpha_2+\frac{1}{3}(\beta_1+\beta_2+\beta_3) \\ \mu+\alpha_3+\frac{1}{3}(\beta_1+\beta_2+\beta_3) \end{pmatrix},\frac{\sigma^2}{3}\begin{pmatrix}1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{pmatrix}) \tag{5.28} \end{equation}\] and \[\begin{equation} \begin{pmatrix}\bar{y}_{1,\cdot}-\bar{y}_{2,\cdot} \\ \bar{y}_{1,\cdot}-\bar{y}_{3,\cdot} \end{pmatrix}\sim N(\begin{pmatrix}\alpha_1-\alpha_2 \\ \alpha_1-\alpha_3 \end{pmatrix},\frac{\sigma^2}{3}\begin{pmatrix}2 & 1 \\ 1 & 2 \end{pmatrix}) \tag{5.29} \end{equation}\] so that \[\begin{equation} \begin{split} SSR&=\frac{1}{\sigma^2}\begin{pmatrix}\bar{y}_{1,\cdot}-\bar{y}_{2,\cdot} \\ \bar{y}_{1,\cdot}-\bar{y}_{3,\cdot} \end{pmatrix}^T\begin{pmatrix}2 & -1 \\ -1 & 2 \end{pmatrix}\begin{pmatrix}\bar{y}_{1,\cdot}-\bar{y}_{2,\cdot} \\ \bar{y}_{1,\cdot}-\bar{y}_{3,\cdot} \end{pmatrix}\\ &=\frac{2}{\sigma^2}\{(\bar{y}_{1,\cdot}-\bar{y}_{2,\cdot})^2-(\bar{y}_{1,\cdot}-\bar{y}_{2,\cdot})(\bar{y}_{1,\cdot}-\bar{y}_{3,\cdot})+(\bar{y}_{1,\cdot}-\bar{y}_{3,\cdot})^2\}\sim\chi_2^2(\eta) \end{split} \tag{5.30} \end{equation}\] where \[\begin{equation} \eta=\frac{1}{2}\begin{pmatrix}\alpha_1-\alpha_2 \\ \alpha_1-\alpha_3 \end{pmatrix}^T\begin{pmatrix}2 & -1 \\ -1 & 2 \end{pmatrix}\begin{pmatrix}\alpha_1-\alpha_2 \\ \alpha_1-\alpha_3 \end{pmatrix} \tag{5.31} \end{equation}\] On the other hand, the unbiased estimaete of the variance \(\sigma^2\) is \[\begin{equation} \tilde{\sigma}^2=\frac{\sum_{i=1}^3\sum_{j=1}^3(y_{i,j}-\hat{y}_{i,j})^2}{4}=\frac{\sum_{i=1}^3\sum_{j=1}^3(y_{i,j}-\bar{y}_{i,\cdot}-\bar{y}_{\cdot,j}+\bar{y}_{\cdot,cdot})^2}{4} \tag{5.32} \end{equation}\] which is independent of SSR and, after appropriate renormalization, follows a \(\chi^2_4\) distribution. Hence, the statistic \[\begin{equation} F=\frac{2\{(\bar{y}_{1,\cdot}-\bar{y}_{2,\cdot})^2-(\bar{y}_{1,\cdot}-\bar{y}_{2,\cdot})(\bar{y}_{1,\cdot}-\bar{y}_{3,\cdot})+(\bar{y}_{1,\cdot}-\bar{y}_{3,\cdot})^2\}}{\sum_{i=1}^3\sum_{j=1}^3(y_{i,j}-\bar{y}_{i,\cdot}-\bar{y}_{\cdot,j}+\bar{y}_{\cdot,cdot})^2}\frac{4}{2} \tag{5.33} \end{equation}\] Follows a non-central \(F\) distribution, \(F_{2,4}(\eta)\). Under the null hypotheses, \(\eta=0\) and an appropriate rejection region can be obtained by finding \(c\) such that \(Pr(F>c|H_0)=0.05\).

Exercise 5.6 (By Abel, FYE 2014 Retake) Consider the model \(y_i=\mu+\beta\sin(x_i+\eta)+\epsilon_i\), where \(x_i=\frac{2\pi i}{n}\) for \(i=0,1,\cdots,n\), \(\epsilon_i\sim N(0,1)\) independently for each \(i\), and \(\eta\in[0,\pi]\).

  1. (25%) Assume for now that \(\eta\) is known. What is the maximum likelihood estimator for \((\mu,\beta)\)?

  2. (35%) Assume now that \(\eta\) is unknown. Use the trigonometric identity, \(\beta\sin(x_i+\eta)=\beta\cos(\eta)\sin(x_i)+\beta\sin(\eta)\cos(x_i)\), to write the model using a linear model formulation with a three-dimensional vector of regression coefficients. Based on the linear model formulation, obtain the maximum likelihood estimator for \((\mu,\beta,\eta)\). Hint: Note that \(\sum_{i=0}^n\sin(x_i)=0\) and \(\sum_{i=0}^n\sin(x_i)\cos(x_i)=0\).

  3. (40%) Assume that you are interested in testing the hypotheses \(H_0: \eta=\pi/4\) vs. \(H_a: \eta\neq\pi/4\). Derive the likelihood ratio test for this pair of hypotheses. (Please keep in mind that we are assuming that the error variance is known!)

Proof. 1. The model can be written as \(\mathbf{y}=\mathbf{X}\boldsymbol{\theta}+\boldsymbol{\epsilon}\) where \[\begin{equation} \mathbf{y}=\begin{pmatrix} y_{1}\\ \vdots \\ y_{n}\end{pmatrix}, \quad \mathbf{X}=\begin{pmatrix} 1 & \sin(x_0+\eta) \\ \vdots & \vdots \\ 1 & \sin(x_n+\eta) \end{pmatrix}, \quad \boldsymbol{\theta}=\begin{pmatrix} \mu \\ \beta \end{pmatrix},\quad \boldsymbol{\epsilon}=\begin{pmatrix} \epsilon_{1}\\ \vdots \\ \epsilon_{n}\end{pmatrix} \tag{5.34} \end{equation}\]

The MLE is simply \(\hat{\boldsymbol{\theta}}=(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}\) with \[\begin{equation} \mathbf{X}^T\mathbf{X}=\begin{pmatrix} n & \sum_{i=0}^n\sin(x_i+\eta) \\ \sum_{i=0}^n\sin(x_i+\eta) &\sum_{i=0}^n\sin^2(x_i+\eta) \end{pmatrix},\mathbf{X}^T\mathbf{y}=\begin{pmatrix} \sum_{i=0}^ny_i\\ \sum_{i=0}^ny_i\sin(x_i+\eta)\end{pmatrix} \tag{5.35} \end{equation}\] so that \[\begin{equation} \hat{\mu}=\frac{\{\sum_{i=0}^ny_i\}\{\sum_{i=0}^n\sin^2(x_i+\eta)\}-\{\sum_{i=0}^ny_i\sin(x_i+\eta)\}\{\sum_{i=0}^n\sin(x_i+\eta)\}}{n\sum_{i=0}^n\sin^2(x_i+\eta)-\{\sum_{i=0}^n\sin(x_i+\eta)\}^2} \tag{5.36} \end{equation}\] and \[\begin{equation} \hat{\beta}=\frac{n\{\sum_{i=0}^ny_i\sin(x_i+\eta)\}-\{\sum_{i=0}^ny_i\}\{\sum_{i=0}^n\sin(x_i+\eta)\}}{n\sum_{i=0}^n\sin^2(x_i+\eta)-\{\sum_{i=0}^n\sin(x_i+\eta)\}^2} \tag{5.37} \end{equation}\]

  1. The proposed alternative representation is valid because of the well-known trigonometric identity \(\sin(\alpha+\gamma)=\sin(\alpha)\cos(\gamma)+\cos(\alpha)\sin(\gamma)\). Note that the model can now be written as \[\begin{equation} \mathbf{y}=\begin{pmatrix} y_{1}\\ \vdots \\ y_{n}\end{pmatrix}, \quad \mathbf{X}=\begin{pmatrix} 1 & \sin(x_0) & \cos(x_0) \\ \vdots & \vdots & \vdots \\ 1 & \sin(x_n) & \cos(x_n) \end{pmatrix}, \quad \boldsymbol{\theta}=\begin{pmatrix} \mu \\ \phi_1 \\ \phi_2 \end{pmatrix},\quad \boldsymbol{\epsilon}=\begin{pmatrix} \epsilon_{1}\\ \vdots \\ \epsilon_{n}\end{pmatrix} \tag{5.38} \end{equation}\] where \(\phi_1=\beta\cos\eta\) and \(\phi_2=\beta\sin\eta\). In this case \[\begin{equation} \begin{split} &\mathbf{X}^T\mathbf{X}=\begin{pmatrix} n & \sum_{i=0}^n\sin(x_i) & \sum_{i=0}^n\cos(x_i) \\ \sum_{i=0}^n\sin(x_i) &\sum_{i=0}^n\sin^2(x_i) & \sum_{i=0}^n\sin(x_i)\cos(x_i) \\ \sum_{i=0}^n\cos(x_i) & \sum_{i=0}^n\sin(x_i)\cos(x_i) & \sum_{i=0}^n\cos^2(x_i) \end{pmatrix}\\ &\mathbf{X}^T\mathbf{y}=\begin{pmatrix} \sum_{i=0}^ny_i\\ \sum_{i=0}^ny_i\sin(x_i) \\ \sum_{i=0}^ny_i\cos(x_i) \end{pmatrix} \end{split} \tag{5.39} \end{equation}\] They can proceed with this general form, but it is easier to note that \(\sum_{i=0}^n\sin(x_i)=0\) and \(\sum_{i=0}^n\sin(x_i)\cos(x_i)=0\) (since both are odd functions around \(\pi\) and the \(x_i\)s are equally spaced), so that \[\begin{equation} \mathbf{X}^T\mathbf{X}=\begin{pmatrix} n & 0 & \sum_{i=0}^n\cos(x_i) \\ 0 &\sum_{i=0}^n\sin^2(x_i) & 0 \\ \sum_{i=0}^n\cos(x_i) & 0 & \sum_{i=0}^n\cos^2(x_i) \end{pmatrix} \tag{5.40} \end{equation}\] Now, if we work with \((\phi_1,\mu,\phi_2)\) instead of \((\mu,\phi_1,\phi_2)\) we obtain a block diagonal matrix that is easier to invert and
    \[\begin{equation} \begin{split} &\hat{\mu}=\frac{\{\sum_{i=0}^ny_i\}\{\sum_{i=0}^n\cos^2(x_i)\}-\{\sum_{i=0}^ny_i\cos(x_i)\}\{\sum_{i=0}^n\cos(x_i)\}}{n\sum_{i=0}^n\cos^2(x_i)-\{\sum_{i=0}^n\cos(x_i)\}^2}\\ &\hat{\phi}_1=\frac{\sum_{i=0}^ny_i\sin(x_i)}{\sum_{i=0}^n\sin^2(x_i)}\\ &\hat{\phi}_2=\frac{n\{\sum_{i=0}^ny_i\cos(x_i)\}-\{\sum_{i=0}^ny_i\}\{\sum_{i=0}^n\cos(x_i)\}}{n\sum_{i=0}^n\cos^2(x_i)-\{\sum_{i=0}^n\cos(x_i)\}^2}\\ \end{split} \tag{5.41} \end{equation}\] Because of invariance, the MLE of \(\eta\) is simply \(\hat{\eta}=arc\tan\{\frac{\hat{\phi}_1}{\hat{\phi}_2}\}\).

  2. Under the null, the model is equivalent to setting \(\phi_1=\phi_2=\beta\). Hence, testing \(H_0: \eta=\pi/4\) vs. \(H_a: \eta\neq\pi/4\) is equivalent to testing the general linear hypotheses: \[\begin{equation} H_0: \begin{pmatrix} 1 & 0 & -1 \end{pmatrix} \begin{pmatrix} \phi_1 \\ \mu \\ \phi_2 \end{pmatrix}=0 \quad vs. \quad H_a: \begin{pmatrix} 1 & 0 & -1 \end{pmatrix} \begin{pmatrix} \phi_1 \\ \mu \\ \phi_2 \end{pmatrix}\neq 0 \tag{5.42} \end{equation}\]

Note that \[\begin{equation} \begin{pmatrix} 1 & 0 & -1 \end{pmatrix} \begin{pmatrix} \hat{\phi}_1 \\ \hat{\mu} \\ \hat{\phi_2} \end{pmatrix}=\frac{\sum_{i=0}^ny_i\sin(x_i)}{\sum_{i=0}^n\sin^2(x_i)}-\frac{n\{\sum_{i=0}^ny_i\cos(x_i)\}-\{\sum_{i=0}^ny_i\}\{\sum_{i=0}^n\cos(x_i)\}}{n\sum_{i=0}^n\cos^2(x_i)-\{\sum_{i=0}^n\cos(x_i)\}^2} \tag{5.43} \end{equation}\] and that \[\begin{equation} \begin{split} \begin{pmatrix} 1 & 0 & -1 \end{pmatrix}&\begin{pmatrix} \frac{1}{\sum_{i=0}^n\sin^2(x_i)} & 0 & 0 \\ 0 & \frac{\sum_{i=0}^n\cos^2(x_i)}{n\sum_{i=0}^n\cos^2(x_i)-\{\sum_{i=0}^n\cos(x_i)\}} & \frac{-\sum_{i=0}^n\cos(x_i)}{n\sum_{i=0}^n\cos^2(x_i)-\{\sum_{i=0}^n\cos(x_i)\}} \\ 0 & \frac{-\sum_{i=0}^n\cos(x_i)}{n\sum_{i=0}^n\cos^2(x_i)-\{\sum_{i=0}^n\cos(x_i)\}} & \frac{n}{n\sum_{i=0}^n\cos^2(x_i)-\{\sum_{i=0}^n\cos(x_i)\}} \end{pmatrix} \begin{pmatrix} 1 \\ 0 \\ -1 \end{pmatrix}\\ &=\frac{1}{\sum_{i=0}^n\sin^2(x_i)}+\frac{n}{n\sum_{i=0}^n\cos^2(x_i)-\{\sum_{i=0}^n\cos(x_i)\}} \end{split} \end{equation}\] Therefore, \[\begin{equation} U=\frac{(\frac{\sum_{i=0}^ny_i\sin(x_i)}{\sum_{i=0}^n\sin^2(x_i)}-\frac{n\{\sum_{i=0}^ny_i\cos(x_i)\}-\{\sum_{i=0}^ny_i\}\{\sum_{i=0}^n\cos(x_i)\}}{n\sum_{i=0}^n\cos^2(x_i)-\{\sum_{i=0}^n\cos(x_i)\}^2})^2}{\frac{1}{\sum_{i=0}^n\sin^2(x_i)}+\frac{n}{n\sum_{i=0}^n\cos^2(x_i)-\{\sum_{i=0}^n\cos(x_i)\}}}\sim\chi_1^2 \tag{5.44} \end{equation}\] So the test proceeds by computing \(U_{obs}\) using the sample and rejecting \(H_0\) if \(U_{obs} > \chi^2_1(1-\alpha)\) where \(\chi^2_1(1-\alpha)\) denotes the \(1-\alpha\) quantile of the chi squared distribution with one degree of freedom.

Exercise 5.7 (By Wu, FYE 2013) For \(i=1,\cdots,N\), let \[\begin{equation} \begin{split} &Y_i=\epsilon_i,\quad i\neq k,k+1,k+2\\ &Y_k=\lambda_1+\epsilon_k\\ &Y_{k+1}=-c\lambda_1+\lambda_2+\epsilon_{k+1}\\ &Y_{k+2}=-c\lambda_2+\epsilon_{k+2} \end{split} \tag{5.45} \end{equation}\] where \(k\) is a fixed integer, \(1\leq k\leq N-2\), \(|c|<1\) is a known constant, and the \(\epsilon_i\) are i.i.d. \(N(0,\sigma^2)\) variables, for \(i=1,\cdots,N\). Let \(\boldsymbol{\beta}=(\lambda_1,\lambda_2)^T\), and suppose \(\sigma^2\) is known.

(a). (30%) Derive the least squares estimate of \(\boldsymbol{\beta}\), and the variance of the estimate.

(b). (35%) Derive the lease squares estimate of \(\boldsymbol{\beta}\) subject to the restriction \(\lambda_1+\lambda_2=0\). What is the variance of this estimate?

(c). (35%) Derive a statistic for testing \(H: \lambda_1+\lambda_2=0\) versus the alternative hypothesis that \(\lambda_1\) and \(\lambda_2\) are unrestricted.
Exercise 5.8 (By Raquel, FYE 2012) Suppose that we have the following data
TABLE 5.1: Regression Data.
Response Type I Response Type II Covariate Type I Covariate Type II
4 2 2 3
6 10 1 2
17 21 -3 -5

Here covariate \(x_{ij}\) is measured at the same time as the response for each experimental unit (\(j=1,2,3\)) in each type \((i=1,2)\).

Consider the following model: \[\begin{equation} y_{ij}=\mu+\alpha_i+\beta x_{ij}+\epsilon_{ij} \tag{5.46} \end{equation}\] with the \(\epsilon_{ij}\stackrel{i.i.d.}{\sim}N(0,\sigma^2)\).

  1. Part I.

  • (a). (15%) Write down the model in the matrix form \(\mathbf{y}=\mathbf{X}\boldsymbol{\beta}+\boldsymbol{\epsilon}\). What is the rank of \(\mathbf{X}\)? What is the rank of \(\mathbf{X}^T\mathbf{X}\)?

  • (b). (10%) Let \(\hat{\boldsymbol{\beta}}\) be a solution to the normal equations \(\mathbf{X}^T\mathbf{X}\boldsymbol{\beta}=\mathbf{X}^T\mathbf{y}\). Is \(\hat{\boldsymbol{\beta}}\) unique? Let \(\hat{\mathbf{y}}=\mathbf{X}\hat{\boldsymbol{\beta}}\). Is \(\hat{\mathbf{y}}\) unique? Justify your answers.

  • (c). (8%) Is \(H_0: \alpha_1=0\) testable? Justify your answer.

  • (d). (9%) Is \(H_0: \alpha_1-\alpha_2=0\) testable? Is the B.L.U.E. of \(\alpha_1-\alpha_2\) unique? Justify your answers.

  • (e). (8%) Is \(H_0: \beta=0\) testable? Justify your answer.

  1. Part II.

Add the following restriction to the model in (1): \(\alpha_2=0\). For the following questions please provide numerical values based on the data in addition to general mathematical expressions.

  • (a). (20%) Find the LSEs of \(\mu,\alpha_1\) and \(\beta\) in the restricted model.

  • (b). (30%) Let \(H_0: \alpha_1=0\). Is \(H_0\) testable in the restricted model? Is so, derive the F-statistic for testing \(H_0\).

Exercise 5.9 (By Raquel, FYE 2012 Retake) Suppose that \(\mathbf{y}=(y_1,\cdots,y_n)^T\) is generated from a model (true model) given by \[\begin{equation} \mathbf{y}=\mathbf{X}_1\boldsymbol{\beta}_1+\mathbf{X}_2\boldsymbol{\beta}_2+\boldsymbol{\epsilon} \tag{5.47} \end{equation}\] where \(\boldsymbol{\epsilon}\sim N_n(0,\sigma^2\mathbf{I})\) with \(\sigma^2\) known and \(X_1,X_2\) full rank matrices of dimensions \(n\times p_1\) and \(n\times p_2\), respectively.

  1. Part I. Assume that the model \[\begin{equation} \mathbf{y}=\mathbf{X}_1\boldsymbol{\beta}^*_1+\boldsymbol{\epsilon}^* \tag{5.48} \end{equation}\] is fitted to the data, with \(\boldsymbol{\epsilon}^*\sim N_n(0,\sigma^2\mathbf{I})\).

  • (a). (15%) Let \(\hat{\boldsymbol{\beta}}_1^*\) be the LSE of \(\boldsymbol{\beta}_1^*\). Is \(\hat{\boldsymbol{\beta}}_1^*\) an unbiased estimator of \(\boldsymbol{\beta}_1\) in general? Please justify your answer providing a detailed calculation.

  • (b). (10%) Now assume that the columns of \(\mathbf{X}_1\) are othogonal to the columns of \(\mathbf{X}_2\). Is \(\hat{\boldsymbol{\beta}}_1^*\) an unbiased estimator of \(\boldsymbol{\beta}_1\) in this case? Please justify your answer.

  • (c). (10%) Find \(Cov(\hat{\boldsymbol{\beta}}_1^*)\).

  • (d). (25%) Let \(\hat{\boldsymbol{\beta}}=\begin{pmatrix} \hat{\boldsymbol{\beta}}_1 \\ \hat{\boldsymbol{\beta}}_2 \end{pmatrix}\) be the LSE of \(\boldsymbol{\beta}=\begin{pmatrix} \boldsymbol{\beta}_1 \\ \boldsymbol{\beta}_2 \end{pmatrix}\) for the model in (5.47) written as \(\mathbf{y}=\mathbf{X}\boldsymbol{\beta}+\boldsymbol{\epsilon}\) with \(\mathbf{X}=(\mathbf{X}_1,\mathbf{X}_2)\). Once again, assume that the columns of \(\mathbf{X}_1\) are othogonal to the columns of \(\mathbf{X}_2\). Is \(Cov(\hat{\boldsymbol{\beta}}_1)=Cov(\hat{\boldsymbol{\beta}}_1^*)\) in this case? Are \(\hat{\boldsymbol{\beta}}_1\) and \(\hat{\boldsymbol{\beta}}_2\) independent? Please justify your answers.

  1. Part II. Suppose that data \(y_1,\cdots,y_n\) are generated from the model \(y_i=\beta_0+\beta_1x_i+\epsilon_i\), but the model \(y_i=\beta_1^*x_i+\epsilon_i^*\) is fitted to the datat instead. Once again assume that \(\boldsymbol{\epsilon}\sim N_n(0,\sigma^2\mathbf{I})\) and \(\boldsymbol{\epsilon}^*\sim N_n(0,\sigma^2\mathbf{I})\) with \(\sigma^2\) known.

  • (a). (15%) Find \(\hat{\beta}_1^*\), the LSE of \(\beta_1^*\).

  • (b). (25%) Derive the statistic to perform the following test: \[\begin{equation} H_0: \beta_1^*=0 \quad\quad\quad H_1: \beta_1^*\neq 0 \tag{5.49} \end{equation}\]

Exercise 5.10 (By Raquel, FYE 2010) Consider the model defined by the following equation: \[\begin{equation} y_{i,j}=\mu+\alpha_i+\epsilon_{i,j},\quad \epsilon_{i,j}\sim N(0,\sigma^2) \tag{5.50} \end{equation}\] for \(i=1,2,j=1,2,3\) and with all \(\epsilon_{i,j}\)s independent.

  1. Part I.

  • (a). (10%) Write the model in matrix form \(\mathbf{y}=\mathbf{X}\boldsymbol{\beta}+\boldsymbol{\epsilon}\). what is the rank of \(\mathbf{X}\)? What is the rank of \(\mathbf{X}^T\mathbf{X}\)?

  • (b). (10%) Is \(H_0: \alpha_1=\alpha_2\) testable? Justify your answer.

  • (c). (10%) Is \(H_0: \mu=0\) testable? Justify your answer.

  • (d). (5%) Are the LSEs of \(\mu,\alpha_1\) and \(\alpha_2\) unique? Justify your answer.

  • (e). (5%) Is the LSE of \(\alpha_1-\alpha_2\) unique? Justify your answer.

  1. Part II: Add the restriction \(\alpha_1=0\) to the model in (5.50).

  • (a). (20%) Find the LSEs of \(\mu\) and \(\alpha_2\) and denote them as \(\hat{\mu}\) and \(\hat{\alpha}_2\). Are these unique?

  • (b). (15%) What is the distribution of \(\hat{\mu}\) and \(\hat{\alpha}_2\)? Are \(\hat{\mu}\) and \(\hat{\alpha}_2\) independent?

  • (c). (10%) Give the form of a 95% C.I. for \(\mu+\alpha_2\).

  • (d). (10%) Find the B.L.U.E. of \(\mu+\alpha_2\) and its associated variance.

  • (e). (5%) Is model (5.50) with the restriction \(\alpha_2=0\) equivalent to model (5.50) with the restriction \(\alpha_1+\alpha_2=0\)? Justify your answer.

Exercise 5.11 (By Raquel, FYE 2009) Consider the model defined as follows. For \(i=1,\cdots,N\), let \[\begin{equation} \begin{split} &y_i=\epsilon_i,\quad i\neq k,i\neq k+1,i\neq k+2\\ &y_k=\lambda_1+\epsilon_k\\ &y_{k+1}=-\lambda_1+\lambda_2+\epsilon_{k+1}\\ &y_{k+2}=-\lambda_2+\epsilon_{k+2} \end{split} \tag{5.51} \end{equation}\] where \(k\) is a fixed integer, \(1\leq k\leq N-2\) nd the \(\epsilon_i\)s are i.i.d. \(N(0,\sigma^2)\) variables. Let \(\boldsymbol{\beta}=(\lambda_1,\lambda_2)^T\) and suppose \(\sigma^2\) is known.

  1. (40%) Derive the LSE of \(\boldsymbol{\beta}\), denoted as \(\hat{\boldsymbol{\beta}}\).

  2. (30%) Find the distribution of \(\hat{\boldsymbol{\beta}}\). Are \(\hat{\lambda}_1\) and \(\hat{\lambda}_2\) independent? Justify your answer.

  3. (30%) Find a \(95\%\) confidence interval for \(\lambda_1+\lambda_2\).

Proof. 1. Writing the model in matrix form \(\mathbf{y}=\mathbf{X}\boldsymbol{\beta}+\boldsymbol{\epsilon}\), we have that \[\begin{equation} (\mathbf{X}^T\mathbf{X})^{-1}=\frac{1}{3}\begin{pmatrix} 2 & 1 \\ 1 & 2 \end{pmatrix},\quad \mathbf{X}^T\mathbf{y}=\begin{pmatrix} y_k-y_{k+1} \\ y_{k+1}-y_{k+2} \end{pmatrix} \tag{5.52} \end{equation}\] and so \[\begin{equation} \hat{\boldsymbol{\beta}}=\begin{pmatrix} \hat{\lambda}_1 \\ \hat{\lambda}_2 \end{pmatrix}=\begin{pmatrix} (2y_k-y_{k+1}-y_{k+2})/3 \\ (y_{k}-y_{k+1}+2y_{k+2})/3 \end{pmatrix} \tag{5.53} \end{equation}\]

  1. \((\hat{\boldsymbol{\beta}}|\sigma^2)\sim N(\boldsymbol{\beta},\sigma^2(\mathbf{X}^T\mathbf{X})^{-1})\) and so \(\hat{\lambda}_1\) and \(\hat{\lambda}_2\) are not independent.

  2. \(96\%\) CI for \(\lambda_1+\lambda_2\) is given by \(\hat{\lambda}_1+\hat{\lambda}_2\pm 1.96\sqrt{2\sigma^2}\).