## 2.1 Matrix Theory

$A= \left[ \begin{array} {cc} a_{11} & a_{12} \\ a_{21} & a_{22} \end{array} \right]$

$A' = \left[ \begin{array} {cc} a_{11} & a_{21} \\ a_{12} & a_{22} \end{array} \right]$

\begin{aligned} \mathbf{(ABC)'} & = \mathbf{C'B'A'} \\ \mathbf{A(B+C)} & = \mathbf{AB + AC} \\ \mathbf{AB} & \neq \mathbf{BA} \\ \mathbf{(A')'} & = \mathbf{A} \\ \mathbf{(A+B)'} & = \mathbf{A' + B'} \\ \mathbf{(AB)'} & = \mathbf{B'A'} \\ \mathbf{(AB)^{-1}} & = \mathbf{B^{-1}A^{-1}} \\ \mathbf{A+B} & = \mathbf{B +A} \\ \mathbf{AA^{-1}} & = \mathbf{I} \end{aligned}

If A has an inverse, it is called invertible. If A is not invertible it is called singular.

\begin{aligned} \mathbf{A} &= \left(\begin{array} {ccc} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \\ \end{array}\right) \left(\begin{array} {ccc} b_{11} & b_{12} & b_{13} \\ b_{21} & b_{22} & b_{23} \\ b_{31} & b_{32} & b_{33} \\ \end{array}\right) \\ &= \left(\begin{array} {ccc} a_{11}b_{11}+a_{12}b_{21}+a_{13}b_{31} & \sum_{i=1}^{3}a_{1i}b_{i2} & \sum_{i=1}^{3}a_{1i}b_{i3} \\ \sum_{i=1}^{3}a_{2i}b_{i1} & \sum_{i=1}^{3}a_{2i}b_{i2} & \sum_{i=1}^{3}a_{2i}b_{i3} \\ \end{array}\right) \end{aligned}

Let $$\mathbf{a}$$ be a $$3 \times 1$$ vector, then the quadratic form is

$\mathbf{a'Ba} = \sum_{i=1}^{3}\sum_{i=1}^{3}a_i b_{ij} a_{j}$

Length of a vector
Let $$\mathbf{a}$$ be a vector, $$||\mathbf{a}||$$ (the 2-norm of the vector) is the length of vector $$\mathbf{a}$$, is the square root of the inner product of the vector with itself:

$||\mathbf{a}|| = \sqrt{\mathbf{a'a}}$

### 2.1.1 Rank

• Dimension of space spanned by its columns (or its rows).
• Number of linearly independent columns/rows

For a $$n \times k$$ matrix A and $$k \times k$$ matrix B

• $$rank(A)\leq min(n,k)$$
• $$rank(A) = rank(A') = rank(A'A)=rank(AA')$$
• $$rank(AB)=min(rank(A),rank(B))$$
• B is invertible if and only if $$rank(B) = k$$ (non-singular)

### 2.1.2 Inverse

In scalar, $$a = 0$$ then $$1/a$$ does not exist.

In matrix, a matrix is invertible when it’s a non-zero matrix.

A non-singular square matrix $$\mathbf{A}$$ is invertible if there exists a non-singular square matrix $$\mathbf{B}$$ such that, $AB=I$ Then $$A^{-1}=B$$. For a $$2\times2$$ matrix,

$A = \left(\begin{array}{cc} a & b \\ c & d \\ \end{array} \right)$

$A^{-1}= \frac{1}{ad-bc} \left(\begin{array}{cc} d & -b \\ -c & a \\ \end{array} \right)$

For the partition matrix,

$\left[\begin{array} {cc} A & B \\ C & D \\ \end{array} \right]^{-1} = \left[\begin{array} {cc} \mathbf{(A-BD^{-1}C)^{-1}} & \mathbf{-(A-BD^{-1}C)^{-1}BD^-1} \\ \mathbf{-DC(A-BD^{-1}C)^{-1}} & \mathbf{D^{-1}+D^{-1}C(A-BD^{-1}C)^{-1}BD^{-1}} \end{array} \right]$

Properties for a non-singular square matrix

• $$\mathbf{A^{-1}}=A$$
• for a non-zero scalar b, $$\mathbf{(bA)^{-1}=b^{-1}A^{-1}}$$
• for a matrix B, $$\mathbf(BA)^{-1}=B^{-1}A^{-1}$$ only if $$\mathbf{B}$$ is non-singular
• $$\mathbf{(A^{-1})'=(A')^{-1}}$$
• Never notate $$\mathbf{1/A}$$

### 2.1.3 Definiteness

A symmetric square k x k matrix, $$\mathbf{A}$$, is Positive Semi-Definite if for any non-zero $$k \times 1$$ vector $$\mathbf{x}$$, $\mathbf{x'Ax \geq 0 }$

A symmetric square k x k matrix, $$\mathbf{A}$$, is Negative Semi-Definite if for any non-zero $$k \times 1$$ vector $$\mathbf{x}$$ $\mathbf{x'Ax \leq 0 }$

$$\mathbf{A}$$ is indefinite if it is neither positive semi-definite or negative semi-definite.

The identity matrix is positive definite

Example Let $$\mathbf{x} =(x_1 x_2)'$$, then for a $$2 \times 2$$ identity matrix,

\begin{aligned} \mathbf{x'Ix} &= (x_1 x_2) \left(\begin{array} {cc} 1 & 0 \\ 0 & 1 \\ \end{array} \right) \left(\begin{array}{c} x_1 \\ x_2 \\ \end{array} \right) \\ &= (x_1 x_2) \left(\begin{array} {c} x_1 \\ x_2 \\ \end{array} \right) \\ &= x_1^2 + x_2^2 >0 \end{aligned}

Definiteness gives us the ability to compare matrices $$\mathbf{A-B}$$ is PSD

This property also helps us show efficiency (which variance covariance matrix of one estimator is smaller than another)

Properties

• any variance matrix is PSD
• a matrix $$\mathbf{A}$$ is PSD if and only if there exists a matrix $$\mathbf{B}$$ such that $$\mathbf{A=B'B}$$
• if $$\mathbf{A}$$ is PSD, then $$\mathbf{B'AB}$$ is PSD
• if $$\mathbf{A}$$ and $$\mathbf{C}$$ are non-singular, then $$\mathbf{A-C}$$ is PSD if and only if $$\mathbf{C^{-1}-A^{-1}}$$
• if $$\mathbf{A}$$ is PD (ND) then $$A^{-1}$$ is PD (ND)

Note

• Indefinite $$\mathbf{A}$$ is neither PSD nor NSD. There is no comparable concept in scalar.
• If a square matrix is PSD and invertible then it is PD

Example:

1. Invertible / Indefinite

$\left[ \begin{array} {cc} -1 & 0 \\ 0 & 10 \end{array} \right]$

1. Non-invertible/ Indefinite

$\left[ \begin{array} {cc} 0 & 1 \\ 0 & 0 \end{array} \right]$

1. Invertible / PSD

$\left[ \begin{array} {cc} 1 & 0 \\ 0 & 1 \end{array} \right]$

1. Non-Invertible / PSD

$\left[ \begin{array} {cc} 0 & 0 \\ 0 & 1 \end{array} \right]$

### 2.1.4 Matrix Calculus

$$y=f(x_1,x_2,...,x_k)=f(x)$$ where $$x$$ is a $$1 \times k$$ row vector.

The Gradient (first order derivative with respect to a vector) is,

$\frac{\partial{f(x)}}{\partial{x}}= \left(\begin{array}{c} \frac{\partial{f(x)}}{\partial{x_1}} \\ \frac{\partial{f(x)}}{\partial{x_2}} \\ \dots \\ \frac{\partial{f(x)}}{\partial{x_k}} \end{array} \right)$

The Hessian (second order derivative with respect to a vector) is,

$\frac{\partial^2{f(x)}}{\partial{x}\partial{x'}}= \left(\begin{array} {cccc} \frac{\partial^2{f(x)}}{\partial{x_1}\partial{x_1}} & \frac{\partial^2{f(x)}}{\partial{x_1}\partial{x_2}} & \dots & \frac{\partial^2{f(x)}}{\partial{x_1}\partial{x_k}} \\ \frac{\partial^2{f(x)}}{\partial{x_1}\partial{x_2}} & \frac{\partial^2{f(x)}}{\partial{x_2}\partial{x_2}} & \dots & \frac{\partial^2{f(x)}}{\partial{x_2}\partial{x_k}} \\ \dots & \dots & \ddots & \dots\\ \frac{\partial^2{f(x)}}{\partial{x_k}\partial{x_1}} & \frac{\partial^2{f(x)}}{\partial{x_k}\partial{x_2}} & \dots & \frac{\partial^2{f(x)}}{\partial{x_k}\partial{x_k}} \end{array} \right)$

Define the derivative of $$f(\mathbf{X})$$ with respect to $$\mathbf{X}_{(n \times p)}$$ as the matrix

$\frac{\partial f(\mathbf{X})}{\partial \mathbf{X}} = (\frac{\partial f(\mathbf{X})}{\partial x_{ij}})$

Define $$\mathbf{a}$$ to be a vector and $$\mathbf{A}$$ to be a matrix which does not depend upon $$\mathbf{y}$$. Then

$\frac{\partial \mathbf{a'y}}{\partial \mathbf{y}} = \mathbf{a}$

$\frac{\partial \mathbf{y'y}}{\partial \mathbf{y}} = 2\mathbf{y}$

$\frac{\partial \mathbf{y'Ay}}{\partial \mathbf{y}} = \mathbf{(A + A')y}$

If $$\mathbf{X}$$ is a symmetric matrix then

$\frac{\partial |\mathbf{X}|}{\partial x_{ij}} = \begin{cases} X_{ii}, i = j \\ X_{ij}, i \neq j \end{cases}$

where $$X_{ij}$$ is the $$(i,j)$$-th cofactor of $$\mathbf{X}$$

If $$\mathbf{X}$$ is symmetric and $$\mathbf{A}$$ is a matrix which does not depend upon $$\mathbf{X}$$ then

$\frac{\partial tr \mathbf{XA}}{\partial \mathbf{X}} = \mathbf{A} + \mathbf{A}' - diag(\mathbf{A})$

If $$\mathbf{X}$$ is symmetric and we let $$\mathbf{J}_{ij}$$ be a matrix which has a 1 in the $$(i,j)$$-th position and 0s elsewhere, then

$\frac{\partial \mathbf{X}6{-1}}{\partial x_{ij}} = \begin{cases} - \mathbf{X}^{-1}\mathbf{J}_{ii} \mathbf{X}^{-1} &, i = j \\ - \mathbf{X}^{-1}(\mathbf{J}_{ij} + \mathbf{J}_{ji}) \mathbf{X}^{-1} &, i \neq j \end{cases}$

### 2.1.5 Optimization

Scalar Optimization Vector Optimization
First Order Condition $\frac{\partial{f(x_0)}}{\partial{x}}=0$ $\frac{\partial{f(x_0)}}{\partial{x}}=\left(\begin{array}{c}0 \\ .\\ .\\ .\\ 0\end{array}\right)$

Second Order Condition

Convex $$\rightarrow$$ Min

$\frac{\partial^2{f(x_0)}}{\partial{x^2}} > 0$ $\frac{\partial^2{f(x_0)}}{\partial{xx'}}>0$
Concave $$\rightarrow$$ Max $\frac{\partial^2{f(x_0)}}{\partial{x^2}} < 0$ $\frac{\partial^2{f(x_0)}}{\partial{xx'}}<0$