Chapter 3 Markov Chain: Definition and Basic Properties (Lecture on 01/12/2021)

The next few chapters will be mainly about discrete time, discrete state space stochastic process, mainly from the context of Markov chain.

Let $X=\{X(t,\omega):t\in T,\omega\in\Omega\}$, $X(t,\omega)\in S$, consider the case both $T$ and $S$ are discrete. Define the random variabel $X_t(\omega)=X(t,\omega)$ and consider the sequence of random variables $\{X_0,X_1,\cdots\}$ which take values in some countable set $S$, called the state space. Each $X_n$ is a discrete random variable that takes one of the $N$ possible values, where $N=|S|$. We are allowing $N=\infty$. This is the set up for discrete time, discrete state stochastic process.

Definition 3.1 (Markov Chain) The process $X=\{X_0,X_1,\cdots\}$ is a Markov chain if it satisfies the Markov condition: $$\begin{equation} P(X_{n}=s|X_0=x_0,X_1=x_1,\cdots,X_{n-1}=x_{n-1})=P(X_{n}=s|X_{n-1}=x_{n-1}),\quad\forall s \tag{3.1} \end{equation}$$

The Markov property described in this way is equivalent to
$$\begin{equation} P(X_{n}=s|X_{n_1}=x_{n_1},\cdots,X_{n_k}=x_{n_k})=P(X_{n}=s|X_{n_k}=x_{n_k}),\quad\forall s \tag{3.2} \end{equation}$$ for all $n_1<n_2<\cdots<n_k\leq n-1$.

We have asssumed that $X$ takes values in some countable set $S$. Since $S$ is countable, it can be put in one-to-one correspondence with some subset $S^{\prime}$ of the integers. Thus, without loss of generality, we can say the following: if $X_n=i$, it actually means that the chain is in the $i$th state at the $n$th time points.

The evolution of a chain is described by the transition probabilities, defined as $P(X_{n+1}=j|X_n=i)$. This probability may depend on $n,i,j$. We will restric our attention to the case when transition probabilities do not depend on $n$.

Definition 3.2 (Homogenous Chain) The chain $X=\{X_0,\cdots\}$ is called homogenous (time homogenous) if $P(X_{n+1}=j|X_n=i)=P(X_1=j|X_0=i),\forall n,i,j$. For a homogenous chain, we define the transition matrix $P=\{p_{ij}\}_{i,j=1}^{|S|}$ as an $|S|\times|S|$ matrix of transition probabilities $p_{ij}=P(X_{n+1}=j|X_n=i)$.

The beauty of homogeneity assumption is that, we can specify the distribution of the whole stochastic process by specify the transition matrix. Suppose the stochastic process have infinite index space $T$ and finite state space $S$. If we assume homogeneity, the problem of specifying the infinite dimensional distribution $(x_0,\cdots,)$ becomes to the problem of specifying a finite dimensional matrix $P$. It simplifies the problem a lot.

Theorem 3.1 (Properties of Transition Probability Matrix) If $\mathbf{P}$ is a transition probability matrix, then

1. $0\leq p_{ij}\leq 1$, $\forall i,j$.

2. $\sum_{j}p_{ij}=1$, $\forall i$.

Definition 3.3 (The n-step Transition Probability Matrix) The n-step transition probability matrix is defined as $P(m,m+n)=\{p_{ij}(m,m+n)\}_{i,j=1}^{|S|}$, where $p_{ij}(m,m+n)=P(X_{m+n}=j|X_m=i)$.

Theorem 3.2 (Chapman-Kolmogorov Equation) $$\begin{equation} p_{ij}(m,m+n+r)=\sum_{k}p_{ik}(m,m+n)p_{kj}(m+n,m+n+r),\quad \forall r \tag{3.3} \end{equation}$$

Intuitively, this means the $n+r$ step transition probability matrix can be decomposed into a $n$ step and a $r$ step transication probability matrix.

Proof. $$\begin{equation} \begin{split} p_{ij}(m,m+n+r)&=P(X_{n+m+r}=j|X_m=i)=\sum_k P(X_{m+n+r}=j,X_{m+n}=k|X_m=i)\\ &=\sum_k P(X_{m+n+r}=j|X_{m+n}=k,X_m=i)P(X_{m+n}=k|X_m=i)\\ &=\sum_k P(X_{m+n+r}=j|X_{m+n}=k)P(X_{m+n}=k|X_m=i) \quad (By\,Markov\,property)\\ &=\sum_k p_{kj}(m+n,m+n+r)p_{ik}(m,m+n) \end{split} \end{equation}$$

Since the transition probability matrix is $P(m,m+n+r)=\{p_{ij}(m,m+n+r)\}_{i,j=1}^{|S|}$, the Chapman-Kolmogorov equation tells us $$\begin{equation} P(m,m+n+r)=P(m,m+n)P(m+n,m+n+r),\quad \forall n,m,r \tag{3.4} \end{equation}$$ Specificly, we can take $r=n=1$ and we have $$\begin{equation} P(m,m+2)=P(m,m+1)P(m+1,m+2),\quad \forall n,m,r \tag{3.5} \end{equation}$$ If we further assume time homogeneity, then $P(m,m+1)=P(m+1,m+2)=P$ and (3.5) becomes $$\begin{equation} P(m,m+2)=p^2 \tag{3.6} \end{equation}$$ In general, we have $$\begin{equation} P(m,m+n)=p^n,\quad \forall n \tag{3.7} \end{equation}$$ That is, if the one step transition probability matrix is $P=\{p_{ij}\}_{i,j=1}^{|S|}$ where $p_{ij}=P(X_{n+1}=j|X_n=i)$, then the $i,j$th entry of the n-step transition probability matrix $P(X_{m+n}=j|X_m=i)=(P^n)_{i,j}$ where $(P^n)_{i,j}$ denotes the $i,j$th entry of $P^n$.

Lemma 3.1 Let $\mu_i(n)=P(X_n=i)$, that is the marginal probability of $X_n$ takes the $i$th state. Write $\boldsymbol{\mu}(n)$ as the row vector $(\mu_i(n),i\in S)$, then $$\begin{equation} \boldsymbol{\mu}(m+n)=\boldsymbol{\mu}(m)P^n \tag{3.8} \end{equation}$$

This lemma gives the relationship between the marginal probability vector of $X$ at time $m$ and at time $m+n$.

Proof. $$\begin{equation} \begin{split} \mu_j^{m+n}&=P(X_{m+n}=j)=\sum_iP(X_{m+n}=j|X_m=i)P(X_m=i)\\ &=\sum_ip_{ij}(m,m+n)\mu_i(m)\\ &=\sum_i(P^n)_{i,j}\mu_i(m)=(\boldsymbol{\mu}(m)P^n)_j \end{split} \end{equation}$$ Since this is true for all $j\in S$, we have $\boldsymbol{\mu}(m+n)=\boldsymbol{\mu}(m)P^n$

Example 3.1 (Simple Random Walk) Suppose $X_n=\left\{\begin{aligned} & 1 & p \\ & -1 & 1-p \end{aligned}\right.$ for all $n\in\mathbb{N}$. Consider the stochastic process given by $S_n(\omega)=X_1(\omega)+\cdots+X_n(\omega)$. The state space of this stochastic process is $S=\{0,\pm 1,\pm 2,\cdots\}$. Then $S_n$ is a Markov chain. The one step transition probability is given by $P(S_n=j|S_{n-1}=i)=\left\{\begin{aligned} & p & j=i+1 \\ & 1-p & j=i-1 \\ & 0 & o.w. \end{aligned}\right.$ Now for the n-step transition probability, we are interested in $P_{ij}(n)=P(X_n=j|X_0=i)$. Suppose there are $a$ upward move and $b$ downward moves, we have $$\begin{equation} \left\{\begin{aligned} & a+b=n \\ & a-b=j-i \end{aligned}\right.\Longrightarrow \left\{\begin{aligned} & a=\frac{n+j-i}{2} \\ & b=\frac{n-j+i}{2} \end{aligned}\right. \tag{3.9} \end{equation}$$ Then, the n-step transition probability is given by $$\begin{equation} p_{ij}(n)=P(X_n=j|X_0=i)=\left\{\begin{aligned} & {n \choose a}p^a(1-p)^b & n+j-i\, even \\ & 0 & n+j-i\,odd \end{aligned}\right. \tag{3.10} \end{equation}$$ where $a$ is given by (3.9).

Example 3.2 (Ehrenfest Diffusion Models) Suppose there are a total of 2A balls in 2 boxes, labeled $b$ and $B$. At each time, we choose a ball at random, and shifted it from its box of origin to the other box. Let $X_n$ be the number of balls at time n in box $b$, then $X_n$ is a Markov chain. We have $$\begin{equation} P(X_{n+1}=A+j|X_n=A+i)=\left\{\begin{aligned} & \frac{A-i}{2A} & j=i+1 \\ & \frac{A+i}{2A} & j=i-1 \end{aligned}\right. \tag{3.11} \end{equation}$$ for all $i=-A,\cdots,A$.

Definition 3.4 (Persistent State) State $i$ is called persistent (recurrent) if $P(X_n=i\, \text{for some}\, n\geq 1|X_0=i)=1$. This is to say that the probability of the chain eventually return to i, having started from i, is 1.

If this probability is less than 1, state $i$ is known as the transient state.

We will be interested in the first passage time defined as $f_{ij}(n)=P(X_1\neq j,\cdots,X_{n-1}\neq j,X_n=j|X_0=i)$. This is the probability that state $j$ is first visited from state $i$ at time n. Write $f_{ij}=\sum_{n=1}^{\infty}f_{ij}(n)$, it is the probability that state $j$ is ever visited from state $i$. If $f_{ij}=1$, we are interested in the constraints it imples on the transition probability.