# Chapter 3 Markov Chain: Definition and Basic Properties (Lecture on 01/12/2021)

The next few chapters will be mainly about discrete time, discrete state space stochastic process, mainly from the context of Markov chain.

Let $$X=\{X(t,\omega):t\in T,\omega\in\Omega\}$$, $$X(t,\omega)\in S$$, consider the case both $$T$$ and $$S$$ are discrete. Define the random variabel $$X_t(\omega)=X(t,\omega)$$ and consider the sequence of random variables $$\{X_0,X_1,\cdots\}$$ which take values in some countable set $$S$$, called the state space. Each $$X_n$$ is a discrete random variable that takes one of the $$N$$ possible values, where $$N=|S|$$. We are allowing $$N=\infty$$. This is the set up for discrete time, discrete state stochastic process.

Definition 3.1 (Markov Chain) The process $$X=\{X_0,X_1,\cdots\}$$ is a Markov chain if it satisfies the Markov condition: $$$P(X_{n}=s|X_0=x_0,X_1=x_1,\cdots,X_{n-1}=x_{n-1})=P(X_{n}=s|X_{n-1}=x_{n-1}),\quad\forall s \tag{3.1}$$$

The Markov property described in this way is equivalent to
$$$P(X_{n}=s|X_{n_1}=x_{n_1},\cdots,X_{n_k}=x_{n_k})=P(X_{n}=s|X_{n_k}=x_{n_k}),\quad\forall s \tag{3.2}$$$ for all $$n_1<n_2<\cdots<n_k\leq n-1$$.

We have asssumed that $$X$$ takes values in some countable set $$S$$. Since $$S$$ is countable, it can be put in one-to-one correspondence with some subset $$S^{\prime}$$ of the integers. Thus, without loss of generality, we can say the following: if $$X_n=i$$, it actually means that the chain is in the $$i$$th state at the $$n$$th time points.

The evolution of a chain is described by the transition probabilities, defined as $$P(X_{n+1}=j|X_n=i)$$. This probability may depend on $$n,i,j$$. We will restric our attention to the case when transition probabilities do not depend on $$n$$.

Definition 3.2 (Homogenous Chain) The chain $$X=\{X_0,\cdots\}$$ is called homogenous (time homogenous) if $$P(X_{n+1}=j|X_n=i)=P(X_1=j|X_0=i),\forall n,i,j$$. For a homogenous chain, we define the transition matrix $$P=\{p_{ij}\}_{i,j=1}^{|S|}$$ as an $$|S|\times|S|$$ matrix of transition probabilities $$p_{ij}=P(X_{n+1}=j|X_n=i)$$.
The beauty of homogeneity assumption is that, we can specify the distribution of the whole stochastic process by specify the transition matrix. Suppose the stochastic process have infinite index space $$T$$ and finite state space $$S$$. If we assume homogeneity, the problem of specifying the infinite dimensional distribution $$(x_0,\cdots,)$$ becomes to the problem of specifying a finite dimensional matrix $$P$$. It simplifies the problem a lot.

Theorem 3.1 (Properties of Transition Probability Matrix) If $$\mathbf{P}$$ is a transition probability matrix, then

1. $$0\leq p_{ij}\leq 1$$, $$\forall i,j$$.

2. $$\sum_{j}p_{ij}=1$$, $$\forall i$$.
Definition 3.3 (The n-step Transition Probability Matrix) The n-step transition probability matrix is defined as $$P(m,m+n)=\{p_{ij}(m,m+n)\}_{i,j=1}^{|S|}$$, where $$p_{ij}(m,m+n)=P(X_{m+n}=j|X_m=i)$$.

Theorem 3.2 (Chapman-Kolmogorov Equation) $$$p_{ij}(m,m+n+r)=\sum_{k}p_{ik}(m,m+n)p_{kj}(m+n,m+n+r),\quad \forall r \tag{3.3}$$$

Intuitively, this means the $$n+r$$ step transition probability matrix can be decomposed into a $$n$$ step and a $$r$$ step transication probability matrix.
Proof. $$$\begin{split} p_{ij}(m,m+n+r)&=P(X_{n+m+r}=j|X_m=i)=\sum_k P(X_{m+n+r}=j,X_{m+n}=k|X_m=i)\\ &=\sum_k P(X_{m+n+r}=j|X_{m+n}=k,X_m=i)P(X_{m+n}=k|X_m=i)\\ &=\sum_k P(X_{m+n+r}=j|X_{m+n}=k)P(X_{m+n}=k|X_m=i) \quad (By\,Markov\,property)\\ &=\sum_k p_{kj}(m+n,m+n+r)p_{ik}(m,m+n) \end{split}$$$

Since the transition probability matrix is $$P(m,m+n+r)=\{p_{ij}(m,m+n+r)\}_{i,j=1}^{|S|}$$, the Chapman-Kolmogorov equation tells us $$$P(m,m+n+r)=P(m,m+n)P(m+n,m+n+r),\quad \forall n,m,r \tag{3.4}$$$ Specificly, we can take $$r=n=1$$ and we have $$$P(m,m+2)=P(m,m+1)P(m+1,m+2),\quad \forall n,m,r \tag{3.5}$$$ If we further assume time homogeneity, then $$P(m,m+1)=P(m+1,m+2)=P$$ and (3.5) becomes $$$P(m,m+2)=p^2 \tag{3.6}$$$ In general, we have $$$P(m,m+n)=p^n,\quad \forall n \tag{3.7}$$$ That is, if the one step transition probability matrix is $$P=\{p_{ij}\}_{i,j=1}^{|S|}$$ where $$p_{ij}=P(X_{n+1}=j|X_n=i)$$, then the $$i,j$$th entry of the n-step transition probability matrix $$P(X_{m+n}=j|X_m=i)=(P^n)_{i,j}$$ where $$(P^n)_{i,j}$$ denotes the $$i,j$$th entry of $$P^n$$.

Lemma 3.1 Let $$\mu_i(n)=P(X_n=i)$$, that is the marginal probability of $$X_n$$ takes the $$i$$th state. Write $$\boldsymbol{\mu}(n)$$ as the row vector $$(\mu_i(n),i\in S)$$, then $$$\boldsymbol{\mu}(m+n)=\boldsymbol{\mu}(m)P^n \tag{3.8}$$$
This lemma gives the relationship between the marginal probability vector of $$X$$ at time $$m$$ and at time $$m+n$$.
Proof. $$$\begin{split} \mu_j^{m+n}&=P(X_{m+n}=j)=\sum_iP(X_{m+n}=j|X_m=i)P(X_m=i)\\ &=\sum_ip_{ij}(m,m+n)\mu_i(m)\\ &=\sum_i(P^n)_{i,j}\mu_i(m)=(\boldsymbol{\mu}(m)P^n)_j \end{split}$$$ Since this is true for all $$j\in S$$, we have $$\boldsymbol{\mu}(m+n)=\boldsymbol{\mu}(m)P^n$$
Example 3.1 (Simple Random Walk) Suppose X_n=\left\{\begin{aligned} & 1 & p \\ & -1 & 1-p \end{aligned}\right. for all $$n\in\mathbb{N}$$. Consider the stochastic process given by $$S_n(\omega)=X_1(\omega)+\cdots+X_n(\omega)$$. The state space of this stochastic process is $$S=\{0,\pm 1,\pm 2,\cdots\}$$. Then $$S_n$$ is a Markov chain. The one step transition probability is given by P(S_n=j|S_{n-1}=i)=\left\{\begin{aligned} & p & j=i+1 \\ & 1-p & j=i-1 \\ & 0 & o.w. \end{aligned}\right. Now for the n-step transition probability, we are interested in $$P_{ij}(n)=P(X_n=j|X_0=i)$$. Suppose there are $$a$$ upward move and $$b$$ downward moves, we have \left\{\begin{aligned} & a+b=n \\ & a-b=j-i \end{aligned}\right.\Longrightarrow \left\{\begin{aligned} & a=\frac{n+j-i}{2} \\ & b=\frac{n-j+i}{2} \end{aligned}\right. \tag{3.9} Then, the n-step transition probability is given by p_{ij}(n)=P(X_n=j|X_0=i)=\left\{\begin{aligned} & {n \choose a}p^a(1-p)^b & n+j-i\, even \\ & 0 & n+j-i\,odd \end{aligned}\right. \tag{3.10} where $$a$$ is given by (3.9).
Example 3.2 (Ehrenfest Diffusion Models) Suppose there are a total of 2A balls in 2 boxes, labeled $$b$$ and $$B$$. At each time, we choose a ball at random, and shifted it from its box of origin to the other box. Let $$X_n$$ be the number of balls at time n in box $$b$$, then $$X_n$$ is a Markov chain. We have P(X_{n+1}=A+j|X_n=A+i)=\left\{\begin{aligned} & \frac{A-i}{2A} & j=i+1 \\ & \frac{A+i}{2A} & j=i-1 \end{aligned}\right. \tag{3.11} for all $$i=-A,\cdots,A$$.

Definition 3.4 (Persistent State) State $$i$$ is called persistent (recurrent) if $$P(X_n=i\, \text{for some}\, n\geq 1|X_0=i)=1$$. This is to say that the probability of the chain eventually return to i, having started from i, is 1.

If this probability is less than 1, state $$i$$ is known as the transient state.

We will be interested in the first passage time defined as $$f_{ij}(n)=P(X_1\neq j,\cdots,X_{n-1}\neq j,X_n=j|X_0=i)$$. This is the probability that state $$j$$ is first visited from state $$i$$ at time n. Write $$f_{ij}=\sum_{n=1}^{\infty}f_{ij}(n)$$, it is the probability that state $$j$$ is ever visited from state $$i$$. If $$f_{ij}=1$$, we are interested in the constraints it imples on the transition probability.