Chapter 5 First Visit Time, Mean Recurrent Time (Lecture on 01/19/2021)

We discussed random walk in one dimension. We have showed $\sum_{n=1}^{\infty}p_{00}(2n)=\infty$ for a one-dimensional random walk. Which implies that state 0 is a recurrent state in a one-dimensional random walk.

Question: Is state 0 continues to be a recurrent state for higher dimensional random walk?

Example 5.1 (Two Dimensional Random Walk) One is able to move to one step up, one step down, one step left or one step right. Here also, if the probability of the four moves are unequal, 0 is going to be a transient state.

If all these moves have equal probability, is state 0 recurrent?

Notice that $p_{00}(2n+1)=0$ and if one want to get back to state 0 in $2n$ step, we would have $i$ step up, $i$ step down, $n-i$ step right and $n-i$ step left. Therefore, we have $$\begin{equation} \begin{split} p_{00}(2n)&= \sum_{i+j = n}\frac{2n!}{i!i!j!j!}(\frac{1}{4})^{2n} \\ & = \frac{2n!}{n!n!}(\frac{1}{4})^{2n} \sum_{i+j=n}\frac{n!n!}{i!i!j!j!} \\ & = {2n \choose n}(\frac{1}{4})^{2n} \sum_{i + j=n}{n \choose j} {n \choose i} \\ & = {2n \choose n}(\frac{1}{4})^{2n} {2n \choose n} \\ & = (\frac{1}{4})^{2n}[\frac{2n!}{n!n!}]^2\quad \text{(Stirling approximation)}\\ &\approx [\frac{(2n)^{2n+1/2}e^{-2n}\sqrt{2\pi}}{(n^{n+1/2})^2e^{-2n}(\sqrt{2\pi})^2}]^2(\frac{1}{4})^{2n}\\ &=[\frac{2^{2n+\frac{1}{2}}}{\sqrt{n}\sqrt{2\pi}}]^2(\frac{1}{4})^{2n}\\ &=\frac{1}{\pi n} \end{split} \tag{5.1} \end{equation}$$

Thus, we have $p_{00}(2n)\approx\frac{1}{\pi n}$ and $p_{00}(2n+1)=0$, we have $\sum_{n=1}^{\infty}p_{00}(n)=\sum_{n=1}^{\infty}p_{00}(2n)\approx\sum_{n=1}^{\infty}\frac{1}{\pi n}=\infty$. Therefore, $0$ is a recurrent state.

Example 5.2 (Three Dimensional Random Walk) For three dimensional random walk, there are three axis, denoted as $X,Y$ and $Z$. For each step, one can choose one of the three axis, then move 1 step on one side, or 1 step on the other side. Therefore, there are in total 6 possible directions to move. We investigate state 0 when all these moves have equal probability $\frac{1}{6}$.

Similarly, we have $p_{00}(2n+1)=0$ and for $2n$th state to revisit state 0, we need to have $i$ step up and $i$ step down in $X$ axis, $j$ step up and $j$ step down in $Y$ axis, and $n-(i+j)$ step up and $n-(i+j)$ step down in $Z$ axis. $$\begin{equation} \begin{split} p_{00}(2n)&= \sum_{0\leq i+j\leq n}\frac{2n!}{i!i!j!j!(n-i-j)!(n-i-j)!}(\frac{1}{6})^{2n} \\ & = \frac{1}{2^{2n}}\frac{2n!}{n!n!}\sum_{0\leq i+j\leq n}\frac{n!n!}{i!i!j!j!(n-i-j)!(n-i-j)!}(\frac{1}{3})^{2n} \\ & = \frac{1}{2^{2n}}{{2n} \choose n}\sum_{0\leq i+j\leq n}[\frac{n!}{i!j!(n-i-j)!}]^2(\frac{1}{3})^{2n}\\ & = \frac{1}{2^{2n}}{{2n} \choose n}(\frac{1}{3})^{n}\sum_{0\leq i+j\leq n}[\frac{n!}{i!j!(n-i-j)!}]^2(\frac{1}{3})^{n}\\ & \leq \frac{1}{2^{2n}}{{2n} \choose n}(\frac{1}{3})^{n}C_n\sum_{0\leq i+j\leq n}[\frac{n!}{i!j!(n-i-j)!}] (\frac{1}{3})^n \end{split} \tag{5.2} \end{equation}$$ where $C_n=\max_{0\leq i+j\leq n}[\frac{n!}{i!j!(n-i-j)!}]$. Notice that by multinomial formula, we have $$\begin{equation} \sum_{0\leq i+j\leq n}[\frac{n!}{i!j!(n-i-j)!}] (\frac{1}{3})^n=(\frac{1}{3}+\frac{1}{3}+\frac{1}{3})^n=1 \tag{5.3} \end{equation}$$ so (5.2) becomes $p_{00}(2n)\leq \frac{1}{2^{2n}}{{2n} \choose n}(\frac{1}{3})^{n}C_n$. For large $n$, $C_n$ will be maximized by $i=j=\frac{n}{3}$ and $C_n\approx\frac{n!}{(\frac{n}{3})!(\frac{n}{3})!(\frac{n}{3})!}$. Using Stirling approximation, we can get $$\begin{equation} p_{00}(2n)\leq\frac{3\sqrt{3}}{2\pi^{3/2}n^{3/2}} \tag{5.4} \end{equation}$$ and therefore we have $$\begin{equation} \sum_{n=1}^{\infty}p_{00}(2n)\leq\sum_{n=1}^{\infty}\frac{3\sqrt{3}}{2\pi^{3/2}n^{3/2}}<\infty \tag{5.5} \end{equation}$$ 0 is a transient state.

Even for random walk in dimension greater than 3, 0 remains a transient state.

We have defined states to be either transient or recurrent. There is a further binnary classification of recurrent states. Let us define and investigate that.

Definition 5.1 (First Visit Time) Let $T_j=\min\{n\geq 1: X_n=j\}$, i.e. the first time to visit state $j$, with convention that $T_j=\infty$ if the visit never occurs. We have $P(T_i=\infty|X_0=i)>0$ if and only if $i$ is transient and $P(X_i=\infty|X_0=i)=0$ if and only if $i$ is recurrent.

A state is recurrent does not guarantee that $E[T_i|X_0=i]<\infty$.

Definition 5.2 (Mean Recurrence Time) We define the mean recurrence time $\mu_i$ of a state $i$ as $\mu_i=E[T_i|X_0=i]=\left\{\begin{aligned} & \sum_n nf_{ii}(n) &\quad \text{i is persistent}\\ & \infty &\quad \text{i is transient} \end{aligned}\right.$

Since it is not guaranteed that for a persisient state, $\sum_n nf_{ii}(n)<\infty$, for persistent state, we call state $i$ null recurrent (persistent) if $\mu_i=\infty$ and non-null recurrent (positive) if $\mu_i<\infty$.

We have already seen that state 0 in a one-dimensional random walk is recurrent when $p=1-p=\frac{1}{2}$. We are interested in whether it is null recurrent or non-null recurrent.

Example 5.3 (Mean recurrence time for one dimensional random walk) For one dimensional random walk, we have $p_{00}(2n)={{2n} \choose n}(\frac{1}{4})^n$. Therefore, for the generating function, we have $$\begin{equation} \begin{split} P_{00}(s)&=\sum_{n=0}^{\infty}p_{00}(n)s^n=1+\sum_{n=1}^{\infty}p_{00}(2n)s^{2n}+\sum_{n=1}^{\infty}p_{00}(2n+1)s^{2n+1}\\ &=1+\sum_{n=1}^{\infty}p_{00}(2n)s^{2n}=1+\sum_{n=1}^{\infty}{{2n} \choose n}(\frac{1}{4})^ns^{2n}\\ &=(1-s^2)^{-1/2}=\frac{1}{\sqrt{1-s^2}} \end{split} \tag{5.6} \end{equation}$$ where the last equation uses the result $$\begin{equation} (1-x)^{-1/2}=\sum_{n=0}^{\infty}{{2n} \choose n}(\frac{x}{4})^n,\quad |x|<1 \tag{5.7} \end{equation}$$

Then from the result $P_{00}(s)=1+F_{00}(s)P_{00}(s)$, it implies $F_{00}(s)=1-\frac{1}{P_{00}(s)}=1-\sqrt{1-s^2}$.

Now since $F_{00}(s)=\sum_{n=0}^{\infty}f_{00}(n)s^n$, we have $$\begin{equation} \sum_{n=1}^{\infty}nf_{00}(n)=\lim_{s\uparrow 1}\frac{d}{ds}F_{00}(s) \tag{5.8} \end{equation}$$ and $\frac{d}{ds}F_{00}(s)=\frac{d}{ds}[1-\sqrt{1-s^2}]=\frac{s}{\sqrt{1-s^2}}$. Thus, $$\begin{equation} \lim_{s\uparrow 1}\frac{d}{ds}F_{00}(s)=\lim_{s\uparrow 1}\frac{s}{\sqrt{1-s^2}}=\infty \tag{5.9} \end{equation}$$ Therefore, $\mu_0=E[T_0|X_0=0]=\infty$, the mean recurrence time of state 0 is infinte. 0 is a null recurrent state.

Consider the following limit $$\begin{equation} \lim_{s\uparrow 1}F_{00}(s)=\sum_{n=0}^{\infty}f_{00}(n)=f_{00}=\lim_{s\uparrow 1}[1-\sqrt{1-s^2}]=1 \end{equation}$$ This also verifies that 0 is recurrent state from another point of view.