Discrete Time Markov Chains: Joint, Conditional, and Marginal Distributions

Every day for lunch you have either a sandwich (state 1), a burrito (state 2), or pizza (state 3). Suppose your lunch choices from one day to the next follow a MC with transition matrix

$P = [\begin{matrix} 0 & 0.5 & 0.5 \\ 0.1 & 0.4 & 0.5 \\ 0.2 & 0.3 & 0.5 \end{matrix}]$

Suppose today is Monday and consider your upcoming lunches.

Monday is day 0, Tuesday is day 1, etc.
You start with pizza on day 0 (Monday).
Let $T$ be the first time (day) you have a sandwich.
Let $V$ be the number of times (days) you have a burrito this five-day work week.
Pizza costs $5, burrito $7, and sandwich $9.

Compute and interpret in context $P (T > 4)$ .

P(T > 4) = 1 - P(T <= 4)

= 1 - (P(T = 1) + P(T = 2) + P(T = 3) + P(T = 4))

= 1 - ((0.2) +

(0.3 * 0.1) + (0.5 * 0.2) +

((0.2 * 0.5 * 0.1) + (0.2 * 0.5 * 0.2) + (0.3 * 0.4 * 0.1) + (0.3 * 0.5 * 0.2) + (0.5 * 0.3 * 0.1) + (0.5 * 0.5 * 0.2)) +

((0.2 * 0.5 * 0.4 * 0.1) + (0.2 * 0.5 * 0.5 * 0.2) + (0.3 * 0.1 *0.5 * 0.1) + (0.3 * 0.1 * 0.5 * 0.2) + (0.3 * 0.4 * 0.4 * 0.1) + (0.3 * 0.4 * 0.5 * 0.2) + (0.3 * 0.5 * 0.3 * 0.1) + (0.3 * 0.5 * 0.5 * 0.2) + (0.5 * 0.2 * 0.5 * 0.1) + (0.5 * 0.2 * 0.5 * 0.2) + (0.5 * 0.3 * 0.4 * 0.1) + (0.5 * 0.3 * 0.5 * 0.2) + (0.5 * 0.5 * 0.3 * 0.1) + (0.5 * 0.5 * 0.5 * 0.2))

= 1 - (0.2 + 0.13 + 0.137 + 0.1233)) = 0.4097

The probability the first day you have a sandwich is after Saturday is 0.533.

Find the marginal distribution of $V$ , and interpret in context $P (V = 2)$ .

library(tidyverse)

── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr     1.1.4     ✔ readr     2.1.5
✔ forcats   1.0.0     ✔ stringr   1.5.1
✔ ggplot2   3.5.1     ✔ tibble    3.2.1
✔ lubridate 1.9.3     ✔ tidyr     1.3.1
✔ purrr     1.0.2     
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors

res <- data.frame()
p <- matrix(
  c(0, 0.5, 0.5,
    0.1, 0.4, 0.5,
    0.2, 0.3, 0.5),
  ncol = 3, byrow = TRUE
)
for (a in 1:3) {
  for (b in 1:3) {
    for (c in 1:3) {
      for (d in 1:3) {
        res <- rbind(res, c(a, b, c, d))
      }
    }
  }
}
for (i in (1:nrow(res))) {
  res[i, 5] <- p[3, res[i, 1]] *
               p[res[i, 1], res[i, 2]] *
               p[res[i, 2], res[i, 3]] *
               p[res[i, 3], res[i, 4]]
}
res$total_burrito <- rowSums(res == "1")
res |> 
  group_by(total_burrito) |> 
  summarize(sum(V5))

# A tibble: 5 × 2
  total_burrito `sum(V5)`
          <dbl>     <dbl>
1             0    0.473 
2             1    0.450 
3             2    0.0765
4             3    0     
5             4    0

The P(V == 2) = 0.0765

Compute the expected total cost of your lunch this work week (Monday through Friday). Interpret this value as a long run average in context.\

Let Xn be the cost of lunch on day n.

E[X0 + X1 + X2 + X3 + X4 | X0 = 5]

= E[X0 | X0 = 5] + E[X1 | X0 = 5] + …

library(expm)

Loading required package: Matrix


Attaching package: 'Matrix'

The following objects are masked from 'package:tidyr':

    expand, pack, unpack


Attaching package: 'expm'

The following object is masked from 'package:Matrix':

    expm

p <- matrix(
  c(0, 0.5, 0.5,
    0.1, 0.4, 0.5,
    0.2, 0.3, 0.5),
  ncol = 3, byrow = TRUE)
5 +
p[3, ] %*% matrix(c(9, 7, 5), nrow = 3) +
(p %^% 2)[3, ] %*% matrix(c(9, 7, 5), nrow = 3) +
(p %^% 3)[3, ] %*% matrix(c(9, 7, 5), nrow = 3) +
(p %^% 4)[3, ] %*% matrix(c(9, 7, 5), nrow = 3)

        [,1]
[1,] 30.2066

$E [\sum_{i = 0}^{4} X_{n} | X_{0} = 5] = 30.2066$

On average, over many many weeks, you spend $30.2066 on lunch every work week.

Describe in detail how, in principle, you could use physical objects (coins, dice, spinners, cards, boxes, etc) to perform by hand a simulation to approximate $E (V | T = 4)$ . Note: this is NOT asking you to compute $E (V | T = 4)$ or how you would compute it using matrices/equations. Rather, you need to describe in words how you would set up and perform the simulation, and how you would use the simulation results to approximate $E (V | T = 5)$ .

Assign a spinner for each row of the transition matrix. So, you have one spinner for states 1, 2, 3. Start with Spinner 3, and keep track of the number of burritos, throwing out all results where the first time you have a sandwich is not Day 4. Take the average to get $E [V | T = 4]$ . For $E [V | T = 5]$ , repeat the same experiment but throw out all results where the 5th day does not have a sandwich.