3.4 Independence

In general, the conditional probability of event $$A$$ given some other event $$B$$ is usually different from the unconditional probability of $$A$$. That is, in general $$\textrm{P}(A | B) \neq \textrm{P}(A)$$. Knowledge of the occurrence of event $$B$$ typically influences the probability of event $$A$$, and vice versa. If so, we say that events $$A$$ and $$B$$ are dependent.

However, in some situations knowledge of the occurrence of one event does not influence the probability of another. For example, if a coin is flipped twice then knowing that the first flip landed on Heads does not change the probability that the second flips lands on Heads. In these situations we say the events are independent.

3.4.1 Independence of events

Example 2.65 Consider the following hypothetical data.

Republican ($$D^c$$) Democrat ($$D$$) Total
Loves puppies ($$L$$) 180 270 450
Does not love puppies ($$L^c$$) 20 30 50
Total 200 300 500

Suppose a person is randomly selected from this group. Consider the events \begin{align*} L & = \{\text{person loves puppies}\}\\ D & = \{\text{person is a Democrat}\} \end{align*}

1. Compute and interpret $$\textrm{P}(L)$$.
2. Compute and interpret $$\textrm{P}(L|D)$$.
3. Compute and interpret $$\textrm{P}(L|D^c)$$.
4. What do you notice about $$\textrm{P}(L)$$, $$\textrm{P}(L|D)$$, and $$\textrm{P}(L|D^c)$$?
5. Compute and interpret $$\textrm{P}(D)$$.
6. Compute and interpret $$\textrm{P}(D|L)$$.
7. Compute and interpret $$\textrm{P}(D|L^c)$$.
8. What do you notice about $$\textrm{P}(D)$$, $$\textrm{P}(D|L)$$, and $$\textrm{P}(D|L^c)$$?
9. Compute and interpret $$\textrm{P}(D \cap L)$$.
10. What is the relationship between $$\textrm{P}(D \cap L$$) and $$\textrm{P}(D)$$ and $$\textrm{P}(L)$$?
11. When randomly selecting a person from this particular group, would you say that events $$D$$ and $$L$$ are independent? Why?
Solution to Example 2.65
1. The probability that the randomly selected person loves puppies is $$\textrm{P}(L)=450/500=0.9$$.
2. The conditional probability that the randomly selected person loves puppies given that the person is a Democrat is $$\textrm{P}(L|D)=270/300=0.9$$.
3. The conditional probability that the randomly selected person loves puppies given that the person is not a Democrat is $$\textrm{P}(L|D^2)=180/200=0.9$$.
4. $(L)=(L|D)=(L|D^c)=0.9. Regardless of whether or not the person is a Democrat the probability that they love puppies is 0.9. 5. The probability that the randomly selected person is a Democrat is $$\textrm{P}(D)=300/500=0.6$$. 6. The conditional probability that the randomly selected person is a Democrat given that the person loves puppies is $$\textrm{P}(D|L)=270/450=0.6$$. 7. The conditional probability that the randomly selected person is a Democrat given that the person does not love puppies is $$\textrm{P}(D|L^c)=30/50=0.6$$. 8.$(D)=(D|L)=(D|L^c)=0.6. Regardless of whether or not the person loves puppies the probability that the person is a Democrat is 0.6.
9. The probability that the random selected person is a Deomcrat and loves puppies is $$\textrm{P}(D \cap L)=270/500=0.54$$.
10. $$\textrm{P}(D \cap L) = 0.54 = (0.6)(0.9)=\textrm{P}(D)\textrm{P}(L)$$. The joint probability is a product of the marginal probabilities.
11. Yes, the events $$D$$ and $$L$$ are independent. Knowing that whether or not the person is a Democrat does not change the probability that the person loves puppies, and vice versa.

As in the example,events $$A$$ and $$B$$ are independent if the knowing whether or not one occurs does not change the probability of the other.

Definition 2.19 Two events $$A$$ and $$B$$ defined on a probability space with probability measure $$\textrm{P}$$ are independent if $$\textrm{P}(A\cap B) = \textrm{P}(A)\textrm{P}(B)$$. That is, two events are independent if their joint probability is the product of their marginal probabilities.

Theorem 2.5 For events $$A$$ and $$B$$ with $$0<\textrm{P}(A)<1$$ and $$0<\textrm{P}(B)<1$$, the following are equivalent. That is, if one is true then they all are true; if one is false, then they all are false.

\begin{align*} \text{A and B} & \text{ are independent}\\ \textrm{P}(A \cap B) & = \textrm{P}(A)\textrm{P}(B)\\ \textrm{P}(A^c \cap B) & = \textrm{P}(A^c)\textrm{P}(B)\\ \textrm{P}(A \cap B^c) & = \textrm{P}(A)\textrm{P}(B^c)\\ \textrm{P}(A^c \cap B^c) & = \textrm{P}(A^c)\textrm{P}(B^c)\\ \textrm{P}(A|B) & = \textrm{P}(A)\\ \textrm{P}(A|B) & = \textrm{P}(A|B^c)\\ \textrm{P}(B|A) & = \textrm{P}(B)\\ \textrm{P}(B|A) & = \textrm{P}(B|A^c)\\ \end{align*}

3.4.2 Independence versus uncorrelatedness

Find $$\textrm{Cov}(X,Y)$$ in each of the following situations. Notice that the marginal distribution of $$X$$ is the same for each part, and similarly for $$Y$$.

$x$    -1     0     1
0   1/6   1/6   1/6
1   1/6   1/6   1/6

$x$     -1      0      1
0   5/24   2/24   5/24
1   3/24   6/24   3/24

$x$     -1      0      1
0   1/12   3/12   2/12
1   3/12   1/12   2/12

In which of the cases are $$X$$ and $$Y$$ independent? How do you know?

If $$X$$ and $$Y$$ are independent then $$X$$ and $$Y$$ are uncorrelated, that is, $$\textrm{Cov}(X, Y) = 0$$.

However, the converse is false.

If $$X$$ and $$Y$$ are independent then $$\textrm{E}(Y|X)=\textrm{E}(Y)$$ and $$\textrm{E}(X|Y)=\textrm{E}(X)$$.

Note: in general, $$\textrm{E}(Y|X)=\textrm{E}(Y)$$ is a nonsensical statement: on the left is a random variable (a function) and on the right is a number. What we really mean is $$\textrm{P}(\textrm{E}(Y|X)=\textrm{E}(Y))=1$$.It doesn’t really . However, we can consider the event $$\{\textrm{E}(Y|X) = \textrm{E}(X)\}=\{\omega\in\Omega:\textrm{E}(Y|X)(\omega) = \textrm{E}(Y)\}$$

If either $$\textrm{E}(Y|X)=\textrm{E}(Y)$$ and $$\textrm{E}(X|Y)=\textrm{E}(X)$$ then $$X$$ and $$Y$$ are uncorrelated.

But the converse is not true.

Example 2.66 Let $$X$$ and $$Y$$ be discrete random variables with joint pmf

 $$x$$ \ $$y$$ -1 0 1 -1 0.1 0 0.25 0 0.3 0 0 1 0 0.2 0.15
1. Find $$\textrm{E}(Y)$$.
2. Find $$\textrm{E}(Y|X=x)$$ for each $$x$$. Is $$\textrm{E}(Y|X)$$ constant?
3. Find $$\textrm{E}(X)$$.
4. Find $$\textrm{E}(X|Y=y)$$ for each $$y$$. Is $$\textrm{E}(X|Y)$$ constant?
5. Find $$\textrm{Cov}(X, Y)$$. Are $$X$$ and $$Y$$ uncorrelated?
6. Are $$X$$ and $$Y$$ independent?
Solution to Example 2.66
1. $$\textrm{E}(Y)= (-1)(0.40) + (0)(0.20)+(1)(0.40) = 0$$.
2. No, as is true in general, $$\textrm{E}(Y|X)$$ is not constant in this example. \begin{align*} \textrm{E}(Y|X=-1) & = (-1)(10/35) + (0)(0)+(1)(20/35) = 2/7\\ \textrm{E}(Y|X=0) & = (-1)(1) + (0)(0)+(1)(0) = -1\\ \textrm{E}(Y|X=1) & = (-1)(0) + (0)(20/35)+(1)(10/35) = 2/7\\ \end{align*} 1.$$\textrm{E}(X)= (-1)(0.40) + (0)(0.20)+(1)(0.40) = 0$$.
3. No, as is true in general, $$\textrm{E}(X|Y)$$ is not constant in this example. \begin{align*} \textrm{E}(X|Y=-1) & = (-1)(1/4) + (0)(3/4)+(1)(0) = -1/4\\ \textrm{E}(X|Y=0) & = (-1)(0) + (0)(0)+(1)(1) = 1\\ \textrm{E}(X|Y=1) & = (-1)(25/40) + (0)(0)+(1)(15/40) = -1/4\\ \end{align*}
4. $$\textrm{E}(XY)= (-1)(-1)(0.1)+(1)(-1)(0.25) + (1)(-1)(0)+(1)(1)(0.15)+0=0$$ so $$\textrm{Cov}(X, Y)=\textrm{E}(XY)-\textrm{E}(X)\textrm{E}(Y)=0$$. Yes, $$X$$ and $$Y$$ are uncorrelated.
5. No, $$X$$ and $$Y$$ are not independent. For example $$\textrm{E}(Y|X=-1)=2/7$$ but $$\textrm{E}(Y|X=0)=-1$$.

$$X$$ and $$Y$$ are independent if and only if $\textrm{E}[g(X)h(Y)] = \textrm{E}[g(X)]\textrm{E}[h(Y)] \qquad \text{for all functions g, h}$