23  Covariance and Correlation

Quantities like expected value and variance summarize characteristics of the marginal distribution of a single random variable. When there are multiple random variables their joint distribution is of interest. Covariance and correlation summarize a characteristic of the joint distribution of two random variables, namely, the degree to which they “co-deviate from the their respective means”.

23.1 Covariance

  • The covariance between two random variables \(X\) and \(Y\) is \[\begin{align*} \text{Cov}(X,Y) & = \text{E}\left[\left(X-\text{E}[X]\right)\left(Y-\text{E}[Y]\right)\right]\\ & = \text{E}(XY) - \text{E}(X)\text{E}(Y) \end{align*}\]
  • Covariance is the long run average of the products of the paired deviations from the mean
  • Covariance is the expected value of the product minus the product of expected values.

Example 23.1

Consider the probability space corresponding to two rolls of a fair four-sided die. Let \(X\) be the sum of the two rolls, \(Y\) the larger of the two rolls, \(W\) the number of rolls equal to 4, and \(Z\) the number of rolls equal to 1.

  1. Specify the joint pmf of \(X\) and \(Y\). (These first few parts are review of things we have covered before.)


  2. Find the marginal pmf of \(X\) and \(\text{E}(X)\).


  3. Find the marginal pmf of \(Y\) and \(\text{E}(Y)\).


  4. Find \(\text{E}(XY)\). Is it equal to \(\text{E}(X)\text{E}(Y)\)?




  5. Find \(\text{Cov}(X, Y)\). Why is the covariance positive?




  6. Is \(\text{Cov}(X, W)\) positive, negative, or zero? Why?




  7. Is \(\text{Cov}(X, Z)\) positive, negative, or zero? Why?




  8. Let \(V=W+Z\). Is \(\text{Cov}(X, V)\) positive, negative, or zero? Why?




  9. Is \(\text{Cov}(W, Z)\) positive, negative, or zero? Why?




  • \(\text{Cov}(X,Y)>0\) (positive association): above average values of \(X\) tend to be associated with above average values of \(Y\)
  • \(\text{Cov}(X,Y)<0\) (negative association): above average values of \(X\) tend to be associated with below average values of \(Y\)
  • \(\text{Cov}(X,Y)=0\) indicates that the random variables are uncorrelated: there is no overall positive or negative association. But be careful: if \(X\) and \(Y\) are uncorrelated there can still be a relationship between \(X\) and \(Y\); there is just no overall positive or negative association.. We will see examples that demonstrate that being uncorrelated does not necessarily imply that random variables are independent.

Example 23.2

What is another name for \(\text{Cov}(X, X)\)?






23.2 Correlation

Example 23.3

The covariance between height in inches and weight in pounds for football players is 96.

  1. What are the measurement units of the covariance?


  2. Suppose height were measured in feet instead of inches. Would the shape of the joint distribution change? Would the strength of the association between height and weight change? Would the value of covariance change?




  • The correlation (coefficient) between random variables \(X\) and \(Y\) is \[\begin{align*} \text{Corr}(X,Y) & = \text{Cov}\left(\frac{X-\text{E}(X)}{\text{SD}(X)},\frac{Y-\text{E}(Y)}{\text{SD}(Y)}\right)\\ & = \frac{\text{Cov}(X, Y)}{\text{SD}(X)\text{SD}(Y)} \end{align*}\]
  • The correlation for two random variables is the covariance between the corresponding standardized random variables. Therefore, correlation is a standardized measure of the association between two random variables.
  • A correlation coefficient has no units and is measured on a universal scale. Regardless of the original measurement units of the random variables \(X\) and \(Y\)

\[ -1\le \textrm{Corr}(X,Y)\le 1 \]

  • \(\textrm{Corr}(X,Y) = 1\) if and only if \(Y=aX+b\) for some \(a>0\)
  • \(\textrm{Corr}(X,Y) = -1\) if and only if \(Y=aX+b\) for some \(a<0\)
  • Therefore, correlation is a standardized measure of the strength of the linear association between two random variables.
  • Covariance is the correlation times the product of the standard deviations.

\[ \text{Cov}(X, Y) = \text{Corr}(X, Y)\text{SD}(X)\text{SD}(Y) \]

Example 23.4

The covariance between height in inches and weight in pounds for football players is 96. Heights have mean 74 and SD 3 inches. Weights have mean 250 pounds and SD 45 pounds. Find the correlation between weight and height.






23.3 Independent versus uncorrelated

Example 23.5

Find \(\text{Cov}(X,Y)\) in each of the following situations. Notice that the marginal distribution of \(X\) is the same for each part, and similarly for \(Y\).

  1. \(p_{X, Y}(x, y)\)
    \(x\) \ \(y\) -1 0 1 \(p_X(x)\)
    0 1/6 1/6 1/6 1/2
    1 1/6 1/6 1/6 1/2
    \(p_Y(y)\) 1/3 1/3 1/3
  2. \(p_{X, Y}(x, y)\)
    \(x\) \ \(y\) -1 0 1 \(p_X(x)\)
    0 5/24 2/24 5/24 1/2
    1 3/24 6/24 3/24 1/2
    \(p_Y(y)\) 1/3 1/3 1/3
  3. In which of the previous parts are \(X\) and \(Y\) independent? Why?




  • If \(X\) and \(Y\) are independent then \(X\) and \(Y\) are uncorrelated, that is, \(\text{Cov}(X, Y) = 0\).
  • However, the converse is not true in general: there are many situations in which \(\text{Cov}(X, Y) = 0\) but \(X\) and \(Y\) are not independent.
  • Independence means there is no probabilistic relationship between two random variables.
  • Covariance only measures a particular kind of probabilistic relationship, namely: how closely do the \((X, Y)\) pairs tend to follow a single straight line with non-zero slope?