1.2 Random Variables

Random variable is a mathematical concept used in probability theory and statistics to model and quantify uncertainty in various situations. It represents a numerical outcome of a random phenomenon or experiment, and its value is determined by chance.

Understanding random variables is crucial in statistical analysis, as they provide a formal way to model and analyze uncertain or random phenomena in a wide range of fields, including finance, physics, biology, and more.

1.2.1 Random variables and probability functions

1.2.1.1 Random variables

Definition:

1. A random variable (r.v.) $X$ is a real-valued (measurable) function which maps the sample space $\Omega$ to the real numbers $\mathbb{R}$, i.e. $X:\Omega \rightarrow \mathbb{R}$. For any sample point $\omega\in\Omega$, we have $X(\omega)=x, x\in \mathbb{R}$.

2. Define the range of a random variable $X$ is $R_X = \{ x|x \in X(\omega), \forall \omega\in\Omega \}$

Remark: The range of a random variable $X$ is also called support, denoted by $supp(X)$.

Here we give the definition of random variable in a simple way. For other definition, see Course 47.

We can divide the random variables into the following two types:

1. If $R_X$ is finite or countable, then it’s a discrete random variable.
2. If $R_X$ is uncountable, then it’s a continuous random variable.

Example:

1. Tossing two fair coins. Define a random variable $X=$ the number of heads. Then $\Omega=\{HH,HT,TH,TT\}, R_X=\{0,1,2\}$.

2. $X=$ the number of car accident in a day. $\Omega=\mathbb{N}=R_X$

3. $X=$ the height (cm) of a person. $\Omega=\mathbb{R}=R_X$ (actually some values are impossible).

1.2.1.2 Discrete Probability Function

Definition:

Suppose $X$ is a discrete random variable with range $R_X$.

1. Define the function $$p_X(x) = \begin{cases} P(X=x), & \forall x \in R_X \\ 0, & \forall x \notin R_X \\ \end{cases}$$ Then $p_X(x)$ is called probability mass function (PMF) of random variable $X$.

2. Define $\displaystyle F_X(a) = P(X \leq a) = \sum_{x \leq a} p_X(x), \quad x \in \mathbb{R}$, then $F_X(x)$ is called the cumulative distribution function (CDF) of random variable $X$.

Proposition:

1. $0 \leq p_X(x) \leq 1, \forall x \in R_X$
2. $\displaystyle \sum_{x \in R_X} p_X(x) = 1$
3. $0 \leq F_X(x) \leq 1, \forall x \in \mathbb{R}$
4. $\displaystyle \lim_{x \to -\infty} F_X(x) = 0, \quad \lim_{x \to \infty} F_X(x) = 1$
5. $\displaystyle \lim_{x \to c^+} F_X(x) = F_X(c^+) = F_X(c), \quad \forall c \in \mathbb{R}$ (right continuous)
6. If $a<b$, then $F_X(a) \leq F_X(b)$ (non-decreasing)
7. $F_X(x)$ is a step function
8. $p_X(x) = F_X(x)-F_X(x^-)$
9. If $a<b$, then $\displaystyle P(a < x \leq b) = F_X(b) - F_X(a) = \sum_{a < x \leq b} p_X(x)$

1.2.1.3 Continuous Probability Function

Now we discuss the continuous case

Definition:

Suppose $X$ is a continuous random variable with range $R_X$. Then

1. $F_X(a) = P(X \leq a), \quad x \in \mathbb{R}$ is called the cumulative distribution function (CDF) of random variable $X$.

2. Define $$f_X(x) = \begin{cases} F'_X(x) = \frac{dF_X(x)}{dx}, & \mbox{if derivative exist at} x \\ 0, & \mbox{otherwise} \\ \end{cases}$$ Then $f_X(x)$ is called probability density function (PDF) of random variable $X$.

Proposition:

1. $f_X(x) \geq 0, \quad \forall x \in R_X$
2. $\int_{x \in R_X} f_X(x) \, dx = 1$
3. $F_X(a) = \int_{-\infty}^a f_X(x) \, dx$
4. $P(X=a) = 0$
5. If $a<b$, then $P(a<x<b) = F_X(b) - F_X(a) = \int_a^b f_X(x) \, dx$
6. $0 \leq F_X(x) \leq 1, \forall x \in \mathbb{R}$
7. $\displaystyle \lim_{x \to -\infty} F_X(x) = 0, \quad \lim_{x \to \infty} F_X(x) = 1$
8. $\displaystyle \lim_{x \to c^-} F_X(x) = F_X(c^-) = F_X(c) = F_X(c^+) = \lim_{x \to c^+} F_X(x), \quad \forall c \in \mathbb{R}$ (continuous)
9. If $a<b$, then $F_X(a) \leq F_X(b)$ (non-decreasing)

1.2.1.4 * Mixed Type Probability Function

A part of random variable is discrete, while another part is continuous

Proposition:

Suppose $X$ is a mixed type random variable. Then $$F_X(x) = aF_d(x) + (1-a)F_c(x), \quad 0<a<1,$$ where $F_d(x)$ is a discrete random variable, $F_c(x)$ is a continuous random variable.

Example: $$F_X(x) = \begin{cases} 0, & x<0 \\ x^2+0.1, & 0 \leq x < 0.5 \\ x, & 0.5 \leq x < 1 \\ 1, & x \geq 1 \end{cases}$$

TBD (write into decomposition and find prob. func.)

1.2.2 Expected values and Variance

Definition:

Let $X$ be a random variable with range $R_X$.

1. Define the expected value of a random variable $\mathbb{E}(X)$ by $$\mu = \mathbb{E}(X) = \begin{cases} \displaystyle \sum_{x \in R_X} xp_X(x), & X \mbox{ is discrete} \\ \displaystyle \int_{x \in R_X} xf_X(x) \, dx, & X \mbox{ is continuous} \\ \end{cases}$$
2. Define the variance of a random variable $\mathrm{Var}(X)$ by $$\sigma^2 = \mathrm{Var}(X) = \mathbb{E}[(X-\mu)^2] = \begin{cases} \displaystyle \sum_{x \in R_X} (x-\mu)^2 p_X(x), & X \mbox{ is discrete} \\ \displaystyle \int_{x \in R_X} (x-\mu)^2 f_X(x) \, dx, & X \mbox{ is continuous} \\ \end{cases}$$
3. Define the standard deviation of a random variable $\mathrm{SD}(X)$ by $$\sigma = \mathrm{SD}(X) = \sqrt{\mathrm{Var}(X)}$$

Theorem: (the rule of the lazy statistician)

Let $X$ be a random variable with range $R_X$, and $g(\cdot)$ be any real-valued function, then $$\mathbb{E}[g(X)] = \begin{cases} \displaystyle \sum_{x \in R_X} g(x) p_X(x), & X \mbox{ is discrete} \\ \displaystyle \int_{x \in R_X} g(x) f_X(x) \, dx, & X \mbox{ is continuous} \\ \end{cases}$$

(the proof is not trivial !)

Proposition:

Suppose $X$ is a random variable, $a,b,c \in \mathbb{R}$, $g(\cdot)$ is any real-valued function, then

1. $\mathbb{E}(X^2) = \sigma^2 + \mu^2$
2. $\mathbb{E}[ag(X)+b] = a\mathbb{E}[g(X)]+b$
3. $\mathrm{Var}(X) \geq 0$
4. $\mathrm{Var}(c)=0$
5. $\mathrm{Var}(ag(X)+b) = a^2\mathrm{Var}(g(X))$
6. $\mathrm{Var}(X) = \mathbb{E}(X^2) - [\mathbb{E}(X)]^2$
7. $\mathrm{SD}(X) \geq 0$
8. $\mathrm{SD}(c)=0$
9. $\mathrm{SD}(ag(X)+b) = |a|\mathrm{SD}(g(X))$

1.2.2.1 *Approximation of a random variable

If we want to calculate the expectation and variance of the transformation of a random variable $g(X)$, then we can use Taylor expansion to approximate it’s value.

Expand $g(X)$ at $X=\mu$: $$\begin{split} g(X) &= \frac{g(\mu)}{0!}(X-\mu)^0 + \frac{g'(\mu)}{1!}(X-\mu)^1 + \frac{g''(\mu)}{2!}(X-\mu)^2 + \cdots \\ &= g(\mu) + g'(\mu)(X-\mu) + \frac{g''(\mu)}{2}(X-\mu)^2 + \cdots \end{split}$$

Take expectation with the first three terms: $$\begin{split} \mathbb{E}[g(X)] &\approx \mathbb{E}[g(\mu)] + g'(\mu)\mathbb{E}[(X-\mu)] + \frac{1}{2}g''(\mu)\mathbb{E}[(X-\mu)^2] \\ &= g(\mu) + \frac{1}{2}g''(\mu)\sigma^2 \end{split}$$

Take variance with the first two terms: $$\begin{split} \mathrm{Var}[g(X)] &\approx \mathrm{Var}[g(\mu)] + [g'(\mu)]^2 \mathrm{Var}[(X-\mu)] \\ &= [g'(\mu)]^2 \sigma^2 \end{split}$$

This is related to a method in statistics called the $\Delta$-method, which is a method of deriving the asymptotic distribution of a random variable. We will introduce in Chapter ??.

1.2.3 Transformation of random variables

A change of variable is a technique used in calculus and statistics to simplify integrals or PDFs by introducing a new variable. This change is often made to simplify the form of an expression, make calculations more manageable, or reveal underlying patterns.

In calculus, a change of variable involves substituting a new variable in place of the original one. This substitution is typically chosen to simplify the integral, making it easier to integrate.

In statistics, change of variable is often used in the context of probability density functions. If you have a random variable $X$ with a known distribution and you want to find the distribution of a function of $X$, you can use the change of variable formula for PDFs. This involves finding the derivative of the transformation function and using it to adjust the PDF of the original random variable.

1.2.3.1 Discrete r.v. transformation

Assume $X$ is a known r.v., let $Y=g(X)$ is a transformation of r.v. $X$. Then we can calculate the PMF of $Y$ by the following formula: $$p_Y(a) = P(Y=a) = P(g(X)=a) = P(X=g^{-1}(a)) = p_X(g^{-1}(a))$$

Note that sometimes $g^{-1}(\cdot)$ may be difficult to calculate or does not exist. In this case, you can make a table to calculate PMF of $Y$ or discuss by different cases.

Example 1.15 Let $X$ be a r.v. with PMF: $p_X(x) = \frac{x}{10}, \quad x=1,2,3,4$. Find the PMF of the r.v. $Y=3X-2$.

• Method 1:

  $p_Y(y) = P(Y=y) = P(3X-2=y) = P(X=\frac{y+2}{3}) = p_X(\frac{y+2}{3}) = \frac{y+2}{30}, \quad y=1,4,7,10$

• Method 2:

Prob.	$X$	$Y=3X-2$
1/10	1	1
2/10	2	4
3/10	3	7
4/10	4	10

Hence $$p_Y(y)= \begin{cases} \frac{1}{10} &, y=1\\ \frac{1}{5} &, y=4\\ \frac{3}{10} &, y=7\\ \frac{2}{5} &, y=10\\ \end{cases}$$

Example 1.16 Let $X$ be a r.v. with PMF: $p_X(x) = \frac{x^2}{10}, \quad x=-2,-1,1,2$. Find the PMF of the r.v. $Y=|X|$.

• Method 1:

  Since $Y=|X|$ does not have inverse function on $[-2,2]$. Discuss by different cases:

  • Case 1: $X\geq0$

    $p_Y(a) = P(Y=a) = P(|X|=a) = P(X=a) = p_X(a) = \frac{a^2}{10}, \quad a=|1|,|2|$

  • Case 2: $X<0$

    $p_Y(a) = P(Y=a) = P(|X|=a) = P(X=-a) = p_X(-a) = \frac{(-a)^2}{10}, \quad a=|-1|,|-2|$

  Hence, $p_Y(y) = \frac{y^2}{5}, \quad y=1,2$

• Method 2:

Prob.	$X$	$Y=|X|$
4/10	-2	2
1/10	-1	1
1/10	1	1
4/10	2	2

Hence $$p_Y(y)= \begin{cases} \frac{1}{5} &, y=1\\ \frac{4}{5} &, y=2\\ \end{cases}$$

1.2.3.2 Continuous r.v. transformation

For continuous r.v., suppose a continuous r.v. has PDF $f_X(x)$, let $Y=g(X)$ is a transformation of r.v. $X$. Then we have the following two methods to find PDF of $Y$, $f_Y(y)$:

• Calculate from CDF:

$$F_Y(a) = P(Y \leq a) = P(g(X)\leq a) = \begin{cases} P(X \leq g^{-1}(a)) = F_X(g^{-1}(a)) &, \mbox{if } g(\cdot) \mbox{ is increasing}\\ P(X \geq g^{-1}(a)) = 1-F_X(g^{-1}(a)) &, \mbox{if } g(\cdot) \mbox{ is decreasing} \end{cases}$$ Hence, $f_Y(y)=\frac{dF_Y(y)}{dy}$.

• Use Jacobian:

If $g(\cdot)$ is one-to-one ($g^{-1}(\cdot)$ exists), then $f_Y(y)=f_X(g^{-1}(y)) \left|\frac{dg^{-1}(y)}{dy}\right|$

Example 1.17 Let $Z$ be a r.v. with PDF $f_Z(z)=\frac{1}{\sqrt{2\pi}}e^{-\frac{z^2}{2}}, \quad z \in \mathbb{R}$, find the PDF of $Y=X^2$

$F_Y(y) = P(Y \leq y) = P(Z^2 \leq y) = P(-\sqrt{y} \leq Z \leq \sqrt{y}) = F_Z(\sqrt{y})-F_Z(-\sqrt{y})$

Therefore, $$\begin{split} f_Y(y) &= F_Y^{'}(y) \\ &= f_Z(\sqrt{y})\frac{1}{2\sqrt{y}}-f_Z(-\sqrt{y})\frac{-1}{2\sqrt{y}} \\ &= \frac{1}{\sqrt{2\pi}}e^{-\frac{(\sqrt{y})^2}{2}}\frac{1}{2\sqrt{y}}+\frac{1}{\sqrt{2\pi}}e^{-\frac{(-\sqrt{y})^2}{2}}\frac{1}{2\sqrt{y}} \\ &= \frac{1}{\sqrt{2\pi y}}e^{-\frac{y}{2}}, \quad y>0 \end{split}$$