19.2 Semi-random Experiment

Chicago Open Enrollment Program

• Students can apply to “choice” schools

• Many schools are oversubscribed (Demand > Supply)

• Resolve scarcity via random lotteries

• Non-random enrollment, we only have random lottery which mean the above

Let

$\delta_j = E(Y_i | Enroll_{ij} = 1; Apply_{ij} = 1) - E(Y_i | Enroll_{ij} = 0; Apply_{ij} = 1)$

and

$\theta_j = E(Y_i | Win_{ij} = 1; Apply_{ij} = 1) - E(Y_i | Win_{ij} = 0; Apply_{ij} = 1)$

Hence, we can clearly see that $$\delta_j \neq \theta_j$$ because you can only enroll, but you cannot ensure that you will win. Thus, intention to treat is different from treatment effect.

Non-random enrollment, we only have random lottery which means we can only estimate $$\theta_j$$

To recover the true treatment effect, we can use

$\delta_j = \frac{E(Y_i|W_{ij} = 1; A_{ij} = 1) - E(Y_i | W_{ij}=0; A_{ij} = 1)}{P(Enroll_{ij} = 1| W_{ij}= 1; A_{ij}=1) - P(Enroll_{ij} = 1| W_{ij}=0; A_{ij}=1)}$

where

• $$\delta_j$$ = treatment effect

• $$W$$ = Whether students win the lottery

• $$A$$ = Whether student apply for the lottery

• i = application

• j = school

Say that we have

10 win

Number students Type Selection effect Treatment effect Total effect
1 Always Takers +0.2 +1 +1.2
2 Compliers 0 +1 +1
7 Never Takers -0.1 0 -0.1

10 lose

Number students Type Selection effect Treatment effect Total effect
1 Always Takers +0.2 +1 +1.2
2 Compliers 0 0 0
7 Never Takers -0.1 0 -0.1

Intent to treatment = Average effect of who you give option to choose

\begin{aligned} E(Y_i | W_{ij}=1; A_{ij} = 1) &= \frac{1*(1.2)+ 2*(1) + 7 * (-0.1)}{10}\\ &= 0.25 \end{aligned}

\begin{aligned} E(Y_i | W_{ij}=0; A_{ij} = 1) &= \frac{1*(1.2)+ 2*(0) + 7 * (-0.1)}{10}\\ &= 0.05 \end{aligned}

Hence,

\begin{aligned} \text{Intent to treatment} &= 0.25 - 0.05 = 0.2 \\ \text{Treatment effect} &= 1 \end{aligned}

\begin{aligned} P(Enroll_{ij} = 1 | W_{ij} = 1; A_{ij}=1 ) &= \frac{1+2}{10} = 0.3 \\ P(Enroll_{ij} = 1 | W_{ij} = 0; A_{ij}=1 ) &= \frac{1}{10} = 0.1 \end{aligned}

$\text{Treatment effect} = \frac{0.2}{0.3-0.1} = 1$

After knowing how to recover the treatment effect, we turn our attention to the main model

$Y_{ia} = \delta W_{ia} + \lambda L_{ia} + e_{ia}$

where

• $$W$$ = whether a student wins a lottery

• $$L$$ = whether student enrolls in the lottery

• $$\delta$$ = intent to treat

Hence,

• Conditional on lottery, the $$\delta$$ is valid

• But without lottery, your $$\delta$$ is not random

• Winning and losing are only identified within lottery

• Each lottery has multiple entries. Thus, we can have within estimator

We can also include other control variables ($$X_i \theta$$)

$Y_{ia} = \delta_1 W_{ia} + \lambda_1 L_{ia} + X_i \theta + u_{ia}$

\begin{aligned} E(\delta) &= E(\delta_1) \\ E(\lambda) &\neq E(\lambda_1) && \text{because choosing a lottery is not random} \end{aligned}

Including $$(X_i \theta)$$ just shifts around control variables (i.e., reweighting of lottery), which would not affect your treatment effect $$E(\delta)$$

References

Cullen, Julie Berry, Brian A Jacob, and Steven D Levitt. 2005. “The Impact of School Choice on Student Outcomes: An Analysis of the Chicago Public Schools.” Journal of Public Economics 89 (5-6): 729–60.