22.5 Advanced Experimental Designs

22.5.1 Semi-Random Experiments

In semi-random experiments, participants are not fully randomized into treatment and control groups. Instead, a structured randomization process ensures fairness while allowing some level of causal inference.

22.5.1.1 Example: Loan Assignment Fairness

A bank wants to evaluate a new loan approval policy while ensuring that the experiment does not unfairly exclude specific demographics.

To maintain fairness:

1. Applicants are first stratified based on income and credit history.
2. Within each stratum, a random subset is assigned to receive the new policy, while others continue under the old policy.

set.seed(123)

# Create stratified groups
data <- data.frame(
  income_group = rep(c("Low", "Medium", "High"), each = 10),
  
  # Stratified randomization
  treatment = sample(rep(c("New Policy", "Old Policy"), each = 15)) 
)

# Display the stratification results
table(data$income_group, data$treatment)
#>         
#>          New Policy Old Policy
#>   High            6          4
#>   Low             6          4
#>   Medium          3          7

This approach ensures that each income group is fairly represented in both treatment and control conditions.

22.5.1.2 Case Study: Chicago Open Enrollment Program

A well-known example of semi-random assignment is the Chicago Open Enrollment Program (Cullen, Jacob, and Levitt 2005), where students could apply to choice schools.

However, since many schools were oversubscribed (i.e., demand exceeded supply), they used a random lottery system to allocate spots.

Thus, while enrollment itself was not fully random, the lottery outcomes were random, allowing researchers to estimate the Intent-to-Treat effect while acknowledging that not all students who won the lottery actually enrolled.

This situation presents a classic case where:

• School choice is not random: Families self-select into applying to certain schools.
• Lottery outcomes are random: Among those who apply, winning or losing the lottery is as good as a random assignment.

Let:

• $Enroll_{ij} = 1$ if student $i$ enrolls in school $j$, and $0$ otherwise.
• $Win_{ij} = 1$ if student $i$ wins the lottery, and $0$ otherwise.
• $Apply_{ij} = 1$ if student $i$ applies to school $j$.

We define:

$$\delta_j = E[Y_i | Enroll_{ij} = 1, Apply_{ij} = 1] - E[Y_i | Enroll_{ij} = 0, Apply_{ij} = 1]$$

$$\theta_j = E[Y_i | Win_{ij} = 1, Apply_{ij} = 1] - E[Y_i | Win_{ij} = 0, Apply_{ij} = 1]$$

where:

• $\delta_j$ is the treatment effect (impact of actual school enrollment).
• $\theta_j$ is the intent-to-treat effect (impact of winning the lottery).

Since not all winners enroll, we know that:

$$\delta_j \neq \theta_j$$

Thus, we can only estimate $\theta_j$ directly, and need an additional method to recover $\delta_j$.

This distinction is crucial because simply comparing lottery winners and losers does not measure the true effect of enrollment—only the effect of being given the opportunity to enroll.

To estimate the treatment effect ($\delta_j$), we use an Instrumental Variable approach:

$$\delta_j = \frac{E[Y_i | W_{ij} = 1, A_{ij} = 1] - E[Y_i | W_{ij} = 0, A_{ij} = 1]}{P(Enroll_{ij} = 1 | W_{ij} = 1, A_{ij} = 1) - P(Enroll_{ij} = 1 | W_{ij} = 0, A_{ij} = 1)}$$

where:

• $P(Enroll_{ij} = 1 | W_{ij} = 1, A_{ij} = 1)$ = probability of enrolling if winning the lottery.
• $P(Enroll_{ij} = 1 | W_{ij} = 0, A_{ij} = 1)$ = probability of enrolling if losing the lottery.

This adjustment accounts for the fact that some lottery winners do not enroll, and thus the observed effect ($\theta_j$) underestimates the true treatment effect ($\delta_j$).

This Instrumental Variable approach corrects for selection bias by leveraging randomized lottery assignment.

---

Numerical Example

Assume 10 students win the lottery and 10 students lose.

For Winners:

Type	Count	Selection Effect	Treatment Effect	Total Effect
Always Takers	1	+0.2	+1	+1.2
Compliers	2	0	+1	+1
Never Takers	7	-0.1	0	-0.1

For Losers:

Type	Count	Selection Effect	Treatment Effect	Total Effect
Always Takers	1	+0.2	+1	+1.2
Compliers	2	0	0	0
Never Takers	7	-0.1	0	-0.1

---

Computing Intent-to-Treat Effect

We compute the expected outcome for those who won and lost the lottery.

$$\begin{aligned} E[Y_i | W_{ij} = 1, A_{ij} = 1] &= \frac{1(1.2) + 2(1) + 7(-0.1)}{10} = 0.25 \\ E[Y_i | W_{ij} = 0, A_{ij} = 1] &= \frac{1(1.2) + 2(0) + 7(-0.1)}{10} = 0.05 \end{aligned}$$

Thus, the Intent-to-Treat Effect is:

$$\text{Intent-to-Treat Effect} = 0.25 - 0.05 = 0.2$$

---

Now, we calculate the probability of enrollment for lottery winners and losers:

$$\begin{aligned} P(Enroll_{ij} = 1 | W_{ij} = 1, A_{ij} = 1) &= \frac{1+2}{10} = 0.3 \\ P(Enroll_{ij} = 1 | W_{ij} = 0, A_{ij} = 1) &= \frac{1}{10} = 0.1 \end{aligned}$$

Using the formula for treatment effect ($\delta$):

$$\text{Treatment Effect} = \frac{0.2}{0.3 - 0.1} = 1$$

This confirms that the true treatment effect is 1 unit.

---

To account for additional factors ($X_i$), we extend the model as follows:

$$Y_{ia} = \delta W_{ia} + \lambda L_{ia} + X_i \theta + u_{ia}$$

where:

• $\delta$ = Intent-to-Treat effect
• $\lambda$ = True treatment effect
• $W$ = Whether a student wins the lottery
• $L$ = Whether a student enrolls in the school
• $X_i \theta$ = Control variables (i.e., reweighting of lottery), but would not affect treatment effect $E(\delta)$

Since choosing to apply to a lottery is not random, we must consider the following:

$$E(\lambda) \neq E(\lambda_1)$$

This demonstrates why lottery-based assignment is a useful but imperfect tool for causal inference—winning the lottery is random, but who applies is not.

---

22.5.2 Re-Randomization

In standard randomization, baseline covariates are only balanced on average, meaning imbalance can still occur due to random chance. Re-randomization eliminates bad randomizations by checking balance before the experiment begins (Morgan and Rubin 2012).

Key Motivations for Re-Randomization

• Randomization does not guarantee balance:
  • Example: For 10 covariates, the probability of at least one imbalance at $\alpha = 0.05$ is:
    $$1 - (1 - 0.05)^{10} = 0.40 = 40\%$$
    
  • This means a high chance of some imbalance across treatment groups.
    
• Re-randomization increases precision: If covariates correlate with the outcome, improving covariate balance improves treatment effect estimation
• Accounting for re-randomization in inference: Since re-randomization filters out bad assignments, it is equivalent to increasing the sample size and must be considered when computing standard errors.
• Alternative balancing techniques:
  • Stratified randomization (Johansson and Schultzberg 2022).
  • Matched randomization (Greevy et al. 2004; Kapelner and Krieger 2014).
  • Minimization (Pocock and Simon 1975).

Figure from USC Schaeffer Center

---

Example: Balancing Experimental Groups

An online retailer is testing two website designs (A and B) but wants to ensure that key customer demographics (e.g., age) are balanced across treatment groups.

We define a balance criterion to check if the mean age difference between groups is acceptable before proceeding.

set.seed(123)

# Define balance criterion: Ensure mean age difference < 1 year
balance_criterion <- function(data) {
  abs(mean(data$age[data$group == "A"]) - mean(data$age[data$group == "B"])) < 1
}

# Generate randomized groups, repeat until balance criterion is met
repeat {
  data <- data.frame(
    age = rnorm(100, mean = 35, sd = 10),
    group = sample(c("A", "B"), 100, replace = TRUE)
  )
  if (balance_criterion(data)) break
}

# Check final group means
tapply(data$age, data$group, mean)
#>        A        B 
#> 35.91079 35.25483

22.5.2.1 Rerandomization Criterion

Re-randomization is based on a function of the covariate matrix ($\mathbf{X}$) and treatment assignments ($\mathbf{W}$).

$$W_i = \begin{cases} 1, & \text{if treated} \\ 0, & \text{if control} \end{cases}$$

A common approach is to use Mahalanobis Distance to measure covariate balance between treatment and control groups:

$$\begin{aligned} M &= (\bar{\mathbf{X}}_T - \bar{\mathbf{X}}_C)' \text{cov}(\bar{\mathbf{X}}_T - \bar{\mathbf{X}}_C)^{-1} (\bar{\mathbf{X}}_T - \bar{\mathbf{X}}_C) \\ &= (\frac{1}{n_T} + \frac{1}{n_C})^{-1} (\bar{\mathbf{X}}_T - \bar{\mathbf{X}}_C)' \text{cov}(\mathbf{X})^{-1} (\bar{\mathbf{X}}_T - \bar{\mathbf{X}}_C) \end{aligned}$$

where:

• $\bar{\mathbf{X}}_T$ and $\bar{\mathbf{X}}_C$ are the mean covariate values for treatment and control groups.
• $\text{cov}(\mathbf{X})$ is the covariance matrix of the covariates.
• $n_T$ and $n_C$ are the sample sizes for treatment and control groups.

If the sample size is large and randomization is pure, then $M$ follows a chi-squared distribution:

$$M \sim \chi^2_k$$

where $k$ is the number of covariates to be balanced.

---

Choosing the Rerandomization Threshold ($M > a$)

Define $p_a$ as the probability of accepting a randomization:

• Smaller $p_a$ → Stronger balance, but longer computation time.
• Larger $p_a$ → Faster randomization, but weaker balance.

A rule of thumb is to re-randomize whenever:

$$M > a$$

where $a$ is chosen based on acceptable balance thresholds.

---

We apply Mahalanobis Distance as a balance criterion to ensure that treatment and control groups are well-matched before proceeding with the experiment.

set.seed(123)
library(MASS)

# Generate a dataset with two covariates
n <- 100
X <- mvrnorm(n, mu = c(0, 0), Sigma = matrix(c(1, 0.5, 0.5, 1), 2, 2))
colnames(X) <- c("Covariate1", "Covariate2")

# Balance function using Mahalanobis Distance
balance_criterion <- function(X, group) {
  X_treat <- X[group == 1, ]
  X_control <- X[group == 0, ]
  
  mean_diff <- colMeans(X_treat) - colMeans(X_control)
  cov_inv <- solve(cov(X))
  M <- t(mean_diff) %*% cov_inv %*% mean_diff
  
  
  # Acceptable threshold 
  # (chi-squared critical value for k = 2, alpha = 0.05)
  return(M < 3.84)  
}

# Repeat randomization until balance is met
repeat {
  group <- sample(c(0, 1), n, replace = TRUE)
  if (balance_criterion(X, group)) break
}

# Display final balance check
table(group)
#> group
#>  0  1 
#> 50 50
colMeans(X[group == 1, ])  # Treatment group means
#> Covariate1 Covariate2 
#>  0.2469635  0.1918521
colMeans(X[group == 0, ])  # Control group means
#>  Covariate1  Covariate2 
#>  0.01717088 -0.14281124

22.5.3 Two-Stage Randomized Experiments

A two-stage randomized experiment involves sequential interventions, where treatment assignments depend on earlier responses. This design is widely used in:

• Adaptive Learning: Adjusting educational content based on student progress.
• Personalized Advertising: Targeting follow-up ads based on engagement.
• Medical Trials: Adapting treatments based on patient response.

By introducing a second randomization stage, researchers can evaluate:

1. The effect of initial treatments.
2. The effect of follow-up treatments.
3. Potential interactions between the two stages.

---

22.5.3.1 Example: Personalized Advertising Experiment

A company tests two initial ad campaigns (Ad A vs. Ad B). After observing customer engagement, they apply a second-stage intervention (e.g., Discount vs. No Discount).

The two-stage experiment can be modeled as:

$$Y_{ijk} = \mu + \alpha_i + \beta_{j(i)} + \epsilon_{ijk}$$

where:

• $\mu$ = Overall mean outcome (e.g., conversion rate).
• $\alpha_i$ = Effect of first-stage intervention ($i$ = Ad A or Ad B).
• $\beta_{j(i)}$ = Effect of second-stage intervention, nested within first-stage groups.
• $\epsilon_{ijk}$ = Random error term.

The nested structure ensures that the second-stage treatment ($\beta_{j(i)}$) is assigned within each first-stage treatment group.

---

set.seed(123)

# Generate first-stage randomization (Initial Ad)
data <- data.frame(
  stage1 = sample(c("Ad A", "Ad B"), 100, replace = TRUE),
  stage2 = rep(NA, 100)  # Placeholder for second-stage randomization
)

# Second-stage assignment based on first-stage response
data$stage2[data$stage1 == "Ad A"] <-
    sample(c("Discount", "No Discount"),
           sum(data$stage1 == "Ad A"),
           replace = TRUE)
data$stage2[data$stage1 == "Ad B"] <-
    sample(c("Discount", "No Discount"),
           sum(data$stage1 == "Ad B"),
           replace = TRUE)

# Display final randomization
table(data$stage1, data$stage2)
#>       
#>        Discount No Discount
#>   Ad A       24          33
#>   Ad B       22          21

This structure ensures:

1. Fair assignment of initial ads.

2. Adaptive targeting in the second stage based on user engagement.

22.5.4 Two-Stage Randomized Experiments with Interference and Noncompliance

In real-world experiments, interference and noncompliance complicate analysis (Imai, Jiang, and Malani 2021):

• Interference: When treatment effects “spill over” from one group to another (e.g., social influence in marketing).

• Noncompliance: When participants do not adhere to their assigned treatment (e.g., a customer ignoring an ad).

To handle noncompliance, we define:

• $Z_{ik}$ = Assigned treatment (e.g., Ad A or Ad B).

• $D_{ik}$ = Actual treatment received (e.g., whether the user actually saw the ad).

• $Y_{ik}$ = Outcome (e.g., purchase).

A two-stage Instrumental Variable model adjusts for noncompliance:

$$\begin{aligned} D_{ik} &= \gamma_0 + \gamma_1 Z_{ik} + v_{ik} \\ Y_{ik} &= \beta_0 + \beta_1 D_{ik} + \epsilon_{ik} \end{aligned}$$ where:

• $\gamma_1$ measures the effect of assignment on actual treatment received.

• $\beta_1$ estimates the treatment effect, adjusting for noncompliance.

If individuals influence each other’s outcomes, traditional randomization is biased. Solutions include:

1. Cluster Randomization: Assigning treatments at the group level (e.g., entire social circles receive the same ad).

2. Partial Interference Models: Assume interference only occurs within predefined groups.

set.seed(123)
library(ivreg)  # Load Instrumental Variable Regression Package

# Generate data for first-stage treatment assignment
n <- 500
data <- data.frame(
  Z = sample(c(0, 1), n, replace = TRUE),  # Initial assignment (randomized)
  D = NA,  # Actual treatment received (affected by compliance)
  Y = NA   # Outcome variable (e.g., purchase)
)

# Introduce noncompliance: 80% compliance rate
data$D <- ifelse(runif(n) < 0.8, data$Z, 1 - data$Z)

# Generate outcome variable (Y) with true treatment effect
# True effect of D on Y is 3
data$Y <- 5 + 3 * data$D + rnorm(n, mean = 0, sd = 2)  

# Estimate Two-Stage Least Squares (2SLS)
iv_model <- ivreg(Y ~ D | Z, data = data)
summary(iv_model)
#> 
#> Call:
#> ivreg(formula = Y ~ D | Z, data = data)
#> 
#> Residuals:
#>    Min     1Q Median     3Q    Max 
#> -5.497 -1.344  0.018  1.303  5.493 
#> 
#> Coefficients:
#>             Estimate Std. Error t value Pr(>|t|)    
#> (Intercept)   5.1285     0.1861  27.557   <2e-16 ***
#> D             2.7487     0.3072   8.949   <2e-16 ***
#> 
#> Diagnostic tests:
#>                  df1 df2 statistic p-value    
#> Weak instruments   1 498    263.54  <2e-16 ***
#> Wu-Hausman         1 497      0.19   0.663    
#> Sargan             0  NA        NA      NA    
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#> 
#> Residual standard error: 2.017 on 498 degrees of freedom
#> Multiple R-Squared: 0.2997,  Adjusted R-squared: 0.2983 
#> Wald test: 80.08 on 1 and 498 DF,  p-value: < 2.2e-16

• The first-stage regression estimates how strongly $Z$ affects $D$ (compliance).

• The second-stage regression estimates the true causal effect of $D$ on $Y$.

If interference is present, the standard IV method may be biased. Researchers should explore network-based randomization or spatial models.

References

Cullen, Julie Berry, Brian A Jacob, and Steven D Levitt. 2005. “The Impact of School Choice on Student Outcomes: An Analysis of the Chicago Public Schools.” Journal of Public Economics 89 (5-6): 729–60.

Greevy, Robert, Bo Lu, Jeffrey H Silber, and Paul Rosenbaum. 2004. “Optimal Multivariate Matching Before Randomization.” Biostatistics 5 (2): 263–75.

Imai, Kosuke, Zhichao Jiang, and Anup Malani. 2021. “Causal Inference with Interference and Noncompliance in Two-Stage Randomized Experiments.” Journal of the American Statistical Association 116 (534): 632–44.

Johansson, Per, and Mårten Schultzberg. 2022. “Rerandomization: A Complement or Substitute for Stratification in Randomized Experiments?” Journal of Statistical Planning and Inference 218: 43–58.

Kapelner, Adam, and Abba Krieger. 2014. “Matching on-the-Fly: Sequential Allocation with Higher Power and Efficiency.” Biometrics 70 (2): 378–88.

Morgan, Kari Lock, and Donald B Rubin. 2012. “Rerandomization to Improve Covariate Balance in Experiments.”

Pocock, Stuart J, and Richard Simon. 1975. “Sequential Treatment Assignment with Balancing for Prognostic Factors in the Controlled Clinical Trial.” Biometrics, 103–15.