27.8 Specification Checks
To validate the credibility of an RD design, researchers perform several specification checks:
27.8.1 Balance Checks
Also known as checking for discontinuities in average covariates, this test examines whether covariates that should not be affected by treatment exhibit a discontinuity at the cutoff.
- Null Hypothesis (H0): The average effect of covariates on pseudo-outcomes (i.e., those that should not be influenced by treatment) is zero.
- If rejected, this raises serious doubts about the RD design, necessitating a strong justification.
27.8.2 Sorting, Bunching, and Manipulation
This test, also known as checking for discontinuities in the distribution of the forcing variable, detects whether subjects manipulate the running variable to sort into or out of treatment.
If individuals can manipulate the running variable—especially when the cutoff is known in advance—this can lead to bunching behavior (i.e., clustering just above or below the cutoff).
- If treatment is desirable, individuals will try to sort into treatment, leading to a gap just below the cutoff.
- If treatment is undesirable, individuals will try to avoid it, leading to a gap just above the cutoff.
Under RD, we assume that there is no manipulation in the running variable. However, bunching behavior, where firms or individuals strategically manipulate their position, violates this assumption.
To address this issue, the bunching approach estimates the counterfactual distribution—what the density of individuals would have been in the absence of manipulation.
The fraction of individuals who engaged in manipulation is then calculated by comparing the observed distribution to this counterfactual distribution.
In a standard RD framework, this step is unnecessary because it assumes that the observed distribution and the counterfactual (manipulation-free) distribution are the same, implying no manipulation
27.8.2.1 McCrary Sorting Test
A widely used formal test is the McCrary density test (McCrary 2008), later refined by (Cattaneo, Idrobo, and Titiunik 2019).
- Null Hypothesis (H0): The density of the running variable is continuous at the cutoff.
- Alternative Hypothesis (Ha): A discontinuity (jump) in the density function at the cutoff, suggesting manipulation.
- Interpretation:
- A significant discontinuity suggests manipulation, violating RD assumptions.
- A failure to reject H0 does not necessarily confirm validity, as some forms of manipulation may remain undetected.
- If there is two-sided manipulation, this test will fail to detect it.
27.8.2.2 Guidelines for Assessing Manipulation
- J. L. Zhang and Rubin (2003), Lee (2009), and Aronow, Baron, and Pinson (2019) provide criteria for evaluating manipulation risks.
- Knowing your research design inside out is crucial to anticipating possible manipulation attempts.
- Manipulation is often one-sided, meaning subjects shift only in one direction relative to the cutoff. In rare cases, two-sided manipulation may occur but often cancels out.
- We could also observe partial manipulation in reality (e.g., when subjects can only imperfectly manipulate). However, since we typically treat it like a fuzzy RD, we would not encounter identification problems. In contrast, complete manipulation would lead to serious identification issues.
Bunching Methodology
- Bunching occurs when individuals self-select into specific values of the running variable (e.g., policy thresholds). See Kleven (2016) for a review.
- The method helps estimate the counterfactual distribution—what the density would have been without manipulation.
- The fraction of individuals who manipulated can be estimated by comparing observed densities to the counterfactual.
If the running variable and outcome are simultaneously determined, a modified RD estimator can be used for consistent estimation (Bajari et al. 2011):
- One-sided manipulation: Individuals shift only in one direction relative to the cutoff (similar to the monotonicity assumption in instrumental variables).
- Bounded manipulation (regularity assumption): The density of individuals far from the threshold remains unaffected (Blomquist et al. 2021; Bertanha, McCallum, and Seegert 2021).
27.8.2.3 Steps for Bunching Analysis
- Identify the window where bunching occurs (based on Bosch, Dekker, and Strohmaier (2020)). Perform robustness checks by varying the manipulation window.
- Estimate the manipulation-free counterfactual distribution.
- Standard errors for inference can be calculated (following Chetty, Hendren, and Katz (2016)), where bootstrap resampling of residuals is used in estimating the counts of individuals within bins. However, this step may be unnecessary for large datasets.
If the bunching test fails to detect manipulation, we proceed to the Placebo Test.
McCrary Density Test (Discontinuity in Forcing Variable)
library(rdd)
set.seed(1)
# Simulated data without discontinuity
x <- runif(100, -1, 1)
DCdensity(x, 0) # No discontinuity
#> [1] 0.06355195
# Simulated data with discontinuity
x <- runif(1000, -1, 1)
x <- x + 2 * (runif(1000, -1, 1) > 0 & x < 0)
DCdensity(x, 0) # Discontinuity detected
#> [1] 0.001936782
Cattaneo Density Test (Improved Version)
library(rddensity)
# Simulated continuous density
set.seed(1)
x <- rnorm(100, mean = -0.5)
rdd <- rddensity(X = x, vce = "jackknife")
summary(rdd)
#>
#> Manipulation testing using local polynomial density estimation.
#>
#> Number of obs = 100
#> Model = unrestricted
#> Kernel = triangular
#> BW method = estimated
#> VCE method = jackknife
#>
#> c = 0 Left of c Right of c
#> Number of obs 66 34
#> Eff. Number of obs 38 25
#> Order est. (p) 2 2
#> Order bias (q) 3 3
#> BW est. (h) 0.922 0.713
#>
#> Method T P > |T|
#> Robust 0.6715 0.5019
#>
#>
#> P-values of binomial tests (H0: p=0.5).
#>
#> Window Length <c >=c P>|T|
#> 0.563 + 0.563 26 20 0.4614
#> 0.603 + 0.580 27 20 0.3817
#> 0.643 + 0.596 30 20 0.2026
#> 0.683 + 0.613 32 21 0.1690
#> 0.722 + 0.630 32 22 0.2203
#> 0.762 + 0.646 33 22 0.1770
#> 0.802 + 0.663 33 23 0.2288
#> 0.842 + 0.680 35 24 0.1925
#> 0.882 + 0.696 36 24 0.1550
#> 0.922 + 0.713 38 25 0.1299
# Plot requires customization (refer to package documentation)
27.8.3 Placebo Tests
Placebo tests, also known as falsification checks, assess whether discontinuities appear at points other than the treatment cutoff. This helps verify that observed effects are causal rather than artifacts of the method or data.
- There should be no jumps in the outcome at values other than the cutoff (Xi<c or Xi≥c).
- The test involves shifting the cutoff along the running variable while using the same bandwidth to check for discontinuities in the conditional mean of the outcome.
- This approach is similar to balance checks in experimental design, ensuring no pre-existing differences. Remember, we can only test on observables, not unobservables.
Under a valid RD design, matching methods are unnecessary. Just as with randomized experiments, balance should naturally occur across the threshold. If adjustments are required, it suggests the RD assumptions may be invalid.
27.8.3.1 Applications of Placebo Tests
- Testing No Discontinuity in Predetermined Covariates: Covariates that should not be affected by treatment should not exhibit a jump at the cutoff.
- Testing Other Discontinuities: Checking for discontinuities at other arbitrary points along the running variable.
- Using Placebo Outcomes: If an outcome variable that should not be affected by treatment shows a significant discontinuity, this raises concerns about RD validity.
- Assessing Sensitivity to Covariates: RD estimates should not be highly sensitive to the inclusion or exclusion of covariates.
27.8.3.2 Mathematical Specification
The balance of observable characteristics on both sides of the threshold can be tested using:
Zi=α0+α1f(Xi)+I(Xi≥c)α2+[f(Xi)×I(Xi≥c)]α3+ui
where:
Xi = running variable
Zi = predetermined characteristics (e.g., age, education, etc.)
α2 should be zero if Zi is unaffected by treatment.
If multiple covariates Zi are tested simultaneously, simulating their joint distribution avoids false positives due to multiple comparisons. This step is unnecessary if covariates are independent, but such independence is unlikely in practice.
27.8.4 Sensitivity to Bandwidth Choice
The choice of bandwidth is crucial in RD estimation. Different bandwidth selection methods exist:
- Ad-hoc or Substantively Driven: Based on theoretical or empirical reasoning.
- Data-Driven Selection (Cross-Validation): Optimizes bandwidth to minimize prediction error.
- Conservative Approach: Uses robust optimal bandwidth selection methods (e.g., (Calonico, Cattaneo, and Farrell 2020)).
The objective is to minimize mean squared error (MSE) between estimated and actual treatment effects.
27.8.5 Assessing Sensitivity
- Results should be consistent across reasonable bandwidth choices.
- The optimal bandwidth for estimating treatment effects may differ from the optimal bandwidth for testing covariates but should be fairly close.
# Load required package
library(rdd)
# Simulate some data
set.seed(123)
n <- 100 # Sample size
running_var <- runif(n, -1, 1) # Running variable centered around 0
treatment <- ifelse(running_var >= 0, 1, 0) # Treatment assigned at cutpoint 0
outcome_var <- 2 * running_var + treatment * 1.5 + rnorm(n) # Outcome variable
# Compute the optimal Imbens-Kalyanaraman (IK) bandwidth
bandwidth <-
IKbandwidth(running_var,
outcome_var,
cutpoint = 0,
kernel = "triangular")
# Print the bandwidth
print(bandwidth)
#> [1] 0.4374142
27.8.6 Manipulation Robust Regression Discontinuity Bounds
McCrary (2008) linked density jumps at cutoffs in RD studies to potential manipulation.
If no jump is detected, researchers proceed with RD analysis; if detected, they halt using the cutoff for inference.
Some studies use the “doughnut-hole” method, excluding near-cutoff observations and extrapolating, which contradicts RD principles.
False negative could be due to a small sample size and can lead to biased estimates, as units near the cutoff may still differ in unobserved ways.
Even correct rejections of no manipulation may overlook that the data can still be informative despite modest manipulation.
Gerard, Rokkanen, and Rothe (2020) introduces a systematic approach to handle potentially manipulated variables in RD designs, addressing both concerns.
The model introduces two types of unobservable units in RD designs:
always-assigned units, which are always on one side of the cutoff,
potentially-assigned units, which fit traditional RD assumptions.
- The standard RD model is a subset of this broader model, which assumes no always-assigned units.
Identifying assumption: manipulation occurs through one-sided selection.
The approach does not make a binary decision on manipulation in RD designs but assesses its extent and worst-case impact.
Two steps are used:
- Determining the proportion of always-assigned units using the discontinuity at the cutoff
- Bounding treatment effects based on the most extreme feasible outcomes for these units.
For sharp RD designs, bounds are established by trimming extreme outcomes near the cutoff; for fuzzy designs, the process involves more complex adjustments due to additional model constraints.
Extensions of the study use covariate information and economic behavior assumptions to refine these bounds and identify covariate distributions among unit types at the cutoff.
Setup
Independent data points (Xi,Yi,Di), where Xi is the running variable, Yi is the outcome, and Di indicates treatment status (1 if treated, 0 otherwise). Treatment is assigned based on Xi≥c.
The design is sharp if Di=I(Xi≥c) and fuzzy otherwise.
The population is divided into:
Potentially-assigned units (Mi=0): Follow the standard RD framework, with potential outcomes Yi(d) and potential treatment states Di(x).
Always-assigned units (Mi=1): These units do not require potential outcomes or states, and always have Xi values beyond the cutoff.
Assumptions
- Local Independence and Continuity:
- P(D=1|X=c+,M=0)>P(D=1|X=c−,M=0)
- No defiers: P(D+≥D−|X=c,M=0)=1
- Continuity in potential outcomes and states at c.
- FX|M=0(x) is differentiable at c, with a positive derivative.
- Smoothness of the Running Variable among Potentially-Assigned Units:
- The derivative of FX|M=0(x) is continuous at c.
- Restrictions on Always-Assigned Units:
- P(X≥c|M=1)=1 and FX|M=1(x) is right-differentiable (or left-differentiable) at c.
- This (local) one-sided manipulation assumption allows identification of the proportion of always-assigned units among all units close to the cutoff.
When always-assigned unit exist, the RD design is fuzzy because we have
- Treated and untreated units among the potentially-assigned (below and above the cutoff)
- Always-assigned units (above the cutoff).
Causal Effects of Interest
causal effects among potentially-assigned units:
Γ=E[Y(1)−Y(0)|X=c,D+>D−,M=0]
This parameter represents the local average treatment effect (LATE) for the subgroup of “compliers”—units that receive treatment if and only if their running variable Xi exceeds a certain cutoff.
The parameter Γ captures the causal effect of changes in the cutoff level on treatment status among potentially-assigned compliers.
RD designs with a manipulated running variable | “Doughnut-Hole” RD Designs: |
---|---|
|
|
Identification of τ in RD Designs
Identification challenges arise due to the inability to distinguish always-assigned from potentially-assigned units, thus Γ is not point identified. We establish sharp bounds on Γ
These bounds are supported by the stochastic dominance of the potential outcome CDFs over observed distributions.
Unit Types and Notation:
- C0: Potentially-assigned compliers.
- A0: Potentially-assigned always-takers.
- N0: Potentially-assigned never-takers.
- T1: Always-assigned treated units.
- U1: Always-assigned untreated units.
The measure τ , representing the proportion of always-assigned units near the cutoff, is point identified by the discontinuity in the observed running variable density fX at the cutoff
Sharp RD:
Units to the left of the cutoff are potentially assigned units. The distribution of their observed outcomes (Y) are the outcomes Y(0) of potentially-assigned compliers (C0) at the cutoff.
To determine the bounds on the treatment effect (Γ), we need to assess the distribution of treated outcomes (Y(1)) for the same potentially-assigned compliers at the cutoff.
Information regarding the treated outcomes (Y(1)) comes exclusively from the subpopulation of treated units, which includes both potentially-assigned compliers (C0) and those always assigned units (T1).
With τ point identified, we can estimate sharp bounds on Γ.
Fuzzy RD:
Subpopulation | Types of units |
---|---|
X=c+,D=1 | C0,A0,T1 |
X=c−,D=1 | A0 |
X=c+,D=0 | N0,U1 |
X=c−,D=0 | C0,N0 |
Unit Types and Combinations: There are five distinct unit types and four combinations of treatment assignments and decisions relevant to the analysis. These distinctions are important because they affect how potential outcomes are analyzed and bounded.
Outcome Distributions: The analysis involves estimating the distribution of potential outcomes (both treated and untreated) among potentially-assigned compliers at the cutoff.
Three-Step Process:
Potential Outcomes Under Treatment: Bounds on the distribution of treated outcomes are determined using data from treated units.
Potential Outcomes Under Non-Treatment: Bounds on the distribution of untreated outcomes are derived using data from untreated units.
Bounds on Parameters of Interest: Using the bounds from the first two steps, sharp upper and lower bounds on the local average treatment effect are derived.
Extreme Value Consideration: The bounds for treatment effects are based on “extreme” scenarios under worst-case assumptions about the distribution of potential outcomes, making them sharp but empirically relevant within the data constraints.
Extensions:
Quantile Treatment Effects: alternative to average effects by focusing on different quantiles of the outcome distribution, which are less affected by extreme values.
Applicability to Discrete Outcomes
Behavioral Assumptions Impact: Assuming a high likelihood of treatment among always-assigned units can narrow the bounds of treatment effects by refining the analysis of potential outcomes.
Utilization of Covariates: Incorporating covariates measured prior to treatment can refine the bounds on treatment effects and help target policies by identifying covariate distributions among different unit types.
Notes:
Quantile Treatment Effects (QTEs): QTE bounds are less sensitive to the tails of the outcome distribution, making them tighter than ATE bounds.
Inference on ATEs is sensitive to the extent of manipulation, with confidence intervals widening significantly with small degrees of assumed manipulation.
Inference on QTEs is less affected by manipulation, remaining meaningful even with larger degrees of manipulation.
Alternative Inference Strategy when manipulation is believed to be unlikely. Try different hypothetical values of τ
library(formattable)
library(data.table)
library(rdbounds)
set.seed(123)
df <- rdbounds_sampledata(1000, covs = FALSE)
#> [1] "True tau: 0.117999815082062"
#> [1] "True treatment effect on potentially-assigned: 2"
#> [1] "True treatment effect on right side of cutoff: 2.35399944524618"
head(df)
#> x y treatment
#> 1 -1.2532616 2.684827 0
#> 2 -0.5146925 5.845219 0
#> 3 3.4853777 6.166070 0
#> 4 0.1576616 3.227139 0
#> 5 0.2890962 7.031685 1
#> 6 3.8350019 10.238570 1
rdbounds_est <-
rdbounds(
y = df$y,
x = df$x,
# covs = as.factor(df$cov),
treatment = df$treatment,
c = 0,
discrete_x = FALSE,
discrete_y = FALSE,
bwsx = c(.2, .5),
bwy = 1,
# for median effect use
# type = "qte",
# percentiles = .5,
kernel = "epanechnikov",
orders = 1,
evaluation_ys = seq(from = 0, to = 15, by = 1),
refinement_A = TRUE,
refinement_B = TRUE,
right_effects = TRUE,
yextremes = c(0, 15),
num_bootstraps = 5
)
#> [1] "The proportion of always-assigned units just to the right of the cutoff is estimated to be 0.38047"
#> [1] "2025-02-08 18:14:33.753569 Estimating CDFs for point estimates"
#> [1] "2025-02-08 18:14:34.145708 .....Estimating CDFs for units just to the right of the cutoff"
#> [1] "2025-02-08 18:14:35.938183 Estimating CDFs with nudged tau (tau_star)"
#> [1] "2025-02-08 18:14:35.993495 .....Estimating CDFs for units just to the right of the cutoff"
#> [1] "2025-02-08 18:14:38.747617 Beginning parallelized output by bootstrap.."
#> [1] "2025-02-08 18:14:43.191747 Computing Confidence Intervals"
#> [1] "2025-02-08 18:14:54.779613 Time taken:0.35 minutes"
rdbounds_summary(rdbounds_est, title_prefix = "Sample Data Results")
#> [1] "Time taken: 0.35 minutes"
#> [1] "Sample size: 1000"
#> [1] "Local Average Treatment Effect:"
#> $tau_hat
#> [1] 0.3804665
#>
#> $tau_hat_CI
#> [1] 0.4180576 1.4333618
#>
#> $takeup_increase
#> [1] 0.6106973
#>
#> $takeup_increase_CI
#> [1] 0.4332132 0.7881814
#>
#> $TE_SRD_naive
#> [1] 1.589962
#>
#> $TE_SRD_naive_CI
#> [1] 1.083977 2.095948
#>
#> $TE_SRD_bounds
#> [1] 0.7165801 2.3454868
#>
#> $TE_SRD_CI
#> [1] -1.200992 7.733214
#>
#> $TE_SRD_covs_bounds
#> [1] NA NA
#>
#> $TE_SRD_covs_CI
#> [1] NA NA
#>
#> $TE_FRD_naive
#> [1] 2.583271
#>
#> $TE_FRD_naive_CI
#> [1] 1.506573 3.659969
#>
#> $TE_FRD_bounds
#> [1] 1.273896 3.684860
#>
#> $TE_FRD_CI
#> [1] -1.266341 10.306116
#>
#> $TE_FRD_bounds_refinementA
#> [1] 1.273896 3.684860
#>
#> $TE_FRD_refinementA_CI
#> [1] -1.266341 10.306116
#>
#> $TE_FRD_bounds_refinementB
#> [1] 1.420594 3.677963
#>
#> $TE_FRD_refinementB_CI
#> [1] NA NA
#>
#> $TE_FRD_covs_bounds
#> [1] NA NA
#>
#> $TE_FRD_covs_CI
#> [1] NA NA
#>
#> $TE_SRD_CIs_manipulation
#> [1] NA NA
#>
#> $TE_FRD_CIs_manipulation
#> [1] NA NA
#>
#> $TE_SRD_right_bounds
#> [1] -2.056178 3.650820
#>
#> $TE_SRD_right_CI
#> [1] -11.044937 9.282217
#>
#> $TE_FRD_right_bounds
#> [1] -2.900703 5.272370
#>
#> $TE_FRD_right_CI
#> [1] -14.75421 15.58640
rdbounds_est_tau <-
rdbounds(
y = df$y,
x = df$x,
# covs = as.factor(df$cov),
treatment = df$treatment,
c = 0,
discrete_x = FALSE,
discrete_y = FALSE,
bwsx = c(.2, .5),
bwy = 1,
kernel = "epanechnikov",
orders = 1,
evaluation_ys = seq(from = 0, to = 15, by = 1),
refinement_A = TRUE,
refinement_B = TRUE,
right_effects = TRUE,
potential_taus = c(.025, .05, .1, .2),
yextremes = c(0, 15),
num_bootstraps = 5
)
#> [1] "The proportion of always-assigned units just to the right of the cutoff is estimated to be 0.38047"
#> [1] "2025-02-08 18:14:56.245033 Estimating CDFs for point estimates"
#> [1] "2025-02-08 18:14:56.449024 .....Estimating CDFs for units just to the right of the cutoff"
#> [1] "2025-02-08 18:14:58.23866 Estimating CDFs with nudged tau (tau_star)"
#> [1] "2025-02-08 18:14:58.276159 .....Estimating CDFs for units just to the right of the cutoff"
#> [1] "2025-02-08 18:15:01.104187 Beginning parallelized output by bootstrap.."
#> [1] "2025-02-08 18:15:05.392548 Estimating CDFs with fixed tau value of: 0.025"
#> [1] "2025-02-08 18:15:05.468534 Estimating CDFs with fixed tau value of: 0.05"
#> [1] "2025-02-08 18:15:05.527238 Estimating CDFs with fixed tau value of: 0.1"
#> [1] "2025-02-08 18:15:05.578757 Estimating CDFs with fixed tau value of: 0.2"
#> [1] "2025-02-08 18:15:06.69184 Beginning parallelized output by bootstrap x fixed tau.."
#> [1] "2025-02-08 18:15:09.681396 Computing Confidence Intervals"
#> [1] "2025-02-08 18:15:22.407704 Time taken:0.44 minutes"