24.2 Specification Checks
24.2.1 Balance Checks
Also known as checking for Discontinuities in Average Covariates
Null Hypothesis: The average effect of covariates on pseudo outcomes (i.e., those qualitatively cannot be affected by the treatment) is 0.
If this hypothesis is rejected, you better have a good reason to why because it can cast serious doubt on your RD design.
24.2.2 Sorting/Bunching/Manipulation
Also known as checking for A Discontinuity in the Distribution of the Forcing Variable
Also known as clustering or density test
Formal test is McCrary sorting test (McCrary 2008) or (Cattaneo, Idrobo, and Titiunik 2019)
Since human subjects can manipulate the running variable to be just above or below the cutoff (assuming that the running variable is manipulable), especially when the cutoff point is known in advance for all subjects, this can result in a discontinuity in the distribution of the running variable at the cutoff (i.e., we will see “bunching” behavior right before or after the cutoff)>
People would like to sort into treatment if it’s desirable. The density of the running variable would be 0 just below the threshold
People would like to be out of treatment if it’s undesirable
(McCrary 2008) proposes a density test (i.e., a formal test for manipulation of the assignment variable).
\(H_0\): The continuity of the density of the running variable (i.e., the covariate that underlies the assignment at the discontinuity point)
\(H_a\): A jump in the density function at that point
Even though it’s not a requirement that the density of the running must be continuous at the cutoff, but a discontinuity can suggest manipulations.
(J. L. Zhang and Rubin 2003; Lee 2009; Aronow, Baron, and Pinson 2019) offers a guide to know when you should warrant the manipulation
Usually it’s better to know your research design inside out so that you can suspect any manipulation attempts.
 We would suspect the direction of the manipulation. And typically, it’s oneway manipulation. In cases where we might have both ways, theoretically they would cancel each other out.
We could also observe partial manipulation in reality (e.g., when subjects can only imperfectly manipulate). But typically, as we treat it like fuzzy RD, we would not have identification problems. But complete manipulation would lead to serious identification issues.
Remember: even in cases where we fail to reject the null hypothesis for the density test, we could not rule out completely that identification problem exists (just like any other hypotheses)
Bunching happens when people selfselect to a specific value in the range of a variable (e.g., key policy thresholds).
Review paper (Kleven 2016)
This test can only detect manipulation that changes the distribution of the running variable. If you can choose the cutoff point or you have 2sided manipulation, this test will fail to detect it.
Histogram in bunching is similar to a density curve (we want narrower bins, wider bins bias elasticity estimates)
We can also use bunching method to study individuals’ or firm’s responsiveness to changes in policy.
Under RD, we assume that we don’t have any manipulation in the running variable. However, bunching behavior is a manipulation by firms or individuals. Thus, violating this assumption.
Bunching can fix this problem by estimating what densities of individuals would have been without manipulation (i.e., manipulationfree counterfactual).
The fraction of persons who manipulated is then calculated by comparing the observed distribution to manipulationfree counterfactual distributions.
Under RD, we do not need this step because the observed and manipulationfree counterfactual distributions are assumed to be the same. RD assume there is no manipulation (i.e., assume the manipulationfree counterfactual distribution)
When running variable and outcome variable are simultaneously determined, we can use a modified RDD estimator to have consistent estimate. (Bajari et al. 2011)
Assumptions:
Manipulation is onesided: People move one way (i.e., either below the threshold to above the threshold or vice versa, but not to or away the threshold), which is similar to the monotonicity assumption under instrumental variable 31.1.3.1
Manipulation is bounded (also known as regularity assumption): so that we can use people far away from this threshold to derive at our counterfactual distribution [Blomquist et al. (2021)](Bertanha, McCallum, and Seegert 2021)
Steps:
 Identify the window in which the running variable contains bunching behavior. We can do this step empirically based on Bosch, Dekker, and Strohmaier (2020). Additionally robustness test is needed (i.e., varying the manipulation window).
 Estimate the manipulationfree counterfactual
 Calculating the standard errors for inference can follow (Chetty, Hendren, and Katz 2016) where we bootstrap resampling residuals in the estimation of the counts of individuals within bins (large data can render this step unnecessary).
If we pass the bunching test, we can move on to the Placebo Test
McCrary (2008) test
A jump in the density at the threshold (i.e., discontinuity) hold can serve as evidence for sorting around the cutoff point
library(rdd)
# you only need the runing variable and the cutoff point
# Example by the package's authors
#No discontinuity
x<runif(1000,1,1)
DCdensity(x,0)
#> [1] 0.6126802
#Discontinuity
x<runif(1000,1,1)
x<x+2*(runif(1000,1,1)>0&x<0)
DCdensity(x,0)
#> [1] 0.0008519227
Cattaneo, Idrobo, and Titiunik (2019) test
library(rddensity)
# Example by the package's authors
# Continuous Density
set.seed(1)
x < rnorm(2000, mean = 0.5)
rdd < rddensity(X = x, vce = "jackknife")
summary(rdd)
#>
#> Manipulation testing using local polynomial density estimation.
#>
#> Number of obs = 2000
#> Model = unrestricted
#> Kernel = triangular
#> BW method = estimated
#> VCE method = jackknife
#>
#> c = 0 Left of c Right of c
#> Number of obs 1376 624
#> Eff. Number of obs 354 345
#> Order est. (p) 2 2
#> Order bias (q) 3 3
#> BW est. (h) 0.514 0.609
#>
#> Method T P > T
#> Robust 0.6798 0.4966
#>
#>
#> Pvalues of binomial tests (H0: p=0.5).
#>
#> Window Length / 2 <c >=c P>T
#> 0.036 28 20 0.3123
#> 0.072 46 39 0.5154
#> 0.107 68 59 0.4779
#> 0.143 94 79 0.2871
#> 0.179 122 103 0.2301
#> 0.215 145 130 0.3986
#> 0.250 163 156 0.7370
#> 0.286 190 176 0.4969
#> 0.322 214 200 0.5229
#> 0.358 249 218 0.1650
# you have to specify your own plot (read package manual)
24.2.3 Placebo Tests
Also known as Discontinuities in Average Outcomes at Other Values
We should not see any jumps at other values (either \(X_i <c\) or \(X_i \ge c\))
 Use the same bandwidth you use for the cutoff, and move it along the running variable: testing for a jump in the conditional mean of the outcome at the median of the running variable.
Also known as falsification checks
Before and after the cutoff point, we can run the placebo test to see whether X’s are different).
The placebo test is where you expect your coefficients to be not different from 0.
This test can be used for
Testing no discontinuity in predetermined variables:
Testing other discontinuities
Placebo outcomes: we should see any changes in other outcomes that shouldn’t have changed.
Inclusion and exclusion of covariates: RDD parameter estimates should not be sensitive to the inclusion or exclusion of other covariates.
This is analogous to Experimental Design where we cannot only test whether the observables are similar in both treatment and control groups (if we reject this, then we don’t have random assignment), but we cannot test unobservables.
Balance on observable characteristics on both sides
\[ Z_i = \alpha_0 + \alpha_1 f(x_i) + [I(x_i \ge c)] \alpha_2 + [f(x_i) \times I(x_i \ge c)]\alpha_3 + u_i \]
where
\(x_i\) is the running variable
\(Z_i\) is other characteristics of people (e.g., age, etc)
Theoretically, \(Z_i\) should no be affected by treatment. Hence, \(E(\alpha_2) = 0\)
Moreover, when you have multiple \(Z_i\), you typically have to simulate joint distribution (to avoid having significant coefficient based on chance).
The only way that you don’t need to generate joint distribution is when all \(Z_i\)’s are independent (unlikely in reality).
Under RD, you shouldn’t have to do any Matching Methods. Because just like when you have random assignment, there is no need to make balanced dataset before and after the cutoff. If you have to do balancing, then your RD assumptions are probably wrong in the first place.
24.2.4 Sensitivity to Bandwidth Choice
Methods for bandwidth selection
Adhoc or substantively driven
Data driven: cross validation
Conservative approach: (Calonico, Cattaneo, and Farrell 2020)
The objective is to minimize the mean squared error between the estimated and actual treatment effects.
Then, we need to see how sensitive our results will be dependent on the choice of bandwidth.
In some cases, the best bandwidth for testing covariates may not be the best bandwidth for treating them, but it may be close.
24.2.5 Manipulation Robust Regression Discontinuity Bounds
McCrary (2008) linked density jumps at cutoffs in RD studies to potential manipulation.
If no jump is detected, researchers proceed with RD analysis; if detected, they halt using the cutoff for inference.
Some studies use the “doughnuthole” method, excluding nearcutoff observations and extrapolating, which contradicts RD principles.
False negative could be due to a small sample size and can lead to biased estimates, as units near the cutoff may still differ in unobserved ways.
Even correct rejections of no manipulation may overlook that the data can still be informative despite modest manipulation.
Gerard, Rokkanen, and Rothe (2020) introduces a systematic approach to handle potentially manipulated variables in RD designs, addressing both concerns.
The model introduces two types of unobservable units in RD designs:
alwaysassigned units, which are always on one side of the cutoff,
potentiallyassigned units, which fit traditional RD assumptions.
 The standard RD model is a subset of this broader model, which assumes no alwaysassigned units.
Identifying assumption: manipulation occurs through onesided selection.
The approach does not make a binary decision on manipulation in RD designs but assesses its extent and worstcase impact.
Two steps are used:
 Determining the proportion of alwaysassigned units using the discontinuity at the cutoff
 Bounding treatment effects based on the most extreme feasible outcomes for these units.
For sharp RD designs, bounds are established by trimming extreme outcomes near the cutoff; for fuzzy designs, the process involves more complex adjustments due to additional model constraints.
Extensions of the study use covariate information and economic behavior assumptions to refine these bounds and identify covariate distributions among unit types at the cutoff.
Setup
Independent data points \((X_i, Y_i, D_i)\), where \(X_i\) is the running variable, \(Y_i\) is the outcome, and \(D_i\) indicates treatment status (1 if treated, 0 otherwise). Treatment is assigned based on \(X_i \geq c\).
The design is sharp if \(D_i = I(X_i \geq c)\) and fuzzy otherwise.
The population is divided into:
Potentiallyassigned units (\(M_i = 0\)): Follow the standard RD framework, with potential outcomes \(Y_i(d)\) and potential treatment states \(D_i(x)\).
Alwaysassigned units (\(M_i = 1\)): These units do not require potential outcomes or states, and always have \(X_i\) values beyond the cutoff.
Assumptions
 Local Independence and Continuity:
 \(P(D = 1X = c^+, M = 0) > P(D = 1X = c^, M = 0)\)
 No defiers: \(P(D^+ \geq D^X = c, M = 0) = 1\)
 Continuity in potential outcomes and states at \(c\).
 \(F_{XM=0}(x)\) is differentiable at \(c\), with a positive derivative.
 Smoothness of the Running Variable among PotentiallyAssigned Units:
 The derivative of \(F_{XM=0}(x)\) is continuous at \(c\).
 Restrictions on AlwaysAssigned Units:
 \(P(X \geq cM = 1) = 1\) and \(F_{XM=1}(x)\) is rightdifferentiable (or leftdifferentiable) at \(c\).
 This (local) onesided manipulation assumption allows identification of the proportion of alwaysassigned units among all units close to the cutoff.
When alwaysassigned unit exist, the RD design is fuzzy because we have
 Treated and untreated units among the potentiallyassigned (below and above the cutoff)
 Alwaysassigned units (above the cutoff).
Causal Effects of Interest
causal effects among potentiallyassigned units:
\[ \Gamma = E[Y(1)  Y(0)  X = c, D^+ > D^, M = 0] \]
This parameter represents the local average treatment effect (LATE) for the subgroup of “compliers”—units that receive treatment if and only if their running variable \(X_i\) exceeds a certain cutoff.
The parameter \(\Gamma\) captures the causal effect of changes in the cutoff level on treatment status among potentiallyassigned compliers.
RD designs with a manipulated running variable  “DoughnutHole” RD Designs: 



Identification of \(\tau\) in RD Designs
Identification challenges arise due to the inability to distinguish alwaysassigned from potentiallyassigned units, thus Γ is not point identified. We establish sharp bounds on Γ
These bounds are supported by the stochastic dominance of the potential outcome CDFs over observed distributions.
Unit Types and Notation:
 \(C_0\): Potentiallyassigned compliers.
 \(A_0\): Potentiallyassigned alwaystakers.
 \(N_0\): Potentiallyassigned nevertakers.
 \(T_1\): Alwaysassigned treated units.
 \(U_1\): Alwaysassigned untreated units.
The measure \(\tau\) , representing the proportion of alwaysassigned units near the cutoff, is point identified by the discontinuity in the observed running variable density \(f_X\) at the cutoff
Sharp RD:
Units to the left of the cutoff are potentially assigned units. The distribution of their observed outcomes (\(Y\)) are the outcomes \(Y(0)\) of potentiallyassigned compliers (\(C_0\)) at the cutoff.
To determine the bounds on the treatment effect (\(\Gamma\)), we need to assess the distribution of treated outcomes (\(Y(1)\)) for the same potentiallyassigned compliers at the cutoff.
Information regarding the treated outcomes (\(Y(1)\)) comes exclusively from the subpopulation of treated units, which includes both potentiallyassigned compliers (\(C_0\)) and those always assigned units (\(T_1\)).
With \(\tau\) point identified, we can estimate sharp bounds on \(\Gamma\).
Fuzzy RD:
Subpopulation  Types of units 

\(X = c^+, D = 1\)  \(C_0, A_0, T_1\) 
\(X = c^, D = 1\)  \(A_0\) 
\(X= c^+, D = 0\)  \(N_0, U_1\) 
\(X = c^, D = 0\)  \(C_0, N_0\) 
Unit Types and Combinations: There are five distinct unit types and four combinations of treatment assignments and decisions relevant to the analysis. These distinctions are important because they affect how potential outcomes are analyzed and bounded.
Outcome Distributions: The analysis involves estimating the distribution of potential outcomes (both treated and untreated) among potentiallyassigned compliers at the cutoff.
ThreeStep Process:
Potential Outcomes Under Treatment: Bounds on the distribution of treated outcomes are determined using data from treated units.
Potential Outcomes Under NonTreatment: Bounds on the distribution of untreated outcomes are derived using data from untreated units.
Bounds on Parameters of Interest: Using the bounds from the first two steps, sharp upper and lower bounds on the local average treatment effect are derived.
Extreme Value Consideration: The bounds for treatment effects are based on “extreme” scenarios under worstcase assumptions about the distribution of potential outcomes, making them sharp but empirically relevant within the data constraints.
Extensions:
Quantile Treatment Effects: alternative to average effects by focusing on different quantiles of the outcome distribution, which are less affected by extreme values.
Applicability to Discrete Outcomes
Behavioral Assumptions Impact: Assuming a high likelihood of treatment among alwaysassigned units can narrow the bounds of treatment effects by refining the analysis of potential outcomes.
Utilization of Covariates: Incorporating covariates measured prior to treatment can refine the bounds on treatment effects and help target policies by identifying covariate distributions among different unit types.
Notes:
Quantile Treatment Effects (QTEs): QTE bounds are less sensitive to the tails of the outcome distribution, making them tighter than ATE bounds.
Inference on ATEs is sensitive to the extent of manipulation, with confidence intervals widening significantly with small degrees of assumed manipulation.
Inference on QTEs is less affected by manipulation, remaining meaningful even with larger degrees of manipulation.
Alternative Inference Strategy when manipulation is believed to be unlikely. Try different hypothetical values of \(\tau\)
library(formattable)
library(data.table)
library(rdbounds)
set.seed(123)
df < rdbounds_sampledata(10000, covs = FALSE)
#> [1] "True tau: 0.117999815082062"
#> [1] "True treatment effect on potentiallyassigned: 2"
#> [1] "True treatment effect on right side of cutoff: 2.35399944524618"
head(df)
#> x y treatment
#> 1 1.2532616 3.489563 0
#> 2 0.5146925 3.365232 0
#> 3 3.4853777 6.193533 0
#> 4 0.1576616 8.820440 1
#> 5 0.2890962 4.791972 0
#> 6 3.8350019 7.316907 0
rdbounds_est <
rdbounds(
y = df$y,
x = df$x,
# covs = as.factor(df$cov),
treatment = df$treatment,
c = 0,
discrete_x = FALSE,
discrete_y = FALSE,
bwsx = c(.2, .5),
bwy = 1,
# for median effect use
# type = "qte",
# percentiles = .5,
kernel = "epanechnikov",
orders = 1,
evaluation_ys = seq(from = 0, to = 15, by = 1),
refinement_A = TRUE,
refinement_B = TRUE,
right_effects = TRUE,
yextremes = c(0, 15),
num_bootstraps = 5
)
#> [1] "The proportion of alwaysassigned units just to the right of the cutoff is estimated to be 0.04209"
#> [1] "20240513 19:12:33 Estimating CDFs for point estimates"
#> [1] "20240513 19:12:33 .....Estimating CDFs for units just to the right of the cutoff"
#> [1] "20240513 19:12:35 Estimating CDFs with nudged tau (tau_star)"
#> [1] "20240513 19:12:35 .....Estimating CDFs for units just to the right of the cutoff"
#> [1] "20240513 19:12:38 Beginning parallelized output by bootstrap.."
#> [1] "20240513 19:12:42 Computing Confidence Intervals"
#> [1] "20240513 19:12:51 Time taken:0.3 minutes"
rdbounds_summary(rdbounds_est, title_prefix = "Sample Data Results")
#> [1] "Time taken: 0.3 minutes"
#> [1] "Sample size: 10000"
#> [1] "Local Average Treatment Effect:"
#> $tau_hat
#> [1] 0.04209028
#>
#> $tau_hat_CI
#> [1] 0.1671043 0.7765031
#>
#> $takeup_increase
#> [1] 0.7521208
#>
#> $takeup_increase_CI
#> [1] 0.7065353 0.7977063
#>
#> $TE_SRD_naive
#> [1] 1.770963
#>
#> $TE_SRD_naive_CI
#> [1] 1.541314 2.000612
#>
#> $TE_SRD_bounds
#> [1] 1.569194 1.912681
#>
#> $TE_SRD_CI
#> [1] 0.1188634 3.5319468
#>
#> $TE_SRD_covs_bounds
#> [1] NA NA
#>
#> $TE_SRD_covs_CI
#> [1] NA NA
#>
#> $TE_FRD_naive
#> [1] 2.356601
#>
#> $TE_FRD_naive_CI
#> [1] 1.995430 2.717772
#>
#> $TE_FRD_bounds
#> [1] 1.980883 2.362344
#>
#> $TE_FRD_CI
#> [1] 0.6950823 4.6112538
#>
#> $TE_FRD_bounds_refinementA
#> [1] 1.980883 2.357499
#>
#> $TE_FRD_refinementA_CI
#> [1] 0.6950823 4.6112538
#>
#> $TE_FRD_bounds_refinementB
#> [1] 1.980883 2.351411
#>
#> $TE_FRD_refinementB_CI
#> [1] 0.6152215 4.2390830
#>
#> $TE_FRD_covs_bounds
#> [1] NA NA
#>
#> $TE_FRD_covs_CI
#> [1] NA NA
#>
#> $TE_SRD_CIs_manipulation
#> [1] NA NA
#>
#> $TE_FRD_CIs_manipulation
#> [1] NA NA
#>
#> $TE_SRD_right_bounds
#> [1] 1.376392 2.007746
#>
#> $TE_SRD_right_CI
#> [1] 5.036752 5.889137
#>
#> $TE_FRD_right_bounds
#> [1] 1.721121 2.511504
#>
#> $TE_FRD_right_CI
#> [1] 6.663269 7.414185
rdbounds_est_tau <
rdbounds(
y = df$y,
x = df$x,
# covs = as.factor(df$cov),
treatment = df$treatment,
c = 0,
discrete_x = FALSE,
discrete_y = FALSE,
bwsx = c(.2, .5),
bwy = 1,
kernel = "epanechnikov",
orders = 1,
evaluation_ys = seq(from = 0, to = 15, by = 1),
refinement_A = TRUE,
refinement_B = TRUE,
right_effects = TRUE,
potential_taus = c(.025, .05, .1, .2),
yextremes = c(0, 15),
num_bootstraps = 5
)
#> [1] "The proportion of alwaysassigned units just to the right of the cutoff is estimated to be 0.04209"
#> [1] "20240513 19:12:52 Estimating CDFs for point estimates"
#> [1] "20240513 19:12:52 .....Estimating CDFs for units just to the right of the cutoff"
#> [1] "20240513 19:12:53 Estimating CDFs with nudged tau (tau_star)"
#> [1] "20240513 19:12:53 .....Estimating CDFs for units just to the right of the cutoff"
#> [1] "20240513 19:12:56 Beginning parallelized output by bootstrap.."
#> [1] "20240513 19:13:02 Estimating CDFs with fixed tau value of: 0.025"
#> [1] "20240513 19:13:02 Estimating CDFs with fixed tau value of: 0.05"
#> [1] "20240513 19:13:02 Estimating CDFs with fixed tau value of: 0.1"
#> [1] "20240513 19:13:02 Estimating CDFs with fixed tau value of: 0.2"
#> [1] "20240513 19:13:03 Beginning parallelized output by bootstrap x fixed tau.."
#> [1] "20240513 19:13:09 Computing Confidence Intervals"
#> [1] "20240513 19:13:19 Time taken:0.46 minutes"