10.6 Confidence Intervals in Nonparametric Regression
Constructing confidence intervals (or bands) for nonparametric regression estimators like kernel smoothers, local polynomials, and smoothing splines is more complex than in parametric models. The key challenges arise from the flexible nature of the models and the dependence of bias and variance on the local data density and smoothing parameters.
10.6.1 Asymptotic Normality
Under regularity conditions, nonparametric estimators are asymptotically normal. For a given point \(x\), we have:
\[ \sqrt{n h} \left\{\hat{m}(x) - m(x)\right\} \overset{\mathcal{D}}{\longrightarrow} \mathcal{N}\left(0, \sigma^2 \, \nu(x)\right), \]
where:
- \(n\) is the sample size,
- \(h\) is the bandwidth (for kernel or local polynomial estimators) or a function of \(\lambda\) (for smoothing splines),
- \(\sigma^2\) is the variance of the errors,
- \(\nu(x)\) is a function that depends on the estimator, kernel, and local data density.
An approximate \((1 - \alpha)\) pointwise confidence interval for \(m(x)\) is given by:
\[ \hat{m}(x) \pm z_{\alpha/2} \cdot \sqrt{\widehat{\operatorname{Var}}[\hat{m}(x)]}, \]
where:
- \(z_{\alpha/2}\) is the \((1 - \alpha/2)\) quantile of the standard normal distribution,
- \(\widehat{\operatorname{Var}}[\hat{m}(x)]\) is an estimate of the variance, which can be obtained using plug-in methods, sandwich estimators, or resampling techniques.
10.6.2 Bootstrap Methods
The bootstrap provides a powerful alternative for constructing confidence intervals and bands, particularly when asymptotic approximations are unreliable (e.g., small sample sizes or near boundaries).
10.6.2.1 Bootstrap Approaches
- Residual Bootstrap:
- Fit the nonparametric model to obtain residuals \(\hat{\varepsilon}_i = y_i - \hat{m}(x_i)\).
- Generate bootstrap samples \(y_i^* = \hat{m}(x_i) + \varepsilon_i^*\), where \(\varepsilon_i^*\) are resampled (with replacement) from \(\{\hat{\varepsilon}_i\}\).
- Refit the model to each bootstrap sample to obtain \(\hat{m}^*(x)\).
- Repeat many times to build an empirical distribution of \(\hat{m}^*(x)\).
- Wild Bootstrap:
Particularly useful for heteroscedastic data. Instead of resampling residuals directly, we multiply them by random variables with mean zero and unit variance to preserve the variance structure.
10.6.2.2 Bootstrap Confidence Bands
While pointwise confidence intervals cover the true function at a specific \(x\) with probability \((1 - \alpha)\), simultaneous confidence bands cover the entire function over an interval with the desired confidence level. Bootstrap methods can be adapted to estimate these bands by capturing the distribution of the maximum deviation between \(\hat{m}(x)\) and \(m(x)\) over the range of \(x\).
10.6.3 Practical Considerations
Bias Correction:
Nonparametric estimators often have non-negligible bias, especially near boundaries. Bias correction techniques or undersmoothing (choosing a smaller bandwidth) are sometimes used to improve interval coverage.Effective Degrees of Freedom:
For smoothing splines, the effective degrees of freedom (related to \(\operatorname{tr}(\mathbf{S}_\lambda)\)) provide insight into model complexity and influence confidence interval construction.