Chapter 5 Multiple Linear Models
Part II presents some statistical methods relating to Travel-Urban Form models. Chapter 5 to 10 introduce the main methods could be applied on VMT-urban form models because the response are continue variables. The models of mode choice are placed in Chapter of Generalized linear models. The last chapter Meta-Analysis introduces basic ideas and approaches of meta-analysis and dealing with publication bias.
These contents focus on some analysis which often be neglected or omitted in travel-urban form models and may give some inspiration for improving current studies. Hence, these are mostly from several textbooks (Casella and Berger 2002; Montgomery, Peck, and Vining 2021; Ravishanker and Dey 2020) and lecture notes by Dr. Robert Fountain, Dr. Nadeeshani Jayasena, and Dr. Jong Sung Kim. It will not involve the front edge of techniques in recent published papers in statistics.
5.1 Assumptions
5.1.1 Additive and linearity
For linear models, the most important assumptions are the additive and linear relationship between the response and predictors. Gravity Law discloses that travel distance has a multiplicative (inverse) relationship with the ‘masses’ of two places. If the population size can be a representative of built environment, the additive relationship will not hold. Previous studies also shows that the effect sizes of built environment with respect of travel are small and complex. There is not sufficient evidence to support or against the linear hypothesis.
5.1.2 Independent Identically Distributed (IID)
Another essential assumption is that random error are Independent Identically Distributed (IID). Random error is also called residual, which refer to the difference between observed yy and fitted ˆy^y. ˆy^y are the linear combinations of predictors XX. residuals represent the part can not be explained by the model.
e=y−ˆy
The expected value, the variances, and the covariances among the random errors are the first- and second-moment of residuals. ‘Identical’ means that random errors should have zero mean and constant variance. The homogeneity of variance is also called homoscedasticity.
E(ε)=0,Var(ε)=σ2
‘Independent’ requires the random errors are uncorrelated. That is
Cov[εi,εj]=0,i≠j
Once the conditions of IID are satisfied, the Gauss - Markov theorem 5.1 proves that least-square method could give the minimum-variance unbiased estimators (MVUE) or called the best linear unbiased estimators (BLUE). These conditions are not strict and make regression method widely applicable.
Theorem 5.1 (Gauss - Markov theorem) For the regression model (1.1) with the assumptions E(ε)=0, Var(ε)=σ2, and uncorrelated errors, the least-squares estimators are unbiased and have minimum variance when compared with all other unbiased estimators that are linear combinations of the yi. (Montgomery, Peck, and Vining 2021)
Another version is that: Under Models II - VII, if λ′β is estimable and ˆβ is any solution to the normal equations, then λ′ˆβ is a linear unbiased estimator of λ′β and, under Model II, the variance of λ′ˆβ is uniformly less than that of any other linear unbiased estimator of λ′β (IX, Theorem E13, p38)
Unfortunately, many of the predictors are correlated. Moreover, the observations from various cities, regions, or counties are very unlikely identical. This issue is called heteroscedasticity. Related contents are in Section of Diagonusis and Validation.
5.1.3 Normality
When conducting hypothesis test and confidence intervals, the required assumption is y|x∼N(Xβ,σ2I). Maximum Likelihood Methods also requires this assumption.
Evidence has demonstrated that travel distance is not Normal distributed. The Zipf’s law also prove that travel distance follows a power distribution. Using logarithm transformations, the skewed distribution can be converted to an approximate normal distribution.
There are some quantitative methods which can examine nomalirty of the transformed distributions.
5.2 Estimations
5.2.1 Least Squares
- Ordinary Least Squares
Least-Squares method can be used to estimate the coefficients β in equation (1.1) The dimension of X is n×p, which means the data contain n observations and p−1 predictors. The p×1 vector of least-squares estimators is denoted as ˆβ and the solution to the normal equations is
ˆβ=(X′X)−1X′y
and
ˆσ2=1n−p(y−Xˆβ)′(y−Xˆβ)
Here requires X′X are invertible, that is, the covariates are linearly independent if X has rank p (Kim 2020, V., Definition, p.22).
Given the estimated coefficients, the model can give the fitted values of response as:
ˆy=Xˆβ=X(X′X)−1X′y=Hy
where H=X(X′X)−1X′ is hat matrix and e=y−ˆy=y−Xˆβ=(I−H)y
- Generalized Least Squares
When the observations are not independent or have unequal variances, the covariance matrix of error is not identity matrix. The assumption of regression model V[ε]=σ2I doesn’t hold. Denote V is a known n×n positive definite matrix and V[ε]=σ2V. Then, there exists an n×n symmetric matrix K with rank n and V=KK′. Let
z=K′y, B=K−1X,and η=K′ε
The linear model becomes z=Bβ+η and V[η]=σ2I. If the model is full rank, that is rank(X)=p then X′V−1X is invertible and the generalized least squares solution is
ˆβGLS=(X′V−1X)−1X′V−1y
and
ˆσ2GLS=1n−p(y−XˆβGLS)′V−1(y−XˆβGLS)
5.2.2 Standardized coefficients
The value of ˆβj means that, given all other coefficients fixed, for each change of one unit in xj, the average change in the mean of Y. However, the units of predictors X are very different. Hence, the values of coefficients are not comparable.
Unit normal scaling or Unit length scaling can convert ˆβj to dimensionless regression coefficient, which is called standardized regression coefficients. Let
zij=xij−ˉxj√∑ni=1(xij−ˉxj)2y0i=yi−ˉy√∑ni=1(yi−ˉy)2
ˆb=(Z′Z)−1Z′y0, orˆbj=ˆβj√∑ni=1(xij−ˉxj)2∑ni=1(yi−ˉy)2, j=1,2,...(p−1), andˆβ0=ˉy−p−1∑j=1ˆβjˉxj
Note that Z′Z correlations matrix.
Z′Z=[1r12r13…r1kr211r23…r2kr31321…r3k⋮⋮⋮⋱⋮rk1rk2k3…1],Z′y0=[r1yr2yr3y⋮rky]
where
rij=∑nu=1(xui−ˉxi)(xuj−ˉxj)√∑nu=1(xui−ˉxi)2∑nu=1(xuj−ˉxj)2rjy=∑nu=1(xuj−ˉxj)(yu−ˉy)√∑nu=1(xuj−ˉxj)2∑nu=1(yu−ˉy)2
where rij is the simple correlation between xi and xj. rjy is the simple correlation between xj and y
It seems that standardized regression coefficients are comparable. However, the value of ˆbj depends on other predictors. Therefore, comparison between different models is still problematic.
5.2.3 Elasticity
Elasticity is commonly used to determine the relative importance of a variable in terms of its influence on a dependent variable. It is generally interpreted as the percent change in the dependent variable induced by a 1% change in the independent variable (McCarthy 2001).
ei=βiXiYi≈∂Yi∂XiXiYi
Model | Marginal.Effects | Elasticity |
---|---|---|
Linear | β | βXiYi |
Log-linear | βYi | βXi |
Linear-log | β1Xi | β1Yi |
Log-log | βYiXi | β |
Logit | βpi(1−pi) | βXi(1−pi) |
Poisson | βλi | βXi |
NB | βλi | βXi |
It might be a typo that Reid Ewing and Cervero (2010) use the formula of βˉX(1−ˉYn) for Logit model. In Poisson model and Negative Binomial model, λi=exp[x′iβ] (Greene, 2018, eq.18-17,21). For truncated Poisson model: δi=(1−Pi,0−λiPi,0)(1−Pi,0)2⋅λiβ (Greene, 2018, eq.18-23). Hurdle model will give separate marginal(partial) effects (Greene, 2018, example 18.20)
5.2.4 Combined effects?
Some studies sums up the standardized coefficients or elasticities and called the summation as combined effects [leeComparingImpactsLocal2020]. Although these values are dimensionless, this method is problematic because different model specifications and data ranges are not comparable.
5.3 Inference
5.3.1 Analysis of Variance
Analysis of Variance (ANOVA) is the fundamental approach in regression analysis. Actually, this method analysis the variation in means rather than variances themselves (Casella & Berger, 2002, Ch.11).
Once the linear relationship holds, the response y can be decomposed to
y′y=y′Hy+y′(I−H)yy′y=ˆβX′y+y′y−ˆβX′yy′y−nˉy2=ˆβX′y−nˉy2+y′y−ˆβX′y∑(y−ˉy)2=∑(ˆy−ˉy)2+∑(y−ˆy)2SST=SSR+SSE
where SST is Sum of Squares Total, SSR is Sum of Squares Regression, and SSE is Sum of Square Error. SSE=e′e represents the unknown part of model.
For Generalized Least Squares method, SST=y′V−1y, SSR=ˆβ′B′z=y′V−1X(X′V−1X)−1X′V−1y, and SSE=SST−SSR
5.3.2 Hypothesis Test
- Significance of Regression
Significance of regression means if the linear relationship between response and predictors is adequate. The hypotheses for testing model adequacy are
H0:β0=β1=⋯=βp−1=0H1:at least one βj≠0, j=0,1,...,(p−1)
By Theorem D14 (XX, p.90), if an n×1random vector y∼N(μ,I), then
y′y∼χ2(n,12μ′μ)
Recall the assumption of y|x∼N(Xβ,σ2I).
By the additive property of χ2 distribution,
MSEσ2=y′(I−H)y(n−p)σ2∼χ2(n−p)MSRσ2=y′Hy(p−1)σ2∼χ2(p−1)
Though σ2 is usually unknown, by the relationship between χ2 and F distributions,
F0=MSEMSR∼F(p−1),(n−p),λ
where λ is the non-centrality parameter. It allows to test the hypotheses given a significance level α. If test statistic F0>Fα,(p−1),(n−p), then one can reject H0.
If a VMT-urban form model added many predictors but adjusted R2 is still low, the association between travel distance and built environment might be spurious.
- Significance of Coefficients
For testing a specific coefficient, the hypothesis is
H0:βj=0H1:βj≠0
ˆβ is a linear combination of y. Based on the assumption of y|x∼N(Xβ,σ2I), it can be proved that ˆβ∼N(β,σ2(X′X)−1) and
t0=ˆβjse(ˆβj)=ˆβj√ˆσ2Cjj∼t(n−p)
where Cjj is the element at the j row and j column of (X′X)−1. If |t0|<tα/2,(n−p), then the test failed to reject the H0, this predictor can be removed from the model. This test is called partial or marginal test because the test statistic for βj depends on all the predictors in the model.