4 Analyses

4.1 Pilot analyses

282 people responded. We may want to further screen these 282 responses for short response durations. According to communications with Qualtrics in early December, this is the number of seconds to complete the entire survey. Currently, the “number of people who clicked on the assessment link (330)” versus valid n (282) may take care of our very low duration respondents. The shortest response duration in the 330 datafile is 101 whereas the lowest in the 282 datafile is 43. All analyses were performed via R version R version 4.0.5 (2021-03-31) (R Core Team, 2021). Demographic consituency of the pilot sample is located in Appendix C.

4.1.1 Classical test theory

In addition to the below interactive plot [via plotly version 4.9.3; Sievert et al. (2021)], a full inter-item correlation matrix is located in Appendix D.

## Some items ( I find it difficult to mentally disconnect from work. ) were negatively correlated with the total scale and 
## probably should be reversed.  
## To do this, run the function again with the 'check.keys=TRUE' option

4.1.1.1 Internal consistency estimates

Condition 1 administered items within the substantive dimensions (with successive randomized blocks of Cognitive, Affective, and Behavioral items). Condition 2 administered items within the attitudinal dimensions (with successive randomized blocks of Absorption, Vigor, and Dedication items). Condition 3 stressed the substantive dimensions (with items fully randomized regardless of attitudinal association). Condition 4 stressed the attitudinal dimensions (with items fully randomized within attitudinal dimension regardless of substantive scale association, see Chapter 3.1 and Appendix B). All internal consistency estimates were generated via psych version 2.1.3 (Revelle, 2021). Alphas for the candidate 12-item scales were:

Dimension Undifferentiated Condition 1 Condition 2 Condition 3 Condition 4
Affective 0.87 0.88 0.84 0.88 0.87
Behavioral 0.79 0.73 0.84 0.76 0.87
Cognitive 0.81 0.79 0.85 0.8 0.87
Absorption 0.74 0.72 0.77 0.72 0.87
Vigor 0.85 0.8 0.88 0.84 0.87
Dedication 0.9 0.9 0.91 0.91 0.87

“Cell” level alphas (4 items each scale, responses collapsed across administrative conditions) were:

Cell Alpha
Affective - Absorption 0.66
Affective - Vigor 0.71
Affective - Dedication 0.75
Behavioral - Absorption 0.56
Behavioral - Vigor 0.7
Behavioral - Dedication 0.64
Cognitive - Absorption 0.59
Cognitive - Vigor 0.62
Cognitive - Dedication 0.83

Corrected item-total correlations are presented in Appendix E

4.1.2 Confirmatory factor analyses

We used lavaan version 0.6.8 (Rosseel et al., 2021) and semPlot version 1.1.2 (Epskamp, 2019)

Model \(\chi^2\) df RMSEA SRMR CFI TLI AIC
3-factor substantive 2159.21 591 0.11 0.1 0.64 0.62 25481.97
Hierarchical substantive 2159.21 591 0.11 0.1 0.64 0.62 25481.97
3-factor attitudinal 2318.92 591 0.11 0.1 0.6 0.58 25641.68
Hierarchical attitudinal 2318.92 591 0.11 0.1 0.6 0.58 25641.68

4.1.2.1 Multigroup analyses

  • using experimental condition as “multiple groups” (measurement invariance)

4.1.2.2 Bifactor analyses

bifactor analysis are most commonly applied in the exploration of common method variance (see, for example, Reise, 2012; Rodriguez et al., 2016). Giordano et al. (2020) provide an overview regarding past and potential applications of exploratory bifactor analysis and cite Reise (2012) as an influential impetus for the resurgence of bifactor models in general.

## lavaan 0.6-8 did NOT end normally after 820 iterations
## ** WARNING ** Estimates below are most likely unreliable
## 
##   Estimator                                         ML
##   Optimization method                           NLMINB
##   Number of model parameters                       114
##                                                       
##                                                   Used       Total
##   Number of observations                           239         282
##                                                                   
## Model Test User Model:
##                                                       
##   Test statistic                                    NA
##   Degrees of freedom                                NA
## 
## Parameter Estimates:
## 
##   Standard errors                             Standard
##   Information                                 Expected
##   Information saturated (h1) model          Structured
## 
## Latent Variables:
##                    Estimate   Std.Err  z-value  P(>|z|)
##   Cognitive =~                                         
##     Item_1             1.000                           
##     Item_2             0.516       NA                  
##     Item_3             1.090       NA                  
##     Item_4             0.353       NA                  
##     Item_13            1.130       NA                  
##     Item_14            0.838       NA                  
##     Item_15            0.798       NA                  
##     Item_16            1.098       NA                  
##     Item_25            0.988       NA                  
##     Item_26            0.909       NA                  
##     Item_27            1.080       NA                  
##     Item_28            0.999       NA                  
##   Affective =~                                         
##     Item_5             1.000                           
##     Item_6             0.656       NA                  
##     Item_7             0.475       NA                  
##     Item_8             1.242       NA                  
##     Item_17            0.933       NA                  
##     Item_18            1.372       NA                  
##     Item_19            1.044       NA                  
##     Item_20            0.536       NA                  
##     Item_29            0.841       NA                  
##     Item_30            0.381       NA                  
##     Item_31            0.724       NA                  
##     Item_32            1.063       NA                  
##   Behavioral =~                                        
##     Item_9             1.000                           
##     Item_10            1.109       NA                  
##     Item_11            0.572       NA                  
##     Item_12            1.176       NA                  
##     Item_21            1.362       NA                  
##     Item_22            1.573       NA                  
##     Item_23            0.873       NA                  
##     Item_24            0.890       NA                  
##     Item_33            0.707       NA                  
##     Item_34            0.879       NA                  
##     Item_35            1.237       NA                  
##     Item_36            0.369       NA                  
##   Absorption =~                                        
##     Item_1             1.000                           
##     Item_2         19131.383       NA                  
##     Item_3          -390.734       NA                  
##     Item_4         21074.473       NA                  
##     Item_5          2937.496       NA                  
##     Item_6         -1569.896       NA                  
##     Item_7           727.592       NA                  
##     Item_8         -1533.787       NA                  
##     Item_9          4326.028       NA                  
##     Item_10         5122.173       NA                  
##     Item_11        -1534.018       NA                  
##     Item_12        -1356.265       NA                  
##   Vigor =~                                             
##     Item_13            1.000                           
##     Item_14           -3.413       NA                  
##     Item_15            0.449       NA                  
##     Item_16           -0.517       NA                  
##     Item_17           -0.246       NA                  
##     Item_18           -1.152       NA                  
##     Item_19            0.766       NA                  
##     Item_20           -3.510       NA                  
##     Item_21            0.381       NA                  
##     Item_22           -0.278       NA                  
##     Item_23            0.324       NA                  
##     Item_24            0.093       NA                  
##   Dedication =~                                        
##     Item_25            1.000                           
##     Item_26            1.378       NA                  
##     Item_27            1.009       NA                  
##     Item_28            0.830       NA                  
##     Item_29            0.885       NA                  
##     Item_30            0.650       NA                  
##     Item_31            0.274       NA                  
##     Item_32            0.512       NA                  
##     Item_33            0.010       NA                  
##     Item_34            0.122       NA                  
##     Item_35            0.888       NA                  
##     Item_36            0.783       NA                  
## 
## Covariances:
##                    Estimate   Std.Err  z-value  P(>|z|)
##   Cognitive ~~                                         
##     Affective          0.549       NA                  
##     Behavioral         0.363       NA                  
##   Affective ~~                                         
##     Behavioral         0.372       NA                  
##   Absorption ~~                                        
##     Vigor              0.000       NA                  
##     Dedication        -0.000       NA                  
##   Vigor ~~                                             
##     Dedication        -0.082       NA                  
##   Cognitive ~~                                         
##     Absorption         0.000                           
##     Vigor              0.000                           
##     Dedication         0.000                           
##   Affective ~~                                         
##     Absorption         0.000                           
##     Vigor              0.000                           
##     Dedication         0.000                           
##   Behavioral ~~                                        
##     Absorption         0.000                           
##     Vigor              0.000                           
##     Dedication         0.000                           
## 
## Variances:
##                    Estimate   Std.Err  z-value  P(>|z|)
##    .Item_1             1.343       NA                  
##    .Item_2             0.350       NA                  
##    .Item_3             0.786       NA                  
##    .Item_4             0.124       NA                  
##    .Item_13            0.809       NA                  
##    .Item_14            0.659       NA                  
##    .Item_15            1.471       NA                  
##    .Item_16            0.760       NA                  
##    .Item_25            1.350       NA                  
##    .Item_26            0.629       NA                  
##    .Item_27            1.282       NA                  
##    .Item_28            0.763       NA                  
##    .Item_5             1.324       NA                  
##    .Item_6             1.195       NA                  
##    .Item_7             0.845       NA                  
##    .Item_8             0.688       NA                  
##    .Item_17            0.484       NA                  
##    .Item_18            0.559       NA                  
##    .Item_19            0.771       NA                  
##    .Item_20            0.780       NA                  
##    .Item_29            0.479       NA                  
##    .Item_30            1.514       NA                  
##    .Item_31            0.538       NA                  
##    .Item_32            0.667       NA                  
##    .Item_9             1.293       NA                  
##    .Item_10            1.961       NA                  
##    .Item_11            1.288       NA                  
##    .Item_12            1.342       NA                  
##    .Item_21            0.894       NA                  
##    .Item_22            0.510       NA                  
##    .Item_23            0.338       NA                  
##    .Item_24            1.026       NA                  
##    .Item_33            0.401       NA                  
##    .Item_34            0.532       NA                  
##    .Item_35            0.547       NA                  
##    .Item_36            0.984       NA                  
##     Cognitive          0.593       NA                  
##     Affective          0.639       NA                  
##     Behavioral         0.283       NA                  
##     Absorption         0.000       NA                  
##     Vigor              0.072       NA                  
##     Dedication         0.596       NA

Currently pretty messed up prior to item deletion (Vigor especially). Model also technically not converging need Renata to fix.

Deese guys also do bifactor stuff: Mansolf & Reise (2017)

4.1.3 Summary

Recommendation for final instrument based on consideration of all of the above pieces of evidence