4 Analyses
4.1 Pilot analyses
282 people responded. We may want to further screen these 282 responses for short response durations. According to communications with Qualtrics in early December, this is the number of seconds to complete the entire survey. Currently, the “number of people who clicked on the assessment link (330)” versus valid n (282) may take care of our very low duration respondents. The shortest response duration in the 330 datafile is 101 whereas the lowest in the 282 datafile is 43. All analyses were performed via R version R version 4.0.3 (2020-10-10) (R Core Team, 2020). Demographic consituency of the pilot sample is located in Appendix C.
4.1.1 Classical test theory
In addition to the below interactive plot (via plotly version 4.9.2.1; Sievert et al., 2020), a full inter-item correlation matrix is located in Appendix D.
## Some items ( I find it difficult to mentally disconnect from work. ) were negatively correlated with the total scale and
## probably should be reversed.
## To do this, run the function again with the 'check.keys=TRUE' option4.1.1.1 Internal consistency estimates
Condition 1 administered items within the substantive dimensions (with successive randomized blocks of Cognitive, Affective, and Behavioral items). Condition 2 administered items within the attitudinal dimensions (with successive randomized blocks of Absorption, Vigor, and Dedication items). Condition 3 stressed the substantive dimensions (with items fully randomized regardless of attitudinal association). Condition 4 stressed the attitudinal dimensions (with items fully randomized within attitudinal dimension regardless of substantive scale association, see Chapter 3.1 and Appendix B). All internal consistency estimates were generated via psych version 2.0.12 (Revelle, 2020). Alphas for the candidate 12-item scales were:
| Dimension | Undifferentiated | Condition 1 | Condition 2 | Condition 3 | Condition 4 |
|---|---|---|---|---|---|
| Affective | 0.87 | 0.88 | 0.84 | 0.88 | 0.87 |
| Behavioral | 0.79 | 0.73 | 0.84 | 0.76 | 0.87 |
| Cognitive | 0.81 | 0.79 | 0.85 | 0.8 | 0.87 |
| Absorption | 0.74 | 0.72 | 0.77 | 0.72 | 0.87 |
| Vigor | 0.85 | 0.8 | 0.88 | 0.84 | 0.87 |
| Dedication | 0.9 | 0.9 | 0.91 | 0.91 | 0.87 |
“Cell” level alphas (4 items each scale, responses collapsed across administrative conditions) were:
| Cell | Alpha |
|---|---|
| Affective - Absorption | 0.66 |
| Affective - Vigor | 0.71 |
| Affective - Dedication | 0.75 |
| Behavioral - Absorption | 0.56 |
| Behavioral - Vigor | 0.7 |
| Behavioral - Dedication | 0.64 |
| Cognitive - Absorption | 0.59 |
| Cognitive - Vigor | 0.62 |
| Cognitive - Dedication | 0.83 |
Corrected item-total correlations are presented in Appendix E
4.1.2 Confirmatory factor analyses
We used lavaan version 0.6.8 (Rosseel et al., 2021) and semPlot version 1.1.2 (Epskamp, 2019)
Bifactor analysis are most commonly applied in the exploration of common method variance (see, for example, Reise, 2012; Rodriguez et al., 2016). Most commonly attributed to Holzinger & Swineford (1937), Giordano et al. (2020) provide an overview regarding past and potential applications of exploratory bifactor analysis and cite Reise (2012) as an influential impetus for the resurgence of bifactor models in general. Giordano & Waller (2020) has a recent review of seven different bifactor model applications.
Deese guys also do bifactor stuff: Mansolf & Reise (2017)
| Model | \(\chi^2\) | df | RMSEA | SRMR | CFI | TLI | AIC |
|---|---|---|---|---|---|---|---|
| 3-factor substantive | 2159.21 | 591 | 0.11 | 0.1 | 0.64 | 0.62 | 25481.97 |
| 3-factor attitudinal | 2318.92 | 591 | 0.11 | 0.1 | 0.6 | 0.58 | 25641.68 |
Note. The bifactor model did not actually converge, so no fit indices are available for the bifactor analysis of the 36-item pilot instrument.
4.2 Final Scale Definitions
Final scale definitions were primarily informed by two sets of indices: 1) corrected item-total correlations, and 2) CFA modification indices. We approached these two procedures as competing item-reduction techniques rather than complementary indices - we did this with the forethought of presenting the competing finalized models within a research context (for example, SIOP). The initially computed corrected item-total correlations can be found in Appendix E. The CFA modification indices extracted from the initial 36-item scale definitions are located in Appendix G.
4.2.1 Corrected item-total informed scale definitions
## npar fmin chisq df
## 39.000 1.055 538.098 132.000
## pvalue baseline.chisq baseline.df baseline.pvalue
## 0.000 2262.874 153.000 0.000
## cfi tli nnfi rfi
## 0.808 0.777 0.777 0.724
## nfi pnfi ifi rni
## 0.762 0.658 0.809 0.808
## logl unrestricted.logl aic bic
## -6456.018 -6186.969 12990.036 13128.145
## ntotal bic2 rmsea rmsea.ci.lower
## 255.000 13004.506 0.110 0.100
## rmsea.ci.upper rmsea.pvalue rmr rmr_nomean
## 0.120 0.000 0.124 0.124
## srmr srmr_bentler srmr_bentler_nomean crmr
## 0.080 0.080 0.080 0.084
## crmr_nomean srmr_mplus srmr_mplus_nomean cn_05
## 0.084 0.080 0.080 76.734
## cn_01 gfi agfi pgfi
## 82.846 0.782 0.718 0.604
## mfi ecvi
## 0.451 2.416
## npar fmin chisq df
## 39.000 1.052 536.537 132.000
## pvalue baseline.chisq baseline.df baseline.pvalue
## 0.000 2262.874 153.000 0.000
## cfi tli nnfi rfi
## 0.808 0.778 0.778 0.725
## nfi pnfi ifi rni
## 0.763 0.658 0.810 0.808
## logl unrestricted.logl aic bic
## -6455.237 -6186.969 12988.474 13126.583
## ntotal bic2 rmsea rmsea.ci.lower
## 255.000 13002.944 0.110 0.100
## rmsea.ci.upper rmsea.pvalue rmr rmr_nomean
## 0.119 0.000 0.131 0.131
## srmr srmr_bentler srmr_bentler_nomean crmr
## 0.076 0.076 0.076 0.080
## crmr_nomean srmr_mplus srmr_mplus_nomean cn_05
## 0.080 0.076 0.076 76.955
## cn_01 gfi agfi pgfi
## 83.084 0.790 0.728 0.610
## mfi ecvi
## 0.452 2.410
| Model | \(\chi^2\) | df | RMSEA | SRMR | CFI | TLI | AIC |
|---|---|---|---|---|---|---|---|
| 3-factor substantive | 538.1 | 132 | 0.11 | 0.08 | 0.81 | 0.78 | 12990.04 |
| 3-factor attitudinal | 536.54 | 132 | 0.11 | 0.08 | 0.81 | 0.78 | 12988.47 |
| Bi-factor model | 214.91 | 111 | 0.06 | 0.04 | 0.95 | 0.93 | 12708.85 |
4.2.2 Modification index-informed scale definitions
4.3 Final proposed structure
The two approaches were applied in the interest of a research question. From a more practical sense, the final scale definitions were informed by these two approaches as well as a consideration of item content (was the indicator an important inclusion for proper content domain sampling?).
4.3.1 Omega
Omega reliability (\(\omega\)) estimates independent latent construct reliability absent the effect of other constructs.
4.4 Multigroup analyses
- using experimental condition as “multiple groups” (measurement invariance)
4.5 Summary
Recommendation for final instrument based on consideration of all of the above pieces of evidence