4 Analyses (Scale Definitions)

4.1 Pilot analyses

282 people responded. We may want to further screen these 282 responses for short response durations. According to communications with Qualtrics in early December, this is the number of seconds to complete the entire survey. Currently, the “number of people who clicked on the assessment link (330)” versus valid n (282) may take care of our very low duration respondents. The shortest response duration in the 330 datafile is 101 whereas the lowest in the 282 datafile is 43. All analyses were performed via R version R version 4.1.3 (2022-03-10) (R Core Team, 2022). Demographic consituency of the pilot sample is located in Appendix C.

4.1.1 Classical test theory

In addition to the below interactive plot [via plotly version 4.10.0; Sievert et al. (2021)], a full inter-item correlation matrix is located in Appendix D.

## Some items ( I find it difficult to mentally disconnect from work. ) were negatively correlated with the total scale and 
## probably should be reversed.  
## To do this, run the function again with the 'check.keys=TRUE' option Internal consistency estimates

Condition 1 administered items within the substantive dimensions (with successive randomized blocks of Cognitive, Affective, and Behavioral items). Condition 2 administered items within the attitudinal dimensions (with successive randomized blocks of Absorption, Vigor, and Dedication items). Condition 3 stressed the substantive dimensions (with items fully randomized regardless of attitudinal association). Condition 4 stressed the attitudinal dimensions (with items fully randomized within attitudinal dimension regardless of substantive scale association, see Chapter 3.1 and Appendix B). All internal consistency estimates were generated via psych version 2.2.3 (Revelle, 2022). Alphas for the candidate 12-item scales were:

Dimension Undifferentiated Condition 1 Condition 2 Condition 3 Condition 4
Affective 0.87 0.88 0.84 0.88 0.87
Behavioral 0.79 0.73 0.84 0.76 0.87
Cognitive 0.81 0.79 0.85 0.8 0.87
Absorption 0.74 0.72 0.77 0.72 0.87
Vigor 0.85 0.8 0.88 0.84 0.87
Dedication 0.9 0.9 0.91 0.91 0.87

“Cell” level alphas (4 items each scale, responses collapsed across administrative conditions) were:

Cell Alpha
Affective - Absorption 0.66
Affective - Vigor 0.71
Affective - Dedication 0.75
Behavioral - Absorption 0.56
Behavioral - Vigor 0.7
Behavioral - Dedication 0.64
Cognitive - Absorption 0.59
Cognitive - Vigor 0.62
Cognitive - Dedication 0.83

Corrected item-total correlations are presented in Appendix E

4.1.2 Confirmatory factor analyses

We used lavaan version 0.6.11 (Rosseel et al., 2022) and semPlot version 1.1.5 (Epskamp, 2022)

Bifactor analysis are most commonly applied in the exploration of common method variance (see, for example, Reise, 2012; Rodriguez et al., 2016). Most commonly attributed to Holzinger & Swineford (1937), Giordano et al. (2020) provide an overview regarding past and potential applications of exploratory bifactor analysis and cite Reise (2012) as an influential impetus for the resurgence of bifactor models in general. Giordano & Waller (2020) has a recent review of seven different bifactor model applications.

Deese guys also do bifactor stuff: Mansolf & Reise (2017)

Model \(\chi^2\) df RMSEA SRMR CFI TLI AIC
3-factor substantive 2159.21 591 0.11 0.1 0.64 0.62 25481.97
3-factor attitudinal 2318.92 591 0.11 0.1 0.6 0.58 25641.68

Note. The bifactor model did not actually converge, so no fit indices are available for the bifactor analysis of the 36-item pilot instrument.

4.2 Scale Reduction

Final scale definitions were primarily informed by two sets of indices: 1) corrected item-total correlations, and 2) CFA modification indices. We approached these two procedures as competing item-reduction techniques rather than complementary indices - we did this with the forethought of presenting the competing finalized models within a research context (for example, SIOP). The initially computed corrected item-total correlations can be found in Appendix E. The CFA modification indices extracted from the initial 36-item scale definitions are located in Appendix G.

4.2.1 Corrected item-total informed scale definitions

4.2.2 Modification index-informed scale definitions

Scale reduction was performed from 36 to 22 items using modification indices from the two independent models. For example, if modification indices implicated residual covariances between substantive model indicators, and these indicators were furthermore potentially explained by the items both being, for example, “Affective”, then those items were retained. Similarly, if modification indices from the attitudinal model CFA implicated residual item covariances, and these items were associated via the substantive model, those two items were given preference for retention. This process became less instructive after the reduction of 14 items. The goal of the assessment was a finalized 18 (6 per dimension and 2 per dimensional “intersection”), so we prioritized empirial factor loadings (from both substantive and attitudinal models) for the final 4 item reductions. This resulted in two possible final scale definitions: one giving “priority” to the attitudinal model, and one giving “priority” to information extracted from the substantive model.

18-Item Model \(\chi^2\) df RMSEA SRMR CFI TLI AIC
substantive (att priority) 430.47 132 0.09 0.07 0.84 0.81 13467.84
attitudinal (att priority) 396.01 132 0.09 0.06 0.85 0.83 13433.39
substantive (sub priority) 460.61 132 0.09 0.07 0.83 0.8 14216.13
attitudinal (sub priority) 447.7 132 0.09 0.07 0.82 0.8 14424.26
bifactor (sub priority) 336.58 126 0.08 0.06 0.89 0.86 15026.75

4.3 Final proposed structure

The two approaches were applied in the interest of a research question. From a more practical sense, the final scale definitions were informed by these two approaches as well as a consideration of item content (was the indicator an important inclusion for proper content domain sampling?).

4.3.1 Omega

Omega reliability (\(\omega\)) estimates independent latent construct reliability absent the effect of other constructs. Rodriguez et al. (2016) identify “Omega Hierarchical” (\(\omega_H\)) as particularly applicable for bifactor models. Specifically, \(\omega_H\) “…estimates the proportion of variance in total scores that can be attributed to a single general factor, thereby, treating variability in scores due to group factors as measurement error” (p. 150).

4.4 Multigroup analyses

  • using experimental condition as “multiple groups” (measurement invariance)

4.5 Summary

Recommendation for final instrument based on consideration of all of the above pieces of evidence