Odds of Doom (%)
Ψlogical
Testing
Chapter 5…
…Vladidity
…Vladidity
House keeping 🧹🗑
- First exam 3/6 (Thurs)
- In class, but administered via Canvas (Ch 1 \(\rightarrow\) 8)
- Lab grade
- Poll incoming
- Project:
- Feedback delivery & revision
- Hinkin (2005)
🔭🌠ASTEROID YR4 PANIC METER!!!🌌💥
Validity is…
…propriety of inferences (from test scores)
“Psychometric” Validity
- aka test validity
- aka construct validity
- different from other uses of the term
- for example:
- internal validity
- external validity
- for example:
Validity is…
- …the interpretation of test scores in terms of a specific psychological construct
- …a matter of degree1
- …based on empirical evidence and theory
Validity is NOT…
…a property of the test itself
…an “all-or-none” issue
…frivilously determined
Practical meaning
- is the name we give to scores appropriate?
- Scores from Form A should be labeled:
- Math Ability?
- Weight?
- Attitude?
Where does this come from?
- the Standards
- published by:
- American Educational Research Association
- American Psychological Association
- National Council on Measurement in Education
The Standards (History)
- 1966, 1974, 1985, 1999, and 2014 (5th ed)
- 2014 revision focused on:
- technological advances
- increased use for accountability and policy-setting
- access for all
- workplace testing
Validity study (aka Validation)
…systematic procedure to gather validity evidence
Typically…
- investigate item content
- determine association with meaningful outcome
- demonstrate association with other measures of same (or similar) construct
Content validation
Question: is the test an adequate representation of the construct domain?
Primary concerns:
- Deficiency
- aka construct underrepresentation
- Contamination
- aka construct-irrelevant variance
Content validation (II)
- Ratings usually collected via Subject Matter Experts (aka SMEs)
- Lawshe (1975) Content Validity Ratio
- Essential
- Useful
- Not necessary
Criterion validation
Question: do test scores relate to anything of real life consequence?
Outcome variables (aka Criteria)…
- …are collected at the same time as your IV
- Concurrent validation
- …are collected at a later time than your IV
- Predictive validation
Criterion validation (II)
Common designs:
- SAT \(\rightarrow\) College GPA?
- Beck’s Depression Inventory \(\rightarrow\) Suicidal Ideation?
- MMPI \(\rightarrow\) Psychopathology?
- NEO \(\rightarrow\) Good Co-Worker?
Construct validation
Question: does the test function similarly as other purported tests of the same construct?
Two lenses:
- Convergence
- …with other measures of similar things
- Discrimination
- …from measures of theoretically distinct things
Activity
- Get in project groups
- Find 2 existing measures of construct (or close to construct)
- Find 1 existing measure of an unrelated construct
Validity coefficient
- typically the focus of a validation study
- usually a correlation coefficient
- like reliability coefficients, range from 0 \(\rightarrow\) 1
- to what extent are inferences appropriate?
- sometimes squared
Coefficient “quality”
- Ask for technical report from test provider
- “More” is better
- larger coefficient
- number of studies
- focus on sampled population(s)
- constituency & size
- validity generalization evidence?
Face validity
- Do the test items appear to be relevant for the testing context?🤨
- Different contexts matter:
- context = purpose
- Applying for a job
- Tradeoff may exist between relevant and more easily fakable
Tip
The primary difference between content validity and face validity is who provides estimates: subject matter experts or test-takers
Validity & Reliability
- reliability is 1st necessary
- cannot have validity without reliability
- you can have reliability without validity
- reliability “puts a cap on” validity
Which of the following is possible?
0% Correct (0/5)
Correlating an employment application with ratings of job performance (collected after a 3 month probation) would give evidence of the application’s…
0% Correct (0/5)
Reliability is freedom from…
0% Correct (0/5)
Methodologically, a criterion used for validation can also be called a/an…
0% Correct (0/5)
Another name for the test being validated is a/an…
0% Correct (0/5)
References
Hinkin, T. R. (2005). Research in organizations: Foundations and methods of inquiry (R. A. Swanson & E. F. Holton III, Eds.; pp. 161–179). Berrett-Koehler Publishers.
Lawshe, C. (1975). A quantitative approach to content validity. Personnel Psychology, 28, 563–575.
Ψlogical Testing Chapter 5… …Vladidity