Ψlogical
Testing

Chapter 5…
…Vladidity

House keeping 🧹🗑

  • First exam 3/6 (Thurs)
    • In class, but administered via Canvas (Ch 1 \(\rightarrow\) 8)
  • Lab grade
    • Poll incoming
  • Project:
    • Feedback delivery & revision
    • Hinkin (2005)

🔭🌠ASTEROID YR4 PANIC METER!!!🌌💥

Validity is…

…propriety of inferences (from test scores)

“Psychometric” Validity

  • aka test validity
  • aka construct validity
  • different from other uses of the term
    • for example:
      • internal validity
      • external validity

Validity is…

  • …the interpretation of test scores in terms of a specific psychological construct
  • …a matter of degree1

  • …based on empirical evidence and theory

Validity is NOT…

  • …a property of the test itself

  • …an “all-or-none” issue

  • …frivilously determined

Practical meaning

  • is the name we give to scores appropriate?
  • Scores from Form A should be labeled:
    • Math Ability?
    • Weight?
    • Attitude?

Where does this come from?

  • the Standards
  • published by:
    • American Educational Research Association
    • American Psychological Association
    • National Council on Measurement in Education

The Standards (History)

  • 1966, 1974, 1985, 1999, and 2014 (5th ed)
  • 2014 revision focused on:
    1. technological advances
    2. increased use for accountability and policy-setting
    3. access for all
    4. workplace testing

Validity study (aka Validation)

…systematic procedure to gather validity evidence

Typically…

  • investigate item content
  • determine association with meaningful outcome
  • demonstrate association with other measures of same (or similar) construct

Content validation

Question: is the test an adequate representation of the construct domain?

Primary concerns:

  • Deficiency
    • aka construct underrepresentation
  • Contamination
    • aka construct-irrelevant variance

Content validation (II)

  • Ratings usually collected via Subject Matter Experts (aka SMEs)
  • Lawshe (1975) Content Validity Ratio
    • Essential
    • Useful
    • Not necessary

Criterion validation

Question: do test scores relate to anything of real life consequence?

Outcome variables (aka Criteria)…

  • …are collected at the same time as your IV
    • Concurrent validation
  • …are collected at a later time than your IV
    • Predictive validation

Criterion validation (II)

Common designs:

  • SAT \(\rightarrow\) College GPA?
  • Beck’s Depression Inventory \(\rightarrow\) Suicidal Ideation?
  • MMPI \(\rightarrow\) Psychopathology?
  • NEO \(\rightarrow\) Good Co-Worker?

Construct validation

Question: does the test function similarly as other purported tests of the same construct?

Two lenses:

  • Convergence
    • …with other measures of similar things
  • Discrimination
    • …from measures of theoretically distinct things

Activity

  1. Get in project groups
  2. Find 2 existing measures of construct (or close to construct)
  3. Find 1 existing measure of an unrelated construct

Validity coefficient

  • typically the focus of a validation study
  • usually a correlation coefficient
    • like reliability coefficients, range from 0 \(\rightarrow\) 1
    • to what extent are inferences appropriate?
    • sometimes squared

Coefficient “quality”

  • Ask for technical report from test provider
  • “More” is better
    • larger coefficient
    • number of studies
  • focus on sampled population(s)
    • constituency & size
  • validity generalization evidence?

Face validity

  • Do the test items appear to be relevant for the testing context?🤨
  • Different contexts matter:
    • context = purpose
    • Applying for a job
  • Tradeoff may exist between relevant and more easily fakable

Tip

The primary difference between content validity and face validity is who provides estimates: subject matter experts or test-takers

Validity & Reliability

  • reliability is 1st necessary
  • cannot have validity without reliability
  • you can have reliability without validity
  • reliability “puts a cap on” validity

Which of the following is possible?

  • high validity, low reliability
  • low validity, high reliability
  • high fidelity, low validity
  • low fidelity, high validity

Correlating an employment application with ratings of job performance (collected after a 3 month probation) would give evidence of the application’s…

  • Predictive validity
  • Concurrent validity
  • Content validity
  • Construct validity

Reliability is freedom from…

  • error
  • taxes
  • validity
  • scores

Methodologically, a criterion used for validation can also be called a/an…

  • DV
  • IV
  • Moderator
  • Confound

Another name for the test being validated is a/an…

  • predictor
  • criterion
  • moderator
  • confound

References

Hinkin, T. R. (2005). Research in organizations: Foundations and methods of inquiry (R. A. Swanson & E. F. Holton III, Eds.; pp. 161–179). Berrett-Koehler Publishers.
Lawshe, C. (1975). A quantitative approach to content validity. Personnel Psychology, 28, 563–575.