9 The warning label

Powerful things usually come with warning labels. Examples include pain medication, chemical drain clog removers, construction equipment, and electrical extension cables.

Much of what we’ve seen in this course is a kind of warning label for the data analysis tools. These are powerful; they are used for takes like guiding public policy, conducting scientific research, adjudicating legal disputes, and directing corporate spending. They need to have a warning label.

9.1 Warning: data analysis tools can be harmful to individuals

• Latanya Sweeney on anonymity and predictive tools

• Prediction is hard in novel situations

• Existing social and cultural biases can be reinforced:

  • Discriminatory results in credit card offers
  • Using predictive methods to make parole and sentencing decisions
    • Paper critical of these predictive methods
    • Paper supportive of these predictive methods

9.2 Warning: data analysis tools can be harmful in public policy

• Recall Adrian Simpson’s critique of education meta-analysis. Meta-meta-analyses for ranking effect sizes get turned into government education policies about what teachers should be doing in their classrooms. These need to make sense!

• The worst hypothesis tests ever

• Statistical tools in drug trials:

  • Ben Goldacre on “what doctors don’t know about the drugs they prescribe”
  • the COMPARE project

• Turning data into medical advice: “results not significant” turns into “procedure does nothing”:

  • “Objective Randomised Blinded Investigation With Optimal Medical Therapy of Angioplasty in Stable Angina (ORBITA) and coronary stents: A case study in the analysis and reporting of clinical trials”

• Statistical arguments in courts of law

  • UK guide on the use of statistics in legal proceedings

  • The tragic case of Sally Clark

  • The Prosecutor’s Fallacy

9.3 Warning: data analysis tools can harm scientific research

• The “Replication Crisis”

• Incentives problems in scientific publishing

  • “p-hacking”
  • “HARKing”
  • “The file drawer”
  • Quantity over quality? How many fields could have a Large Hadron Collider type project?

• Publication bias

  • Registered Reports
  • Preregistration

• Communicating science

9.4 Warning: data analysis tools can foster both excessive cynicism and excessive credulity

• “You can make the data say anything”

• “The numbers don’t lie”

• Constructive criticism