B.2 False positive psychology

B.2.1 Introduction

To highlight problematic research practices within psychology, Simmons, Nelson and Simonsohn (2011) published a controversial article with a necessarily false finding. By conducting simulations and two simple behavioral experiments, the authors show that flexibility in data collection, analysis, and reporting dramatically increases the rate of false-positive findings.

B.2.2 Data sources

Articles reporting the original research:

  • Simmons, J.P., Nelson, L.D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. doi: https://doi.org/10.1177/0956797611417632

Article on the data used here:

  • Simmons, J.P., Nelson, L.D., & Simonsohn, U. (2014). Data from paper “False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant.” Journal of Open Psychology Data, 2(1), e1. doi: http://doi.org/10.5334/jopd.aa

See https://openpsychologydata.metajnl.com/articles/10.5334/jopd.aa/ for data. (A zip-archive with txt files is available at http://dx.doi.org/10.5281/zenodo.7664.)

B.2.3 Codebook

The study data is stored in 2 seperate files: study1.xlsx & study2.xlsx. Both data files contain the same information about each participant in 17 variables:

  1. age: Days since participant was born (based on their self-reported birthday).

  2. dad: Father’s age (in years).

  3. mom: Mother’s age (in years).

  4. female: Is the participant a woman?

    • 1: yes
    • 2: no

  5. root: Did they geht correctly the square root of 100?

    • 1: yes
    • 2: no

  6. bird: Imagine a restaurant you really like offered a 30% discount for dining between 4 pm and 6 pm.
    How likely would you be to take advantage of that offer?

    • 1: very unlikely to 7: very likely

  7. political: In the political spectrum, where would you place yourself?

    • 1: very liberal
    • 2: liberal
    • 3: centrist
    • 4: conservative
    • 5: very conservative

  8. quarterback: If you had to guess who was chosen the quarterback of the year in Canada last year, which of the following four options would you choose?

    • 1: Dalton Bell
    • 2: Daryll Clark
    • 3: Jarious Jackson
    • 4: Frank Wilczynski

  9. olddays: How often have you referred to some past part of your life as “the good old days?” Scale from

    • 11: never
    • 12: almost never
    • 13: sometimes
    • 14: often
    • 15: very often

  10. potato: Did the participant hear the song ‘Hot Potato’ by The Wiggles?

    • 1: yes
    • 2: no

  11. when64: Did the participant hear the song ‘When I am 64’ by The Beatles?

    • 1: yes
    • 2: no

  12. kalimba: Did the participant hear the song ‘Kalimba’ by Mr. Scrub?

    • 1: yes
    • 2: no

  13. feelold: How old do you feel?

    • 1: very young
    • 2: young
    • 3: neither young nor old
    • 4: old
    • 5: very old

  14. computer: “Computers are complicated machines.”

    • Scale from 1: strongly disagree to 5: strongly agree.

  15. diner: Imagine you were going to a diner for dinner tonight, how much do you think you would like the food?

    • Scale from 1: dislike extremely to 9: like extremely.

  16. cond: In which condition was the participant?

    • control: Participant heard the song ‘Kalimba’ by Mr. Scrub;
    • potato: Participant heard the song ‘Hot Potato’ by The Wiggles;
    • 64: Participant heard the song ‘When I am 64’ by The Beatles.

  17. aged365: Participant age (in years).

B.2.4 Getting the data

Files available

The following file was generated from the original data files (and saved in .csv format):

  • falsePosPsy_all.csv: Combines the 2 original datasets in one file:
    http://rpository.com/ds4psy/data/falsePosPsy_all.csv.
    2 variables that denote the original study (1 vs. 2) and a unique participant ID (ranging from 1 to 78) have been added, so that the data file now contains 78 cases and 19 variables.

Loading data

# Load csv-data files from online links:
falsePosPsy_all <- readr::read_csv(file = "http://rpository.com/ds4psy/data/falsePosPsy_all.csv")

# Check: 
dim(falsePosPsy_all)  # 78 x 19 
#> [1] 78 19

# Check number of missing values: 
sum(is.na(falsePosPsy_all))  # 0 missing values  
#> [1] 0

B.2.5 References

Articles

  • Simmons, J.P., Nelson, L.D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. doi: https://doi.org/10.1177/0956797611417632

  • Simmons, J.P., Nelson, L.D., & Simonsohn, U. (2014). Data from paper “False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant.” Journal of Open Psychology Data, 2(1), e1. doi: http://doi.org/10.5334/jopd.aa