1.3 Reproducible research

Results should be explicit and reproducible. Anyone repeating the same steps on the same data should obtain the same results.

1.3.1 What?

Peng & Hicks (2020) make the following general distinction:

  • reproducibility is the ability of independent analysts to re-create the results claimed by the original authors using the original data and analysis techniques.

  • replicability is the ability for confirming scientific results and claims by completely independent investigations.

While the concepts of reproduciblity and replicability are related, it is worth noting that they are focused on quite different goals and address different aspects of scientific progress.

Replicability is a great normative idea, but — like many ideals — very difficult to implement in practice. For instance, if two laboratories conduct the “same” study, how similar do their participants need to be? Does it matter on which day or time the study is conducted? What about differences due to different statistical procedures?

Overall, while most serious researchers agree that replicability is desirable ideal, the issue is a mine-field and boomerang for the entire field of psychology — and many others, who perhaps have not yet investigated the issue in such a self-deprecating manner.

1.3.2 Why?

Introduce general idea and goals of reproducible research (see Peng, 2011):

General goal of transparency: Others should not only obtain the final results and interpretation (e.g., in the form of a published article), but also how results were derived from data.

1.3.3 How?

Providing information: Details on all steps (prior to/during/after data collection and analysis).

More specifically,

  • reveal the process of collecting the data and conducting the research

Practically, this implies

  • publish the data and every step of the analysis process.
The spectrum of reproducibility (Source: Fig. 1 from Peng, 2011).

Figure 1.8: The spectrum of reproducibility (Source: Fig. 1 from Peng, 2011).

Figure 1.8 illustrates that we can think of reproducibility as a continuum. On one end, there is the traditional publication. For instance, a typical article in a scientific journal describes how data was acquired, how it was processed and analyzed, and which results were obtained. These results are then interpreted, discussed, and possible conclusions drawn. Interestingly, courses of scientific writing typically emphasize that a goal of writing both Methods and Results sections is to be as explicit and transparent as possible.

An alternative view is provided as the “gold standard” at the other end of the continuum. Enabling a complete replication would require much more detail than any description of methods or results in journal article provides. For instance, many experimental designs allow for a large variety of statistical tests. If this flexibility remains hidden (e.g., if the original study design or data are not published) and undisclosed, researchers have an immense opportunities to fish for all kinds of statistical effects (see Simmons et al., 2011 for striking examples).

While reproducibility is simpler than replicability and may seem straightforward to implement, it was often thwarted by a lack of availability of the data or computer code that was used in the original data analyses.

Today, there are no excuses left: Free software tools and online repositories. However, this does not mean that all problems are solved. For instance, malicious researchers could still cheat on some or all steps along the way and only pretend to be transparent. Ultimately, we will always need some trust, and solid checks and balances in the system (e.g., peer review, independent replications).

Peng & Hicks (2020) mention another positive side effect:

  • readers also get the data and the computer code, both of which are valuable to the extent that they can be reused for future research.

If the analysis was conducted in R, this is an excellent opportunity to learn R and new methods of data analysis.

1.3.4 R Markdown

Using R Markdown (which is developed and supported by the RStudio.

For instructions on combining text and code, see


Neth, H. (2022a). Data science for psychologists. Social Psychology; Decision Sciences, University of Konstanz. https://bookdown.org/hneth/ds4psy/
Peng, R. D. (2011). Reproducible research in computational science. Science, 334(6060), 1226–1227. https://doi.org/10.1126/science.1213847
Peng, R. D., & Hicks, S. C. (2020). Reproducible research: A retrospective. arXiv Preprint arXiv:2007.12210. https://arxiv.org/abs/2007.12210
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. https://doi.org/10.1177/0956797611417632
Wickham, H., & Grolemund, G. (2017). R for data science: Import, tidy, transform, visualize, and model data. O’Reilly Media, Inc. http://r4ds.had.co.nz