Reproducibility
InstructionsReproducibility
Enabling and Redoing
x
Defining Reproducibility
Defining reproducibility and replicability, has been a challenge in the research community, as different interpretations and even contradicting definitions are often used. Defining these terms has proven to be challenging as their use and understanding differs between fields of research. However, the European funded iRise consortium developed a reproducibility glossary by critically reviewing existing scientific literature. The glossary provides working definitions for the use of terms reproducibility, replicability and replication, as well as related concepts.
References
Voelkl, B., Heyard, R., Fanelli, D., Wever, K., Held, L., Würbel, H., Zellers, S., & Maniadis, Z. (2024). Glossary of common terminology resulting from scoping reviews. https://osf.io/ewybt.Futures of Reproducibility
Improving reproducibility is a multifaceted challenge requiring both behavioural and cultural change. The adoption of reproducibility practices has been sparked and embraced by the Open Science movement. However, a lot of researchers are not fully aware of the implications of reproducibility and how Open Science and reproducibility are connected and intertwined (Haven et al., 2022). To increase awareness and change research practices several steps should be taken (Nosek, 2019). First, the infrastructure for the desired behaviour should be provided to make it possible. Second, the user interface and experience of the infrastructure should be improved to make the behaviour easy. Third, communities of practice should be fostered to make the behaviour visible and so increasingly normative. Fourth, incentives to enact the behaviour should be provided to make it rewarding. Last, policies should be enacted to make the behaviour required (Nosek, 2019). To further this work, we sought to explore the future of reproducibility for different stakeholders and question what should be the next steps for reproducibility and how diverse epistemic contexts can adopt reproducibility in different forms. In this deliverable, we aim to add nuance to the reproducibility debate through flexible investigation of diverse epistemic contexts (researchers from the field of machine learning and researchers working with qualitative methods), exploring the future of reproducibility through the lens of diverse research stakeholders – researchers, funders, and publishers.
In this context, we look to the future of reproducibility by exploring the preferred scenarios for multiple stakeholders, including how these scenarios can be realized. We reflect on the steps that are necessary for adherence to reproducibility-enabling practices and what different epistemic contexts need to make reproducibility a priority. Lastly, we reflect on what are the new problems that we may face when aiming to improve reproducibility. We believe exploring the possible futures for reproducibility is essential to discover the next steps for different members of the scientific community to take to realize the preferred future and the actions to avoid steering away from the dystopian futures.
We aim to highlight the essential role of institutions, funders and publishers in this endeavor to make reproducibility a priority by recognizing, rewarding, evaluating and monitoring reproducibility. Ultimately, we hope to steer and move forward the debate on reproducibility in the research community by addressing a set of core research questions related to how key stakeholders in the academic community envision the way in which matters of reproducibility should be addressed in the future. More specifically, it asks representatives from research funders, scholarly publishers, and researchers from diverse disciplinary backgrounds:
1. What are the preferred futures of reproducibility?
2. What are the enablers and barriers on the way to the preferred future or reproducibility more generally?
Below we present the results of our study below.
