What is reproducibility?
InstructionsWhat is reproducibility?
Enabling and Redoing
The research team within TIER2 present an analytical framework that supports epistemic diversity by examining the potential relevance and degree of feasibility of reproducibility for different modes of knowledge production. The research team find current general typologies with the same aim wanting. They propose top-down derived enumerative lists of kinds of reproducibility organised according to vaguely defined fields, disciplines, methods or so-called research types. Current typologies cannot sufficiently characterise different kinds of research and their varying research context at the granularity needed to deal with how epistemic diversity and reproducibility relate. They also do not clarify the prevailing conceptual confusion surrounding reproducibility and replication. To clarify matters, we propose redoing to commonly describe the acts of reproducing and replicating and enabling to describe the acts of making something reproducible and replicable. We suggest mapping practices and epistemic functions to characterise what parts of a study should be redone or enabled and for what intended purposes. We propose knowledge production modes (KPM) as an organising construct to situate redoing and enabling within knowledge production’s epistemic, social, and contextual conditions. Epistemologies determine epistemic norms and criteria. Social conditions influence how research is organised, practised, rewarded, reported, and discussed. Contextual conditions put boundaries and restrictions on research, for example, due to subject matter, environment, availability of resources, and technologies, which are the ‘local’ conditions. Our framework clarifies the potential relevance of redoing and the degree of feasibility of redoing and enabling for a specific knowledge production mode. Relevance comprises research goals and epistemology. Epistemology is the basic assumption behind knowledge production modes. It determines how knowledge claims are produced and justified with systems of justification, the criteria for good/trustworthy research, and, thus, the epistemic norms. Different ways of knowing have different epistemic norms, practices, and criteria. Feasibility comprises the nature and complexity of the subject under investigation, the necessary investment for redoing or enabling, and the degree of theoretical and methodological uncertainty associated with the actual research. The proposed framework works bottom-up in that knowledge production modes are not defined a priori but derived from the analytical framework. The framework, therefore, supports epistemic diversity by being open and non-hierarchical and working at a sufficient level of granularity to discern the diverse conditions of knowledge production. The research team propose a framework that can clarify, not a cookbook. Enabling in some form always seems relevant in empirical work irrespective of ways of knowing—the same is not true for redoing.
For the full paper click here: MetaArXiv Preprints - Knowledge Production Modes: The Relevance and Feasibility of Reproducibility
Reference
Ulpts, S., & Schneider, J. W. (2023, September 25). Knowledge Production Modes: The Relevance and Feasibility of Reproducibility. https://doi.org/10.31222/osf.io/ujnd9Defining Reproducibility
Defining reproducibility and replicability, has been a challenge in the research community, as different interpretations and even contradicting definitions are often used. Defining these terms has proven to be challenging as their use and understanding differs between fields of research. However, the European funded iRise consortium developed a reproducibility glossary by critically reviewing existing scientific literature. The glossary provides working definitions for the use of terms reproducibility, replicability and replication, as well as related concepts.
References
Voelkl, B., Heyard, R., Fanelli, D., Wever, K., Held, L., Würbel, H., Zellers, S., & Maniadis, Z. (2024). Glossary of common terminology resulting from scoping reviews. https://osf.io/ewybt.Futures of Reproducibility
Improving reproducibility is a multifaceted challenge requiring both behavioural and cultural change. The adoption of reproducibility practices has been sparked and embraced by the Open Science movement. However, a lot of researchers are not fully aware of the implications of reproducibility and how Open Science and reproducibility are connected and intertwined (Haven et al., 2022). To increase awareness and change research practices several steps should be taken (Nosek, 2019). First, the infrastructure for the desired behaviour should be provided to make it possible. Second, the user interface and experience of the infrastructure should be improved to make the behaviour easy. Third, communities of practice should be fostered to make the behaviour visible and so increasingly normative. Fourth, incentives to enact the behaviour should be provided to make it rewarding. Last, policies should be enacted to make the behaviour required (Nosek, 2019). To further this work, we sought to explore the future of reproducibility for different stakeholders and question what should be the next steps for reproducibility and how diverse epistemic contexts can adopt reproducibility in different forms. In this deliverable, we aim to add nuance to the reproducibility debate through flexible investigation of diverse epistemic contexts (researchers from the field of machine learning and researchers working with qualitative methods), exploring the future of reproducibility through the lens of diverse research stakeholders – researchers, funders, and publishers.
In this context, we look to the future of reproducibility by exploring the preferred scenarios for multiple stakeholders, including how these scenarios can be realized. We reflect on the steps that are necessary for adherence to reproducibility-enabling practices and what different epistemic contexts need to make reproducibility a priority. Lastly, we reflect on what are the new problems that we may face when aiming to improve reproducibility. We believe exploring the possible futures for reproducibility is essential to discover the next steps for different members of the scientific community to take to realize the preferred future and the actions to avoid steering away from the dystopian futures.
We aim to highlight the essential role of institutions, funders and publishers in this endeavor to make reproducibility a priority by recognizing, rewarding, evaluating and monitoring reproducibility. Ultimately, we hope to steer and move forward the debate on reproducibility in the research community by addressing a set of core research questions related to how key stakeholders in the academic community envision the way in which matters of reproducibility should be addressed in the future. More specifically, it asks representatives from research funders, scholarly publishers, and researchers from diverse disciplinary backgrounds:
1. What are the preferred futures of reproducibility?
2. What are the enablers and barriers on the way to the preferred future or reproducibility more generally?
Below we present the results of our study below.
