View source for Instruction:62bee3f2-de7c-4bc6-ae88-fe8b93f5d1f6

{{Instruction
|Title=What is reproducibility?
|Has Related Initiative=Initiative:286109fc-03cb-4a08-bd45-c0276eae3079
|Instruction Goal=This module provides an in-depth introduction to reproducibility through the work of TIER2 and iRise. The purpose and concept of reproducibility is introduced, working definitions are provided for important concepts in relation to reproducibility and replicability, and the futures of reproducibility and illustrated from the perspective of different stakeholders.
|Has Duration=2
|Important For=All stakeholders in research; Funders; Publishers; Researchers
}}
{{Custom TabContent Trainee Open}}
{{Instruction Steps Foldout Trainee}}
{{Instruction Perspective Trainee}}
{{Instruction Step Trainee
|Instruction Step Title=Enabling and Redoing
|Instruction Step Text=<span lang="EN-GB">The research team within TIER2 present an analytical framework that supports epistemic diversity by examining the potential relevance and degree of feasibility of reproducibility for different modes of knowledge production. The research team find current general typologies with the same aim wanting. They propose top-down derived enumerative lists of kinds of reproducibility organised according to vaguely defined fields, disciplines, methods or so-called research types. Current typologies cannot sufficiently characterise different kinds of research and their varying research context at the granularity needed to deal with how epistemic diversity and reproducibility relate. They also do not clarify the prevailing conceptual confusion surrounding reproducibility and replication. To clarify matters, we propose ''redoing'' to commonly describe the acts of reproducing and replicating and ''enabling'' to describe the acts of making something reproducible and replicable. We suggest mapping practices and epistemic functions to characterise what parts of a study should be redone or enabled and for what intended purposes. We propose knowledge production modes (KPM) as an organising construct to situate redoing and enabling within knowledge production’s epistemic, social, and contextual conditions. Epistemologies determine epistemic norms and criteria. Social conditions influence how research is organised, practised, rewarded, reported, and discussed. Contextual conditions put boundaries and restrictions on research, for example, due to subject matter, environment, availability of resources, and technologies, which are the ‘local’ conditions. Our framework clarifies the potential ''relevance'' of redoing and the degree of ''feasibility'' of redoing and enabling for a specific knowledge production mode. Relevance comprises research goals and epistemology. Epistemology is the basic assumption behind knowledge production modes. It determines how knowledge claims are produced and justified with systems of justification, the criteria for good/trustworthy research, and, thus, the epistemic norms. Different ways of knowing have different epistemic norms, practices, and criteria. Feasibility comprises the nature and complexity of the subject under investigation, the necessary investment for redoing or enabling, and the degree of theoretical and methodological uncertainty associated with the actual research. The proposed framework works bottom-up in that knowledge production modes are not defined a priori but derived from the analytical framework. The framework, therefore, supports epistemic diversity by being open and non-hierarchical and working at a sufficient level of granularity to discern the diverse conditions of knowledge production. The research team propose a framework that can clarify, not a cookbook. Enabling in some form always seems relevant in empirical work irrespective of ways of knowing—the same is not true for redoing.</span> 

For the full paper click here: [https://osf.io/preprints/metaarxiv/ujnd9_v1 MetaArXiv Preprints - Knowledge Production Modes: The Relevance and Feasibility of Reproducibility]

Reference

Ulpts, S., & Schneider, J. W. (2023, September 25). Knowledge Production Modes: The Relevance and Feasibility of Reproducibility. <nowiki>https://doi.org/10.31222/osf.io/ujnd9</nowiki>
}}
{{Instruction Step Trainee
|Instruction Step Title=Defining Reproducibility
|Instruction Step Text=Defining reproducibility and replicability, has been a challenge in the research community, as different interpretations and even contradicting definitions are often used. Defining these terms has proven to be challenging as their use and understanding differs between fields of research. However, the European funded iRise consortium developed a reproducibility glossary  by critically reviewing existing scientific literature. The glossary provides working definitions for the use of terms reproducibility, replicability and replication, as well as related concepts.

'''References'''

Voelkl, B., Heyard, R., Fanelli, D., Wever, K., Held, L., Würbel, H., Zellers, S., & Maniadis, Z. (2024). Glossary of common terminology resulting from scoping reviews. https://osf.io/ewybt.
}}
{{Instruction Step Trainee
|Instruction Step Title=Futures of Reproducibility
|Instruction Step Text=<div>
<span lang="EN-GB">Improving reproducibility is a multifaceted challenge requiring both behavioural and cultural change.</span> <span lang="EN-US">The adoption of reproducibility practices has been sparked and embraced by the Open Science movement. However, a lot of researchers are not fully aware of the implications of reproducibility and how Open Science and reproducibility are connected and intertwined (Haven et al., 2022). To increase awareness and change research practices several steps should be taken (Nosek, 2019). First, the infrastructure for the desired behaviour should be provided to make it possible. Second, the user interface and experience of the infrastructure should be improved to make the behaviour easy. Third, communities of practice should be fostered to make the behaviour visible and so increasingly normative.  Fourth, incentives to enact the behaviour should be provided to make it rewarding. Last, policies should be enacted to make the behaviour required (Nosek, 2019).  To further this work, we sought to explore the future of reproducibility for different stakeholders and question what should be the next steps for reproducibility and how diverse epistemic contexts can adopt reproducibility in different forms. In this deliverable, we aim to add nuance to the reproducibility debate through flexible investigation of diverse epistemic contexts (researchers from the field of machine learning and researchers working with qualitative methods), exploring the future of reproducibility through the lens of diverse research stakeholders – researchers, funders, and publishers.</span>  
</div><div>
<span lang="EN-US">In this context, we look to the future of reproducibility by exploring the preferred scenarios for multiple stakeholders, including how these scenarios can be realized. We reflect on the steps that are necessary for adherence to reproducibility-enabling practices and what different epistemic contexts need to make reproducibility a priority. Lastly, we reflect on what are the new problems that we may face when aiming to improve reproducibility. We believe exploring the possible futures for reproducibility is essential to discover the next steps for different members of the scientific community to take to realize the preferred future and the actions to avoid steering away from the dystopian futures.</span>  
</div><div>
<span lang="EN-US">We aim to highlight the essential role of institutions, funders and publishers in this endeavor to make reproducibility a priority by recognizing, rewarding, evaluating and monitoring reproducibility. Ultimately, we hope to steer and move forward the debate on reproducibility in the research community by</span> <span lang="EN-GB">addressing a set of core research questions related to how key stakeholders in the academic community envision the way in which matters of reproducibility should be addressed in the future. More specifically, it asks representatives from research funders, scholarly publishers, and researchers from diverse disciplinary backgrounds:</span> 
</div><div>
<span lang="EN-GB">1. What are the preferred futures of reproducibility?</span>  
</div><div>
<span lang="EN-GB">2. What are the enablers and barriers on the way to the preferred future or reproducibility more generally?</span>  
</div><div>
Below we present the results of our study below. 
</div>
}}
{{Instruction Remarks Trainee}}
{{Custom TabContent Close Trainee}}
{{Custom TabContent Trainer Open}}
{{Instruction Steps Foldout Trainer}}
{{Instruction Perspective Trainer}}
{{Instruction Remarks Trainer}}
{{Custom TabContent Close Trainer}}
{{Related To}}
{{Tags
|Involves=TIER2
|Has Good Practice And Misconduct=Reproducibility
}}