Instruction:26f898cb-a1e1-485c-b3d3-e7cc564d6dac

Instructions

What is the evidence for reproducibility in different epistemic contexts?

Instructions for:TraineeTrainer

Related Initiative

TIER2

Goal

Conceptions, roles & pertinence of reproducibility are closely linked to different principal modes of knowledge production within & across research fields, so-called ‘epistemic contexts’. TIER2 mapped the contextual factors to various dimensions of ‘reproducibility’, as well as pertinent framework conditions (e.g., political, ethical, social, legal) that may affect the uptake of reproducibility practices in qualitative and machine learning fields. The literature reviews enabled analyses of diverse conceptions, roles & barriers of reproducibility & permits identification of relevant & targeted tools.

Duration (hours)

1

For whom is this important?

Academic institutionsAll stakeholders in researchFundersPublishersresearchersreserach integrity offices

Steps

1Reproducibility in Machine Learning-based Research: Overview, Barriers and Drivers

2Reproducibility and replicability of qualitative research: an integrative review of concepts, barriers and enablers

1

Reproducibility in Machine Learning-based Research: Overview, Barriers and Drivers

"Many research fields are currently reckoning with issues of poor levels of reproducibility. Some label it a "crisis", and research employing or building Machine Learning (ML) models is no exception. Issues including lack of transparency, data or code, poor adherence to standards, and the sensitivity of ML training conditions mean that many papers are not even reproducible in principle. Where they are, though, reproducibility experiments have found worryingly low degrees of similarity with original results. Despite previous appeals from ML researchers on this topic and various initiatives from conference reproducibility tracks to the ACM's new Emerging Interest Group on Reproducibility and Replicability, we contend that the general community continues to take this issue too lightly. Poor reproducibility threatens trust in and integrity of research results. Therefore, in this article, we lay out a new perspective on the key barriers and drivers (both procedural and technical) to increased reproducibility at various levels (methods, code, data, and experiments). We then map the drivers to the barriers to give concrete advice for strategies for researchers to mitigate reproducibility issues in their own work, to lay out key areas where further research is needed in specific areas, and to further ignite discussion on the threat presented by these urgent issues."

Find the full paper here: Reproducibility in Machine Learning-based Research: Overview, Barriers and Drivers

References:

Semmelrock, H., Ross-Hellauer, T., Kopeinik, S., Theiler, D., Haberl, A., Thalmann, S., & Kowald, D. (2024). Reproducibility in Machine Learning-based Research: Overview, Barriers and Drivers. arXiv preprint arXiv:2406.14325.

2

Reproducibility and replicability of qualitative research: an integrative review of concepts, barriers and enablers

"The integrative review presented here examines how reproducibility and replicability are conceptualized and discussed in relation to qualitative research, and which factors and practices enable or undermine them. Both peer-reviewed and grey English-language literature that address reproducibility and/or Open Science in relation to qualitative research were eligible for inclusion. Initial searches were conducted in Scopus, Web of Science, Dimensions, PubMed, APA PsychInfo, and JSTOR, and followed with snowball sampling from included literature. Studies were screened and both quantitative and qualitative data were extracted using the SyRF online platform, with 248 papers included. We found that conceptualizations that stem from quantitative standpoints are overwhelmingly framed as inappropriate practices and epistemic criteria for (most) qualitative research. When conceptualized in alternative ways that are adapted to the epistemic conditions, aims and practices of qualitative research, they can be both applicable and appropriate. Key barriers include the ontological and epistemological misalignment of reproducibility, replicability and Open Science and qualitative research, and ethical and practical concerns surrounding data sharing and reuse. Key enablers include practices that respond to ethical and practical concerns around data sharing and reuse (anonymization, ethical consent practices, context documentation, and ethical access management), adapting expectations and norms of openness, and established qualitative practices including documentation, reflexivity, and considering positionality. We conclude that reproducibility, replicability and Open Science practices must be adapted to the aims and epistemic conditions of qualitative research for them to be applicable and feasible, and that they will not always be both for all qualitative research."

Find the full paper here: MetaArXiv Preprints - Reproducibility and replicability of qualitative research: an integrative review of concepts, barriers and enablers.

Reference:

Cole, N. L., Ulpts, S., Bochynska, A., Kormann, E., Good, M., Leitner, B., & Ross-Hellauer, T. (2024, December 23). Reproducibility and replicability of qualitative research: an integrative review of concepts, barriers and enablers. https://doi.org/10.31222/osf.io/n5zkw_v1

Steps

Other information