Text (Instruction Step Text)
From The Embassy of Good Science
Describe the actions the user should take to experience the material (including preparation and follow up if any). Write in an active way.
- ⧼SA Foundation Data Type⧽: Text
3
<span lang="EN-US">In this pilot, [https://schema-lab.hypatia-comp.athenarc.gr/ SCHEMA] was developed as an open-source framework, comprising SCHEMA api for programmatic execution and SCHEMA lab for a user-friendly web interface to support reproducible computational research through containerized execution, metadata capturing and experiment management. It provides a scalable environment that enables researchers to design and run reproducible computational workflows and experiments.</span> +
In this lecture, Rosemarie Bernabe addresses the ethical challenges associated with implementing Open Science in practice. The lecture provides an overview through various examples of ethical and integrity issues encountered in Open Science practice. These include the risk of spreading misinformation, the emergence of new biases related to alternative metrics, among other concerns.
'''Watch the lecture and then answer the questions.'''
'''Further reading:'''
ROSiE General Guidelines on Responsible Open Science. https://doi.org/10.5281/zenodo.10046520
UNESCO (2023). Open science outlook 1: status and trends around the world. https://doi.org/10.54677/GIIC6829
Kingsley, D. (2025, March 30). Show your working: How the ‘open science’ movement tackles scientific misconduct. The Conversation. http://theconversation.com/show-your-working-how-the-open-science-movement-tackles-scientific-misconduct-249020
Düwell, M. (2019). Open science and ethics. Ethical Theory and Moral Practice, 22(5), 1051-1053. https://doi.org/10.1007/s10677-019-10053-3
Lindemann, T., & Häberlein, L. (2023). Contours of a research ethics and integrity perspective on open science. Frontiers in Research Metrics and Analytics, 8, 1052353. https://doi.org/10.3389/frma.2023.1052353
Laine, H. (2018). Open science and codes of conduct on research integrity. Informaatiotutkimus, 37(4). https://doi.org/10.23978/inf.77414 +
<span lang="EN-US">The primer initiative from the United Kingdom Reproducibility Network was designed introduce a broad audience to important topics and aspects of open and reproducible scholarship. Each primer is a 3-to-4-page summary document which describes the open or reproducible research practice, why one might be interested in it, and how they can be applied or adopted with thorough guidance.</span>
More information on the Primers Initative can be found here: <u>[https://www.ukrn.org/primers/ <span lang="EN-US">https://www.ukrn.org/primers/</span>]</u> . +
Introduction to the evaluation of the effectiveness of Research Ethics and Integrity (REI) training +
REI training effectiveness should be approached as a system. While measuring reactions and knowledge (or learning process) is quite simple, it is also important to measure common practices and the environment (impact on the institutions). Training outcomes are influenced by many factors, not least the match between perceived needs and what the training offers, that is, the alignment between needs, intended learning outcomes, training objectives, and training implementation.
To get a holistic picture (i.e. to triangulate measurement) of the effectiveness of training it is recommended to combine different measurement tools, use them at various measurement points and align them with a common taxonomy. It is important to consider what to do with the results – what kind of changes are necessary to improve teaching and/or the environment to build a culture of integrity? +
Doing research with communities affected by climate change: Climate-conscious methodologies matrix (for students and citizen scientists) +
The climate conscious methodology matrix helps researchers, students, and citizen scientists make ethical and responsible decisions in research. It is especially useful when working with climate-related impacts (such as floods, heatwaves, or droughts), with vulnerable or affected communities.
The tool encourages you to '''ask good questions''' throughout all stages of your research (from the early design to communication of results) and helps you adapt your methods while keeping fairness, safety, and community needs at the centre.
In a changing climate, traditional research methods may not work or may cause harm. Communities may be under stress. Researchers may face risks. By using the matrix, you can:
*'''Stay flexible''', but still do high-quality research
*'''Respect local people and ecosystems'''
*'''Build trust''' through careful planning and honest communication
*'''Support climate justice''' by including affected voices in all stages of research.
In the following steps, five card deks are presented each of them addresses a different methodological component:
* Assessment and measurement strategies
* Data evaluation and interpretation
* Ethical issues and scientific integrity
* Communication of research findings
Have a look at how to use the cards by clicking on the link below. +
Taxonomies are usually implemented in learning situations. But in some contexts, we may need to evaluate ethical awareness/sensitivity outside the learning context, for example when analysing comments that people have added as open answers to vignettes in retention checks or REI surveys.
We needed to modify the instrument for measuring ethical sensitivity in a non-training context (see Table 2). The instrument can be used in deductive content analysis.
Table 2. Ethical Awareness and Sensitivity Meter (Tammeleht et al., forthcoming).
[[File:Img6.png|center|frameless|500x500px]]
By ethical awareness we mean recognition of ethical aspects, interpreting a situation. In the ethical sensitivity we include awareness, conscious scrutiny of an ethical topic, and recognising intricacies of the issues/participants. +
Theories of normative ethics can be used to analyse participants’ responses to a variety of learning tasks such as analyses of cases or essays, in which participants describe their own approaches to an ethical question or their thinking about ethical matters. If familiarity of ethical approaches or ethical theories are part of the course objectives or intended learning outcomes of the training, an analysis of how the learner has addressed these is very suitable.
For example, the analysis of authentic responses to learning tasks can involve the identification if deontology, virtue, utilitarian ethics, or other approaches, as relevant. The Virt2ue project materials on the Embassy of Good Science [[Instruction:17705907-d9b2-4f33-bc4f-088d84b4d971|Preparatory Viewing: Introduction to Concepts & Themes (embassy.science)]] provides helpful guidance (e.g. a video) into understanding common ethics theories.
The analysis takes a bit of time but may yield interesting information about how learners have understood the concepts. The analysis can focus on the presence of the theoretical concepts in a text, the depth at which the learner uses the theoretical knowledge, or the levels of understanding that the learner displays regarding pertinent ethical theories or approaches. The depth of thinking, which the learner displays, can be analysed with a scheme of levels of reflection or the SOLO taxonomy / ECAG Grid.
[[File:Img11.png|center|frameless|600x600px]]
Figure 3. Example from a case study analysis done by 7 people displaying ethical approaches (visual from MaxQDA programme).
Figure 3 displays ethical approaches displayed in the case analysis and the level of understanding (SOLO taxonomy): the most prevalent in this group is the rule-based approach, but consequentialism is also quite common. SOLO levels indicate that the level of understanding was mostly on the relational and extended abstract level. +
Focusing and attention are important in obtaining new skills and competencies, the same applies to ethics and integrity. Monitoring physiological markers in the context of ethics training has not been researched much. In the context of education there are examples of studying engagement and heart-rate, but eye-tracking is novel.
Eye-tracking/gaze-tracking may have the potential of revealing new insights about how learners process information for moral judgment in learning situations. Gaze-tracking collects a very large number of data points and thus allows within-person comparisons through focusing on events that are similar and frequent. This enables the analysis of how an individual (re)acts across these situations (Kirkpatrick’s level 2). The data can be viewed for example through heat maps.
To facilitate the analysis, visual markers (QR codes) can be used to locate and synchronise the data. The latest SeeTrue eye tracking device model we have used has a software component to recognize code markers. The gaze tracking device will recognize markers as they appear depending on where the person wearing the device is looking at. Thus, with an additional software component developed by researchers at the University of Helsinki, it has been possible to consolidate the gaze points coordinates of various markers into a unified pair of coordinates, allowing tracking the gaze of the person wearing the device on an idealized static version of the scene which contains the areas of interest, in our case posters on ethical principles and ethics theories (Figure 13a).
Eye-tracking data are analysed statistically and displayed often in density plots or heat-maps (Figure 13b). With these plots we can see what the salient areas of interest observed by a subject during a given time interval in the discussion taking place during an episode happen to be. A sensible temporal analysis of the gaze patterns and scan paths is also possible once this representation is available. An initial analysis suggests that the participant whose gaze is visualized in Figure 13b has focused especially on the ethical principle of beneficence and virtue ethics theory. This information can then be considered in light of the case that the participants were discussing, as well as under prior and subsequent arguments presented by the participant. If understanding ethical principles and ethics theory is an intended learning outcome of the training, then, the way in which learners apply the theories and principles can indicate how effectively the training conveys its content. It is noteworthy that the information in the poster boards should correspond with content related to the intended learning outcomes of the training for this method to be used in assessing training effectiveness.
The analysis is time-consuming and would not be used for large-scale measurement of learning. Specific equipment and software are necessary for collecting and analysing the information. If the facilitator has access to these tools, this measurement gives interesting insights about the learning process. Ethical considerations must be considered: while the equipment is easy to remove, it nevertheless attaches to the body (Hannula et al., 2022).
[[File:Img17.png|center|frameless|500x500px]]
Figure 13. a) to the left code recognition (with fisheye distortion correction using our own software) and b) to the right density plot (heat map) of the gaze point locations during a short interval.
The tool may be most suitable for students and early-career researchers as the target group of training and in case of small groups.
Please match the key terms related to the Crisis Tree with their descriptions +
<span lang="en-DE">After exploring the image, consider the following guided reflection prompts</span> +
<span lang="EN-US">Choose the cases you would like to play. The groups can play either the same cases (cases can then be projected on slides) or different ones (cases can be projected by expanding the window below or printed handouts).</span><span lang="EN-US">The cases for handouts can also be found [https://zenodo.org/records/17733497 here].</span>
*Prepare the cases to be shared with the players (print them, put them on slides or other visuals to be used in the activity).
*<span lang="EN-US">Decide whether to hold joint discussions about particular cases across groups or have each group discuss separately.</span>
<div><div><div><span name="_msocom_1"></span></div></div></div> +
Look at slide 8-12 (in step 1) and ask participants to go over the REI leadership principles and take a role of a REI leader.
After having explained what the activity is about you may ask the participants to turn to the posters and work independently. Or you can go over the case analysis as a group following tasks on the slides (both slides and posters include same tasks). The tasks involve: Read the case. The case has a mentor-mentee situation and a dilemma for the leader/supervisor to deal with.
*Identify the ethical principles that may be at stake in this case.
*Use the ethical analysis to solve the situation described in the case.
*Analyse which ethical approaches the possible courses of action follow.
*Reflect back at the REI leadership framework – the group should discuss which REI leadership principles would contribute to solving this case and reflect if they displayed any of the principles. +
<span lang="EN-US">Inspired by the examples shared in the podcast, this task invites you to slow down and imagine a small ritual that embodies care for something within your research environment (e.g., water, soil, lab equipment, participants, data, community).</span>
<span lang="EN-US">Reflect and respond to the prompts below:</span> +
<span lang="EN-US">Now that you have identified possible blind spots and challenges in your research, we invite you to use storytelling to craft an original short story. By doing so, you can reflect on hidden assumptions, neglected impacts, silenced voices, and power relations within your own research context or field.</span> <span lang="EN-US">Storytelling helps us see what academic prose often hides.</span> +
<span lang="EN-US">In the two steps above, we have learned about the PEPE framework.</span>
*'''Pluralizing''' invites researchers to <span lang="EN-GB">go beyond the superficial combining of different perspectives that's inattentive to power and to genuinely reflect on the question: how to attend different knowledge forms and navigate that plurality of different perspectives?</span>
*'''Empowering''' <span lang="EN-GB">tries to address the fact that in research we often communicate dry data or facts and publish them, but we don't attend to all of the emotions and perspectives that play a critical role in how we do research and the importance of ''whose'' voice matters.</span>
*'''Politicizing''' <span lang="EN-GB">that attends to the issue of power. We all have very different assumptions about how research might challenge power structures. And that could be more of a scholar activist perspective, but it could also be deeply collaborative.</span>
*'''Embedding''' <span lang="EN-GB">addresses the fact that we're operating in systems that are not necessarily always supportive of what we're doing. H</span><span lang="EN-US">ow do we move beyond temporary experiments to lasting institutional change?</span> +
Fieldwork can have several environmental consequences related to energy use, habitat disturbance, waste production and methods of sampling. Practicing safe, inclusive, and equitable fieldwork can help reduce these risks and costs. In this session, you will learn about various environmental impacts of research fieldwork and explore practical strategies to reduce their impacts.
Watch the video on “Minimising the Environmental Impact of Research Fieldwork” which provides guidelines for mitigating environmental impacts of research fieldwork. +
Take a piece of paper a draw the good researcher. What would characterise a good researcher, scientist, or innovator? What is in their hands, on their mind, in/on their clothes, or their equipment? +
[[File:Ge2Image3.png|center|frameless|600x600px]]
What do you think are the main ethics issues for this case? Some of the main ones are listed below. Sort each issue into one of the four categories: Autonomy, Future generations;Potential harms and benefits;and Slippery slope. +
In order to monitor how ethics competencies develop during the training session and also see if the competencies are retained over a longer period of time reflective learning diaries can be used to monitor the development of REI competencies as well as measure the effectiveness of the training in the long term (Kirkpatrick’s levels 2 and 3). Reflection is a crucial part of ethics education as it supports the development of ethical sensitivity and ethical decision-making (Mustajoki and Mustajoki, 2017;Löfström and Tammeleht, 2023). Written reflection tasks may provide good results as writing offers a chance to pause and have an inner dialogue with oneself.
Implementation is relatively easy, but analysis may take some time. There are different ways to elicit learner’s texts – from pen-and-paper format to individual digital diaries (on various platforms) as well as forum format (where learners can see and respond to each other’s entries). Forum format has provided the best results (Tammeleht et al., 2024). Diaries should be kept during a longer period of time (at least a few months) and include weekly or bimonthly submissions. Some guiding questions or topics should be provided for each entry – this helps to keep focus and guide discussion.
Various instruments can be used for analysis: the SOLO taxonomy to evaluate the level of understanding, reflection levels to see the advancement of reflective skills, content criteria (ethical principles, ethical analysis, ethical approaches) or specific topics relevant for the course content.
There are various ethical aspects to consider when collecting and analysing learning diaries (Thorpe, 2004;Gibbs et al., 2007). Learners can be asked if they would prefer individual or forum format diary-keeping. Depending on the level of trust in the group, forum format may be uncomfortable for some learners. In HE context learners are adults, who take responsibility for writing the entries on time and honestly. This may call for keeping the content confidential unless agreed otherwise
The tool is suitable for students and ECRs. Supervisors may find it challenging to keep a diary in addition to their regular work-load.
<div>
<span lang="EN-GB">Improving reproducibility is a multifaceted challenge requiring both behavioural and cultural change.</span> <span lang="EN-US">The adoption of reproducibility practices has been sparked and embraced by the Open Science movement. However, a lot of researchers are not fully aware of the implications of reproducibility and how Open Science and reproducibility are connected and intertwined (Haven et al., 2022). To increase awareness and change research practices several steps should be taken (Nosek, 2019). First, the infrastructure for the desired behaviour should be provided to make it possible. Second, the user interface and experience of the infrastructure should be improved to make the behaviour easy. Third, communities of practice should be fostered to make the behaviour visible and so increasingly normative. Fourth, incentives to enact the behaviour should be provided to make it rewarding. Last, policies should be enacted to make the behaviour required (Nosek, 2019). To further this work, we sought to explore the future of reproducibility for different stakeholders and question what should be the next steps for reproducibility and how diverse epistemic contexts can adopt reproducibility in different forms. In this deliverable, we aim to add nuance to the reproducibility debate through flexible investigation of diverse epistemic contexts (researchers from the field of machine learning and researchers working with qualitative methods), exploring the future of reproducibility through the lens of diverse research stakeholders – researchers, funders, and publishers.</span>
</div><div>
<span lang="EN-US">In this context, we look to the future of reproducibility by exploring the preferred scenarios for multiple stakeholders, including how these scenarios can be realized. We reflect on the steps that are necessary for adherence to reproducibility-enabling practices and what different epistemic contexts need to make reproducibility a priority. Lastly, we reflect on what are the new problems that we may face when aiming to improve reproducibility. We believe exploring the possible futures for reproducibility is essential to discover the next steps for different members of the scientific community to take to realize the preferred future and the actions to avoid steering away from the dystopian futures.</span>
</div><div>
<span lang="EN-US">We aim to highlight the essential role of institutions, funders and publishers in this endeavor to make reproducibility a priority by recognizing, rewarding, evaluating and monitoring reproducibility. Ultimately, we hope to steer and move forward the debate on reproducibility in the research community by</span> <span lang="EN-GB">addressing a set of core research questions related to how key stakeholders in the academic community envision the way in which matters of reproducibility should be addressed in the future. More specifically, it asks representatives from research funders, scholarly publishers, and researchers from diverse disciplinary backgrounds:</span>
</div><div>
<span lang="EN-GB">1. What are the preferred futures of reproducibility?</span>
</div><div>
<span lang="EN-GB">2. What are the enablers and barriers on the way to the preferred future or reproducibility more generally?</span>
</div><div>
Below we present the results of our study below.
</div>
For more information, please refer to the full paper: [https://osf.io/preprints/metaarxiv/gx9jq_v1 MetaArXiv Preprints - How to get there from here? Barriers and enablers on the road towards reproducibility in research].
