Text (Instruction Step Text)
From The Embassy of Good Science
Describe the actions the user should take to experience the material (including preparation and follow up if any). Write in an active way.
- ⧼SA Foundation Data Type⧽: Text
2
Open science and research ethics have in common their foundation on the universality of human rights. In full alignment with the [https://www.un.org/en/about-us/universal-declaration-of-human-rights Universal Declaration of Human Rights], open science assumes and serves the principle that “All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood.” Additionally, article 27 of the Universal Declaration states that “Everyone has the right freely to participate in the cultural life of the community, to enjoy the arts and to share in scientific advancement and its benefits”, as well as “Everyone has the right to the protection of the moral and material interests resulting from any scientific, literary or artistic production of which he is the author”.
The UNESCO Recommendation on Open Science identifies the core values of open science. These values, applicable to the entire scientific research process, include (1) quality and integrity, (2) collective benefit, (3) equity and fairness, and (4) diversity and inclusiveness (UNESCO, 2021). The UNESCO Recommendation also introduces guiding principles of open science, such as transparency, scrutiny, critique, reproducibility, equality of opportunities, responsibility, respect, accountability, collaboration, participation, inclusion, flexibility, and sustainability (UNESCO, 2021).
Open science practices are in principle reciprocal and symmetrical. Everyone contributes knowledge and data by making them openly accessible, and everyone can then use the knowledge and data for further research. Citizen and participatory science is part of open science as one of the types of engagement of social actors and is defined as "models of scientific research conducted by non-professional scientists, following scientifically valid methodologies and frequently carried out in association with formal, scientific programmes or with professional scientists with web-based platforms and social media, as well as open source hardware and software (especially low-cost sensors and mobile apps) as important agents of interaction." (UNESCO, 2021, p. 18-19)
'''References'''
#UNESCO (2021). Recommendation on Open Science. https://doi.org/10.54677/MNMH8546
In many cases participation in research does not pose '''risks to research participants''', for example, filling in an anonymous questionnaire usually is easy, and no risks are associated with it (nevertheless, sometimes sensitive questions may pose psychological risk). In some other types of research, participation can pose physical or psychological risks. For example, participants of biomedical research who are exposed to experimental treatments might face risks to physical well-being; research in psychology may lead to emotional distress among participants; studies dealing with sensitive information may raise risks for the privacy and confidentiality of participants; some research topics may be socially sensitive and research participants might face social consequences or stigma. In citizen science, sharing data sometimes might pose a privacy risk to the citizen scientists themselves. This might be a case when, for example, management of citizen science programs requires collecting private information about volunteers.
The '''rights and interests of research participants''' are arguably the cornerstone of research ethics and in the traditional research ethics setting there has been developed a certain framework of how these rights are applied in different fields of research. Citizen science however introduces some additional challenges that need to be addressed. Many citizen science projects are conducted outside traditional academic or commercial settings. This raises the issue of ethics oversight of these studies and whether citizen scientists have the necessary research ethics training.
Research involving human research participants is guided by various laws and ethical guidelines. These legal and ethical standards embody important ethical principles and requirements (Emanuel et al 2000; Resnik 2019):
*'''Social value''' means that to justify the participation of human subjects, research should be expected to yield results that can benefit society.
*Evaluation of the '''risk/benefit ratio''' means that risks posed by participation in a research study should be minimized and justified in terms of the potential benefits to the research participants and society.
*'''Informed consent''' means that research participants should receive adequate information about the planned research and their voluntary consent should be sought and appropriately documented.
*'''Confidentiality''' is required to protect personal data and privacy of research participants.
*'''Data safety''' monitoring means that research data should be adequately protected to avoid harming, e.g., stigmatizing research participants.
*'''Fair selection of subjects''' means that the selection of research participants should be based on sound scientific and ethical criteria.
*'''Protection of vulnerable subjects''' requires to ensure additional protections for research participants who may be vulnerable to coercion, exploitation, or harm.
*'''Independent review''' is a requirement applied to some types of research, e.g. biomedical research involving human research participants should be reviewed by an independent research ethics committee according to the national legal framework.
For a research study to be ethical, researchers, including citizen scientists, must comply with all the requirements and principles mentioned above. For example, poorly designed studies will not yield valuable results and therefore, the risks that research participants have been exposed to during the study will be unjustified. One of the suggested ways to avoid these problems is to closely collaborate with professional scientists who are experts in a particular field of research (Resnik 2019).
'''References'''
#Emanuel, E. J., Wendler, D., & Grady, C. (2000). What Makes Clinical Research Ethical? ''JAMA'', 283(20), 2701–2711. https://doi.org/10.1001/jama.283.20.2701
#Resnik, D. B. (2019). Citizen scientists as human subjects: Ethical issues. ''Citizen Science: Theory and Practice'', 4(1). https://doi.org/10.5334/cstp.150
Citizen science offers valuable opportunities for all stakeholders involved; however, it also raises new issues regarding research ethics and integrity. Some authors have expressed concerns regarding the potential '''exploitation and instrumentalization''' of citizen scientists, where their unpaid work is utilized without proper acknowledgment of their contributions (Resnik, 2019). Therefore, '''recognizing the contributions''' of citizen scientists in all phases of research especially in scientific publications is essential to acknowledge their valuable research inputs. In some cases, citizen scientists may qualify for co-authorship if they have made substantial intellectual contributions to the research, including contributions to study design, data analysis, manuscript writing, and agreement to be accountable for all aspects of the research ([https://bit.ly/N7uoq3 <span lang="EN-GB">ICMJE</span>]<span lang="EN-GB">). While traditional academic authorship criteria may not always directly apply to citizen scientists, there are various other ways to appropriately recognize their involvement. Citizen scientists who have contributed to the research but whose contributions do not justify authorship may be acknowledged as contributors, with their roles and specific tasks described in a contributorship statement or acknowledgments. Open and transparent communication with citizen scientists throughout the research process, involving them in discussions about authorship and recognition, is crucial for building trust and ensuring that everyone involved feels appropriately acknowledged for their contributions.</span>
Additionally, issues of '''data quality and ownership''' have been raised in the context of citizen science, as citizen scientists are often not specifically trained in research ethics and methodologies. The quality of data collected by citizen scientists can be ensured through various methods. Researchers can provide appropriate training to citizen scientists on data collection techniques and emphasize the importance of maintaining good research records. It is also crucial to ensure that the technological solutions chosen for citizen science projects are comprehensible and user-friendly, which can help minimize errors or misunderstandings during data collection and improve the overall quality of the collected data. Moreover, facilitating discussions between professional researchers and citizen scientists on questions of data ownership and future data accessibility is important to establish clear agreements on how the data will be used, shared, and accessed.
Citizen scientists should also be provided with information regarding research integrity to ensure ethical conduct. This includes informing them about potential financial and non-financial '''conflicts of interest''', such as relationships with organizations sponsoring research or personal interests (Resnik, 2019). Openly discussing the expectations and motivations of citizen scientists within the research team can help foster transparency and compliance with research ethics principles.
To provide a framework for conducting citizen science projects the European Citizen Science Association (ECSA) has developed the 10 principles of citizen science. Before moving to the next step, please, read: [http://doi.org/10.17605/OSF.IO/XPR2N ECSA (European Citizen Science Association). (2015). Ten Principles of Citizen Science]
'''References'''
#ICMJE. [https://bit.ly/N7uoq3 Defining the role of authors and contributors.]
#Resnik, D.B. (2019). Citizen scientists as human subjects: Ethical issues. ''Citizen Science: Theory and Practice'', 4(1). https://doi.org/10.5334/cstp.150
#The Embassy of Good Science: “[https://embassy.science/wiki-wiki/index.php/Theme:Cbe88760-7f0e-4d6d-952b-b724bb0f375e Authorship criteria]”
Citizen science projects collect and share diverse types of data. As pointed out by Balázs et al.: "Some projects are solely quantitative data projects, while others are solely qualitative. Mixed-method citizen science projects also exist which include both quantitative and qualitative data collection, generation, and manipulation." (Balázs et al., 2021) Due to this variety of data and other reasons, data quality in citizen science encounters various challenges that can impact the reliability and usability of the collected information. For example, analysis of the data collected by iNaturalist project revealed that the data suffers from various kinds of biases, for example, towards certain taxa (such as birds, plants, and mammals). Also, there is some evidence of spatial sampling bias. For example, about 58% of all threatened species observations in iNaturalist come from the U.S., Canada, Mexico, Russia and New Zealand (Soroye et al., 2022).
Balázs et al. point out the two main aspects of data quality in citizen science - reliability and validity. Reliability refers to the stability and consistency of data over time. In the context of citizen science, reliable data means that results can be replicated consistently. (Balázs et al., 2021) For example, in a project tracking water quality in a river, if different citizen scientists using the same measurement tools consistently report similar results for the same water samples, the data is deemed reliable. Validity in data refers to the extent to which the data accurately represents what it is supposed to measure or describe. For example, in a citizen science project on weather monitoring, if citizen scientists consistently report all relevant weather parameters (temperature, humidity, precipitation), the data is valid as it provides a comprehensive view of weather conditions.
Data contextualization refers to the practice of providing essential context and information surrounding a dataset, enabling a better understanding of how the data was generated, its purpose, and its quality. It includes metadata, attribution, and curation details to situate the data within its broader context. (Balázs et al., 2021) For example, in a climate monitoring citizen science project, metadata could include details about the creation of data set, contributors, methodology, instruments used, calibration procedures, and the temporal and spatial resolution of data. Metadata enhances the understanding and usability of the data.
''Four aspects of data accuracy in citizen science''. Balázs B. et al. https://doi.org/10.1007/978-3-030-58278-4_8, [https://creativecommons.org/licenses/by/4.0/ CC BY 4.0]
'''References'''
#Balázs, B., Mooney, P., Nováková, E., Bastin, L., Jokar Arsanjani, J. (2021). Data Quality in Citizen Science. In: ''The Science of Citizen Science''. Springer https://doi.org/10.1007/978-3-030-58278-4_8
#Soroye, P. et al. (2022). The risks and rewards of community science for threatened species monitoring. ''Conservation Science and Practice'', 4(9), e12788. https://doi.org/10.1111/csp2.12788
The term “conflict of interest” refers to situations where a person or an organisation has more than one interest (personal, professional, financial, etc.) and pursuing one of them could potentially involve conflict with others. There are two main types of conflicts of interest – financial and non-financial. An example of a '''financial conflict of interest''' is a physician who works for a pharmaceutical company that produces medicine for the same group of patients that she treats. In this case physician’s interest in earning more money conflicts with her role as a physician whose main duty is to find and prescribe the best available treatment to each patient. An example of a '''non-financial conflict of interest''' is a scientist whose personal beliefs or affiliations may impact the interpretation of his research findings. The same applies to a scientist who makes a biased hypothesis that tends to support her preferred theory.
It is important to note that conflict of interests also includes the potential for conflict, and these should always be declared. Whether financial or non-financial conflicts of interests threaten the core virtue of scientific enterprise as it interferes with the role of the scientist as a seeker of truth. Besides that, it also might undermine the public’s trust in science.
Investigators in citizen science projects might not have their pet theory that they might want to see proven true. However, laypeople who are involved in collaboration with scientists might have some political or personal interests that motivate them to participate in the research in the first place. For example, a person might have some strong beliefs about an environmental issue, and she might see involvement in the research as a way of solving the problem. There is some evidence that one of the key reasons why some citizen scientists engage in helping researchers to collect data is to advance their political aims (Riesch & Potter, 2014). These non-financial conflicts of interest might be more common in citizen science than financial conflicts of interest. An example of the latter would be a citizen scientist who receives funding from an environmental group or serves on its board of directors.
A common strategy for dealing with conflicts of interest is to declare them. Although by itself it will not solve all the problems, timely disclosure of a potential conflict of interest avoids situations where the conflict is discovered after the fact. Thus, one might avoid suspicions and loss of trust (Resnik, 2015). The importance of a potential conflict of interests may vary, some might be negligible, and some, on the other hand, very severe. Whatever the case, it is always better to inform about it upfront. One unique problem with this strategy in the context of citizen science is that lead investigators of a study might have to deal with a large number of such disclosures as many citizen scientists might be involved in the study and sharing all this information might be impractical. One strategy to solve the problem could be to disclose the conflict of interest in aggregate (Resnik, 2015). Another strategy, how one can deal with damage, that might be caused by a conflict of interests is to make all the data publicly available. This enables everybody to analyse the data and assess the results independently (Resnik, 2015).
'''References'''
#Riesch, H., & Potter, C. (2014). Citizen science as seen by scientists: Methodological, epistemological and ethical dimensions. Public Understanding of Science, 23(1), 107–120. https://doi.org/10.1177/0963662513497324
#Resnik, D. B., Elliott, K. C., & Miller, A. K. (2015). A framework for addressing ethical issues in citizen science. Environmental Science & Policy, 54, 475–481. https://doi.org/10.1016/j.envsci.2015.05.008
The data collected by citizen scientists are increasingly used in different fields of scientific research. One of the most prominent examples is animal and plant population monitoring programs. This development brings many '''benefits'''. It is a cost-effective way to gather substantial amounts of data for research purposes that otherwise would be impossible or too expensive to collect. The involvement of citizen scientists in monitoring animal and plant populations could also help improve public understanding of science and promote public engagement in conservation. Additionally, these citizen science projects can inform policies.
However, some '''risks''' have to be addressed as well. Publishing information about the location of threatened animal and plant species might inadvertently enable poaching. '''Poaching''' refers to illegal hunting, capturing, or harvesting of wildlife, typically for commercial purposes or personal gain. For example, Soroye et al. point out that "human disturbance or poaching and harvesting are listed as major threats for 57.9% of threatened species reported in iNaturalist" compared with 38% of all Red List threatened species are at risk of these threats. (Soroye et al., 2022) This suggests “that the threatened species reported to iNaturalist disproportionately tend to be threatened by disturbance and harvesting.” (Soroye et al., 2022) Moreover, incentivising non-professional monitoring creates a potential for harm even to the species that are not threatened by poaching as some species can be negatively affected just by disturbance (Quinn, 2021).
Citizen scientists can greatly contribute to monitoring threatened species by complementing traditional methods and addressing monitoring gaps. '''To avoid or mitigate''' the above-mentioned risks citizen scientists should be provided with information or trained on species identification and monitoring, citizen science projects should ensure a robust data vetting process and involve threatened species experts, as well as developing plans for data use and security. Some citizen science projects are even directly aimed at fighting against poaching (See this [https://news.mongabay.com/2018/01/crowdsourcing-the-fight-against-poaching-with-the-help-of-remote-cameras/ project in South Africa].)
'''References'''
#Quinn, A. (2021). Transparency and secrecy in citizen science: Lessons from herping. ''Studies in History and Philosophy of Science Part A'', 85, 208–217. https://doi.org/10.1016/j.shpsa.2020.10.010
#Soroye, P. et al. (2022). The risks and rewards of community science for threatened species monitoring. ''Conservation Science and Practice'', 4(9), e12788. https://doi.org/10.1111/csp2.12788
Examine the Rotterdam Dilemma Game cases and familiarize yourself with the classification criteria.'"`UNIQ--ref-00000057-QINU`"' Select the dilemmas you want to discuss. If using the Dilemma Game [https://www.eur.nl/en/about-eur/policy-and-regulations/integrity/research-integrity/dilemma-game app], start a new 'room' in group mode, creating a name for the room and specifying that it will be used for a 'lecture'. You will then be able to select the cases.
Please note that cases are grouped per topic. If the training is specifically aimed at reflecting on issues such as research processes, roles of different parties or publication ethics, the trainer might pick cases which correspond to those topics. Besides, while selecting the cases, take the attributes of the trainee group into account as well. For example, if you are going to play the game with a group of PhD students, then you should pick the cases suitable for them.
'"`UNIQ--references-00000058-QINU`"' +
Click below to watch the annotated video from the PREPARED meeting in Paris! +
The TRUST Code aims to promote equitable research partnerhsips in international research.
Why is this important? Browse through the next four steps of this module to find out. +
''Target Audience: undergraduate and graduate students.''
The training games developed by the [https://www.academicintegrity.eu/wp/bridge/ BRIDGE project] use the “gamification” approach to raise awareness and provide very basic knowledge on research integrity for university students. Online, board, role-play and scenario games can be found [https://www.academicintegrity.eu/wp/bridge-games/ here]. +
Defining reproducibility and replicability, has been a challenge in the research community, as different interpretations and even contradicting definitions are often used. Defining these terms has proven to be challenging as their use and understanding differs between fields of research. However, the European funded iRise consortium developed a reproducibility glossary by critically reviewing existing scientific literature. The glossary provides working definitions for the use of terms reproducibility, replicability and replication, as well as related concepts.
'''References'''
Voelkl, B., Heyard, R., Fanelli, D., Wever, K., Held, L., Würbel, H., Zellers, S., & Maniadis, Z. (2024). Glossary of common terminology resulting from scoping reviews. https://osf.io/ewybt. +
Six different modules on responsible open science can be found [[Guide:E525ee0d-0d7e-4ba5-b19b-89e4a5029b2f|here]], on the Embassy of Good Science and [https://classroom.eneri.eu/node/82 ENERI] platforms. [https://zenodo.org/records/11671024 Open Science Learning Gate] developed by [https://community.embassy.science/c/nerq/105 NERQ] offers the possibility to align trainings with the principles of the research community. To enhance the quality of open science practices, the ‘Open Science Learning Gate’ seeks to unite the research community while aligning and standardizing both a) customized and b) high-quality OS training programs. +
Most of the resources available are made by recommendations and guidelines on the use of responsible AI. Published by the European Commission, the [https://research-and-innovation.ec.europa.eu/document/download/2b6cf7e5-36ac-41cb-aab5-0d32050143dc_en?filename=ec_rtd_ai-guidelines.pdf Living Guidelines on the Responsible Use of Generative AI in Research] provides recommendations for researchers, research institutions and funding organizations. [https://www.academicintegrity.eu/ ENAI] developed [https://www.academicintegrity.eu/wp/wp-content/uploads/2024/06/Using-GenAI-Tools-Practical-Guide-for-Students-Ver1-MAR2024.pdf a practical guide for students] (see also [https://www.academicintegrity.eu/wp/wp-content/uploads/2024/06/USING-GENAI-TOOLS_guide_for_students_ver2.pdf here]) on the use of generative AI. [https://ukrio.org/ UKRIO] provides a full list of resources on [https://ukrio.org/ukrio-resources/ai-in-research/#list the use of AI in research]. +
'''''Target audience:''' Master and bachelor students, doctoral students and early career researchers''
The developed by the [https://cordis.europa.eu/project/id/787580 VIRT2UE] project presents a modified version of the developed by Erasmus University Rotterdam. This exercise supports participants in identifying research integrity principles, virtues and questionable research practices in a hypothetical case. It provides a framework to consider, choose and defend alternative courses of action regarding realistic research integrity dilemmas. +
In this step, you will learn how to add your own theme pages on all topics related to research ethics and research integrity.
Click the video below to see how you can make your own!
<div class="video-button" data-href="https://www.youtube.com/embed/nuxciN6pT0g">
<span class="video-button-label">Adding Theme Pages</span>
<span class="video-button-duration">0:58 min</span>
</div> +
'''''TARGET audience:''' bachelor and master students, doctoral students and early career researchers''
The e-learning modules produced by the [[Guide:Bbe860a3-56a9-45f7-b787-031689729e52|VIRT2UE]] project represent an introduction to relevant topics in research integrity, These modules, which foster self-learning introduce [[Instruction:6ceba4e4-fb32-4953-9138-5436807fcde6|research integrity]], [[Instruction:86f47366-a189-4395-9301-36ddb6d1fc68|virtue ethics relevant for RI]] and virtue ethics under current research conditions. Moreover, the training has developed a series of [[Instruction:17705907-d9b2-4f33-bc4f-088d84b4d971|videos]] and [[Instruction:7ce7ad50-499a-4cca-b09d-b2c1573d94f3|reading material]] introducing specific concepts and themes relevant to the field and practice of research integrity, such as ethical decision making in research and moral disengagement in research. +
The BEYOND approach - ‘it’s not the apple, but the orchard’ - reflects the idea that integrity is upheld as a collaborative effort. This is why it is important that training also models the collaborative way. Cases have the capacity to open up discussion space for the complexities of integrity and ethics in research, again, guiding learners to think of the full complexity, not just individuals, but also other systemic levels, including meso and macro levels, that is organisation, research community, and national, international and global context. Scaffolding provides a technique acknowledging where the individual or even a team or research community is at and designing the next steps to facilitate learning and development eventually leading to better alignment with the highest ethical and integrity standards. The point of departure is that there is always room for improvement, even in the strongest of research communities and the work starts with acknowledging status quo and identifying the next goals, which are within reach, irrespective of whether we envision the learning of individuals or communities. With these approaches; case-based and collaborative learning and scaffolding we believe training is well geared towards nurturing the orchard.
The BEYOND Trainer Guide goes beyond simply listing training materials; it adds value by explaining various pedagogical approaches that can be applied to enhance the use of different materials. It shows how learning taxonomies can be applied to create learning-focused training (as opposed to mere information transmission) irrespective of which materials produced in EU-funded projects that are implemented. We have structured the material according to target group, so that trainers can easily identify materials that are suitable for the target group they are training.
Additionally, the content is also structured according to the type of learning activities to support those trainers who wish to work using specific activities but may hesitate whether they are suitable for a particular target group, or simply would like to know more about the activity itself.
To summarise, the BEYOND approach is manifested in the Trainer Guide as:
- A proposal for a research-based approach to an ‘orchard pedagogy’
- Suggestions for measuring training effect to gain an indication of the preparedness of the research community to develop a culture of integrity
Facilitation for using existing RE/RI training resources by providing two alternative structures for trainers, including one, which addresses various actors in ‘the orchard’ through a career-level approach. We wish trainers and other readers, as well as learners taking part in trainings and learning activities utilising the resources referred to in the BEYOND Trainer Guide, a joyful journey through the orchard!
<div>
During the session, participants analyse a case using role play. In subgroups (up to 6 participants in each group) they each impersonate a member of an expert group who has been formed by the executive board of a prestigious institution to examine a difficult case and provide advice. <div>
Every participant plays one of following roles: Healthcare professional (physician); R<span lang="EN-US">epresentative of “HealthAI”;</span> <span lang="EN-US">Patient rights advocacy</span>; <span lang="EN-US">Medical ethicist;</span> <span lang="EN-US">Representative of human resources of the hospital</span><span lang="EN-US">;</span> <span lang="EN-US">Representative of a health insurance company</span>.
</div><div>
The experts are invited to have a dialogue and to learn more from each other’s perspectives. The aim is to formulate an advice for the executive board.
</div></div><div>
Before starting the exercise, it can be useful to emphasize that the groups are invited to engage in [[Theme:6217d06b-c907-4b09-af4e-b4c8a17b9847|dialogue]] rather than debate.
</div><div>
To encourage the dialogue a list of questions has been prepared (see step 5).
</div> +
The BEYOND Trainer Guide introduces effectiveness measures to help trainers assess whether the training provided is impactful and beneficial. The versatile evaluation tools are designed to be applicable to various target groups and compatible with a variety of training activities and resources. Such evaluation measures are often absent in training resources, yet they provide trainers with a valuable mechanism to ensure how effectively training supports learning. Understanding how training facilitates learning and development is necessary in the process of fostering and strengthening integrity in the research community. Provision of training is a necessary component of the overall building of a culture of integrity. Yet training, the effects of which are not monitored, falls short of its potential to mirror the change it contributes to the research community. Therefore, in the orchard approach, learning and development provides important information about the readiness of the community to build a culture of integrity. Evaluating training effectiveness to ensure training programmes achieve their intended outcomes is crucial because it connects training investments to tangible results, ensuring that the effort put into developing and delivering training is worthwhile, and for pinpointing further development needs.
Effectiveness of research ethics and integrity (REI) training can be viewed through an established effectiveness framework, which identifies four outcome domains, namely:
1. reactions (participants’ self-assessment),
2. learning (knowledge, content),
3. behaviour (acting in the research community),
4. results (e.g. institutional outcomes).'"`UNIQ--ref-000000BF-QINU`"''"`UNIQ--ref-000000C0-QINU`"'
Evaluating development of ethical competencies should be determined through done as a system to get a more holistic picture. To do this, one can combine different forms of measurement, such as self-assessment and facilitator feedback as well as attitudes and behaviour treats (in tasks that display REI competencies in the research community, like research proposals, ethics sections of theses, articles, etc.). Furthermore, measurement could take place at different times to gain insight into the learning process, learning outcome, and long-term implications, namely:
• during the training (learning process),
• right after the training – students' and facilitator’s self-reports,
• later as part of another event or course where the display of REI competencies is expected (like RE section in theses and articles, research proposal, evaluation of RE situation in the department, etc.)
It is also important to consider what to do with the results, that is what kind of changes are necessary to improve teaching and/or the environment to build a culture of integrity.
Different tools can be used to collect various learning outputs and analysis instruments can be implemented to analyse the information that has been collected (Table 2). By analysis instruments we mean the taxonomies of learning and application of theoretical models, such as levels of reflection, ethical principles and so on (if data available are mainly in a qualitative format) or statistics and learning analytics (if the data are mainly in quantitative format).
{| class="wikitable"
|+Table 1: Tools and analytical instruments for collecting learning outputs in research ethics and integrity training
!Tool for collecting learning outputs
!Details
!Analysis instrument
|-
|'''''ProLearning'' app'''
|''ProLearning'': [http://www.prolearning.realto.ch/ www.prolearning.realto.ch]
|learning analytics
|-
|'''Engagement app'''
|App under development, [https://forms.office.com/Pages/ShareFormPage.aspx?id=WXWumNwQiEKOLkWT5i_j7twYn7PlpvpDlgGDpz2LgIdUMk5XRTVYQTVKRFRDWDlHOUdGU1FHTUlFVi4u&sharetoken=03epmvYBRpmfXvpRg9os form] (for copying and editing)
|SOLO taxonomy, reflection levels, content criteria
|-
|'''Self-Reflection Form/Compass'''
|App under development, [https://forms.office.com/Pages/ShareFormPage.aspx?id=WXWumNwQiEKOLkWT5i_j7twYn7PlpvpDlgGDpz2LgIdUMk5XRTVYQTVKRFRDWDlHOUdGU1FHTUlFVi4u&sharetoken=03epmvYBRpmfXvpRg9os form] (for copying and editing)
|SOLO taxonomy, reflection levels, content criteria
|-
|'''Pre-post texts'''
|Collect a short text (e.g. a response to a case or short essay) before the training and after the training
|SOLO taxonomy, reflection levels, content criteria
|-
|'''Learning diaries'''
|Ask learners keep a diary over a certain period, for each submission provide some guiding questions or topics
|SOLO taxonomy, reflection levels, content criteria
|-
|'''Group reports'''
|Ask groups working together to provide a (short) group report (or provide a template with points to work on)
|SOLO taxonomy, content criteria
|-
|'''Group discussions'''
|Monitor the group discussions to evaluate the level of understanding and content discussed (scaffold as appropriate)
|SOLO taxonomy, content criteria
|-
|'''Group dynamics'''
|''CoTrack'' application: https://www.cotrack.website/en/
|learning analytics
|-
|'''Online learning platform'''
|Make use of accumulated authentic learning outputs in the learning platform.
|statistics, SOLO taxonomy, reflection scale, content criteria
|-
|'''Domain-specific/ domain-transcending measure'''
|Use either of the two forms measuring recognition and exemplifying of ethical issues.
|statistics, SOLO taxonomy, content criteria
|-
|'''Retention check'''
|After a certain time (few weeks/months) ask learners to provide a short text (analysis of a case, short essay on an ethics topic/question). Compare the levels of understanding to another piece collected during or right after the training.
|SOLO taxonomy, content criteria
|-
|'''Vignettes'''
|This can be used for measuring ethical sensitivity in (non-)training context
|statistics, EASM (based on the SOLO taxonomy), content criteria
|-
|'''National surveys'''
|Can be used for analysing training-related content in reports and monitoring the display of REI leadership.
|statistics, REI leadership framework
|}
Evaluation tools can give further insight into the effectiveness of the training and materials proposed. This will help trainers to adjust training content and delivery methods to improve trainees’ learning experience and outcomes. We propose mixing various tools for collecting learning outputs and adjusting them to the intended target groups (throughout the training guide suggestions are provided on which tools would be most suitable for various target groups).
'"`UNIQ--references-000000C1-QINU`"'
<span lang="EN-US">Trainers should develop specific learning goals for their session.</span> The learning objectives for these sessions should align with those of the e-learning modules. General expected outcomes at the end of these sessions include the following:
*Participants should be knowledgeable on relevant literature, developments and regulations with regards to the topic addressed
*They should be able to indicate what ethical issues are pressing regarding research concerning specific technologies
*They should be able to apply relevant ethical concerns on realistic cases
*<span lang="EN-US">They should be aware how learning materials are relevant for their professional/academic context</span> +
