Measurement tools for collecting learning outputs: medium term effects
ModulesMeasurement tools for collecting learning outputs: medium term effects
What is this about?
This module will give an overview of measurement tools and evaluate the possible use of the identified measurement methods for mid-term effects.
We have divided the tools according to Kirkpatrick’s framework (1959) for training effectiveness. The framework has been used for training evaluation in REI context (Steele et al., 2016; Stoesz & Yudintseva, 2018) as well as HE context (Praslova, 2010), and includes the following levels (different kinds of tools may provide information about the achievement of the level):
- reactions (participants’ self-assessment) – different kinds of instrument may be used to collect learners’ affective and utility judgements;
- learning process (knowledge, content) – content tests, performance tasks, other course-work that is graded/evaluated, pre-post texts (tests);
- behaviour and practices (acting in the research community) – end-of-programme/course integration paper/project, learning diaries/journals (kept over a longer period), documentation of integrative work, tasks completed as part of other courses;
- results (e.g. institutional outcomes) – results can be monitored via alumni and employer surveys, media coverage, awards or recognition. In addition, nation-wide surveys may indicate the ‘health’ of RE/RI.
Pre- and post-texts
While pre- and post-test are very common as a training effectiveness measure, we are proposing a pre- and post-text measure.
Of course, tests are easy to implement and analyse (if statistics is used), but the improvement of the average scores may not provide the entire picture of the learning process. In addition, post-post texts can be used as a measure implemented several months after the training to assess the retention of the competencies – this may also provide insights into the potential change in the learner’s behaviour or practices (Kirkpatrick’s levels 2 and 3).
The learner would provide a text (either an essay or short reflection of a case) prior to the training, after that participates in training activities and then submits another text (can again be a short essay or discussion of a case). Optionally, another text can be produced several months after the end of the training. If the same analysis tool is used, the long-term impact can be measured.
This measure is suitable for HE context and in all disciplines. The measure is simple to implement. Common analysis tools make the work simpler and the progress levels comparable. The text can be evaluated based on the SOLO taxonomy and the reflection levels. Content criteria, like ethical principles, ethical analysis, ethical approaches can also be sought for in the texts. It may be challenging to use it in case of large groups as reading and analysis may take time. It may be difficult to find the learners months after the end of training.
Ethics sections in doctoral dissertations can also be analysed as ‘pre- and post-texts’ if the final product can be compared to earlier drafts.
The tool is suitable for use in training for all target groups in HE context.Domain-specific, domain-transcending measure
Ethical awareness can be investigated through domain-specific and domain-transcending measures (see Jordan, 2007). Domain-specific measure can be used to measure awareness and knowledge of a specific field. Domain-transcending measure provides an opportunity to obtain information about more general ethical issues irrespective of the discipline. These measures can be used simultaneously or separately, and they may provide information about the outcomes of training but also perhaps about the impact on practices and behaviour (mostly Kirkpatrick’s level 2). (Löfström, 2012)
A domain-specific measure contains ethical issues typical of a field and may rely on authentic examples, such as the following example, which is a fictive research proposal with multiple choice questions about ethical issues addressed in the proposal (Löfström, 2012). Such a measure is relatively easy to compose based on a fictive but realistic research proposal or other realistic academic text. The measure is suitable for use in HE context and can be adopted for various disciplines when research proposals of the filed as well as ethical issues are modified. A score can be calculated for each correct response (2 per section totalling a maximum score of 10).
Instruction: Read through the following four excerpts from a research proposal. Sentences with a number following it contain an ethical issue. Please, pair the number in the parenthesis with the corresponding ethical issue in the column to the right.
| The goal of the proposed research is to gain a broader understanding of education and wellbeing issues and concerns of youth. To accomplish this, aim a focus group involving youth aged 9-15 is formed (1). During regular bi-monthly meetings the youths’ concerns relating to education and wellbeing will be identified and discussed. The project also aims at developing the dissemination of wellbeing-related information through web-based and printed resources and materials (2). Local development needs in the area of educational, recreational and health services for youth will be identified. Participants are encouraged to disseminate information resulting from the project. | Confidentiality *
Right to withdraw Vulnerable populations Reporting of results Risk-benefit analysis |
| Research participants will be recruited at local schools, and if necessary, through the snowball technique (3). Parental permission will be sought from the youth volunteering for the focus group. Youth receiving parental permission will be included in the project. Informed consent and parental consent will be obtained in writing. The nature and purpose of the research project, its potential risks and benefits to participation will be explained to the participating adolescents and their parents or legal guardians (4). | Informed consent
Right to withdraw Vulnerable populations Voluntary participation Reporting of results |
| Participants will be asked to share their experiences and thoughts about education and wellbeing-related issues and concerns with participants in the focus group and with the researchers. Participants have the right to determine what and how much information they disclose. Identifiable personal information will not be disclosed (5). Participants may discontinue at any time without penalty or inquiries about their decision (6). | Anonymity
Confidentiality Informed consent Right to withdraw Vulnerable populations |
| The researchers will monitor and facilitate focus group discussions as needed. The youth participating in the focus group are encouraged to show respect for their peers and not to disclose information about the other participants outside the research project (7). Participants are informed of their obligation to report information that indicates potential risk or harm to self or others (8). | Confidentiality
Right to withdraw Voluntary participation Reporting of results Risk-benefit analysis |
| Discussion sessions are taped and recorded, and transcribed verbatim, and also survey data will be collected from the participants. All data will be maintained in a secure location on campus with only project researchers having access to it (9). Data will be stored a year after project completion, after which it will be destroyed. Participants and their parents will be informed that discussions may cover sensitive areas and that participants may be provided with psychological and medical information pertaining to questions that may arise during focus group sessions. A psychologist will be available to assist participating youth who may feel discomfort due to issues raised in the project (10). | Anonymity
Confidentiality Informed consent Voluntary participation Risk-benefit analysis |
Learning diaries
In order to monitor how ethics competencies develop during the training session and also see if the competencies are retained over a longer period of time reflective learning diaries can be used to monitor the development of REI competencies as well as measure the effectiveness of the training in the long term (Kirkpatrick’s levels 2 and 3). Reflection is a crucial part of ethics education as it supports the development of ethical sensitivity and ethical decision-making (Mustajoki and Mustajoki, 2017; Löfström and Tammeleht, 2023). Written reflection tasks may provide good results as writing offers a chance to pause and have an inner dialogue with oneself.
Implementation is relatively easy, but analysis may take some time. There are different ways to elicit learner’s texts – from pen-and-paper format to individual digital diaries (on various platforms) as well as forum format (where learners can see and respond to each other’s entries). Forum format has provided the best results (Tammeleht et al., 2024). Diaries should be kept during a longer period of time (at least a few months) and include weekly or bimonthly submissions. Some guiding questions or topics should be provided for each entry – this helps to keep focus and guide discussion.
Various instruments can be used for analysis: the SOLO taxonomy to evaluate the level of understanding, reflection levels to see the advancement of reflective skills, content criteria (ethical principles, ethical analysis, ethical approaches) or specific topics relevant for the course content.
There are various ethical aspects to consider when collecting and analysing learning diaries (Thorpe, 2004; Gibbs et al., 2007). Learners can be asked if they would prefer individual or forum format diary-keeping. Depending on the level of trust in the group, forum format may be uncomfortable for some learners. In HE context learners are adults, who take responsibility for writing the entries on time and honestly. This may call for keeping the content confidential unless agreed otherwise
Group reports/portfolios
Prior research has confirmed (Tammeleht et al., 2019; 2022) that using ‘the epistemic artefacts’ in the form of group reports or portfolios reflect the advancement of knowledge and co-creation of knowledge. In addition, a written collaborative ‘artefact’ with facilitator induced guidelines provides structural scaffolding like decomposing the task, asking leading questions, redirecting learners, goal orientation and highlighting important aspects (Tammeleht et al., 2020). Thus, providing a written group report to be filled during the training may act as a didactical as well as a measurement tool as after the training the facilitator can monitor the advancement of ethics competencies of learners based on the report (Kirkpatrick’s level 2).
Group reports/portfolios can be analysed by using the ECAG (the SOLO taxonomy) and content criteria (ethical principles, ethical analysis, ethical approaches). Occasionally, it is possible to monitor the advancement of understanding of separate topics and this can also be visualised (see figure 1).
Figure 1. Display of the learning process of one group based on their group portfolio. T1-T8 are topics; Ül1-4 are separate tasks outlined in the group report; numbers 0-4 indicate the SOLO levels.
The tool is suitable for all target groups in HE context.
An example template for a group portfolio can be found in Appendix 1.Group discussions
Group discussions (e.g. groups solving cases, analysing vignettes, making a decision, discussing possible courses of action or value conflicts) reflect the groups’ understanding of the topics and are suitable in HE context and in different disciplines. Sometimes the entire training may consist of discussions but only at the end the group presents their outcome, and the facilitator evaluates that part (Kirkpatrick’s level 2). But the facilitator may choose to record the discussion and evaluate the entire discussion process. Group discussions can be monitored by facilitators on site with e.g. ECAG template or recoded and then evaluated afterwards based on the SOLO taxonomy. In addition, content criteria can be identified in the discussions and discourse analysis can be used to monitor attitudes and values.
For example, the figure 2 (from Tammeleht et al., 2022) displays a (recorded) group discussion timeline indicating the levels of understanding, the time devoted to certain topics (A-D) and the order of topics. Indeed, this takes a long time but can be done in case of small groups. ECAG is more convenient and can be used in-class for on-site evaluation.
Figure 2. Group discussion timeline (from Tammeleht et al., 2022).
This tool is suitable for all target groups in HE context.
== Group discussionsGroup discussions (e.g. groups solving cases, analysing vignettes, making a decision, discussing possible courses of action or value conflicts) reflect the groups’ understanding of the topics and are suitable in HE context and in different disciplines. Sometimes the entire training may consist of discussions but only at the end the group presents their outcome, and the facilitator evaluates that part (Kirkpatrick’s level 2). But the facilitator may choose to record the discussion and evaluate the entire discussion process. Group discussions can be monitored by facilitators on site with e.g. ECAG template or recoded and then evaluated afterwards based on the SOLO taxonomy. In addition, content criteria can be identified in the discussions and discourse analysis can be used to monitor attitudes and values. ==Monitoring the online learning environment
Online learning environments (e.g. Moodle) can be used as good tools to monitor acquisition of learning outcomes and making conclusions of training effectiveness (Kirkpatrick’s level 2). Indeed, the course (both the online content as well as f2f sessions) needs to be well-planned. Accumulation of authentic learning tasks and activities reflect the materialisation of learning objectives and can be used as indicators of training effectiveness.
Most courses nowadays have an online learning environment support. Tasks included in the environment should be well-planned and have a pedagogical aspect. For instance, one online REI course that usually hosts 400 students per semester includes the following elements:
The course includes a series of online lecture videos and written texts (including links to further online resources) on each topic, followed by Quizzes or Reflective Activities about the topic in question (Assessment 5). Each topic also includes Case Studies, research ethics cases, that students discuss in groups of 5 (Assessment 3). After the first week students submit their first essay (approx. 2 pages) in which they describe the ethical questions and challenges of their own research (Assessment 1). These essays are then distributed among the students so that each gets one essay from a student of their own discipline (or close to it) and one from any other discipline. During the second week students write short peer reviews of these essays (Assessment 2). In the final essay (again approx. 2 p.) in the end of the course students are asked to consider ethics of their own research again by reflecting the comments they have received in peer reviews. They should also show that they are now able to recognize ethical issues at each stage of their research (planning, conducting, publishing, communication) (Assessment 4). In this sense, even though it is an online course, it is a kind of application of flipped learning (DeLozier & Rhodes, 2017; Tucker, 2012). In the course, students familiarize themselves with the course's online materials and lectures on different areas of research ethics and apply this in the Assessments. Evaluation is designed as following:
During the course:
• Self-evaluation through Reflective Activities and Quizzes (figure 17)
• Peer review of the 1st essays (figure 16)
• Peer evaluation in Group Discussions about the Case Studies (figure 18)
At the end of the course:
• Self-evaluation: responding to the peer reviews and reflecting on one’s development needs
• Teacher evaluates the final, 2nd essay that includes topics from all modules.
The analysis instruments may vary depending on the task – quizzes can be statistically analysed (usually the online environment provides options for creating small quizzes); essays and other texts (including written group discussions) can be analysed using the SOLO taxonomy and reflection level; content can be analysed based on content criteria (ethical principles, ethical analysis, ethical approaches), etc. Various measurement points can be chosen and self-, peer- and facilitator evaluation can be combined.
Some example assessment tasks:
Figure 3. Peer assessment example.
Figure 5. Screenshot of a case study forum in Moodle.
This tool is suitable for students and ECRs participating in a longer course.Impact of group dynamics
Collaboration has proven to be a crucial element in REI education (cf. Tammeleht et al., 2019). This means that when integrating group activities in training sessions, group dynamics and active participation may indicate how learners develop.
CoTrack monitors group dynamics and turn-taking of participants during group-work (Chejara et al., 2021). CoTrack has previously been used in REI training context (see Tammeleht et al., 2022). We piloted it also with WP3 to monitor group dynamics during a discussion game on ethics (see figure 6). The application provides information about group dynamics: how active participants are during group discussions and how their interaction is directed. This MMLA tool does not provide information about the degree to which training content has been learned, but if the learning objectives include engagement in dialogue (like for example the Debate and Dialogue exercise developed in the Virt2UE project) (Stolper & Inguaggiato, n.d.), then this tool can be used to capture the learning process (Kirkpatrick’s level 2).
Figure 6. Piloting results with CoTrack (with 4 participants). (from Parder et al., 2024)
This tool is suitable for use in training with various target groups with a limited number of participants.
CoTrack application: https://www.cotrack.website/en/Remarks
Authors: Erika Löfström, Anu Tammeleht, Simo Kyllönen,
This course was produced on behalf of the BEYOND project. The BEYOND project was finded by the European Union uder the grat agreement n. 101094714
