Non-disclosure of changes to the research design

Themes

Non-disclosure of changes to the research design

What is this about?

Not disclosing changes to the research design is considered a questionable research practice. At the start of research, the study design should be determined to ensure validity and verifiability. Whilst performing research, it may turn out that the study design contains flaws or is insufficient. Changing the study design without disclosing those changes undermines the transparency of the research process. Altering the design without disclosing the changes provides researchers with the opportunity to present their results in a skewed way. For example, the researcher could select the most spectacular results, or choose a different test that leads to statistical significance.

Why is this important?

Not disclosing changes creates a biased view of the research performed. Some of the changes that researchers perform after the first analyses include P-hacking, HARK-ing, cherry picking results, or performing explorative subgroup analyses. In qualitative research the methods can also be changed, for instance, changing the research question after data collection.

For whom is this important?

All stakeholders in research

What are the best practices?

There are several ways to deal with this questionable research practice. The first is disclosing the changes made to the research design. The second is preregistration of studies.

Disclosing changes

Deviating from the research design is allowed under certain instances. For instance, new sub-questions can surface when further progressing with the project. These can only be answered by performing extra analyses or different tests. In those cases, disclosing changes to the research design is considered a good practice. These analyses or changes should be presented as explorative, rather than final.

Preregistration

Preregistration is the process of submitting the research design before performing the study. Preregistration can be seen as an effective way to address researchers from not ‘luring’ them into changing methods to present results more spectacularly.

Some journals also publish protocols and/or accept studies based in their design, proposed methods and relevance – and make a commitment to publish results. The outcome of the study is made of lesser importance, and the relevance of the study and rigor of the study design more important.

Iris Lechner, Natalie Evans contributed to this theme. Latest contribution was Oct 27, 2020

Other information

Virtues & Values

• Honesty
• Reliability