Statistical pre-registration
ThemesStatistical pre-registration
What is this about?
In a statistical pre-registration, the purpose and type of statistical analysis to be conducted in a study should be disclosed before the study begins. Statistical pre-registration helps to clearly distinguish between hypothesis generating (exploratory) studies, for which statistical significance is not meaningful, and hypothesis testing (confirmatory) studies for which statistical significance is meaningful[1].
- ↑ Joe Leif Uri. Data Colada. How to Properly Preregister a Study. 2017 Nov 6. [cited 2020 Aug 12]. Available from: http://datacolada.org/64
Why is this important?
Exploratory and confirmatory parts of research are two different stages and it is important to clearly distinguish them. However, researchers very often don’t make that distinction; they generate or change their hypotheses after observing the data and yet they report their study as exploratory or hypothesis-generating (see HARKing thematic page)[1]. Observing the data before choosing which statistical tests to conduct can show statistical significance where there actually isn’t any[1]. Subsequently, this leads to false positive results. Stopping these practices is important in every research area, but especially in medicine. Hypotheses regarding drug effects generated in exploratory research are tested in confirmatory research[2]. Without careful hypotheses testing, patients are put at risk[3].
To avoid this kind of inappropriate research, and all the consequences that can arise from it, pre-registration of a study is essential. Pre-registration consists of specifying and recording your study plans before you even start observing the data[1][4]. It refers to recording hypotheses, subjects, statistical analysis and predictions[5]. This will help clearly distinguish the exploratory and confirmatory data[1][4][6], prevent bias[6], avoid p-hacking[7], ensure transparency and openness of the analysis plans[7], and increase trustworthiness and reproducibility of study findings[4].
- ↑ 1.0 1.1 1.2 1.3 Lindsay DS, Simons DJ, Lilienfeld SO. Research Preregistration 101. APS. 2016 Nov 30. [cited 2020 Aug 12]. Available from: https://www.psychologicalscience.org/observer/research-preregistration-101
- ↑ Kimmelman J, Mogil JS, Dirnagl U. Distinguishing between Exploratory and Confirmatory Preclinical Research Will Improve Translation. PLoS Biol. 2014;12(5):e1001863
- ↑ Carlisle B, Federico CA, Kimmelman J. Trials that say ‘maybe’: the disconnect between exploratory and confirmatory testing after drug approval. BMJ 2018;360:k959
- ↑ 4.0 4.1 4.2 Nosek BA, Ebersole CR, DeHaven AC, Mellor DT. The preregistration revolution. PNAS 2018;115(11):2600-2606.
- ↑ APS. Easier Done Than Said: Lessons from 6 Years of Preregistration. 2020 March 11. [cited 2020 Aug 12]. Available from: https://www.psychologicalscience.org/publications/observer/obsonline/easier-done-than-said-lessons-from-6-years-of-preregistration.html
- ↑ 6.0 6.1 Wagenmakers EJ, Dutilh G. Seven Selfish Reasons for Preregistration. APS. 2016 Oct 31. [cited 2020 Aug 12]. Available from: https://www.psychologicalscience.org/observer/seven-selfish-reasons-for-preregistration
- ↑ 7.0 7.1 APS. Preregistration of Research Plans. [cited 2020 Aug 12]. Available from: https://www.psychologicalscience.org/publications/psychological_science/preregistration
For whom is this important?
What are the best practices?
The most relevant examples are studies needed for drug approval. The approval procedure usually requires a series of clinical trials divided into three phases. Phase I and II can involve model building according to European Medicines Agency (EMA), however phase III trial is always designed as a confirmatory trial[1]. Both the EMA and the Food and Drug Administration (FDA) require statistical pre-registration before the beginning of a trial. Exploratory trials aim to produce evidence of effectiveness of new drugs. These results then lead to confirmatory trials[1]. The need for pre-registrations of trials is demonstrated by the following case. In 2004, the New York attorney general’s office filed a lawsuit against pharmaceutical company GlaxoSmithKline. Four unpublished clinical trials showed evidence that use of the antidepressant Paxil increases the risk of suicidal tendencies amongst young people. This lawsuit helped raise awareness that studies need to be pre-registered[2]. That year the International Committee of Medical Journal Editors required pre-registration of clinical trials[2].
US laws also require clinical trials to be pre-registered[3]. Journals and research funders support pre-registration and some organizations promote it as an important step towards openness and transparency in research[4]. There have been some other interesting efforts that promote pre-registration, such as the Preregistration Challenge, sponsored by the Center for Open Science. This campaign offered $1,000 awards to researchers who pre-register their studies and publish their results within a deadline[3][4][5]. These initiatives have led to a certain “cultural shift” - there are now more than 8,000 pre-registrations on Open Science Framework for research in different disciplines[3]. APS journals also began to practice pre-registration in 2014. From 2014 to 2019, 43 of 154 articles published in Psychological Science earned “Preregistered badge” due to pre-registration of design and analysis plan of their studies[6].
Although pre-registration has demonstrated benefits for the trustworthiness of research, the practice still needs to be widely adopted across the scientific community.
- ↑ 1.0 1.1 Carlisle B, Federico CA, Kimmelman J. Trials that say ‘maybe’: the disconnect between exploratory and confirmatory testing after drug approval. BMJ 2018;360:k959
- ↑ 2.0 2.1 Should You Pre-Register Your Research Study? A Quick Guide. Enago Academy. 2020 April 3. [cited 2020 Aug 12]. Available from: https://www.enago.com/academy/pre-registration-of-your-research/
- ↑ 3.0 3.1 3.2 Nosek BA, Ebersole CR, DeHaven AC, Mellor DT. The preregistration revolution. PNAS 2018;115(11):2600-2606.
- ↑ 4.0 4.1 Mellor D. Preregistration: improved rigor, improved workflow. BMC. 2016 Feb 17. [cited 2020 Aug 12]. Available from: https://blogs.biomedcentral.com/bmcblog/2016/02/17/preregistration-improved-rigor-improved-workflow/
- ↑ Mellor DT, Esposito J, Hardwicke TE, Nosek BA, Cohoon J, Soderberg CK, et al. Preregistration Challenge: Plan, Test, Discover. OSF HOME. 2019 Sep 20. [cited 2020 Aug 12]. Available from: https://osf.io/x5w7h/
- ↑ APS. Open Practice Badges. [cited 2020 Aug 12]. Available from: https://www.psychologicalscience.org/publications/badges
Ružica Tokalić, Natalie Evans, Andrijana Perković Paloš contributed to this theme. Latest contribution was Oct 24, 2020
