LoginSign up

Difference between revisions of "Theme:349f9eb9-b796-46cb-9a98-214c06db9046"

From The Embassy of Good Science
(Created page with "{{Theme |Theme Type=Good Practices |Title=Statistical pre-registration |Is About=In a statistical pre-registration, the purpose and type of statistical analysis to be conducte...")
 
Line 3: Line 3:
 
|Title=Statistical pre-registration
 
|Title=Statistical pre-registration
 
|Is About=In a statistical pre-registration, the purpose and type of statistical analysis to be conducted in a study should be disclosed before the study begins. Statistical pre-registration helps to clearly distinguish between hypothesis generating (exploratory) studies, for which statistical significance is not meaningful, and hypothesis testing (confirmatory) studies for which statistical significance is meaningful<ref>Joe Leif Uri. Data Colada. How to Properly Preregister a Study. 2017 Nov 6. [cited 2020 Aug 12]. Available from: http://datacolada.org/64</ref>.
 
|Is About=In a statistical pre-registration, the purpose and type of statistical analysis to be conducted in a study should be disclosed before the study begins. Statistical pre-registration helps to clearly distinguish between hypothesis generating (exploratory) studies, for which statistical significance is not meaningful, and hypothesis testing (confirmatory) studies for which statistical significance is meaningful<ref>Joe Leif Uri. Data Colada. How to Properly Preregister a Study. 2017 Nov 6. [cited 2020 Aug 12]. Available from: http://datacolada.org/64</ref>.
 +
<references />
 
|Important Because=Exploratory and confirmatory parts of research are two different stages and it is important to clearly distinguish them. However, researchers very often don’t make that distinction; they generate or change their hypotheses after observing the data and yet they report their study as exploratory or hypothesis-generating (see HARKing thematic page)<ref name=":0">Lindsay DS, Simons DJ, Lilienfeld SO. Research Preregistration 101. APS. 2016 Nov 30. [cited 2020 Aug 12]. Available from: https://www.psychologicalscience.org/observer/research-preregistration-101</ref>. Observing the data before choosing which statistical tests to conduct can show statistical significance where there actually isn’t any<ref name=":0" />. Subsequently, this leads to false positive results. Stopping these practices is important in every research area, but especially in medicine. Hypotheses regarding drug effects generated in exploratory research are tested in confirmatory research<ref>Kimmelman J, Mogil JS, Dirnagl U. Distinguishing between Exploratory and Confirmatory Preclinical Research Will Improve Translation. PLoS Biol. 2014;12(5):e1001863</ref>. Without careful hypotheses testing, patients are put at risk<ref>Carlisle B, Federico CA, Kimmelman J. Trials that say ‘maybe’: the disconnect between exploratory and confirmatory testing after drug approval. BMJ 2018;360:k959</ref>.
 
|Important Because=Exploratory and confirmatory parts of research are two different stages and it is important to clearly distinguish them. However, researchers very often don’t make that distinction; they generate or change their hypotheses after observing the data and yet they report their study as exploratory or hypothesis-generating (see HARKing thematic page)<ref name=":0">Lindsay DS, Simons DJ, Lilienfeld SO. Research Preregistration 101. APS. 2016 Nov 30. [cited 2020 Aug 12]. Available from: https://www.psychologicalscience.org/observer/research-preregistration-101</ref>. Observing the data before choosing which statistical tests to conduct can show statistical significance where there actually isn’t any<ref name=":0" />. Subsequently, this leads to false positive results. Stopping these practices is important in every research area, but especially in medicine. Hypotheses regarding drug effects generated in exploratory research are tested in confirmatory research<ref>Kimmelman J, Mogil JS, Dirnagl U. Distinguishing between Exploratory and Confirmatory Preclinical Research Will Improve Translation. PLoS Biol. 2014;12(5):e1001863</ref>. Without careful hypotheses testing, patients are put at risk<ref>Carlisle B, Federico CA, Kimmelman J. Trials that say ‘maybe’: the disconnect between exploratory and confirmatory testing after drug approval. BMJ 2018;360:k959</ref>.
  
 
To avoid this kind of inappropriate research, and all the consequences that can arise from it, pre-registration of a study is essential. Pre-registration consists of specifying and recording your study plans before you even start observing the data<ref name=":0" /><ref name=":1">Nosek BA, Ebersole CR, DeHaven AC, Mellor DT. The preregistration revolution. PNAS 2018;115(11):2600-2606.</ref>. It refers to recording hypotheses, subjects, statistical analysis and predictions<ref>APS. Easier Done Than Said: Lessons from 6 Years of Preregistration. 2020 March 11. [cited 2020 Aug 12]. Available from: https://www.psychologicalscience.org/publications/observer/obsonline/easier-done-than-said-lessons-from-6-years-of-preregistration.html</ref>. This will help clearly distinguish the exploratory and confirmatory data<ref name=":0" /><ref name=":1" /><ref name=":2">Wagenmakers EJ, Dutilh G. Seven Selfish Reasons for Preregistration. APS. 2016 Oct 31. [cited 2020 Aug 12]. Available from: https://www.psychologicalscience.org/observer/seven-selfish-reasons-for-preregistration</ref>, prevent bias<ref name=":2" />, avoid p-hacking<ref name=":3">APS. Preregistration of Research Plans. [cited 2020 Aug 12]. Available from: https://www.psychologicalscience.org/publications/psychological_science/preregistration</ref>, ensure transparency and openness of the analysis plans<ref name=":3" />, and increase trustworthiness and reproducibility of study findings<ref name=":1" />.
 
To avoid this kind of inappropriate research, and all the consequences that can arise from it, pre-registration of a study is essential. Pre-registration consists of specifying and recording your study plans before you even start observing the data<ref name=":0" /><ref name=":1">Nosek BA, Ebersole CR, DeHaven AC, Mellor DT. The preregistration revolution. PNAS 2018;115(11):2600-2606.</ref>. It refers to recording hypotheses, subjects, statistical analysis and predictions<ref>APS. Easier Done Than Said: Lessons from 6 Years of Preregistration. 2020 March 11. [cited 2020 Aug 12]. Available from: https://www.psychologicalscience.org/publications/observer/obsonline/easier-done-than-said-lessons-from-6-years-of-preregistration.html</ref>. This will help clearly distinguish the exploratory and confirmatory data<ref name=":0" /><ref name=":1" /><ref name=":2">Wagenmakers EJ, Dutilh G. Seven Selfish Reasons for Preregistration. APS. 2016 Oct 31. [cited 2020 Aug 12]. Available from: https://www.psychologicalscience.org/observer/seven-selfish-reasons-for-preregistration</ref>, prevent bias<ref name=":2" />, avoid p-hacking<ref name=":3">APS. Preregistration of Research Plans. [cited 2020 Aug 12]. Available from: https://www.psychologicalscience.org/publications/psychological_science/preregistration</ref>, ensure transparency and openness of the analysis plans<ref name=":3" />, and increase trustworthiness and reproducibility of study findings<ref name=":1" />.
 +
<references />
 
|Important For=Researchers; Policy makers; Journal editors
 
|Important For=Researchers; Policy makers; Journal editors
 
|Has Best Practice=The most relevant examples are studies needed for drug approval. The approval procedure usually requires a series of clinical trials divided into three phases. Phase I and II can involve model building according to European Medicines Agency (EMA), however phase III trial is always designed as a confirmatory trial<ref name=":0">Carlisle B, Federico CA, Kimmelman J. Trials that say ‘maybe’: the disconnect between exploratory and confirmatory testing after drug approval. BMJ 2018;360:k959</ref>. Both the EMA and the Food and Drug Administration (FDA) require statistical pre-registration before the beginning of a trial. Exploratory trials aim to produce evidence of effectiveness of new drugs. These results then lead to confirmatory trials<ref name=":0" />. The need for pre-registrations of trials is demonstrated by the following case. In 2004, the New York attorney general’s office filed a lawsuit against pharmaceutical company GlaxoSmithKline. Four unpublished clinical trials showed evidence that use of the antidepressant Paxil increases the risk of suicidal tendencies amongst young people. This lawsuit helped raise awareness that studies need to be pre-registered<ref name=":1">Should You Pre-Register Your Research Study? A Quick Guide. Enago Academy. 2020 April 3. [cited 2020 Aug 12]. Available from: https://www.enago.com/academy/pre-registration-of-your-research/</ref>. That year the International Committee of Medical Journal Editors required pre-registration of clinical trials<ref name=":1" />.  
 
|Has Best Practice=The most relevant examples are studies needed for drug approval. The approval procedure usually requires a series of clinical trials divided into three phases. Phase I and II can involve model building according to European Medicines Agency (EMA), however phase III trial is always designed as a confirmatory trial<ref name=":0">Carlisle B, Federico CA, Kimmelman J. Trials that say ‘maybe’: the disconnect between exploratory and confirmatory testing after drug approval. BMJ 2018;360:k959</ref>. Both the EMA and the Food and Drug Administration (FDA) require statistical pre-registration before the beginning of a trial. Exploratory trials aim to produce evidence of effectiveness of new drugs. These results then lead to confirmatory trials<ref name=":0" />. The need for pre-registrations of trials is demonstrated by the following case. In 2004, the New York attorney general’s office filed a lawsuit against pharmaceutical company GlaxoSmithKline. Four unpublished clinical trials showed evidence that use of the antidepressant Paxil increases the risk of suicidal tendencies amongst young people. This lawsuit helped raise awareness that studies need to be pre-registered<ref name=":1">Should You Pre-Register Your Research Study? A Quick Guide. Enago Academy. 2020 April 3. [cited 2020 Aug 12]. Available from: https://www.enago.com/academy/pre-registration-of-your-research/</ref>. That year the International Committee of Medical Journal Editors required pre-registration of clinical trials<ref name=":1" />.  
Line 12: Line 14:
  
 
Although pre-registration has demonstrated benefits for the trustworthiness of research, the practice still needs to be widely adopted across the scientific community.
 
Although pre-registration has demonstrated benefits for the trustworthiness of research, the practice still needs to be widely adopted across the scientific community.
 +
<references />
 
}}
 
}}
 
{{Related To
 
{{Related To

Revision as of 11:19, 21 October 2020

Icon-back.svgThemes

Statistical pre-registration

What is this about?

In a statistical pre-registration, the purpose and type of statistical analysis to be conducted in a study should be disclosed before the study begins. Statistical pre-registration helps to clearly distinguish between hypothesis generating (exploratory) studies, for which statistical significance is not meaningful, and hypothesis testing (confirmatory) studies for which statistical significance is meaningful[1].

  1. ↑ Joe Leif Uri. Data Colada. How to Properly Preregister a Study. 2017 Nov 6. [cited 2020 Aug 12]. Available from: http://datacolada.org/64

Why is this important?

Exploratory and confirmatory parts of research are two different stages and it is important to clearly distinguish them. However, researchers very often don’t make that distinction; they generate or change their hypotheses after observing the data and yet they report their study as exploratory or hypothesis-generating (see HARKing thematic page)[1]. Observing the data before choosing which statistical tests to conduct can show statistical significance where there actually isn’t any[1]. Subsequently, this leads to false positive results. Stopping these practices is important in every research area, but especially in medicine. Hypotheses regarding drug effects generated in exploratory research are tested in confirmatory research[2]. Without careful hypotheses testing, patients are put at risk[3].

To avoid this kind of inappropriate research, and all the consequences that can arise from it, pre-registration of a study is essential. Pre-registration consists of specifying and recording your study plans before you even start observing the data[1][4]. It refers to recording hypotheses, subjects, statistical analysis and predictions[5]. This will help clearly distinguish the exploratory and confirmatory data[1][4][6], prevent bias[6], avoid p-hacking[7], ensure transparency and openness of the analysis plans[7], and increase trustworthiness and reproducibility of study findings[4].

  1. ↑ 1.0 1.1 1.2 1.3 Lindsay DS, Simons DJ, Lilienfeld SO. Research Preregistration 101. APS. 2016 Nov 30. [cited 2020 Aug 12]. Available from: https://www.psychologicalscience.org/observer/research-preregistration-101
  2. ↑ Kimmelman J, Mogil JS, Dirnagl U. Distinguishing between Exploratory and Confirmatory Preclinical Research Will Improve Translation. PLoS Biol. 2014;12(5):e1001863
  3. ↑ Carlisle B, Federico CA, Kimmelman J. Trials that say ‘maybe’: the disconnect between exploratory and confirmatory testing after drug approval. BMJ 2018;360:k959
  4. ↑ 4.0 4.1 4.2 Nosek BA, Ebersole CR, DeHaven AC, Mellor DT. The preregistration revolution. PNAS 2018;115(11):2600-2606.
  5. ↑ APS. Easier Done Than Said: Lessons from 6 Years of Preregistration. 2020 March 11. [cited 2020 Aug 12]. Available from: https://www.psychologicalscience.org/publications/observer/obsonline/easier-done-than-said-lessons-from-6-years-of-preregistration.html
  6. ↑ 6.0 6.1 Wagenmakers EJ, Dutilh G. Seven Selfish Reasons for Preregistration. APS. 2016 Oct 31. [cited 2020 Aug 12]. Available from: https://www.psychologicalscience.org/observer/seven-selfish-reasons-for-preregistration
  7. ↑ 7.0 7.1 APS. Preregistration of Research Plans. [cited 2020 Aug 12]. Available from: https://www.psychologicalscience.org/publications/psychological_science/preregistration

For whom is this important?

ResearchersPolicy makersJournal editors

What are the best practices?

The most relevant examples are studies needed for drug approval. The approval procedure usually requires a series of clinical trials divided into three phases. Phase I and II can involve model building according to European Medicines Agency (EMA), however phase III trial is always designed as a confirmatory trial[1]. Both the EMA and the Food and Drug Administration (FDA) require statistical pre-registration before the beginning of a trial. Exploratory trials aim to produce evidence of effectiveness of new drugs. These results then lead to confirmatory trials[1]. The need for pre-registrations of trials is demonstrated by the following case. In 2004, the New York attorney general’s office filed a lawsuit against pharmaceutical company GlaxoSmithKline. Four unpublished clinical trials showed evidence that use of the antidepressant Paxil increases the risk of suicidal tendencies amongst young people. This lawsuit helped raise awareness that studies need to be pre-registered[2]. That year the International Committee of Medical Journal Editors required pre-registration of clinical trials[2].

US laws also require clinical trials to be pre-registered[3]. Journals and research funders support pre-registration and some organizations promote it as an important step towards openness and transparency in research[4]. There have been some other interesting efforts that promote pre-registration, such as the Preregistration Challenge, sponsored by the Center for Open Science. This campaign offered $1,000 awards to researchers who pre-register their studies and publish their results within a deadline[3][4][5]. These initiatives have led to a certain “cultural shift” - there are now more than 8,000 pre-registrations on Open Science Framework for research in different disciplines[3]. APS journals also began to practice pre-registration in 2014. From 2014 to 2019, 43 of 154 articles published in Psychological Science earned “Preregistered badge” due to pre-registration of design and analysis plan of their studies[6].

Although pre-registration has demonstrated benefits for the trustworthiness of research, the practice still needs to be widely adopted across the scientific community.

  1. ↑ 1.0 1.1 Carlisle B, Federico CA, Kimmelman J. Trials that say ‘maybe’: the disconnect between exploratory and confirmatory testing after drug approval. BMJ 2018;360:k959
  2. ↑ 2.0 2.1 Should You Pre-Register Your Research Study? A Quick Guide. Enago Academy. 2020 April 3. [cited 2020 Aug 12]. Available from: https://www.enago.com/academy/pre-registration-of-your-research/
  3. ↑ 3.0 3.1 3.2 Nosek BA, Ebersole CR, DeHaven AC, Mellor DT. The preregistration revolution. PNAS 2018;115(11):2600-2606.
  4. ↑ 4.0 4.1 Mellor D. Preregistration: improved rigor, improved workflow. BMC. 2016 Feb 17. [cited 2020 Aug 12]. Available from: https://blogs.biomedcentral.com/bmcblog/2016/02/17/preregistration-improved-rigor-improved-workflow/
  5. ↑ Mellor DT, Esposito J, Hardwicke TE, Nosek BA, Cohoon J, Soderberg CK, et al. Preregistration Challenge: Plan, Test, Discover. OSF HOME. 2019 Sep 20. [cited 2020 Aug 12]. Available from:  https://osf.io/x5w7h/
  6. ↑ APS. Open Practice Badges. [cited 2020 Aug 12]. Available from: https://www.psychologicalscience.org/publications/badges

Ružica Tokalić, Natalie Evans, Andrijana Perković Paloš contributed to this theme. Latest contribution was Oct 24, 2020

Other information

Who
Virtues & Values
Good Practices & Misconduct
Research Area
Retrieved from "https://embassy.science:443/wiki-wiki/index.php?title=Theme:349f9eb9-b796-46cb-9a98-214c06db9046&oldid=4755"
Cookies help us deliver our services. By using our services, you agree to our use of cookies.
More information

Your platform for research integrity and ethics

Privacy statementTerms of serviceTake down policyContact
Let us keep you posted

Twitter Logo Dark.svg

The EnTIRE and VIRT2UE projects have received funding from the European Union’s Horizon 2020 research programme under grant agreements N 741782 and N 787580.

5.1.6