LoginSign up

Difference between revisions of "Theme:7e07b7fa-793a-4282-a036-6fefe8480b3a"

From The Embassy of Good Science
(Created page with "{{Theme |Theme Type=Good Practices |Title=Replicability |Is About=Replicability or replication in science refers to being able to repeat findings of another experiment. Succes...")
 
 
Line 6: Line 6:
  
 
Replication can be divided into direct and conceptual<ref name=":0">Nosek BA, Errington TM. Making sense of replications. Elife. 2017;6:e23383.</ref>. Direct replication is an exact replication of an experiment and it ensures that the phenomenon is reproducible; however, it does not guarantee that the theory behind the phenomenon is true. Therefore, confirming the same results with a different methodology or a different experimental system adds more credibility to the proposed theory or model<ref name=":0" />. Nevertheless, we cannot expect that every experiment can be replicated down to the last detail, especially in psychology and medicine. We can always expect to see random deviation in the results and conclusions when conducting an independent experiment<ref>Bretz F, Maurer W, Xi D. Replicability, Reproducibility, and Multiplicity in Drug Development. Chance. 32(4):4-11. </ref>.
 
Replication can be divided into direct and conceptual<ref name=":0">Nosek BA, Errington TM. Making sense of replications. Elife. 2017;6:e23383.</ref>. Direct replication is an exact replication of an experiment and it ensures that the phenomenon is reproducible; however, it does not guarantee that the theory behind the phenomenon is true. Therefore, confirming the same results with a different methodology or a different experimental system adds more credibility to the proposed theory or model<ref name=":0" />. Nevertheless, we cannot expect that every experiment can be replicated down to the last detail, especially in psychology and medicine. We can always expect to see random deviation in the results and conclusions when conducting an independent experiment<ref>Bretz F, Maurer W, Xi D. Replicability, Reproducibility, and Multiplicity in Drug Development. Chance. 32(4):4-11. </ref>.
 +
<references />
 
|Important For=All stakeholders in research; PhD Students; Funders; Journal editors
 
|Important For=All stakeholders in research; PhD Students; Funders; Journal editors
 
|Has Best Practice=Some of the most common examples of replication failures come from drug discovery and development. Usually drugs are developed in several stages, beginning with cells and animal studies and ultimately advancing to human trials. Failures in both conceptual and direct replication are frequent in this branch of science. Conceptual failure, for example, can occur when testing a drug that has promising action in animals for the first time in humans<ref>Bracken MB. Why animal studies are often poor predictors of human reactions to exposure. JRSM. 2009;102(3):120-22.</ref><ref>The Conversation. Of mice and men: why animal trial results don't always translate to humans. 2017 Aug 29. [cited 2020 June 12]. Available from: <nowiki>https://theconversation.com/of-mice-and-men-why-animal-trial-results-dont-always-translate-to-humans-73354</nowiki></ref>, whereas a direct replication failure might occur when testing the same drug on a similar group of people<ref>Ioannidis JPA. Contradicted and Initially Stronger Effects in Highly Cited Clinical Research. JAMA. 2005;294(2):218-228.</ref>. Since successful replications enhance public trust in science and medicine, the increasing number of non-replicable studies in various disciplines, mainly psychology, have resulted in what has been described as a “replication crisis” and raised serious concerns<ref>Diener E, Biswas-Diener R. NOBA: The replication crisis in psychology. 2020. [cited 2020 June 12]. Available from: <nowiki>https://nobaproject.com/modules/the-replication-crisis-in-psychology</nowiki></ref>. A study conducted by a team of 270 scientists at the University of Virginia in Charlottesville showed that only 35 of 100 studies published in one of the prominent psychology journals in 2008 could be replicated<ref>Raloff J. ScienceNewsforStudents: When a study can't be replicated. 2015 Sep 11. [cited 2020 June 12]. Available from:  <nowiki>https://www.sciencenewsforstudents.org/article/when-study-cant-be-replicated</nowiki></ref>. Some argue however that there is no such thing as a “replication crisis”; moreover, sometimes the “non-replicability” could be helpful to science<ref name=":0">Miceli S. Reproducibility and Replicability in Research. The National Academies In Focus. 2019;18(1):12-14. </ref>.
 
|Has Best Practice=Some of the most common examples of replication failures come from drug discovery and development. Usually drugs are developed in several stages, beginning with cells and animal studies and ultimately advancing to human trials. Failures in both conceptual and direct replication are frequent in this branch of science. Conceptual failure, for example, can occur when testing a drug that has promising action in animals for the first time in humans<ref>Bracken MB. Why animal studies are often poor predictors of human reactions to exposure. JRSM. 2009;102(3):120-22.</ref><ref>The Conversation. Of mice and men: why animal trial results don't always translate to humans. 2017 Aug 29. [cited 2020 June 12]. Available from: <nowiki>https://theconversation.com/of-mice-and-men-why-animal-trial-results-dont-always-translate-to-humans-73354</nowiki></ref>, whereas a direct replication failure might occur when testing the same drug on a similar group of people<ref>Ioannidis JPA. Contradicted and Initially Stronger Effects in Highly Cited Clinical Research. JAMA. 2005;294(2):218-228.</ref>. Since successful replications enhance public trust in science and medicine, the increasing number of non-replicable studies in various disciplines, mainly psychology, have resulted in what has been described as a “replication crisis” and raised serious concerns<ref>Diener E, Biswas-Diener R. NOBA: The replication crisis in psychology. 2020. [cited 2020 June 12]. Available from: <nowiki>https://nobaproject.com/modules/the-replication-crisis-in-psychology</nowiki></ref>. A study conducted by a team of 270 scientists at the University of Virginia in Charlottesville showed that only 35 of 100 studies published in one of the prominent psychology journals in 2008 could be replicated<ref>Raloff J. ScienceNewsforStudents: When a study can't be replicated. 2015 Sep 11. [cited 2020 June 12]. Available from:  <nowiki>https://www.sciencenewsforstudents.org/article/when-study-cant-be-replicated</nowiki></ref>. Some argue however that there is no such thing as a “replication crisis”; moreover, sometimes the “non-replicability” could be helpful to science<ref name=":0">Miceli S. Reproducibility and Replicability in Research. The National Academies In Focus. 2019;18(1):12-14. </ref>.
Line 12: Line 13:
  
 
In the last few years, leading scientific institutions in the United States have taken some steps to improve replicability. In 2014, the National Institutes of Health (NIH) provided training modules for postdoctoral fellows and a list of publications regarding replicability on their website, and emphasized addressing transparency in grant applications<ref>Lofgren K. Replicability in Clinical Research. NBER-IFS International Network on the Value of Medical Research White Paper. 2018 Nov 13. [cited 2020 June 16]. Available from: <nowiki>http://www.nber.org/aging/valmed/WhitePaper-Lofgren11.2018.pdf</nowiki></ref>. The National Science Foundation (NSF) and the Institute for Education Sciences (IES) have published Companion Guidelines on Replication and Reproducibility in Education Research in 2018. The guidelines suggest several actions to enhance replicability. For example, proposals for replication studies should guarantee objectivity, pre-registration of the research design and methods should ensure transparency, research should be described in detail, and all research data should be publically available<ref>Companion Guidelines on Replication & Reproducibility in Education Research. A Supplement to the ''Common Guidelines for Education Research and Development''. A Report from the National Science Foundation and The Institute of Education Sciences, U.S. Department of Education. 2018 Nov 28. [cited 2020 June 16]. Available from:  
 
In the last few years, leading scientific institutions in the United States have taken some steps to improve replicability. In 2014, the National Institutes of Health (NIH) provided training modules for postdoctoral fellows and a list of publications regarding replicability on their website, and emphasized addressing transparency in grant applications<ref>Lofgren K. Replicability in Clinical Research. NBER-IFS International Network on the Value of Medical Research White Paper. 2018 Nov 13. [cited 2020 June 16]. Available from: <nowiki>http://www.nber.org/aging/valmed/WhitePaper-Lofgren11.2018.pdf</nowiki></ref>. The National Science Foundation (NSF) and the Institute for Education Sciences (IES) have published Companion Guidelines on Replication and Reproducibility in Education Research in 2018. The guidelines suggest several actions to enhance replicability. For example, proposals for replication studies should guarantee objectivity, pre-registration of the research design and methods should ensure transparency, research should be described in detail, and all research data should be publically available<ref>Companion Guidelines on Replication & Reproducibility in Education Research. A Supplement to the ''Common Guidelines for Education Research and Development''. A Report from the National Science Foundation and The Institute of Education Sciences, U.S. Department of Education. 2018 Nov 28. [cited 2020 June 16]. Available from:  
−
 
 
<nowiki>https://www.nsf.gov/pubs/2019/nsf19022/nsf19022.pdf</nowiki></ref>. Taking these important steps calls for a significant culture shift so that accuracy in research would be valued more than swiftness<ref>ATTC. Reproducibility is broken and scientists are paying the price. [cited 2020 June 15]. Available from: <nowiki>https://www.lgcstandards-atcc.org/~/media/PDFs/Marketing%20Material/Reproducibility/Reproducibility%20Infographic.ashx</nowiki></ref>.
 
<nowiki>https://www.nsf.gov/pubs/2019/nsf19022/nsf19022.pdf</nowiki></ref>. Taking these important steps calls for a significant culture shift so that accuracy in research would be valued more than swiftness<ref>ATTC. Reproducibility is broken and scientists are paying the price. [cited 2020 June 15]. Available from: <nowiki>https://www.lgcstandards-atcc.org/~/media/PDFs/Marketing%20Material/Reproducibility/Reproducibility%20Infographic.ashx</nowiki></ref>.
 +
<references />
 
}}
 
}}
 
{{Related To
 
{{Related To

Latest revision as of 12:17, 21 October 2020

Icon-back.svgThemes

Replicability

What is this about?

Replicability or replication in science refers to being able to repeat findings of another experiment. Successful replication supports the validity of a certain discovery, increases public trust in science and impacts public health.

Why is this important?

Replication is of great importance to science, because science aims to discover laws of nature. Since such laws are permanent, experiments on which they are based should be infinitely replicable[1]. This concept is highly important to medicine. Being able to replicate, for example, an epidemiologic study to determine health effects of certain risk factors could build up existing scientific evidence and impact decision making that might affect the public health[2]. Replicability also represents a direct public interest since science is significantly funded by public resources. If a study cannot be replicated, the money invested in it is wasted. It is estimated that annual costs of non-replicable preclinical research are approximately US$28 billion[3].

Replication can be divided into direct and conceptual[4]. Direct replication is an exact replication of an experiment and it ensures that the phenomenon is reproducible; however, it does not guarantee that the theory behind the phenomenon is true. Therefore, confirming the same results with a different methodology or a different experimental system adds more credibility to the proposed theory or model[4]. Nevertheless, we cannot expect that every experiment can be replicated down to the last detail, especially in psychology and medicine. We can always expect to see random deviation in the results and conclusions when conducting an independent experiment[5].

  1. ↑ Broad CD. On the Relation between Induction and Probability (Part I.). Mind. 1918;27(108):389-404.
  2. ↑ Peng, RD, Dominici F, Zeger SL. Reporoducible Epidemiologic Research. Am J Epidemiol. 2006;163(9):783-89.
  3. ↑ Freedman LP, Cockburn IM, Simcoe TS. The Economics of Reproducibility in Preclinical Research. PLoS Biol. 2015;13(6):e1002165.
  4. ↑ 4.0 4.1 Nosek BA, Errington TM. Making sense of replications. Elife. 2017;6:e23383.
  5. ↑ Bretz F, Maurer W, Xi D. Replicability, Reproducibility, and Multiplicity in Drug Development. Chance. 32(4):4-11.

For whom is this important?

All stakeholders in researchPhD StudentsFundersJournal editors

What are the best practices?

Some of the most common examples of replication failures come from drug discovery and development. Usually drugs are developed in several stages, beginning with cells and animal studies and ultimately advancing to human trials. Failures in both conceptual and direct replication are frequent in this branch of science. Conceptual failure, for example, can occur when testing a drug that has promising action in animals for the first time in humans[1][2], whereas a direct replication failure might occur when testing the same drug on a similar group of people[3]. Since successful replications enhance public trust in science and medicine, the increasing number of non-replicable studies in various disciplines, mainly psychology, have resulted in what has been described as a “replication crisis” and raised serious concerns[4]. A study conducted by a team of 270 scientists at the University of Virginia in Charlottesville showed that only 35 of 100 studies published in one of the prominent psychology journals in 2008 could be replicated[5]. Some argue however that there is no such thing as a “replication crisis”; moreover, sometimes the “non-replicability” could be helpful to science[6].

If replication fails, it does not necessarily mean that the original result of the experiment which is being replicated is false. It indicates some unknown factors are different in the replication experiment vs. the original experiment and an attempt should be made to investigate these [6][7]. If such factors are found (either of a technical or knowledge domain specific nature) they can substantially improve the understanding of the phenomena being studied.

In the last few years, leading scientific institutions in the United States have taken some steps to improve replicability. In 2014, the National Institutes of Health (NIH) provided training modules for postdoctoral fellows and a list of publications regarding replicability on their website, and emphasized addressing transparency in grant applications[8]. The National Science Foundation (NSF) and the Institute for Education Sciences (IES) have published Companion Guidelines on Replication and Reproducibility in Education Research in 2018. The guidelines suggest several actions to enhance replicability. For example, proposals for replication studies should guarantee objectivity, pre-registration of the research design and methods should ensure transparency, research should be described in detail, and all research data should be publically available[9]. Taking these important steps calls for a significant culture shift so that accuracy in research would be valued more than swiftness[10].

  1. ↑ Bracken MB. Why animal studies are often poor predictors of human reactions to exposure. JRSM. 2009;102(3):120-22.
  2. ↑ The Conversation. Of mice and men: why animal trial results don't always translate to humans. 2017 Aug 29. [cited 2020 June 12]. Available from: https://theconversation.com/of-mice-and-men-why-animal-trial-results-dont-always-translate-to-humans-73354
  3. ↑ Ioannidis JPA. Contradicted and Initially Stronger Effects in Highly Cited Clinical Research. JAMA. 2005;294(2):218-228.
  4. ↑ Diener E, Biswas-Diener R. NOBA: The replication crisis in psychology. 2020. [cited 2020 June 12]. Available from: https://nobaproject.com/modules/the-replication-crisis-in-psychology
  5. ↑ Raloff J. ScienceNewsforStudents: When a study can't be replicated. 2015 Sep 11. [cited 2020 June 12]. Available from:  https://www.sciencenewsforstudents.org/article/when-study-cant-be-replicated
  6. ↑ 6.0 6.1 Miceli S. Reproducibility and Replicability in Research. The National Academies In Focus. 2019;18(1):12-14.
  7. ↑ Nosek BA, Errington TM. Making sense of replications. Elife. 2017;6:e23383.
  8. ↑ Lofgren K. Replicability in Clinical Research. NBER-IFS International Network on the Value of Medical Research White Paper. 2018 Nov 13. [cited 2020 June 16]. Available from: http://www.nber.org/aging/valmed/WhitePaper-Lofgren11.2018.pdf
  9. ↑ Companion Guidelines on Replication & Reproducibility in Education Research. A Supplement to the Common Guidelines for Education Research and Development. A Report from the National Science Foundation and The Institute of Education Sciences, U.S. Department of Education. 2018 Nov 28. [cited 2020 June 16]. Available from: https://www.nsf.gov/pubs/2019/nsf19022/nsf19022.pdf
  10. ↑ ATTC. Reproducibility is broken and scientists are paying the price. [cited 2020 June 15]. Available from: https://www.lgcstandards-atcc.org/~/media/PDFs/Marketing%20Material/Reproducibility/Reproducibility%20Infographic.ashx

Ružica Tokalić, Andrijana Perković Paloš contributed to this theme. Latest contribution was Oct 21, 2020

Other information

Who
Virtues & Values
Good Practices & Misconduct
Research Area
Retrieved from "https://embassy.science:443/wiki-wiki/index.php?title=Theme:7e07b7fa-793a-4282-a036-6fefe8480b3a&oldid=4754"
Cookies help us deliver our services. By using our services, you agree to our use of cookies.
More information

Your platform for research integrity and ethics

Privacy statementTerms of serviceTake down policyContact
Let us keep you posted

Twitter Logo Dark.svg

The EnTIRE and VIRT2UE projects have received funding from the European Union’s Horizon 2020 research programme under grant agreements N 741782 and N 787580.

5.1.6