Replicability
ThemesReplicability
What is this about?
Why is this important?
Replication is of great importance to science, because science aims to discover laws of nature. Since such laws are permanent, experiments on which they are based should be infinitely replicable[1]. This concept is highly important to medicine. Being able to replicate, for example, an epidemiologic study to determine health effects of certain risk factors could build up existing scientific evidence and impact decision making that might affect the public health[2]. Replicability also represents a direct public interest since science is significantly funded by public resources. If a study cannot be replicated, the money invested in it is wasted. It is estimated that annual costs of non-replicable preclinical research are approximately US$28 billion[3].
Replication can be divided into direct and conceptual[4]. Direct replication is an exact replication of an experiment and it ensures that the phenomenon is reproducible; however, it does not guarantee that the theory behind the phenomenon is true. Therefore, confirming the same results with a different methodology or a different experimental system adds more credibility to the proposed theory or model[4]. Nevertheless, we cannot expect that every experiment can be replicated down to the last detail, especially in psychology and medicine. We can always expect to see random deviation in the results and conclusions when conducting an independent experiment[5].
- ↑ Broad CD. On the Relation between Induction and Probability (Part I.). Mind. 1918;27(108):389-404.
- ↑ Peng, RD, Dominici F, Zeger SL. Reporoducible Epidemiologic Research. Am J Epidemiol. 2006;163(9):783-89.
- ↑ Freedman LP, Cockburn IM, Simcoe TS. The Economics of Reproducibility in Preclinical Research. PLoS Biol. 2015;13(6):e1002165.
- ↑ 4.0 4.1 Nosek BA, Errington TM. Making sense of replications. Elife. 2017;6:e23383.
- ↑ Bretz F, Maurer W, Xi D. Replicability, Reproducibility, and Multiplicity in Drug Development. Chance. 32(4):4-11.
For whom is this important?
What are the best practices?
Some of the most common examples of replication failures come from drug discovery and development. Usually drugs are developed in several stages, beginning with cells and animal studies and ultimately advancing to human trials. Failures in both conceptual and direct replication are frequent in this branch of science. Conceptual failure, for example, can occur when testing a drug that has promising action in animals for the first time in humans[1][2], whereas a direct replication failure might occur when testing the same drug on a similar group of people[3]. Since successful replications enhance public trust in science and medicine, the increasing number of non-replicable studies in various disciplines, mainly psychology, have resulted in what has been described as a “replication crisis” and raised serious concerns[4]. A study conducted by a team of 270 scientists at the University of Virginia in Charlottesville showed that only 35 of 100 studies published in one of the prominent psychology journals in 2008 could be replicated[5]. Some argue however that there is no such thing as a “replication crisis”; moreover, sometimes the “non-replicability” could be helpful to science[6].
If replication fails, it does not necessarily mean that the original result of the experiment which is being replicated is false. It indicates some unknown factors are different in the replication experiment vs. the original experiment and an attempt should be made to investigate these [6][7]. If such factors are found (either of a technical or knowledge domain specific nature) they can substantially improve the understanding of the phenomena being studied.
In the last few years, leading scientific institutions in the United States have taken some steps to improve replicability. In 2014, the National Institutes of Health (NIH) provided training modules for postdoctoral fellows and a list of publications regarding replicability on their website, and emphasized addressing transparency in grant applications[8]. The National Science Foundation (NSF) and the Institute for Education Sciences (IES) have published Companion Guidelines on Replication and Reproducibility in Education Research in 2018. The guidelines suggest several actions to enhance replicability. For example, proposals for replication studies should guarantee objectivity, pre-registration of the research design and methods should ensure transparency, research should be described in detail, and all research data should be publically available[9]. Taking these important steps calls for a significant culture shift so that accuracy in research would be valued more than swiftness[10].
- ↑ Bracken MB. Why animal studies are often poor predictors of human reactions to exposure. JRSM. 2009;102(3):120-22.
- ↑ The Conversation. Of mice and men: why animal trial results don't always translate to humans. 2017 Aug 29. [cited 2020 June 12]. Available from: https://theconversation.com/of-mice-and-men-why-animal-trial-results-dont-always-translate-to-humans-73354
- ↑ Ioannidis JPA. Contradicted and Initially Stronger Effects in Highly Cited Clinical Research. JAMA. 2005;294(2):218-228.
- ↑ Diener E, Biswas-Diener R. NOBA: The replication crisis in psychology. 2020. [cited 2020 June 12]. Available from: https://nobaproject.com/modules/the-replication-crisis-in-psychology
- ↑ Raloff J. ScienceNewsforStudents: When a study can't be replicated. 2015 Sep 11. [cited 2020 June 12]. Available from: https://www.sciencenewsforstudents.org/article/when-study-cant-be-replicated
- ↑ 6.0 6.1 Miceli S. Reproducibility and Replicability in Research. The National Academies In Focus. 2019;18(1):12-14.
- ↑ Nosek BA, Errington TM. Making sense of replications. Elife. 2017;6:e23383.
- ↑ Lofgren K. Replicability in Clinical Research. NBER-IFS International Network on the Value of Medical Research White Paper. 2018 Nov 13. [cited 2020 June 16]. Available from: http://www.nber.org/aging/valmed/WhitePaper-Lofgren11.2018.pdf
- ↑ Companion Guidelines on Replication & Reproducibility in Education Research. A Supplement to the Common Guidelines for Education Research and Development. A Report from the National Science Foundation and The Institute of Education Sciences, U.S. Department of Education. 2018 Nov 28. [cited 2020 June 16]. Available from: https://www.nsf.gov/pubs/2019/nsf19022/nsf19022.pdf
- ↑ ATTC. Reproducibility is broken and scientists are paying the price. [cited 2020 June 15]. Available from: https://www.lgcstandards-atcc.org/~/media/PDFs/Marketing%20Material/Reproducibility/Reproducibility%20Infographic.ashx
Ružica Tokalić, Andrijana Perković Paloš contributed to this theme. Latest contribution was Oct 21, 2020
