Insufficiently reported study flaws and limitations
ThemesInsufficiently reported study flaws and limitations
What is this about?
Every research has its flaws and limitations, a failure to report these however is a questionable research practice. Insufficiently reported study flaws and limitations are considered one of the most common questionable research practices or examples of ‘sloppy science’. Since these ‘sloppy’ practices are much more frequent, they are arguably more detrimental to science than research misconduct (falsification, fabrication, and plagiarism) [1].
- ↑ Bouter LM. Ranking major and minor research misbehaviors: results from a survey among participants of four World Conferences on Research Integrity. Res Integr Peer Rev. 2016;1-17.
Why is this important?
There is no such thing as perfect research. Every study, whether it is experimental or observational, has its limitations and deficiencies that can influence outcomes of the research [1]. Not reporting them properly is dangerous, because it could lead to over-confidence in flawed findings [2]. This, consequently, could reflect negatively on trust between scientists and decrease the public trust in science [3]. Different types of limitations that can occur during the research process in the first place are related to sample size, methodology, lack of resources and time constraints [4].
Since limitations are a natural part of the research process, they should be fully reported and described. Reporting flaws and limitations shows that a researcher fully understands the topic [5], informs and gives readers an opportunity to detect existing gaps in research, [1] and provides them with a possibility to identify new research questions [2]. Finally, presenting study limitations also shows a researcher’s honesty and transparency [1][5]. Knowing limitations and deficiencies of a certain study is an essential tool which helps to modify every decision based on considerations of risks and benefits.
- ↑ 1.0 1.1 1.2 Ross PT, Bibler Zaidi NL. Limited by our limitations. Perspect Med Educ. 2019;8:261-264.
- ↑ 2.0 2.1 Ioannidis JPA. Limitations are not properly acknowledged in the scientific literature. J Clin Epidemiol. 2007;60:324-329.
- ↑ Bouter LM. Ranking major and minor research misbehaviors: results from a survey among participants of four World Conferences on Research Integrity. Res Integr Peer Rev. 2016;1-17.
- ↑ Proacademic writers. Limitations in Research: Best Tips and Examples for Everyone. [cited June 30, 2020]. Available from: https://pro-academic-writers.com/blog/limitations-in-research.
- ↑ 5.0 5.1 Editage Insights. What should I write in limitations? [cited June 30, 2020]. Available from: https://www.editage.com/insights/what-should-i-write-in-limitations.
For whom is this important?
What are the best practices?
Regardless of the importance and necessity to fully report study limitations, in practice researchers still need to be encouraged to report their limitations and to describe them properly and thoroughly. The following example demonstrates that scientists in medicine do not fully discuss and present limitations of their research [1]. A study was conducted on 400 articles published in 2005 in journals with the highest number of citations, among them two open-access journals. Full-texts of these articles were electronically searched, looking for words ‘limitation’, ‘caveat’ or ‘caution’. The results showed that only 67 articles (17%) used at least one of the mentioned words when presenting their own research. Furthermore, only four articles (1%) used the word ‘limitation’ in their abstract, while not one article mentioned limitations of their research that had impact on the conclusions [2].
Researchers do not present their study limitations because perhaps they do not fully understand the significance, outcomes and implications of these limitations to the study results. Maybe they think that probability for publication of their work would be higher by not addressing them [3]. Journals also bear great responsibility in this matter because of the word limits that prevent authors from reporting and thoroughly describing their limitations [3]. When researchers do mention their study limitations, they usually provide only a list, they do not fully describe them [3].
There are several things researchers and journals can do to responsibly report study flaws and limitations. When describing them, researchers should clearly classify the type of limitation so that readers could interpret the research findings correctly [3]. They should not only describe the limitations, but also explain their implications. Assessing impact of limitations on conclusions of the research and its validity is also very important and can help to avoid bias. Researchers should explain why they did not take some alternative approaches or maybe provide some alternative explanations of their findings. Finally, researchers should describe efforts taken to mitigate the implications of study limitations [3]. Journals, on the other hand, should encourage authors to present their study limitations and provide them with some guidelines [2].
Reporting study flaws and limitations should enter the everyday research practice. The only way to deal with such uncertainties is to present data, methodology, limitations and study deficiencies transparently so that decision makers can be fully aware of quality and potential errors in inference.
- ↑ Puhan MA, Akl EA, Bryant D, Xie F, Apolone G, ter Riet G. Discussing study limitations in reports of biomedical studies – the need for more transparency. Health Qual Life Out. 2012;10-23.
- ↑ 2.0 2.1 Ioannidis JPA. Limitations are not properly acknowledged in the scientific literature. J Clin Epidemiol. 2007;60:324-329.
- ↑ 3.0 3.1 3.2 3.3 3.4 Ross PT, Bibler Zaidi NL. Limited by our limitations. Perspect Med Educ. 2019;8:261-264.
Natalie Evans, Andrijana Perković Paloš contributed to this theme. Latest contribution was Oct 28, 2020
