A growing number of research journals are requiring scientists to be more forthcoming in disclosing funding, potential conflicts of interest, raw data and detailed research protocols. So, is it working? When it comes to determining whether research is being done right, Yale School of Public Health Assistant Professor Joshua Wallach, Ph.D., is a good person to ask. Affiliated with Yale’s Collaboration for Research Integrity and Transparency (CRIT), Wallach is an expert in meta-research—the study of research itself. He is the lead author on a new study in PLOS Biology that looks at whether policies promoting more open science practices are having an impact. Wallach, a member of the Department of Environmental Health Sciences, discusses the findings below. Joining Wallach on the study were John Ioannidis of Stanford University’s Meta-Research Innovation Center and Kevin Boyack of SciTech Strategies, a private scientific research organization.
Your earlier research found a significant lack of transparency and reproducibility in biomedical research articles published between 2000-2014. This time you looked at more recent biomedical literature (2015-2017). What did you learn?
JW: We were eager to evaluate whether recent initiatives to promote open science practices, as well as an increased awareness of the importance of reproducibility across the scientific community, may have improved transparency and reproducibility. Unlike our previous study, we found that most published biomedical articles in our sample provided information about funding sources and potential conflicts of interest. These are important measures of transparency since funding and conflicts can influence how researchers design, conduct, and interpret studies. It was also promising that one in five articles included a statement related to data sharing. Data sharing is essential because it allows independent authors to re-analyze, replicate, or build upon previous findings. However, we observed areas where future progress is necessary. Only one article included a link to a study protocol, which outlines the steps to conduct a study, and most articles reported novel findings instead of replication efforts.
How does a lack of transparency and reproducibility hinder the scientific process?
JW: There are many ways that a lack of transparency and reproducibility can undermine the scientific process. Without transparency regarding funders or conflicts, it is unclear whether external pressures may have influenced reported findings. Without protocol and data sharing, independent investigators are unable to evaluate or repeat previous investigations. Without replication attempts, it is unclear how evidence accumulates over time and whether individual results are chance findings. These, and other key issues, can undermine the credibility of the published literature, can lead to confusion about the potential implications of a study, and can waste resources.
Why is transparency and reproducibility particularly important in biomedical research?
JW: While science is often self-correcting, and it is perfectly normal for individual research findings to be refuted by subsequent evidence, there are concerns that biomedical reproducibility is lower than desired. In certain cases, the consequences of irreproducible research may be more severe when it comes to biomedical research. For instance, when treatment decisions are based on overstated or random findings, patients can be harmed. Without transparency, it is also difficult for researchers, patients, and physicians to evaluate the robustness of published results that have the potential to guide clinical or public health decisions.
There are many ways that a lack of transparency and reproducibility can undermine the scientific process.
Joshua Wallach
Multiple initiatives have been launched in an attempt to address the growing demand for transparency and openness, including new policies for fully disclosing funding, possible conflicts of interest, research protocols and raw data. In your opinion, what is working well and what isn’t working?
JW: The International Committee of Medical Journal Editors, which is a group of medical journal editors that develops guidelines, has an electronic conflict of interest disclosure form that is currently supported by hundreds of biomedical journals. As a result, significant progress has been made is the public disclosure of potential conflicts of interest. On the other hand, most journals do not mandate protocol or data sharing. Instead, authors are often required to include a data sharing “plan” in their paper. Some evidence suggests that this may not be leading to improved transparency. In other words, a statement outlining a willingness to share does not actually imply that authors will share data. Moving forward, it will be necessary to monitor how well this effort works and whether more rigorous requirements are necessary.
Many researchers submit studies for publication that claim to present novel discoveries. You say there is much less incentive for researchers to publish studies that focus solely on validating or replicating existing knowledge. Why is this a concern and what can be done to change this scientific culture?
JW: In our survey, we found that over half of the articles claimed to have novel findings. However, it seems unlikely that the majority of studies actually resulted in drastically new discoveries. Instead, this phenomenon may be a result of the current scientific culture and academic incentive structure. For instance, “novel” findings are often preferred by journals and funders. With the pressure to generate new findings, it is possible that unfavorable or non-novel findings, which can inform science, are not published or spun in a favorable light. This undermines the validity of the published literature. In order to address this issue, the scientific culture of demanding new or groundbreaking results needs to change. There are already some promising developments, with certain journals promoting the publication of replication studies or non-significant findings. More progress can be made if the scientific community, including high impact journals, funders, and academic institutions, stop perceiving non-novel research as second-rate science.
Things seem to be getting better. Are there additional steps that can be taken to make science practices even more open and transparent?
JW: I am extremely optimistic about the recent trends. I believe that most scientists are now aware of the importance of scientific transparency and reproducibility but may be unsure about what realistic changes can be easily accomplished. At the individual level, researchers should review conflict disclosure policies to ensure that they are complying with the guidelines. When realistic, researchers should voluntarily strive to share their protocols, data, software, code and other materials that are necessary to fully replicate a study, either as supplementary documents or on an established platform. Journals could promote the use of various protocol sharing platforms, including the Open Science Framework and Protocols.io. Funders and academic institutions can play a bigger role promoting open-science practices by incentivizing or requiring greater transparency. Ongoing training about the importance of reproducibility and open-science developments can ultimately change the current research culture.
Learn more about Joshua Wallach’s work.