Issues with data and analyses: Errors, underlying themes, and potential solutions
There is a sense of irony that our paper about issues with data and how to prevent and correct errors came out just after our letter to the editor attempting to correct a misanalyzed cluster randomized trial was dismissed by the authors. Perhaps had we been able to share our article describing these errors and a variety of ways to prevent and correct them, the response would have been different.
Given our experiences with other attempts to correct the literature, though, that is doubtful.
In our letter, we lay out that the authors prespecified the appropriate analysis, ignored the correct analysis for certain comparisons in their paper, and then used the misanalyzed results as the focus for their press-releases. Although we offered to work with the authors in private because we believe collaboration can be more constructive than letters back and forth, the authors turned us down. In turn, their reply to our letter did not acknowledge their error. In fact, they doubled down by claiming that because others made the mistake in the past it was okay. Ted Kyle with ConscienHealth describes it more.
Our PNAS article describes challenges such as these and others we faced when trying to identify and correct the literature.
As of October 1, 2017, I am now an Assistant Professor in the Department of Applied Health Science in the School of Public Health-Bloomington at Indiana University.
My six plus year journey at the University of Alabama at Birmingham provided me with many excellent experiences, and I am both excited to move on to the next part of my career, and saddened to leave many friends and colleagues behind.
The importance of nutrition in medicine is increasingly being discussed. In response, a new book was compiled entitled Nutrition in Lifestyle Medicine edited by Dr. James Rippe. In it, my wife and colleague Dr. Michelle Bohan Brown and I wrote a chapter about “Critical Evaluation of Nutrition Research.”
The chapter walks through a number of challenges in evaluating and communicating research in general and nutrition in particular. We try to keep the topic lighthearted, with the hope that encouraging critical evaluation will not induce unabashed cynicism.
After presenting a talk for the ConAgra Food Science Institute Nutri-bites Webinars organized by Dr. Rippe entitled, “In the Eye of the Beholder: Critical Evaluation of Nutrition Research,” I was honored to be invited to contribute a chapter on a similar topic. This topic is one that Michelle and I have conducted research on, such as our paper about the reporting of the relationships between skipping breakfast and obesity. I was delighted we were able to work together again on this chapter.
Reflecting on our experiences writing letters to editors and authors to try to improve the literature through post-publication peer review, we were invited to write a comment in Nature, which we titled “Reproducibility: A tragedy of errors”.
Over the course of at least 18 months, we wrote numerous letters to editors and authors, including direct contacts, contacts through submission systems, and PubMed Commons. The effort took much time and was often frustratingly slow, including sometimes ending without a resolution. Continue reading