By Roy Benaroch, M.D., Emory University
What is shoddy science? It is a study that bypasses good scientific methods to produce results that shouldn’t be taken seriously. Let’s look at an example of shoddy science. This example is not fraud, exactly, but it’s definitely an example of cutting corners.
In September 2012, French researcher Gilles-Eric Seralini published a paper in the journal Food and Chemical Toxicology. It showed that rats fed GMO corn and exposed to a common herbicide developed cancers and other health problems. The study made a huge media splash, with headlines like “Monsanto Corn Study in France Finds Tumors and Organ Damage in Rats,” written by Reuters; and “Monsanto GM Maize May Face Europe Ban after French Study Links to Cancer,” published by French media company RFI.
Many of these news reports were accompanied by striking images of rats with huge and sometimes grotesque tumors. Those pictures are hard to forget and added to the emotional impact of the stories.
Learn more about the media and weight loss.
First Signs of Shoddy Science
But even the first few news reports on the study contained some, let’s say, red flags. As reported by the BBC in their piece “French GM-Red Rat Study Triggers Furore,” the authors insisted that journalists, who have been given access to the story, sign an unusual nondisclosure agreement. This nondisclosure agreement prevented journalists from reviewing the study with any outside experts prior to publishing their reports.
That meant that for the first few days, during that critical window of focused media attention, people with the expertise to understand potential shortfalls of the study didn’t get to see it. And those photos of the rats with tumors implied that tumors like that only happened in exposed rats, but they occurred in other rats, too. The rats used in the study were a special breed with a very high baseline incidence of tumors and cancer.
Further Signs of Shoddy Science
Skepticism mounted quickly. A Forbes writer, two days after the initial media release, wrote an article titled “Proof Perfect That the Seralini Paper on GM Corn and Cancer in Rats is Rubbish.” He pointed out that lab rats in the U.S. and other parts of the world have been routinely fed GMO corn-based food for at least a decade, and no corresponding jump in tumor rates has been observed.
Other reporters questioned the statistical power of the study to show a meaningful difference. The math is complicated, but Seralini didn’t study enough rats to draw firm conclusions from his data, and his statistical analysis was flawed.
By November 2012, about two months after the study was made public, the European Food Safety authority weighed in with a summary of their assessment of the paper. Their press release was actually an assessment by both a multinational team and of six independent assessments by individual EU states. They said the author’s conclusions “cannot be regarded as scientifically sound because of inadequacies in the design, analysis, and reporting of the study”.
This is a transcript from the video series The Skeptic’s Guide to Health, Medicine, and the Media. Watch it now, on The Great Courses Plus.
The Retraction of the Study
About a year after the Seralini study was published, it was formally retracted by the journal. As quoted in the Huffington Post, the publisher’s retraction statement said “Ultimately, the results presented—while not incorrect—are inconclusive, and therefore do not reach the threshold of publication.”
In other words, what was presented wasn’t wrong or false or fabricated, but it didn’t show what the authors had claimed it showed. That’s subtle, in a way, but crucial. For a study to be reliable, its methods, its statistics, and its interpretations have to support the thesis.
Learn more about alternative medicine in the news.
The Crucial Role of Media in Detecting Shoddy Science
The Seralini story illustrates the crucial role of the media in conveying health information, though this time with a twist. Since the study authors insisted in this case that traditional journalistic practices should be bypassed. Reporters given early access to the study, so they could write their stories, were not allowed to discuss the study with outside experts; hence, the initial reporting was only from the author’s perspective.
Presenting Both Sides of the Scale
Good health journalism should present both sides of the scale, meaning articles should include multiple viewpoints. That wasn’t allowed to happen with the Seralini rat study. Initial news reports fed an immediate hysteria. Russia immediately banned imports of this variety of corn, and Kenya announced that they’d stop planting it. But within a few days, journalistic integrity changed the tone of many of the follow-up articles.
Experts weighed in, and the shortcomings of the study became apparent. There were many examples of good reporting during that aftermath, but the initial splash, especially with those photos of rats with tumors, made big headlines and misled a lot of people.
Common Questions about Shoddy Science
Good health journalism should present both sides of the scale, meaning articles should include multiple viewpoints.
Media has a very important role in popularizing and evaluating the merits of studies. However, at times media, just like science, is swayed by the potential or pressure of success.
One of the examples of shoddy science is the French GM rat study, which was proven inconclusive by the experts.
