(p. 10) How much of biomedical research is actually wrong? John Ioannidis, an epidemiologist and health-policy researcher at Stanford, was among the first to sound the alarm with a 2005 article in the journal PLOS Medicine. He showed that small sample sizes and bias in study design were chronic problems in the field and served to grossly overestimate positive results. His dramatic bottom line was that “most published research findings are false.”
The problem is especially acute in laboratory studies with animals, in which scientists often use just a few animals and fail to select them randomly. Such errors inevitably introduce bias. Large-scale human studies, of the sort used in drug testing, are less likely to be compromised in this way, but they have their own failings: It’s tempting for scientists (like everyone else) (p. C2) to see what they want to see in their findings, and data may be cherry-picked or massaged to arrive at a desired conclusion.
A paper published in February [2017] in the journal PLOS One by Estelle Dumas-Mallet and colleagues at the University of Bordeaux tracked 156 biomedical studies that had been the subject of stories in major English-language newspapers. Follow-up studies, they showed, overturned half of those initial positive results (though such disconfirmation rarely got follow-up news coverage). The studies dealt with a wide range of issues, including the biology of attention-deficit hyperactivity disorder, new breast-cancer susceptibility genes, a reported link between pesticide exposure and Parkinson’s disease, and the role of a virus in autism.
Reviews by pharmaceutical companies have delivered equally grim numbers. In 2011, scientists at Bayer published a paper in the journal Nature Reviews Drug Discovery showing that they could replicate only 25% of the findings of various studies. The following year, C. Glenn Begley, the head of cancer research at Amgen, reported in the journal Nature that he and his colleagues could reproduce only six of 53 seemingly promising studies, even after enlisting help from some of the original scientists.
With millions of dollars on the line, industry scientists overseeing clinical trials with human subjects have a stronger incentive to follow high standards. Such studies are often designed in cooperation with the U.S. Food and Drug Administration, which ultimately reviews the findings. Still, most clinical trials produce disappointing results, often because the lab studies on which they are based were themselves flawed.
For the full essay see:
(Note: bracketed year added.)
(Note: the online version of the essay was updated April 7, 2017 [sic], and has the title “The Breakdown in Biomedical Research.”)
The essay quoted above is adapted from Mr. Harris’s book:
Harris, Richard. Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions. New York: Basic Books, 2017.
The 2005 paper by Ioannidis mentioned above is: