The standard scientific method is more fallible, and less logically rigorous, than is generally admitted. One implication is to strengthen the case for allowing patients considerable freedom in choosing their own treatments.
(p. D1) It has been jarring to learn in recent years that a reproducible result may actually be the rarest of birds. Replication, the ability of another lab to reproduce a finding, is the gold standard of science, reassurance that you have discovered something true. But that is getting harder all the time. With the most accessible truths already discovered, what remains are often subtle effects, some so delicate that they can be conjured up only under ideal circumstances, using highly specialized techniques.
Fears that this is resulting in some questionable findings began to emerge in 2005, when Dr. John P. A. Ioannidis, a kind of meta-scientist who researches research, wrote a paper pointedly titled “Why Most Published Research Findings Are False.”
. . .
. . . he published another blockbuster, examining more than a decade’s worth of highly regarded papers — the effect of a daily aspirin on cardiac disease, for example, or the risks of hormone replacement therapy for older women. He found that a large proportion of the conclusions were undermined or contradicted by later studies.
His work was just the beginning. Concern about the problem has reached the point that the journal Nature has assembled an archive, filled with reports and analyses, called Challenges in Irreproducible Research.
Among them is a paper in which C. Glenn Begley, who is chief scientific officer at TetraLogic Pharmaceuticals, described an experience he had while at Amgen, another drug company. He and his colleagues could not replicate 47 of 53 landmark papers about cancer. Some of the results could not be reproduced even with the help of the original scientists working in their own labs.
For the full commentary, see:
GEORGE JOHNSON. “Raw Data; New Truths That Only One Can See.” The New York Times (Tues., JAN. 21, 2014): D1 & D6.
(Note: ellipses added.)
(Note: the online version of the commentary has the date JAN. 20, 2014.)
The first Ioannidis article mentioned above is:
Ioannidis, John P. A. “Why Most Published Research Findings Are False.” PLoS Medicine 2, no. 8 (August 2005): 696-701.
The second Ioannidis article mentioned above is:
Ioannidis, John P. A. “Contradicted and Initially Stronger Effects in Highly Cited Clinical Research.” JAMA 294, no. 2 (July 13, 2005): 218-28.
The Begley article mentioned above is:
Begley, C. Glenn, and Lee M. Ellis. “Drug Development: Raise Standards for Preclinical Cancer Research.” Nature 483, no. 7391 (March 29, 2012): 531-33.