Methodology – Page 24 – artdiamondblog.com

“Theory-Induced Blindness”

(p. 276) The mystery is how a conception of the utility of outcomes that is vulnerable to . . . obvious counterexamples survived for so long. I can explain (p. 277) it only by a weakness of the scholarly mind that I have often observed in myself. I call it theory-induced blindness: once you have accepted a theory and used it as a tool in your thinking, it is extraordinarily difficult to notice its flaws. If you come upon an observation that does not seem to fit the model, you assume that there must be a perfectly good explanation that you are somehow missing. You give the theory the benefit of the doubt, trusting the community of experts who have accepted it. . . . As the psychologist Daniel Gilbert observed, disbelieving is hard work, and System 2 is easily tired.

Source:
Kahneman, Daniel. Thinking, Fast and Slow. New York: Farrar, Straus and Giroux, 2011.
(Note: ellipses added.)

How Politics Trumps Peer Review in Medical Research

Abstract

The U.S. public biomedical research system is renowned for its peer review process that awards federal funds to meritorious research performers. Although congressional appropriators do not earmark federal funds for biomedical research performers, I argue that they support allocations for those research fields that are most likely to benefit performers in their constituencies. Such disguised transfers mitigate the reputational penalties to appropriators of interfering with a merit‐driven system. I use data on all peer‐reviewed grants by the National Institutes of Health during the years 1984-2003 and find that performers in the states of certain House Appropriations Committee members receive 5.9-10.3 percent more research funds than those at unrepresented institutions. The returns to representation are concentrated in state universities and small businesses. Members support funding for the projects of represented performers in fields in which they are relatively weak and counteract the distributive effect of the peer review process.

Source:
Hegde, Deepak. “Political Influence Behind the Veil of Peer Review: An Analysis of Public Biomedical Research Funding in the United States.” Journal of Law and Economics 52, no. 4 (Nov. 2009): 665-90.

Economists Have “the Tools to Slap Together a Model to ‘Explain’ Any and All Phenomena”

(p. 755) The economist of today has the tools to slap together a model to ‘explain’ any and all phenomena that come to mind. The flood of models is rising higher and higher, spouting from an ever increasing number of journal outlets. In the midst of all this evidence of highly trained cleverness, it is difficult to retain the realisation that we are confronting a complex system ‘the working of which we do not understand’. . . . That the economics profession might be humbled by recent events is a realisation devoutly to be wished.

Source:
Leijonhufvud, Axel. “Out of the Corridor: Keynes and the Crisis.” Cambridge Journal of Economics 33, no. 4 (July 2009): 741-57.
(Note: ellipsis added.)
(Note: the passage above was quoted on the back cover of The Cato Journal 30, no. 2 (Spring/Summer 2010).)

Simple Algorithms Predict Better than Trained Experts

(p. 222) I never met Meehl, but he was one of my heroes from the time I read his Clinical vs. Statistical Prediction: A Theoretical Analysis and a Review of the Evidence.
In the slim volume that he later called “my disturbing little book,” Meehl reviewed the results of 20 studies that had analyzed whether clinical predictions based on the subjective impressions of trained professionals were more accurate than statistical predictions made by combining a few scores or ratings according to a rule. In a typical study, trained counselors predicted the grades of freshmen at the end of the school year. The counselors interviewed each student for forty-five minutes. They also had access to high school grades, several aptitude tests, and a four-page personal statement. The statistical algorithm used only a fraction of this information: high school grades and one aptitude test. Nevertheless, the formula was more accurate than 11 of the 14 counselors. Meehl reported generally sim-(p. 223)ilar results across a variety of other forecast outcomes, including violations of parole, success in pilot training, and criminal recidivism.
Not surprisingly, Meehl’s book provoked shock and disbelief among clinical psychologists, and the controversy it started has engendered a stream of research that is still flowing today, more than fifty years after its publication. The number of studies reporting comparisons of clinical and statistical predictions has increased to roughly two hundred, but the score in the contest between algorithms and humans has not changed. About 60% of the studies have shown significantly better accuracy for the algorithms. The other comparisons scored a draw in accuracy, but a tie is tantamount to a win for the statistical rules, which are normally much less expensive to use than expert judgment. No exception has been convincingly documented.

Source:
Kahneman, Daniel. Thinking, Fast and Slow. New York: Farrar, Straus and Giroux, 2011.
(Note: italics in original.)

Observed Climate “Not in Good Agreement with Model Predictions”

The author of the following commentary is a Princeton physics professor:

(p. A13) What is happening to global temperatures in reality? The answer is: almost nothing for more than 10 years. Monthly values of the global temperature anomaly of the lower atmosphere, compiled at the University of Alabama from NASA satellite data, can be found at the website http://www.drroyspencer.com/latest-global-temperatures/. The latest (February 2012) monthly global temperature anomaly for the lower atmosphere was minus 0.12 degrees Celsius, slightly less than the average since the satellite record of temperatures began in 1979.
The lack of any statistically significant warming for over a decade has made it more difficult for the United Nations Intergovernmental Panel on Climate Change (IPCC) and its supporters to demonize the atmospheric gas CO2 which is released when fossil fuels are burned.
. . .
Frustrated by the lack of computer-predicted warming over the past decade, some IPCC supporters have been claiming that “extreme weather” has become more common because of more CO2. But there is no hard evidence this is true.
. . .
Large fluctuations from warm to cold winters have been the rule for the U.S., as one can see from records kept by the National Ocean and Atmospheric Administration, NOAA. For example, the winters of 1932 and 1934 were as warm as or warmer than the 2011-2012 one and the winter of 1936 was much colder.
. . .
It is easy to be confused about climate, because we are constantly being warned about the horrible things that will happen or are already happening as a result of mankind’s use of fossil fuels. But these ominous predictions are based on computer models. It is important to distinguish between what the climate is actually doing and what computer models predict. The observed response of the climate to more CO2 is not in good agreement with model predictions.
. . .
. . . we should . . . remember the description of how science works by the late, great physicist, Richard Feynman:
“In general we look for a new law by the following process. First we guess it. Then we compute the consequences of the guess to see what would be implied if this law that we guessed is right. Then we compare the result of the computation to nature, with experiment or experience; compare it directly with observation, to see if it works. If it disagrees with experiment it is wrong.”

For the full commentary, see:
WILLIAM HAPPER. “Global Warming Models Are Wrong Again; The observed response of the climate to more CO2 is not in good agreement with predictions.” The Wall Street Journal (Tues., March 27, 2012): A13.
(Note: ellipses added.)

Physicist Says “Financial Models Are Only Mediocre Metaphors”

Source of book image: online version of the WSJ review quoted and cited below.

(p. A19) Trained as a physicist, Emanuel Derman once served as the head of quantitative analysis at Goldman Sachs and is currently a professor of industrial engineering and operations research at Columbia University. With “Models Behaving Badly” he offers a readable, even eloquent combination of personal history, philosophical musing and honest confession concerning the dangers of relying on numerical models not only on Wall Street but also in life.

Mr. Derman’s particular thesis can be stated simply: Although financial models employ the mathematics and style of physics, they are fundamentally different from the models that science produces. Physical models can provide an accurate description of reality. Financial models, despite their mathematical sophistication, can at best provide a vast oversimplification of reality. In the universe of finance, the behavior of individuals determines value–and, as he says, “people change their minds.”
In short, beware of physics envy. When we make models involving human beings, Mr. Derman notes, “we are trying to force the ugly stepsister’s foot into Cinderella’s pretty glass slipper. It doesn’t fit without cutting off some of the essential parts.” As the collapse of the subprime collateralized debt market in 2008 made clear, it is a terrible mistake to put too much faith in models purporting to value financial instruments. “In crises,” Mr. Derman writes, “the behavior of people changes and normal models fail. While quantum electrodynamics is a genuine theory of all reality, financial models are only mediocre metaphors for a part of it.”

For the full review, see:
BURTON G. MALKIEL. “BOOKSHELF; Physics Envy; Creating financial models involving human behavior is like forcing ‘the ugly stepsister’s foot into Cinderella’s pretty glass slipper.'” The Wall Street Journal (Weds., December 14, 2011): A19.

The book under review is:
Derman, Emanuel. Models.Behaving.Badly: Why Confusing Illusion with Reality Can Lead to Disaster, on Wall Street and in Life. New York: Free Press, 2011.

Faraday and Einstein Were Visual and Physical Thinkers, Not Mathematicians

Source of book image: http://www.rsc.org/images/Faraday_Chemical_History-of-a-Candle_180_tcm18-210390.jpg

(p. C6) Michael Faraday is one of the most beguiling and lovable figures in the history of science. Though he could not understand a single equation, he deduced the essential structure of the laws of electromagnetism through visualization and physical intuition. (James Clerk Maxwell would later give them mathematical form.) Albert Einstein kept a picture of Faraday over his desk, for Einstein also thought of himself primarily as a visual and physical thinker, not an abstract mathematician.
. . .
Faraday’s text is still charming and rich, a judgment that few popular works on science could sustain after so many years. Though he addresses himself to an “auditory of juveniles,” he calls for his audience to follow a close chain of reasoning presented through a series of experiments and deductions.
. . .
. . . : “In every one of us there is a living process of combustion going on very similar to that of a candle,” as Faraday illustrates in his experiments.
In his closing, he turns from our metabolic resemblance to a candle to his deeper wish that “you may, like it, shine as lights to those about you.”

For the full review, see:
PETER PESIC. “BOOKSHELF; Keeper of the Flame.” The Wall Street Journal (Sat., January 7, 2012): C6.
(Note: ellipses added.)

Book under review:
Faraday, Michael. The Chemical History of a Candle. Oxford, UK: Oxford University Press 2011.

In History, Documenting Your Sources Matters More than Your Credentials

George Dyson. Source of photo: online version of the NYT interview quoted and cited below.

(p. D11) BELLINGHAM, Wash. — More than most of us, the science historian George Dyson spends his days thinking about technologies, old and very new.
. . .
Though this 58-year-old author’s works are centered on technology, they often have an autobiographical subtext. Freeman Dyson, the physicist and mathematician who was a protagonist of Project Orion, is his father. Esther Dyson, the Internet philosopher and high-tech investor, is his sister. We spoke for three hours at his cottage here, and later by telephone. A condensed and edited version of the conversations follows.
. . .
. . . today you make your living as a historian of science and technology. How does a high school dropout get to do that?
Hey, this is America. You can do what you want! I love this idea that someone who didn’t finish high school can write books that get taken seriously. History is one of the only fields where contributions by amateurs are taken seriously, providing you follow the rules and document your sources. In history, it’s what you write, not what your credentials are.

For the full interview, see:
CLAUDIA DREIFUS, interviewer. “Looking Backward to Put New Technologies in Focus.” The New York Times (Tues., December 6, 2011): D11.
(Note: question bolded in original; ellipses added.)
(Note: the online version of the interview is dated December 5, 2011.)

Dyson’s most recent book is:
Dyson, George. Turing’s Cathedral: The Origins of the Digital Universe. New York: Pantheon Books, 2012.

Simple Heuristics Can Work Better than Complex Formulas

(p. C4) Most business people and physicians privately admit that many of their decisions are based on intuition rather than on detailed cost-benefit analysis. In public, of course, it’s different. To stand up in court and say you made a decision based on what your thumb or gut told you is to invite damages. So both business people and doctors go to some lengths to suppress or disguise the role that intuition plays in their work.
Prof. Gerd Gigerenzer, the director of the Max Planck Institute for Human Development in Berlin, thinks that instead they should boast about using heuristics. In articles and books over the past five years, Dr. Gigerenzer has developed the startling claim that intuition makes our decisions not just quicker but better.
. . .
The economist Harry Markowitz won the Nobel prize for designing a complex mathematical formula for picking fund managers. Yet when he retired, he himself, like most people, used a simpler heuristic that generally works better: He divided his retirement funds equally among a number of fund managers.
A few years ago, a Michigan hospital saw that doctors, concerned with liability, were sending too many patients with chest pains straight to the coronary-care unit, where they both cost the hospital more and ran higher risks of infection if they were not suffering a heart attack. The hospital introduced a complex logistical model to sift patients more efficiently, but the doctors hated it and went back to defensive decision-making.
As an alternative, Dr. Gigerenzer and his colleagues came up with a “fast-and-frugal” tree that asked the doctors just three sequential yes-no questions about each patient’s electrocardiographs and other data. Compared with both the complex logistical model and the defensive status quo, this heuristic helped the doctors to send more patients to the coronary-care unit who belonged there and fewer who did not.

For the full commentary, see:
By MATT RIDLEY. “MIND & MATTER; All Hail the Hunch–and Damn the Details.” The Wall Street Journal (Sat., December 24, 2011): C4.
(Note: ellipsis added.)

A couple of Gigerenzer’s relevant books are:
Gigerenzer, Gerd. Gut Feelings: The Intelligence of the Unconscious. New York: Penguin Books, 2007.
Gigerenzer, Gerd. Rationality for Mortals: How People Cope with Uncertainty. New York: Oxford University Press, USA, 2008.

Amateurs Can Advance Science

(p. C4) The more specialized and sophisticated scientific research becomes, the farther it recedes from everyday experience. The clergymen-amateurs who made 19th-century scientific breakthroughs are a distant memory. Or are they? Paradoxically, in an increasing variety of fields, computers are coming to the rescue of the amateur, through crowd-sourced science.
Last month, computer gamers working from home redesigned an enzyme. Last year, a gene-testing company used its customers to find mutations that increase or decrease the risk of Parkinson’s disease. Astronomers are drawing amateurs into searching for galaxies and signs of extraterrestrial intelligence. The modern equivalent of the Victorian scientific vicar is an ordinary person who volunteers his or her time to solving a small piece of a big scientific puzzle.
Crowd-sourced science is not a recent invention. In the U.S., tens of thousands of people record the number and species of birds that they see during the Christmas season, a practice that dates back more than a century. What’s new is having amateurs contribute in highly technical areas.

For the full commentary, see:
MATT RIDLEY. “MIND & MATTER; Following the Crowd to Citizen Science.” The Wall Street Journal (Sat., FEBRUARY 11, 2012): C4.

Big Data Opportunity for Economics and Business

(p. 7) Data is not only becoming more available but also more understandable to computers. Most of the Big Data surge is data in the wild — unruly stuff like words, images and video on the Web and those streams of sensor data. It is called unstructured data and is not typically grist for traditional databases.
But the computer tools for gleaning knowledge and insights from the Internet era’s vast trove of unstructured data are fast gaining ground. At the forefront are the rapidly advancing techniques of artificial intelligence like natural-language processing, pattern recognition and machine learning.
Those artificial-intelligence technologies can be applied in many fields. For example, Google’s search and ad business and its experimental robot cars, which have navigated thousands of miles of California roads, both use a bundle of artificial-intelligence tricks. Both are daunting Big Data challenges, parsing vast quantities of data and making decisions instantaneously.
. . .
To grasp the potential impact of Big Data, look to the microscope, says Erik Brynjolfsson, an economist at Massachusetts Institute of Technology’s Sloan School of Management. The microscope, invented four centuries ago, allowed people to see and measure things as never before — at the cellular level. It was a revolution in measurement.
Data measurement, Professor Brynjolfsson explains, is the modern equivalent of the microscope. Google searches, Facebook posts and Twitter messages, for example, make it possible to measure behavior and sentiment in fine detail and as it happens.
In business, economics and other fields, Professor Brynjolfsson says, decisions will increasingly be based on data and analysis rather than on experience and intuition. “We can start being a lot more scientific,” he observes.
. . .
Research by Professor Brynjolfsson and two other colleagues, published last year, suggests that data-guided management is spreading across corporate America and starting to pay off. They studied 179 large companies and found that those adopting “data-driven decision making” achieved productivity gains that were 5 percent to 6 percent higher than other factors could explain.
The predictive power of Big Data is being explored — and shows promise — in fields like public health, economic development and economic forecasting. Researchers have found a spike in Google search requests for terms like “flu symptoms” and “flu treatments” a couple of weeks before there is an increase in flu patients coming to hospital emergency rooms in a region (and emergency room reports usually lag behind visits by two weeks or so).
. . .
In economic forecasting, research has shown that trends in increasing or decreasing volumes of housing-related search queries in Google are a more accurate predictor of house sales in the next quarter than the forecasts of real estate economists. The Federal Reserve, among others, has taken notice. In July, the National Bureau of Economic Research is holding a workshop on “Opportunities in Big Data” and its implications for the economics profession.

For the full story, see:

STEVE LOHR. “NEWS ANALYSIS; The Age of Big Data.” The New York Times, SundayReview (Sun., February 12, 2012): 1 & 7.
(Note: ellipses added.)
(Note: the online version of the article is dated February 11, 2012.)