John List Shows Limitations of Randomized Controlled Trials (RCTs)

(p. A15) John List’s “The Voltage Effect” is marketed as a generic business title on how and whether to scale up an idea or product. Mr. List, an economics professor at the University of Chicago, explores why some ideas attain “voltage” and catch fire while others die out. This angle suggests that it will be another book about how to turn that great invention in your garage into the next Hewlett-Packard. But Mr. List is far too thoughtful to write something gimmicky or simple.

. . .

“The Voltage Effect” is a fine business book, though in many ways it works better as a meditation on the shortcomings of our increasingly data-driven world. The business community and academia have been taken over by data science. Mr. List seemingly argues that good and helpful data analysis may not scale well. It takes tremendous skill and talent to distinguish a scalable idea from one that is doomed to flop when you are working with a limited set of data and have an incentive to overhype your results. Data is the new currency; companies are presumed to have an unfair advantage if they have access to more of it. What gets less attention is the shortage of people who know how to make sense of statistical experiments and generalize them to a larger population.

The fields of business, policy and economics have all become enthralled with Randomized Control Trials. These are statistical experiments in which researchers take two populations: a “treatment” group that may be given cash or some other incentive and a “control” group that is not given anything. Researchers then observe any difference in outcomes from the experiment to make policy recommendations. RCTs can be a useful tool. But taking Mr. List’s lessons to heart, you see how limited they are.

Even the best-designed experiment may not give you insights that scale. For example, studies have found that it is more effective to give people cash in Kenya than to distribute aid through arcane development programs. The mantra in the development community has become “just give people money.” But just because cash is better than aid in Kenya, it doesn’t necessarily mean that a Universal Basic Income will work well in California.

For the full review, see:

Allison Schrager. “BOOKSHELF; Do We Have a Winner?” The Wall Street Journal (Monday, March 28, 2022): A15.

(Note: ellipsis added.)

(Note: the online version of the review has the date March 27, 2022, and has the title “BOOKSHELF; ‘The Voltage Effect’ Review: Do We Have a Winner?”)

The book under review is:

List, John A. The Voltage Effect: How to Make Good Ideas Great and Great Ideas Scale. New York: Currency, 2022.

Leave a Reply

Your email address will not be published.