Methodology – Page 31 – artdiamondblog.com

Highly Reputed Academic Science Journal Found Similar Error Rates in Britannica and Wikipedia

(p. 208) Wikipedia was already highly regarded, anecdotally, but it got a glowing evaluation from the prestigious Nature magazine in December 2005, when it concluded that Wikipedia “comes close” to Britannica in the quality of its science articles. “Our reviewers identified an average of four errors in each Wikipedia article, and three in each Britannica article.”

The news came as a bit of a surprise. Many folks felt Wikipedia did better than they’d have thought, and Britannica did, well, worse than they expected. The result of the study was hotly debated between Nature and Britannica, but to most Wikipedians it was a vindication. They knew that Wikipedia was a minefield of errors, but to be in such close proximity in quality to a traditionally edited encyclopedia, while using such a grassroots process, was the external validation they had been waiting for.
Britannica wasn’t pleased with the methodology, and posted a rebuttal with this criticism: “Almost everything about the journal’s investigation, from the criteria for identifying inaccuracies to the discrepancy between the article text and its headline, was wrong and misleading.” Nature and Britannica exchanged barbs and rebuttals, but in the end, the overall result seemed clear.
“The Nature (sic) article showed that we are on the right track with our current methods. We just need better ways to prevent the display of obvious vandalism at any time,” wrote longtime Wikipedian Daniel Mayer on the mailing list.

Source:
Lih, Andrew. The Wikipedia Revolution: How a Bunch of Nobodies Created the World’s Greatest Encyclopedia. New York: Hyperion, 2009.
(Note: italics in original.)

Quants Confused Mathematical Models and Reality

Source of book image: http://seekingalpha.com/article/188632-the-quants-review-when-the-money-grid-went-dark

(p. 7) The virtually exclusive use of mathematical models, Mr. Patterson says, was what separated the younger cohorts of quants from their Wall Street forebears. Unlike Warren Buffett or Peter Lynch, the quants did not focus on so-called market fundamentals like what goods or services a particular company actually produced. Seldom if ever did they act on old-fashioned gut instinct. Instead, they focused on factors like how cheap a stock was relative to the rest of the market or how quickly its price had risen or fallen.

Therein was the quants’ flaw, according to Mr. Patterson. Pioneers like Mr. Thorp understood that while the math world and the financial world have much in common, they aren’t always in sync. The quant traders’ model emphasized the most likely moves a stock or bond price could make. It largely ignored the possibility of big jolts caused by human factors, especially investor panics.
“The model soon became so ubiquitous that, hall-of-mirrors-like, it became difficult to tell the difference between the model and the market itself,” Mr. Patterson declares.
Move ahead to August 2007 and beyond, when markets swooned on doubts about subprime mortgages. Stocks that the model predicted were bound to go up went sharply down, and vice versa. Events that were supposed to happen only once in 10,000 years happened three days in a row.

For the full review, see:

HARRY HURT III. “Off the Shelf; In Practice, Stock Formulas Weren’t Perfect.” The New York Times, SundayBusiness Section (Sun., February 21, 2010): 7.
(Note: the online version of the article is dated February 20, 2010.)

The reference to Patterson’s book, is:
Patterson, Scott. The Quants: How a New Breed of Math Whizzes Conquered Wall Street and Nearly Destroyed It. New York: Crown Business, 2010.

Philosopher Duped by Hoax Because He Failed to Consult Wikipedia

(p. A4) PARIS — For the debut of his latest weighty title, “On War in Philosophy,” the French philosopher Bernard-Henri Lévy made the glossy spreads of French magazines with his trademark panache: crisp, unbuttoned white Charvet shirts, golden tan and a windswept silvery mane of hair.

But this glamorous literary campaign was suddenly marred by an absolute philosophical truth: Mr. Lévy backed up the book’s theories by citing the thought of a fake philosopher. In fact, the sham philosopher has never been a secret, and even has his own Wikipedia entry.
In the uproar that followed over the rigors of his research, Mr. Lévy on Tuesday summed up his situation with one e-mailed sentence: “My source of information is books, not Wikipedia.”

For the full story, see:

DOREEN CARVAJAL. “Philosopher Left to Muse on Ridicule Over a Hoax.” The New York Times (Weds., February 10, 2010): A4.
(Note: the online version of the article is dated February 9, 2010.)

“Expert Scholarship” Versus “People of Dubious Background”

(p. 71) The acknowledgment, by name, of volunteers in the preface sections of the OED is akin to Wikipedia’s edit history, where one can inspect who contributed to each article. Some Oxford contributors were professors, some royalty, but most were ordinary folks who answered the call. Winchester, in The Professor and the Madman: A Tale of Murder, Insanity, and the Making of the Oxford English Dictionary, tells the story of the “madman” William Chester Minor, a U.S. Civil War survivor whose “strange and erratic behavior” resulted in him shooting an “innocent working man” to death in the street in Lambeth. He was sent to Broadmoor asylum for criminal lunatics. He discovered the OED as a project around 1881, when he saw the “Appeal for Readers” in the library, and worked for the next twenty-one years contributing to the project, receiving notoriety as a contributor “second only to the contributions of Dr. Fitzedward Hall in enhancing our illustration of the literary history of individual words, phrases and constructions.” Minor did something unusual in not just sending submissions, but having his own cataloging system such that the dictionary editors could send a postcard and “out the details flowed, in abundance and always with unerring accuracy.” Until Minor and Murray met in January 1891, no one working with (p. 72) the OED knew their prolific contributor was a madman and murderer housed at Broadmoor.

As we will see in later chapters, a common question of the wiki method is whether one can trust information created by strangers and people of dubious background. But the example of the OED shows that using contributors rather than original expert scholarship is not a new phenomenon, and that projects built as a compendium of primary sources are well suited for harnessing the power of distributed volunteers.

Source:
Lih, Andrew. The Wikipedia Revolution: How a Bunch of Nobodies Created the World’s Greatest Encyclopedia. New York: Hyperion, 2009.
(Note: italics in original.)

Like Wikipedia, Oxford English Dictionary Was Built by Amateur Volunteers

(p. 70) The venerable Oxford English Dictionary (OED), the history of which is masterfully documented by Simon Winchester in The Meaning of Everything and The Professor and the Madman, was in fact possible only through the soliciting of contributions, and the receipt of thousands of “slips” of paper, each with words and definitions found by readers and volunteers.

The OED didn’t start out with such a grand title, and was first a project of the Philological Society in Great Britian (sic), as a response to what they saw as the popular dictionaries of Noah Webster and Samuel Johnson not doing the “English language justice.” In 1857, it was started as the Unregistered Words Committee, and the job was to comb through all forms of media of the era (printed matter, song, spoken word) leading to the inventorying and cataloging of English words. The three founders, Chenevix Trench, Herbert Coleridge, and Frederick Furnivall, sent out a notice in November of that year: “AN APPEAL TO THE ENGLISH-SPEAKING AND ENGLISH-READING PUBLIC TO READ BOOKS AND MAKE EXTRACTS FOR THE PHILOLOGICAL SOCIETY’S NEW ENGLISH DICTIONARY.” Specifically, it described the project thusly:

Accordingly, in January 1859. the Society issued their Proposal for the publication of a New English Dictionary, in which the characteristics of the proposed work were explained, and an appeal made to the English and American public to assist in collecting the raw materials for the work, these materials consisting of quotations illustrating the use of English words by all writers of all ages and in all senses, each quotation being made on a uniform plan on a half-sheet of notepaper that they might in due course be arranged and classified alphabetically and significantly. This Appeal met with generous response: some hundreds of volunteers began to read books, make quotations and send in their slips to “sub-editors who volunteered each to take charge of a letter or part of one, and by whom the slips were in turn further arranged, classified, and (p. 71) to some extent used as the basis of definitions and skeleton schemes of the meanings of words in preparation for the Dictionary.

The notice was sent to “bookshops and libraries across the English-speaking world” and, under the direction of Scottish lexicographer James Murray, saw its growth blossom. In 1879, Oxford University Press formally agreed to be publisher and employed Murray to take on the editorship. Slips sent in to the effort were filed away in pigeonholes at the Scriptorium, a corrugated metal building Mill Hill School erected specifically for the effort of sorting and housing the staff to work on the dictionary.

Source:
Lih, Andrew. The Wikipedia Revolution: How a Bunch of Nobodies Created the World’s Greatest Encyclopedia. New York: Hyperion, 2009.
(Note: italics and caps in original.)

The block quote within the Lih block quote is from p. 108 of:
Winchester, Simon. The Meaning of Everything: The Story of the Oxford English Dictionary. paperback ed. New York: Oxford University Press, USA, 2003.

At Odds with Academic Culture, Wiki Programmer Adams Released Early and Released Often

(p. 67) Adams did something unexpected for the academic community, but common in open source culture–release early and release often. Within weeks of its launch, one of the biggest annoyances of Wikipedia was resolved directly by the software’s author. It was not because of monetary compensation or any formal request, but simply because the author was interested in solving it on his own time, and sharing it with others. It was the hacker ethos, and it had crossed from the domain of tech programmers into the world of encyclopedias.

Source:
Lih, Andrew. The Wikipedia Revolution: How a Bunch of Nobodies Created the World’s Greatest Encyclopedia. New York: Hyperion, 2009.

“Strategy, as We Knew It, Is Dead”

(p. B7) During the recession, as business forecasts based on seemingly plausible swings in sales smacked up against reality, executives discovered that strategic planning doesn’t always work.

Some business leaders came away convinced that the new priority was to be able to shift course on the fly. Office Depot Inc., for example, began updating its annual budget every month, starting in early 2009. Other companies started to factor more extreme scenarios into their thinking. A few even set up “situation rooms,” where staffers glued to computer screens monitored developments affecting sales and finances.
Now, even though the economy is slowly picking up, those fresh habits aren’t fading. “This downturn has changed the way we will think about our business for many years to come,” says Steve Odland, Office Depot’s chairman and chief executive.
Walt Shill, head of the North American management consulting practice for Accenture Ltd., is even more blunt: “Strategy, as we knew it, is dead,” he contends. “Corporate clients decided that increased flexibility and accelerated decision making are much more important than simply predicting the future.”
Companies have long planned for changing circumstances. What’s new–and a switch from the distant calendars and rigid forecasts of the past–is the heavy dose of opportunism. Office Depot stuck with its three-year planning process after the recession hit, largely to make sure employees had a common plan to rally around, Mr. Odland says. But the CEO decided to review the budget every month rather than quarterly so the office-supply chain could react faster to changes in customers’ needs.

For the full story, see:
JOANN S. LUBLIN and DANA MATTIOLI. “Theory & Practice; Strategic Plans Lose Favor; Slump Showed Bosses Value of Flexibility, Quick Decisions.” The Wall Street Journal (Mon., January 22, 2010): B7.

Wikipedia Works in Practice, Not in Theory

(p. 20) Jimmy walked into the offices of Chicago Options Associates in 1994 and met the CEO Michael Davis for a job interview. Davis had looked over Wales’s academic publication about options pricing.

“It was impressive looking,” says Wales wryly about the paper. “It was a very theoretical paper but it wasn’t very practical.” But Davis was sufficiently intrigued, as he wanted someone like Wales to pore over the firm’s financial models and help improve them. So he took on young Wales, who seemed to be sharp and had acumen for numbers. Little did either of them know they would have a long road ahead together, with Wikipedia in the future.
Wales’s first job was to go over the firm’s current pricing models. “What was really fascinating was that it was truly a step beyond what I’d seen in academia,” he recalls. “It was very practical, and didn’t have a real theoretical foundation.” Wales was intrigued that the firm traded on principles that worked in practice, not in theory. (This is something he would say about his future endeavor Wikipedia.) “Basically they just knew in the marketplace that the existing models were wrong.”

Source:
Lih, Andrew. The Wikipedia Revolution: How a Bunch of Nobodies Created the World’s Greatest Encyclopedia. New York: Hyperion, 2009.
(Note: italics in original.)

The Entrepreneurial Epistemology of Wikipedia

Source of book image: http://kellylowenstein.files.wordpress.com/2009/04/wikipedia-revolution1.jpg

Wikipedia is a very unexpected and disruptive institution. Amateurs have produced an encyclopedia that is bigger, deeper, more up-to-date, and arguably of at least equal accuracy, with the best professional encyclopedias, such as Britannica.
I learned a lot from Lih’s book. For instance I did not know that the founders of Wikipedia were admirers of Ayn Rand. And I did not know that the Oxford English Dictionary was constructed mainly by volunteer amateurs.
I also did not know anything about the information technology precursors and the back-history of the institutions that helped Wikipedia to work.
I learned much about the background, values, and choices of Wikipedia entrepreneur “Jimbo” Wales. (Jimbo Wales seems not to be perfect, but on balance to be one of the ‘good guys’ in the world—one of those entrepreneurs who can be admired for something beyond their particular entrepreneurial innovation.)
Lih’s book also does a good job of sketching the problems and tensions within Wikipedia.
I believe that Wikipedia is a key step in the development of faster and better institutions of knowledge generation and communication. I also believe that substantial further improvements can and will be made.
Most importantly, I think that you can only go so far with volunteers–ways must be found to reward and compensate.
In the meantime, much can be learned from Lih. In the next few weeks, I will be quoting a few passages that I found especially illuminating.

Book discussed:
Lih, Andrew. The Wikipedia Revolution: How a Bunch of Nobodies Created the World’s Greatest Encyclopedia. New York: Hyperion, 2009.

Global Warming “Consensus” Achieved by Suppressing Skeptical Research

(p. A25) When scientists make putative compendia of that literature, such as is done by the U.N. climate change panel every six years, the writers assume that the peer-reviewed literature is a true and unbiased sample of the state of climate science.

That can no longer be the case. The alliance of scientists at East Anglia, Penn State and the University Corporation for Atmospheric Research (in Boulder, Colo.) has done its best to bias it.
A refereed journal, Climate Research, published two particular papers that offended Michael Mann of Penn State and Tom Wigley of the University Corporation for Atmospheric Research. One of the papers, published in 2003 by Willie Soon and Sallie Baliunas (of the Harvard-Smithsonian Center for Astrophysics), was a meta-analysis of dozens of “paleoclimate” studies that extended back 1,000 years. They concluded that 20th-century temperatures could not confidently be considered to be warmer than those indicated at the beginning of the last millennium.
In fact, that period, known as the “Medieval Warm Period” (MWP), was generally considered warmer than the 20th century in climate textbooks and climate compendia, including those in the 1990s from the IPCC.
Then, in 1999, Mr. Mann published his famous “hockey stick” article in Geophysical Research Letters (GRL), which, through the magic of multivariate statistics and questionable data weighting, wiped out both the Medieval Warm Period and the subsequent “Little Ice Age” (a cold period from the late 16th century to the mid-19th century), leaving only the 20th-century warming as an anomaly of note.
Messrs. Mann and Wigley also didn’t like a paper I published in Climate Research in 2002. It said human activity was warming surface temperatures, and that this was consistent with the mathematical form (but not the size) of projections from computer models. Why? The magnitude of the warming in CRU’s own data was not as great as in the models, so therefore the models merely were a bit enthusiastic about the effects of atmospheric carbon dioxide.
Mr. Mann called upon his colleagues to try and put Climate Research out of business. “Perhaps we should encourage our colleagues in the climate research community to no longer submit to, or cite papers in, this journal,” he wrote in one of the emails. “We would also need to consider what we tell or request of our more reasonable colleagues who currently sit on the editorial board.”
After Messrs. Jones and Mann threatened a boycott of publications and reviews, half the editorial board of Climate Research resigned. People who didn’t toe Messrs. Wigley, Mann and Jones’s line began to experience increasing difficulty in publishing their results.

For the full commentary, see:
PATRICK J. MICHAELS. “OPINION; How to Manufacture a Climate Consensus; The East Anglia emails are just the tip of the iceberg.” The Wall Street Journal (Fri., DECEMBER 18, 2009): A25.
(Note: the online version of the article is dated DECEMBER 17, 2009.)

Establishments Assume New Methods Are Unsound Methods

(p. 188) For the next two years, Conway coordinated her efforts under Sutherland at PARC with Mead’s ongoing work at Caltech. But she was frustrated with the pace of progress. There was no shortage of innovative design ideas; computerized design tools had advanced dramatically since Mead’s first efforts several years before. Yet the industry as a whole continued in the old rut. As Conway put it later, the problem was “How can you take methods that are new, methods that are not in common use and therefore perhaps considered unsound methods, and turn them into sound methods?” [Conway’s italics].

She saw the challenge in the terms described in Thomas Kuhn’s popular book The Structure of Scientific Revolutions. it was the problem that took Boltzmann to his grave. It was the problem of innovation depicted by economist Joseph Schumpeter in his essays on entrepreneurship: new systems lay waste to the systems of the past. Creativity is a solution for the creator and the new ventures he launches. But it wreaks dissolution–“creative destruction,” in Schumpeter’s words– for the defenders of old methods. In fact, no matter how persuasive the advocates of change, it is very rare that an entrenched establishment will reform its ways. Establishments die or retire or fall in revolution; they only rarely transform themselves.

Source:

Gilder, George. Microcosm: The Quantum Revolution in Economics and Technology. Paperback ed. New York: Touchstone, 1990.
(Note: italics in original.)