Category: Methodology

Improved AI Models Do Worse at Identifying Prime Numbers

(p. A2) . . . new research released this week reveals a fundamental challenge of developing artificial intelligence: ChatGPT has become worse at performing certain basic math operations.

The researchers at Stanford University and the University of California, Berkeley said the deterioration is an example of a phenomenon known to AI developers as drift, where attempts to improve one part of the enormously complex AI models make other parts of the models perform worse.

“Changing it in one direction can worsen it in other directions,” said James Zou, a Stanford professor who is affiliated with the school’s AI lab and is one of the authors of the new research. “It makes it very challenging to consistently improve.”

. . .

The goal of the team of researchers, consisting of Lingjiao Chen, a computer-science Ph.D. student at Stanford, along with Zou and Berkeley’s Matei Zaharia, is to systematically and repeatedly see how the models perform over time at a range of tasks.

Thus far, they have tested two versions of ChatGPT: version 3.5, available free online to anyone, and version 4.0, available via a premium subscription.

The results aren’t entirely promising. They gave the chatbot a basic task: identify whether a particular number is a prime number. This is the sort of math problem that is complicated for people but simple for computers.

Is 17,077 prime? Is 17,947 prime? Unless you are a savant you can’t work this out in your head, but it is easy for computers to evaluate. A computer can just brute force the problem—try dividing by two, three, five, etc., and see if anything works.

To track performance, the researchers fed ChatGPT 1,000 different numbers. In March, the premium GPT-4, correctly identified whether 84% of the numbers were prime or not. (Pretty mediocre performance for a computer, frankly.) By June its success rate had dropped to 51%.

. . .

The phenomenon of unpredictable drift is known to researchers who study machine learning and AI, Zou said. “We had the suspicion it could happen here, but we were very surprised at how fast the drift is happening.”

For the full commentary, see:

Josh Zumbrun. “THE NUMBERS; AI Surprise: It’s Unlearning Basic Math.” The Wall Street Journal (Saturday, Aug. 5, 2023): A2.

(Note: ellipses added.)

(Note: the online version of the commentary has the date August 4, 2023, and has the title “THE NUMBERS; Why ChatGPT Is Getting Dumber at Basic Math.”)

Bullshit Is Worse Than a Lie

(p. A17) Professor Frankfurt became best known for a single, irreverent paper largely unrelated to his life’s main work.

The paper, written in the mid-1980s under the same title as his eventual book, discussed what to his mind was a pervasive but underanalyzed feature of our culture: a form of dishonesty akin to lying but even less considerate of reality. Whereas the liar is at least mindful of the truth (if only to avoid it), the “bullshitter,” Professor Frankfurt wrote, is distinguished by his complete indifference to how things are.

Whether its purveyor is an advertiser, a political spin doctor or a cocktail-party blowhard, he argued, this form of dishonesty is rooted in a desire to make an impression on the listener, with no real interest in the underlying facts. “By virtue of this,” Professor Frankfurt concluded, “bullshit is the greater enemy of truth than lies are.”

. . .

For all this sang-froid, Professor Frankfurt was heartfelt in his philosophical pursuits. Throughout his career, he was drawn to lines of inquiry — about freedom, love, selfhood and purpose — that he said appealed to him not only as an academic but also “as a human being trying to cope in a modestly systematic manner with the ordinary difficulties of a thoughtful life.”

For the full obituary, see:

James Ryerson. “Harry G. Frankfurt, a Philosopher Eager to Cut the Bull, Dies at 94.” The New York Times (Tuesday, July 18, 2023): A17.

(Note: ellipsis added.)

(Note: the online version of the obituary has the date July 17, 2023, and has the title “Harry G. Frankfurt, Philosopher With a Surprise Best Seller, Dies at 94.”)

Frankfurt’s best-known book is:

Frankfurt, Harry G. On Bullshit. Princeton, NJ: Princeton University Press, 2005.

Scientists Had Political Motives for Dismissing Wuhan Lab-Based Covid Origin

(p. A17) On March 17, 2020, the journal Nature Medicine published a paper by five scientists, “The Proximal Origin of SARS-CoV-2,” that dismissed “any type of laboratory based scenario” for the origin of the pandemic. It was cited by thousands of news outlets to claim that the virus emerged naturally. But Slack messages and emails subpoenaed and released by the House Oversight Select Subcommittee on the Coronavirus Pandemic suggest that some of the authors didn’t believe their own conclusions. Before, during and even after the publication of their paper, they worried privately that Covid-19 was caused by a laboratory escape, perhaps even of a genetically engineered virus.

. . .

On April 16, a month after publication, Mr. Andersen wrote that “I’m still not fully convinced that no culture was involved” and “we also can’t fully rule out engineering”—i.e., that the virus not only was released from the lab but had been genetically manipulated there. He worried about the Wuhan lab’s research on live SARS-like viruses from bats at low biosafety levels: “it’s definitely concerning work, no question about it.”

So why did they publish a paper denying that laboratory origin was plausible? The answer may lie in their messages. In early February 2020, Mr. Rambaut wrote: “Given the s— show that would happen if anyone serious accused the Chinese of even accidental release, my feeling is we should say that given there is no evidence of a specifically engineered virus, we cannot possibly distinguish between natural evolution and escape so we are content to ascribing it to natural processes.”

Mr. Andersen replied: “I totally agree that that’s a very reasonable conclusion. Although I hate when politics is injected into science—but it’s impossible not to.”

. . .

To adjust the conclusions in a scientific paper for political reasons isn’t part of the scientific process. The world was misled with serious consequences.

For the full commentary, see:

Matt Ridley and Alina Chan. “The Covid Lab-Leak Deception.” The Wall Street Journal (Thursday, July 27, 2023): A17.

(Note: ellipses added.)

(Note: the online version of the commentary has the date July 26, 2023, and has the same title as the print version.)

Scientist Latta Knows, but Cannot Prove, That Ivory-Billed Woodpecker Is Not Extinct

I respect and admire Dr. Latta for having the courage to affirm what he saw with his own two eyes. Other scientists should not be so quick to ‘give him the bird’ (so to speak ;).

(p. A19) If there’s new hope, it’s blurry. What’s certain: The roller coaster tale of the ivory-billed woodpecker, a majestic bird whose presumed extinction has been punctuated by a series of contested rediscoveries, is going strong.

The latest twist is a peer-reviewed study Thursday [May 18, 2023] in the journal Ecology and Evolution presenting sighting reports, audio recordings, trail camera images and drone video. Collected over the last decade in a Louisiana swamp forest, the precise location omitted for the birds’ protection, the authors write that the evidence suggests the “intermittent but repeated presence” of birds that look and behave like ivory-billed woodpeckers.

But are they?

“It’s this cumulative evidence from our multiyear search that leaves us very confident that this iconic species exists, and it persists in Louisiana and probably other places as well,” said Steven C. Latta, one of the study’s authors and director of conservation and field research at the National Aviary, a nonprofit bird zoo in Pittsburgh that helps lead a program that searches for the species.

But Dr. Latta acknowledges that no single piece of evidence is definitive, and the study is carefully tempered with words like “putative” and “possible.”

. . .

. . . Dr. Latta, the study co-author, insisted that he had seen one clearly with his own eyes. He was in the field in 2019 to set up recording units, and he figures he spooked the bird. As it flew up and away, he got a close, unimpeded view of its signature markings.

“I couldn’t sleep for, like, three days,” Dr. Latta said. “It was because I had this opportunity and I felt this responsibility to establish for the rest of the world, or at least the conservation world, that this bird actually does exist.”

For the full story, see:

Catrin Einhorn. “Experts Strive to Prove ‘This Bird Actually Does Exist’.” The New York Times (Friday, May 19, 2023): A19.

(Note: ellipses, and bracketed date, added. The online version of the article says that the print version appears on p. 21. My national edition of the print version appeared on p. 19.)

(Note: the online version of the story was updated May 18, 2023, and has the title “A Vanished Bird Might Live On, or Not. The Video Is Grainy.”)

The peer-reviewed paper, co-authored by Latta and mentioned above, is:

Latta, Steven C., Mark A. Michaels, Thomas C. Michot, Peggy L. Shrum, Patricia Johnson, Jay Tischendorf, Michael Weeks, John Trochet, Don Scheifler, and Bob Ford. “Multiple Lines of Evidence Suggest the Persistence of the Ivory-Billed Woodpecker (Campephilus Principalis) in Louisiana.” Ecology and Evolution 13, no. 5 (2023): e10017 https://onlinelibrary.wiley.com/doi/full/10.1002/ece3.10017.

Data Set Too Small to Support Claim of 250,000 Annual ER Deaths Due to Misdiagnosis

(p. A17) A shocking headline recently claimed that every year 250,000 people in the U.S. die after misdiagnosis in the emergency room. Even more shocking, the statistic was extrapolated from the death of one man—in a Canadian emergency room more than a decade ago.

. . .

The statistical methods used to arrive at the report’s estimate of 250,000 deaths are very bad, resulting in inaccurate findings that exaggerate potential harm in ERs. The estimate was derived from a single study that included only 503 patients discharged from two Canadian emergency rooms from August to December 2004. Researchers found that among the 503 patients, one person unexpectedly died related to a delay in diagnosis by an ER physician. The patient had signs of an aortic dissection—a tear in the major vessel that carries blood from the heart. For reasons we don’t know, the diagnosis was delayed for seven hours.

The goal of the Canadian study was to measure all kinds of medical errors, not to estimate the death rate from erroneous or late diagnoses. The sample size wasn’t big enough for that. Had nobody in the study sample died, would that mean that ERs never make fatal errors? Obviously not.

The AHRQ report misused this single death to estimate a death rate across the entire U.S. Dividing one death by 503 patients, the researchers estimate a death rate of 0.2%. They then multiply 0.2% by total annual ER visits in the U.S.—130 million—and come up with 250,000 deaths.

For the full commentary, see:

Kristen Panthagani. “A Study Sounds a False Alarm About America’s Emergency Rooms.” The Wall Street Journal (Friday, Dec. 30, 2022): A17.

(Note: ellipsis added.)

(Note: the online version of the commentary has the date December 29, 2022, and has the same title as the print version.)

China Still Seeks to Censor International Journals and Databases on Covid

(p. A1) Early in 2020, on the same day that a frightening new illness officially got the name Covid-19, a team of scientists from the United States and China released critical data showing how quickly the virus was spreading, and who was dying.

The study was cited in health warnings around the world and appeared to be a model of international collaboration in a moment of crisis.

Within days, though, the researchers quietly withdrew the paper, which was replaced online by a message telling scientists not to cite it. A few observers took note of the peculiar move, but the whole episode quickly faded amid the frenzy of the coronavirus pandemic.

What is now clear is that the study was not removed because of faulty research. Instead, it was withdrawn at the direction of Chinese health officials amid a crackdown on science. That effort kicked up a cloud of dust around the dates of early Covid cases, like those reported in the study.

“It was so hard to get any information out of China,” said one of the authors, Ira Longini, of the University of Florida, who described the back story of the removal publicly for the first time in a recent interview. “There was so much covered up, and so much hidden.”

That the Chinese government muzzled scientists, hindered international investigations and censored online discussion of the pandemic is well documented. But Beijing’s stranglehold on information goes far deeper than even many pandemic researchers are aware of. Its censorship campaign has targeted international journals and scientific databases, shaking the foundations of shared scientific knowledge, a New York Times investigation found.

Under pressure from their government, Chinese scientists have withheld data, withdrawn genetic sequences from public databases and altered crucial details in journal submissions. Western journal editors enabled those efforts by agreeing to those edits or withdrawing papers for murky reasons, a review by The Times of over a dozen retracted papers found.

Groups including the World Health Organization have given credence to muddled data and inaccurate timelines.

For the full story, see:

Mara Hvistendahl and Benjamin Mueller. “China Censors Are Thwarting Covid Science.” The New York Times (Monday, April 24, 2023): A1 & A9.

(Note: the online version of the story has the date April 23, 2023, and has the title “Chinese Censorship Is Quietly Rewriting the Covid-19 Story.”)

Critique of Fed Claim That a Year of Above Average Temps Increases Odds of a Recession

(p. A17) . . . recently I published a critique of a study from the Federal Reserve Board claiming that a year of above-normal temperatures in countries around the world makes economic contraction more likely.

. . .

There are two main reasons why the Fed study appeared at first to show a statistically significant effect of temperatures on economic growth. First, each country in the sample had equal weight in the analysis. China had the same weight as St. Vincent though China’s population is 13,000 times as large. Equal weighting means that some small countries with unusual histories of economic growth greatly influenced the results.

The paper’s results disappeared when countries like Rwanda and Equatorial Guinea—which had economic catastrophes and bonanzas unrelated to climate change—were omitted. Omitting similar countries representing less than 1% of world gross domestic product was enough to eliminate the paper’s result. The complicated statistical techniques used in the Fed study magnified the influence of these unusual countries.

There’s a second reason why the Fed study appears to find that temperature affects growth: Many poor countries have warm climates. A warm climate doesn’t preclude economic growth, as is demonstrated by Florida, Arizona, Taiwan, Singapore and several Persian Gulf states. But the average poor country is warmer than the average rich country. Debate continues as to whether this correlation is random or causal, but the hypothesis of the Fed paper is that year-to-year increases in temperature reduce annual economic growth. The paper claims that its method controls for long-term differences in climate, but using simulated data I found that the Fed paper’s method can be fooled into finding an effect that doesn’t exist.

For the full commentary, see:

David Barker. “The Fed’s Climate Studies Are Full of Hot Air.” The Wall Street Journal (Monday, April 10, 2023): A17.

(Note: ellipses added.)

(Note: the online version of the commentary has the date April 9, 2023, and has the same title as the print version.)

Barker’s critique mentioned above is:

Barker, David. “Temperature and Economic Growth: Comment on Kiley.” Econ Journal Watch 20, no. 1 (March 2023): 69–84.

Mainstream Media Suppresses Scientific Debate on Cochrane Meta-Analysis of Mask Efficacy

(p. A11) On Jan. 30, [2023] the Cochrane Collaboration, highly regarded for its rigorous systematic reviews, published an update of its meta-analysis of masking and other physical methods to prevent respiratory illnesses. It found no strong evidence for masking, and the initial media response was silence. After conservative media covered the study, the mainstream press went on the attack. The Washington Post, the New York Times, the Atlantic and others piled on.

As an epidemiologist, I hoped the review might dent the politicized discourse surrounding masks and other prevention measures. I sent an article to Time magazine, which had recently published my work.

. . .

Eventually the editor . . . asked for some further changes, which I readily accepted. “I think we’re set,” he wrote. “My colleague will finalize and publish.” I heard nothing from him until two weeks later, when he finally confirmed Time had decided to kill the piece.

. . .

“Follow the science,” they said throughout the pandemic. You can’t do that if you suppress scientific debate.

For the full commentary, see:

Daniel Halperin. “Media Keep Stifling the Covid Debate.” The Wall Street Journal (Saturday, April 1, 2023): A11.

(Note: ellipses, and bracketed year, added.)

(Note: the online version of the commentary has the date March 31, 2023, and has the same title as the print version.)

The Cochrane meta-analysis mentioned above is:

Jefferson, T., L. Dooley, E. Ferroni, L. A. Al-Ansary, M. L. van Driel, G. A. Bawazeer, M. A. Jones, T. C. Hoffmann, J. Clark, E. M. Beller, and et al. “Physical Interventions to Interrupt or Reduce the Spread of Respiratory Viruses.” Cochrane Database of Systematic Reviews (Jan. 30 2023).

“The Reliability of Science Is Based” on Free Speech

Theoretical physicist Carlo Rovelli’s argument should be pondered by global warming and Covid scientists who want to censor and cancel those with whom they disagree. They should also read John Stuart Mill’s On Liberty.

(p. C5) Science is a process that builds upon existing theories and knowledge by continuously revising them. Every aspect of scientific knowledge can be questioned, including the general rules of thinking that appear to be most certain.

. . .

Consider a folk healer’s herbal medicine. Can we say this treatment is “scientific”? Yes, if it is proven to be effective, even if we have no idea why it works. In fact, several common medications used today have their origin in folk treatments, and we are still not sure how they work. This does not imply that folk treatments are generally effective. To the contrary, most of them are not. What distinguishes scientific medicine from nonscientific medicine is the readiness to seriously test a treatment and to be ready to change our minds if something is shown not to work.

Exaggerating a bit, one could say that the core of modern medicine is not much more than the accurate testing of treatments. A homeopathic doctor is not interested in rigorously testing his remedies: He continues to administer the same remedy even if a statistical analysis shows that the remedy is ineffective. He prefers to stick to his theory. A research doctor in a modern hospital, on the contrary, must be ready to change his theory if a more effective way of understanding illness, or treating it, becomes available.

. . .

What makes modern science uniquely powerful is its refusal to believe that it already possesses ultimate truth. The reliability of science is based not on certainty but on a radical lack of certainty. As John Stuart Mill wrote in “On Liberty” in 1859, “The beliefs which we have most warrant for, have no safeguard to rest on, but a standing invitation to the whole world to prove them unfounded.”

. . .

There is no secure method for avoiding error. Our point of departure is always just the ramshackle, error-filled totality of what we think we know. But uncertainty does not make knowledge worthless. If our theory is contradicted by experiment, this remains a real fact, solid as rock, even if we don’t yet know with clarity where our mistake lies. The fact that the assumptions in our reasoning can be mistaken doesn’t change the fact that scientific reasoning is our best cognitive tool.

For the full essay, see:

Carlo Rovelli. “The Best Reason to Trust Science.” The Wall Street Journal (Saturday, March 11, 2023): C5.

(Note: ellipses added.)

(Note: the online version of the essay has the date March 9, 2023, and has the same title as the print version.)

Rovelli’s essay quoted above is based on his book:

Rovelli, Carlo. Anaximander and the Birth of Science. New York: Riverhead Books, 2023 (2011).

Mill’s wonderful defense of freedom, mentioned above, is:

Mill, John Stuart. On Liberty and Other Essays, Oxford World’s Classics. New York: Oxford University Press, USA, 2008 (1859).

Cochrane Study Finds No Benefits of Mandatory Masking

During the pandemic, I wrote an op-ed piece advocating the voluntary (not mandatory) use of masks. I still believe that, based on the mechanics of disease spread, and the mechanics of physically blocking virus particles, that masks can have a modest effect in reducing the viral load we spread to others. I also still believe in free speech and believe that it was wrong to censor those who were skeptical of masks.

(p. A19) The most rigorous and comprehensive analysis of scientific studies conducted on the efficacy of masks for reducing the spread of respiratory illnesses — including Covid-19 — was published late last month. Its conclusions, said Tom Jefferson, the Oxford epidemiologist who is its lead author, were unambiguous.

“There is just no evidence that they” — masks — “make any difference,” he told the journalist Maryanne Demasi. “Full stop.”

But, wait, hold on. What about N-95 masks, as opposed to lower-quality surgical or cloth masks?

“Makes no difference — none of it,” said Jefferson.

What about the studies that initially persuaded policymakers to impose mask mandates?

“They were convinced by nonrandomized studies, flawed observational studies.”

. . .

These observations don’t come from just anywhere. Jefferson and 11 colleagues conducted the study for Cochrane, a British nonprofit that is widely considered the gold standard for its reviews of health care data. The conclusions were based on 78 randomized controlled trials, six of them during the Covid pandemic, with a total of 610,872 participants in multiple countries. And they track what has been widely observed in the United States: States with mask mandates fared no better against Covid than those without.

No study — or study of studies — is ever perfect. Science is never absolutely settled. What’s more, the analysis does not prove that proper masks, properly worn, had no benefit at an individual level. People may have good personal reasons to wear masks, and they may have the discipline to wear them consistently. Their choices are their own.

. . .

The C.D.C.’s increasingly mindless adherence to its masking guidance is none of those things. It isn’t merely undermining the trust it requires to operate as an effective public institution. It is turning itself into an unwitting accomplice to the genuine enemies of reason and science — conspiracy theorists and quack-cure peddlers — by so badly representing the values and practices that science is supposed to exemplify.

It also betrays the technocratic mind-set that has the unpleasant habit of assuming that nothing is ever wrong with the bureaucracy’s well-laid plans — provided nobody gets in its way, nobody has a dissenting point of view, everyone does exactly what it asks, and for as long as officialdom demands. This is the mentality that once believed that China provided a highly successful model for pandemic response.

For the full commentary, see:

Bret Stephens. “‘Do Something’ Is Not Science.” The New York Times (Wednesday, February 22, 2023): A19.

(Note: ellipses added.)

(Note: the online version of the commentary has the date Feb. 21, 2023, and has the title “The Mask Mandates Did Nothing. Will Any Lessons Be Learned?”)

Heat Deaths Rise Mostly Due to Rise in Fragile Aging Population

(p. A17) One recent and much-cited Lancet report appears deliberately deceptive.

The study offers a frightening statistic: Rapidly rising temperatures have increased annual global heat deaths among older people by 68% in less than two decades. That stark figure has been cited all over, from the BBC and Time to the Washington Post and the Times of India, the world’s largest-selling English-language daily.

. . .

Annual heat deaths have increased significantly among people 65 and older world-wide. The average deaths per year increased 68% from the early 2000s to the late 2010s. But that is almost entirely because there are so many more older people today than there were 20 years ago, in no small part thanks to medical innovations that keep us alive longer. Measured across the same time span the Lancet maps heat deaths, the number of people 65 and older has risen by 60%, or almost as much as heat deaths. When the increase in heat mortality is adjusted for this population growth, the actual rise that can be attributed to rising temperatures is only 5%.

It is hard not to see the Lancet study’s failure to adjust this figure as a deliberate act of deception.

For the full commentary, see:

Bjorn Lomborg. “The Lancet’s ‘Heat Death’ Deception.” The Wall Street Journal (Saturday, November 5, 2022): A17.

(Note: ellipsis added.)

(Note: the online version of the commentary has the date November 4, 2022, and has the title “Climate Change and the Lancet’s ‘Heat Death’ Deception.”)