Models Only Say What They’re Told to Say -- The Paper!

The paper (more like a glorified note, really) "Models Only Say What They’re Told to Say" (by me) will appear in the Springer book Prediction and Causality in Econometrics and Related Topics shortly. I'm providing a PDF of the paper in advance. There is also a coronadoom example or two!

The paper has a fraction of math in it, but none anybody who paid attention in middle school would have trouble with. It is mostly, as with most of my professional work, a matter of philosophy.

Now on that subject, I want to clear up a misunderstanding some have about the necessarily true statement that "all models only say what they're told to say."

First, it is indeed necessarily true, the proof of which you can find in the paper, and I won't bore you with here.

Second, it is neither good nor bad that "all models only say what they're told to say." It is just The Way Things Are. It is also no limitations on models or on our understanding of how the world works. Our understanding is models.

Nevertheless, it often serves as a pithy reminder to say "all models only say what they're told to say" when people are being terrorized by a model, as they were during the (still lingering) coronadoom panic, and as they are in an increasing number of ways.

Models can be good or bad. The ones we're terrorized with, and called "Denier!" for doubting, are the bad ones. Which is why it's good to highlight that the model is only saying what its builders want it to say.

Here's an example from the paper, adapted and modified from The Price of Panic.

Here is a press headline from a Minnesota news source on 13 May 2020: “Updated Model Predicts COVID-19 Peak In Late-July With SAHO Extended Through May; 25K Deaths Possible” [the source is listed in the paper]. This was of course produced during the coronavirus panic.

The article stated:

An updated model from the University of Minnesota and state’s health department is predicting that COVID-19 cases will peak in late-July with 25,000 deaths possible -- if the stay-at-home order is extended until the end of May...In Scenario 5 [the model scenario relied up by government], the stay-at-home order is extended for all until the end of May. With that happening, the model predicts that the COVID-19 peak will happen on July 27, with the top intensive care units (ICU) demand being 4,000 and 25,000 possible deaths.

Another estimate, Scenario 4, predicts that if the stay-at-home order is extended by a month into mid-July, the peak would occur on July 13 with 3,700 as the top ICU demand and 22,000 possible deaths.

In previous briefings, Minnesota’s Governor Tim Walz had asked Minnesotans not to focus on specific numbers, but rather focus on when the peaks might occur. “Modeling was never meant to provide a number,” Governor Walz said on Wednesday. “It was meant to show trend and direction, that if you social distance you buy more time.”

This is false. It, and many similar comments made by numerous official sources during the panic, were common. They were all not only false, but misleading, too. Enforced stay-at-home social distance working was an input to all of these models. We cannot therefore point to model output and say, at least with a straight face, “See? The model says stay-at-home social social distancing works. Which is why we need to implement it.” This mistake made countless times during the crisis. This is detailed at length in [16], including a discussion of how the World Health Organization made the same mistake, based on a high-school science project, to conclude social distancing worked, see [17], when, as always, this was a premise of the model.

The models reported on here were built saying social distancing reduces death. This assumption was an integral part of the models. It was not a “discovery” of the models, it was a condition of them. The models had to say social distancing worked because they started with the premise social distancing worked.

You cannot “discover” stay-at-home social distancing worked via any model---though it may be discovered via after-the-fact observation. You had to have built in that possibility in the first place. You knew in advance that it worked because that’s what you told the model.

You cannot run the model, wait for the output, run to your Governor and say “The latest model says social distancing works.” If your Governor had any sense he would say, “Didn’t you write the model code? And didn’t the code say somewhere that social distancing worked?”

The rush to embrace these models as if they were oracular says more about the goals and desires and decision makers during the panic than it says about model making.

Incidentally, according to the CDC, as of 13 September, attributed COVID-19 deaths in Minnesota were 1,803, with the peak occurring in mid-May, [18]. These represent all attributed deaths, including those deaths where the individuals died of multiple causes. The error of the models relied upon to make decisions was therefore at least 12 times, an enormous and horrendous mistake. The restrictions is Minnesota did stay in place, but were interrupted by the Minneapolis riots, a time when social distancing was not observed. If the models had any bearing on reality, since social distancing did not obtain, the deaths should have been higher than 25,000.

You can read the rest in the paper.

Subscribe or donate to support this site and its wholly independent host using credit card or PayPal click here