The Reason Why Much Science Is Broken: Every Use Of P-Values Is A Logical Fallacy

I made this picture for my talk at the first ever public Broken Science event (videos coming soon):

Everybody has heard the saying “correlation doesn’t imply causation.” Taken loosely, it seems wrong, because when there is causation there is also correlation. And scientists sometimes investigate problems in which they suspect causation is present.

The difficulty is with imply. In the loose, vulgar sense, it means something like goes with. So we’d have “correlation goes with causation”, which is true, more or less.

The original slogan is meant to be logical, where there is a strict definition of imply. As in logically follows. This means another way of saying the slogan is “causation does not follow from correlation.” Or “correlation could be a coincidence and not causation.”

There are also difficulties with coincidence. Here I mean it in the sense of Diaconis and Mosteller, which are events which seem surprising and unusually close together in essential aspects, yet have no causal link, with a stress on no causal link. (We also unfortunately sometimes use coincidence to mean there is a causal link, and that it is suspicious.)

You have the idea.

In any case, every single time, with no exceptions, no, not even if you are the greatest scientist in the world, when we say “Here is a correlation, thus it is causation”, it is a fallacy. A strict logical fallacy. Every time. Each time. No wiggling out. Even if your grant is large.

From this it does not follow that causation is never present when correlation is. It may be. The more wisely you choose your correlations the more likely causation is there, somewhere, maybe.

But it remains a simple logical inescapable fact that moving from correlation to causation without understanding the causation, without putting the premises of the causation into your argument, is a fallacy.

Every use of a p-value is a fallacy.

The p-value says, “The null is true that a coincidence happened, and here is the probability of something that happened; therefore, my correlation is causation.” This is false, a fallacy.

Again, every use of a p-value is a fallacy. Every as in every. Each one. Every time. No exceptions. All p-values are logical fallacies. This is so no matter what. It is so even if you say, “Yes, p-values have problems, but there are some good uses for them.” Whether or not that is true, and I say it is not, each time a p-value is used a logical fallacy has been invoked.

If, that is, a causation has been imagined, hinted at, teased, whispered, or outright declared when the p is wee.

Which always happens. That, after all, is why the p-value is used. To “reject” the hypothesis of correlation—which is to say, of coincidence. To reject the “null” is to say “The correlation I have observed is not a coincidence. Something is going on here.” And that something is causation.

Perhaps not direct causation, as in the parameter associated with the wee p is not thought in itself to be the cause, but is itself caused by something else, perhaps not measured. Still, this is causation. So this use of the p-value is a strict logical fallacy, too.

Coincidences abound. There are, if not an infinite number of interesting correlations, then there are so many that it is near enough infinity. Which means there is an infinity, or near enough, of data sets that will evince wee p-values, but where no causality is present. And if we merely dress up the data sets in some pretty science language, we can make our coincidences sound like science.

Which is what happens.

I’ve used this example countless times, but it hasn’t worn out its usefulness yet. The site Spurious Correlations—spurious indicating lack of cause—has a graph of US spending on science, space and technology with Suicides by hanging, strangulation and suffocation. Nearly perfect correlation, a cute coincidence. Spurious.

But that correlation would give a wee p-value. Which I use to reject the rejection of the “null” hypothesis, because why? Because it’s obvious there is no causal connection between the two.

Which judgement—our premise of supposing there is no causal connections—proves we often bring in information outside the p-value paradigm (sounds like a 80s cold war spy novel). So why not bring it every time?

Indeed, why not. That’s kinda sorta what Bayesians do, except they obsess over unobservable parameters to which they also assign bizarre causal powers, which is a story for another time.

Subscribe or donate to support this site and its wholly independent host using credit card click here. Or use the paid subscription here. For Zelle, use my email: matt@wmbriggs.com, and please include yours so I know who to thank.