Reminder: The Thursday Class is only for those interested in studying uncertainty. I don’t expect all want to read these posts. Pease don’t feel like you must. Yet, I have nowhere else to put them. Your support makes this Class possible for those who need it. Thank you.
Once again, the eternal battle against P-values. I tried to write this one with as little technical apparatus as possible to appeal to a larger audience. However, it’s still part of the Class and not easy going.
Video
Links: YouTube * Twitter – X * Rumble * Bitchute * Class Page * Jaynes Book * Uncertainty
HOMEWORK: Review.
Lecture
The other day, as is my wont, after an intelligent and worthy fellow was posting on X/Twitter about some study and lamenting the p-values were not quite there, I pointed out not to use these creatures, and that every p-value use is a formal fallacy. This is exactly the kind of behavior that makes we statisticians such fun a parties.
Our man said this:
In the Finnish basic income study, women randomly given extra monthly income were less likely to get pregnant. Women whose husband/spouse was randomly given more income were more likely to get pregnant.
At least, that’s what they reported, but the p values are kinda shit, so we can’t really say.
Now I in no way want to make this about the gentlemen in question, who is, as I said without guile, intelligent and worthy. So I will not link to this conversation as I do not wish the man any grief. But because the course of the conversation I had with him is so common (to me), I thought I would reproduce the best bits of it, all in service to this question:
If you think there are “good uses of P-values”, please tell us what they are.
I say there are none; I say that each and every use of one, to the services which they are put, is a formal fallacy. I say they ought never be used. I say they are horrible tools when better one are available, and for free!
But maybe I’m wrong. It has happened. Many will naturally dispute this, but I assure you, as disquieting as it is to contemplate, it is true. It may be true here, too. I do not think it is. Yet, in the spirit of fair play, I’m willing to test my ideas.
I have written much on this subject, and recorded many videos. I offered one of the videos to our man above after he said p-values “have some good uses”. He refused to watch it. Which is the right move. I, too, do not watch unsolicited videos, except rarely. I prefer words over video by hundreds to one. Which is why, after refusing to watch the videos, I sent him links to some of my award-eligible articles on the subject.
Which he would not read. And which he ought not to have, either. Because none of the screeds are short, each takes a substantial investment in time, and why should anybody make this investment over what is, after all, such a seemingly small question?
So I gave it to him short and sweet, in a tweet. Which he would not consider, except to repeat he understood me not, and that p-values had some good uses, and that anyway he wasn’t using them in the way I accused him of. And he was right to dismiss these attempts, too. Because who the hell am I? Just some random (ha ha!) guy on the interwebs chirruping, unwanted, about some stupid measure which every other statistician he’s ever worked with assured him is fine, and have “good uses.”
This is why the battle to remove these sad sorry statistics is so long and hard. Inertia. Consider: I could not argue via authority, which too is a fallacy, but which is a damned useful tool, because I have no authority of any kind. And can’t gain any, for many reasons, to include my acerbic personality, but also because I, and a few compatriots, cannot convince people like our man of the rightness of our quest.
So I propose to listen to any of you who will respond, and I’ll show you in the next Class (or next after next) why your attempts fail.
Our Man’s Example
I’ll start with the attempt used by our man, an attempt he denied.
Let us take this sentence: “Women given extra monthly income were less likely to become pregnant than women whose husbands were given extra monthly income.”
If we assume that “were less likely” to mean something like “observed proportionately” (less or more), then, assuming the accuracy of our man’s data (in science anything can happen to data), that sentence is true. Another way to say that is that the sentence has probability 1, conditional on these assumptions and data.
If all we wanted to know about was this sentence, given these assumptions, then we are done.
We don’t need p-values. What would we do with them? We don’t need “confidence intervals”, Bayes factors, information criteria, models of any kind. What use are any of these here? None.
Here are two entirely different questions: Was the extra monthly income of men the cause of their wives’ greater frequency of births? Was the extra monthly income of women the cause of their lower frequency of births?
The money can’t have always been a cause, because not every wife of every man granted extra monthly income became pregnant, and some women who were given extra monthly income became pregnant. But money could have been a cause sometimes, or rather two possible causes, since we have two separate kinds of events.
Can a p-value tell us if either was a cause? No, of course not. P-values cannot tell cause. Every writer on the theory of p-values agrees with that. Can a Bayes factor? No, of course not. Can the estimated value of a parameter in a model tells us if either was a cause? No, of course not.
Can p-values give us the chance, or an approximation to the chance, either of these causes was operative? Not only no, but—well, let’s not be rude. P-values explicitly are anti-probability of cause, or “hypotheses” of any kind.
“Briggs, what is all this about cause? Why are you complicating things?”
Why did they do the experiment if not to test whether their “intervention” was going to cause a change in births? Or, rather, how often the money would cause a change in births. Since this is so, we want to have an analysis method that helps us determine things about this cause. Yes?
As any frequentist theorist will insist, P-values say nothing about this. Of course, everybody who uses P-values uses them to “confirm” cause, or make the proposed cause “more likely” to be true in his mind. Every. This is forbidden by P-value theory. To act against this theory is a cardinal sin. Unforgivable. Everybody does it. Everybody.
We can now interpret that sentence in a new way, one in which we’re not certain about cause, but where we want to know something about future measurements on other women and men (and their wives) who are given extra monthly income. Since writing about that long sentence is a pain in the keister, let’s call the “money-birth” sentence for short. That okay?
If we assume extra money is causal, we can form probability statements like this:
“Pr(Birth | Woman money-birth, Old data, Model) < Pr(Birth | Man money-birth, Old data, Model),”
written to show the explicit assumption on money-birth being causal, in a way, and where we account for the old data we observed (in the first sentence) and on some model which we assume ties these things together. We can also easily form probabilities for any related propositions, too, like
“Pr(Birth | Man money-birth, Old data, Model) > Pr(Birth | Man no-money-birth, Old data, Model),”
where “no-money-birth” indicates no extra money. Or we can condition on some number of new couples and calculate probabilities of numbers of new births under the conditions we specify. And on and on about anything we choose. It’s so easy using straight probability you just have to laugh. And then cry when you realize this simple system was overthrown for P-values, which can do none of these things, yet which all believe they can.
Careful readers will have noticed that proviso about “money-birth being causal, in a way“. That “in a way” is important. The assumption is merely that the money is involved in some way, but not in an obvious way that we can specify. It may be, for instance, the money itself means nothing alone, but that wive’s of the men who get it feel lucky and thus have warm thoughts about making babies. Hey, it’s possible. There is nothing here stating that the cause operates in the same way in each couple, either. There are lots of possibilities. We are agnostic about them all in this model.
This you might have noticed. But few notice something else, to our point about P-value use being a sin against frequentist theory. But it’s devastating. It’s that in the discussions to papers like this one, the authors will have no hesitation whatsoever about discussing the (potential) cause and its many possibilities. They will say, or hint, that some of these possibilities are more likely than others. This, too, is utterly forbidden in P-value theory. And is anyway not needed, because if you’re going to calculate probabilities about causes, why not do it the right way?
The P-value also requires a model, and the same model as in the probability statements. In P-value theory, it is forbidden to put probabilities on “hypotheses”, or in plain English, sentences (which all are free to do in just-plain probability). Only statements about data can “have” probabilities, because in frequentist theory probabilities are, in a sense, alive. They imbue data, or “events”. They are a real physical presence, and have energy, somehow, that can be measured. This is why frequentists talk about “true values” of probabilities, or worse, parameters inside ad hoc models. and, when these true values cannot be had, they speak of “estimating” them. This is like “estimating” the length of unicorn horns.
The P-value is this: it (a) first imagines that same experiment that gave rise to the original data is repeated an infinite (no less!) number of times, (b) assumes the posited cause is FALSE, and (c) then calculates the probability of data that was not seen if that unseen data were to happen in these experimental repetitions, and finally (d) it makes a decision about cause for you, stating that it is false—or might someday be false; which (e) everybody takes to mean true.
If the P-value is less than the magic number, it is said the results are “statistically significant.” And what is “statistically significant”? Only that the P-value is less the magic number. A circular definition.
“No, Briggs, it means the results were not due to chance.”
Due to? You mean caused?
“Well, I guess so.”
But chance cannot cause anything. Chance is a matter of the mind. Chance has no causal powers. Nothing is “due to” or caused by chance. Not ever.
“Well…”
What you instead mean when you say “due to chance” is that the money in this experiment had no causal powers, direct or indirect, and that the results we observed instead had other causes, causes we did not measure.
“I guess so.”
So that if the P is wee (less than the magic number), you “reject” the hypothesis that the results were “due to chance”. Yes?
“Yes, that’s right.”
And if the results were not “due to chance”, they cannot have been caused by chance, which means they were caused by something else, right?
“Yes, that’s true.”
And our only other candidate for cause is that money. So that “rejecting” the “null” hypothesis that results were caused by chance, means you insist with absolute certainty that the results were caused by the money.
“No! I never said anything about absolute certainty. I just reject that chance caused the results.”
But if you reject chance, then logically, since you offered no other choices, you accept money must be the cause, and with certainty, since there aren’t any other possibilities. You offered none.
“No, I don’t accept that money hypothesis, because some day I might reject it.”
What.
“Yes, we never accept the alternate hypothesis. We merely fail to reject it.”
Which logically means, if you do not reject it, you accept it.
“No, because some day I might reject it.”
P-values are time dependent? Do they have expiration dates? Just what in the world are you talking about.
“I only allow that the money cause might be true.”
Might? You mean it’s more likely to be true since the P was wee?
“Something like that.”
But that’s forbidden. That’s just you putting a probability on a hypothesis, albeit one that is unquantified. You aren’t, in P-value theory, even allowed to think that hypothesis is more or less likely. It has no, and can have no, probability according to theory. It is only true or false. Anything less than total certainty in its truth or falsity is forbidden. You can’t do it.
“Well, it’s not likely to be chance with such a small P.”
There you go putting probabilities on propositions again, which is outlawed in frequentism. You may as well just do it like I do, using just-plain probabilities, and keep your conscience clean. Anyway, what you are doing, even if you don’t put a probability, numerical or implied, on any proposition, it is still a fallacy. You are committing a formal fallacy.
“How? Everybody does it like I do. How can what all these great minds advocate, and even you admit they are great minds, be a fallacy?”
You have calculated a small probability of data you did not see, correct?
“Yes, that’s what the P-value is.”
And you use that statement, that wee P, to accept the hypothesis that the money was the cause. Or equivalently, you reject with certainty that chance was a cause, right?
“Except for the possibility I might someday reject the money cause hypothesis, yes.”
Then this is a formal fallacy. You calculate the probability of data you did not see, and then say if that probability is small, that the money cause is certain. There is no way to get from the premise of a wee P to the certainty of the cause. This is called Bernoulli’s Fallacy.You can’t even get from the premise of the wee P to that cause being more or less certain. This is your argument:
The cause is false;
If the P assuming the cause is false is wee, then the cause is true;
The P is wee;
Therefore, the cause is true.
You started by assuming the cause is false. The second premise is a mere assertion, with no logic to back it up. None of this makes any sense. Why not just calculate probabilities, as I did above? You can even do more, and calculate probabilities the cause is true, or that it is this or that “strength.” Or anything! You can calculate probabilities of anything. You don’t need all this weird indirect rigmarole.
“I see what you mean. But still. P-values are fine is they’re used correctly.”
How! How are they “used correctly”. I’m asking you—all of you. How are they “used correctly”?
All my work is free and user-supported. Here’s how to help:
A small sample: when my wife and I were dating, I just happened to show her my pay stub (I had a good job at he time). She was impressed and commented "that makes it worth working." This was forty years ago and that pay now would be minimum wage. Anyway, that money and my baby blues allowed her to overlook my acerbic personality and proceed to the altar. That pay allowed her to concentrate on procreation rather than wage slavery and it is now thirty-eight years and five babies later. My point: male income that allows for a single paycheck is a great promoter of procreation. Female employment and procreation don't mix well.
Easy.
P values are great source of amusement, and even moreso when they are the targets of much mockery.