William, here's the rub with hypothesis testing: we calculate a p value claiming H0 is true but then use that as evidence it is not true. Huh? yes that's what we are doing. Furthermore it only tells you the data is extreme if H0 is true. That's it. It tells you nothing useful at all! It doesn't mean the alternative is true, it doesn't mean H0 is false. All it tells you is how compatible the data is with H0. It's basically worthless.
This gets at something that came up for me during the video. Namely, my understanding was similar to what Paul Fischer says above regarding the data "is extreme if H0 is true" as opposed to what William stated that the frequentists assert -> p is *equally* distributed between 0 and 1. In other words, I was taught that the p value indicates the probability of seeing an observed value given the assumed distribution associated with the null hypothesis.
The problem is that the 0.05 p-value that is "standard" is too large. It leads to too many false positives. In physics, they use a much smaller p-value. I think it's 1 in 35 million instead of 1 in 20.
Another "nice" feature of the 0.05 p-value is that it self-adjusts to the sample size. It doesn't matter if you have a sample size of 10 or a million. The 0.05 p-value standard will give a false positive 5% of the time either way.
“We want Pr(H|Evidence), we get Pr(Data we didn’t see | H false). And then everybody pretends they are the same. Why why why.”
Just a comment more on the “why” aspect of today’s commentary. Part of the perversion of statistics and statistical inference I believe lies in the ease of computing, which contributes to a lack of understanding. Let me elaborate, but correct me if you see error in my “understanding/theory”. Hey, it’s been a long time…. ;-)
I was blessed (I now realize) to be taught statistics *before* the age of computer math/stat packages like SPSS and BMDP became popular. I was not allowed to use those packages in class before I did such problems by hand. Yep, I did factor analysis by multiplying matrices by hand and teasing out factors with a protractor. A simple problem solution took most of the afternoon at my desk.
It was through those painful exercises that I got a feel for where the statistics came from and what they were indicating—or rather the extent we could make inferences from the data.
Shortly after, while ABD, I took my first full-time employment in the university computer center. I was in charge of installing and consulting on the large math/stat packages. There I saw one faculty person after another making the same mistakes over and over again wrt “p-values”. Social science of course was the worst. Large arrays of data points were gathered and large matrices of significant (?) intercorrelations produced. Find those “wee P’s”, write up nonsense assertions, publish, repeat.
The mainframe computers and math/stat packages we bought were a great tool to the knowledgeable, but at the same time permitted the unknowledgeable to flood the zone. Sigh.
Statistics education is a mess. Instructors barely grasp the subject themselves, failing to explain what p-values actually are. Instead, they parrot flawed dogma to students, who then feed piles of data into stats software, churning out thousands of meaningless p-values with zero clue about what’s actually happening. It’s getting worse every year. Back when I started college, we didn’t have computers we had slide rules. We couldn't mindlessly crunch numbers. But after grad school, they were everywhere, turning stats and research into a complete farce.
I spent a small fortune (relative to my resources) on my K&E slide rule in 1971 , thinking this is an investment in my career. It served me well through undergraduate engineering but by graduate school the HP calculator and computer center had relegated the K&E to my desk drawer. Still, it inculcated a good discipline in estimating, order of magnitude thinking and scientific notation. That is still a useful skill.
p-values, confidence intervals, and all that NHST junk have already proven themselves pretty pathetic in the realm of science, but I’d be interested in hearing your take on the use of testing in the Deborah Mayo style and all that talk about severe testing (I know almost no one takes old-school Popperianism seriously in philosophy departments anymore, but in the sciences it’s still frequently cited—even if only in its fairy-tale version).
Not convincing in and of itself. IF p value is uniformly distributed under the null hypothesis, then a measured wee p value is going to be improbable.
I wonder about that IF. Still need to rework the frequentist math.
Many other explanations abound for bad replication. The science press jumps on interesting and counterintuitive results. A one in 20 chance of a silly experiment meeting the threshold is going to provide. And, of course, ANY time series is going to have a control problem. Lots of things happen over the course of time. They can produce similar curves for unconnected phenomena. Mann's Hockey Stick was a notorious example of playing games with short curve matches.
And the matches weren't all that good! The noise in the bristlecone pine rings vs. temperature was way bigger than the signal. Your call to have paper show the raw data is spot on!
p-values are Uniform(0,1) under H0 from a simple probability transform. If T is the test statistic and has some distribution F. So a p value is the probability under H0 true of seeing a
T as or more extreme as the observed one.
p=P(T>=Tobs| H0). So now F is the CDF of T assuming H0 is true. then p=1-F(Tobs) is uniform (0,1). All it means is if H0 is true every p has equally likely. If you play around with an example it's easy to see.
OK, I looked at the Wikipedia entry on P values and what you wrote makes sense.
BUT, is H0 unique? For the trivial fair dice example, perhaps. The Wikipedia article talked about assumed distributions and whatnot. This is what I haven't wrapped my head around. I hit the wall reading Bulmer because I didn't work the problems as I went along. Finding time/focus to read more closely and work enough problems is challenging as I am dealing with gamma rays and tennis ball sized lymph nodes currently. Really kills the energy and concentration.
30+ years ago when I was doing QA work for the GPS system I did a lot of staring at residuals and comparing the stats on the residuals with the predicted stats -- as well as eyeballing things to see if they looked like Gaussian noise. The ideal was to fit a model that the residuals (measured vs. predicted range values) looked like Gaussian noise. This ideal was not attained. Yet GPS kind of works.
What I'm trying to figure out is if scientists are simply assuming certain a priori stats or are they applying as much double checking as engineers using Kalman filters.
I wonder if deep within a great statistician there is found a profound compulsion to rule the Black Jack table.
(just as surgeons are with a knife... engineers are with a build... )
Beat The Dealer by Dr. Edward Thorpe.
William, here's the rub with hypothesis testing: we calculate a p value claiming H0 is true but then use that as evidence it is not true. Huh? yes that's what we are doing. Furthermore it only tells you the data is extreme if H0 is true. That's it. It tells you nothing useful at all! It doesn't mean the alternative is true, it doesn't mean H0 is false. All it tells you is how compatible the data is with H0. It's basically worthless.
This gets at something that came up for me during the video. Namely, my understanding was similar to what Paul Fischer says above regarding the data "is extreme if H0 is true" as opposed to what William stated that the frequentists assert -> p is *equally* distributed between 0 and 1. In other words, I was taught that the p value indicates the probability of seeing an observed value given the assumed distribution associated with the null hypothesis.
The problem is that the 0.05 p-value that is "standard" is too large. It leads to too many false positives. In physics, they use a much smaller p-value. I think it's 1 in 35 million instead of 1 in 20.
https://xkcd.com/882/
Another "nice" feature of the 0.05 p-value is that it self-adjusts to the sample size. It doesn't matter if you have a sample size of 10 or a million. The 0.05 p-value standard will give a false positive 5% of the time either way.
P-values will stay, because people need a third party to blame for failures (medical, corporate, all of it). “B-b-but the DATA said…!”
You're on fire Briggs!
🔥🔥🙌
“We want Pr(H|Evidence), we get Pr(Data we didn’t see | H false). And then everybody pretends they are the same. Why why why.”
Just a comment more on the “why” aspect of today’s commentary. Part of the perversion of statistics and statistical inference I believe lies in the ease of computing, which contributes to a lack of understanding. Let me elaborate, but correct me if you see error in my “understanding/theory”. Hey, it’s been a long time…. ;-)
I was blessed (I now realize) to be taught statistics *before* the age of computer math/stat packages like SPSS and BMDP became popular. I was not allowed to use those packages in class before I did such problems by hand. Yep, I did factor analysis by multiplying matrices by hand and teasing out factors with a protractor. A simple problem solution took most of the afternoon at my desk.
It was through those painful exercises that I got a feel for where the statistics came from and what they were indicating—or rather the extent we could make inferences from the data.
Shortly after, while ABD, I took my first full-time employment in the university computer center. I was in charge of installing and consulting on the large math/stat packages. There I saw one faculty person after another making the same mistakes over and over again wrt “p-values”. Social science of course was the worst. Large arrays of data points were gathered and large matrices of significant (?) intercorrelations produced. Find those “wee P’s”, write up nonsense assertions, publish, repeat.
The mainframe computers and math/stat packages we bought were a great tool to the knowledgeable, but at the same time permitted the unknowledgeable to flood the zone. Sigh.
Statistics education is a mess. Instructors barely grasp the subject themselves, failing to explain what p-values actually are. Instead, they parrot flawed dogma to students, who then feed piles of data into stats software, churning out thousands of meaningless p-values with zero clue about what’s actually happening. It’s getting worse every year. Back when I started college, we didn’t have computers we had slide rules. We couldn't mindlessly crunch numbers. But after grad school, they were everywhere, turning stats and research into a complete farce.
I spent a small fortune (relative to my resources) on my K&E slide rule in 1971 , thinking this is an investment in my career. It served me well through undergraduate engineering but by graduate school the HP calculator and computer center had relegated the K&E to my desk drawer. Still, it inculcated a good discipline in estimating, order of magnitude thinking and scientific notation. That is still a useful skill.
Yes indeed. Computers are magic, too.
p-values, confidence intervals, and all that NHST junk have already proven themselves pretty pathetic in the realm of science, but I’d be interested in hearing your take on the use of testing in the Deborah Mayo style and all that talk about severe testing (I know almost no one takes old-school Popperianism seriously in philosophy departments anymore, but in the sciences it’s still frequently cited—even if only in its fairy-tale version).
We’ll come to that. Thanks.
Not convincing in and of itself. IF p value is uniformly distributed under the null hypothesis, then a measured wee p value is going to be improbable.
I wonder about that IF. Still need to rework the frequentist math.
Many other explanations abound for bad replication. The science press jumps on interesting and counterintuitive results. A one in 20 chance of a silly experiment meeting the threshold is going to provide. And, of course, ANY time series is going to have a control problem. Lots of things happen over the course of time. They can produce similar curves for unconnected phenomena. Mann's Hockey Stick was a notorious example of playing games with short curve matches.
And the matches weren't all that good! The noise in the bristlecone pine rings vs. temperature was way bigger than the signal. Your call to have paper show the raw data is spot on!
It brings us nowhere near Pr(H|E). And every use of Pr(data we didn’t see| H false) to infer anything about Pr(H|E) is fallacious. End of story.
p-values are Uniform(0,1) under H0 from a simple probability transform. If T is the test statistic and has some distribution F. So a p value is the probability under H0 true of seeing a
T as or more extreme as the observed one.
p=P(T>=Tobs| H0). So now F is the CDF of T assuming H0 is true. then p=1-F(Tobs) is uniform (0,1). All it means is if H0 is true every p has equally likely. If you play around with an example it's easy to see.
OK, I looked at the Wikipedia entry on P values and what you wrote makes sense.
BUT, is H0 unique? For the trivial fair dice example, perhaps. The Wikipedia article talked about assumed distributions and whatnot. This is what I haven't wrapped my head around. I hit the wall reading Bulmer because I didn't work the problems as I went along. Finding time/focus to read more closely and work enough problems is challenging as I am dealing with gamma rays and tennis ball sized lymph nodes currently. Really kills the energy and concentration.
30+ years ago when I was doing QA work for the GPS system I did a lot of staring at residuals and comparing the stats on the residuals with the predicted stats -- as well as eyeballing things to see if they looked like Gaussian noise. The ideal was to fit a model that the residuals (measured vs. predicted range values) looked like Gaussian noise. This ideal was not attained. Yet GPS kind of works.
What I'm trying to figure out is if scientists are simply assuming certain a priori stats or are they applying as much double checking as engineers using Kalman filters.
The null is not unique. No probability is unique! I demonstrated that in the Does This Guy Have ESP (I think the second).
I have two more weeks of this for clarification. Won't that be fun.
I really liked that ESP article. Your getting down to a concrete example vs. going through tedious notation was very helpful.
Many thanks.