The Common Cause In All Reproducibility Crises
Here is a cartoon, but a revealing one. Type "reproducibility crisis in" in your search engine and let it suggest the ending of the sentence, as we did here:
Science, medicine of all kinds, psychology, physics, finance, economics. And so on.
Now these are all academic crises. They happen inside universities and similar institutions. The crises are in formal academic disciples and manifest in formal publishing, and are comprised of the growing horror that published papers, especially prominent---i.e. popular---papers in which that which was claimed true turns out to be false, or at least far from proven. Attempts to duplicate well established results fail at an extraordinary rate.
Of course, not all admit to the crisis. Here's a lady (in the Proceedings of the National Academy of Science) who says the failures seen across all these disciplines aren't really failures. The are "epochal changes" and represent "empowerment" and that if the succession of shocking re-non-results were given these more pleasant names, then this would be "inspiring, and compelling."
Well, for some, feelings are more important than dull metrics, like model accuracy and Theory-Reality concordance. For others, what counts most are those dull metrics.
The crises are well recognized, and to a certain extent not political. Even Vox admitted (two years ago) that "science" has been in a replication crisis "for a decade."
Here's a conference on the crisis in, if you can believe it, "ML-based science" (thanks to Stephen Shipman for the tip; ML is a fancy way to say statistical models). Speaking for many fields, one of the goals of this conference is to "Identify root causes of the observed reproducibility failures and explain why they have occurred in dozens of fields that adopted ML methods."
Nature did a survey of working scientists and found only 3% of respondents said there was no crisis. That does not inspire confidence.
Take a look again at the cartoon, which turns out to be pretty accurate. Did you notice what is missing from the list? Seeing what is not there---hearing (so to speak) the dog that didn't bark---is often the hardest evidence to discover.
It's engineering.
But why is engineering missing? Engineering also suffers from the crisis, but to a far lesser extent. There is one key reason. Better, it will turn out that this strange reason is why the other fields, if they wanted, could be absent from the list, too. But aren't.
It's not DIE. Academic engineering is, of course, as subject to the cancer of DIE as the other fields. Injecting Diversity for the sake of Diversity weakens engineering just as much as it degrades any subject. DIE means what it says. Standards will fall. Certainly DIE contributes to the reproducibility crises, but it is not, after all, the premiere cause of them. That is something else.
The advantage is this: engineering, at least in its unwoke and least academic state, is in maximum contact with Reality. Theory exists, but is largely subservient to Reality. Not wholly: largely.
Theory also goes by the alias Models.
In engineering, there is much Theory, many Models, and must be. Theory is an explanation of the way the world works. The more accurate the Theory/Model is, the better the match to Reality, because the better it explains how the world works. Sure, there are Diversity bridges that fail, and even non-Diversity bridges that collapse because of false Theory and ugly Models. But it's rare (but not impossible) in engineering for a Theory to be retained after it fails its confrontation with Reality.
It is not as rare in the other fields. There is variability in this, and some of these disciplines are more Reality-based than others, but all of the other areas are much more in love with Theory and Models.
The crisis arises when false or shaky, but beautiful, Models and Theory is believed because they are beautiful, appealing, complimentary, lucrative, political, or cultural, and not because they accurately explains the way the world works.
To extent a field relies on statistics, the deeper it will be in the crisis. Statistics in its "testing" sense---whether Bayesian or frequentist---is nothing but a way to provide evidence for the Theories in the minds of scientists. It's not only statistics, naturally. This happens whenever scientists search only or mainly for confirmatory evidence, and have lost the ability to accept criticism. They can't criticize themselves, and they certainly will not accept criticism from non-Experts.
Not so coincidentally, this was pointed out to me early this morning:
This is in psychology, a field which is saturated in statistics, would be dead without it. Those p-values represent the "testing", which "confirms" Theory. Having the true belief of the originator of the Theory is what helps it replicate.
Now I realize I haven't proved my own claim here, in these 800 words. But I ask you to think back to the years of analysis of failed science you and I have examined to provide that proof. Or consider this article a "Part I". We'll certainly be doing more.
Accepting our contention, the way out of these crises is obvious. Restore Reality. Make Theories and Models meet the test of Reality. In statistics in particular, it means abandoning "testing" and Model-centric thinking. And moving, like in engineering, to making skillful predictions.
Buy my new book and learn to argue against the regime: Everything You Believe Is Wrong.
Visit wmbriggs.com.
I tried to reproduce your lady claim, and found the lady to have an impressive beard.