Massive Over-Certainties In Medicine

Bad Statistics, Bad Experiments

Apologies to email readers. The html edits I did earlier did not stick. Don’t know why.

Ready to Follow The Science? Let's listen to the lamentations of its lecturers and lovers.

The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue. Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance, science has taken a turn towards darkness.

Richard Horton, 2015, Editor in chief The Lancet

We earlier did an article looking into Horton's comments. It's more than half.

Evidence-based medicine is actually so corrupt as to be useless or harmful...

It is simply no longer possible to believe much of the clinical research that is published, or to rely on the judgement of trusted physicians or authoritative medical guidelines. I take no pleasure in this conclusion, which I reached slowly and reluctantly over my two decades as an editor.

Marcia Angell, 2009, Former editor in chief New England Journal of Medicine

And how about this?

The medical profession is being bought by the pharmaceutical industry, not only in terms of the practise of medicine, but also in terms of teaching and research. The academic institutions of this country are allowing themselves to be paid agents of the pharmaceutical industry. I think it’s disgraceful.

Dr Arnold Relman, 2002, Former editor New England Journal of Medicine

And this?

I think we have to call it what it is. It is a corruption of the scientific process...It’s led me and others to increasingly question the idea that the manufacturer of the drug could ever be considered the right people to evaluate its effectiveness and safety.

Fiona Godlee, 2016 Editor, BMJ

My favorite, and maybe yours, too, is this one:

If peer review was a drug it would never get on the market because we have lots of evidence of its adverse effects and don’t have evidence of its benefit.

It’s time to slaughter the sacred cow.

Dr Richard Smith, 2015, Former Editor BMJ

There's more if you need them at the site that collects them.

You will have noticed none of these specimens are from random bloggers or other dissidents, but from Experts themselves.

Now let's look at the 2017 BMJ review paper "Availability of evidence of benefits on overall survival and quality of life of cancer drugs approved by European Medicines Agency: retrospective cohort study of drug approvals 2009-13". The objective was "To determine the availability of data on overall survival and quality of life benefits of cancer drugs approved in Europe."

Obviously, we'd like the survival prospects to have increased when taking the drugs. The "quality of life" measures are trickier, both because of attempts to quantify the unquantifiable and because these measures are less impressive.

Results:

From 2009 to 2013, the EMA approved the use of 48 cancer drugs for 68 indications. Of these, eight indications (12%) were approved on the basis of a single arm study. At the time of market approval, there was significant prolongation of survival in 24 of the 68 (35%). The magnitude of the benefit on overall survival ranged from 1.0 to 5.8 months (median 2.7 months).

Expensive drugs buy an average 2.7 months extra life. In only 35% of the studies were these "significant". Meaning had a wee p-value, the worst ever measure of performance. Hint: it should be banned.

Conclusion:

This systematic evaluation of oncology approvals by the EMA in 2009-13 shows that most drugs entered the market without evidence of benefit on survival or quality of life. At a minimum of 3.3 years after market entry, there was still no conclusive evidence that these drugs either extended or improved life for most cancer indications. When there were survival gains over existing treatment options or placebo, they were often marginal.

Now let's look at a Science News piece: "A massive 8-year effort finds that much cancer research can’t be replicated".

Researchers with the Reproducibility Project: Cancer Biology aimed to replicate 193 experiments from 53 top cancer papers published from 2010 to 2012. But only a quarter of those experiments were able to be reproduced, the team reports in two papers published December 7 in eLife...

What’s more, of the 50 experiments from 23 papers that were reproduced, effect sizes were, on average, 85 percent lower than those reported in the original experiments. Effect sizes indicate how big the effect found in a study is. For example, two studies might find that a certain chemical kills cancer cells, but the chemical kills 30 percent of cells in one experiment and 80 percent of cells in a different experiment. The first experiment has less than half the effect size seen in the second one.

Score that one against p-values, too.

Not only p-values, of course. Experimenter control of original data is also huge. This doesn't always mean cheating, though it means that at least sometimes. But it's too easy to push things in desired directions, even unthinkingly, when your hand is on all the levers.

When the experiment meets Reality and that control disappears? Well, we see what we have seen here. These two papers are not unusual. There are scores of them, all with the same story and with the identical conclusion: Be not too certain. Do not blindly trust The Science, for it is more often wrong than right.

Then we recall the past two years and the vexxine mania. Authorities <em>who should know better</em> were touting 95-100% efficacy rates---against infection, forsooth!---based on Pfizer's initial data. (Regular readers will recall I said it couldn't be greater than half that in real life.)

We were screamed at "Follow The Science!" That should have meant evincing considerable skepticism and cautious use. And not a moral panicked headline rush headline into the abyss.

The results have been entirely as we on Team Reality predicted.

Similar results will continue to hold, not just in medicine, but in all areas of science in which the practitioners have a strong interest in the outcomes.

Buy my new book and learn to argue against the regime: Everything You Believe Is Wrong.

Visit me at wmbriggs.com.