A Statistician Reads JAMA

[R_cubed]

@simongates

Lovely demonstration of the mess that frequentist stats leads us into!

I don’t think this is entirely fair to characterize all of “frequentist” stats like this. This is a result of an institutionalizing certain interpretations of frequentist procedures that have now long veered off into misinterpretations.

Now if by “frequentist” you mean “fixed level \alpha testing”, then I agree wholeheartedly.

Is Any Method So Obsolete It Shouldn't Be Used?

With out a doubt, fixed level testing at the magical levels of 0.1, 0.05, 0.025, or 0.01 should be relegated to the historical dustbin. That doesn’t imply any “ban” on testing, however. It is responsible for the litany of p-value fallacies we see in the scientific journals. It is also responsible for a vast literature on the so-called “Jeffreys-Lindley” paradox. There is no paradox if a declines at a suitable rate as 1-\beta increases. I’m exploring the relationship of this adaptive \alpha …

Note: None of the following should be interpreted as any defense of p value methods in general. I just point out that

1. they are reasonable shortcuts in certain areas and certain stages of scientific research and
2. they are in widespread use, so one needs to know how to interpret them, and know when the reports cananot possibly be correct, or conflict with the narrative of the report.

The problem as far as I can tell, is that all statistical theory recommends is that in order to keep the relationship between the admissible frequentist procedure that reports a p value, and the Bayesian Posterior intact, p values from the frequentist method need to be adjusted for multiple looks at the data. I came to that conclusion after comparing Royall’s discussion of 1/k bound on the likelihood ratio of favoring B to A when A is true, to Tippett’s minimum p value procedure:

\alpha^* = 1- (1 - \alpha)^{1/k}

where a^* is the error (or assertion) probability conditional on the reference null that all studies came from the same distribution of no effect, and \alpha is the error (or assertion) probability from a single study.

There are more formal, and probably better ways of proving this. How to maintain a fixed error rate on the entire procedure can be done in a multitude of ways.

There needs to be a movement that revisits current scientific norms in medical research in light of modern computing power, and a decision theoretic approach that incorporate multiple stakeholders, not just one, who gets to specify the cost function of a procedure, that currently escapes examination.

References

Greenland, S. (2021). Analysis goals, error‐cost sensitivity, and analysis hacking: Essential considerations in hypothesis testing and multiple comparisons. Paediatric and perinatal epidemiology, 35(1), 8-23.

Greenland, S., & Hofman, A. (2019). Multiple comparisons controversies are about context and costs, not frequentism versus Bayesianism. European journal of epidemiology, 34, 801-808.

Berry, D. A., & Hochberg, Y. (1999). Bayesian perspectives on multiple comparisons. Journal of Statistical Planning and Inference, 82(1-2), 215-227. https://www.sciencedirect.com/science/article/abs/pii/S0378375899000440?via=ihub