If you could fix one topic in Modern Statistics, what would it be?

[R_cubed]

For those looking for a further elaboration of this argument about \alpha, see this thread:

What is type I error?

Statisticians, clinical trialists, and drug regulators frequently claim that they want to control the probability of a type I error, and they go on to say that this equates to a probability of a false positive result. This thinking is oversimplified, and I wonder if type I error is an error in the usual sense of the word. For example, a researcher may go through the following thought process. I want to limit the number of misleading findings over the long run of repeated experiments like mine…

If you study closely these 3 papers:

1. Evans, M. (2016). Measuring statistical evidence using relative belief. Computational and structural biotechnology journal, 14, 91-96. (HTML)

2. Bayarri, M. J., Benjamin, D. J., Berger, J. O., & Sellke, T. M. (2016). Rejection odds and rejection ratios: A proposal for statistical practice in testing hypotheses. Journal of Mathematical Psychology, 72, 90-103. (HTML)

3. Rafi, Z., & Greenland, S. (2020). Semantic and cognitive tools to aid statistical science: replace confidence and significance by compatibility and surprise. BMC medical research methodology, 20(1), 244. (HTML)

you can see (with some simple algebra) that \alpha is a ratio of what Evans calls the relative belief ratio divided by frequentist power. Bayarri et. al. make a similar argument. This makes \alpha a measure of discrepancy from a particular reference point \theta = 0) prior to the data being seen. After the data, the p value is the observed discrepancy (or information) by placing the sample estimate in relation to some quantile of an assumed distribution.

It should not take 3 papers to discuss these relationships!

There was an extensive discussion on p values in this video that deserves review for those of us who were not formally trained in statistics. James Berger in the first half hour describes how to use Bayesian reasoning to justify your \alpha level, and how this improved replication in a certain area of genetic research.

More videos related to p values are found in this thread:

P values and Multiple Comparison Procedures: Some resources

I discovered an interesting video from the NISS channel on Youtube related to the use of p values in decision making that might be of interest. [The Use of p-Values in Making Decisions: A Third NISS Webinar - May, 2020] [Digging Deeper Into p-values: NISS Webinar Follow-up with Three Authors - November, 2020] [NISS Webinar: Alternatives to the Traditional p-value - May, 2020] Related Threads https://link.springer.com/article/10.1007/s11229-021-03276…

What I wish statistics courses made clear are the important relationships between Bayesian and Frequentist methods. I’d go as far as to say (based on Wald and Duanmu’s complete class theorems) that Frequentist methods are important special cases of Bayesian methods.

Likewise, Frequentist methods (via simulation) can aid us in checking the validity and impact of our assumptions before taking action based upon an analysis. Frequentists have figured out a number of computational methods to use when doing a direct Bayesian calculation is not easy or obvious.

For some food for thought, read a few of the papers that discuss stat foundations, which can be found here: Bayesian, Fiducial and Frequentist = BFF = Best Friend Forever