Reference Collection to push back against "Common Statistical Myths"

[R_cubed]

The following threads had a discussion of this issue. It seems like it could fall under the p value misconception heading, but I think it deserves a section on its own.

I think the Bayesian POV provides a framework for guidance on this. @Sander provided references to Bayesian justifications for adjustment.

Transforming confidence intervals into credible intervals

Sorry, this may only be a matter of words, but: I would not say “It is necessary to not adjust for multiplicity to get a Bayesian-frequentist equivalence”. The way I’d put it, a Bayesian method does not “adjust for multiplicity” because the account for multiplicity is pre-loaded (as it were) in its framing of the question and model to address what a frequentist calls a “multiplicity problem”. As per their culture, frequentists differ only in allowing a zoo of incompletely specified questions and…

Some of his own writing on the issue:

Multiple comparisons controversies are about context and costs, not...

https://onlinelibrary.wiley.com/doi/full/10.1111/ppe.12711

I posted a few references in the context of clinical trials in this thread:

Multiplicity in clinical trials with hierarchical strategy

It will be hard to improve on what Frank wrote above. I’ll just post a link to the short thread that occurred here a few months ago were I posted some references to this issue: If neither 1. ordering the research questions or 2. doing an alpha spending adjustment are acceptable, it might be worth considering the FDR procedure. The following is open access: Watson, J.; Robertson, D. (2020) Controlling type I error rates in multi‐arm clinical trials: A case for the false discovery rate. Phar…

Multiple test correction for several independent DVs?

You might find this older post, with a section on multiple comparisons by Dr. Harrell helpful: Taking for granted that all analyses are to be conducted within the frequentist perspective: If the comparisons were specified ahead of the data analysis, the consensus is that no adjustments are necessary. This makes sense a if decision based on the experiment is to be used only for the purposes of the experimenter. An interesting dilemma is whether a party not part of the design phase should …

The following is as close to a complete theory of MCPs as we are going to get as the perceived need to adjust is closely related to the ratio of \frac{1-\beta} {\alpha} an experimenter wishes to make, or the plausibility of the hypothesis under consideration.

https://onlinelibrary.wiley.com/doi/abs/10.1002/bimj.200810425