# Multiple test correction for several independent DVs?

**URL:** <https://discourse.datamethods.org/t/multiple-test-correction-for-several-independent-dvs/3770>\
**Category:** general\
**Tags:** multiple-endpoints\
**Created:** [September 22, 2020, 10:02pm UTC](https://discourse.datamethods.org/t/multiple-test-correction-for-several-independent-dvs/3770 "2020-09-22T22:02:59Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![R\_cubed](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/r_cubed/32/1518_2.png) [@R\_cubed](https://discourse.datamethods.org/u/R_cubed)\
**Post date:** [September 23, 2020, 2:04pm UTC](https://discourse.datamethods.org/t/multiple-test-correction-for-several-independent-dvs/3770/2 "2020-09-23T14:04:49Z")

</div>

You might find this older post, with a section on multiple comparisons by Dr. Harrell helpful:

> [@Author Checklist](https://discourse.datamethods.org/t/author-checklist/3407):
>
> Statistical Problems to Document and to Avoid Checklist for Authors [References](https://hbiostat.org/bib) | [Ten Simple Rules](http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1004961) | [Checklist to Avoid p-Hacking](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5122713) | [Research Reliability and Publishing](https://www.frontiersin.org/articles/10.3389/fnhum.2018.00037/full) | [Guidelines for Reporting of Statistics](https://www.ncbi.nlm.nih.gov/pubmed/30537407) | [Guidelines for Figures and Tables](https://www.europeanurology.com/article/S0302-2838(20)3036-X/pdf) | [Statistical Myths](https://discourse.datamethods.org/t/reference-collection-to-push-back-against-common-statistical-myths) | [Video](https://youtu.be/5HnW1YP3vTk?t=20413) | [Common Pitfalls](https://journals.stfm.org/primer/2022/van-smeden-2022-0059/#.YvQyN_kxOPE.twitter) This list is not all-inclusive, and nominations for additional entries are welcomed. The list is not meant to imply that a lead author should wait until the end to correct statistical errors, especially w…

Taking for granted that all analyses are to be conducted within the frequentist perspective:

If the comparisons were specified ahead of the data analysis, the consensus is that no adjustments are necessary. This makes sense a if decision based on the experiment is to be used only for the purposes of the experimenter. An interesting dilemma is whether a party not part of the design phase should accept this argument. I’m not sure if there is a consensus on that. Don’t be surprised if you receive resistance to the prespecified hypotheses argument.

For post hoc hypotheses, there are a number of methods. Older work focus on control of the Family Wide Error Rate (FWER); since the 1995 Benjamini and Hochberg paper, newer methods focus on the False Discovery Rate (FDR). There are interesting relationships between the methods, with the FDR controlling the FWER in a weaker sense, but providing more power.

I’m not exactly sure when we can substitute FDR control for FWER control. It seems that the FDR is more appropriate for the exploratory stages of research (ie. GWAS studies looking at thousands of test results), while FWER is more important in the later stages of confirmation.

Some references (in no particular order):  
**Cook, RJ; Farewell, VT (1996).** Multiplicity Considerations in the Design of Clinical Trials _Journal of the Royal Statistical Society_ Vol 159 No. 1 Pages 93-110. (Referenced in multiple posts by Dr. Harrell).

**Westfall, P; Young, S. (1993)** Resampling-Based Multiple Testing: Examples and Methods for p-Value Adjustment. _Wiley Series in Probability and Mathematical Statistics_

**Hochberg, Y; Tamhane, A. (1987)** Multiple Comparison Procedures. _Wiley Series in Probability and Statistics_

**Benjamini, Y; Hochberg, Y. (1995)** Controlling the False Discovery Rate: a Practical and Powerful Approach to Multiple Testing. _Journal of the Royal Statistical Society: Series B (Methodological) Vol. 57, No 1._ (Landmark paper describing False Discovery Rate control and the relation to other methods)

---

_[View the full topic](https://discourse.datamethods.org/t/multiple-test-correction-for-several-independent-dvs/3770)._