Hi,
A researcher I know recently was asked by a reviewer to correct their results for multiple comparisons.
They ran four linear regression models with the same data for the independent variables (the same 4 IVs in each model), but different outcome measures for each model (these were all conceptually distinct DVs). They pre-planned their analysis but did not pre-register.
They are using frequentist null hypothesis significance testing and the results will be significant in either case - the issue is whether it is correct to do multiple test correction in this case. A senior researcher in the area says that it is not at all common in this research community to correct for multiple comparisons unless you think the DVs are strongly correlated with each other.
What should they do in this case? I had never heard of the dependence of the outcome measures mattering for this, but this post seems to possibly indicate that they are correct?
https://discourse.datamethods.org/t/multiplicity-adjustments-in-bayesian-analysis/
I’d love to learn more about this since I will probably run across this in my own research. Is there any accessible textbook or online resource that covers this problem?
All the best, and thank you for any help you can provide,
Jacob Ritchie