Blockquote
What is important to understand is that the probability of a Type I error does not necessarily equal the chosen significance level … [following a discussion of \alpha under strict normality] if the random sample does not come from a normal distribution, … then Type I error can differ, sometimes substantially from \alpha, so we do not have as much control over Type I error as the chosen \alpha leads us to believe.
The following section talks about constructing exact “confidence” (aka. compatibility) intervals from permutation tests. This was not mentioned, but it seems to leave open the possibility of constructing an exact “confidence distribution” (the set of all intervals at any \alpha level).
There was also a brief discussion of the Wilcoxon rank test, which is an elegant example of a permutation test.
The paper doesn’t address Frank’s criticism regarding adjustment for covariates, however.
Addendum: The following paper does develop a permutation approach that adjusts for covariates. I’m still in the process of reading it.
My guess is that model based adjustment can be more easily applied, while permutation analyses need to be re-derived for each study based upon the design.
Still, I think the logic behind permutation methods is important to understand, even if the modelling approach is preferred.
Tang, L., Duan, N., Klap, R., Asarnow, J.R. and Belin, T.R. (2009), Applying permutation tests with adjustment for covariates and attrition weights to randomized trials of health-services interventions. Statist. Med., 28: 65-74.
This recent dissertation links permutation methods to estimation of confidence distributions and rare event meta-analysis.
Related Threads
Permutation methods are a crucial component of valid adaptive estimation procedures, created by “inverting the test” – ie. searching for values not rejected. These frequentist techniques can be used in place of robust estimates and the invalid procedure of testing assumptions. Thomas O’Gorman’s books on this should be more widely recognized for those with an applied frequentist philosophy.