P values in clinical trials can always be justified (and calculated) by permutation tests based upon the design. Wasn’t that Fisher’s key justification for the randomization procedure?
The same cannot be said for observational studies. I’m not sure how big of an error this is, however. At least in simulations, the t test results generally matches the permutation test results.
Addendum: I re-read the first paper I posted by Berger, Lunneborg, Ernst, and Levene who argue that RCTs should be examined using permutation tests. I find their reasons for permutation techniques (as well as approximate permutation methods) in this context unassailable and logically correct.
Blockquote
If credibility for the parametric test derives from assurances that its p-value will likely be close to the corresponding exact one, then this is tantamount to an admission that the exact test is the gold standard (or, perhaps, the platinum standard). Approximate tests cannot be any more exact than the exact tests they are trying to approximate and, as approximations to the exact tests, are correct only to the extent that they agree with the exact test. A “heads I win, tails you lose” situation then arises, because if the parametric and exact tests lead to essentially the same inference, then this is as much an argument in favor of the exact test as it is for the parametric test, and there is no benefit to using the parametric test. If they do not agree, then the exact test needs to be used.