"Observed Power" and other "Power" Issues

[ADAlthousePhD]

@RGNewcombe

However, as others have rightly pointed out, to re-assess power retrospectively by taking the observed effect size as the true value is a total fiction. In fact, posterior power is best regarded as merely a rescaling of the p-value. For example, assume that the study was designed to have an 80% power to detect a specified size of difference using a test at the conventional 2-sided 5% alpha level. Then, if the parameter estimates from the data are just what was assumed in advance, we expect to end up with z or t around 2.8, not 2 - because the planned power was 80%, not 50%. Or, if the eventual test is chi-square, this would be around 8, instead of 4. A retrospective power of 80% simply means that p was about 0.005.

Correct.

It’s amazing how statisticians can spend 20 years writing papers that explain this, and yet the fiction of a retrospective power calculation using the observed effect size lives on in clinical journals as a thing people think is not only useful, but necessary. Even after it’s been explained to them…the response came and showed that they had learned nothing:

https://insights.ovid.com/pubmed?pmid=29979247

“We fully understand that P value and post hoc power based on observed effect size are mathematically redundant; however, we would point out that being redundant is not the same as being incorrect. As such, we believe that post hoc power is not wrong, but instead a necessary first assistant in interpreting results.”

@zad and I have just submitted a letter in reply to the surgeon’s double-down. If the journal accepts, I will link; if the journal declines, I will post the full content here, possibly with a guest post on someone’s blog as well.