Suppose there is a treatment studied in large trials. It has recently been approved by the FDA. The trials were analyzed under a frequentist framework. They were not powered (individually or cumulatively) to study some adverse effects we suspect might indeed exist when the treatment hits general use. They did not show increase in risk of this adverse event.
Now suppose we want to conduct a retrospective study using routine clinical and administrative databases to look at those adverse effects by comparing this new drug to outcomes without this drug. The drug is new enough that many eligible patients don’t yet have it prescribed because doctors aren’t yet familiar with it.
The limitations of this sort of study are well appreciated. When the sample size is large enough, even small amounts of confounding common in such studies will produce “statistically significant” results in the frequentist framework. We all view these small effects with skepticism.
What if we were to analyze this study using a weak skeptical prior – skeptical against higher risk of the adverse effect in question. I haven’t seen this before but am considering doing.
Do you have any examples or guidance here?