You make excellent points about other interventions not having enough funding for an RCT. I’ve asked myself similar questions.
In terms of the primary study – if you assume an effect distribution around zero before seeing the data, then use their data to update your prior, you can provide evidence for a range of effects that were not initially planned for in the study.
I would also think a Bayesian analysis of the subgroups could be persuasive evidence in terms of the effect.
Edit: Links to related and relevant posts on this topic:
From a broader POV, a better analysis of RCT data can (when examined from a Bayesian decision theory POV) lead to the derivation of an experiment that will decide the clinically relevant question.
Here is what I think after having given this issue a lot of thought. It seems reasonable to me, but I would value some additional input from scholars in this area.
A quick way to describe my thoughts would be a Bayesian parametric meta-analysis of non-parametric primary effect sizes.
My main emphasis would be on the effect estimate (regardless of significance), and the design (to see if there need to be downward adjustments to precision based on errors in the analysis such as dichotomization, improper change score analysis, etc. Dr. Harrell lists a number in his free booklet Biostats for Biomedical Research AKA BBR).
My preferred estimate of effect would be some sort of odds ratio related to the logistic model. I think parametric effect sizes based on standardized means are more fragile than is understood.
Standardized mean differences are easily translated into log odds.
See the following for an informal proof of translating means into odds:
The actual ratio to multiply standardized mean effect by is \frac{\pi}{\sqrt{3}}.
I guess you can say I share Dr. Harrell’s preference for the Wilcoxon-Mann-Whitney as my default 2 sample test, and the logistic model from which it is derived.
You could do a meta-analysis of the relevant trials, adjust for publication bias, then do a bootstrap on the corrected effect size estimates.
Points inside the bootstrap CI could be defensible point estimates to base a Bayesian prior distribution on. If the 25th percentile of the bootstrap distribution is assumed to be the mean of a normal distribution – is it far enough from 0 that another study would be hard to justify?
More complicated models would require the use of meta-regression. The logistic model would be a natural fit here.
Empirical Bayes techniques have been described in this area that might help you persuade the dogmatic frequentists. Using an Empirical Bayes approach gives you a posterior distribution than can be interpreted as a predictive model for future studies.
I’ve already collected a number of papers related to the issue of meta-analysis in this thread