Critique of paper on generalizability of oncology trials

I agree with most of your post, except for the following:

I’ve discussed GRADE in a number of threads, and it falls far short of what reasonable people would do in a community of cooperative skeptics interested in truth.

Empirical assessment (linked to in that thread) indicate that applying GRADE overvalues RCTs (granting them more credibility than they contain) and undervalues observational evidence, when the outcome measure is predicting what a future study would show.

The right way was discussed in @Sander in his paper on modelling bias in observational studies, which is inherently Bayesian in perspective.

Greenland, S. (2005). Multiple-bias modelling for analysis of observational data. Journal of the Royal Statistical Society Series A: Statistics in Society, 168(2), 267-306 https://academic.oup.com/jrsssa/article/168/2/267/7084313

Conventional analytic results do not reflect any source of uncertainty other than random error, and as a result readers must rely on informal judgments regarding the effect of possible biases. (my emphasis)

I pointed out in another thread that this “informal judgement” was actually formalized in a field of AI and computer science, known as Subjective Logic, which is an application of Bayesian hierarchical models.

4 Likes