Reporting frequentist and Bayesian results side by side in physiotherapy evidence: helpful or confusing?

Hi all,

I would value the community’s views on a reporting question that came up in a recent project.

I reanalysed the meta-analyses in a recent systematic review on exercise for rotator cuff-related shoulder pain (Lafrance et al., JOSPT 2024). The review concluded that motor control exercise is “probably slightly superior” to nonspecific exercise and recommended it to clinicians, interpreting effects with Cohen’s effect-size categories rather than a clinical threshold.

Using group-level data extracted from the review’s forest plots (19 RCTs), I fitted Bayesian random-effects meta-analyses (bayesmeta in R) with neutral, sceptical and enthusiastic priors calibrated to a clinical threshold, and half-normal priors on τ. For pain (0–10 scale), I used an MCID of 1.1 points, deliberately choosing the lowest published value.

Two results illustrate my question:

  1. Motor control vs nonspecific exercise, pain at ~3 months (5 trials): The original review reported SMD 0.38, p = 0.03. In the Bayesian analysis (neutral prior), the posterior mean difference was 0.69 points (95% CrI −0.10 to 1.50); P(benefit) = 96%, but P(effect > MCID) = 12%. The two analyses agree on direction but tell a different clinical story.
  2. Same comparison at ~12 months (only 2 trials, 96 patients): A classical random-effects analysis gives p = 0.002, the “strongest” result in the dataset. The Bayesian analysis gives a 95% CrI that includes zero, and P(effect > MCID) falls from 73% (neutral prior) to 23% (sceptical prior), largely reflecting the uncertainty in τ with two studies.

My questions: In physiotherapy, most readers and reviewers still think in terms of p-values. I am considering recommending that both frequentist and Bayesian results be presented together in future trials and reanalyses (e.g., frequentist primary analysis plus a pre-specified Bayesian analysis reporting probabilities of exceeding clinically meaningful thresholds).

  • Is side-by-side reporting genuinely useful, or does it risk confusing readers or inviting “pick the more favourable result”?
  • If both are reported, which should be primary, and how should discordant results (as in example 2) be communicated?
  • Would you instead recommend dropping p-values and reporting only compatibility intervals alongside the Bayesian results?
  • Any concerns about using within-patient MCIDs as thresholds for between-group effects in this setting?

I would be grateful for any thoughts, references, or examples of good practice.

Thanks in advance.

2 Likes

Reporting likelihood functions along with the compatible confidence distributions would satisfy a very broad range of statistical philosophies, and make meta-analysis somewhat easier.

How to discuss the results in text is up for debate. In a number of threads @f2harrell has persuasively argued mixing up frequentist concepts with Bayesian ones especially after the data have been collected is a confusing mess, and I’ve (finally) come to understand why he holds that POV.

So, I think it is defensible for a Bayesian to put frequentist summaries like confidence distributions in an appendix, and use the main body of the paper to discuss the specification of the prior, and the design that produces the data summarized by the likelihood function.

A Bayesian can make direct use of a likelihood function, but might need to work out how to extract it if only a confidence distribution is reported. So to make things convenient for everyone, just report both.

Study of the monograph Confidence, Likelihood, and Probability will provide a lot of ideas on how to use sets of confidence intervals as frequentist summaries, and how they relate to Bayesian inference.

FWIW, I’m less enthusiastic about their claims regarding confidence distributions as a replacement for Bayesian inference, and I still find the frequentist notion of “epistemic confidence” opaque. Regardless, the relationship between confidence distributions and likelihood functions is well worth exploring.

Further Reading

  1. Pawitan, Y., & Lee, Y. (2021). Confidence as likelihood. Statistical Science, 36(4), 509-517.

  2. Lee, Y. (2025). Epistemic Confidence Statement via Extended Likelihood. arXiv preprint arXiv:2512.24701. link

  3. Efron, B. (1993). Bayes and likelihood calculations from confidence intervals. Biometrika, 80(1), 3-26.

  4. Infanger, D., & Schmidt‐Trucksäss, A. (2019). P value functions: An underused method to present research results and to promote quantitative reasoning. Statistics in medicine, 38(21), 4189-4197.

  5. Rosenthal, R., & Rubin, D. B. (1994). The counternull value of an effect size: A new statistic. Psychological Science, 5(6), 329-334.

The second paper by Lee (2025) partially addresses Frank’s concerns about extended likelihood in this thread:

3 Likes

While I think we’d be better off if p-values and confidence limits were never used, I think having the results side-by-side is a lot better than presenting frequentist measures along.
Note that if the posterior median effect equals the MCID, you get P(effect > MCID) = only 1/2 so we should be computing instead P(effect > MCID/2) or P(effect > MCID/3) or using whatever the trivial effect threshold is.

For a pain scale a normal model is very unlikely to fit; I would use an ordinal model.

1 Like

Thank you, @f2harrell and @R_cubed, for these thoughtful replies. The presentation is now done, so my questions are really about how to plan future studies.

@f2harrell, your points have already changed my approach. I see that P(effect > MCID) sets a very demanding bar, since an effect exactly at the MCID would give only 0.5. In future work I plan to report probabilities across a range of thresholds, including a trivial-effect threshold. I also agree that a normal model is unlikely to fit a 0–10 pain scale. In this reanalysis I only had group-level means and SDs from forest plots, so I will report this as a limitation; in my own trials I intend to use a Bayesian proportional odds model.

@R_cubed, thank you for the reading list, especially the work on confidence distributions and P value functions. If I understand correctly, you suggest that the main text should be fully Bayesian (prior specification, design and posterior results), with frequentist summaries such as confidence distributions reported in an appendix, rather than mixing both frameworks in the discussion.

This differs from what I had in mind: a pre-specified frequentist primary analysis, mainly because reviewers, ethics committees and trial registries in physiotherapy still largely expect one, with a pre-specified Bayesian analysis reported alongside it.

For a future studies in which everything can be pre-specified, I would be grateful for your views on:

  1. Would you endorse R_cubed’s structure (Bayesian main text, frequentist summaries in an appendix) for a field where readers are mostly used to p-values? If so, how would you pre-specify the Bayesian primary analysis in the protocol and trial registration (e.g., decision criteria in terms of posterior probabilities)?
  2. If a reviewer insists on a frequentist primary analysis, what is the least confusing way to present both without mixing the concepts?
  3. How would you suggest choosing the trivial-effect threshold when no smallest worthwhile effect has been established for the population?
  4. Would your advice differ for observational designs (cross-sectional or longitudinal), where pre-specification is less standard?

Thank you both again for your time.

That depends upon how you and your team wishes to approach a problem. In the ideal world, we would have a large community of scientists who find no objection in principle to the Bayesian decision theoretic analysis, so you could go ahead and just write up a report using Bayesian language and concepts. None of that means others will accept your analysis; you will simply be able to avoid complaints non-Bayesians may express, which further confuse people.

If you were to use frequentist methods, I’d say 1. a Bayesian will greatly appreciate a full likelihood function (both raw and penalty used – if any). 2. you could discuss the data using Reverse Bayesian methods (aka. Bayesian Analysis of Credibility) discussed by Robert Matthews and Leonhard Held in an appendix, but again, that might still confuse people.

Because of the institutional bias toward frequentist methods, the Bayesian perspective has to do a bit more to be accepted. A frequentist paper unfortunately, doesn’t have to address Bayesian concerns, but if you want to be helpful, report the likelihood function somewhere, which will also be acceptable to frequentists.

Anything you can do to aid readers in checking the computations, would be very helpful.

There is less agreement between Bayesians and Frequentists if you use testing approaches, so this is only likely to lead to controversy and undermine you message after the data are in. Pick a framework, stick with it, and insert other perspectives or summaries in an appendix.

Some Further References

  1. Matthews, R. A. (2001). Methods for assessing the credibility of clinical trial outcomes. Drug information journal: DIJ/Drug Information Association, 35(4), 1469-1478.

  2. Held, L., Matthews, R., Ott, M., & Pawel, S. (2022). Reverse‐Bayes methods for evidence assessment and research synthesis. Research Synthesis Methods, 13(3), 295-314.

  3. Greenland, S. (2005). Multiple-bias modelling for analysis of observational data. Journal of the Royal Statistical Society Series A: Statistics in Society, 168(2), 267-306.

  4. Hildreth, C. (1963). Bayesian statisticians and remote clients. Econometrica: Journal of the Econometric Society, 422-438.

Paper 3 by @Sander gives some considerations you might use in a Bayesian perspective of a body of evidence when discussing how credible it is, and how it influences your priors.

This ancient post by Andrew Grieve refers to paper 4, which was published in econometrics that explores some of the issues you ask about.

1 Like

I had the wonderful experience of being a co-author on a paper with @Matthew in which The Lancet asked us to remove all frequentist analyses (which were there for secondary endpoints). We’re at the point now of having purely Bayesian articles + traditional descriptive statistics (without p-values and confidence intervals). It is important to address Bayesian operating characteristics in the article, primarily the probability of making correct decisions when the distribution of universe of effects (the study generating prior) departs from the analysis prior.

The above link also partially addresses the triviality threshold. It is best to elicit this from experts in the field, and barring that, to base it on measurement error magnitudes, e.g., mean absolute disagreement in technical replicates. Instead of attempting to be an MCID, this is more like MDD (minimum detectable difference).

4 Likes