# A Bayes-Frequentist approach to the Analysis of Credibility of meta-analyses using p-values

**URL:** <https://discourse.datamethods.org/t/a-bayes-frequentist-approach-to-the-analysis-of-credibility-of-meta-analyses-using-p-values/5004>\
**Category:** bayes\
**Tags:** meta-analysis\
**Created:** [October 11, 2021, 1:16am UTC](https://discourse.datamethods.org/t/a-bayes-frequentist-approach-to-the-analysis-of-credibility-of-meta-analyses-using-p-values/5004 "2021-10-11T01:16:23Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![R\_cubed](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/r_cubed/32/1518_2.png) [@R\_cubed](https://discourse.datamethods.org/u/R_cubed)\
**Post date:** [October 11, 2021, 1:16am UTC](https://discourse.datamethods.org/t/a-bayes-frequentist-approach-to-the-analysis-of-credibility-of-meta-analyses-using-p-values/5004/1 "2021-10-11T01:16:23Z")

</div>

I wanted to post this for some input to see if there was any flaw in my reasoning for the proposed procedure. It incorporates a lot of ideas from dozens of papers and posts. Following the ideas from  
[A Theory of Experimenters](https://www.nber.org/papers/w23867), this description is how an individual with no need to convince an external audience, might use it. There are some adaptations I can imagine that would bring it in line with [Robert Matthew’s Analysis of Credibility](https://discourse.datamethods.org/t/frequentist-vs-bayesian-debate-applied-to-real-world-data-analysis-any-philosophical-insights/1695/9), to model a dialogue with a skeptical audience.

> Blockquote:  
> Statistical practice changes slowly because the teaching of statistics changes slowly … Once sociologists and physicians have learned about significance levels well enough to use them, a major reorganization of the thought process is required to adapt to decision theoretic or Bayesian analysis … With the help of theory, I have developed insights and intuitions that prevent me from giving weight to data dredging and other forms of statistical heresy. This feeling of freedom and ease does not exist until I have a decision theoretic, Bayesian view of the problem. I am a Bayesian decision theorist in spite of my use of Fisherian tools.  
> — **Herman Chernoff** in commentary on “why Isn’t Everyone Bayesian” by Bradley Efron (1986)

**Procedure Description**  
**Inputs:**

- sequence of signed test statistics and/or 2 sided p-values,
- Prior and Posterior Odds,
- ~~Sequence of Posterior Odds changes as number of studies increase,~~
- Sequence of changes \alpha ~~allocation~~ as studies increase (see this article on [adaptive \alpha](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3912868/) levels),
- Expected Power for individual study
- Step procedure (Holm or Hochberg)
- P value combination procedure (Stouffer-Liptak, Fisher, Logit)

> 

**Output:**  
~~Statement of Posterior Odds~~ There exists s of N studies indicated a clear direction of effect, where

- s = H(r) + C(b)
- H(r) = null rejected by Hochberg step-up method (or Holm step-down)
- C(b) = -1 or +1 if combination procedure of studies not rejected in Hochberg are rejected at remaining alpha, 0 otherwise.

The familywide \alpha level will be set using Bayesian reasoning. Following Cheng and Sheng, \alpha will be allocated between a step procedure and a combination procedure. The input for combination procedure will be the remaining statistics not rejected by step procedure. Following Naaman and Pericchi and Periera, familywide \alpha will decline as number of studies increases. i ~~In addition, less error will be allocated to the combination procedure, permitting more power for the sequential testing procedure to find a local effect.~~

All values will be specified by user prior to data collection as part of SAP (statistical analysis plan).

**Rationale:** Contrary to Chernoff’s quote, current discussions of a scientific replication crisis indicates that “significance” levels are not understood well enough to use them correctly in a large areas of science. This suboptimal state of affairs continues to exist because:

> Blockquote:  
> Classical Fisherian significance testing is immensely popular because it requires so little from the scientist… – **Bradley Efron** Large Scale Inference: Empirical Bayes for Large Scale Testing, Estimation, and Prediction (p. 48)

This major reorganization of thought begins by placing elementary statistical procedures in a Bayesian context. Particularly important are are [correction for multiplicity](https://journals.sagepub.com/doi/abs/10.3102/10769986024001042), and the information fusion approach via p-value combination have [Bayesian](https://www.jstor.org/stable/2337467) and information theoretic justifications.

Omnibus p-value procedures remain useful and research into extensions of these methods is ongoing.  
Hedges and Olkin (1985) note:

> Blockquote  
> An important application of omnibus test procedures is to screen for any effect … Alternatively, combined test procedures can be used to combine effect size analyses based on different outcome variables.

In the context of meta-analysis, it is far from unusual to have what would ordinarily be a small sample (ie. no more than 20 studies). This limits the application of preferred methods (ie. effect size aggregation or meta-regression) when explanation of variability would be useful. Rather than produce a misleading effect size estimate from heterogeneous studies, it would be preferable to compare and contrast the local results of individual studies to design more informative ones given current knowledge.

The following procedure builds upon Cheng and Sheng (2017)

> **[Combination of “combinations of p values” - Empirical Economics](https://link.springer.com/article/10.1007/s00181-017-1230-9)**
>
> We investigate the impact of an uncertain number of false individual null hypotheses on commonly used p value combination methods. Under such uncertainty, these methods perform quite differently and often yield conflicting results. Consequently, we...

Their procedure divides the familywise \alpha among 2 uncorrelated p-value combination procedures. This \alpha division is essentially a Bonferroni adjustment for multiple tests.

Taking guidance from Bayarri, Berger, Benjamin, and Selke (2016), Pericchi and Periera (2016), and Naaman (2016), this Bayes-Frequentist procedure will overcome limitations of classical combination procedures, and provide a more intuitive Bayesian interpretation.

> **[Rejection odds and rejection ratios: A proposal for statistical practice in...](https://pmc.ncbi.nlm.nih.gov/articles/PMC6358203/)**
>
> Much of science is (rightly or wrongly) driven by hypothesis testing. Even in situations where the hypothesis testing paradigm is correct, the common practice of basing inferences solely on p-values has been under intense criticism for over 50 ...

[https://projecteuclid.org/journals/brazilian-journal-of-probability-and-statistics/volume-30/issue-1/Adaptative-significance-levels-using-optimal-decision-rules--Balancing-by/10.1214/14-BJPS257.full](https://projecteuclid.org/journals/brazilian-journal-of-probability-and-statistics/volume-30/issue-1/Adaptative-significance-levels-using-optimal-decision-rules--Balancing-by/10.1214/14-BJPS257.full)  
[https://projecteuclid.org/journals/electronic-journal-of-statistics/volume-10/issue-1/Almost-sure-hypothesis-testing-and-a-resolution-of-the-Jeffreys/10.1214/16-EJS1146.full](https://projecteuclid.org/journals/electronic-journal-of-statistics/volume-10/issue-1/Almost-sure-hypothesis-testing-and-a-resolution-of-the-Jeffreys/10.1214/16-EJS1146.full)

Related threads:

> [@Using p-values, S-values, and the Bayesian Rejection Ratio to re-examine a meta-analysis on knee abduction kinematics and ACL injury risk](https://discourse.datamethods.org/t/using-p-values-s-values-and-the-bayesian-rejection-ratio-to-re-examine-a-meta-analysis-on-knee-abduction-kinematics-and-acl-injury-risk/3887):
>
> A colleague brought to my attention the following peer-reviewed meta-analysis of 12 prognostic studies that attempted to predict future ACL injury risk from knee kinematics. Title: Do knee abduction kinematics and kinetics predict future anterior cruciate ligament injury risk? A systematic review and meta-analysis of prospective studies The authors found 9 studies, using 3 different assessments, that attempt to detect excess angular motion as a predictor of future ACL injury. They reported …

> [@Bayesian Qualitative Analysis of Credibility for Meta-analysis of Heterogeneous Data from RCTs](https://discourse.datamethods.org/t/bayesian-qualitative-analysis-of-credibility-for-meta-analysis-of-heterogeneous-data-from-rcts/3424):
>
> Synopsis: There exist clinical questions where effect size aggregation procedures are inappropriate. In the context of a highly heterogeneous data, frequentists can use various p value combination procedures. The research hypothesis would be: Blockquote “Assuming there is zero effect in all studies, is this collection of research results sufficiently surprising for me conclude there is at least 1 study with a true effect?” For most clinicians, that meta-analytic hypothesis is not very sat…

---

<div class="post-metadata">

**Author:** ![davidcnorrismd](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/davidcnorrismd/32/3502_2.png) [@davidcnorrismd](https://discourse.datamethods.org/u/davidcnorrismd)\
**Post date:** [October 11, 2021, 9:30am UTC](https://discourse.datamethods.org/t/a-bayes-frequentist-approach-to-the-analysis-of-credibility-of-meta-analyses-using-p-values/5004/2 "2021-10-11T09:30:47Z")

</div>

> [@R\_cubed](#):
>
> this description is how **an individual with no need to convince an external audience** , might use it. There are some adaptations I can imagine that would bring it in line with [Robert Matthew’s Analysis of Credibility](https://discourse.datamethods.org/t/frequentist-vs-bayesian-debate-applied-to-real-world-data-analysis-any-philosophical-insights/1695/9), to model a dialogue with a skeptical audience. [**emphasis** added]

Why would this be at all useful in _science_? Shouldn’t we always aim to be our own, harshest critics – Einsteins, not amoebas?  
[https://twitter.com/davidcnorrismd/status/991054068708605952?s=20](https://twitter.com/davidcnorrismd/status/991054068708605952?s=20)

---

<div class="post-metadata">

**Author:** ![R\_cubed](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/r_cubed/32/1518_2.png) [@R\_cubed](https://discourse.datamethods.org/u/R_cubed)\
**Post date:** [October 11, 2021, 10:24am UTC](https://discourse.datamethods.org/t/a-bayes-frequentist-approach-to-the-analysis-of-credibility-of-meta-analyses-using-p-values/5004/3 "2021-10-11T10:24:43Z")

</div>

> [@davidcnorrismd](#):
>
> Why would this be at all useful in _science_? Shouldn’t we always aim to be our own, harshest critics

From an information theoretic (and decision theoretic) sense, this is not realistic, nor necessary.

Scientific concerns and decision theoretic ones coincide, when maximizing information is equivalent to maximizing utility. An agent might not have enough of a budget to run a “definitive” experiment, and must factor in the cost of information within the context with which the decision will be made.

Put another way, being a “harsh critic” perspective might entail having such a low \alpha for your experiment that it has no power to detect anything surprising.

---

<div class="post-metadata">

**Author:** ![davidcnorrismd](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/davidcnorrismd/32/3502_2.png) [@davidcnorrismd](https://discourse.datamethods.org/u/davidcnorrismd)\
**Post date:** [October 11, 2021, 11:20am UTC](https://discourse.datamethods.org/t/a-bayes-frequentist-approach-to-the-analysis-of-credibility-of-meta-analyses-using-p-values/5004/4 "2021-10-11T11:20:21Z")

</div>

It sounds to me as if you are concerned with the problem of constructing _beliefs_ that are ‘useful’, rather than constructing _theories_ that are true. (I can’t help pointing out that we have in the US a neofascist party utterly committed to the programe of constructing what are for them highly useful ‘beliefs’, without regard for truth). By making budgetary (and other such) excuses for ourselves, and setting out to find a science that is useful, we obtain neither science nor usefulness. I’m sure you’ve heard the old saw about how ‘we ultimately have to make a clinical decision’ and so _must_ dichotomize.

---

<div class="post-metadata">

**Author:** ![R\_cubed](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/r_cubed/32/1518_2.png) [@R\_cubed](https://discourse.datamethods.org/u/R_cubed)\
**Post date:** [October 11, 2021, 11:29am UTC](https://discourse.datamethods.org/t/a-bayes-frequentist-approach-to-the-analysis-of-credibility-of-meta-analyses-using-p-values/5004/5 "2021-10-11T11:29:34Z")

</div>

I don’t want to get bogged down into philosophy or politics, although I agree that the Feynman idea of a scientist with integrity seems to have been long forgotten.

Suffice it to say there is modification that can incorporate a range of priors to model the dialogue between an honest advocate and an honest skeptic that will satisfy your concerns. I just want to know if I am making some glaring error in logic before I write any simulations.

Related Thread:

> [@Who should define reasonable priors and how?](https://discourse.datamethods.org/t/who-should-define-reasonable-priors-and-how/2935/13):
>
> I thought I’d bump this thread after finding a valuable discussion between Andrew Gelman and Sander Greenland on this precise issue of setting priors. Specifying a prior distribution for a clinical trial: What would Sander Greenland do? [https://statmodeling.stat.columbia.edu/2008/02/05/specifying\_a\_pr\_1/](https://statmodeling.stat.columbia.edu/2008/02/05/specifying_a_pr_1/) Specifying a prior distribution for a clinical trial [https://statmodeling.stat.columbia.edu/2008/01/24/specifying\_a\_pr/](https://statmodeling.stat.columbia.edu/2008/01/24/specifying_a_pr/) My posts in this thread were some speculations I had about how 2 hone…

---

<div class="post-metadata">

**Author:** ![davidcnorrismd](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/davidcnorrismd/32/3502_2.png) [@davidcnorrismd](https://discourse.datamethods.org/u/davidcnorrismd)\
**Post date:** [October 11, 2021, 12:22pm UTC](https://discourse.datamethods.org/t/a-bayes-frequentist-approach-to-the-analysis-of-credibility-of-meta-analyses-using-p-values/5004/6 "2021-10-11T12:22:21Z")

</div>

Arguably, your #inductivism IS the logical error 😉 . It will be interesting to see if your simulations prove telling on this point. I think sufficiently good simulations will!

> <https://twitter.com/davidcnorrismd/status/989516919861489664>

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [October 11, 2021, 12:22pm UTC](https://discourse.datamethods.org/t/a-bayes-frequentist-approach-to-the-analysis-of-credibility-of-meta-analyses-using-p-values/5004/7 "2021-10-11T12:22:47Z")

</div>

I don’t understand @R_cubed Robert’s motivation well enough. The procedure he outlined is a complex mix of Bayesian and frequentist ideas. Mixing the two seems to cause nothing but confusion. And the use of odds implies point hypotheses for both reference and alternative conditions. Even though zero is special in a certain way for a reference hypothesis, there is no special alternative value. So the procedure should use posterior distributions instead of odds even if it mixes in some frequentist ideas. But I would recommend just going with a Bayesian hierarchical model and being done with it.

Regarding multiplicity, we will stay clear if we distringuish likelihood-modifying multiplicity from non-likelihood-modifying multiplicity.

---

<div class="post-metadata">

**Author:** ![R\_cubed](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/r_cubed/32/1518_2.png) [@R\_cubed](https://discourse.datamethods.org/u/R_cubed)\
**Post date:** [October 11, 2021, 1:35pm UTC](https://discourse.datamethods.org/t/a-bayes-frequentist-approach-to-the-analysis-of-credibility-of-meta-analyses-using-p-values/5004/8 "2021-10-11T13:35:38Z")

</div>

> [@f2harrell](#):
>
> I don’t understand @R_cubed Robert’s motivation well enough. The procedure he outlined is a complex mix of Bayesian and frequentist ideas. Mixing the two seems to cause nothing but confusion. And the use of odds implies point hypotheses for both reference and alternative conditions.

🆕 Fair question. Sticking strictly to frequentist criteria, do you see any issue where the familywide \alpha would be higher than the nominal for this method? Is there any frequentist reason why someone would reject the procedure? That seems to be the most critical point for me ATM.

As for the “alternative condition” – in the analysis of p-values the default assumption that none of the studies are surprising enough to reject the no effect model (ie no clear sign in any study). By placing odds on this omnibus hypothesis allows me to have a procedure that states “These specific studies indicate a clear direction (sign), and this set of studies indicate at least 1 additional study with direction, with a posterior odds of X to 1” that will satisfy both Frequentists and Bayesians.

By using the step procedure, I’m able to argue that the estimated signs are the “true” signs based upon the frequentist criteria. For the studies that aren’t rejected, they still contain information about the sign, and at least according to the frequentist interpretation, and count as at least 1 study.

One problem I commonly find in rehabilitation is the aggregation of heterogeneous studies with different outcome scales. Take for example this [meta-analysis of functional electrical stimulation for CVA.](https://www.archives-pmr.org/article/S0003-9993(15)00044-1/fulltext) or the meta-analysis I linked to above on ACL injury prediction.

It has been awhile since I read the CVA paper, but they lumped together a variety of patient populations, outcome assessments, and statistical procedures using the “standardized mean difference” which we know is problematic. There was a very similar meta-analysis published in that journal a decade earlier, using similar analysis methods.

For the sake of discussion, assume that each study used reasonable procedures for the particular study conducted (ie. locally rational). I think various ordinal scales provide information; I also think interval and ratio measurements also provide value.

But in the face of such heterogeneity of methods, populations, etc, it isn’t clear how to combine them on an estimation scale to argue to a skeptic (who engages in what Sander calls nullism) that either better studies would be useful, or there is at least a preponderance of evidence to believe an intervention is worthwhile.

I have no doubt your suggestion is correct – when there is a well defined question to ask (and answer). In rehabilitation, we are usually in the stage where there is a jungle of ill-specified models, along with misapplied frequentist methods, to an equally large variety of assessment methods.

Other motivations:

1. Clinicians will continue to see p-values for a long time to come for the very reason Efron noted – they don’t ask for much from the researcher. Since [p-values have a Bayesian interpretation](https://journals.lww.com/epidem/Fulltext/2013/01000/Living_with_P_Values__Resurrecting_a_Bayesian.9.aspx) as a lower bound for the sign of the effect, there is no need to stick with the fixed cut-offs used in textbooks. I view this as part of a campaign to embrace, extend, and then extinguish this misleading “null” hypothesis significance testing and move toward more decision theoretic ideas.

2. Suppose we have a meta-analysis that finds “no evidence” via effect size averaging, but with 1 study is rejected (with multiplicity adjustment), but the aggregate of other studies is rejected with the opposite sign. We can then open the discussion to the differences among the studies, talk about covariates that need to be accounted for, and then introduce some of the ideas from RMS on what future studies should account for.

3. The method of analysis should be determined by the amount of information contained in the data.  
A large majority of meta-analyses in rehabilitation contain much less than 20 studies. Given the sources of heterogeneity above, I don’t see how textbook suggestions like meta-regression would be helpful. It is more honest to discuss direction of effect with small samples, and then figure out how to estimate it appropriately.

4. I can later argue for your ordinal longitudinal approach after they understand p-values well enough to comprehend that the parametric methods used on ordinal data don’t even calculate a valid p-value (or estimate) that can be used in a synthesis.

I have other motivations, but that is good for now. I always appreciate your input.

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [October 11, 2021, 6:07pm UTC](https://discourse.datamethods.org/t/a-bayes-frequentist-approach-to-the-analysis-of-credibility-of-meta-analyses-using-p-values/5004/9 "2021-10-11T18:07:41Z")

</div>

Boiling down the question to whether a sequential add-one-study-at-a-time procedure has any \alpha issues in the frequentist sense, it depends on what you _might do_ upon completing one of the early analyses. If you _might stop_ with a conclusion that you have evidence for an effect, then it seems that multiplicity adjustment is needed. Bayesians can argue that this frequentist procedure is not very logical, but if you are going to be a frequentist you should probably be a frequentist.

---

<div class="post-metadata">

**Author:** ![R\_cubed](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/r_cubed/32/1518_2.png) [@R\_cubed](https://discourse.datamethods.org/u/R_cubed)\
**Post date:** [October 11, 2021, 6:28pm UTC](https://discourse.datamethods.org/t/a-bayes-frequentist-approach-to-the-analysis-of-credibility-of-meta-analyses-using-p-values/5004/10 "2021-10-11T18:28:40Z")

</div>

One application is more of a sanity check on published syntheses that mangle the analysis.

One last question: Safe to assume you aren’t all that much enthusiastic for the Berger proposal in his Bayesian Rejection Ratio paper? This very method was used in large N genetic data to substantially improve prediction regarding future studies (by dramatically lowering \alpha for individual tests).

If there is a complaint – it is the notion of “Rejection” that might be troublesome for those who prefer estimation. Perhaps a better title would be “The unreasonable effectiveness of Bayesian methods to derive frequentist procedures.”

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [October 11, 2021, 6:44pm UTC](https://discourse.datamethods.org/t/a-bayes-frequentist-approach-to-the-analysis-of-credibility-of-meta-analyses-using-p-values/5004/11 "2021-10-11T18:44:23Z")

</div>

I think it may miss the point. The point is what is the prior for the signals in the high dimensional setting. This prior can be viewed equally as the prior information about the effect of one feature, or the population distribution of effects of all features together. This prior should be chosen carefully. The horseshoe prior is one good possibility. Then all posterior inference flows without modification for dimensionality.