# A Statistician Reads JAMA

**URL:** <https://discourse.datamethods.org/t/a-statistician-reads-jama/28225>\
**Category:** general\
**Tags:** p-value, multiplicity\
**Created:** [June 30, 2025, 7:21pm UTC](https://discourse.datamethods.org/t/a-statistician-reads-jama/28225 "2025-06-30T19:21:29Z")\
**Posts on this page:** 1\
**Showing post:** 15

<div class="post-metadata">

**Author:** ![R\_cubed](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/r_cubed/32/1518_2.png) [@R\_cubed](https://discourse.datamethods.org/u/R_cubed)\
**Post date:** [July 6, 2025, 3:16pm UTC](https://discourse.datamethods.org/t/a-statistician-reads-jama/28225/15 "2025-07-06T15:16:26Z")

</div>

> [@DylanArmbruster](#):
>
> To your question, I think it’s crucial to distinguish between the formal outcome of a Frequentist hypothesis test and one’s personal scientific assessment. I believe you’re asking me about the latter. In this case of 3 studies failing to reject, I don’t see the problem with thinking there is some credence toward that hypothesis.

Neyman advised setting different \alpha levels for different experiments. His test also formally used a likelihood ratio, not a p-value.

**Neyman, J. (1977).** Frequentist probability and frequentist statistics. _Synthese_, _36_(1), 97-131. [PDF](https://statmodeling.stat.columbia.edu/wp-content/uploads/2024/07/Neyman-FrequentistProbabilityFrequentist-1977.pdf)

> The elements common to all the situations typified by situation 5, will  
> be: (1) a hypothesis Ht to be tested against an alternative Ha and **(2) a  
> subjective appraisal of the relative importance of the two kinds of error,**

This gets overshadowed by idea that 0.05 is some “objective” line of demarcation, which is total nonsense and irrational. He was concerned about **rules for rational behavior** not reasoning under uncertainty, as Fisher was.

Fisher’s closely related, but distinct procedure only specifies 1 hypothesis and a sample size. He noted that p-values could be combined to detect departures from the reference null, even if there was no clear signal in any single study.

He was more willing to consider that the mathematical assumptions that specify the data generation process may not be entirely correct, reducing the ability to detect a signal at the individual study level, but not eliminating the information contained in the study.

> **[Fisher's method](https://en.wikipedia.org/wiki/Fisher%27s_method)**
>
> In statistics, Fisher's method, also known as Fisher's combined probability test, is a technique for data fusion or "meta-analysis" (analysis of analyses). It was developed by and named for Ronald Fisher. In its basic form, it is used to combine the results from several independence tests bearing upon the same overall hypothesis (H0).
> Fisher's method combines extreme value probabilities from each test, commonly known as "p-values", into one test statistic (X2) using the formula
> where pi is t...

The following has a quote from Fisher on combined tests:

> **[The logic of the Fisher method to combine P-values](https://brainder.org/2012/05/11/the-logic-of-the-fisher-method-to-combine-p-values/)**
>
> Consider a set of $latex k$ independent tests, each of these to test a certain null hypothesis $latex \\mathcal{H}\_{0|i}$, $latex i=\\{1, 2, \\ldots, k\\}$. For each test, a significance level $latex P…

On information grounds alone, a 2 sided p value ignores the sign, so we can halve the reported p value to 0.03.

Based on the base 2 log transformation of p values and the assumption that the statistics from the reported studies came from a distribution N(0,1), we have 3 \times -log\_2(0.03) \approx 15 bits of information against that hypothesis.

Alternatively, we could say 15 bits of information is lost by using the null reference model to compress the observed data. That is a huge amount of information loss. The Higgs Boson discovery in physics reported 22 bits of information against the reference null.

It is _not correct_ to say any study that “fails to reject” contains “no evidence” (aka. information).

> [@Relating S-values to other measures of statistical information](https://discourse.datamethods.org/t/relating-s-values-to-other-measures-of-statistical-information/4293):
>
> After reading this paper again: I wanted to share some R code I wrote to improve my intuition on how information measures relate to other common statistical quantities (ie Z-scores). # Converting among various measures of information and evidence # Shannons, p values, Z\_scores, Bayes Factor Bounds. bayes\_factor \<- function(p) { e \<- 2.71828 if (p \>= 1/e) { bayes\_factor \<- 1.00 } else { bayes\_factor \<- 1/(-e\*p\*log(p)) } } shannon\_table \<- function(lower, upper) { bits …

---

_[View the full topic](https://discourse.datamethods.org/t/a-statistician-reads-jama/28225)._