Some thoughts on uniform prior probabilities when estimating P values and confidence intervals

[R_cubed]

@HuwLlewelyn

I regard uniform priors as the prior probabilities of possible true results and possible observed results of any study conditional on the universal set of all numbers (i.e. before any information about the nature of the study is known). These priors are uniform.

In order to compute a “replication probability” of an already conducted study, what are you conditioning on?

I think @EvZ was pretty clear in the various types of hypotheses one could ask, their statistical assumptions, and the computations.

Perhaps someone could correct me if I’m wrong, but this is how I understand his calculations for a replication study where “replication” means the same sign, and a SNR \ge Z_1.

Addendum: The correction to my reasoning is found in the Steve Goodman paper mentioned above. A true Bayesian answer results in shrinkage a reduction of replication probabilities relative to the assumption Z_{init} = SNR_{true} even with a uniform prior; my reasoning assumes the initial estimate in Z_{init} = SNR_{true}, so I have re-named it a “naive frequentist” analysis.

Goodman, S. N. (1992). A comment on replication, p‐values and evidence. Statistics in medicine, 11(7), 875-879. (link)

Naive Frequentist Analysis

Before study 1:

1. Prior:(Uniform over \Re) None;
2. Sampling Distribution N(0,1)
3. Predictive distribution: Uniform Normal with parameters estimated from data

After Study 1, before replication :

1. Prior: N(\theta, 1), \theta = 1.96 \pm 1,
2. Sampling Distribution N(0,1);
3. Predictive Distribution: N(\theta, 1), \theta = 1.96 \pm 1.

The N(\theta,1), \theta \ne 0 gives a (naive) 68% confidence interval for the SNR after the first study.

Standard error

After the first study, the naive frequentist observation of the SNR (Z score) provides no shrinkage to the observed results, leading someone who was ignorant before seeing the data to conclude that the sign of the parameter is in the same direction as the sign of the estimate, and his/her “best guess” at the true SNR is the MLE of the collected data, which is equal to the 50th percentile (ie. p=0.5) under the assumed sampling model. This is only credible with a large amount of data.

Even with shrinkage a discounting of the probability of replication provided by the uniform prior, it is very unrealistic, as was noted here:

Then too a lot of Bayesians (e.g., Gelman) object to the uniform prior because it assigns higher prior probability to β falling outside any finite interval (−b,b) than to falling inside, no matter how large b; e.g., it appears to say that we think it more probable that OR = exp(β) > 100 or OR < 0.01 than 100>OR>0.01, which is absurd in almost every real application I’ve seen.

What are credible priors and what are skeptical priors?

A few weeks ago Dan Scharfstein asked a group of colleagues about how to report an odds ratio of 1.70 with 95% confidence limits of 0.96 and 3.02. Back-calculating from these statistics gives a two-sided P of 0.06 or 0.07, corresponding to an S-value (surprisal, log base 2 of P) of about 4 bits of information against the null hypothesis of OR=1. So, not much evidence against the null from the result, but still favoring a positive association over an inverse one, and so thought worthy of reportin…

The empirically derived priors place most of the probability mass near the center of the distribution (closer to 0), and provide better shrinkage discounting of the replication probabilities, improving prediction of future studies.

If you also notice, this definition of a replication study can simultaneously have a Z_{rep}:

1. attain a p > Z_1
2. have a \frac{Z_{max} - Z_{min}}{\sqrt{2}} \lt 1 which would show the estimates are quite compatible.

This definition of “replication” ignores 1 side of a symmetrical distribution.