I used ‘star’ as my multiplying sign in my last post, which was interpreted by Datamethods as a ‘begin italics’ code and distorted my expressions in places. I will resume the discussion when I have time to create proper formulae in Word and then copy and paste them into Datamethods as images. It looks as if I should also revisit some earlier exchanges to check for misunderstandings. I am preparing to give two talks on Saturday so I will respond after the week-end.
It looks as if I should also revisit some earlier exchanges to check for misunderstandings.
Sure. It’s probably best if you have a look at my post 136, which has the set-up and what I believe to be your claim.
I would like to ask that you do not retroactively change earlier posts (139) because that is very confusing for others who might try to follow the discussion.
Before you send any more formulas after the weekend, I’ll take the opportunity to explain why I think you made a simple bracketing error.
To be clear, the set-up is: We start with \beta which is the unkown true effect. We have two studies; the “original” and the “replication” that both target beta. Let’s assume they have the same sample size n and standard deviation s. Suppose they yield estimates b and b_\text{repl} which are unbiased with the same standard error se=s/\sqrt{n}. In other words, conditionally on \beta, b and b_\text{repl} are independent, normally distributed with mean \beta and standard deviation se. Now define \text{SNR}=\beta/se, z=b/se and z_\text{repl}=b_\text{repl}/se. It follows that conditionally on SNR, z and z_\text{repl} are independent, normally distributed with mean SNR and standard deviation 1. We are interested in the conditional probability of a statistically significant replication, given the z-statistic of the first study. So, that’s P(z_\text{repl} > 1.96 \mid z).
Now, you’re claiming the following:
If we assume the (improper) uniform (or “flat”) prior for SNR, then
z_\text{repl} \mid z \sim N(z/\sqrt{2},1). This implies
P(z_\text{repl} > 1.96 \mid z) = 1 - \Phi(1.96 - z/\sqrt{2}).
Unfortunately, this is an unambiguous mathematical mistake. The correct statement is in Goodman (1992) and vZ and Goodman (2022). It is
If we assume the (improper) uniform (or “flat”) prior for SNR, then
z_\text{repl} \mid z \sim N(z,\sqrt{2}). This implies
P(z_\text{repl} > 1.96 \mid z) = 1 - \Phi((1.96 - z)/\sqrt{2}).
Comparing your formula
P(z_\text{repl} > 1.96 \mid z) = 1 - \Phi(1.96 - z/\sqrt{2})
to the correct formula
P(z_\text{repl} > 1.96 \mid z) = 1 - \Phi((1.96 - z)/\sqrt{2})
strongly suggests that you made a bracketing error.
In your post 135, you even admitted being unaware of the distinction:
However, what I am used to seeing is p(z_repln>1.96|z) = pnorm((1.96-z)/sqrt(2)). So I’m sorry that I missed the subtle difference where you replaced pnorm((1.96-z)/sqrt(2)) with 1-pnorm(1.96-z/sqrt(2)). As you say, this gives the same result as my version of pnorm(1.96/sqrt(2) - 1.96), which is a simplified version of p(z_repln>1.96 | b, s, n1, n2) = pnorm(b/sqrt(n1) +s/sqrt(n))-1.96).
Finally, I want to stress that it is a mere coincidence that your erroneous formula leads to similar results as Goodman and I obtained with an empirical prior for the SNR. By using the flat prior you explicitly do not take any empirical information into account.
2 Likes
Please use \LaTeX math notation here, set off with single dollar signs.
2 Likes
There was no bracketing error on my part as explained below. My expression is:
P(Z_{\text{repln}} > 1.96 \mid b, s, n_1, n_2) = \Phi\left( \frac{b}{\sqrt{\left( \frac{s}{\sqrt{n_1}} \right)^2 + \left( \frac{s}{\sqrt{n_2}} \right)^2}} - 1.96 \right)
Define:
v_1 = \left( \frac{s}{\sqrt{n_1}} \right)^2, \quad v_2 = \left( \frac{s}{\sqrt{n_2}} \right)^2
The above expression simplifies to:
\Phi\left( \frac{b}{\sqrt{v_1 + v_2}} - 1.96 \right)
In the special case where v_1 = v_2 , we get:
\Phi\left( \frac{b}{\sqrt{2 \cdot v_1}} - 1.96 \right)
= \Phi\left( \frac{b}{\sqrt{2} \cdot \sqrt{v_1}} - 1.96 \right)
= \Phi\left( \frac{b}{\sqrt{2} \cdot \text{SE}_1} - 1.96 \right)
= \Phi\left( \frac{b}{\text{SE}_1 \cdot \sqrt{2}} - 1.96 \right)
However, you chose to rearrange my formula to:
1 - \Phi\left( 1.96 - \left( \frac{b}{\text{SE}_1 \cdot \sqrt{2}} \right) \right)
Now, when b = 2 and \text{SE}_1 = 1 , my formula as rearranged by you yields:
1 - \Phi\left( 1.96 - \left( \frac{2}{\text{1} \cdot \sqrt{2}} \right) \right) = 1 - \Phi(0.547) \approx 1 - 0.707 = 0.293
However, under your formula:
P(Z_{\text{repln}} > 1.96 \mid b, s, n_1, n_2) = 1 - \Phi\left( \frac{1.96 - \left( \frac{b}{\text{SE}_1} \right)}{\sqrt{2}} \right)
we get:
1 - \Phi\left( \frac{1.96 - \frac{2}{1}}{\sqrt{2}} \right) = 1 - \Phi(-0.0283) \approx 1 - 0.4887 = 0.511
Therefore, there was no bracketing error on my part or your part. So the result was not just a happy far-fetched coincidence from an error. It appears to be an interesting estimate of the findings in your 2022 paper with Goodman by applying a careful reasoning process. This appears to have interesting implications.
Regarding this:
P(Z_{\text{repln}} > 1.96 \mid b, s, n_1, n_2) = \Phi\left( \frac{b}{\sqrt{\left( \frac{s}{\sqrt{n_1}} \right)^2 + \left( \frac{s}{\sqrt{n_2}} \right)^2}} - 1.96 \right)
This notation looks confused to me. IIRC,
- b stands for the unknown parameter,
- s stands for the estimate, and
- n_1 and n_2 stand for sample sizes.
AFAICT:
I. The variable in 1 is not known and can be ignored for the purpose of calculation of the numerator of the Signal-Noise Ratio.
II. The variable in 2 has no defined value (ie. you aren’t conditioning on any particular estimate or standardized estimate ie. a Z score.)
III. Once you standardize your estimate via the normality assumption, the sample sizes aren’t needed. It should be understood that Z scores carry a standard error of \pm 1. The sample sizes are used in T test procedures when you estimate the variance from the data via Welch’s formula:
t=\frac{\bar X_1 - \bar X_2}{\sqrt{(s_1^2/\sqrt{n_1}) + (s_2^2/\sqrt{n_2})}}
There is an additional variable to account for degrees of freedom as described here:
If your question is: Conditional on H_0 = 0, P(Z_1 \ge 1.96 and Z_2 \ge 1.96 | b = 0), then that should simply be \alpha=0.05^2. Again, it is not exactly clear what your assumptions are.
As defined previously in the discussion:
b is the observed average, raw effect of the completed first study
s is the standard deviation of the data in the first complete study
n1 is the number of observations or sample size of the completed first study
n2 is the planned sample size of a second replicating study not yet done
This isn’t compatible with @EvZ’s notation, then. If you are conditioning on b, it should be to the right of the vertical bar “|” with b=[number].
So, what values of b, s, n1 and n2 are you using? And are you assuming normality?
See the last 4 expressions of my post (145) based on b=2, s=10, n1 = 100 and n2=100 if used in the first expression in the post. I did assume normality as was assumed when Goodman derived his expression.
Just to be clear, this is how I would have expected your assumptions to be written:
P(Z_{rep} > 1.96 | b \sim N(2,1) )
So now, what are you assuming about the distribution of study 2? Is it coming from a distribution N(2,1)?
If so, the predictive distribution is what @EvZ wrote above:
z_{repl}|z \sim N(z,\sqrt{2})
Substituting your values:
z_{repl}|z \sim N(2,\sqrt{2})
The value of z differs between the two methods. I therefore prefer to express results in terms of b, s, n1, n2. Thus @EvZ’s expression is
P(Z_{\text{repl}} > 1.96 \mid b, s, n_1, n_2) = 1 - \Phi\left( \frac{1.96 - \dfrac{b \sqrt{n_1}}{s}}{\sqrt{1 + \dfrac{n_1}{n_2}}} \right) .
Compare this to my expression on the top of post 145 repeated here:
P(Z_{\text{repln}} > 1.96 \mid b, s, n_1, n_2) = \Phi\left( \frac{b}{\sqrt{\left( \frac{s}{\sqrt{n_1}} \right)^2 + \left( \frac{s}{\sqrt{n_2}} \right)^2}} - 1.96 \right)
Many of the assumptions are similar but with important nuances that are not reflected easily by the notation N(x, y). Perhaps you could suggest how you think that the distributions should be described in the light of my expression and that of @EvZ.
If that is true, you are contradicting one of your own assumptions – ie. that the 2 estimates come from the same distribution N(\theta, \sigma), or that the variance is known.
I don’t know how you compute anything without making a distributional assumption about your hypothetical data set.
The notation N(\theta, \sigma) makes clear we are conditioning on a mean value \theta, with the uncertainty expressed as a variance parameter \sigma.
The prediction interval for a future study, conditional on the observed study only and the assumption of normality, is going to be centered around the observed sample mean, with the variance of the prediction being a sum of the variance of 2 normals.
What I do is estimate the distribution of beta conditional on b, s, n1, n2 and a uniform prior. For each possible value of beta, I estimate the distribution of p(b_repln) by convoluting the two distributions each with mean b and SE =
SE(b, s, n1, n2) = \left( \frac{b}{\sqrt{v_1 + v_2}} \right)
when
v_1 = \left( \frac{s}{\sqrt{n_1}} \right)^2, \quad v_2 = \left( \frac{s}{\sqrt{n_2}} \right)^2
Is your prior proper or improper?
It is improper if the small number p(beta_i) is 0 when 0/0 is undefined but proper if the small number p(beta_i) is delta and delta/delta = 1 and ∑p(beta_i) = 1 without knowing the exact value of beta, but the result is basically the same.
I don’t understand what you are doing here. The improper uniform prior is only relevant before the first estimate is observed. After that first estimate is observed, your posterior before the replication study has a form of N(z,\sqrt{2}). The uniform prior doesn’t add a term to the numerator (so no shrinkage in the sense of the mean), but it adds variance compared to the naive assumption that the unknown parameter \beta = b (the estimate).
If I’m thinking about this correctly, the form of the Bayesian credible interval (derived from a uniform prior over the SNR) has the same form as the frequentist prediction interval: N(z,\sqrt{2}) The Bayesian credible interval refers to the parameter that best describes the observed results for this particular data set. The frequentist prediction interval has the 1-\alpha coverage for future observations (if the assumptions used to derive it are correct).
A Bayesian prediction interval should incorporate the uncertainty in the yet to be conducted study, given our knowledge and modelling assumptions, so it should have the form N(z, \sqrt{3})
If I wish to estimate the posterior distribution of beta for the current study, then I do so by assuming a flat prior and that SE(b, s, n1) = \frac{s}{\sqrt{n_1}} . If I wish to estimate the posterior distribution for b_repln or or the possible means of a repeat study, I do so directly by assuming a flat prior again and a single convolved distribution with b and s so that SE(b, s, n1, n2) is based on a combination of two samples n1 and n2 so that when
v_1 = \left( \frac{s}{\sqrt{n_1}} \right)^2, \quad v_2 = \left( \frac{s}{\sqrt{n_2}} \right)^2
then
SE(b, s, n1, n2) = \left( \frac{b}{\sqrt{v_1 + v_2}} \right)
Why do you assume a flat prior after your first estimate? That amounts to ignoring the likelihood function from the first study.
Please study the following:
Kass, R. E. (1990). Data-translated likelihood and Jeffreys’s rules. Biometrika, 77(1), 107-114. (PDF)
Because they are totally independent calculations for two different distributions.
Regarding using uniform priors after the first estimate, you wrote:
This is wrong. The notation N(\theta, \sigma) denotes the distribution, so there is nothing preventing the realization of random variables from such a distribution from having different values. Random number generators do this all the time. This use of a uniform distribution after the first study amounts to ignoring your information (ie. the form of the likelihood) from the first estimate.
When doing computations, we substitute an expectation (measure of location), and make our uncertainty in the possible sample values explicit with the variance (measure of scale).
Do read that paper by Kass on uniform distributions and Jeffrey’s rules for non-informative priors.