Where have I said something like this or analogous to this? Can you specify the line in Post 288?
I’m done with this. Read my posts above.
1 Like
Hi Huw
Is this a correct summary of the three steps you provided in your first post?
- You observe b = 2 & v1 = 1, but this is just one draw from β∼N(β,v1) = N(2, 1), the latter because you’re uncertain about β
- Then you model a replication as:
b2∼N(β,v2) - Putting these together and assuming v1 ≈ v2:
- Predictive distribution for replication b2∼N(b,v1+v2) = N(2, 2)
The shaded area above z = 1.96 under this distribution is the replication probability ≈ 29.3%.
I am not going to enter this discussion - just appreciate your feedback on the above.
Thank you for your interest. Yes what you say does summarise my approach.
I have been doing my best to convey what is intuitively clear to me (e.g. when working in Excel). I have been trying to do so in mathematical notation written via LaTex, (as suggested by @f2harrell) with which I am not familiar but on a steep learning curve. Following @EvZ 's recent reply to @R_cubed where he summarised the derivation of an expression that he uses, I have tried to derive the expression that I use in the same brief but hopefully mathematically sound way using the same notation. Here it is in case you or anyone else is interested.
Assume a flat prior on \beta and
b \mid \beta, v_1 \sim N(\beta, v_1)\quad \text{and} \quad b_{\mathrm{repl}} \mid beta, v_2 \sim N(\beta, v_2),
where b and b_{\mathrm{repl}} are conditionally independent given \beta.
-
Since b \mid \beta, v_1 \sim N(\beta, v_1) and \beta has a flat (uniform) prior, it follows that \beta \mid b, v_1 \sim N(b, v_1).
-
Since b_{\mathrm{repl}} \mid \beta, v_2 \sim N(\beta, v_2) and b and b_{\mathrm{repl}} are conditionally independent given \beta, it follows that :b_{\mathrm{repl}} \mid b, v_1, v_2 \sim N(b, v_1 + v_2).
-
Standardize the (conditional) predictive distribution by dividing b_{\mathrm{repl}} by the posterior predictive standard deviation \sqrt{v_1 + v_2}.
Replication success is defined on the same two-sided 0.05 significance scale, i.e.
P(z_{\mathrm{repl}} > 1.96 \mid b, v_1, v_2) = P\!\left(\frac{b_{\mathrm{repl}}}{\sqrt{v_1 + v_2}} > 1.96 \right). -
Conditionally on b, v_1, v_2, the standardized predictive replication distribution is
\frac{b_{\mathrm{repl}}}{\sqrt{v_1 + v_2}} \sim N\!\!\left( \frac{b}{\sqrt{v_1 + v_2}},\, 1 \right).
Hence the replication success probability is
P(z_{\mathrm{repl}} > 1.96 \mid b, v_1, v_2) = \Phi\!\!\left(\frac{b}{\sqrt{v_1 + v_2}} -1.96\right).
Finally, substituting v_1 = \left( \frac{s}{\sqrt{n_1}} \right)^2 and v_2 = \left( \frac{s}{\sqrt{n_2}} \right)^2, we obtain P(Z_{\text{repln}} > 1.96 \mid b, s, n_1, n_2) = \Phi\left( \frac{b}{\sqrt{\left( \frac{s}{\sqrt{n_1}} \right)^2 + \left( \frac{s}{\sqrt{n_2}} \right)^2}} - 1.96 \right)
To be specific, b_\text{repl}/\sqrt{v_1+v_2} cannot be considered as a “replication statistic”. So, your “replication success probability” has nothing to do with replication success. It’s misleading. Stop it!
You are correct of course that \frac{b_{\mathrm{repl}}}{\sqrt{v_1 + v_2}} is not a test statistic nor a replication statistic. I have been asking for advice for some time during the past few weeks as to what to call it. I have moved on from calling it a ‘replication statistic’ as advised by @f2harrell. It was never meant to be the replication study’s own test statistic by the way because I don’t focus on the replication study in isolation. Instead, I am predicting, from the vantage point of the original study, the probability that a future replication study result would again be significant at the 0.05 level.
Therefore, \frac{b_{\mathrm{repl}}}{\sqrt{v_1 + v_2}} is a predictive standardization arising from combining uncertainty about \beta after the original study (v_1) with replication noise (v_2) arrived at by convolving the two distributions. It is the scale for computing the predictive probability that the replication study’s own test will again exceed the usual 0.05 cut-off.
So I don’t think that I am ignoring @f2harrell 's advice because I have taken it and moved on by changing my terminology and regarding \frac{b_{\mathrm{repl}}}{\sqrt{v_1 + v_2}} as a predictive standardization arising from combining uncertainty about \beta from the original study’s (v_1), with replication noise (v_2).
Instead, I am predicting, from the vantage point of the original study, the probability that a future replication study result would again be significant at the
0.05 level.
That would be
P(b_\text{repl}/\sqrt{v_2} > 1.96 \mid b,v_1,v_2) = \Phi\left( \frac{b - 1.96\sqrt{v_2}}{\sqrt{v_1+v_2}} \right)
as I’ve explained many times. See also Goodman (1992).
Your b_\text{repl}/\sqrt{v_1+v_2} is not a useful statistic. It is misleading to refer to the (conditional) probability that it exceeds 1.96 as the “replication success probability”. By continuing to do so, you are essentially forcing me to keep correcting you. You are abusing this platform, and you need to stop it!
That expression does not appear in Goodman (1992). Can you point us to the publication where that expression is described and discussed?
RE:The derivation posted @EvZ
This simply follows from the axioms for arithmetic on random variables and standardization as I’ve posted numerous times.
You should be able to follow the derivation, considering you claim to understand “convolution” of probability distributions, which at this point, is doubtful.
Of course I can follow it and it is clear to me at last why it produces probabilities of replication that are different to the expression that I have described. The mathematics of both are of course sound, but the assumptions and understanding of what is meant by ‘probability of replication’ on which the calculations are based are different. I was interested to know if there was a publication where these assumptions were discussed in detail. I have not able to find such a publication so far by searching the web.
It’s absolutely obvious to everyone but you that significance of the replication study means |b_\text{repl}/\sqrt{v_2}|>1.96. Now can you please stop this inane thread!
I would like to see published evidence for this, or at least a publication that has allowed it to be discussed openly, calmly and thoughtfully. However, I am prepared to accept the above very strong assertion as an assumption which forms the basis for deriving an expression. Thank you for an enlightening discussion.
Hi Huw,
I am happy to take this conversation offline with you as I am interested. However note that framing it as an alternative to conventional replication methods will not fly. You will have to clarify in your mind exactly what you are computing (and there is no need for this to be framed in replication terms) and how it helps within the framework of medical research. Then work out what this adds vis a vis the conventional methods. To put this in context, perhaps your preprint should follow the style of Senn’s paper “Trying to be precise about vagueness” (look it up on PubMed) as that sort of approach will allow your ideas to focus on the problem at hand.
Hi Suhail
Thank you for your interest. In my pre-print I try to frame the issue as an attempt to improve mutual understanding between statisticians, doctors and biological scientists by looking at fresh ideas from a different viewpoint, especially from a medical reasoning point of view. Perhaps I need to emphasise this more strongly. There may strong resistance to new ideas of course.
Hi Huw, I agree. As Mahoney puts it, “It is only contrary-to-prediction experiments which carry logical implications” and the mandate for us all must exclude this “dogmatically confirmatory tradition”.
This appears to have been the object from early on. It was by resisting it and asking questions that I learnt so much. I am grateful for what I learnt, although the process was unsettling. I don’t understand how the central limit theorem has any bearing on the validity of these discussions. Thank you for your interest.
Hi again Suhail
I agree. like many other people, I have some understanding of replication from my clinical work (e.g. as an ‘attending physician’ with responsibility for trying to replicate the findings of resident doctors many times a day) as well as in research and interpreting clinical research publications. I have tried to express my understanding in terms of a model based on probability theory (but not previously in the notation and much vocabulary used by @EvZ, who challenged me to do so). As you suggest (and as you point out according to Mahoney), it is important to test the logical implications of such a model against data from experiment, which is what I am trying to do.
One other useful tip since you use excel is a MC simulation tool we created as a plugin for excel. This can be downloaded from www.epigear.com and is called Ersatz. This will help you visualize your ideas clearly. If you need help installing it please send me a PM. The install is a bit complicated as it was formerly paid and we made it free now.
This is obvious, and any argument against this must now cease. Any concept of a replication statistic must reflect how practitioners actually compute such statistics.
3 Likes
Additionally, Huw, you are well aware that the replication crisis exists because there is an assumption somewhere that statistical significance implies a replicable finding and this assumption is perhaps based on inadvertently falling into the trap of glorifying p < 0.05 as a marker of scientific truth – which it is not. As such, in line with Franks comment, best you steer clear of replication (it is not a useful concept) and find a meaning to what you observe and focus on that. No one will help you clarify your thoughts if you keep bringing back replication. I think you should just reflect on this privately until you find an application for your idea and then start a new thread focusing on that application. Finally, I have immensely benefitted from your input in very many threads on this forum and look forward to seeing your input somewhere else.
2 Likes