I think we have a different philosophy about the role of mathematical models. Let’s leave it at that.
Sure, if your philosophy of mathematical modelling doesn’t include correctness of derivations then it’s best to end this discussion.
I would like to ask you to refrain from claiming that your “Predictive Probability of Replication Success” is a mathematical consequence of certain assumptions (in particular assuming a flat prior). Just say it’s based on your experience of replication in practice when working as diagnostician, decision maker and researcher. Or say it came to you in a dream. It makes little difference.
I’m sorry, if I did that, I would be telling a lie! Please read post 175 again and reflect!
From the abstract of your preprint:
You’re saying that you’re not claiming that your “Predictive Probability of Replication Success” is a mathematical consequence of certain assumptions (in particular assuming a flat prior)?
Sure looks like it, though.
On the contrary, I have been claiming this all along and you asked me in post 182 to refrain from claiming it!
The remaining part of the abstract said this:
“The same model can be used to estimate sample sizes when planning the required power of the initial study. If this requires doubling the variance, then this will require double the sample size estimated using current power calculations, suggesting that studies using current methods are underpowered. These considerations might be used to explain the replication crisis.”
I could use the Open Science Data paper as an example to ask the DataMethods readers for their advice about this last sentence in the abstract, which is closely connected to my other posts.
The fundamental problem both @EvZ and I have is that, as far as he or I can tell, your formula conflicts with well established results from the algebra of random variables.
Specifically, for normal distribution, the variance of a sum (or difference) 2 independent, normally distributed random variables is N(E(X) \pm E(Y), var(x) + var(y)).
The difference between 2 independent random variables entails subtracting their expectations, but adding their variances. (see first link above)
I think both EvZ and I are assuming both estimates come from a Normal distribution. If that is so, then the difference is distributed as a normal N(0, 2). We can predict the probability of any difference by returning to the standard normal N(0,1) by dividing the observed difference by \sqrt{2}.
None of the attempts at formalizing your argument mathematically up to this point, agree with this.
This is all very confusing. You claim that your assumptions (including the uniform prior) imply that
P(z_\text{repl} > 1.96 \mid b,s,n_1,n_2) = \Phi\left( \frac{b}{\sqrt{\frac{s^2}{n_1} + \frac{s^2}{n_2}}} - 1.96\right).
Correct?
But then I demonstrated to you that your own assumptions actually imply
P(z_\text{repl} > 1.96 \mid b,s,n_1,n_2)= \Phi\left( \frac{b - 1.96\frac{s}{\sqrt{n_2}}}{\sqrt{\frac{s^2}{n_1} + \frac{s^2}{n_2}}} \right).
You agreed “jolly well” with my derivation, but then mysteriously claimed that the discrepancy is due to a “different philosophy about the role of mathematical models”.
Can you please explain which special philosophy allows you to violate the rules of algebra which we all learn in high school? I couldn’t find such an explanation in your pre-print.
As best as I could tell, he wants to assume a uniform prior after the observation of the first estimate.
The two different philosophies about the role of mathematical models are:
- If your model is inconsistent with the data, then the data is wrong and the model is right so ignore the data and keep the model
- If your model is inconsistent with the data, then the model is wrong and data is right so you have to change your model.
The data in Post 175 (from you very own paper) is inconsistent with with the model that you propose and consistent with the model that I propose. This suggests that you think that your own data is wrong and should be ignored and your model is right, so you follow philosophy 1 above.
I am following philosophy 2 above and adopted model that is consistent with your data. Why have you not responded to my Post 175 and you data but ignored it?
Not at all. I assumed a uniform prior because I was going to use numbers to make my measurements before I even knew what I am going to investigate (let alone after making observations)..
I would be interested to know where this happened.
I’m definitely not proposing the uniform prior. In fact, I wrote a paper about that with Steve Goodman (2022).
The point is that you are – or at least appear to be – proposing a model based on the uniform prior. Now, I understand that you are unhappy with the consequence, namely
P(z_\text{repl} > 1.96 \mid b,s,n_1,n_2)= \Phi\left( \frac{b - 1.96\frac{s}{\sqrt{n_2}}}{\sqrt{\frac{s^2}{n_1} + \frac{s^2}{n_2}}} \right),
because it doesn’t agree with your experience of replication in practice when working as diagnostician, decision maker and researcher. According to your philosophy 2 (which I agree with), you should change the model. Instead, you only change the consequence of your model into
P(z_\text{repl} > 1.96 \mid b,s,n_1,n_2) = \Phi\left( \frac{b}{\sqrt{\frac{s^2}{n_1} + \frac{s^2}{n_2}}} - 1.96\right).
Now there’s no coherent probability model anymore. In your post 169 you write
P(b_\text{repl} \mid b,s,n_1,n_2,U)∼N(b, \frac{s^2}{n_1}+\frac{s^2}{n_2})
and then present your new formula for the Predictive Probability of Replication Success. But those two probability statements are incompatible.
I am unhappy with the top expression because it is inconsistent with your data in Post 175 and also suggests that the probability of replication increases with less data when P>0.05. If, as you say, you follow philosophy 2 you should change your model. However, the bottom expression proposed by me is compatible with the data, so you should change to my model (or another model compatible with the data). Please address this issue, otherwise you are just taking us around in circles with diversionary tactics in keeping with philosophy 1.
In statistics, we try to come up with coherent probability models that involve the relevant variables, and that agree with the important aspects of the data. In all modesty, a good example is my paper with Goodman.
You, on the other hand, have an incoherent model. For example, you make the following two statements which are actually incompatible:
P(b_\text{repl} \mid b,s,n_1,n_2,U)∼N(b, \frac{s^2}{n_1}+\frac{s^2}{n_2})
and
P(z_\text{repl} > 1.96 \mid b,s,n_1,n_2) = \Phi\left( \frac{b}{\sqrt{\frac{s^2}{n_1} + \frac{s^2}{n_2}}} - 1.96\right).
The second statement happens to agree reasonably well with the results of my paper with Goodman. But those results are motivated by an actual data analysis, and they are part of a fully coherent probability model.
It just ocurred to me that our confusion/disagreement might be explained if you referred to the standalone statement
P(z_\text{repl} > 1.96 \mid b,s,n_1,n_2) = \Phi\left( \frac{b}{\sqrt{\frac{s^2}{n_1} + \frac{s^2}{n_2}}} - 1.96\right)
as your “model”. Is that so?
By claiming that the two expressions are incompatible, you appear to be confusing a predictive distribution with a probability of a specific event. The distribution over b_repl (on the original scale) must be standardized to compute the probability that the replication test statistic exceeds 1.96. The second expression is simply the result of applying that standardization and threshold to the first.
How do you explain why the second statement happens to agree reasonably well with the results of your paper with Goodman if it is not due to appropriate modelling?.
By claiming that the two expressions are incompatible, you appear to be confusing a predictive distribution with a probability of a specific event.
As I demonstrated to you (and you claimed to understand), the statement
P(b_\text{repl} \mid b,s,n_1,n_2,U)∼N(b, \frac{s^2}{n_1}+\frac{s^2}{n_2})
is equivalent to the statement
P(z_\text{repl} > 1.96 \mid b,s,n_1,n_2)= \Phi\left( \frac{b - 1.96\frac{s}{\sqrt{n_2}}}{\sqrt{\frac{s^2}{n_1} + \frac{s^2}{n_2}}} \right).
Therefore, it is incompatible with the obviously different statement
P(z_\text{repl} > 1.96 \mid b,s,n_1,n_2) = \Phi\left( \frac{b}{\sqrt{\frac{s^2}{n_1} + \frac{s^2}{n_2}}} - 1.96\right).
Is that really so difficult to understand?
How do you explain why the second statement happens to agree reasonably well with the results of your paper with Goodman if it is not due to appropriate modelling?
The results in my paper with Goodman are specific to the clinical trials in the Cochrane database. They would have been different if we’d considered only phase III clinical trials in oncology or experiments in parapsychology or whatever else. You appear to claim that your formula always applies. Or at least you haven’t indicated when it applies and when it doesn’t.
1 Like
The closest you’ve come to specifying an actual model is your post 169. Unfortunately, the final statement of that post is incompatible with the earlier statements. Are you able to correct that? In other words, can you specify a model for \beta, b, z, b_\text{repl} and z_\text{repl} that is not contradictory?
for the above statement the equivalent statement is
P(z_{\text{repl}} \mid b, s, n_1, n_2, U) \sim \mathcal{N}\left( \frac{b \sqrt{n_1}}{s},\ 1 + \frac{n_1}{n_2} \right)
However for the following statement:
P(z_\text{repl} > 1.96 \mid b,s,n_1,n_2) = \Phi\left( \frac{b}{\sqrt{\frac{s^2}{n_1} + \frac{s^2}{n_2}}} - 1.96\right).
The equivalent statement is
P(b_\text{repl} \mid b,s,n_1,n_2,U)∼N(b, \frac{s^2}{n_1}+\frac{s^2}{n_2})
NO. It has to be tested against other data. My expression is compatible with your data and on first glance, the Open Science Collaboration and other data. However, your expression is not compatible with your own data, which shows already that there is something wrong with it.
This is getting positively weird now. Contrary to your claim, I proved that
P(b_\text{repl} \mid b,s,n_1,n_2,U)∼N(b, \frac{s^2}{n_1}+\frac{s^2}{n_2})
is equivalent to
P(z_\text{repl} > 1.96 \mid b,s,n_1,n_2)= \Phi\left( \frac{b - 1.96\frac{s}{\sqrt{n_2}}}{\sqrt{\frac{s^2}{n_1} + \frac{s^2}{n_2}}} \right).
Can you point to a mistake in my proof? And can you prove your equivalence?