Some thoughts on uniform prior probabilities when estimating P values and confidence intervals

[R_cubed]

You refer to:

@HuwLlewelyn

The Open Science Collaboration. Estimating the reproducibility of psychological science. (link added)

I’m very skeptical of anything psychologists publish, given that they naively use parametric methods on ordinal data. I never felt comfortable using assessments designed like this, and I finally convinced myself it produces no information, which I argued in this thread. You can’t rely on their statistics without a laborous examination of of the included studies.

Preprint: Analysis of Likert Type Data Using Metric Methods

Dennis Boos and Judy Chen (2022). Analysis of Likert-Type Data Using Metric Methods. https://researchers.one/articles/22.01.00002 I edited this to bring the main points earlier, and the decision theoretic justification later in the post. Introduction The authors defend the common practice of using parametric models on ordinal data on the basis of pragmatics (ie.“easier” to for researchers understand), and perceived robustness in that parametric methods on empirical or simulated data se…

I have much more confidence in these publications, which are closely related to the questions you bring up in this thread:

van Zwet, E., Gelman, A., Greenland, S., Imbens, G., Schwab, S., & Goodman, S. N. (2023). A new look at P values for randomized clinical trials. Nejm evidence, 3(1), EVIDoa2300003.

van Zwet, E. W., & Goodman, S. N. (2022). How large should the next study be? Predictive power and sample size requirements for replication studies. Statistics in Medicine, 41(16), 3090-3101. (link)

Goodman, S. N. (1992). A comment on replication, p‐values and evidence. Statistics in medicine, 11(7), 875-879. (link)