Some thoughts on uniform prior probabilities when estimating P values and confidence intervals

I would be happy to debate the validity of these results from psychological studies with them. When 2 mathematical psychologists (Joel Michell and Paul Barret) characterize psychometrics as a “pathological science”, and the entire “replication crisis” is centered in psychology, I think it is reasonable to severely discount anything coming out of that area of study.

They aren’t the only ones who argue that psychological data is at best ordinal, and should be statistically examined using ordinal methods. Psychologist Norman Cliff published 2 books on ordinal methods that share many similarities with the recommendations of @f2harrell, even though he didn’t go into as much detail on the proportional odds model.

Suffice it to say, the Cochrane Database of medical RCTs is more relevant to your question.

Related Thread:

Although various statistical methods are available for the analysis of PROs in RCT settings, there is no consensus on what statistical methods are the most appropriate for use.

Instead of deriving the right path from first principles, they go through a fruitless exercise of empirically fitting various models, much like is done in psychometrics. But they are correct in that most researchers argue about how to properly analyze the data, with those favoring treating ordinal data as interval having positions of authority. That makes the reported statistics useless, and also makes any meta-analysis also worthless.

2 Likes