Quality of remdesivir trials

[f2harrell]

Those are good points. But regarding the paper, they are discussing analysis of outcomes that are already ordinal.

[Pavlos_Msaouel]

I agree but the counterargument from their side is that these PRO scores are sums of ordinal scales that can be treated as interval. An underlying assumption here is that the ordinal vs interval debate in this situation is trivial. I think that treating PRO scores as interval causes more trouble than it is worth but those who advocate for linear models would beg to differ.

[f2harrell]

It would be nice to see some of the distributions, to check for absence of floor effects, ceiling effects, bimodality, heavy tails, etc. On the other hand why take a risk by still using parametric methods when the semiparametric methods are so powerful?

[pmbrown]

in the jama paper you tweeted (hydroxychloroquine RCT) it says they used simulations for power. Have you used simulations to see how floor effects, bimodality, non-PO etc affect power? And why simulations, ie is it not possible to just use these equations?: Size and power estimation for the Wilcoxon–Mann–Whitney test for ordered categorical data

[f2harrell]

I think a better place to seek that is one of the papers that studied what goes wrong when you use linear models to analyze ordinal data.

[R_cubed]

Blockquote
I agree but the counterargument from their side is that these PRO scores are sums of ordinal scales that can be treated as interval.

This is a common assumption in psychometrics, but I’ve never been able to find any rigorous mathematical argument in favor of it.

On the contrary, much like the literature on “significance” testing vs. p values as surprise measures, there is a debate within social science on whether it is permissible to treat ordinal data as interval.

Representational measurement theory would advise analysis that takes into account only the order properties when spacing cannot be verified as equal at all points of the scale.

Joel Mitchell (a mathematical psychologist) as written numerous papers on this issue and considers psychometrics a “pathological science” because of it.

Michell J. Normal Science, Pathological Science and Psychometrics. Theory & Psychology. 2000;10(5):639-667. doi:10.1177/0959354300105004

The consequences of treating these ordinal scales as interval has been explored in this paper:

Article: Robustness of statistical methods when measure is affected by ceiling and/or floor effect

I was following up on citations to a paper @f2harrell cited in a number of threads supporting his preference for ordinal methods in applied problems: The following PLOS paper approaches the problem from a formal measurement theory perspective. I’m still digesting the paper, but their reference section is an excellent historical review of the simulation studies and the longstanding argument that compares the performance of textbook parametric and nonparametric methods on ordinal data. Th…

After reading this, I’ve come to the unfortunate conclusion that parametric analysis of PRO measures at the individual study level makes any synthesis or meta-analysis (without access to individual data) unreliable.

If researchers would simply use ordinal methods at the individual study level, then meta-analysis could be a useful tool. The methods recommended by @f2harrell seem correct if the goal is to learn at both the individual level, and to aggregate studies via meta-analysis.

[davidcnorrismd]

@Pavlos_Msaouel

the preference for linear (or even logistic) models by all stakeholders (including statisticians) is that they are less familiar/comfortable with ordinal models,

It sounds as if Medicine’s mantra of “continuous learning” needs to be brought somehow to Statistics.

[llynn]

Latest

https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(22)00519-0/fulltext?hss_channel=tw-27013292#seccestitle160

The discussion is quite interesting.

[llynn]

https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(22)00789-9/fulltext

[pmbrown]

they dont know what ‘multivariate’ means. Also, wondering why they would do meta-analysis, i hope that doesnt become some new standard

[f2harrell]

Individual patient meta-analysis may have a role but I’d rather label it is “efficient analysis of studies with available individual level data, accounting for study heterogeneity”.

[arthur_albuquerque]

@f2harrell

And what is so hard to interpret about an ordinal outcome? SImplifying to 5 outcomes levels (at home with no shortness of breath, at home with significant shortness of breath, hospitalized, invasive ventilation, death), the interpretation of the trial can be stated these ways simultaneously:

• The estimated probability that a randomly chosen patient given treatment B has a better clinical outcome than a randomly chosen patient on treatment A is 0.7
• The estimated probabilities for treatments A and B of the patient having significant shortness of breath or worse are x.xx and x.xx
• The probabilities of the patient needing hospitalization or worse are x.xx an x.xx
• The probabilities of needing invasive ventilation or dying are x.xx and x.xx
• The probabilities of death are x.xx and x.xx
• The Bayesian posterior probability that treatment B affects mortality differently than it affects shortness of breath, hospitalization, or need for a ventilator is x.xx

These indeed are very interesting outputs. However, I haven’t found any recent RCT reporting such statements. Would you please have any reference?

[f2harrell]

You see several trials reporting proportions of Y >=y but not as estimated from the model. Yet.

[arthur_albuquerque]

Fingers crossed!

Referring back to this:

@f2harrell

The estimated probability that a randomly chosen patient given treatment B has a better clinical outcome than a randomly chosen patient on treatment A is 0.7

How relatable is it to Agresti’s Generalized Odds Ratio (GOR)?

Your quote sounded similar (not equal) to this explanation about GOR in another paper:

I haven’t seen you talking about GOR. Looking forward to know your opinion about it. Thanks!

[R_cubed]

The GenOR is closely related to the Mann-Whitney-Wilcoxon statistic, which is a special case of the proportional odds.

I started a different thread that discusses why proportional odds are preferable to metric models for ordinal data, despite decades of debate to the contrary.

Preprint: Analysis of Likert Type Data Using Metric Methods

Dennis Boos and Judy Chen (2022). Analysis of Likert-Type Data Using Metric Methods. I edited this to bring the main points earlier, and the decision theoretic justification later in the post. Introduction The authors defend the common practice of using parametric models on ordinal data on the basis of pragmatics (ie.“easier” to for researchers understand), and perceived robustness in that parametric methods on empirical or simulated data sets are not sufficiently inaccurate that skepti…

[arthur_albuquerque]

Great resource, thanks!

Of note, the authors from the paper I mentioned before cite this another article, where they describe this concept:

“The proposed Wilcoxon-Mann-Whitney generalised odds ratio (WMW GenOR) follows the same logic as Agresti’s GenOR but does not ignore the ties; tied observations are split evenly between better and worse outcomes”

[f2harrell]

You can easily convert the regular PO model odds ratio to the concordance probability (that counts ties as 1/2) as detailed here. I hope that someone can figure out how that relates to GenOR or Agresti’s.

[arthur_albuquerque]

@f2harrell

• The Bayesian posterior probability that treatment B affects mortality differently than it affects shortness of breath, hospitalization, or need for a ventilator is x.xx

Were you referring to a Constrained Partial Proportional Odds model in this topic?

If so, would that be the posterior probability of \tau being greater/lower than 0 (whichever probability is greater) in this model?

[f2harrell]

Yes; I just wouldn’t say “whichever probability is greater” but would pre-specify the direct of interest. Or use a non-null assertion as done in the example, e.g., compute the probability that the treatment effects mortality differently by an “interesting” amount, e.g. \Pr(|\tau| > 0.2). Instead of 0.2 choose the log of the OR you care about.

[arthur_albuquerque]

This is such an interesting model. Many possible estimands. I think I finally wrapped my mind around the model.

Thanks again!