How to compare a diagnostic test with binary outcome to one with an ordinal outcome

topical areas diagnosis
, ,
1

Hi,

I’m looking to compare two diagnostic methods that are currently in use. The current gold standard method has a binary outcome (positive/negative), while the less established test has an ordinal outcome with 5 levels. These levels are Negative, Unlikely, Indeterminate, Likely, Positive.

I am looking for some best practice methods to compare these. I figured I’ll start simple by collapsing the two highest and two lowest levels into Positive and Negative respectively, while omitting the Indeterminate level. Individual levels or other collapsed levels could also be compared. However, I’m aware of the loss of information here, so was wondering if there are any other methods that could be used here and what to watch out when comparing different (collapsed) levels.

Secondly, the less established test probably has inferior predictive performance compared to the gold standard. I was wondering if anybody might have any tips for establishing inferiority or non-inferiority for diagnostics.

NB. I’m not trying to develop a new prediction model, simply to compare two existing ones.

Many thanks

1 Like
2

This is only a partial solution, but consider the use of rank correlation indexes that do not penalize for ties on diagnostic categories, e.g. Somers’ D_{xy}.

1 Like
3

Great suggestion, I’ll include this, many thanks Frank

1 Like
5

Sensitivity and specificity and resulting ROC curves do not use proper conditioning (e.g. computes things like P(X | y > c) instead of P(y | X=c)) and are highly indirect ways to answer the question. In addition they only work for a binary Y, not for ordinal Y.

1 Like