Hello,

I need to assess the effect of a continuous metric obtained from a model on a binary outcome and determine a population-level cut-point(s).

Can anyone kindly guide whether there is a similar approach as ROC in Bayes approach?

Thanks in advance.

I have seen Emax, but not sure if I can interpret it as ROC.

Thanks for your hints and guidance.

Before dealing with a Bayesian approach it’s best to think hard about ROC curves. As has been written about on `datamethods`

and fharrell.com, ROC curves are not consistent with individual patient decision making because every point on the curve conditions on the unknown to calculation the probability of the (already observed!) known. Also, the c-index is not sensitive enough for comparing two models.

The severe problems with the ROC curve culminate in its tendency to beg users to create cutpoints. This is not valid. It is arbitrary and information-losing. It is inconsistent with individual decision making.

Professor Harrell,

Thanks a lot for your response and guidance.

Dr. Vehatri also brought this point to our attention. While I totally agree with your point. However, when I presented my metric which is very strong by itself to predict outcomes, I was asked how I can use this in the clinic if I do not have a range or cut-point.

I rather use a continuous metric and show that can change per person over time than using a cut-point as I also believe the individual-based changes. Perhaps, I should just add this to the discussion of the manuscript.

Thanks again.

See my RMS course notes for a detailed list of what goes wrong with that approach. One the the most severe and subtle problems is that when you seek cutpoints on input variables, the cutpoints must mathematically be functions of all the other input variables. For example if you model is additive and linear so that part of the model is \beta_1 x_1 + \beta_2 x_2, any cutoff for x_1 must be a function of the continuous x_2 value. This is a consequence of seeking a cutpoint at the wrong time in the information flow. A cutpoint can only be logical on \hat{Y} once you know the utilities for the decision to be made. In other words, cutpoints can only be logical for outputs (if even then) never for inputs.

In the Bayes approach, we put a cut-point for a continuous Y hat, usually for an ordinal outcome.

What if I use ROC for a model that just one continuous variable predict a binary outcome such as mortality at different cutpoints of another continuous variable like the age to assess Sensitivity 1-specificity using the ROC approach. The goal is to find out how well the continuous predictor predicts mortality at various ages and report AUC.

Does this approach also have caveats? If yes, so I drop it, if not, can I use a Bayes approach?

Thanks

That approach has many problems. Forget cutpoints and think about the distribution of Y|X, and remember that sens and spec are conditioning on what you are trying to predict to compute the probability of what you already know.

Thanks a lot for all your guidance.