Assessing predictive accuracy in prospective studies and the effect of interventions

data analysis models
1

I have a question that has been an issue for me in conducting prospective analyses of risk prediction tools and accounting for the effect of intervention. Let’s say you have created an instrument (e.g. nomogram, checklist, actuarial instrument) from retrospective data and you want to see how accurate the instrument is when used in the field by professionals. The problem, of course, is that professionals act on the results of the risk instrument and may apply intervention/treatment to those who are deemed to be at higher risk of the outcome. The intervention/treatment could obviously change the probability of the outcome occurring and therefore affect the resulting accuracy of your instrument.

In an ideal world, you would run some sort of RCT comparing TAU or no treatment/intervention to a group receiving a standardised intervention on the basis of the instrument. However, in the setting I work in, this is not feasible. In fact, I can’t even really run any sort of comparison group randomised or otherwise. In these situations, is it possible to look at discrimination based statistics like the AUC while accounting for variable levels of intervention at each ‘risk level’ of the instrument? I have seen that AUCs can be calculated with covariates, but the illustrative examples I have looked at (e.g. In Margaret Pepe’s book) do not really cover the situation I am looking at. How would you approach this issue analytically?

Thanks,
B

2

Without an intervention study it is impossible to really know the impact of an estimation tool. But if you are going to take your best shot using the original observational data, you need to state a quantity of real interest, on a one-patient basis, and estimate that. The c-index or other measures related to ROCs will not be what you need here.

3

Thanks. So, let’s say I have a quantity of interest related to an intervention (e.g. number of minutes delivering psychotherapy), I have my rating on the estimation tool and I have my outcome (e.g. hospitalisation for psychotic episode as a binary outcome). Is a regression based model the way to go with my intervention as a covariate?

4

This sounds reasonable. If the intervention were to be randomized you would need minimal covariate adjustment. Otherwise you’ll have to go to extra effort to elicit form experts the set of variables they may possibly use in deciding on the number of minutes, and then collect those variables for adjustment.