I have a question that has been an issue for me in conducting prospective analyses of risk prediction tools and accounting for the effect of intervention. Let’s say you have created an instrument (e.g. nomogram, checklist, actuarial instrument) from retrospective data and you want to see how accurate the instrument is when used in the field by professionals. The problem, of course, is that professionals act on the results of the risk instrument and may apply intervention/treatment to those who are deemed to be at higher risk of the outcome. The intervention/treatment could obviously change the probability of the outcome occurring and therefore affect the resulting accuracy of your instrument.
In an ideal world, you would run some sort of RCT comparing TAU or no treatment/intervention to a group receiving a standardised intervention on the basis of the instrument. However, in the setting I work in, this is not feasible. In fact, I can’t even really run any sort of comparison group randomised or otherwise. In these situations, is it possible to look at discrimination based statistics like the AUC while accounting for variable levels of intervention at each ‘risk level’ of the instrument? I have seen that AUCs can be calculated with covariates, but the illustrative examples I have looked at (e.g. In Margaret Pepe’s book) do not really cover the situation I am looking at. How would you approach this issue analytically?