Strategies to overcome referral \ workup bias in Diagnosis Prediction Models

data analysis modeling strategy
,
1

I wonder what are some common strategies to overcome referral \ workup bias:

Some patients might not get the full diagnosis because of existing pre-test protocol and the outcome will be labaled as Non-Event (0) even though the underlying outcome exists (1).

2

This is a problem when analyzing data the indirect way using sensitivity and specificity. If you are on the other hand in forward-time predictive mode where you estimate probabilities of disease on the basis of current data, I donā€™t think you need to do anything special with, for example, logistic regression.

3

The current data does not indicate the underlying outcome for some cases and the probabilities will be biased regardless of the performance metrics.

If ā€œDavidā€ has a true outcome 1 and a reported outcome 0 because he didnā€™t do the riskier and more accurate diagnosis we must adjust for it somehow.

4

To within the resolution of available data, you use pre-test patient characteristics to model risk, and the predictors in the model should include indicators of what makes a patient tend to not get the ultimate test. So based on best available evidence youā€™ve accounted for what needs to be accounted for. Unless Iā€™m missing something.

5

If I understand correctly Ben van Calster talked about the same issue in his (very cool) lecture on enemies of reliable prediction models:

(The link directs to the relevant part of the lecture)

1 Like
6

Benā€™s presentation is excellent. I was hoping to hear something about resistance to workup bias that results from having enough representation of patients of a certain type in the complete data. For example if females seldom go the final diagnosis but you have 100 females in your dataset who did, the model might be OK.

7

It is indeed an excelent presentation! I showed it to my team two days ago.
For me the most disturbing points are 8 for diagnosis (this thread) and point 9 for prognosis that leads me to counterfactual predictions.

I sent Ben email asking about the subject and he sent me some links to work done by Joris A H de Groot and a related tutorial in R:

Verification problems in diagnostic accuracy studies: consequences and solutions
 Adjusting for differential-verification bias in diagnostic-accuracy studies: a Bayesian approach
Correcting for partial verification bias in diagnostic accuracy studies: A tutorial using R

I wonder if you have any thoughts about this work.