I am not sure what Judea Pearl was trying to do by offering this example based on alleles and tasty medication. His and Scott’s paper contains the necessary data to illustrate their argument of how to place bounds on the probabilities of 4 counterfactual situations of (1) survival on control and treatment (2) no survival on control but survival on treatment (‘benefit’), (3) survival on control but not on treatment (‘harm’) and non-survival on control and treatment. I proposed one hypothetical narrative that would make sense of their data from a clinical point of view (see Individual response - #65 by HuwLlewelyn). With imagination, there could be many such appealing narratives but the allele / tasty drug example as proposed by Judea Pearl does not appear to be one of them.
It seems to me that dreaming up appealing illustrative narratives is not the issue. From my viewpoint there are 4 important questions:
- The paper’s use of the word ‘benefit’ and ‘harm’ in a counterfactual situation differs from that used when describing the probabilities of an outcome conditional on control and treatment and designating the treatment beneficial or harmful.
- If we could derive probabilities for the above, how would they be used to make medical decisions by also taking into account probabilities of adverse effects and their various utilities (i.e. effects on well-being)? In other words, what is the purpose of calculating probabilities of these 4 counterfactual situations?
- They are estimating the probabilities of counterfactual situations that are by definition inaccessible for the purposes of verification or calibration, unlike other models of prediction for example.
- In view of (3) are their assumptions and reasoning about using various results from RCTs and observational studies to arrive at inequality probabilities of these counterfactual situations sound, culminating on page 8 of the paper?