Individual response

data analysis
, ,
225

I’m sure there will be some who can follow the arguments above- unfortunately, I’m not among them. It would be very helpful if the authors of the paper in question could propose, in a simple, narrative way (no math, no symbols), how they would take a drug that showed promise in animal studies through the regulatory process to approval. I would like to see estimates of how many patients would be recruited to clinical trials, where patients would come from who might be included in observational studies, and when each type of study would occur relative to regulatory approval. All other arguments are moot if proposals for how such studies would be (ethically) conducted are based in a fantastic conceptualization of drug regulatory processes.

On another note, possibly of interest to you:

From post 104 in this thread:

“In any set of patients, with binary treatment and outcome, there are four types of patients: never-recoverer, benefiter, harmed-by-treatment, and always-recoverer.”

I wonder if this proposed approach is derived from the econometrics literature (?) See section 6.1 of this paper by Guido Imbens. It seems like this idea of “4 latent groups” might have originated in econometrics in the context of describing potential reactions of of young men to the Vietnam war draft:

https://onlinelibrary.wiley.com/doi/10.3982/ECTA21204?af=R

"Although we initially worked within that traditional latent index framework, our then- colleague at Harvard, Gary Chamberlain, suggested that it would improve transparency to remove what he called “the somewhat mysterious variable νi,” and to use a potential outcome notation not just for the outcomes, but also for the decision to serve in the military. Here, the pair of potential treatment values,

Wi(0)􏰀Wi(1) 􏰀

denotes whether a particular individual would serve if draft-eligible (the potential out- come Wi (1) ∈ {0􏰀 1}), and whether they would serve if not draft-eligible (the potential outcome Wi (0) ∈ {0􏰀 1}). This notation greatly clarified our argument and made clear that there are, in principle, four different types of individuals, as presented in Table I.14 There are never-takers, who do not serve irrespective of their draft-eligibility status, always- takers who serve irrespective of their draft-eligibility status, compliers, who only serve if draft eligible, and defiers, who only serve if not draft-eligible."

The authors seem to be trying to extrapolate the “four latent group” econometrics concept to human biologic responses to drug treatments. The extrapolation fails in this context, but this fact might not be obvious to those trained in computer science rather than biologic science…

2 Likes
226

The 4 part individual effect of an intervention proposed by the causal inference community is based on counterfactuals. For example, if a group of 10 people are treated and 6 survive and then we go back in time and don’t treat, 4 survive. However 2 individuals would have survived with or without treatment (always survivors), 4 would have survived with treatment but not without (benefited), 2 would have survived without treatment but not with treatment (harmed) and 2 would not have survived with or without treatment (never survivors).

In order to discover what happened to each individual above we would need a Time Machine to treat, go back in time and not treat and then compare what happened to each individual. However Pearl & Muller calculated the above proportions (but not what happened to each individual) using various inequalities from a combination of RCTs and observational studies.

There is also a question of stochastic processes. In the messy real world if the above counterfactual study was repeated a few days later the above 2 individuals ‘harmed’ in the first study might appear in the benefit group the second time and 2 of those in the ‘benefit’ group in the first study might appear in the ‘harm’ group during the second study. The overall proportions of 6/10 and 4/10 would stay the same suggesting that the treatment was beneficial on the whole. Individuals from all 4 groups would probably jump around leaving the overall proportions the same.

The problem is that Pearl and Muller don’t explain how knowing the above 4 proportions changes the decision of how to advise an individual when making a decision about whether to accept or decline a treatment. @Stephen and @phildawid have written a paper recently explaining why "the approach is dangerously misguided and should not be used in practice” https://arxiv.org/pdf/2301.11976.pdf. I agree that the 4 proportions are of theoretical interest only and have no place in practical decisions including those made using established decision theory.

In my latest post 220 Individual response - #224 by HuwLlewelyn I suggest that the 4 proportions (for what they are worth) can be arrived at by using traditional diagnostic reasoning from RCT results alone using covariants (e.g. those that represent disease severity or other information such as genetic markers). Observational studies are not necessary. Also the 4 proportions provide less information than that of diagnostic reasoning as explained in my ‘P Maps’.

The only way that I can envision individuals really being harmed and also benefiting from a single treatment are via two different causal mechanisms. For example a drug might benefit by killing cancer cells but harm by killing bone marrow cells. You would then have 2x4 theoretical proportions, 4 for each of the 2 causal mechanisms for what they are worth.

5 Likes
227

Thank you for this terrific narrative explanation; all your hard work to arrive at this point is greatly appreciated. But I can’t help but think that it shouldn’t have required such slogging by any reader to arrive at this elegant summary. Also, thanks for the link to the arxiv publication- mathematically-inclined readers will surely appreciate its insights.

3 Likes
228

How would we put that to the test? Here in this link the RCT find no effect on the rate of postoperative complications using perioperative pulse oximetry yet it is the standard of care. Why?

First relying solely on RCT data, without “observational evidence”, whatever that is, pulse oximetry might be abandoned. Indeed considered from a utilitarian perspective perioperative pulse oximetry may be a waste of money. .However from the perspective of "individual response’ pulse oximetry is viewed as pivotal by clinicians. If there is individual benefit from pulse oximetry, how does individual harm from of pulse oximetry balance that out to render no net benefit for the group under test?

Arguably an RCT is too simple to allow discovery of the individual response which renders the extant theory of the balancing harm and benefit from perioperative pulse oximetry. For this reason clinicians may largely ignore the results of the RCT when they are applied to in the study of treatment or testing applied to a largely unmeasured bucket of highly complex, heterogeneous pathophysiologic conditions.

So RCT have limits which are more definable when observational studies are contemporaneously or previously applied.

1 Like
229

I am not sure this logic follows. A well conducted observational study may give more information than a poorly conducted RCT but clearly a well conducted RCT is better. These RCTs seem poorly conducted to me as both arms had the intervention and the fact that oximetry is “limited” in the control arm seems unethical to say the least. As Huw said previously, we do not need a RCT to compare outcomes between jumpers with intact parachutes and parachutes with a hole in them so the research question needs to be ethical and well chosen

230

The observation study described by Scott and Pearl involved not giving treatment to those in one (refusnik) group (they had mild disease perhaps) and giving it to those in another compliant group (perhaps they had severe disease). Therefore, a comparison of treatment against no treatment in both groups was not possible in the observational study. However, they ASSUMED that the proportion surviving in the ‘mild’ refusnik no treatment observation group was the same as in the RCT (implying that they could have discovered it from the RCT). They also assumed that the proportion dying in the treated compliant ‘severe’ group was the same as in the RCT.

In practice, an observational study and RCT should be done using the mild/severe split to check that the above proportions are the same and if not to question the fact (e.g. Was the treatment given properly in the RCT but not in the observational study?). The observational study could also contain many more subjects than the RCT and might detect rare adverse effects that were not detected in the smaller sized RCT. Even so, there could be an objection that the adverse effects could have occurred equally often in a large control group that was of course absent in the observational study.

Regarding the oximeter example, perhaps they should have by way of analogy included patients at high and low risk of hypoxaemia (or better still a range of risks from very low to very high) in their RCTs. They might then be able to tell which patients with different levels of risk (if any) would benefit.

231

Why? Pulse oximetry clearly can be harmful. Without an RCT how do we know whether the ATE is positive or negative?. So why is it unethical. Furthermore the studies were done so the instant discussion relates to the science and math of the studies.

Pulse oximeters are not a simple parachutes but more importantly the many pathophysiologies of unexpected death and their evolution in the hospital is not comparable to death caused by precipitous deceleration after an inadequately impeded force of gravity…

Interesting idea but it is actually the timing of the hypoxemia in relation to the death pattern not its occurrence which present theory suggests determines the benefit or risk of pulse oximetry and that cannot be known prior to the RCT.

I provide this example to show that, when engaging complex and dynamic questions in the setting of highly heterogenous pathophysiology the RCT cannot provide the “why” which is often needed. to understand results. Why didn’t the RCT show a benefit or harm? We are left speculating and worse, if it does not give us the result we were sure it would we default to arguing the RCT were poorly done an ignore the results. But here, if we believe that pulse oximeter provide benefit then the result of the RCTs suggest that it causes harm which balanced out the benefit. So there is important information derived from the RCT, you just have to put the bias aside and trust the results and then explore the reasons why pulse oximetry might be harmful. .

232

I agree. The ‘why’ in the form of a possible subsequent explanation for an outcome with and without intervention should have been part of the original hypothesis being tested by that RCT. We should try to reason why and under what circumstances a pulse oximter should help by reducing the frequency of some unwanted outcome and design a RCT to test this hypothesis.

1 Like
233

Yes. It’s important not to indict the RCT method because some people, historically, haven’t designed them with enough foresight/care.

Assay sensitivity seems to be the key ingredient missing in the design of many of the RCTs Lawrence is concerned about. “Post-operative status” isn’t a disease, so maybe it’s not a great inclusion criterion for an RCT (in contrast to e.g,. acute occlusion MI or gallstone pancreatitis).

Randomly assigning an experimental intervention to subjects who are in a “physiologic state” that can be arrived at by many different biologic pathways, without a deep understanding of the prognostic distribution of untreated subjects, isn’t an ideal approach. If we could turn back the clock to a time before perioperative pulse oximetry became routine, we could maybe imagine a better way to design RCTs to reveal its benefits. For example, a reasonable first step might have been careful analysis of cases involving patients who died suddenly in the perioperative period while not being monitored. Attention to cause of death and measures that might plausibly have averted a bad outcome (e.g., a pulse oximeter alarming), might have identified a patient subset that is more likely to benefit from oximetry. After that, an RCT aiming to corroborate a benefit for oximetry could have been enriched with higher risk patients. Observing more adverse outcomes might have allowed any intrinsic benefit of oximetry to be detected more efficiently. For example, maybe a trial that enrolled only post-op patients with COPD or neuromuscular disease could show a benefit, whereas an RCT involving “all-comers” in the post-op period, would not.

The above process sounds logical enough. But once a medical practice has become firmly established, there will be many who argue that clinical equipoise has been lost. This is especially true if the intervention is cheap and doesn’t use a lot of resources, where downsides to empiric intervention, even without RCT “proof” of benefit, are hard to fathom, and where the potential consequences of not intervening are serious. The second to last point is the real nub of the issue. Often, some stakeholders perceive potential downsides to an intervention, where other stakeholders don’t (see the endless debate re mask mandates during the pandemic). In the case of pulse oximetry, every anesthetist probably can recall a few cases where a pulse oximeter was the first indicator of a patient’s unanticipated abrupt postoperative decompensation- those types of cases probably stick with a person for a very long time…

Finally, it seems important not to start seeing the potential for qualitative interactions everywhere we look. While their presence might be more plausible in a poorly-designed RCT that has lumped a pile of patients together who have no business being part of the same experiment, a well-designed RCT, focusing on patients with more homogeneous disease (e.g., acute occlusion MI) would probably be much less likely to involve important treatment by patient qualitative interactions.

Arguing that EVERY ostensibly neutral RCT plausibly might be “hiding” signals of efficacy that have simply been “obscured” by qualitative interactions, assumes that EVERY treatment we can imagine plausibly has the potential to benefit some patients and harm others- we just need to keep examining people on a more and more granular level in order to distinguish “responders” from “non-responders.” But of course, this argument is susceptible to infinite regress and isn’t a realistic basis for approving new drugs and devices.

4 Likes
234

Yes, here they were studying a continuous testing device generating a dynamic testing result. Such dynamic testing invariably transition from true negative to false negative to true positive. The period of false negativity poses risk to any subset of patients requiring time sensitive intervention because it induces a false sense of security which may cause a delay in critical, time sensitive, intervention .

Now we see the mix of pathophysiologies renders a mix of patients wherein the period of false negativity is long (harm) and wherein it is short (benefit). So this is the same fundamental problem I discussed previously in relation to RCT of poorly defined synthetic syndromes, like sleep apnea, sepsis, and ARDS.

The key here is that there has been decades long a general misunderstanding of the applicability of RCT in the investigation of testing or intervention in poorly defined populations where the measurement (e.g. “all patients having procedure X”) is not a valid measurement for the treatment or test being studied.

Finally, the dynamic behavior of the individual confusion matrices in specific relation to the range of pathophysiologies under test must be understood. All of these require deep observational research to learn the dynamic relational patterns of the target adverse conditions. The idea that an RCT can routinely replace discovery in complex heterogenous environments is not true. OS are the source of requisite initial discovery.

I bring this to this “individual response” discussion because it shows that the nuanced relationship between individual harm and individual benefit, how these can be routinely hidden within the average and how this can result in wrongful conclusions. Applying the test in an RCT to a select population most likely to benefit might bias the result towards benefit if the test is then applied to a more broad population. because the number of those most likely to be harmed might be diminished.

These fundamental considerations underlie the potential effects of the severity disparities induced by choice. Unless the OS is constructed with an informed and narrow focus, not only might the severity be different in the refusniks, the pathophysiology itself might be different.

235

I cannot see why such a delay would be attributed to oximetry rather than to the monitoring process itself? If we assume that the monitoring process was indeed fully understood by the study investigators then the question that should be asked is why such a study was conceptualized and actually done?

236

Pulse oximetry and the “monitoring process” are the same thing here. The investigators were trying to determine if perioperative pulse oximetry reduced complications which many take for granted (hence the parachute analogy). .They failed to understand how the broad entry criteria (measurement) might effect the heterogeneity of treatment effects (HTE) (as explained below) and probably failed to recognize that pulse oximetry can cause harm. This lack of understanding of the relationship of “individual response” (and particularly HTE) to the entry criteria is ubiquitous.

I bring this to this “individual response” discussion because HTE underlies the type of analysis under discussion here. If the entry criteria (measurements) are broad and include different groups of pathophysiologies (eg anxiety, depression, Sleep apnea, sepsis, ARDS, the perioperative state) then HTE is high and the average treatment effect (ATE) will be biased by the subset mix of the pathophysiologies captured in the instant RCT or OS. This may be a much larger effect than “refusenik bias” of the OS. However, if, in the alternative, the criteria were narrowly chosen with a measurement which reliably captures the target pathophysiology so the relevant (target) pathophysiology is generally present in the study population (eg a throat culture positive for group A strep in the investigation of the efficacy of a new antibiotic) then refusenik bias would rise as a more relevant issue…

HTE is a function of the entry criteria (measurement). HTE as it relates to the mix of captured groupings of pathophysiologies can have a similar effect on OS so they key here in some settings is to use the OS as exploratory to find the measurements which identify the target population so the RCT can be narrowed with a reliable measurement (eg a biomarker or other mathematical tool)…

Oncologic RCT have evolved over the past 2 decades to reduce the HTE by narrowing (and rendering more homogeneous) the population under test relevant the target pathophysiology. Many other disciplines have failed to make these moves and continue to produce useless RCT cargo cult ruminations (pathological science of Langmuir). .

My point is that HTE as a function of broad entry criteria induces the potential for high bidirectional "individual response’ and marked individual response heterogeneity which may dominate and render refusenik bias moot in many fields. .

237

So lets actually look at a couple of potential “individual responses” to continuous Pulse Oximetry monitoring. **(By the way knowing these things may help you next time the nurse discounts your concern about a loved one’s shortness of breath by pointing out that your loved one’s pulse oximetry readings are satisfactory).

Pulse Oximetry Harm
This is a common harm by classic false sense of security caused by reliance on the pulse oximetry signal as indicative of respiratory stability, since the patient progressively increases ventilation in the Type I pattern moving more air (and therefore more oxygen) into the lungs so the oxygen saturation by pulse oximetry (SpO2) may remain normal till near death.

Here I present a typical Type I Pattern of Unexpected Hospital Death (PUHD). This is a typical relational time series pattern of signals in sepsis, congestive heart failure and other common conditions. Note the SPO2 (oxygen saturation by pulse oximetry actually can rise early in the death process and then can remain stable and normal till close to death.) (This is caused by compensatory response of increasing ventilation volume (Ve) and respiratory rate (RR).

Note the potentially fatal false sense of security provided by the pulse oximeter. Of course, death may not occur, rather there may be complications such as organ injury or prolonged hospital stay due to late detection.

image
image799×641 77.2 KB

Pulse Oximetry Benefit
In stark contrast (but studied in the same black box RCT) here is a Type II PUHD. Note here pulse oximetry will provide benefit because the SPO2 falls early in the death pattern providing early warning potentially preventing death or complications. (Particularly if the patient is not receiving supplemental oxygen). If the SPO2 falls early

image
image801×485 40.7 KB

Now the details are different from the @scott example cited by @Stephen but the fundamentals are the same.

There is individual benefit, individual harm, and potential for refusal (which refusal may benefit or harm). The harm or benefit cannot be predicted prior to the study without more knowledge (which might be provided by a prior high density data OS ). The authors are probably unaware that the individual pathophysiology here affects the probability of harm or benefit. The 2 RCT could be done at different centers or on different wards with different pathophysiology sets…

You see how the trialists and statisticians were certain of the validity as long as N was sufficient and compliance was good enough… RCT in black box format (i.e. Does X cause benefit or harm when applied to population Y, as defined by broad and/or capricious, criteria Z?)

Here we see the contrast and consequence of this during the pandemic.

Does dexamethasone improve survival when given to patients with ARDS?
RCT answer-No

Does dexamethasone improve survival when given to patients with ARDS due to COVID?
RCT answer-Yes

How do you reconcile that without deciding “RCT for ARDS” is a pitfall? The answer is the same as for pulse oximetry. Combining the pathophysiologies by using “the perioperative state” as criteria rendered a SET with a mix of individual responses which averaged out to show no harm or benefit for both the pulse oximetry RCT just as occurred with the pathophysiologic mix under ARDS RCT test, as derived from the threshold based nonspecific criteria of the many ARDS RCT. The mix of individual responses in the SET under test determined the average treatment effect of the study. However that ATE may be markedly different then the average treatment effect of another population defined by exactly the same broad criteria. . .

So “individual response”, the focus of this thread, is not just a function of the individual but also a function of the specific pathophysiology affecting the individual, but more importantly it is a function of the percent mix of individual in the SET under test without the fundamental target pathophysiology but which meet the criteria for entry into the RCT.

Finding means to narrow the criteria for the population under test to those with the target pathophysiology is the first step, before deciding if a valid RCT can be reliably performed…

Here, the synergies between OS are RCT are clear. OS, may provide an incorrect answer due to lack of randomization or suitable controls. Yet OS should be performed in complex populations because they provide much greater density of time data, for example time series matrix data from EMR, to render transparent potential individual responses. This was shown in the instant example wherein, before the RCT can be reliably performed, one must learn the lessons of the pathophysiologic basis for pulse oximetry benefit or harm.

During the COVID pandemic most trusting in evidence based medicine were absolutely sure they had RCT evidence that corticosteroids did not work for COVID ARDS and were highly critical of its empiric use. (Given that death was often due to an overwhelming inflammatory response, empiric dexamethasone would have made pathophysiologic sense if the RCT did not exist). The number of lives lost due to that false EBM was likely quite high. We could not have known that, or what to do, but, like the pulse oximetry in Type I pattern of unexpected hospital death, we had a false sense of security that dexamethasone would NOT work. Perhaps clinicians would have been less sure, if everyone was a little more forthcoming about the potential weakness of RCT when applied with broad criteria. This is what failure to consider the heterogeneity of “individual response” as noted by @scott can cause false EBM and harm to the public.

We love RCT based EBM, its our base. Yet lack of reform is robbing EBM of its standing. This article shows the negative drift of the image of EBM, which could be prevented by converting into objective terms the qualities required for entry criteria.

I know I am off track from what the group wants to talk about in this thread and its a great thread and I do not want it to be ended. I will cease so you can get back to it. Regards.

Ref.

238

I can see how important patient/treatment qualitative interactions could be missed as a result of poor RCT design (e.g., inappropriate “lumping” of patients in disparate clinical states into a single trial). Failure to do adequate preparatory study to optimize disease definition, trial inclusion criteria, and measurement tools would be analogous to a drug company skipping preclinical or early phase clinical studies and jumping to phase III- the chance of success would be very low (see below).

I’m not sure whether this problem (which seems much more prevalent in certain medical specialties than others) could be described as suboptimal “construct validity”(?) Whatever it’s called, we’ve discussed how it could lead to noisy trial results, with a net benefit in some subgroups plausibly being obscured by net harm experienced by other subgroups (yielding an overall neutral trial result). Having said this though, I suspect that poor construct validity probably isn’t the “rate-limiting” step in the effort to discover efficacious new therapies in most disease areas. The fact is that it’s really hard to discover new treatments, even for stakeholders with every possible resource at their disposal- the pharmaceutical industry:

“Drug discovery and development is a long, costly, and high-risk process that takes over 10–15 years with an average cost of over $1–2 billion for each new drug to be approved for clinical use1. For any pharmaceutical company or academic institution, it is a big achievement to advance a drug candidate to phase I clinical trial after drug candidates are rigorously optimized at preclinical stage. However, nine out of ten drug candidates after they have entered clinical studies would fail during phase I, II, III clinical trials and drug approval2,3. It is also worth noting that the 90% failure rate is for the drug candidates that are already advanced to phase I clinical trial, which does not include the drug candidates in the preclinical stages. If drug candidates in the preclinical stage are also counted, the failure rate of drug discovery/development is even higher than 90%.”

Everybody knows how rigorous the drug development process is. Pharmaceutical companies expend colossal effort trying to optimize drug dose and trial inclusion criteria, in order to tease out the intrinsic efficacy of a new molecule, if it’s present. Since financial stakes are very high, every effort is made to minimize “noise” in trial results that could obscure an efficacy signal. And yet, even these maximally-financially-incentivized stakeholders have abysmal success rates for bringing new drugs to market. So viewing the situation in this light, maybe it’s not so surprising that researchers who are NOT affiliated with pharmaceutical companies (and therefore have fewer resources at their disposal) and who are often testing complex, nonspecific interventions (e.g., “sepsis bundles”, perioperative pulse oximetry) for heterogeneous/poorly-defined conditions, rather than intensively-targeted new molecules directed at highly-specific biologic pathways for homogeneous/well-defined conditions, rarely meet with success…

Discovering efficacious new treatments is very hard in medicine, across the board, even under “optimal” testing conditions. Since success is infrequent even in the noise-minimizing conditions created by pharmaceutical companies testing new molecules, should we really be surprised that success rates are near zero in fields where noise is rampant? “Insanity is doing the same thing over and over” and all that…