The more I think about RNCTs the more I conclude that statisticians who endorse these need to have their ethics examined.
2 Likes
Maybe this is how events play out (?):
-
Researchers really donât want to risk missing an efficacy signal for a new therapy in a rare disease;
-
They know that the most reliable way to assess whether one therapy is superior to another is by conducting an RCT i.e., randomizing some patients to the new drug and other patients to the standard of care, then comparing their clinical trajectories;
-
But researchers also know that a small RCT will be less capable of detecting a new therapyâs superiority over standard of care than a large RCT. Since few people will remain alive at each time point after randomization (due to the small number of trial participants), between-arm differences in survival rates at any given time point will be hard to interpret;
-
Therefore, they worry that the small size of their trial could cause them to miss the signal for superiority that they are seeking from the new drug and which might actually be present;
-
To address this fear, they propose a backup plan- an alternate way to search for a potential superiority signal for the new therapy, should the within-trial between-arm comparison fail to detect one;
-
They decide, when designing their study, that if the between-arm comparison within their small RCT ends up being non-informative, they will instead look for an efficacy signal by comparing the clinical trajectory of patients treated with the new drug in their current trial with the clinical trajectory of patients treated with standard of care therapy in historical trials. But this type of comparison, with historical controls, is no longer consistent with an âRCTâ design. Rather, itâs a type of observational cohort study (?), with the progress of the two arms potentially being observed in different time periods and using patients who are members of different convenience samples;
-
Researchers settled on the term âRandomized Non-Comparative Trialâ because they couldnât justifiably call their study an âRCTâ if they werenât, realistically, planning to focus their statistical analysis on the randomized between-arm comparison within their single convenience sample. They would plan to âeyeballâ the within-trial between-arm comparison behind the scenes, but then ignore it in their subsequent publication if they didnât see any obvious separation of the trial arms. By randomizing, they would preserve their ability to 1) be happily surprised by unexpected detection of a superiority signal for the new drug; or 2) contribute their trialâs data to a future meta-analysis of RCTs involving the same treatment;
-
The problem: If they donât end up reporting the within-trial between-arm comparison (because they donât see a superiority signal when they eyeball their results) and instead end up focusing their analysis on the comparison with a historical control, any failure to admit, explicitly, that their main treatment comparison has become observational in nature (rather than randomized) will mean that readers might not recognize that they havenât met the standards required for a valid observational treatment comparison. Readers who donât recognize that they are actually reading about an observational treatment comparison (because the word ârandomizedâ is used in the studyâs title) might not register that the authors havenât tried to identify important between-trial, between-arm untreated prognostic distribution differences. These between-trial differences need to be addressed because the patients in the arms being compared will come from two different convenience samples (the current trialâs convenience sample and the historical studyâs convenience sample)- not from a single convenience sample, as would have been the case if the authors had compared the trialâs two randomized arms;
RNCT authors need to take the precautions required for a valid observational treatment comparison- the type of comparison that will usually end up being the main focus of an "R"NCT.
1 Like
Yup. But if RNCTs are as absurd as we think they are then those who dare to compare should gain a meaningful advantage. We have done our part raising awareness. Leading by example is the strategy we are henceforth predominantly adopting to counter RNCTs.
2 Likes
@ESMD I think your layout of the logic that is likely being used by proponents of RNCTs is excellent. At the core of their misunderstandings is mean squared error of treatment effect estimates. Even a small bias from historical controls will render their perceived variance reduction moot, as exemplified here - EHRs and RCTs: Outcome Prediction vs. Optimal Treatment Selection â Statistical Thinking.
1 Like
Compare this with the esotericism in Project Optimus, as discussed in this PubPeer post,
My conjecture now is that FDA OCE ultimately came to view âdose optimizationâ as fundamentally an esoteric practiceâone wholly driven by idiographic considerations, and utterly inaccessible to a nomothetic perspective. Thus, ultimately, Project Optimus degenerated into a syncretistic doctrine that embraced and tolerated contradictions (Eco 1995, Popper 1940).
To reveal the origins of this insight, hereâs a bit of my interaction with an FDA OCE Reviewer 2 of my 2023 PSP paper, How large must a dose-sptimization trial be?:
Apropos of Erinâs âeyeballâ specifically, compare the concept of gestalt randomization I helpfully offer a bit later on in that exchange:
1 Like
Again apropos of Erinâs âeyeballâ, it occurs to me that the isotonic transformation applied in the MERIT design Iâve criticized in another thread could be seen as a way to indulge the illusion so vital to RNCTâs, that this pesky randomness statisticians are always going on about can somehow be swept away without their assistance:
In any case, the upshot of these last 2 posts is that, if we are looking for empirical evidence on the mental substrate for RNCTâs, then Project Optimus provides plenty of material. Indeed, FDA OCE developed and advocated its ârandomized, parallel [yet non-comparative] doseâresponse trialsâ out in the open (on YouTube, in a âwhite paperâ, in JCO and the NEJM), much as Project 2025 did for the current US Administration.
2 Likes