some of the statements about medical RCTs miss the mark (e.g., the assertion that subgroup analysis is rarely performed…).
I completely agree with you on this. The same error is committed by one of my otherwise favorite statistics books, Herbert Weisberg’s Willful Ignorance (Wiley, 2014). There was a survey done a few years ago by NEJM showing how widespread subgroup analysis has become in the literature. There are a few other nit picks I have with Hubbard et al, which I’ll enumerate elsewhere, but I generally found it to be a good perspective.
There was a survey done a few years ago by NEJM showing how widespread subgroup analysis has become in the literature
Correct. For example we found that among 785 oncology phase 3 RCTs, 379 (48%) reported at least one subgroup analysis. The vast majority of these subgroup analyses (93%) was of low or very low quality. The related commentary here notes that these issues have been common across medicine over decades.
Similarly, we found in another study that the majority of subgroup comparisons using forest plots in studies presented at the American Society of Clinical Oncology Annual Meetings are inconclusive.
Thank you for the comments, @s_doi ! A few specific reactions:
#3 can potentially undermine everything else. My paper takes a very extreme perspective on this, one that few may find to be pragmatic. Even if one is to be less extreme, I think point #3 should be top of mind always. It is easy to let it fall out of view.
#4 bothers me a little. In the case of medical diagnostic tests, there was a temptation by some to take the reported Se/Sp estimates of various COVID diagnostic tests (both the point estimates and UIs) to compare which tests were “better” than others. I contend that unless the estimates were derived from using the same set of patient samples to evaluate all the tests, it is not possible to make such a comparison. Vendor A may have selected an “easy” set of samples relative to Vendor B. Geographic regions of the patients in the two sample sets could be non-comparable. Moreover, the stage of the pandemic (eg, before or after the delta variant came out) in which samples were taken might affect the results. So I don’t see how either the point estimates nor UIs could be used to compare across studies. A reader can generalize this to other examples (eg, vaccine efficacy of the initial COVID-19 vaccines (published in late 2020 and throughout 2021 in NEJM) - it is unfair in my opinion to compare the “worse” results of the Janssen vaccine with those of the Moderna and Pfizer ones, because the latter two were based on trials initiated in the late summer of 2020, while the Janssen trial started in the fall, and included the time period when the Delta variant started circulating).
@f2harrell is correct, my paper focuses on the frequentist perspective. As P. Diaconis and others alluded decades ago, due to the Likelihood Principle, Bayesian and Likelihoodist inference also depend on a prespecified model, and are thus subject to the same critiques if the modeler lets the data influence the model form. I believe Frank’s point is that it seems to be straightforward in Bayes to build in extra model parameters up front to reflect hypothetical effects and sources of uncertainty, thus allowing model uncertainty to be more explicitly represented in the model outputs. The last paragraph of my paper’s Appendix A.1 alludes to this in more detail.
I look forward to reading the Sander Greenland article posted by @R_cubed ! My internet access has been intermittent and will continue to be so for the next week, so it might be a while before I can catch up.
Taken literally, the recommendations in that paper cited by Christopher Tong would require scientists to so severely discount any source of data that he or she did not personally collect, it might as well be treated as having no credibility. That would include meta-analysis of RCT’s, since meta-analyses of RCTs are also merely “observational”.
This seems to be a non-sequitur, unless you’re referring to the potential for undisclosed post-hoc decisions made regarding trial execution and/or data analysis that were not pre-specified in the protocol and statistical analysis plan. This is why regulators prefer the sponsor to file the protocol and SAP for phase III trials in advance, and such pre-registrations are becoming more prevalent among academic studies as well (see the paper by Nosek et al cited in my 2019 paper).
I am in fact a bit skeptical of meta-analyses, but not for the reason you gave. Variations in study design/execution, enrolled patients, standard of care, study quality, data quality, etc., make it challenging in my mind to “combine” the estimates mathematically.
If the frequentists are going to complain about the Bayesian prior, why should the Bayesian meekly accept that a study that claims to use randomization is inherently more credible than any other type of formal model for the data collection procedure, when other claims from that same agent are treated with doubt, if not strong skepticism?
As a former regulator, I had access to sponsor-provided proprietary information to verify to a large extent what the sponsor was actually doing with regard to “randomization”. It would take almost as much effort to fabricate these records as to actually perform the planned randomization plan (which itself would not have been blessed in advance if it were flawed).
I have no reason to doubt your personal experience, but I think there is at least a preponderance of evidence from various data sources I’ve come across to doubt your experience is representative, or regulatory procedures are strong enough to discourage scientific misconduct in important policy relevant circumstances, especially in the context of LLMs. Excessive trust in “random sampling” or “random assignment” that is controlled by an untrusted party just won’t work.
I think this forum would agree that the computer science researchers that deal with topics close to statistics would benefit from a study of statistical fundamentals. Looking at it from the other perspective, statisticians would benefit from understanding fundamental computing and information security concepts, especially examination of trust networks, as well as protocol design and verification.
Hamming, in his book The Art of Probability had this to say about the concept of “randomness”
*Randomness is a negative property…Randomness can never be proved, only a lack of it can be shown.
In the context of secure communication, randomness is used by the defender to guarantee any message (information) passed through a network can only be read by its intended recipient via an encryption algorithm. The defender trusts randomization because they know that the function that maps plaintext to ciphertext relies only on the key, which he or she controls, and the algorithm that uses it.
The clinical trial context can be modelled as a network protocol (or a game as in game theory) where “nature” passes a message to our agent “the regulator” via an untrusted proxy – the experimenter – who has incentives to get the regulator to accept claims of beneficial treatment, regardless of whether the treatment is actually valuable or not.
In the clinical trial context, our communication channel has none of the properties that any communication system engineer would attempt to guarantee as correct by construction. The introductory chapters of Gerard Holtzman’s classic work The Design and Validation of Computer Protocols had this to say:
All freshly designed protocols, no matter how disciplined their designers have been must be treated with suspicion. Every protocol should be considered to be incorrect until the opposite is proven (italics in original).
Rigorous examination of communication protocols, especially those that implement cryptography, assume the following (taken from Bella’s Formal Correctness of Security Protocols):
Each [security] property implicitly assumes the existence of an agent the Spy whose aim is exactly to violate the communication profitably.
Unneeded computing power in input handling code is a hand-out to attackers. Reduce computing power needed for your protocols, reduce parsing exposure to the necessary minimum! The power that is not there cannot be hijacked.
Hard-to-parse protocols require complex parsers. Complex, buggy parsers become weird machines for exploits to run on. Help stop weird machines today: Make your protocol context-free or regular!
Protocols and file formats that are Turing-complete input languages are the worst offenders, because for them, recognizing valid or expected inputs is UNDECIDABLE: no amount of programming or testing will get it right.
My thesis: granting an untrusted party the ability to input random samples into our decision procedure creates a computationally undecidable problem for our trusted regulator. We have shifted the burden of proof from those making a positive claim, to forcing the trusted regulator to proving error or falsehood.
Case study: Fraud and Alzheimer’s research
For Further Study:
This wikipedia entry is a good intro, with lots of links to scholarly articles:
Here is a reasonable discussion of the halting problem and its applications in secure computing:
Agree on all points. There’s much more protection against shenanigans/bad actors/incompetence built into RCTs (many of which are done to support drug approval) than there is for most observational studies (which lack regulatory agency oversight). Of course, this doesn’t mean that big oversight failures never occur in the RCT context, but they are going to be much less common than in the observational realm. Like you, I’m not a big meta-analysis fan, other than those used primarily for qualitative purposes.
Thanks @ChristopherTong , I understand your concern and as you can read from this thread this is certainly a major problem that, as you say, can undermine everything else. However if we combine #2 & #3 then it becomes clear that UIs are just a measure of precision and not accuracy and the latter requires that we remember model assumptions (agreed that nobody seems to want to do so). However, blaming the method for how people misuse it is like blaming a car for reckless driving — the problem isn’t the vehicle, it’s the driver. However there is another problem too. The limits of the vehicle could have been misunderstood by the driver, thus making it seem like reckless driving. The result however is the same - the method is not at fault. So what do we do? The answer is to put the idea into layman terms so that no one can claim it is something that it is not. I will think about how to state this in layman terms and make another post if I think it may be useful.
But Suhail, isn’t the point being made in several of the articles cited in this thread that precision also hinges on a lot of assumptions and that, in the overwhelming majority of cases, at least as far as observational studies are concerned, these assumptions are NEVER going to be justified? So, in this context, maybe the method shouldn’t even be applied, as it generates numbers that are essentially meaningless ?
edited for clarity
No Erin, that’s not the point Chris is making in his paper as well as in other papers. The point being made is that precision can be computed for any estimator so long as we are happy with the probability model but if we are complacent and always believe that the target of estimation is indeed the “truth” in the population then we tend to interpret the UI in that way and therefore make a major error of judgment. In most cases statistical models are not correctly specified, and we misuse UIs by assuming they are. That’s the reason you asked this question at the beginning of the thread so @ChristopherTong is right. However we disagree on the solution to this problem.
Addendum to make things clear
A statistical model tells us how to build an estimator — the mathematical rule or formula we apply to the data. The assumptions (e.g., correct specification, right covariates, functional form) are what make that estimator converge to the true parameter we care about. If these assumptions are wrong, the estimator can systematically miss the target (bias). This is where the concern has been raised about model assumptions.
A probability model tells us how that estimator behaves when applied repeatedly to random realizations of the data. In any one sample we get an estimate (a single numerical value). The probability model, via distributional theory and often the central limit theorem, lets us describe how much that estimate would vary across hypothetical repetitions — i.e., its precision. This is how the UI is generated
I’m in no position to argue with experts on the substance of these matters, given that I’m not one. I’ll let the experts on this forum argue amongst themselves, if they disagree with each other. All I’ll say is that I perceive, as a novice, some pretty profound disagreement on exactly these issues, even if I lack the training to articulate it coherently…
Think of it like using a bathroom scale. The statistical model is like trusting that the scale is well-calibrated: if it’s built correctly, then when you step on it, the number should reflect your true weight. But if the scale is off-balance or miscalibrated, then no matter how many times you step on it, you’ll keep getting a number that is consistently too high or too low — that’s bias.
The probability model is about something different: it tells us how much your reading might jiggle if you step on and off the same scale many times. Even a perfectly working scale won’t give exactly the same number each time — maybe one time it says 70.1 kg, next time 70.0 kg, then 70.2 kg. That jiggle is random variability, and from it we can compute how precise the readings are. This is what the uncertainty interval (UI) is trying to capture.
The problem is when people mix these two up. If the scale is miscalibrated (bad assumptions in the statistical model), then the readings are systematically off. But if we just look at the little jiggle (the precision/UI) and assume the average number must be our “true” weight, we’re fooling ourselves. We’re very precisely measuring the wrong thing.
And that’s the disagreement: some people say the fix is to always distrust the model, others say we can still use it carefully. But the key is: precision (the wiggle we can compute) is not the same as truth (the target we care about).
I understand the difference between accuracy and precision- this is not the issue at hand. Rather, the issue that I’m trying to highlight is that a non-negligible proportion of statisticians seems to put no stock in the bounds OR width of "confidence intervals” generated from observational studies because they consider that the assumptions that underlie generation of such intervals are almost never justifiable. In contrast, my reading of your position is that you don’t agree that indefensible assumptions underlie the generation of these error intervals.
An argument that “some error bars are better than no error bars” would be analogous to arguing that “some data are always better than no data” (e.g., when trying to justify conduct of an observational study in certain clinical contexts where RCTs are considered infeasible). I disagree with this view. Sometimes, accepting that we can’t get a sufficiently reliable answer to a question will cause us to make better decisions for patients.
I came across these nice statistics blogs from a few years ago while noodling around on the Internet:
An excerpt:
“In Statistics courses, we teach how to interpret things like estimated regression coefficients, credible/confidence/compatibility intervals, and p-values. But there is rarely a meta-interpretation layer that honors the disagreements in many of those interpretations, the unsettled foundations of how they should be used in science, and the downstream effects on science and decision making.”
Thanks, as always Suhail, for continuing to engage.
That’s a frequentist view, which is fine. In the Bayesian world we may not have estimators. We may find that the most useful and honest output is a posterior distribution for an individual patient-type risk of a clinical event. A statistical model in general tells us how to connect unknown parameter values to data, and there is not necessarily any hypothetical repetitions.
I want to second that. Many an observational study has resulting in the public having false confidence in a result, and acting on it (remember when bacon used to cause pancreatic cancer?).
The phenomenon of “medical reversals” is a good example.
Medical reversals are caused when a treatment is widely adopted even when there is not compelling evidence for its safety and effectiveness. For example, an intervention may be adopted because it “makes sense”, or because there are observational studies supporting its putative benefits.The negative effects of such reversals include harm to patients who received the intervention when it was considered relatively safe and effective, as well as reducing public trust in medicine.
Hi again Erin. This is my understanding of use of confidence intervals for observational studies. I would be grateful for your views.
If one conducts an observational study to examine the difference in outcome between a group exposed and not exposed to a potential cause, then the method and results should be described in a way that can be repeated consistently. By doing this you could assume that you are creating (or ‘selecting’) in an identical random way elements of two different sets of observations where one contains the potential cause and the other does not. If you continue in this way, you could theoretically create two sets with a near infinite number of elements in each, thus arriving at their ‘true’ means.
At any point during this process you can assume that you have a two ‘samples’ ‘selected randomly’ from the two final populations. From these you can estimate the normal likelihood distribution of the observed data conditional on each the possible true means (or if you prefer the normal probability distribution of the possible eventual true means). You can combine the two distributions to give the likelihood or probability distributions of the differences between the true means. From the latter normal distribution, you can estimate the 95% confidence interval of the likelihood distribution or the 95% credibility interval of a corresponding probability distribution.
Now if the mean of the distribution of differences is zero suggesting no difference, then depending on the width of the distribution (i.e., the size of the Uncertainty Interval in the form of a confidence or credibility interval), you might conclude that there is little evidence so far in support of your hypothesis that your potential cause is a real one. You might then decided to abandon the notion. So this is a useful application of a confidence or credibility interval in an observational study, which is basically exploratory. The chosen Uncertainty Interval is therefore a preliminary assessment to decide whether further work is worthwhile.
If the Uncertainty Interval excludes zero difference, then you have a problem. This means assessing (a) whether your hypothetical cause actually caused the outcome, (b) whether the outcome caused your hypothetical cause or (c) whether something else caused your cause and the outcome. This will involve a complex argument that even a mild sceptic will be disinclined to accept and if wrong could cause harm. The best way to settle the argument is to do a carefully designed RCT and again estimate the distribution of possible differences in the true means.
You can then use the distribution of true means to not only to create some Uncertainty Interval in the form of a confidence or credibility interval or P-value or even the theoretical probability of replication if you repeat the study with similar sample sizes or sample sizes of near infinity. You can also speculate about how the findings in the study population might apply (or ‘be transportable’) to other populations (e.g. of different disease severity, gender, age etc).
That way of stating things perhaps pays insufficient attention to the trustworthiness of the center of the interval when sampling or outcome ascertainment is biased.
