Dear all
I am evaluating a surgical risk factor in a small retrospective study (N=20, 6 events). The potential risk factor is a continuous variable measured during the operation. I am fully aware of the limitations of such a small sample size, but I’ve applied several robust methods to address potential bias and instability. I would like to know if these findings hold some clinical/statistical weight, or if they should be dismissed as “nonsense.”
Methods & Results:
-
Univariable Firth’s Penalized Regression: OR 1.259 (95% CI: 1.028–1.759), p=0.02.
-
BCa Bootstrap (1,000 resamples): 95% CI for OR was 1.008–1.810 (does not cross 1.0).
-
Internal Validation of the model: Apparent AUC was 0.7738. Using bootstrap-based optimism correction, the mean optimism was only 0.0016 (Bias-corrected AUC: 0.7722).
The Question: Does the fact that the signal survived both Firth’s penalization and BCa-bootstrap correction provide some compelling evidence for a pilot study? Or is it still statistically non-sensical to draw any conclusions from such a small dataset?
I’d appreciate your critical and candid views.
It is mathematically impossible to assess differences when there are 6 events. That’s because you cannot even estimate risk in your situation if there are no predictors, i.e., you can’t even estimate the intercept in a logistic model were all slopes known to be zero. See this. So this is a futile project, unfortunately.
When doing penalization empirically (i.e., when not specifying a prior distribution to a Bayesian model) the sample size required to choose the right penalty can be quite large. It’s a lose-lose situation unfortunately.
The best you can do is to report a Wilson 0.95 confidence interval for the unknown outcome probability assuming homogeneous risk, which is [0.15, 0.52]. The point estimate of 0.3 is not meaningful. To nail down the average risk to 0.15 - 0.52 means we don’t know very much.
6 Likes
I really appreciate your comment. I understand that the sample size is insufficient to estimate the occurrence of this event. I realize I cannot statistically discuss anything about the event when the average risk is as uncertain as 0.15 - 0.52.
Thank you again for your time and for providing such a clear lesson.
3 Likes
You’re very welcome. I hope you have the opportunity to get a larger sample size. Some sample size calculations are discussed here and here.