I am working on the development of a multivariable logistic regression prognostic model for adverse outcomes post intensive care discharge.
Candidate predictors were prespecified from prior literature…Predictor selection is being performed by fitting a group LASSO within each of 10 imputed datasets, using 10-fold cross-validation to select lambda-min, and retaining predictors selected in more than 50% of the imputed datasets.
We have gone back and forth with this because certain predictors are modelled as restricted cubic splines (known non-linear relationships). As far as we can tell, the joint multiple-imputation LASSO methods - MI-LASSO (Chen and Wang 2013 ) and galasso and the stacked adaptive elastic net (Du 2022) use their group penalty for grouping each variable’s coefficient across imputed datasets, leaving no room to also group the spline terms… Per-imputation group LASSO with a selection frequency threshold is the only route with appropriate R package we’ve found that keeps the splines grouped while still handling the imputations.
I’d appreciate your advice on if this is a defensible way to combine grouped spline selection with multiple imputation or if you would recommend a different approach. We have chosen to select rather than fit the full model with shrinkage to prioritise usability of the final model at the point of ICU discharge.
My other question was regarding confidence intervals of our optimism corrected estimates after bootstrap validation. We follow predictor selection by fitting an unpenalised model with retained predictors in the imputed data to calculate performance metrics (we plan to account for fitting an unpenalised model by applying a uniform shrinkage factor (= the optimism corrected calibration slope) to apparent model coefficients). We’re performing internal validation with bootstrap optimism correction, reimputing within each resample (B=1000/m=10). We’ll calculate optimism and derive optimism corrected performance measures.
I’ve read your recent bootcal post on ABCLOC but as far as I can tell, that’s for single complete datasets in rms? So for the CI, we were going to use Noma et al’s location-shifted bootstrap approach (shifting the 2.5th/97.5th percentiles of the apparent bootstrap distribution by the estimated optimism). We considered the 2-stage process but it would be too computationally intensive to run predboot in R, given we are working within a secure data environment…Does the location-shifted approach seem reasonable here or would you suggest something else?
A quick follow-up: The location shifted interval degenerates on dummy data specifically for the calibration slope and intercept because the apparent slope (~1) and intercept (~0) are fixed by construction, their bootstrap distributions have minimal spread, so the bias-corrected confidence interval comes out spuriously narrow. C-statistic and Brier intervals look sensible… I wonder if this is something ABCLOC can handle in this setting?
Thanks very much for your time on both these points
using bootstrap selection frequencies for feature selection is just another way to use p-values and other measures of strength of evidence, and it doesn’t handle competing collinearities well. Selection frequencies don’t offer anything new.
the validation strategy does not properly take shrinkage into account and especially does not take uncertainties in amount of shrinkage into account
Please provide the sample size, the distribution of outcomes, and the number of candidate features so that we can check that the following recommendations are indeed good ones. And if the sample size is not very large, it is unlikely that the information content in the data can properly inform you regarding the amount of shrinkage to use.
For most problems of this type I recommend data reduction techniques over feature selection. Nonlinear principal components analysis is one approach. Sparse PCA is a very good approach, though not worked out for nonlinearities to my knowledge.
Since the effective sample size (ESS) is seldom large enough to bring enough certainty into the feature selection process, it is better to live within the confines of the ESS. Data reduction (unsupervised learning) does that.
See this for a case study in nonlinear data reduction with multiple imputation using stacking.
If you still want to do feature selection, first show that bootstrap confidence intervals for relative explained variation are not embarrassingly wide. You can’t reliably select features without estimating feature importance.
I think that my simulations in the ABCLOC article show that Noma et al’s method is not accurate enough. ABCLOC should handle the slope and intercept issue you mentioned.
To clarify our approach: predictor selection used a pragmatic >50% selection-frequency rule — group lasso (λmin) applied within each of the 10 imputed datasets, retaining terms selected in more than half of them. The unpenalised selected model was then refit across the 10 imputations (Rubin-pooled) for the apparent statistics. Then, the entire process — imputation, group-lasso selection, and refit — was repeated inside every bootstrap resample, so the optimism correction reflects the whole modelling pipeline rather than a frozen model.
PCA seems a sensible route — the collinear clusters (Hb/haematocrit and pH/pCO₂/HCO₃/base excess) would be handled naturally. My uncertainty is the nonlinear predictors: am I right that a nonlinear PCA approach (as in the bacteremia study) estimates the transformations within the reduction itself, rather than us carrying pre-specified RCS terms into it? Several of these variables have clearly nonlinear relationships with our outcome (readmission or death within 30 days of ICU discharge), so I’d want that structure preserved.
Finally, on ABCLOC — is it applicable in the multiple-imputation setting? Our bootstrap stores both the bootstrap- and original-sample performance per resample, but pooled across the 10 imputations.
That’s a pretty good sample size. The effective sample size is 3np(1-p) = 2010 with p=0.082.
I question the use of feature selections in almost any context and a little more so in yours. The 50% is very arbitrary and selection that does not combine collinear predictors first (and then use multi-degree of freedom chunk tests) becomes even more arbitrary, as when there are competing predictors the choice is very random.
Yes, nonlinear PCA (e.g., R Hmisctranscan function) finds transformations that make variables maximally relate to each other. If you’re lucky this is similar to how they relate to outcome. A lucky example may be found here.
ABCLOC has only been studied for complete data cases. We need to find a way to extend that to multiple imputation, probably using stacking.
That doesn’t sound like a repetition of the entire process since you are refitting the “selected” model and are not respecting penalization.
Look forward to applying ABCLOC within an imputation framework! For now, we will have to honestly report the limitation of Noma’s CI for the calibration slope and intercept…
I take your broader point on penalisation and on feature selection more generally, the selection frequency threshold is very arbitrary. For the definitive analysis we’ll look seriously at nonlinear data reduction (transcan), mindful of your caveat.
Just to clarify the process point: within each bootstrap resample the whole procedure is re-run from scratch — imputation, group-lasso selection, and refit — so the selected predictors genuinely differ from one resample to the next. We aren’t treating any selected set as fixed - our sole interest is the optimism-corrected performance statistics averaged across B=1000, which this provides.
Mean optimism is derived by averaging optimism across 1000 bootstrap iterations, where optimism is calculated by subtracting test performance from bootstrap performance, where test performance is obtained by applying each bootstrap-derived model to the original dataset, with 10 imputations of the original dataset run using the imputation models trained for that bootstrap sample.
Sounds right. If you do take all sources of uncertainty into account you’ll see wild changes in the list of features selected from one bootstrap to the next, making you ask “why am I doing this?”. And the whole reason for doing this is to validate an example model since with feature selection there is no such thing as the model.
To check the order of the steps:
run 20 multiple imputations to get 20 completed datasets
for each dataset do a full analysis and fresh bootstraps