# Choosing penalty parameters for elastic net

**URL:** <https://discourse.datamethods.org/t/choosing-penalty-parameters-for-elastic-net/922>\
**Category:** variable selection\
**Tags:** penalization, prediction\
**Created:** [November 11, 2018, 1:57pm UTC](https://discourse.datamethods.org/t/choosing-penalty-parameters-for-elastic-net/922 "2018-11-11T13:57:02Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![bobmcpop](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/bobmcpop/32/797_2.png) [@bobmcpop](https://discourse.datamethods.org/u/bobmcpop)\
**Post date:** [November 11, 2018, 1:57pm UTC](https://discourse.datamethods.org/t/choosing-penalty-parameters-for-elastic-net/922/1 "2018-11-11T13:57:02Z")

</div>

I’m trying to develop a classification model with \>70 variables and training set of only 500 (case prevalence 10%). I’ve used cross-validation before to choose the penalty (lamba) for lasso, similar to what’s described [here](https://rpubs.com/kaz_yos/alasso). But often there were too few variables selected. It was suggested to me that elastic net is worth trying.

But now there are two tuning parameters [alpha (mixture of lasso/ridge) and lamba]. How do I choose both at the same time, are there any recommended steps?

For the first time (!) @f2harrell 's BBR did not have the answer for me.  
 ![Capture](https://discourse.datamethods.org/uploads/default/original/1X/ca4d8937d8639c5086c4814ae539187099040bef.jpeg)  
Thank you

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [November 11, 2018, 4:09pm UTC](https://discourse.datamethods.org/t/choosing-penalty-parameters-for-elastic-net/922/2 "2018-11-11T16:09:28Z")

</div>

Like _lasso_, _elastic net_ is not used for classification but rather for estimation, e.g., risk estimation.

Your effective sample size is about 50 so the number of candidate features that you can reliably analyze is in the range of 1-10. With \> 70 candidate features the analysis is doomed. For example, if you bootstrap the entire process to check the stability of selected features, you’ll find there is none.

As you’ve found, when it is possible to use elastic net (say if your sample size were much, much larger), it is tricky to simultaneously choose 2 penalty parameters. Many people take the easy shortcut that the ratio of the two penalties is fixed, and the software invites you to do this. I know of no theory that backs up that procedure.

I suggest using severe data reduction (e.g., sparse principal components) followed by ordinary modeling of the 1-5 summary scores coming out of that.

---

<div class="post-metadata">

**Author:** ![bobmcpop](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/bobmcpop/32/797_2.png) [@bobmcpop](https://discourse.datamethods.org/u/bobmcpop)\
**Post date:** [November 11, 2018, 6:54pm UTC](https://discourse.datamethods.org/t/choosing-penalty-parameters-for-elastic-net/922/3 "2018-11-11T18:54:47Z")

</div>

Thank you! I had previously used lasso to predict each individual’s probability of being a “case”, and then classify them using a probability threshold derived from a desired specificity.

> For example, if you bootstrap the entire process to check the stability of selected features, you’ll find there is none.

I came across this exact problem, so instead surrogate-labeled the much larger original dataset (n\>15k, from which the aforementioned 500 was randomly selected) to first reduce the number of variables, before applying them to the 500 gold-standard training set. So two cycles of feature selection (?validity using [this approach](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6080726/)).

> Many people take the easy shortcut that the ratio of the two penalties is fixed, and the software invites you to do this. I know of no theory that backs up that procedure.

The sparsity of literature on this made me question using elastic net too. I was surprised that such a well-established, well-used and frequently taught method had such a glaring “gap” in it’s procedure, and thought that I was missing something.

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [November 11, 2018, 9:32pm UTC](https://discourse.datamethods.org/t/choosing-penalty-parameters-for-elastic-net/922/4 "2018-11-11T21:32:45Z")

</div>

> [@bobmcpop](#):
>
> I had previously used lasso to predict each individual’s probability of being a “case”, and then classify them using a probability threshold derived from a desired specificity.

This is at odds with what the model does. It gives a direct estimate of a probability. Don’t bring sensitivity, specificity, or thresholds into the procedure.

> [@bobmcpop](#):
>
> first reduce the number of variables, before applying them to the 500 gold-standard training set

Such a reduction of the number of variables must use only unsupervised learning. Using Y to drive the variable selection, when this selection is “outside” of the penalized modeling, will result in severe bias.

---

<div class="post-metadata">

**Author:** ![MarkvandeWiel](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/markvandewiel/32/3231_2.png) [@MarkvandeWiel](https://discourse.datamethods.org/u/MarkvandeWiel)\
**Post date:** [September 6, 2022, 9:06am UTC](https://discourse.datamethods.org/t/choosing-penalty-parameters-for-elastic-net/922/5 "2022-09-06T09:06:23Z")

</div>

Late reply, but if you’re still interested: here’s a blog post that explains why indeed it is tricky to tune both hyperparameters of the elastic net simultaneously.

> **[Sparsity can not be estimated (Why tuning the elastic net is hard)](https://markvandewielblog.wordpress.com/2022/09/02/sparsity-can-not-be-estimated-why-tuning-the-elastic-net-is-hard/)**
>
> THIS POST IS MORE TECHNICAL THAN MY PREVIOUS ONES. READ IT WHEN YOU’RE INTERESTED IN REGRESSION WITH PENALTIES OR SHRINKAGE PRIORS. SparsitySparsity is an important concept in high-dimensiona…

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [September 6, 2022, 11:08am UTC](https://discourse.datamethods.org/t/choosing-penalty-parameters-for-elastic-net/922/6 "2022-09-06T11:08:34Z")

</div>

Very nice article. Sorry you would voted down on [stats.stackexchange.com](http://stats.stackexchange.com) for pointing to your own article. I’ve never been voted down for doing that.

A few random thoughts:

- The 3-d plot showing how hard it is to pick a combination of two penalty parameters is the kind of display we should see more often
- There is some recent work showing sample sizes needed to estimate a single penalty parameter (wish I had a reference). It would be worthwhile to estimate the sample size for adequately estimating two penalty parameters.
- A good solution is probably to use ridge regression with Bayesian model projection: [Projection predictive variable selection – A review and recommendations for the practicing statistician](https://avehtari.github.io/modelselection/bodyfat.html)
- I don’t recommend using _lasso_

---

<div class="post-metadata">

**Author:** ![scboone](https://discourse.datamethods.org/letter_avatar_proxy/v4/letter/s/edb3f5/32.png) [@scboone](https://discourse.datamethods.org/u/scboone)\
**Post date:** [September 6, 2022, 3:53pm UTC](https://discourse.datamethods.org/t/choosing-penalty-parameters-for-elastic-net/922/7 "2022-09-06T15:53:11Z")

</div>

Perhaps not quite the one you were looking for, but this one does a nice illustration of sample size on the variability of performance estimates from shrinkage methods too:  
[https://journals.sagepub.com/doi/abs/10.1177/0962280220921415](https://journals.sagepub.com/doi/abs/10.1177/0962280220921415)

Why specifically would you not recommend lasso? Because it penalises estimates to 0 which can lead to arbitrary variable selection?

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [September 6, 2022, 4:58pm UTC](https://discourse.datamethods.org/t/choosing-penalty-parameters-for-elastic-net/922/8 "2022-09-06T16:58:31Z")

</div>

Yes; the variables selected by `lasso` are entirely unreliable unless you have an enormous sample size and the predictors are uncorrelated with each other. More [here](https://hbiostat.org/bbr/hdata.html).

---

<div class="post-metadata">

**Author:** ![MarkvandeWiel](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/markvandewiel/32/3231_2.png) [@MarkvandeWiel](https://discourse.datamethods.org/u/MarkvandeWiel)\
**Post date:** [September 6, 2022, 7:09pm UTC](https://discourse.datamethods.org/t/choosing-penalty-parameters-for-elastic-net/922/9 "2022-09-06T19:09:46Z")

</div>

Thanks for the nice words and suggestion. Indeed (Bayesian) ridge + posterior selection can be a very good alternative. Some use stability selection (by Meinshausen et al.) as a bootstrapping based technique to improve variable selection with lasso.

As for sample size: increasing sample size will of course help in terms of more stable regression coefficients and variable selection. However, increasing sample size does not overcome the identifiability issue of the two penalty parameters. This seems, unfortunately, rather intrinsic.

---

<div class="post-metadata">

**Author:** ![MarkvandeWiel](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/markvandewiel/32/3231_2.png) [@MarkvandeWiel](https://discourse.datamethods.org/u/MarkvandeWiel)\
**Post date:** [September 6, 2022, 7:42pm UTC](https://discourse.datamethods.org/t/choosing-penalty-parameters-for-elastic-net/922/10 "2022-09-06T19:42:39Z")

</div>

And about the down-voting: apparently the rule is that when referring to one’s own blog, one should be very clear about that. That’s what I did now. Also they prefer to have at least part of the argumentation in the reply. Not unreasonable, so I added that as well.