# Fitting unrestricted model after LASSO to avoid over-shrinkage?

**URL:** <https://discourse.datamethods.org/t/fitting-unrestricted-model-after-lasso-to-avoid-over-shrinkage/365>\
**Category:** modeling strategy\
**Tags:** penalization, variable-selection\
**Created:** [August 9, 2018, 4:38am UTC](https://discourse.datamethods.org/t/fitting-unrestricted-model-after-lasso-to-avoid-over-shrinkage/365 "2018-08-09T04:38:39Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Pavel\_Roshanov](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/pavel_roshanov/32/3309_2.png) [@Pavel\_Roshanov](https://discourse.datamethods.org/u/Pavel_Roshanov)\
**Post date:** [August 9, 2018, 4:38am UTC](https://discourse.datamethods.org/t/fitting-unrestricted-model-after-lasso-to-avoid-over-shrinkage/365/1 "2018-08-09T04:38:39Z")

</div>

Hi everyone - I love the new site!

I’ve been experimenting with LASSO lately to develop a model in a relatively small dataset. I came across something in Friedman, Tibshirani, and Hastie’s 2009 Elements of Statistical Learning book that I thought contradicts my interpretation of Frank’s treatment of shrinkage in his RMS book and wondered about your interpretation:

" **Section 3.8.5 - Further properties of the Lasso**

Regarding the coefficients themselves, the lasso shrinkage causes the estimates of the non-zero coefficients to be biased towards zero, and in general they are not consistent. One approach for reducing this bias is to run the lasso to identify the set of non-zero coefficients, **and then fit an un-restricted linear model to the selected set of features**. "

I recognize there is evidence of over-shrinkage with lasso, but I worry about losing the benefits of coefficient shrinkage with the above suggestion. The benefits seem to outweigh the risk of overshrinkage in most simulations I’ve seen.

Thanks for considering,  
Pavel

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [August 9, 2018, 11:36am UTC](https://discourse.datamethods.org/t/fitting-unrestricted-model-after-lasso-to-avoid-over-shrinkage/365/2 "2018-08-09T11:36:06Z")

</div>

You can only make the statement “fit an un-restrited linear model” if you don’t care whatsoever in demonstrating absolute accuracy of the resulting model though an unbiased high-resolution calibration curve. The sole reason for _lasso_ is to penalize selected variables’ effects for the biased selection process. If we were going to trust unpenalized estimates we never needed the _lasso_ to select variables for us.

---

<div class="post-metadata">

**Author:** ![Pavel\_Roshanov](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/pavel_roshanov/32/3309_2.png) [@Pavel\_Roshanov](https://discourse.datamethods.org/u/Pavel_Roshanov)\
**Post date:** [August 9, 2018, 12:04pm UTC](https://discourse.datamethods.org/t/fitting-unrestricted-model-after-lasso-to-avoid-over-shrinkage/365/3 "2018-08-09T12:04:02Z")

</div>

That was my understanding. Thanks!

---

<div class="post-metadata">

**Author:** ![thomas.stewart](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/thomas.stewart/32/131_2.png) [@thomas.stewart](https://discourse.datamethods.org/u/thomas.stewart)\
**Post date:** [August 9, 2018, 2:39pm UTC](https://discourse.datamethods.org/t/fitting-unrestricted-model-after-lasso-to-avoid-over-shrinkage/365/4 "2018-08-09T14:39:26Z")

</div>

You may want to take a look at the relaxed lasso, a two step procedure with lasso in the first step and a touch of shrinkage in the second step. The procedure described in your post is a special case of the relaxed lasso with the second-step shrinkage set to zero.

Definitely agree with Frank that you will still want to generate an optimism-corrected calibration curve.

---

<div class="post-metadata">

**Author:** ![Pavel\_Roshanov](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/pavel_roshanov/32/3309_2.png) [@Pavel\_Roshanov](https://discourse.datamethods.org/u/Pavel_Roshanov)\
**Post date:** [August 9, 2018, 3:01pm UTC](https://discourse.datamethods.org/t/fitting-unrestricted-model-after-lasso-to-avoid-over-shrinkage/365/5 "2018-08-09T15:01:14Z")

</div>

Unfortunately haven’t seen easy way to do (like Frank’s calibrate function in rms). Can program the bootstrap myself. Maybe I’ll adapt Frank’s calibrate code.

---

<div class="post-metadata">

**Author:** ![f2harrell](https://discourse.datamethods.org/user_avatar/discourse.datamethods.org/f2harrell/32/165_2.png) [@f2harrell](https://discourse.datamethods.org/u/f2harrell)\
**Post date:** [August 9, 2018, 3:04pm UTC](https://discourse.datamethods.org/t/fitting-unrestricted-model-after-lasso-to-avoid-over-shrinkage/365/6 "2018-08-09T15:04:06Z")

</div>

What does the second stage do? I fear it lacks sufficient shrinkage, and begs the question of what the first stage was supposed to do.