Not exactly spase solution of l1-penalized least-squares

[Clément Lejeune]

I am trying to solve some least-squares problems regularized by convex non-smooth penalties (such as l1-matrix i.e LASSO, or nuclear norm). Although the LASSO solution of a least-squares problem is supposed to be exactly sparse, i.e the solution vector (or matrix in my case) entails components which are exactly zero, I noticed that the LASSO solution returned by the CVXR solver is not exactly sparse. In other words, some components of the solution are very small (~10^-15), but not exactly 0, and the relevant ones much bigger (~10^-1).

I noticed the issue with the generated data used for the tutorial on ElasticNet on CVRX website, where one can see that in the table1 (LASSO-path), the "almost zero components" of the solutions have been rounded to 3 digits.

Is there a way to get an exact sparse solution with CVXR ?

To get an exact sparse solution, I did as in the CVXR ElasticNet tutorial i.e I rounded the components of the returned solution to 2 or 3 digits. However, it is a pretty empirical approach isn't it ? In practice, it is not possible to know whether the true non-zero components of the solution are order 10^1, 10^0,... hence it is hard to threshold the digits of the true zero components of the solution (and not really in the spirit of LASSO-like problems).

NOTE
I know that I could solve the problem outside CVXR, with proximal methods, and the solution would result exactly sparse but for some convex penaIties it is hard for me to compute the proximal operators analytically. I would like to experiment different convex penalties quickly on my problem, and that's why I decided to learn to use CVXR.

Thanks !

[Balasubramanian Narasimhan]

The solvers all deal with doubles and so anything ~1e-15 is effectively zero. So you can set a zero threshold and force the solution components that are this small in absolute value to be zero upon return from CVXR.

But the larger point is that this is solver behavior. Some solvers like glmnet use active sets, where the coefficients for variables not in the active set are directly set to zero. Other solvers, of course, work differently.

The natural follow-up is then: can a solver like glmnet be used with CVXR? Yes, in the experiments we have done. But no promises yet.