LP in statistics: The Dantzig Selector

Lots of statistical procedures are based on an underlying optimization problem. Least squares regression and maximum likelihood estimation are two obvious examples. In a few cases, linear programming is used. Some examples are:

• Least absolute deviation (LAD) regression [1]
• Chebyshev regression [2]
• Quantile regression [3]

Here, \(||v||_1 = \sum_i |v_i| \) and \(||v||_\infty = \max_i |v_i|\). The resulting model is an LP.

Note that if \(\color{darkblue}\delta=0\) we solve a standard OLS (Ordinary Least Squares) problem. The constraint becomes: \(\color{darkblue}X^T(\color{darkblue}y-\color{darkblue}X\color{darkred}\beta)=0\) which corresponds to the so-called normal equations: \(\color{darkblue}X^T \color{darkblue}X\color{darkred}\beta=\color{darkblue}X^T \color{darkblue}y\). This system of linear equations, when solved for \(\color{darkred}\beta\), gives us a least squares solution.

The intuition is here that \(\color{darkblue}\delta\) models the trade-off between a full OLS estimate and a sparser solution.

Note that the more popular Lasso approach creates an unconstrained model using a penalty term:

To understand what the function dantzig.delta in the R package GDSARM is doing, I tried to replicate their example:

> library(GDSARM)> data(dataHamadaWu)> print(dataHamadaWu)
   V1 V2 V3 V4 V5 V6 V7    V8
111-1111-16.05821-1111-1-14.7333-1111-1-1-14.6254111-1-1-115.899511-1-1-11-17.00061-1-1-11-115.7527-1-1-11-1115.6828-1-11-111-16.6079-11-111-115.818101-111-1115.91711-111-11115.86312-1-1-1-1-1-1-14.809> X = dataHamadaWu[,-8]> Y = dataHamadaWu[,8]>#scale and center X and y> scaleX = base::scale(X, center=TRUE, scale =TRUE)> scaleY = base::scale(Y, center=TRUE, scale =FALSE)>> maxDelta =max(abs(t(scaleX)%*%matrix(scaleY, ncol=1)))># Dantzig Selector on 4 equally spaced delta values between 0 and maxDelta> dantzig.delta(scaleX, scaleY, delta = seq(0,maxDelta,length.out=4)) 
                         V1        V2         V3         V4         V5        V6         V7
00.170160910.1534495-0.1283823-0.26955930.078247910.47793020.095655671.752410749563350.010850840.00000000.0000000-0.11024920.000000000.31862010.000000003.50482149912670.000000000.00000000.00000000.00000000.000000000.15931010.000000005.257232248690050.000000000.00000000.00000000.00000000.000000000.00000000.00000000>

When we increase delta, the estimates become sparser. Note that this data set has \(n \gt p\), so not really the original target for this method.

It helps me better understand statistical techniques when I reimplement the underlying optimization problem, rather than just running a black-box function. The GAMS model (listed in the appendix) is not very complex. We have to linearize the absolute values in two places. In the objective and in the constraint. In this case, things are convex, so linearizing the absolute values is not very difficult.

The objective is \(\min_j |\color{darkred}\beta_j|\), which can be modeled as \[\begin{align}\min &\sum_j \color{darkred}b_j\\ & \color{darkred}b_j  \ge -\color{darkred}\beta_j\\&\color{darkred}b_j\ge\color{darkred}\beta_j  \\ & \color{darkred}b_j\ge 0\end{align}\]

The constraints \(||\color{darkred}v||_\infty \le \color{darkblue}\delta\) can be written as: \[ -\color{darkblue}\delta \le\color{darkred}v_j \le \color{darkblue}\delta\]

When I run my GAMS model , I can indeed reproduce the R results:

----158 PARAMETER results  estimations

          delta          V1          V2          V3          V4          V5          V6          V7

OLS                   0.1700.153-0.128-0.2700.0780.4780.096
k1                    0.1700.153-0.128-0.2700.0780.4780.096
k2        1.7520.011-0.1100.319
k3        3.5050.159
k4        5.257

References

1. Linear Programming and LAD Regression, https://yetanothermathprogrammingconsultant.blogspot.com/2017/11/lp-and-lad-regression.html
2. Linear Programming and Chebyshev Regression, https://yetanothermathprogrammingconsultant.blogspot.com/2017/11/lp-and-chebyshev-regression.html
3. Median, quantiles and quantile regression as linear programming problems, https://yetanothermathprogrammingconsultant.blogspot.com/2021/06/median-quantiles-and-quantile.html
4. Emmanuel Candes and Terence Tao, “The Dantzig Selector: Statistical Estimation When p Is Much Larger than n.” The Annals of Statistics, vol. 35, no. 6, 2007, pp. 2313–51.
5. Robert Tibshirani, “Regression Shrinkage and Selection via the Lasso.” Journal of the Royal Statistical Society. Series B (Methodological), vol. 58, no. 1, 1996, pp. 267–88.

Appendix: GAMS Model