In [1] a non-negative least squares problem is solved using Cplex's matlab interface (function cplexlsqnonneglin [2]).
The model this function tries to solve is:
\[\begin{align}\min\>&||Cx-d||_2^2\\&x\ge 0\end{align}\]
Unfortunately we see:
This looks like a numerical or tolerance issue: least-squares problems should be convex.
I often propose a different model:
\[\begin{align}\min\>&||r||_2^2\\&r=Cx-d\\&x\ge 0, r\text{ free}\end{align}\]
This looks like a bad idea: we add variables \(r\) and extra constraints. However this model has a big advantage: the \(Q\) matrix is much more well-behaved. It is basically a (very well-scaled) diagonal matrix. Using this model we see:
This reformulation is not only useful for Cplex. Some other solvers show the same behavior: the first model is declared to be non-convex while the second model solves just fine. Some other solvers solve both models ok. These solvers add a very tiny value to the diagonal elements of the \(Q\) matrix to make it numerically positive semi-definite [3].
The \(Q\) matrix is formed by \(C^TC\). We can show that \(C^TC\) is always positive semi-definite. This follows from: \(x^T(C^TC)x=(Cx)^T(Cx)\ge 0\). However the inner product \(Q=C^TC\) can accumulate some small floating point errors, causing \(Q\) to be no longer positive semi-definite. When I calculate \(Q=C^TC\) and print its eigenvalues I see:
The presence of negative eigenvalues indicate this calculated matrix \(Q\) is indeed not pos def. Another verification is to see if we can form a Cholesky factorization [4]. This will fail if the matrix is not positive definite:
![]()
This factorization is typically used inside solvers on the \(Q\) matrix. Finally, it is noted that it is also possible that \(Q\) is actually pos def but both tests (eigenvalues and Cholesky decomposition) fail to see this. Of course for all practical purposes that means the matrix is not numerically positive definite.
Model 1
The model this function tries to solve is:
\[\begin{align}\min\>&||Cx-d||_2^2\\&x\ge 0\end{align}\]
Unfortunately we see:
Number of nonzeros in lower triangle of Q = 861
Using Approximate Minimum Degree ordering
Total time for automatic ordering = 0.00 sec. (0.02 ticks)
Summary statistics for factor of Q:
Rows in Factor = 42
Integer space required = 42
Total non-zeros in factor = 903
Total FP ops to factor = 25585
CPLEX Error 5002: objective is not convex.
QP with an indefinite objective can be solved
to local optimality with optimality target 2,
or to global optimality with optimality target 3.
Presolve time = 0.03 sec. (0.06 ticks)
Barrier time = 0.03 sec. (0.06 ticks)
This looks like a numerical or tolerance issue: least-squares problems should be convex.
Model 2
I often propose a different model:
\[\begin{align}\min\>&||r||_2^2\\&r=Cx-d\\&x\ge 0, r\text{ free}\end{align}\]
This looks like a bad idea: we add variables \(r\) and extra constraints. However this model has a big advantage: the \(Q\) matrix is much more well-behaved. It is basically a (very well-scaled) diagonal matrix. Using this model we see:
Tried aggregator 1 time.
QP Presolve eliminated 1 rows and 1 columns.
Reduced QP has 401 rows, 443 columns, and 17201 nonzeros.
Reduced QP objective Q matrix has 401 nonzeros.
Presolve time = 0.02 sec. (1.21 ticks)
Parallel mode: using up to 8 threads for barrier.
Number of nonzeros in lower triangle of A*A' = 80200
Using Approximate Minimum Degree ordering
Total time for automatic ordering = 0.00 sec. (3.57 ticks)
Summary statistics for Cholesky factor:
Threads = 8
Rows in Factor = 401
Integer space required = 401
Total non-zeros in factor = 80601
Total FP ops to factor = 21574201
Itn Primal Obj Dual Obj Prim Inf Upper Inf Dual Inf
03.3391791e-01-3.3391791e-019.70e+030.00e+004.20e+04
19.6533667e+02-3.0509942e+031.21e-120.00e+001.71e-11
26.4361775e+01-3.6729243e+023.08e-130.00e+001.71e-11
32.2399862e+01-6.8231454e+011.14e-130.00e+003.75e-12
46.8012056e+00-2.0011575e+012.45e-130.00e+001.04e-12
53.3548410e+00-1.9547176e+001.18e-130.00e+003.55e-13
61.9866256e+006.0981384e-015.55e-130.00e+001.86e-13
71.4271894e+001.0119284e+002.82e-120.00e+001.15e-13
81.1434804e+001.1081026e+006.93e-120.00e+001.09e-13
91.1163905e+001.1149752e+005.89e-120.00e+001.14e-13
101.1153877e+001.1153509e+002.52e-110.00e+009.71e-14
111.1153611e+001.1153602e+002.10e-110.00e+008.69e-14
121.1153604e+001.1153604e+001.10e-110.00e+008.96e-14
Barrier time = 0.17 sec. (38.31 ticks)
Total time on 8 threads = 0.17 sec. (38.31 ticks)
QP status(1): optimal
Cplex Time: 0.17sec (det. 38.31 ticks)
Optimal solution found.
Objective : 1.115360
This reformulation is not only useful for Cplex. Some other solvers show the same behavior: the first model is declared to be non-convex while the second model solves just fine. Some other solvers solve both models ok. These solvers add a very tiny value to the diagonal elements of the \(Q\) matrix to make it numerically positive semi-definite [3].
A little background in numerics
The \(Q\) matrix is formed by \(C^TC\). We can show that \(C^TC\) is always positive semi-definite. This follows from: \(x^T(C^TC)x=(Cx)^T(Cx)\ge 0\). However the inner product \(Q=C^TC\) can accumulate some small floating point errors, causing \(Q\) to be no longer positive semi-definite. When I calculate \(Q=C^TC\) and print its eigenvalues I see:
[1] 6.651381e+034.436663e+023.117601e+022.344391e+021.377582e+024.842287e+01
[7] 2.471843e+014.970353e+004.283370e+003.878515e+009.313360e-014.146773e-01
[13] 2.711321e-012.437365e-012.270969e-018.583981e-022.784610e-022.047456e-02
[19] 1.670033e-028.465492e-033.948085e-032.352994e-031.726044e-031.287873e-03
[25] 1.095779e-032.341448e-041.776231e-041.266256e-044.064795e-052.980187e-05
[31] 1.056841e-052.409130e-062.039620e-065.597181e-071.367130e-072.376693e-08
[37] 3.535393e-091.355988e-112.996032e-14-1.468230e-13-1.622616e-13-2.217776e-13
The presence of negative eigenvalues indicate this calculated matrix \(Q\) is indeed not pos def. Another verification is to see if we can form a Cholesky factorization [4]. This will fail if the matrix is not positive definite:
This factorization is typically used inside solvers on the \(Q\) matrix. Finally, it is noted that it is also possible that \(Q\) is actually pos def but both tests (eigenvalues and Cholesky decomposition) fail to see this. Of course for all practical purposes that means the matrix is not numerically positive definite.
 |
| André-Louis Cholesky (1875-1918) |
References
- https://stackoverflow.com/questions/48164626/linear-combination-of-curves-to-match-a-single-curve-with-integer-constraints
- Cplex, cplexlsqnonneglin, https://www.ibm.com/support/knowledgecenter/SSSA5P_12.8.0/ilog.odms.cplex.help/refmatlabcplex/html/cplexlsqnonneglin-m.html
- Gurobi, PSDtol, http://www.gurobi.com/documentation/7.5/refman/psdtol.html
- Cholesky decomposition, https://en.wikipedia.org/wiki/Cholesky_decomposition