I was playing with some large sparse square systems of non-linear equations (actually resulting from some spreadsheets). R has a package called BB [1] which is based on [2,3]. Technically: this is Barzilai-Borwein spectral method. It is both derivative-free and matrix-free (only needs a few vectors of length \(n\)).
For some problems it works rather nicely:
This solves very quickly especially given BB is written in pure R (no C or Fortran):
There is always to say something. First note how clever the test function was implemented. By setting up the vector i we don't need to do any loops. But also note that the test function is not correctly evaluated with \(n=2\). In R the sequence "\(2:1\)" results in counting backwards. It generates the numbers \([2\> 1]\).
The second thing: the first number \(||F(x^0)||\) looks a bit funny. Indeed we have:
Looking at the source I see:
I do not understand the reason for the division by \(\sqrt{n}\). It is only doing this for the initial value. Subsequent log lines are correctly printing \(||F(x)||\).
For some problems it works rather nicely:
n <- 100
x0 <- -runif(n)
fbroyden <- function(x) {
n <- length(x)
F<- rep(NA,n)
F[1] <- x[1]*(3-0.5*x[1]) - 2*x[2] + 1
i <- 2:(n-1)
F[i] <- x[i]*(3-0.5*x[i]) - x[i-1] - 2*x[i+1] + 1
F[n] <- x[n]*(3-0.5*x[n]) - x[n-1] + 1
F
}
library(BB)
dfsane(par=x0,fn=fbroyden,control=list(triter=1))
This solves very quickly especially given BB is written in pure R (no C or Fortran):
Iteration: 0 ||F(x0)||: 1.441474
iteration: 1 ||F(xn)|| = 11.23496
iteration: 2 ||F(xn)|| = 7.900779
iteration: 3 ||F(xn)|| = 6.542131
iteration: 4 ||F(xn)|| = 3.139281
iteration: 5 ||F(xn)|| = 4.031394
iteration: 6 ||F(xn)|| = 5.183635
iteration: 7 ||F(xn)|| = 0.5650939
iteration: 8 ||F(xn)|| = 0.1870383
iteration: 9 ||F(xn)|| = 0.09536832
iteration: 10 ||F(xn)|| = 0.04728994
iteration: 11 ||F(xn)|| = 0.0231393
iteration: 12 ||F(xn)|| = 0.01204457
iteration: 13 ||F(xn)|| = 0.01253904
iteration: 14 ||F(xn)|| = 0.004658623
iteration: 15 ||F(xn)|| = 0.00671189
iteration: 16 ||F(xn)|| = 0.002562044
iteration: 17 ||F(xn)|| = 0.0007190788
iteration: 18 ||F(xn)|| = 0.000526917
iteration: 19 ||F(xn)|| = 0.0004205647
iteration: 20 ||F(xn)|| = 0.0003359958
iteration: 21 ||F(xn)|| = 0.0002087812
iteration: 22 ||F(xn)|| = 2.471228e-05
iteration: 23 ||F(xn)|| = 7.605672e-06
iteration: 24 ||F(xn)|| = 1.304184e-05
iteration: 25 ||F(xn)|| = 3.089383e-05
iteration: 26 ||F(xn)|| = 6.874306e-05
iteration: 27 ||F(xn)|| = 3.30784e-05
iteration: 28 ||F(xn)|| = 1.16808e-05
iteration: 29 ||F(xn)|| = 3.541111e-06
iteration: 30 ||F(xn)|| = 8.387173e-07
$par
[1] -1.0323920-1.3150464-1.3887103-1.4076713-1.4125364-1.4137837-1.4141034-1.4141853-1.4142063-1.4142117-1.4142131
[12] -1.4142135-1.4142135-1.4142135-1.4142135-1.4142136-1.4142135-1.4142136-1.4142136-1.4142136-1.4142136-1.4142136
[23] -1.4142136-1.4142136-1.4142135-1.4142136-1.4142135-1.4142136-1.4142135-1.4142136-1.4142135-1.4142136-1.4142135
[34] -1.4142136-1.4142136-1.4142136-1.4142135-1.4142136-1.4142136-1.4142136-1.4142136-1.4142136-1.4142135-1.4142136
[45] -1.4142136-1.4142136-1.4142136-1.4142136-1.4142136-1.4142136-1.4142136-1.4142135-1.4142136-1.4142135-1.4142136
[56] -1.4142136-1.4142136-1.4142136-1.4142136-1.4142136-1.4142136-1.4142136-1.4142135-1.4142136-1.4142136-1.4142136
[67] -1.4142136-1.4142136-1.4142135-1.4142136-1.4142136-1.4142135-1.4142135-1.4142135-1.4142135-1.4142135-1.4142134
[78] -1.4142132-1.4142128-1.4142121-1.4142107-1.4142080-1.4142027-1.4141924-1.4141722-1.4141328-1.4140561-1.4139064
[89] -1.4136144-1.4130448-1.4119339-1.4097678-1.4055460-1.3973251-1.3813439-1.3503811-1.2907820-1.1775120-0.9675106
[100] -0.5965290
$residual
[1] 8.387173e-08
$fn.reduction
[1] 14.41473
$feval
[1] 31
$iter
[1] 30
$convergence
[1] 0
$message
[1] "Successful convergence"
There is always to say something. First note how clever the test function was implemented. By setting up the vector i we don't need to do any loops. But also note that the test function is not correctly evaluated with \(n=2\). In R the sequence "\(2:1\)" results in counting backwards. It generates the numbers \([2\> 1]\).
The second thing: the first number \(||F(x^0)||\) looks a bit funny. Indeed we have:
> f <- fbroyden(x0)
> norm(f,type="2")
[1] 16.11595
>
Looking at the source I see:
F0 <- normF <- sqrt(sum(F * F))
if (trace)
cat("Iteration: ", 0, " ||F(x0)||: ", F0/sqrt(n), "\n")
I do not understand the reason for the division by \(\sqrt{n}\). It is only doing this for the initial value. Subsequent log lines are correctly printing \(||F(x)||\).
References
- Varadhan, Ravi, and Paul D. Gilbert. 2009. “BB: An R Package for Solving a Large System of Nonlinear Equations and for Optimizing a High-Dimensional Nonlinear Objective Function.” Journal of Statistical Software 32 (4): 1–26. http://www.jstatsoft.org/v32/i04/.
- Cruz, William La, and Marcos Raydan. 2003. “Nonmonotone Spectral Methods for Large-Scale Nonlinear Systems.” Optimization Methods and Software 18: 583–99.
- Cruz, William La, Jose Mario Martınez, and Marcos Raydan. 2006. “Spectral Residual Method Without Gradient Information for Solving Large-Scale Nonlinear Systems of Equations.” Mathematics of Computation 75: 1449–66.