A secant update for the Jacobian

In solving F(x)=0, where $F:{\bf {\rm R}}^n\rightarrow{\bf {\rm R}}^n$, one uses x^(k), F(x^(k)), and J(x^(k)), or an approximation to it, to produce x^(k+1). I assume that I have a nonsingular matrix A_k approximating J(x^(k)) and I wish to produce an approximation A_k+1 to J(x^(k+1)). Since

$$F(x^{(k)})\doteq F(x^{(k+1)})+J(x^{(k+1)})(x^{(k)}-x^{(k+1)})$$

should hold, the secant equation will be

 

A_k+1s^(k)=y^(k), (2)

where

$$s^{(k)}=x^{(k+1)}-x^{(k)},\ y^{(k)}=F(x^{(k+1)})-F(x^{(k)}).$$

Since I have new information only in the direction s^(k), I require that A_k and A_k+1 agree on all vectors orthogonal to s^(k):

$$p\cdot s^{(k)}=0\ \Rightarrow\ A_{k+1}p=A_kp.$$ (3)

I now wish to use (2) and (3) to choose A_k+1.

Condition (2) implies that the matrix A_k+1-A_k has a null space of dimension n-1 (there are n-1 independent vectors orthogonal to s^(k)). Therefore, the Fundamental Theorem of Linear Algebra² implies that the rank of A_k+1-A_kmust be 1. Therefore, every column of A_k+1-A_k must be a multiple of a common vector u:

$$A_{k+1}-A_k=\left[v_1u\vert v_2u\vert\cdots\vert v_nu\right].$$

The matrix on the right can be written as uv^T, where u and v are vectors, regarded as $n\times 1$ matrices. The reader should verify the following fundamental formula:

$$uv^Tx=(v\cdot x)u\ \mbox{for all}\x\in{\bf {\rm R}}^n.$$

Thus A_k+1 is to be chosen as a rank-one update of A_k:

 

A_k+1=A_k+uv^T. (4)

I can now determine u and v. Conditions (3) and (4) imply that

$$(uv^T)p=0\ \mbox{for all}\p\ \mbox{such that}\ p\cdot s^{(k)}=0,$$

or, equivalently,

$$(v\cdot p)u=0\ \mbox{for all}\p\ \mbox{such that}\ p\cdot s^{(k)}=0.$$

This implies that

$$v\cdot p=0\ \mbox{for all}\p\ \mbox{such that}\ p\cdot s^{(k)}=0.$$

But the only vector v satisfying this condition is s^(k) itself (or a multiple of it). Therefore, I take v=s^(k) and use (2) to determine u:

$$\begin{eqnarray*}A_{k+1}s^{(k)}=y^{(k)}&\Rightarrow&A_ks^{(k)}+(s^{(k)}\cdot s^{... ... &\Rightarrow&u=\frac{y^{(k)}-A_ks^{(k)}}{s^{(k)}\cdot s^{(k)}}. \end{eqnarray*}$$

Therefore,

$$A_{k+1}=A_k+\frac{\left(y^{(k)}-A_ks^{(k)}\right)(s^{(k)})^T}{s^{(k)}\cdot s^{(k)}}.$$ (5)

Equation (5) is referred to as Broyden's update.

Normally when using Broyden's update, the initial Jacobian estimate is taken to be J(x⁽⁰⁾) or a finite-difference estimate of it.³ It can be shown that, under certain conditions, the local convergence of Broyden's method for solving F(x)=0 is superlinear. This is not as fast as Newton's method, which converges quadratically; however, since Broyden's method can use much less time per iteration by avoiding the computation of J(x^(k)), it is more efficient than Newton's method on some problems.

I will give more details about Broyden's method for solving F(x)=0later.