A secant update for the Hessian

I now return to the question of finding a secant update for the Hessian that preserves symmetry and positive definiteness. That is, given a symmetric and positive definite matrix H_k and

$$s^{(k)}=x^{(k+1)}-x^{(k)},\ y^{(k)}=\nabla f(x^{(k+1)})-\nabla f(x^{(k)}),$$

I want to find a symmetric positive definite matrix H_k+1 that satisfies

 

H_k+1s^(k)=y^(k). (6)

Such an H_k+1 does not necessarily exist. If (6) holds, where H_k+1 is positive definite, then

$$s^{(k)}\cdot y^{(k)}=s^{(k)}\cdot H_{k+1}s^{(k)}>0.$$

Therefore, if $s^{(k)}\cdot y^{(k)}>0$ fails, it will be impossible to find H_k+1. The condition $s^{(k)}\cdot y^{(k)}>0$ is equivalent to

$$\left(\nabla f(x^{(k+1)})-\nabla f(x^{(k)})\right)\cdot\left(x^{(k+1)}-x^{(k)}\right)>0.$$ (7)

A strictly convex function f would satisfy (7) for any points x^(k) and x^(k+1). The condition (7) therefore means that $\nabla f(x^{(k)})$ and $\nabla f(x^{(k+1)})$ are consistent with f's having positive curvature on the line segment between x^(k) and x^(k+1). If this condition were to fail, it would be impossible to find a positive definite matrix H_k+1 satisfying the secant equation.

Next I wish to point out the following: No rank-one update can produce H_k+1 that is symmetric, positive definite, and satisfies the secant equation, at least not in every case. Indeed, to preserve symmetry and positive definiteness, a rank-one update would have to take the form

H_k+1=H_k+uu^T.

However, it is easy to show that, in some cases (even when $s^{(k)}\cdot y^{(k)}>0$holds), no vector u causes H_k+1 to satisfy the secant equation.⁴

Since a rank one update cannot be found, I will try a different approach. To simplify the notation in the following derivation, I will write H=H_k, H₊=H_k+1, s=s^(k), and y=y^(k). The positive definite matrix H has a Cholesky factorization H=LL^T. I will look for H₊ in the form H₊=JJ^T, where J is nonsingular and close to L in some sense. The reader should notice that, if J is nonsingular, then JJ^T is necessarily symmetric and positive definite.

If H₊ is to satisfy the secant equation, then

JJ^Ts=y

must hold. The matrix J will be found by a two-step process:

1.

Given any vector v, choose J so that Jv=y and J is as close as possible to L. Broyden's method shows how to do this:

$$J=L+\frac{\left(y-Lv\right)v^T}{v\cdot v}.$$ (8)

2.

Choose v so that J^Ts=v also holds (then JJ^T will satisfy the secant equation (6)). Using (8),

$$\begin{eqnarray*}J^Ts=v&\Rightarrow&L^Ts+\frac{v(y-Lv)^Ts}{v\cdot v}=v\\ &\Righ... ...\frac{y\cdot s}{v\cdot v}+ \frac{v\cdot L^Ts}{v\cdot v}\right)v. \end{eqnarray*}$$

This last equation is quite nonlinear in v, and to solve it directly would be difficult or impossible. However, it shows that the vectors v and L^Ts must point in the same direction. Therefore, there exists $\alpha$such that

$$v=\alpha L^Ts.$$

Substituting this formula for v into (8) and simplifying yields

$$J=L+\frac{ys^TL}{\alpha s\cdot Hs}-\frac{Hss^TL}{s\cdot Hs}$$

(to arrive at this formula for J, I used the fact that H=LL^T).

I now use the equation J^Ts=v, where $v=\alpha L^Ts$, to solve for $\alpha$:

$$\begin{eqnarray*}J^Ts=v&\Rightarrow&L^Ts+\frac{L^Tsy^Ts}{\alpha s\cdot Hs}- \fra... ...\alpha L^Ts\\ &\Rightarrow&\alpha^2=\frac{y\cdot s}{s\cdot Hs}. \end{eqnarray*}$$

There are two solutions for $\alpha$; the positive solution is the correct one to take, since then J=L if H already satisfies the secant equation. (The reader should notice that the requirement that $s\cdot y$ be positive appears here.)

Therefore, if

$$J=L+\frac{ys^TL}{\alpha s\cdot Hs}-\frac{Hss^TL}{s\cdot Hs},$$ (9)

where

$$\alpha=\sqrt{\frac{y\cdot s}{s\cdot Hs}},$$

then H₊=JJ^T is symmetric, positive definite, and satisfies

H₊s=y.

Since J, as given by (9), is not necessarily lower triangular (even though L is), it is more convenient to express the update in terms of H and H₊ rather than in terms of L and J. A tedious calculation, which simplifies nicely in the end, shows that

$$JJ^T=H-\frac{Hss^TH}{s\cdot Hs}+\frac{yy^T}{y\cdot s}.$$

The resulting update is known as the BFGS update:⁵

$$H_{k+1}=H_k-\frac{H_ks^{(k)}(s^{(k)})^TH_k}{s^{(k)}\cdot H_ks^{(k)}}+ \frac{y^{(k)}(y^{(k)})^T}{y^{(k)}\cdot s^{(k)}}.$$ (10)