Next: The steepest-descent method Up: Globalizing Newton's method: Descent Previous: Introduction

# Descent Directions

The reader will recall that is a descent direction for f at x(k) if

This condition implies that

Indeed, if represents the one-dimensional slice'' of f in the direction of p,

then the descent condition implies that and hence that for all sufficiently small (see Figure 1).

Given x(k) and a descent direction p, it is possible to reduce f by moving in the direction of p, that is, by choosing an appropriate and defining . A procedure for choosing is referred to as a line search (since x(k+1) is found on the (half-)line parametrized as ). I will discuss line searches (a solution to the third difficulty described above in the Introduction) later. For now, I want to concentrate on methods for producing descent directions.

Newton's method produces the direction

This is a descent direction if

that is, if

that is, if

 (1)

Condition (1) will hold if is positive definite. The reader may recall that the eigenvalues of H-1 are simply the reciprocals of the eigenvalues of H,1 and therefore is positive definite if and only if is positive definite. Of course, if is positive definite, then it is nonsingular, and in this case the first two difficulties mentioned in the Introduction disappear. If is positive definite, then the Newton step is well-defined and represents a descent direction.

The following observation is essential: If H is any symmetric positive definite matrix, then is a descent direction for f at x(k). This suggests the following modification of Newton's method:

 (2)

where Hk is a positive definite approximation of and is a step-length parameter that is chosen by a line search. An iteration of the form (2) is referred to as a quasi-Newton iteration. In order for the rapid local convergence of Newton's method to be preserved, Hk should be a good approximation to (or equal to) when x(k) is close to x*.

I will now describe two specific quasi-Newton methods.

Next: The steepest-descent method Up: Globalizing Newton's method: Descent Previous: Introduction
Mark S. Gockenbach
2003-01-30