1.2 Principle of Optimization Transfer

There is a general principle that will be repeated in this book that Kenneth Lange calls “optimization transfer”. The basic idea applies to the problem of maximizing a function \(f\).

1. We want to maximize \(f\), but it is difficult to do so.

2. We can compute an approximation to \(f\), call it \(g\), based on local information about \(f\).

3. Instead of maximizing \(f\), we “transfer” the maximization problem to \(g\) and maximize \(g\) instead.

4. We iterate Steps 2 and 3 until convergence.

The difficult problem of maximizing \(f\) is replaced with the simpler problem of maximizing \(g\) coupled with iteration (otherwise known as computation). This is the optimization “transfer”.

Note that all of the above applies to minimization problems, because maximizing \(f\) is equivalent to minimizing \(-f\).