27.1 Root finding

27.1.1 Newton’s method (Newton–Raphson algorithm)

27.1.2 Gauss–Newton algorithm

27.1.3 Gradient Descent

27.1.3.1 Stochastic gradient descent (SGD)

Implicit updates (ISGD)

27.1.3.2 Momentum

27.1.3.3 AdaGrad

27.1.3.4 RMSProp

27.1.3.5 Adaptive Moment Estimation (Adam)

coordinate descent

27.1.4 Conjugate gradient method

27.1.5 quasi-Newton method

Broyden’s method

Symmetric rank-one

27.1.6 Nelder–Mead method

27.1.7 Broyden–Fletcher–Goldfarb–Shanno (BFGS) algorithm

BFGS and L-BFGS