9.3 Deterministic DynProg

For a given deterministic dynamic programming without end point constraints: \[ \begin{align} \min \quad & J = \phi \left(\mathbf{x}_{N}\right) + \sum_{i=0}^{N-1} L_{i}\left(\mathbf{x}_{i}, \mathbf{u}_{i}\right) \\ \text{s.t.} \quad & \mathbf{x}_{i+1} = f_{i} \left(\mathbf{x}_{i}, \mathbf{u}_{i}\right) \quad \\ & \mathbf{x}_{0} = \underline{\mathbf{x}}_{0} \\ & \mathbf{u}_{i} \in \mathcal{U}_{i} \end{align} \]

9.3.1 Pontryagins Maximum principle

The necessary condition for \(\hat{\mathbf{u}}_0, \hat{\mathbf{u}}_1, ..., \hat{\mathbf{u}}_{N-1}\) being critical points is given by the Euler-Lagrange equations: \[ \begin{align} x_{i+1, j} &= f_{i} \left(x_{i, j}, u_{i, j} \right) \quad \text{for } j = 1, 2, ..., M \text{ and } i = 0, 1, 2, ..., N \quad \text{(state equations)} \\ \lambda_{i, j} &=\frac{\partial H_{i}}{\partial x_{i, j}} \quad \text{for } j = 1, 2, ..., M \text{ and } i = 0, 1, 2, ..., N \quad \text{(costate equations)} \\ u_{i, j} &= \arg \min _{u_{i} \in \mathcal{U}_{i}} H_{i} \quad \text{for } j = 1, 2, ..., M \text{ and } i = 0, 1, 2, ..., N \quad \text{(optimality conditions)} \\ \end{align} \] where the boundary conditions are: \[ \begin{align} \mathbf{x}_{0} &= \underline{\mathbf{x}}_{0} \\ \lambda_{i, N}^{T} &= \frac{\partial}{\partial x_{i, N}} \phi \left(\mathbf{x}_{N}\right) \quad \text{for } j = 1, 2, ..., M \text{ and } i = 0, 1, 2, ..., N \end{align} \]

Exm 9.2 with 2-D Expressions

If the original two-dimensional expressions are to be used, the deterministic dynamic programming becomes: \[ \begin{align} \min_{\textbf{u}, \textbf{v}} \quad & J = \sum_{i=0}^{N-1} \left[ m g y_{i} + \frac{1}{2} m g l sin(\theta_i) \right] \\ \text{s.t.} \quad & \left[\begin{array}{l}{z} \\ {y}\end{array}\right]_{i+1} = \left[ \begin{array}{l}{z} \\ {y}\end{array} \right]_{i} + l \left[ \begin{array}{l}{u} \\ {v}\end{array} \right]_i \quad \text{for } i = 0, 1, ..., N-1 \\ & u_i^2 + v_i^2 = l^2 \quad \text{for } i = 0, 1, ..., N-1 \\ & \left[ \begin{array}{l}{z} \\ {y}\end{array} \right]_{0} = \left[ \begin{array}{l}{0} \\ {0}\end{array} \right] \\ & \left[ \begin{array}{l}{z} \\ {y}\end{array} \right]_{N} = \left[ \begin{array}{l}{h} \\ {0}\end{array} \right] \end{align} \] where \(z_1, z_2, ..., z_{N-1}\) and \(y_1, y_2, ..., y_{N-1}\) are decision variables as well, but they are determined once \(\theta_0, \theta_1, ..., \theta_{N-1}\) are chosen.

The \(i\)-th Hamiltonian function becomes:

\[ H_i = m g y_{i} + \frac{1}{2} m g v_{i} + \lambda^z_{i+1} \left( z_{i} + u_{i} \right) + \lambda^y_{i+1} \left( y_{i} + v_{i} \right) \]

according to Pontryagins Maximum principle, if we consider :

\[ \begin{align} \lambda^z_{i+1} &= \lambda^{z}_{i} \\ \lambda^y_{i+1} &= \lambda^{y}_{i} - m g \\ \left[u_i, v_i \right]^T &= \arg \min_{u_i^2 + v_i^2 = l^2} \left\{ m g y_{i} + \frac{1}{2} m g v_i + \lambda^z_{i+1} \left( z_{i} + u_i \right) + \lambda^y_{i+1} \left( y_{i} + v_i \right) \right\} \\ z_{i+1} &= z_{i} + u_i \\ y_{i+1} &= y_{i} + v_i \\ \end{align} \]

The optimization algorithms and the inputed original values have large influence on the final result. If the correct values are inputed as the original values, we can get the correct values.