9  Quadratic From

A quadratic form is a type of polynomial expression that is a sum of terms where each term is either a variable squared or the product of two variables, often associated with a symmetric matrix. Quadratic forms are useful in various areas of mathematics, physics, statistics, and optimization.

9.1 Definition of Quadratic Form

A quadratic form is a type of mathematical expression that can be written as a sum of terms, each of which involves the product of a variable with itself (squared terms) or with another variable (cross terms). In general, a quadratic form is a homogeneous polynomial of degree 2 in a set of variables.

9.1.1 2D Quadratic Form

Let the vector and symmetric matrix in 2D be defined as follows:

$$\mathbf{x}=\left[\begin{array}{c}{x}_1\\ x_2\end{array}\right],\mathbf{Q}=\left[\begin{array}{cc}{q}_{11}& q_{12}\\ q_{12}& q_{22}\end{array}\right].$$

The quadratic form is expressed as:

$$Q(\mathbf{x})={\mathbf{x}}^T\mathbf{Q}\mathbf{x}.$$

Expanding this expression gives:

$$Q(\mathbf{x})=\left[\begin{array}{cc}{x}_1& x_2\end{array}\right]\left[\begin{array}{cc}{q}_{11}& q_{12}\\ q_{12}& q_{22}\end{array}\right]\left[\begin{array}{c}{x}_1\\ x_2\end{array}\right].$$

The resulting quadratic form is:

$$Q(x_1,x_2)=q_{11}{x}_1^2+q_{22}{x}_2^2+2q_{12}{x}_1{x}_2.$$

9.1.2 3D Quadratic Form

For 3D, the vector and symmetric matrix are defined as follows:

$$\mathbf{x}=\left[\begin{array}{c}{x}_1\\ x_2\\ x_3\end{array}\right],\mathbf{Q}=\left[\begin{array}{ccc}{q}_{11}& q_{12}& q_{13}\\ q_{12}& q_{22}& q_{23}\\ q_{13}& q_{23}& q_{33}\end{array}\right].$$

The quadratic form in this case is expressed as:

$$Q(\mathbf{x})={\mathbf{x}}^T\mathbf{Q}\mathbf{x}.$$

Expanding it gives:

$$Q(\mathbf{x})=\left[\begin{array}{ccc}{x}_1& x_2& x_3\end{array}\right]\left[\begin{array}{ccc}{q}_{11}& q_{12}& q_{13}\\ q_{12}& q_{22}& q_{23}\\ q_{13}& q_{23}& q_{33}\end{array}\right]\left[\begin{array}{c}{x}_1\\ x_2\\ x_3\end{array}\right].$$

The expanded quadratic form is:

$$Q(x_1,x_2,x_3)=q_{11}{x}_1^2+q_{22}{x}_2^2+q_{33}{x}_3^2+2q_{12}{x}_1{x}_2+2q_{13}{x}_1{x}_3+2q_{23}{x}_2{x}_3.$$

9.1.3 nD Quadratic Form

In $n$-dimensions, the vector and symmetric matrix are defined as:

$$\mathbf{x}=\left[\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_n\end{array}\right],\mathbf{Q}=\left[\begin{array}{cccc}{q}_{11}& q_{12}& \dots & q_{1n}\\ q_{12}& q_{22}& \dots & q_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ q_{1n}& q_{2n}& \dots & q_{nn}\end{array}\right].$$

The quadratic form is expressed as:

$$Q(\mathbf{x})={\mathbf{x}}^T\mathbf{Q}\mathbf{x}.$$

Steps to Calculate a Quadratic Form:

1. Compute the transpose of $\mathbf{x}$:
   First, express $\mathbf{x}$ as a column vector of variables: $$\mathbf{x}=\left[\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_n\end{array}\right].$$
   The transpose of $\mathbf{x}$, denoted as ${\mathbf{x}}^T$, is: $${\mathbf{x}}^T=[x_1{x}_2\dots x_n].$$

2. Multiply ${\mathbf{x}}^T$ with the matrix $\mathbf{Q}$:
   Multiply the row vector ${\mathbf{x}}^T$ by the symmetric matrix $\mathbf{Q}$: $${\mathbf{x}}^T\mathbf{Q}=[x_1{x}_2\dots x_n]\left[\begin{array}{cccc}{q}_{11}& q_{12}& \dots & q_{1n}\\ q_{12}& q_{22}& \dots & q_{2n}\\ ⋮& ⋮& \ddots & ⋮\\ q_{1n}& q_{2n}& \dots & q_{nn}\end{array}\right].$$
   This produces a row vector: $$[\sum _{j=1}^n{q}_{1j}{x}_j,\sum _{j=1}^n{q}_{2j}{x}_j,\dots ,\sum _{j=1}^n{q}_{nj}{x}_j].$$

3. Multiply the result by $\mathbf{x}$:
   Multiply the resulting row vector by the column vector $\mathbf{x}$:

   $$Q(\mathbf{x})=[\sum _{j=1}^n{q}_{1j}{x}_j,\sum _{j=1}^n{q}_{2j}{x}_j,\dots ,\sum _{j=1}^n{q}_{nj}{x}_j]\left[\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_n\end{array}\right].$$
   This produces the final quadratic form expression.

4. Final Expression:

   In summation form: $$Q(\mathbf{x})=\sum _{i=1}^n\sum _{j=1}^n{q}_{ij}{x}_i{x}_j.$$

   Separating diagonal and off-diagonal terms: $$Q(\mathbf{x})=\sum _{i=1}^n{q}_{ii}{x}_i^2+2\sum _{i=1}^n\sum _{j=i+1}^n{q}_{ij}{x}_i{x}_j.$$

   Or, in fully expanded form: $$Q(\mathbf{x})=q_{11}{x}_1^2+q_{22}{x}_2^2+\cdots +q_{nn}{x}_n^2+2(q_{12}{x}_1{x}_2+q_{13}{x}_1{x}_3+\cdots +q_{n-1,n}{x}_{n-1}{x}_n).$$

Key Insights:

• The diagonal terms $q_{ii}$ correspond to the squared variables $x_i^2$.
• The off-diagonal terms $q_{ij}$ (where $i\ne j$) represent the cross terms $x_i{x}_j$.

9.2 Key Concepts

1. Symmetry of the Matrix $\mathbf{Q}$:
   The matrix $\mathbf{Q}$ must be symmetric, meaning that $q_{ij}=q_{ji}$. This symmetry ensures that the quadratic form only contains real coefficients and simplifies the expression of the form.

2. Diagonal and Off-Diagonal Terms:
   The quadratic form contains two types of terms:

   • Diagonal terms ($q_{ii}$), which represent the squared terms of the variables $x_i^2$.
   • Off-diagonal terms ($q_{ij}$, for $i\ne j$), which correspond to the interactions between different variables $x_i$ and $x_j$ (cross terms like $x_i{x}_j$).

3. Scalar Output:
   The result of applying the quadratic form to the vector $\mathbf{x}$ is a scalar, i.e., a single numerical value.

9.3 Geometric Interpretation of Quadratic Forms

In the context of quadratic forms, the matrix $\mathbf{Q}$ defines the geometric properties of the surface associated with the quadratic form $f(\mathbf{x})={\mathbf{x}}^T\mathbf{Q}\mathbf{x}$. The geometry of the surface described depends on the properties of $\mathbf{Q}$.

9.3.1 Ellipsoid

If $\mathbf{Q}$ is a positive definite matrix, all the eigenvalues of $\mathbf{Q}$ are positive. This means that the quadratic form represents a surface where all directions curve outward, such as an ellipsoid in 3D or an ellipse in 2D. In this case, the quadratic form is always positive for any non-zero vector $\mathbf{x}$, indicating a closed surface. An ellipsoid can be described with the equation:

$$\frac{x^2}{a^2}+\frac{y^2}{b^2}+\frac{z^2}{c^2}=1$$

9.3.2 Hyperboloid

If $\mathbf{Q}$ is indefinite, meaning that $\mathbf{Q}$ has both positive and negative eigenvalues, the quadratic form describes a hyperboloid. In this case, the surface could take on one of two general forms:

• A one-sheeted hyperboloid, if there’s a pair of positive and negative eigenvalues.A one-sheeted hyperboloid can be described with the equation:

$$\frac{x^2}{a^2}-\frac{y^2}{b^2}-\frac{z^2}{c^2}=1$$ - A two-sheeted hyperboloid, if there’s more complex mixing of positive and negative eigenvalues. A two-sheeted hyperboloid can be described with the equation:

$$\frac{x^2}{a^2}-\frac{y^2}{b^2}-\frac{z^2}{c^2}=-1$$

A hyperboloid is an open surface, unlike the ellipsoid. Let consider the following one-sheeted hyperboloid visualization:

9.3.3 Paraboloid

A paraboloid arises when the quadratic form has a special structure, often reflecting a situation where the matrix $\mathbf{Q}$ has both positive and zero eigenvalues. This typically occurs when there is a critical point along one of the axes. A paraboloid can open upwards, downwards, or sideways, depending on the sign of the nonzero eigenvalue.

• An elliptic paraboloid can be described with the equation:

$$\frac{x^2}{a^2}+\frac{y^2}{b^2}=z$$

This involves using a positive definite matrix for the elliptic paraboloid.

• A hyperbolic paraboloid can be described with the equation:

$$\frac{x^2}{a^2}-\frac{y^2}{b^2}=z$$

The type of surface described by a quadratic form is determined by the positive, negative, or zero nature of the eigenvalues of the matrix $\mathbf{Q}$, which governs the curvature and openness of the graph.

9.4 Applications of Quadratic Forms

Quadratic forms are essential in various disciplines:

• Optimization: In finding the minimum or maximum of a function, especially quadratic functions in constrained optimization problems.
• Statistics: In the context of variance-covariance matrices and regression analysis.
• Physics and Engineering: For representing the energy of a system or describing various physical systems.
• Machine Learning: In algorithms like Support Vector Machines (SVM) and in kernel methods where the quadratic form is used to map data into higher-dimensional spaces.

9.5 Simple Implementation in Python

Below is a Python implementation of a quadratic form:

import numpy as np

# Define values for x1 and x2
x1, x2 = 1, 2  # Example 

# Matrix Q and vector x
Q = np.array([[2, 2],
              [2, 3]])
x = np.array([[x1], [x2]])

# Compute the quadratic form
Q_x = np.dot(x.T, np.dot(Q, x))
print("Quadratic Form Q(x):", Q_x)

Quadratic Form Q(x): [[22]]