4. INNER PRODUCTS AND ORTHOGONALITY

In making the definition of a vector space, we generalized the linear structure (addition and scalar multiplication) of ℝ² an ℝ³. We ignored other important features, such as the notions of length and angle. These ideas are embedded in the concept we now investigate, inner products.

4.1 Inner product spaces

We begin with a definition.

Using the linearity axiom (I1) and the symmetry axiom (I2), we obtain

⟨u,av₁ + bv₂⟩ = ⟨av₁ + bv₂, u⟩ = a⟨v₁, u⟩ + b⟨v₂, u⟩ = a⟨u, v₁⟩ + b⟨u, v₂⟩.

That is, the inner product function is also linear in its second entry. By induction, we obtain

⟨a₁ u₁ + a₂ u₂ + … + ar ur, v⟩ = a₁⟨u₁, v⟩ + a₂⟨u₂, v⟩ + … + a_r⟨ur, v⟩

and

⟨u, b₁ v₁ + b₂ v₂ + … + bs vs⟩ = b₁⟨u, v₁⟩ + b₂⟨u, v₂⟩ + … + bs⟨u, vs⟩.

Combining these two properties yields the following general formula:

$$\begin{array}{r}⟨\sum _{i=1}^r a_i{\mathbf{u}}_i,\sum _{j=1}^s b_j{\mathbf{v}}_j⟩=\sum _{i=1}^r \sum _{j=1}^s a_i{b}_j⟨{\mathbf{u}}_i,{\mathbf{v}}_j⟩\\ =\left[\begin{array}{llll}{a}_1& a_2& \dots & a_r\end{array}\right]\left[\begin{array}{cccc}⟨{\mathbf{u}}_1,{\mathbf{v}}_1⟩& ⟨{\mathbf{u}}_1,{\mathbf{v}}_2⟩& \dots & ⟨{\mathbf{u}}_1,{\mathbf{v}}_s⟩\\ ⟨{\mathbf{u}}_2,{\mathbf{v}}_1⟩& ⟨{\mathbf{u}}_2,{\mathbf{v}}_2⟩& \dots & ⟨{\mathbf{u}}_2,{\mathbf{v}}_s⟩\\ \dots & \dots & \dots & \dots \\ ⟨{\mathbf{u}}_r,{\mathbf{v}}_1⟩& ⟨{\mathbf{u}}_r,{\mathbf{v}}_2⟩& \dots & ⟨{\mathbf{u}}_r,{\mathbf{v}}_s⟩\end{array}\right]\left[\begin{array}{c}{b}_1\\ b_2\\ ⋮\\ b_s\end{array}\right]\end{array} (4.1)$$

By axiom (I3), ⟨u, u⟩ is a non negative number and hence its positive real square root exists. We use the notation

$$\Vert \mathbf{u}\Vert =\sqrt{⟨\mathbf{u},\mathbf{u}⟩}.$$

This non negative real number ∥u∥ is called the norm of u. The relation ∥u∥² = ⟨u, u⟩ will be used frequently.

4.2 Gram matrices of an inner product

Let V be an inner product space and let $\mathcal{B}$ = {b₁, b₂, …, b_n} be a basis for V. Since any vectors u, v ∈ V can be written as a linear combination of the basis using their coordinates as weights, we can write the inner product using (4.1) as follows

$$⟨\mathbf{u},\mathbf{v}⟩=[\mathbf{u}{]}_{\mathcal{B}}^{\mathrm{\top }}\left[\begin{array}{cccc}⟨{\mathbf{b}}_1,{\mathbf{b}}_1⟩& ⟨{\mathbf{b}}_1,{\mathbf{b}}_2⟩& \dots & ⟨{\mathbf{b}}_1,{\mathbf{b}}_n⟩\\ ⟨{\mathbf{b}}_2,{\mathbf{b}}_1⟩& ⟨{\mathbf{b}}_2,{\mathbf{b}}_2⟩& \dots & ⟨{\mathbf{b}}_2,{\mathbf{b}}_n⟩\\ \dots & \dots & \dots & \dots \\ ⟨{\mathbf{b}}_n,{\mathbf{b}}_1⟩& ⟨{\mathbf{b}}_n,{\mathbf{b}}_2⟩& \dots & ⟨{\mathbf{b}}_n,{\mathbf{b}}_n⟩\end{array}\right][\mathbf{v}{]}_{\mathcal{B}}.$$

The n × n matrix

$$G^{\mathcal{B}}=\left[\begin{array}{cccc}⟨{\mathbf{b}}_1,{\mathbf{b}}_1⟩& ⟨{\mathbf{b}}_1,{\mathbf{b}}_2⟩& \dots & ⟨{\mathbf{b}}_1,{\mathbf{b}}_n⟩\\ ⟨{\mathbf{b}}_2,{\mathbf{b}}_1⟩& ⟨{\mathbf{b}}_2,{\mathbf{b}}_2⟩& \dots & ⟨{\mathbf{b}}_2,{\mathbf{b}}_n⟩\\ \dots & \dots & \dots & \dots \\ ⟨{\mathbf{b}}_n,{\mathbf{b}}_1⟩& ⟨{\mathbf{b}}_n,{\mathbf{b}}_2⟩& \dots & ⟨{\mathbf{b}}_n,{\mathbf{b}}_n⟩\end{array}\right]$$

is called the Gram matrix if the inner product relative to the basis $\mathcal{B}$. Notice that by axioms (I2) and (I3), the Gram matrix must satisfy

$${\left(G^{\mathcal{B}}\right)}^{\mathrm{\top }}=G^{\mathcal{B}}\text{ and }0\le {\mathbf{x}}^{\mathrm{\top }}{G}^{\mathcal{B}}\mathbf{x},\text{ for all }\mathbf{x}\in {\mathbb{R}}^n.$$

The definition of an inner product is independent of the choice of basis. Nevertheless, the Gram matrices of an inner product relative to an old basis $\mathcal{B}$ and a new basis $\mathcal{C}$ are, in general, not equal; that is, G^{$\mathcal{B}$} ≠ G^{$\mathcal{C}$}. So it is sensible to conclude that the two Gram matrices are related in some way. We explore this relation here.

First, using the Gram matrix G^{$\mathcal{B}$}, we can write

$$⟨\mathbf{u},\mathbf{v}⟩=[\mathbf{u}{]}_{\mathcal{B}}^{\mathrm{\top }}{G}^{\mathcal{B}}[\mathbf{v}{]}_{\mathcal{B}}.$$

Recall that for any vector x ∈ V, the coordinate vector [x]_{$\mathcal{B}$} is related to the coordinate vector [x]_{$\mathcal{C}$} by means of the change of basis matrix $P_{\mathcal{B}}^{\mathcal{C}}$:

$$[\mathbf{x}{]}_{\mathcal{B}}=P_{\mathcal{B}}^{\mathcal{C}}[\mathbf{x}{]}_{\mathcal{C}}.$$

Therefore, we can use the change of basis matrix to change the coordinate vectors relative to the old basis to coordinate vectors relative to the new basis:

$$⟨\mathbf{u},\mathbf{v}⟩={\left(P_{\mathcal{B}}^{\mathcal{C}}[\mathbf{u}{]}_{\mathcal{C}}\right)}^{\mathrm{\top }}{G}^{\mathcal{B}}\left(P_{\mathcal{B}}^{\mathcal{C}}[\mathbf{v}{]}_{\mathcal{C}}\right)=[\mathbf{u}{]}_{\mathcal{C}}^{\mathrm{\top }}\left({\left(P_{\mathcal{B}}^{\mathcal{C}}\right)}^{\mathrm{\top }}{G}^{\mathcal{B}}{P}_{\mathcal{B}}^{\mathcal{C}}\right)[\mathbf{v}{]}_{\mathcal{C}}.$$

Comparing the last equation with

$$⟨\mathbf{u},\mathbf{v}⟩=[\mathbf{u}{]}_{\mathcal{C}}^{\mathrm{\top }}{G}^{\mathcal{C}}[\mathbf{v}{]}_{\mathcal{C}},$$

we conclude that

$$G^{\mathcal{C}}={\left(P_{\mathcal{B}}^{\mathcal{C}}\right)}^{\mathrm{\top }}{G}^{\mathcal{B}}{P}_{\mathcal{B}}^{\mathcal{C}}.$$

4.3 Cauchy-Schwarz inequality

The following inequality is called the Cauchy-Schwarz inequality; it is used in many branches of mathematics.

Proof. Let u, v ε V. If v = 0, then both sides of the inequality are equal to 0 and the desired result holds. Thus we can assume that v ≠ 0.

By axiom (I3), for all λ ε ℝ, we get:

$$⟨\mathbf{u}-\lambda \mathbf{v},\mathbf{u}-\lambda \mathbf{v}⟩\ge 0$$

Consequently,

$$\Vert \mathbf{u}{\Vert }^2-2\lambda ⟨\mathbf{u},\mathbf{v}⟩+{\lambda }^2\Vert \mathbf{v}{\Vert }^2\ge 0$$

Taking $\lambda =\frac{⟨\mathbf{u},\mathbf{v}⟩}{\Vert \mathbf{v}{\Vert }^2}$, we obtain

$$\Vert \mathbf{u}{\Vert }^2-2\frac{⟨\mathbf{u},\mathbf{v}{⟩}^2}{\Vert \mathbf{v}{\Vert }^2}+\frac{⟨\mathbf{u},\mathbf{v}{⟩}^2}{\Vert \mathbf{v}{\Vert }^2}\ge 0$$

That is,

$$\Vert \mathbf{u}{\Vert }^2-\frac{⟨\mathbf{u},\mathbf{v}{⟩}^2}{\Vert \mathbf{v}{\Vert }^2}\ge 0$$

Multiplying both sides by ∥v∥²,

$$\Vert \mathbf{u}{\Vert }^2\Vert \mathbf{v}{\Vert }^2-⟨\mathbf{u},\mathbf{v}{⟩}^2\ge 0$$

which implies

$$⟨\mathbf{u},\mathbf{v}{⟩}^2\le \Vert \mathbf{u}{\Vert }^2\Vert \mathbf{v}{\Vert }^2.$$

Taking the positive square root of both sides, we finally get

$$|⟨\mathbf{u},\mathbf{v}⟩|\le \Vert \mathbf{u}\Vert \Vert \mathbf{v}\Vert .$$

Observe that if u = λ v, then

⟨u - λv, u - λv⟩ = 0.

In this case, each step of the proof is valid with ≤ and ≥ replaced with equality signs. Moreover, each step (with equality replacing inequalities) is reversible. Therefore, the Cauchy-Schwarz inequality is an equality if and only if u or v is a scalar multiple of the other.

The next theorem gives basic properties of the norm; the proof of the third property requires the Cauchy-Schwarz inequality.

Proof. Property (N1) is a direct consequence of axiom (I3). Property (N2) is a direct consequence of the definition of norm and the linearity of the inner product in both entries:

$$\Vert k\mathbf{v}\Vert =\sqrt{⟨k\mathbf{v},k\mathbf{v}⟩}=\sqrt{k^2⟨\mathbf{v},\mathbf{v}⟩}=|k|\Vert \mathbf{v}\Vert .$$

Most of our effort is in proving property (N3). For u, v ∈ V,

$$\begin{array}{r}\Vert \mathbf{u}+\mathbf{v}{\Vert }^2=⟨\mathbf{u}+\mathbf{v},\mathbf{u}+\mathbf{v}⟩\\ =\Vert \mathbf{u}{\Vert }^2+2⟨\mathbf{u},\mathbf{v}⟩+\Vert \mathbf{v}{\Vert }^2\\ \le \Vert \mathbf{u}{\Vert }^2+2|⟨\mathbf{u},\mathbf{v}⟩|+\Vert \mathbf{v}{\Vert }^2\\ \le \Vert \mathbf{u}{\Vert }^2+2\Vert \mathbf{u}\Vert \Vert \mathbf{v}\Vert +\Vert \mathbf{v}{\Vert }^2\text{ (by Cauchy-Schwartz) }\\ =(\Vert \mathbf{u}\Vert +\Vert \mathbf{v}\Vert {)}^2.\end{array}$$

Taking square roots of both sides of the inequality above gives the triangle inequality. The proof above shows that the triangle inequality is an equality if and only if ⟨u, v⟩ = ∥u∥ ∥v∥. If one u, v is a non negative multiple of the other, then the above equality holds. Conversely, if the above equality holds, then the condition for the Cauchy-Schwarz inequality implies that one of u, v must be a scalar multiple of the other. If u = λ v, then bye (N2), ⟨u, v⟩ = ∥u∥ ∥v∥ implies λ = |λ|, which forces the scalar in question to be non negative, as desired.

If ∥u∥ = 1, or, equivalently, if ⟨u, u⟩ = 1, then u is called a unit vector and is said to be normalized. Every non zero vector v ∈ V can be multiplied by the reciprocal of its norm to obtain a unit vector

$$\widehat{\mathbf{v}}=\frac{\mathbf{v}}{\Vert \mathbf{v}\Vert }$$

which is a positive multiple of v. This process is called normalizing v.

The non negative real number

d(u, v) = ∥u- v∥

is called the distance between u and v. For any non zero vectors u, v ∈ V, the angle between u and v is defined to be the angle θ such that 0 ≤ θ ≤ π and

$$\cos \theta =\frac{⟨\mathbf{u},\mathbf{v}⟩}{\Vert \mathbf{u}\Vert \Vert \mathbf{v}\Vert }.$$

By the Cauchy-Schwartz inequality, -1 ≤ cos θ ≤ 1 and so the angle always exists and is unique.

4.4 Orthogonality

Let V be an inner product space. The vectors u, v ∈ V are said to be orthogonal and u is said to be orthogonal to v if

⟨u, v⟩ = 0.

The relation is clearly symmetric; that is, if u is orthogonal to v, then ⟨v, u⟩ = 0 and so v is orthogonal to u. We note that 0 ∈ V is orthogonal to every vector v ∈ V for

⟨0, v⟩ = ⟨0v,v⟩ = 0⟨v, v⟩ = 0.

Conversely, if u is orthogonal to every v ∈ V, then ⟨u, u⟩ = 0 and hence u = 0 by axiom (I3). Observe that u and v are orthogonal if and only if cos θ = 0 where θ is the angle between u and v, and this is true if and only if θ = π/2.

Let S be a subset of an inner product space V. The orthogonal complement of S, denoted by S^⊥ (read “S perp”) consists of those vectors in V which are orthogonal to every vector u ∈ S:

S^⊥ = {v ∈ V : ⟨v, u⟩ = 0 for every u ∈ S}.

We show that S^⊥ is a subspace of V. Clearly 0 ∈ S^⊥ since 0 is orthogonal to every vector in V. Now suppose that u, w ∈ S^⊥. Then, for any scalars a and b and any vector u ∈ S, we have

⟨av + bw, u⟩ = a ⟨v, u⟩ + b ⟨w, u⟩ = a₀ + b₀ = 0.

Thus a v + b w ∈ S^⊥ and therefore S^⊥ is a subspace of V.

The next theorem is a basic result in linear algebra.

Proof. Let λ be a scalar. Then

x = λv + (x - λv).

Notice that λ ∈ S. Thus we need to choose λ so that v is orthogonal to (x - λv). In other words, we want

0 = ⟨x-λv,v⟩ = ⟨x, v⟩ - λ ∥v∥².

The equation above shows that we should choose λ to ⟨x, v⟩/∥v∥². Making this choice of λ, we can write

$$\mathrm{x}=\frac{⟨\mathrm{x},\mathrm{v}⟩}{\Vert \mathrm{v}{\Vert }^2}\mathrm{v}+\left(\mathrm{x}-\frac{⟨\mathrm{x},\mathrm{v}⟩}{\Vert \mathrm{v}{\Vert }^2}\mathrm{v}\right).$$

4.5 Orthogonal sets and bases

A set S of vectors in V is called orthogonal if each pair of vectors in S are orthogonal, and S is called orthonormal if S is orthogonal and each vector in S has unit norm. In other words, S = {u₁, u₂, …, u_r} is orthogonal if

⟨ui, uj ⟩ = 0 for i ≠ j

and is orthonormal if

$$⟨{\mathbf{u}}_i,{\mathbf{u}}_j⟩={\delta }_{i,j}=\left\{\begin{array}{l}0,\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\text{ if }i\ne j\\ 1,\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\hspace{0.25em}\text{ if }i=j\end{array}\right.$$

Normalizing an orthogonal set S refers to the process of multiplying each vector in S by the reciprocal of its norm in order to transform S into an orthonormal set of vectors. The following theorems apply.

Proof. Let S = {u₁, u₂, …, u_r} be an orthogonal set. Suppose that the scalars a₁, a₂, …, a_r satisfy

a₁ u₁ + a₂ u₂ + … + a_r u_r = 0.

Computing the inner product of both sides of the above equation with ui for i = 1, 2, …, r, by the orthogonality of S we obtain

$$\begin{array}{r}0=⟨{\mathbf{u}}_i,\mathbf{0}⟩=⟨{\mathbf{u}}_i,a_1{\mathbf{u}}_1+a_2{\mathbf{u}}_2+\cdots +a_r{\mathbf{u}}_r⟩\\ =a_1⟨{\mathbf{u}}_i,{\mathbf{u}}_1⟩+\cdots +a_i⟨{\mathbf{u}}_i,{\mathbf{u}}_i⟩+\cdots +a_r⟨{\mathbf{u}}_i,{\mathbf{u}}_r⟩=a_i{‖{\mathbf{u}}_i‖}^2,\text{ for }i=1,2,\dots ,r\end{array}$$

This implies that a_i = 0 for i = 1, 2, …, r is the unique solution of the homogeneous equation. Therefore S is a linearly independent set.

Proof. Suppose that ⟨u, v⟩ = 0. Then

$$\Vert \mathbf{u}+\mathbf{v}{\Vert }^2=⟨\mathbf{u}+\mathbf{v},\mathbf{u}+\mathbf{v}⟩=⟨\mathbf{u},\mathbf{u}⟩+2⟨\mathbf{u},\mathbf{v}⟩+⟨\mathbf{v},\mathbf{v}⟩=⟨\mathbf{u},\mathbf{u}⟩+⟨\mathbf{v},\mathbf{v}⟩=\Vert \mathbf{u}{\Vert }^2+\Vert \mathbf{v}{\Vert }^2$$

which gives the desired result.

The most useful property of orthogonal bases is that the coordinates of any vector relative to an orthogonal basis can be computed directly one by one, without the need to solve a system of linear equations.

Suppose that {u₁, …, u_n} is an orthogonal basis for V. Then, for any v ∈ V,

Proof. Since {u₁, …, u_n} is a basis for V, any vector v ∈ V can be written as a linear combination of the basis; that is,

v = a₁u₁ +a₂u₂ + … + anu_n

for suitable scalars a₁, a₂, …, a_n. Using the orthogonality of the basis, we compute

$$\begin{array}{r}⟨\mathbf{v},{\mathbf{u}}_i⟩=⟨a_1{\mathbf{u}}_1+a_2{\mathbf{u}}_2+\cdots +a_n{\mathbf{u}}_n,{\mathbf{u}}_i⟩\\ =a_1⟨{\mathbf{u}}_1,{\mathbf{u}}_i⟩+\cdots +a_i⟨{\mathbf{u}}_i,{\mathbf{u}}_i⟩+\cdots +a_n⟨{\mathbf{u}}_n,{\mathbf{u}}_i⟩=a_i⟨{\mathbf{u}}_i,{\mathbf{u}}_i⟩,\text{ for }i=1,2,\dots ,n\end{array}$$

This implies that ai = ⟨v, u_i⟩/⟨u_i, u_i⟩ for i = 1, 2, …, n, which is the desired result.

The above scalars

$$a_i=\frac{⟨\mathbf{v},{\mathbf{u}}_i⟩}{⟨{\mathbf{u}}_i,{\mathbf{u}}_i⟩}=\frac{⟨\mathbf{v},{\mathbf{u}}_i⟩}{{‖{\mathbf{u}}_i‖}^2},\text{ for }i=1,2,\dots ,n$$

are called the Fourier coefficients of v with respect to {u₁, …, u_n}.

Consider a non zero vector w in an inner product space V. For any v ∈ V, we have showed that

$$\lambda =\frac{⟨\mathbf{v},\mathbf{w}⟩}{⟨\mathbf{w},\mathbf{w}⟩}=\frac{⟨\mathbf{v},\mathbf{w}⟩}{\Vert \mathbf{w}{\Vert }^2}$$

is the unique scalar such that $\widehat{\mathbf{v}}$ = v - λ w is orthogonal to w. The projection of v along w, denoted by proj_wv, is defined as

$${\mathrm{proj}}_{\mathbf{w}}\mathbf{v}=\frac{⟨\mathbf{v},\mathbf{w}⟩}{⟨\mathbf{w},\mathbf{w}⟩}\mathbf{w}=\frac{⟨\mathbf{v},\mathbf{w}⟩}{\Vert \mathbf{w}{\Vert }^2}\mathbf{w}$$

The above notion is generalized as follows.

Proof. We verify that $\widehat{\mathbf{v}}$ is orthogonal to u₁, u₂, …, u_r. For i = 1, 2, …, r, compute using the orthogonality of u₁, u₂, …, u_r,

$$\begin{array}{r}⟨\widehat{\mathbf{v}},{\mathbf{u}}_i⟩=⟨\mathbf{v}-\left(c_1{\mathbf{u}}_1+c_2{\mathbf{u}}_2+\cdots +c_r{\mathbf{u}}_r\right),{\mathbf{u}}_i⟩\\ =⟨\mathbf{v},{\mathbf{u}}_i⟩-c_1⟨{\mathbf{u}}_1,{\mathbf{u}}_i⟩-c_2⟨{\mathbf{u}}_2,{\mathbf{u}}_i⟩-\cdots -c_i⟨{\mathbf{u}}_i,{\mathbf{u}}_i⟩-\cdots -c_r⟨{\mathbf{u}}_r,{\mathbf{u}}_i⟩\\ =⟨\mathbf{v},{\mathbf{u}}_i⟩-c_i⟨{\mathbf{u}}_i,{\mathbf{u}}_i⟩\\ =⟨\mathbf{v},{\mathbf{u}}_i⟩-\frac{⟨\mathbf{v},{\mathbf{u}}_i⟩}{⟨{\mathbf{u}}_i,{\mathbf{u}}_i⟩}⟨{\mathbf{u}}_i,{\mathbf{u}}_i⟩=⟨\mathbf{v},{\mathbf{u}}_i⟩-⟨\mathbf{v},{\mathbf{u}}_i⟩=0\end{array}$$

Therefore $\widehat{\mathbf{v}}$ is orthogonal to u₁, u₂, …, ur.

Notice that each ci in the theorem above is the Fourier coefficient of v along the corresponding vector ui.

The notion of the projection of a vector v ∈ V along a subspace W of V is defined as follows. If W = Span{w₁, w₂, …, w_r} where the w_i form an orthogonal set, then the projection of v along W, denoted by proj_W v, is defined as

$$\begin{array}{r}{\mathbf{w}}_1={\mathbf{v}}_1,\\ {\mathbf{w}}_2={\mathbf{v}}_2-{\mathrm{proj}}_{{\mathbf{w}}_1}{\mathbf{v}}_2\text{, }\\ {\mathbf{w}}_3={\mathbf{v}}_3-{\mathrm{proj}}_{{\mathbf{w}}_1}{\mathbf{v}}_3-{\mathrm{proj}}_{{\mathbf{w}}_2}{\mathbf{v}}_3,\\ {\mathbf{w}}_n={\mathbf{v}}_n-{\mathrm{proj}}_{{\mathbf{w}}_1}{\mathbf{v}}_n-{\mathrm{proj}}_{{\mathbf{w}}_2}{\mathbf{v}}_n-\cdots -{\mathrm{proj}}_{{\mathbf{w}}_{n-1}}{\mathbf{v}}_n.\end{array}$$

4.6 Gram-Schmidt orthogonalization

Suppose that {v₁, v₂, …, v_n} is a basis of an inner product space V. One can use this to construct an orthogonal basis {w₁, w₂, …, w_n} of V as follows. Set

$$\begin{array}{r}{\mathbf{w}}_1={\mathbf{v}}_1\\ {\mathbf{w}}_2={\mathbf{v}}_2-{\mathrm{proj}}_{{\mathbf{w}}_1}{\mathbf{v}}_2\\ {\mathbf{w}}_3={\mathbf{v}}_3-{\mathrm{proj}}_{{\mathbf{w}}_1}{\mathbf{v}}_3-{\mathrm{proj}}_{{\mathbf{w}}_2}{\mathbf{v}}_3\\ \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots {\mathrm{proj}}_{{\mathbf{w}}_{n-1}}{\mathbf{v}}_n.\end{array}$$

In other words, for k = 2, 3, …, n, we define

$${\mathbf{w}}_k={\mathbf{v}}_k-{\mathrm{proj}}_{W_{k-1}}{\mathbf{v}}_k,\text{ where }{W}_{k-1}=\mathrm{Span}\left\{{\mathbf{w}}_1,{\mathbf{w}}_2,\dots ,{\mathbf{w}}_{k-1}\right\}.$$

By Theorem 4.7, each w_k is orthogonal to the preceding w’s. Thus, {w₁, w₂, …, w_n} forms an orthogonal basis for V as claimed. Normalizing will then yield an orthonormal basis for V.

The above construction is known as the Gram-Schmidt orthogonalization process. Each vector w_k is a linear combination of vk and the preceding w’s. Hence, it can easily be shown, by induction, that each w_k is a linear combination of v₁, v₂, …, v_k. Because taking multiples of vectors does not affect orthogonality, it may be simpler in hand calculations to clear fractions in any new w_k, by multiplying w_k by an appropriate scalar, before obtaining the next w_k+1.

Suppose u₁, u₂, …, u_r are linearly independent, and so they form a basis for

U = Span{u₁, u₂, …, u_r}.

Applying Gram-Schmidt orthogonalization to the u’s yields an orthogonal basis for U. The following theorems use the above algorithm and remarks.

Proof. Since S is a linearly independent set in V, we can extend S to a basis of V. That is, we can find vectors ur+1, …, u_n such that {w₁, …, wr, u_r+1, … u_n} is a basis for V. For k = r + 1, …, n, we define

$${\mathbf{w}}_k={\mathbf{u}}_k-{\mathrm{proj}}_{W_{k-1}}{\mathbf{u}}_k,\text{ where }{W}_{k-1}=\mathrm{Span}\left\{{\mathbf{w}}_1,{\mathbf{w}}_2,\dots ,{\mathbf{w}}_{k-1}\right\}$$

This is just the Gram-Schmidt orthogonalization process applied to {w₁, …, w_r, u_r+1, … u_n}, which yields an orthogonal basis for V.

Proof. Let {u₁, …, u_r} be a basis for W. Applying the Gram-Schmidt orthogonalization process, we obtain an orthogonal basis for W: S = {w₁, …, w_r}. Using S, for every v ∈ V, we can write

v = proj_W v + (v − proj_W v).

Clearly, proj_W v ∈ W. By Theorem 4.7, (v − proj_W v) is orthogonal to every vector in S and, thus, to every vector in W. Therefore, (v − proj_W v) ∈ W^⊥.

Suppose that we can write

$$\mathbf{v}=\mathbf{w}+\widehat{\mathbf{w}},\mathbf{w}\in W,\widehat{\mathbf{w}}\in W^{\perp }.$$

Then subtracting the above expression from v = proj_W v + (v − proj_W v), we get

$$\mathbf{0}=\left({\mathrm{proj}}_W\mathbf{v}-\mathbf{w}\right)+\left(\mathbf{v}-{\mathrm{proj}}_W\mathbf{v}-\widehat{\mathbf{w}}\right)$$

Observe that (proj_W v − w) ∈ W and (v − proj_Wv − $\widehat{W}$) ∈ W^⊥. Therefore,

$$\begin{array}{r}0=⟨\mathbf{0},{\mathrm{proj}}_W\mathbf{v}-\mathbf{w}⟩=⟨\left({\mathrm{proj}}_W\mathbf{v}-\mathbf{w}\right)+\left(\mathbf{v}-{\mathrm{proj}}_W\mathbf{v}-\widehat{\mathbf{w}}\right),{\mathrm{proj}}_W\mathbf{v}-\mathbf{w}⟩\\ =⟨{\mathrm{proj}}_W\mathbf{v}-\mathbf{w},{\mathrm{proj}}_W\mathbf{v}-\mathbf{w}⟩.\end{array}$$

This implies that proj_Wv − w = 0 or, equivalently, w = proj_W v. Similarly,

$$\begin{array}{r}0=⟨\mathbf{0},\mathbf{v}-{\mathrm{proj}}_W\mathbf{v}-\widehat{\mathbf{w}}⟩=⟨\left({\mathrm{proj}}_W\mathbf{v}-\mathbf{w}\right)+\left(\mathbf{v}-{\mathrm{proj}}_W\mathbf{v}-\widehat{\mathbf{w}}\right),\mathbf{v}-{\mathrm{proj}}_W\mathbf{v}-\widehat{\mathbf{w}}⟩\\ =⟨\mathbf{v}-{\mathrm{proj}}_W\mathbf{v}-\widehat{\mathbf{w}},\mathbf{v}-{\mathrm{proj}}_W\mathbf{v}-\widehat{\mathbf{w}}⟩\end{array}$$

Therefore, v - proj_Wv - w = 0 or, equivalently, w = v - proj_W v. This proves that each v ∈ V is expressed uniquely as v = proj_W v + (v - proj_W v).

4.7 Orthogonal projections and minimization

Suppose that U is a subspace of an inner product space V. Then each vector v ∈ V can be written uniquely in the form

v = u + w,

where u ∈ U and w ∈ U^⊥. We use this decomposition to define a linear mapping on V, denoted by proj_U, called the orthogonal projection of V along U. For v ∈ V, we define proj_Uv to be the vector u in the decomposition above.

The orthogonal projection along a subspace satisfies the following properties.

Proof. 1. If u ∈ U, then u = proj_Uu. Therefore, u ∈ range proj_U. Conversely, if u ∈ range proj_U, then by definition, u ∈ U. Altogether, we have U ⊆ range proj_U and range proj_U ⊆ U, which implies range proj_U = U.

2. If u ∈ U^⊥, then it is expressed uniquely as u = 0+u since it has no component in U other than the zero vector. Then proj_U u = 0 which means that u ∈ ker proj_U. Conversely, if u ∈ ker proj_U then proj_U u = 0. But u is expressed uniquely as u = 0+w where w ∈ U^⊥. It follows that u ∈ U^⊥. Altogether, we have U^⊥ ⊆ ker proj_U and ker proj_U ⊆ U^⊥, which implies ker proj_U = U^⊥.

3. Since v can be expressed as v = u + w where w ∈ U^⊥ and, by definition, u = proj_U v. It follows that w = v - proj_U v and, thus, v - proj_U v ∈ U^⊥.

4. Consider any vector v ∈ V. Then, by the first and second statements above, it can be written as v = proj_U v + w where w ∈ ker proj_U. Therefore, applying the orthogonal projection to both sides of this equation, we get

$${\mathrm{proj}}_U\mathbf{v}={\mathrm{proj}}_U^2\mathbf{v}+{\mathrm{proj}}_U\mathbf{w}={\mathrm{proj}}_U^2\mathbf{v}$$

But this is true for any v ∈ V. Then, ${\mathrm{proj}}_U^2={\mathrm{proj}}_U$.

5. For every v ∈ V, by Pythagoras,

$${‖{\mathrm{proj}}_U\mathbf{v}‖}^2⩽{‖{\mathrm{proj}}_U\mathbf{v}‖}^2+{‖\mathbf{v}-{\mathrm{proj}}_U\mathbf{v}‖}^2=\Vert \mathbf{v}{\Vert }^2.$$

Taking the positive square root of both sides, we get $‖{\mathrm{proj}}_U\mathbf{v}‖⩽\Vert \mathbf{v}\Vert$.

By the definition of orthogonal projection, if U = Span{u₁, u₂, …, u_r} where the ui form an orthogonal basis, then

$${\mathrm{proj}}_U\mathbf{v}=\frac{⟨\mathbf{v},{\mathbf{u}}_1⟩}{⟨{\mathbf{u}}_1,{\mathbf{u}}_1⟩}{\mathbf{u}}_1+\frac{⟨\mathbf{v},{\mathbf{u}}_2⟩}{⟨{\mathbf{u}}_2,{\mathbf{u}}_2⟩}{\mathbf{u}}_2+\cdots +\frac{⟨\mathbf{v},{\mathbf{u}}_r⟩}{⟨{\mathbf{u}}_r,{\mathbf{u}}_r⟩}{\mathbf{u}}_r.$$

The following problem often arises: given a subspace U of V and a vector v ∈ V, find a vector u ∈ U such that ∥v - u∥ is as small as possible. The next theorem shows that this minimization problem is solved by taking u = proj_Uv.

Proof. Suppose that u ∈ U. Then

$$\begin{array}{r}{‖\mathbf{v}-{\mathrm{proj}}_U\mathbf{v}‖}^2\le {‖\mathbf{v}-{\mathrm{proj}}_U\mathbf{v}‖}^2+{‖{\mathrm{proj}}_U\mathbf{v}-\mathbf{u}‖}^2\\ ={‖\left(\mathbf{v}-{\mathrm{proj}}_U\mathbf{v}\right)+\left({\mathrm{proj}}_U\mathbf{v}-\mathbf{u}\right)‖}^2\\ =\Vert \mathbf{v}-\mathbf{u}{\Vert }^2\end{array}$$