Linear Algebra Home

## Inner Product Spaces

Definition. Let $$V$$ be a real vector space. An inner product on $$V$$, denoted by $$\langle\cdot,\cdot\rangle$$, is a function from $$V\times V$$ to $$\mathbb R$$ for which the following hold for all $$\overrightarrow{u},\overrightarrow{v},\overrightarrow{w}\in V$$ and $$c,d\in \mathbb R$$:

1. $$\langle\overrightarrow{u},\overrightarrow{v}\rangle=\langle\overrightarrow{v},\overrightarrow{u}\rangle$$. (symmetry)

2. $$\langle c\overrightarrow{u}+d\overrightarrow{v},\overrightarrow{w}\rangle =c\langle\overrightarrow{u},\overrightarrow{w}\rangle+d\langle\overrightarrow{v},\overrightarrow{w}\rangle$$. (linearity)

3. $$\langle\overrightarrow{u},\overrightarrow{u}\rangle\geq 0$$ where $$\langle\overrightarrow{u},\overrightarrow{u}\rangle=0$$ if and only if $$\overrightarrow{u}=\overrightarrow{0}$$. (nonnegativity)

A real vector space with an inner product defined on it is called a real inner product space.

Example.

1. The real vector space $$\mathbb R^n$$ is a real inner product space with the standard inner product or the dot product: $\langle\overrightarrow{u},\overrightarrow{v}\rangle= \overrightarrow{u}\cdot \overrightarrow{v} = \overrightarrow{u}^T\overrightarrow{v}.$ We call $$\mathbb R^n$$ as the $$n$$-dimensional Euclidean space.

2. Consider the real vector space $$\ell^2(\mathbb R)$$ of square-summable real sequences: $\ell^2(\mathbb R)=\{\overrightarrow{a}=(a_1,a_2,a_3,\ldots)\in \mathbb R^{\infty} \;|\; \sum_{n=1}^{\infty} a_n^2 < \infty\}.$ $$\ell^2(\mathbb R)$$ is a real inner product space with the following inner product: $\langle\overrightarrow{a},\overrightarrow{b}\rangle= \sum_{n=1}^{\infty} a_nb_n.$

3. The real vector space $$C[0,1]$$ of all continuous real-valued functions on $$[0,1]$$ is a real inner product space with the following inner product: $\langle f,g\rangle= \int_0^1 f(x)g(x)\,dx.$ To show the above is an inner product, let $$f,g,h\in C[0,1]$$ and $$c,d\in \mathbb R$$.
1. $$\langle f,g\rangle=\int_0^1 f(x)g(x)\,dx=\int_0^1 g(x)f(x)\,dx=\langle g,f\rangle$$. (symmetry)

2. $$\langle cf+dg,h\rangle =\int_0^1 (cf(x)+dg(x))h(x)\,dx =c\int_0^1 f(x)h(x)\,dx+d\int_0^1 g(x)h(x)\,dx =c\langle f,h\rangle+d\langle g,h\rangle$$. (linearity)

3. $$\langle f,f\rangle=\int_0^1 (f(x))^2\,dx\geq 0$$ where $$\langle f,f\rangle=\int_0^1 (f(x))^2\,dx=0$$ if and only if $$f=0$$. (nonnegativity)

Definition. Let $$\overrightarrow{u}$$ and $$\overrightarrow{v}$$ be in a real inner product space $$V$$. The length or norm of $$\overrightarrow{v}$$, denoted by $$\left\lVert\overrightarrow{v}\right\rVert$$, is defined by $\left\lVert\overrightarrow{v}\right\rVert=\sqrt{\langle\overrightarrow{v},\overrightarrow{v}\rangle}.$ $$\overrightarrow{v}\in V$$ is a unit vector if $$\left\lVert\overrightarrow{v}\right\rVert=1$$. The distance between $$\overrightarrow{u},\overrightarrow{v}$$, denoted by $$\operatorname{d}(\overrightarrow{u},\overrightarrow{v})$$, is defined by $\operatorname{d}(\overrightarrow{u},\overrightarrow{v})=\left\lVert\overrightarrow{u}-\overrightarrow{v}\right\rVert.$ Theorem. The following are true for all vectors $$\overrightarrow{u}$$ and $$\overrightarrow{v}$$ of a real inner product space $$V$$ and for all scalars $$c$$ in $$\mathbb R$$.

1. $$\left\lVert\overrightarrow{v}\right\rVert^2=\langle\overrightarrow{v},\overrightarrow{v}\rangle$$.

2. $$\left\lVert c\overrightarrow{v}\right\rVert=|c| \left\lVert\overrightarrow{v}\right\rVert$$.

3. Triangle inequality: $$\left\lVert\overrightarrow{u}+\overrightarrow{v}\right\rVert \leq \left\lVert\overrightarrow{u}\right\rVert+ \left\lVert\overrightarrow{v}\right\rVert$$.

4. Parallelogram law: $$\left\lVert\overrightarrow{u}+\overrightarrow{v}\right\rVert^2 =2\left\lVert\overrightarrow{u}\right\rVert^2+ 2\left\lVert\overrightarrow{v}\right\rVert^2$$.

5. Cauchy-Schwarz inequality: $$|\langle\overrightarrow{u},\overrightarrow{v}\rangle| \leq \left\lVert\overrightarrow{u}\right\rVert \left\lVert\overrightarrow{v}\right\rVert$$ where the equality holds if and only if $$\{\overrightarrow{u},\overrightarrow{v} \}$$ is linearly dependent.

Exercise.

Example. Find the distance between $$f(x)=x$$ and $$g(x)=x^2$$ in the real inner product space $$C[0,1]$$.

Solution. \begin{align*} \operatorname{d}(f,g) =\left\lVert f-g\right\rVert &=\sqrt{\langle f-g,f-g\rangle}\\ &= \sqrt{\int_0^1 (f(x)-g(x))^2\,dx}\\ &= \sqrt{\int_0^1 (x-x^2)^2\,dx}\\ &= \sqrt{\int_0^1 (x^2-2x^3+x^4)\,dx}\\ &= \frac{1}{\sqrt{30}} \end{align*}

Definition. Two vectors $$\overrightarrow{u}$$ and $$\overrightarrow{v}$$ of a real inner product space $$V$$ are orthogonal if $\langle\overrightarrow{u},\overrightarrow{v}\rangle=0.$ Example. Show that $$f(x)=\sin(\pi x)$$ and $$g(x)=\cos(\pi x)$$ are orthogonal in the real inner product space $$C[0,1]$$.

Solution. \begin{align*} \langle f,g\rangle &= \int_0^1 f(x)g(x)\,dx\\ &= \int_0^1 \sin(\pi x) \cos(\pi x) \,dx\\ &= \frac{1}{2}\int_0^1 \sin(2\pi x) \,dx\\ &= -\frac{1}{4\pi} \left. \cos(2\pi x) \right|_0^1\\ &= 0 \end{align*} Since $$\langle f,g\rangle=0$$, $$f(x)=\sin(\pi x)$$ and $$g(x)=\cos(\pi x)$$ are orthogonal in the real inner product space $$C[0,1]$$. Alternatively we can show it by Pythagorean Theorem.

Theorem.(Pythagorean Theorem) Two vectors $$\overrightarrow{u}$$ and $$\overrightarrow{v}$$ of a real inner product space $$V$$ are orthogonal if and only if $$\left\lVert\overrightarrow{v}+\overrightarrow{v}\right\rVert^2 =\left\lVert\overrightarrow{u}\right\rVert^2+\left\lVert\overrightarrow{v}\right\rVert^2$$.

Similar to the proof of Pythagorean Theorem in Orthogonal Vectors in $$\mathbb R^n$$.

Definition. The angle $$\theta$$ between two vectors $$\overrightarrow{u}$$ and $$\overrightarrow{v}$$ of a real inner product space $$V$$ is the angle in $$[0,\pi]$$ satisfying $\langle\overrightarrow{u},\overrightarrow{v}\rangle =\left\lVert\overrightarrow{u}\right\rVert \left\lVert\overrightarrow{v}\right\rVert \cos \theta.$ Definition. Let $$W$$ be a subspace of a real inner product space $$V$$. A vector $$\overrightarrow{v}\in V$$ is orthogonal to $$W$$ if $$\langle\overrightarrow{v},\overrightarrow{w}\rangle =0$$ for all $$\overrightarrow{w}\in W$$. The orthogonal complement of $$W$$, denoted by $$W^{\perp}$$, is the set of all vectors in $$V$$ that are orthogonal to $$W$$, i.e., $W^{\perp}=\{\overrightarrow{v}\in V \;|\; \langle\overrightarrow{v},\overrightarrow{w}\rangle =0 \text{ for all } \overrightarrow{w}\in W\}.$

Theorem. Let $$W$$ be a subspace of a real inner product space $$V$$. Then

1. $$\overrightarrow{v} \in W^{\perp}$$ if and only if $$\overrightarrow{v}$$ is orthogonal to each vector $$\overrightarrow{w}$$ of a basis of $$W$$.

2. $$W^{\perp}$$ is a subspace of $$V$$.

3. $$W\subseteq (W^{\perp})^{\perp}$$ where the equality holds for finite dimensional $$W$$.

4. $$W\cap W^{\perp}=\{\overrightarrow{0_V}\}$$.

Similar to the proof of the third theorem in Orthogonal Vectors in $$\mathbb R^n$$.

Example. The real vector space $$P_1[0,1]$$ of all polynomials of degree at most 1 on $$[0,1]$$ is a real inner product space with the following inner product: $\langle f,g\rangle= \int_0^1 f(x)g(x)\,dx.$ Find $$W^{\perp}$$ for $$W=\operatorname{Span}\{x\}$$.

Solution. First note that $$\{x\}$$ is a basis of $$W=\operatorname{Span}\{x\}$$. Let $$p(x)=a+bx \in W^{\perp}$$. Then \begin{align*} 0=\langle p,x\rangle &= \int_0^1 (a+bx)x \,dx\\ &= \left. \frac{ax^2}{2}+\frac{bx^3}{3} \right|_0^1\\ &= \frac{a}{2}+\frac{b}{3}. \end{align*} Then $$b=-\frac{3a}{2}$$ and $$p(x)=a-\frac{3a}{2}x=\frac{a}{2}(2-3x)$$. Thus $W^{\perp}=\left\lbrace \frac{a}{2}(2-3x)\;|\; a\in \mathbb R\right\rbrace=\operatorname{Span}\{2-3x\}.$ We can verify that $$(W^{\perp})^{\perp}=W$$ and $$W\cap W^{\perp}=\{0\}$$.

Definition. A subset $$\{\overrightarrow{v_1},\overrightarrow{v_2},\ldots,\overrightarrow{v_k}\}$$ of a real inner product space $$V$$ is called an orthogonal set if $$\langle\overrightarrow{v_i},\overrightarrow{v_j}\rangle=0$$ for all distinct $$i,j=1,2,\ldots,k$$. Also $$\{\overrightarrow{v_1},\overrightarrow{v_2},\ldots,\overrightarrow{v_k}\}$$ is called an orthonormal set if it is an orthogonal set of unit vectors.

Theorem. If $$\{\overrightarrow{v_1},\overrightarrow{v_2},\ldots,\overrightarrow{v_k}\}$$ is an orthogonal set of nonzero vectors in a real inner product space $$V$$, then $$\{\overrightarrow{v_1},\overrightarrow{v_2},\ldots,\overrightarrow{v_k}\}$$ is linearly independent and consequently forms a basis of $$\operatorname{Span} \{\overrightarrow{v_1},\overrightarrow{v_2},\ldots,\overrightarrow{v_k}\}$$.

Similar to the proof of the first theorem in in Orthogonal Bases and Matrices in $$\mathbb R^n$$.

Definition. Let $$W$$ be a subspace of a real inner product space $$V$$. An orthogonal basis of $$W$$ is a basis of $$W$$ that is an orthogonal set. Similarly an orthonormal basis of $$W$$ is a basis of $$W$$ that is an orthonormal set.

Theorem. Let $$W$$ be a subspace of a real inner product space $$V$$ and $$\{\overrightarrow{w_1},\overrightarrow{w_2},\ldots,\overrightarrow{w_k}\}$$ is an orthogonal basis of $$W$$. If $$\overrightarrow{v}\in W$$, then $\overrightarrow{v}=\frac{\langle\overrightarrow{v},\overrightarrow{w_1}\rangle}{\langle\overrightarrow{w_1},\overrightarrow{w_1}\rangle}\overrightarrow{w_1} +\frac{\langle\overrightarrow{v},\overrightarrow{w_2}\rangle}{\langle\overrightarrow{w_2},\overrightarrow{w_2}\rangle}\overrightarrow{w_2} +\cdots+ \frac{\langle\overrightarrow{v},\overrightarrow{w_k}\rangle}{\langle\overrightarrow{w_k},\overrightarrow{w_k}\rangle}\overrightarrow{w_k}.$

Similar to the proof of the second theorem in in Orthogonal Bases and Matrices in $$\mathbb R^n$$.

Theorem.(Orthogonal Decomposition Theorem) Let $$W$$ be a subspace of a real inner product space $$V$$ and $$\overrightarrow{y}\in V$$. Then $\overrightarrow{y}=\overrightarrow{w}+\overrightarrow{z}$ for unique vectors $$\overrightarrow{w}\in W$$ and $$\overrightarrow{z}\in W^{\perp}$$. Moreover, if $$\{\overrightarrow{w_1},\overrightarrow{w_2},\ldots,\overrightarrow{w_k}\}$$ is an orthogonal basis of $$W$$, then $\overrightarrow{w}=\frac{\langle\overrightarrow{y},\overrightarrow{w_1}\rangle}{\langle\overrightarrow{w_1},\overrightarrow{w_1}\rangle}\overrightarrow{w_1} +\frac{\langle\overrightarrow{y},\overrightarrow{w_2}\rangle}{\langle\overrightarrow{w_2},\overrightarrow{w_2}\rangle}\overrightarrow{w_2} +\cdots+ \frac{\langle\overrightarrow{y},\overrightarrow{w_k}\rangle}{\langle\overrightarrow{w_k},\overrightarrow{w_k}\rangle}\overrightarrow{w_k} \text{ and } \overrightarrow{z}=\overrightarrow{y}-\overrightarrow{w}.$

Similar to the proof of the Orthogonal Decomposition Theorem in Orthogonal Projections in $$\mathbb R^n$$.

Definition. Let $$W$$ be a subspace of a real inner product space $$V$$. Each vector $$\overrightarrow{y}\in V$$ can be uniquely written as $$\overrightarrow{y}=\overrightarrow{w}+\overrightarrow{z}$$ where $$\overrightarrow{w}\in W$$ and $$\overrightarrow{z}\in W^{\perp}$$. The unique vector $$\overrightarrow{w}\in W$$ is called the orthogonal projection of $$\overrightarrow{y}$$ onto $$W$$ and it is denoted by $$\operatorname{proj}_W \overrightarrow{y}$$.

Corollary. Let $$W$$ be a subspace of a real inner product space $$V$$ with an orthonormal basis $$\{\overrightarrow{w_1},\overrightarrow{w_2},\ldots,\overrightarrow{w_k}\}$$. Then for each $$\overrightarrow{y}\in V$$, $\operatorname{proj}_W \overrightarrow{y}= \langle\overrightarrow{y},\overrightarrow{w_1}\rangle \overrightarrow{w_1} +\langle\overrightarrow{y},\overrightarrow{w_2}\rangle \overrightarrow{w_2} +\cdots+ \langle\overrightarrow{y},\overrightarrow{w_k}\rangle \overrightarrow{w_k}.$
Theorem.(Best Approximation Theorem) Let $$W$$ be a subspace of a real inner product space $$V$$ and $$\overrightarrow{b}\in V$$. Then $\min_{\overrightarrow{w}\in W}\left\lVert\overrightarrow{b}-\overrightarrow{w}\right\rVert =\left\lVert\overrightarrow{b}-\operatorname{proj}_W \overrightarrow{b}\right\rVert.$

Similar to the proof of the Best Approximation Theorem in Orthogonal Projections in $$\mathbb R^n$$.

Theorem.(Gram-Schmidt Process) Let $$W$$ be a subspace of a real inner product space $$V$$ with a basis $$\{\overrightarrow{w_1},\overrightarrow{w_2},\ldots,\overrightarrow{w_k}\}$$. There is an orthogonal basis $$\{\overrightarrow{v_1},\overrightarrow{v_2},\ldots,\overrightarrow{v_k}\}$$ of $$W$$ where $\overrightarrow{v_1}=\overrightarrow{w_1} \text{ and } \overrightarrow{v_i}=\overrightarrow{w_i} -\sum_{j=1}^{i-1} \frac{\langle\overrightarrow{w_i},\overrightarrow{v_j}\rangle}{\langle\overrightarrow{v_j},\overrightarrow{v_j}\rangle} \overrightarrow{v_j}, \; i=2,3,\ldots,k.$ Moreover, $$\operatorname{Span} \{\overrightarrow{v_1},\overrightarrow{v_2},\ldots,\overrightarrow{v_i}\} =\operatorname{Span} \{\overrightarrow{w_1},\overrightarrow{w_2},\ldots,\overrightarrow{w_i}\}$$ for $$i=1,2,\ldots,k$$.

Similar to the proof of Gram-Schmidt Process in Gram-Schmidt Process in $$\mathbb R^n$$.

Example Consider the real inner product space $$C[0,1]$$ with the following inner product: $\langle f,g\rangle= \int_0^1 f(x)g(x)\,dx.$

1. Find an orthogonal basis of the subspace $$W=\operatorname{Span}\{1,x,x^2\}$$ of $$C[0,1]$$.

2. Find the best approximation of $$e^x\in C[0,1]$$ by a polynomial in $$W$$.

Solution.

1. We use Gram-Schmidt Process. First note that $$\{1,x,x^2\}$$ is a basis of $$W$$. Now we find a vector in $$W$$ that is orthogonal to $$1$$: $x-\frac{\langle x,1\rangle}{\langle 1,1\rangle} 1 =x-\frac{\int_0^1 x\cdot 1\,dx}{\int_0^1 1^2\,dx}=x-\frac{1}{2}$ Next we find a vector in $$W$$ that is orthogonal to both $$1$$ and $$x-\frac{1}{2}$$: \begin{align*} & x^2-\frac{\langle x^2,1\rangle}{\langle 1,1\rangle} 1 - \frac{\langle x^2,x-\frac{1}{2}\rangle}{\langle x-\frac{1}{2},x-\frac{1}{2}\rangle} \left(x-\frac{1}{2} \right)\\ =\;& x^2 -\frac{\int_0^1 x^2\cdot 1\,dx}{\int_0^1 1^2\,dx} -\frac{\int_0^1 x^2\left(x-\frac{1}{2}\right)\,dx}{\int_0^1 \left(x-\frac{1}{2}\right)^2\,dx}\left(x-\frac{1}{2} \right)\\ =\;& x^2-\frac{\frac{1}{3}}{1}-\frac{\frac{1}{12}}{\frac{1}{12}}\left(x-\frac{1}{2} \right)\\ =\;& x^2-x+\frac{1}{6} \end{align*} Thus $$\{ 1,x-\frac{1}{2},x^2-x+\frac{1}{6} \}$$ is an orthogonal basis of $$W=\operatorname{Span}\{1,x,x^2\}$$.

2. The best approximation of $$e^x\in C[0,1]$$ by a polynomial in $$W$$ is \begin{align*} \operatorname{proj}_{W} e^x=\;&\frac{\langle e^x,1\rangle}{\langle 1,1\rangle} 1 + \frac{\langle e^x,x-\frac{1}{2}\rangle}{\langle x-\frac{1}{2},x-\frac{1}{2}\rangle} \left(x-\frac{1}{2} \right) +\frac{\langle e^x,x^2-x+\frac{1}{6}\rangle}{\langle x^2-x+\frac{1}{6},x^2-x+\frac{1}{6}\rangle}\left(x^2-x+\frac{1}{6}\right)\\ =\;&\frac{\int_0^1 e^x\cdot 1\,dx}{\int_0^1 1^2\,dx} +\frac{\int_0^1 e^x\left(x-\frac{1}{2}\right)\,dx}{\int_0^1 \left(x-\frac{1}{2}\right)^2\,dx}\left(x-\frac{1}{2} \right) +\frac{\int_0^1 e^x\left(x^2-x+\frac{1}{6}\right)\,dx}{\int_0^1 \left(x^2-x+\frac{1}{6}\right)^2\,dx}\left(x^2-x+\frac{1}{6}\right)\\ =\;& \frac{(e-1)}{1}+\frac{(3-e)/2}{1/12}\left(x-\frac{1}{2} \right)+\frac{(7e-19)/6}{1/180} \left(x^2-x+\frac{1}{6} \right)\\ =\;&(e-1)+6(3-e) \left(x-\frac{1}{2} \right)+30(7e-19) \left(x^2-x+\frac{1}{6} \right)\\ =\;&(210e-570)x^2+(-216e+588)x+39e-105. \end{align*}

Last edited