Definition. Let \(V\) be a real vector space. An inner product on \(V\), denoted by \(\langle\cdot,\cdot\rangle\), is a function from \(V\times V\) to \(\mathbb R\) for which the following hold for all \(\overrightarrow{u},\overrightarrow{v},\overrightarrow{w}\in V\) and \(c,d\in \mathbb R\):

1. \(\langle\overrightarrow{u},\overrightarrow{v}\rangle=\langle\overrightarrow{v},\overrightarrow{u}\rangle\). (symmetry)


2. \(\langle c\overrightarrow{u}+d\overrightarrow{v},\overrightarrow{w}\rangle =c\langle\overrightarrow{u},\overrightarrow{w}\rangle+d\langle\overrightarrow{v},\overrightarrow{w}\rangle\). (linearity)


3. \(\langle\overrightarrow{u},\overrightarrow{u}\rangle\geq 0\) where \(\langle\overrightarrow{u},\overrightarrow{u}\rangle=0\) if and only if \(\overrightarrow{u}=\overrightarrow{0}\). (nonnegativity)

A real vector space with an inner product defined on it is called a real inner product space.

Example.

1. The real vector space \(\mathbb R^n\) is a real inner product space with the standard inner product or the dot product: \[\langle\overrightarrow{u},\overrightarrow{v}\rangle= \overrightarrow{u}\cdot \overrightarrow{v} = \overrightarrow{u}^T\overrightarrow{v}.\] We call \(\mathbb R^n\) as the \(n\)-dimensional Euclidean space.


2. Consider the real vector space \(\ell^2(\mathbb R)\) of square-summable real sequences: \[\ell^2(\mathbb R)=\{\overrightarrow{a}=(a_1,a_2,a_3,\ldots)\in \mathbb R^{\infty} \;|\; \sum_{n=1}^{\infty} a_n^2 < \infty\}.\] \(\ell^2(\mathbb R)\) is a real inner product space with the following inner product: \[\langle\overrightarrow{a},\overrightarrow{b}\rangle= \sum_{n=1}^{\infty} a_nb_n.\]


3. The real vector space \(C[0,1]\) of all continuous real-valued functions on \([0,1]\) is a real inner product space with the following inner product: \[\langle f,g\rangle= \int_0^1 f(x)g(x)\,dx.\] To show the above is an inner product, let \(f,g,h\in C[0,1]\) and \(c,d\in \mathbb R\).
   1. \(\langle f,g\rangle=\int_0^1 f(x)g(x)\,dx=\int_0^1 g(x)f(x)\,dx=\langle g,f\rangle\). (symmetry)
   
   
   2. \(\langle cf+dg,h\rangle =\int_0^1 (cf(x)+dg(x))h(x)\,dx =c\int_0^1 f(x)h(x)\,dx+d\int_0^1 g(x)h(x)\,dx =c\langle f,h\rangle+d\langle g,h\rangle\). (linearity)
   
   
   3. \(\langle f,f\rangle=\int_0^1 (f(x))^2\,dx\geq 0\) where \(\langle f,f\rangle=\int_0^1 (f(x))^2\,dx=0\) if and only if \(f=0\). (nonnegativity)

Definition. Let \(\overrightarrow{u}\) and \(\overrightarrow{v}\) be in a real inner product space \(V\). The length or norm of \(\overrightarrow{v}\), denoted by \(\left\lVert\overrightarrow{v}\right\rVert\), is defined by \[\left\lVert\overrightarrow{v}\right\rVert=\sqrt{\langle\overrightarrow{v},\overrightarrow{v}\rangle}.\] \(\overrightarrow{v}\in V\) is a unit vector if \(\left\lVert\overrightarrow{v}\right\rVert=1\). The distance between \(\overrightarrow{u},\overrightarrow{v}\), denoted by \(\operatorname{d}(\overrightarrow{u},\overrightarrow{v})\), is defined by \[\operatorname{d}(\overrightarrow{u},\overrightarrow{v})=\left\lVert\overrightarrow{u}-\overrightarrow{v}\right\rVert.\] Theorem. The following are true for all vectors \(\overrightarrow{u}\) and \(\overrightarrow{v}\) of a real inner product space \(V\) and for all scalars \(c\) in \(\mathbb R\).

1. \(\left\lVert\overrightarrow{v}\right\rVert^2=\langle\overrightarrow{v},\overrightarrow{v}\rangle\).


2. \(\left\lVert c\overrightarrow{v}\right\rVert=|c| \left\lVert\overrightarrow{v}\right\rVert\).


3. Triangle inequality: \(\left\lVert\overrightarrow{u}+\overrightarrow{v}\right\rVert \leq \left\lVert\overrightarrow{u}\right\rVert+ \left\lVert\overrightarrow{v}\right\rVert\).


4. Parallelogram law: \(\left\lVert\overrightarrow{u}+\overrightarrow{v}\right\rVert^2 =2\left\lVert\overrightarrow{u}\right\rVert^2+ 2\left\lVert\overrightarrow{v}\right\rVert^2\).


5. Cauchy-Schwarz inequality: \(|\langle\overrightarrow{u},\overrightarrow{v}\rangle| \leq \left\lVert\overrightarrow{u}\right\rVert \left\lVert\overrightarrow{v}\right\rVert\) where the equality holds if and only if \(\{\overrightarrow{u},\overrightarrow{v} \}\) is linearly dependent.

Example. Find the distance between \(f(x)=x\) and \(g(x)=x^2\) in the real inner product space \(C[0,1]\).

Solution. \[\begin{align*} \operatorname{d}(f,g) =\left\lVert f-g\right\rVert &=\sqrt{\langle f-g,f-g\rangle}\\ &= \sqrt{\int_0^1 (f(x)-g(x))^2\,dx}\\ &= \sqrt{\int_0^1 (x-x^2)^2\,dx}\\ &= \sqrt{\int_0^1 (x^2-2x^3+x^4)\,dx}\\ &= \frac{1}{\sqrt{30}} \end{align*}\]

Definition. Two vectors \(\overrightarrow{u}\) and \(\overrightarrow{v}\) of a real inner product space \(V\) are orthogonal if \[\langle\overrightarrow{u},\overrightarrow{v}\rangle=0.\] Example. Show that \(f(x)=\sin(\pi x)\) and \(g(x)=\cos(\pi x)\) are orthogonal in the real inner product space \(C[0,1]\).

Solution. \[\begin{align*} \langle f,g\rangle &= \int_0^1 f(x)g(x)\,dx\\ &= \int_0^1 \sin(\pi x) \cos(\pi x) \,dx\\ &= \frac{1}{2}\int_0^1 \sin(2\pi x) \,dx\\ &= -\frac{1}{4\pi} \left. \cos(2\pi x) \right|_0^1\\ &= 0 \end{align*}\] Since \(\langle f,g\rangle=0\), \(f(x)=\sin(\pi x)\) and \(g(x)=\cos(\pi x)\) are orthogonal in the real inner product space \(C[0,1]\). Alternatively we can show it by Pythagorean Theorem.

Theorem.(Pythagorean Theorem) Two vectors \(\overrightarrow{u}\) and \(\overrightarrow{v}\) of a real inner product space \(V\) are orthogonal if and only if \(\left\lVert\overrightarrow{v}+\overrightarrow{v}\right\rVert^2 =\left\lVert\overrightarrow{u}\right\rVert^2+\left\lVert\overrightarrow{v}\right\rVert^2\).

Definition. The angle \(\theta\) between two vectors \(\overrightarrow{u}\) and \(\overrightarrow{v}\) of a real inner product space \(V\) is the angle in \([0,\pi]\) satisfying \[\langle\overrightarrow{u},\overrightarrow{v}\rangle =\left\lVert\overrightarrow{u}\right\rVert \left\lVert\overrightarrow{v}\right\rVert \cos \theta.\] Definition. Let \(W\) be a subspace of a real inner product space \(V\). A vector \(\overrightarrow{v}\in V\) is orthogonal to \(W\) if \(\langle\overrightarrow{v},\overrightarrow{w}\rangle =0\) for all \(\overrightarrow{w}\in W\). The orthogonal complement of \(W\), denoted by \(W^{\perp}\), is the set of all vectors in \(V\) that are orthogonal to \(W\), i.e., \[W^{\perp}=\{\overrightarrow{v}\in V \;|\; \langle\overrightarrow{v},\overrightarrow{w}\rangle =0 \text{ for all } \overrightarrow{w}\in W\}.\]

Theorem. Let \(W\) be a subspace of a real inner product space \(V\). Then

1. \(\overrightarrow{v} \in W^{\perp}\) if and only if \(\overrightarrow{v}\) is orthogonal to each vector \(\overrightarrow{w}\) of a basis of \(W\).


2. \(W^{\perp}\) is a subspace of \(V\).


3. \(W\subseteq (W^{\perp})^{\perp}\) where the equality holds for finite dimensional \(W\).


4. \(W\cap W^{\perp}=\{\overrightarrow{0_V}\}\).

Example. The real vector space \(P_1[0,1]\) of all polynomials of degree at most 1 on \([0,1]\) is a real inner product space with the following inner product: \[\langle f,g\rangle= \int_0^1 f(x)g(x)\,dx.\] Find \(W^{\perp}\) for \(W=\operatorname{Span}\{x\}\).

Solution. First note that \(\{x\}\) is a basis of \(W=\operatorname{Span}\{x\}\). Let \(p(x)=a+bx \in W^{\perp}\). Then \begin{align*} 0=\langle p,x\rangle &= \int_0^1 (a+bx)x \,dx\\ &= \left. \frac{ax^2}{2}+\frac{bx^3}{3} \right|_0^1\\ &= \frac{a}{2}+\frac{b}{3}. \end{align*} Then \(b=-\frac{3a}{2}\) and \(p(x)=a-\frac{3a}{2}x=\frac{a}{2}(2-3x)\). Thus \[W^{\perp}=\left\lbrace \frac{a}{2}(2-3x)\;|\; a\in \mathbb R\right\rbrace=\operatorname{Span}\{2-3x\}.\] We can verify that \((W^{\perp})^{\perp}=W\) and \(W\cap W^{\perp}=\{0\}\).

Definition. A subset \(\{\overrightarrow{v_1},\overrightarrow{v_2},\ldots,\overrightarrow{v_k}\}\) of a real inner product space \(V\) is called an orthogonal set if \(\langle\overrightarrow{v_i},\overrightarrow{v_j}\rangle=0\) for all distinct \(i,j=1,2,\ldots,k\). Also \(\{\overrightarrow{v_1},\overrightarrow{v_2},\ldots,\overrightarrow{v_k}\}\) is called an orthonormal set if it is an orthogonal set of unit vectors.

Theorem. If \(\{\overrightarrow{v_1},\overrightarrow{v_2},\ldots,\overrightarrow{v_k}\}\) is an orthogonal set of nonzero vectors in a real inner product space \(V\), then \(\{\overrightarrow{v_1},\overrightarrow{v_2},\ldots,\overrightarrow{v_k}\}\) is linearly independent and consequently forms a basis of \(\operatorname{Span} \{\overrightarrow{v_1},\overrightarrow{v_2},\ldots,\overrightarrow{v_k}\}\).

Definition. Let \(W\) be a subspace of a real inner product space \(V\). An orthogonal basis of \(W\) is a basis of \(W\) that is an orthogonal set. Similarly an orthonormal basis of \(W\) is a basis of \(W\) that is an orthonormal set.

Theorem. Let \(W\) be a subspace of a real inner product space \(V\) and \(\{\overrightarrow{w_1},\overrightarrow{w_2},\ldots,\overrightarrow{w_k}\}\) is an orthogonal basis of \(W\). If \(\overrightarrow{v}\in W\), then \[\overrightarrow{v}=\frac{\langle\overrightarrow{v},\overrightarrow{w_1}\rangle}{\langle\overrightarrow{w_1},\overrightarrow{w_1}\rangle}\overrightarrow{w_1} +\frac{\langle\overrightarrow{v},\overrightarrow{w_2}\rangle}{\langle\overrightarrow{w_2},\overrightarrow{w_2}\rangle}\overrightarrow{w_2} +\cdots+ \frac{\langle\overrightarrow{v},\overrightarrow{w_k}\rangle}{\langle\overrightarrow{w_k},\overrightarrow{w_k}\rangle}\overrightarrow{w_k}.\]

Theorem.(Orthogonal Decomposition Theorem) Let \(W\) be a subspace of a real inner product space \(V\) and \(\overrightarrow{y}\in V\). Then \[\overrightarrow{y}=\overrightarrow{w}+\overrightarrow{z}\] for unique vectors \(\overrightarrow{w}\in W\) and \(\overrightarrow{z}\in W^{\perp}\). Moreover, if \(\{\overrightarrow{w_1},\overrightarrow{w_2},\ldots,\overrightarrow{w_k}\}\) is an orthogonal basis of \(W\), then \[\overrightarrow{w}=\frac{\langle\overrightarrow{y},\overrightarrow{w_1}\rangle}{\langle\overrightarrow{w_1},\overrightarrow{w_1}\rangle}\overrightarrow{w_1} +\frac{\langle\overrightarrow{y},\overrightarrow{w_2}\rangle}{\langle\overrightarrow{w_2},\overrightarrow{w_2}\rangle}\overrightarrow{w_2} +\cdots+ \frac{\langle\overrightarrow{y},\overrightarrow{w_k}\rangle}{\langle\overrightarrow{w_k},\overrightarrow{w_k}\rangle}\overrightarrow{w_k} \text{ and } \overrightarrow{z}=\overrightarrow{y}-\overrightarrow{w}.\]

Definition. Let \(W\) be a subspace of a real inner product space \(V\). Each vector \(\overrightarrow{y}\in V\) can be uniquely written as \(\overrightarrow{y}=\overrightarrow{w}+\overrightarrow{z}\) where \(\overrightarrow{w}\in W\) and \(\overrightarrow{z}\in W^{\perp}\). The unique vector \(\overrightarrow{w}\in W\) is called the orthogonal projection of \(\overrightarrow{y}\) onto \(W\) and it is denoted by \(\operatorname{proj}_W \overrightarrow{y}\).

Corollary. Let \(W\) be a subspace of a real inner product space \(V\) with an orthonormal basis \(\{\overrightarrow{w_1},\overrightarrow{w_2},\ldots,\overrightarrow{w_k}\}\). Then for each \(\overrightarrow{y}\in V\), \[\operatorname{proj}_W \overrightarrow{y}= \langle\overrightarrow{y},\overrightarrow{w_1}\rangle \overrightarrow{w_1} +\langle\overrightarrow{y},\overrightarrow{w_2}\rangle \overrightarrow{w_2} +\cdots+ \langle\overrightarrow{y},\overrightarrow{w_k}\rangle \overrightarrow{w_k}.\]
Theorem.(Best Approximation Theorem) Let \(W\) be a subspace of a real inner product space \(V\) and \(\overrightarrow{b}\in V\). Then \[\min_{\overrightarrow{w}\in W}\left\lVert\overrightarrow{b}-\overrightarrow{w}\right\rVert =\left\lVert\overrightarrow{b}-\operatorname{proj}_W \overrightarrow{b}\right\rVert.\]

Theorem.(Gram-Schmidt Process) Let \(W\) be a subspace of a real inner product space \(V\) with a basis \(\{\overrightarrow{w_1},\overrightarrow{w_2},\ldots,\overrightarrow{w_k}\}\). There is an orthogonal basis \(\{\overrightarrow{v_1},\overrightarrow{v_2},\ldots,\overrightarrow{v_k}\}\) of \(W\) where \[\overrightarrow{v_1}=\overrightarrow{w_1} \text{ and } \overrightarrow{v_i}=\overrightarrow{w_i} -\sum_{j=1}^{i-1} \frac{\langle\overrightarrow{w_i},\overrightarrow{v_j}\rangle}{\langle\overrightarrow{v_j},\overrightarrow{v_j}\rangle} \overrightarrow{v_j}, \; i=2,3,\ldots,k.\] Moreover, \(\operatorname{Span} \{\overrightarrow{v_1},\overrightarrow{v_2},\ldots,\overrightarrow{v_i}\} =\operatorname{Span} \{\overrightarrow{w_1},\overrightarrow{w_2},\ldots,\overrightarrow{w_i}\}\) for \(i=1,2,\ldots,k\).

Example Consider the real inner product space \(C[0,1]\) with the following inner product: \[\langle f,g\rangle= \int_0^1 f(x)g(x)\,dx.\]

1. Find an orthogonal basis of the subspace \(W=\operatorname{Span}\{1,x,x^2\}\) of \(C[0,1]\).


2. Find the best approximation of \(e^x\in C[0,1]\) by a polynomial in \(W\).

Solution.

1. We use Gram-Schmidt Process. First note that \(\{1,x,x^2\}\) is a basis of \(W\). Now we find a vector in \(W\) that is orthogonal to \(1\): \[x-\frac{\langle x,1\rangle}{\langle 1,1\rangle} 1 =x-\frac{\int_0^1 x\cdot 1\,dx}{\int_0^1 1^2\,dx}=x-\frac{1}{2}\] Next we find a vector in \(W\) that is orthogonal to both \(1\) and \(x-\frac{1}{2}\): \[\begin{align*} & x^2-\frac{\langle x^2,1\rangle}{\langle 1,1\rangle} 1 - \frac{\langle x^2,x-\frac{1}{2}\rangle}{\langle x-\frac{1}{2},x-\frac{1}{2}\rangle} \left(x-\frac{1}{2} \right)\\ =\;& x^2 -\frac{\int_0^1 x^2\cdot 1\,dx}{\int_0^1 1^2\,dx} -\frac{\int_0^1 x^2\left(x-\frac{1}{2}\right)\,dx}{\int_0^1 \left(x-\frac{1}{2}\right)^2\,dx}\left(x-\frac{1}{2} \right)\\ =\;& x^2-\frac{\frac{1}{3}}{1}-\frac{\frac{1}{12}}{\frac{1}{12}}\left(x-\frac{1}{2} \right)\\ =\;& x^2-x+\frac{1}{6} \end{align*}\] Thus \(\{ 1,x-\frac{1}{2},x^2-x+\frac{1}{6} \}\) is an orthogonal basis of \(W=\operatorname{Span}\{1,x,x^2\}\).


2. The best approximation of \(e^x\in C[0,1]\) by a polynomial in \(W\) is \[\begin{align*} \operatorname{proj}_{W} e^x=\;&\frac{\langle e^x,1\rangle}{\langle 1,1\rangle} 1 + \frac{\langle e^x,x-\frac{1}{2}\rangle}{\langle x-\frac{1}{2},x-\frac{1}{2}\rangle} \left(x-\frac{1}{2} \right) +\frac{\langle e^x,x^2-x+\frac{1}{6}\rangle}{\langle x^2-x+\frac{1}{6},x^2-x+\frac{1}{6}\rangle}\left(x^2-x+\frac{1}{6}\right)\\ =\;&\frac{\int_0^1 e^x\cdot 1\,dx}{\int_0^1 1^2\,dx} +\frac{\int_0^1 e^x\left(x-\frac{1}{2}\right)\,dx}{\int_0^1 \left(x-\frac{1}{2}\right)^2\,dx}\left(x-\frac{1}{2} \right) +\frac{\int_0^1 e^x\left(x^2-x+\frac{1}{6}\right)\,dx}{\int_0^1 \left(x^2-x+\frac{1}{6}\right)^2\,dx}\left(x^2-x+\frac{1}{6}\right)\\ =\;& \frac{(e-1)}{1}+\frac{(3-e)/2}{1/12}\left(x-\frac{1}{2} \right)+\frac{(7e-19)/6}{1/180} \left(x^2-x+\frac{1}{6} \right)\\ =\;&(e-1)+6(3-e) \left(x-\frac{1}{2} \right)+30(7e-19) \left(x^2-x+\frac{1}{6} \right)\\ =\;&(210e-570)x^2+(-216e+588)x+39e-105. \end{align*}\]

   
   

   Last edited