Linear Algebra Home

## Orthogonal Projections

Theorem.(Orthogonal Decomposition Theorem) Let $$W$$ be a subspace of $$\mathbb R^n$$ and $$\overrightarrow{y}\in \mathbb R^n$$. Then $\overrightarrow{y}=\overrightarrow{w}+\overrightarrow{z}$ for unique vectors $$\overrightarrow{w}\in W$$ and $$\overrightarrow{z}\in W^{\perp}$$. Moreover, if $$\{\overrightarrow{w_1},\overrightarrow{w_2},\ldots,\overrightarrow{w_k}\}$$ is an orthogonal basis of $$W$$, then $\overrightarrow{w}=\frac{\overrightarrow{y}\cdot \overrightarrow{w_1}}{\overrightarrow{w_1} \cdot \overrightarrow{w_1}}\overrightarrow{w_1}+ \frac{\overrightarrow{y}\cdot \overrightarrow{w_2}}{\overrightarrow{w_2} \cdot \overrightarrow{w_2}}\overrightarrow{w_2}+\cdots+ \frac{\overrightarrow{y}\cdot \overrightarrow{w_k}}{\overrightarrow{w_k} \cdot \overrightarrow{w_k}}\overrightarrow{w_k} \text{ and } \overrightarrow{z}=\overrightarrow{y}-\overrightarrow{w}.$

Suppose $$\{\overrightarrow{w_1},\overrightarrow{w_2},\ldots,\overrightarrow{w_k}\}$$ is an orthogonal basis of $$W$$. Then $\overrightarrow{w}=\frac{\overrightarrow{y}\cdot \overrightarrow{w_1}}{\overrightarrow{w_1} \cdot \overrightarrow{w_1}}\overrightarrow{w_1}+ \frac{\overrightarrow{y}\cdot \overrightarrow{w_2}}{\overrightarrow{w_2} \cdot \overrightarrow{w_2}}\overrightarrow{w_2}+\cdots+ \frac{\overrightarrow{y}\cdot \overrightarrow{w_k}}{\overrightarrow{w_k} \cdot \overrightarrow{w_k}}\overrightarrow{w_k} \in \operatorname{Span} \{\overrightarrow{w_1},\overrightarrow{w_2},\ldots,\overrightarrow{w_k}\}=W.$ Let $$\overrightarrow{z}=\overrightarrow{y}-\overrightarrow{w}$$. We show that $$\overrightarrow{z}=\overrightarrow{y}-\overrightarrow{w}\in W^{\perp}$$. For $$i=1,2,\ldots,k$$, \begin{align*} \overrightarrow{z} \cdot \overrightarrow{w_i}&= (\overrightarrow{y}-\overrightarrow{w}) \cdot \overrightarrow{w_i}\\ &= \overrightarrow{y} \cdot \overrightarrow{w_i}-\overrightarrow{w} \cdot \overrightarrow{w_i}\\ &= \overrightarrow{y} \cdot \overrightarrow{w_i}-\left( \frac{\overrightarrow{y}\cdot \overrightarrow{w_1}}{\overrightarrow{w_1} \cdot \overrightarrow{w_1}}\overrightarrow{w_1}+ \frac{\overrightarrow{y}\cdot \overrightarrow{w_2}}{\overrightarrow{w_2} \cdot \overrightarrow{w_2}}\overrightarrow{w_2}+\cdots+ \frac{\overrightarrow{y}\cdot \overrightarrow{w_k}}{\overrightarrow{w_k} \cdot \overrightarrow{w_k}}\overrightarrow{w_k} \right) \cdot \overrightarrow{w_i}\\ &= \overrightarrow{y} \cdot \overrightarrow{w_i}-\left(0+\cdots+0+ \frac{\overrightarrow{y}\cdot \overrightarrow{w_i}}{\overrightarrow{w_i} \cdot \overrightarrow{w_i}}\overrightarrow{w_i}\cdot \overrightarrow{w_i}+0+\cdots+0 \right)\\ &= 0. \end{align*} Since $$\overrightarrow{z} \cdot \overrightarrow{w_i}=0$$ for $$i=1,2,\ldots,k$$, $$\overrightarrow{z} \cdot \overrightarrow{w}=0$$ for all $$\overrightarrow{w}\in W=\operatorname{Span} \{\overrightarrow{w_1},\overrightarrow{w_2},\ldots,\overrightarrow{w_k}\}$$ and consequently $$\overrightarrow{z}\in W^{\perp}$$. To show the uniqueness of the decomposition $$\overrightarrow{y}=\overrightarrow{w}+\overrightarrow{z}$$, let $$\overrightarrow{y}=\overrightarrow{w}'+\overrightarrow{z}'$$ for some $$\overrightarrow{w}' \in W$$ and $$\overrightarrow{z}' \in W^{\perp}$$. Then $\begin{array}{rrl} &\overrightarrow{0} = &\overrightarrow{y}-\overrightarrow{y}=(\overrightarrow{w}+\overrightarrow{z})-(\overrightarrow{w}'+\overrightarrow{z}')\\ \implies & \overrightarrow{w}'-\overrightarrow{w} =& \overrightarrow{z}-\overrightarrow{z}' \in W\cap W^{\perp}=\{\overrightarrow{0} \}\\ \implies & \overrightarrow{w}'=\overrightarrow{w}, & \overrightarrow{z}'=\overrightarrow{z}. \end{array}$

Definition. Let $$W$$ be a subspace of $$\mathbb R^n$$. Each vector $$\overrightarrow{y}\in \mathbb R^n$$ can be uniquely written as $$\overrightarrow{y}=\overrightarrow{w}+\overrightarrow{z}$$ where $$\overrightarrow{w}\in W$$ and $$\overrightarrow{z}\in W^{\perp}$$. The unique vector $$\overrightarrow{w}\in W$$ is called the orthogonal projection of $$\overrightarrow{y}$$ onto $$W$$ and it is denoted by $$\operatorname{proj}_W \overrightarrow{y}$$.

Example.

1. Let $$\overrightarrow{w}=[2,1]^T$$ and $$W=\operatorname{Span} \{\overrightarrow{w}\}$$. For $$\overrightarrow{y}=[2,3]^T$$, find $$\operatorname{proj}_W \overrightarrow{y}$$ and the orthogonal decomposition of $$\overrightarrow{y}$$ with respect to $$W$$.

Solution. $$\operatorname{proj}_W \overrightarrow{y}=\frac{\overrightarrow{y}\cdot \overrightarrow{w}}{\overrightarrow{w} \cdot \overrightarrow{w}}\overrightarrow{w}= \frac{7}{5}[2,1]^T \in W$$ and $$\overrightarrow{y}-\operatorname{proj}_W \overrightarrow{y}=\frac{1}{5}[-4,8]^T \in W^{\perp}$$. The orthogonal decomposition of $$\overrightarrow{y}$$ with respect to $$W$$ is $\overrightarrow{y}=[2,3]^T=\frac{7}{5}[2,1]^T+\frac{1}{5}[-4,8]^T.$

2. Let $$\overrightarrow{w_1}=\left[\begin{array}{r}2\\3\\0\end{array} \right]$$, $$\overrightarrow{w_2}=\left[\begin{array}{r}0\\0\\2\end{array} \right]$$, and $$W=\operatorname{Span} \{\overrightarrow{w_1},\overrightarrow{w_2}\}$$. For $$\overrightarrow{y}=\left[\begin{array}{r}1\\0\\1\end{array} \right]$$, find $$\operatorname{proj}_W \overrightarrow{y}$$ and the orthogonal decomposition of $$\overrightarrow{y}$$ with respect to $$W$$.

Solution. \begin{align*} \operatorname{proj}_W \overrightarrow{y}&=\frac{\overrightarrow{y}\cdot \overrightarrow{w_1}}{\overrightarrow{w_1} \cdot \overrightarrow{w_1}}\overrightarrow{w_1} +\frac{\overrightarrow{y}\cdot \overrightarrow{w_2}}{\overrightarrow{w_2} \cdot \overrightarrow{w_2}}\overrightarrow{w_2}\\ &=\frac{2}{13}\left[\begin{array}{r}2\\3\\0\end{array} \right] +\frac{2}{4}\left[\begin{array}{r}0\\0\\2\end{array} \right]\\ &=\frac{1}{13}\left[\begin{array}{r}4\\6\\13\end{array} \right]\in W,\\ \overrightarrow{y}-\operatorname{proj}_W \overrightarrow{y}&=\frac{1}{13}\left[\begin{array}{r}9\\-6\\0\end{array} \right] \in W^{\perp}. \end{align*} The orthogonal decomposition of $$\overrightarrow{y}$$ with respect to $$W$$ is $\overrightarrow{y}=\left[\begin{array}{r}1\\0\\1\end{array} \right] =\frac{1}{13}\left[\begin{array}{r}4\\6\\13\end{array} \right]+\frac{1}{13}\left[\begin{array}{r}9\\-6\\0\end{array} \right].$

Corollary. Let $$W$$ be a subspace of $$\mathbb R^n$$ with an orthonormal basis $$\{\overrightarrow{w_1},\overrightarrow{w_2},\ldots,\overrightarrow{w_k}\}$$. Let $$U=[\overrightarrow{w_1}\; \overrightarrow{w_2}\; \cdots \;\overrightarrow{w_k}]$$. Then for each $$\overrightarrow{y}\in \mathbb R^n$$, $\operatorname{proj}_W \overrightarrow{y}=UU^Ty= (\overrightarrow{y}\cdot \overrightarrow{w_1}) \overrightarrow{w_1} +(\overrightarrow{y}\cdot \overrightarrow{w_2}) \overrightarrow{w_2} +\cdots+ (\overrightarrow{y}\cdot \overrightarrow{w_k}) \overrightarrow{w_k}.$

$U^T\overrightarrow{y}=\left[\begin{array}{c}\overrightarrow{w_1}^T\\\overrightarrow{w_2}^T\\ \vdots\\\overrightarrow{w_k}^T \end{array} \right] \overrightarrow{y} =\left[\begin{array}{c}\overrightarrow{w_1}^T\overrightarrow{y} \\\overrightarrow{w_2}^T\overrightarrow{y} \\ \vdots\\\overrightarrow{w_k}^T\overrightarrow{y} \end{array} \right] =\left[\begin{array}{c}\overrightarrow{w_1}\cdot \overrightarrow{y}\\\overrightarrow{w_2}\cdot \overrightarrow{y}\\ \vdots\\\overrightarrow{w_k}\cdot \overrightarrow{y} \end{array} \right].$ $UU^T\overrightarrow{y} =[\overrightarrow{w_1}\; \overrightarrow{w_2}\; \cdots \;\overrightarrow{w_k}] \left[\begin{array}{c}\overrightarrow{w_1}\cdot \overrightarrow{y}\\\overrightarrow{w_2}\cdot \overrightarrow{y}\\ \vdots\\\overrightarrow{w_k}\cdot \overrightarrow{y} \end{array} \right] =(\overrightarrow{y}\cdot \overrightarrow{w_1}) \overrightarrow{w_1} +(\overrightarrow{y}\cdot \overrightarrow{w_2}) \overrightarrow{w_2} +\cdots+ (\overrightarrow{y}\cdot \overrightarrow{w_k}) \overrightarrow{w_k} =\operatorname{proj}_W \overrightarrow{y}.$

Remark. Recall that for an $$m\times n$$ real matrix $$A$$, $$A\overrightarrow{x}=\overrightarrow{b}$$ has a solution if and only if $$\overrightarrow{b} \in \operatorname{CS}\left(A\right)$$. So $$A\overrightarrow{x}=\overrightarrow{b}$$ has no solution if and only if $$\overrightarrow{b} \notin \operatorname{CS}\left(A\right)$$. We find $$\overrightarrow{w}\in \operatorname{CS}\left(A\right)$$ that is closest to $$\overrightarrow{b}$$, i.e., the best approximation to $$\overrightarrow{b}$$ by a vector $$\overrightarrow{w}\in \operatorname{CS}\left(A\right)$$.

Theorem.(Best Approximation Theorem) Let $$W$$ be a subspace of $$\mathbb R^n$$ and $$\overrightarrow{b}\in \mathbb R^n$$. Then $\min_{\overrightarrow{w}\in W}\left\lVert\overrightarrow{b}-\overrightarrow{w}\right\rVert =\left\lVert\overrightarrow{b}-\operatorname{proj}_W \overrightarrow{b}\right\rVert.$

It suffices to show that $$\left\lVert\overrightarrow{b}-\operatorname{proj}_W \overrightarrow{b}\right\rVert < \left\lVert\overrightarrow{b}- \overrightarrow{w}\right\rVert$$ for all $$\overrightarrow{w}\in W$$ when $$\overrightarrow{w}\neq \operatorname{proj}_W \overrightarrow{b}$$. Let $$\overrightarrow{w}\in W$$ and $$\overrightarrow{w}\neq \operatorname{proj}_W \overrightarrow{b}$$. Then $$\overrightarrow{0}\neq \operatorname{proj}_W \overrightarrow{b}-\overrightarrow{w}\in W$$. Since $$\operatorname{proj}_W \overrightarrow{b} \in W$$, $$\overrightarrow{b}-\operatorname{proj}_W \overrightarrow{b} \in W^{\perp}$$ by the orthogonal decomposition. Then $(\overrightarrow{b}-\operatorname{proj}_W \overrightarrow{b})\cdot (\operatorname{proj}_W \overrightarrow{b}-\overrightarrow{w})=0.$ By Pythagorean theorem, \begin{align*} & \left\lVert(\overrightarrow{b}-\operatorname{proj}_W \overrightarrow{b})+ (\operatorname{proj}_W \overrightarrow{b}-\overrightarrow{w})\right\rVert^2 =\left\lVert\overrightarrow{b}-\operatorname{proj}_W \overrightarrow{b}\right\rVert^2 +\left\lVert\operatorname{proj}_W \overrightarrow{b}-\overrightarrow{w}\right\rVert^2\\ \implies & \left\lVert\overrightarrow{b}-\overrightarrow{w}\right\rVert^2= \left\lVert\overrightarrow{b}-\operatorname{proj}_W \overrightarrow{b}\right\rVert^2 +\left\lVert\operatorname{proj}_W \overrightarrow{b}-\overrightarrow{w}\right\rVert^2 > \left\lVert\overrightarrow{b}-\operatorname{proj}_W \overrightarrow{b}\right\rVert^2 \end{align*} because $$\operatorname{proj}_W \overrightarrow{b}-\overrightarrow{w}\neq \overrightarrow{0}$$. Thus $$\left\lVert\overrightarrow{b}- \overrightarrow{w}\right\rVert > \left\lVert\overrightarrow{b}-\operatorname{proj}_W \overrightarrow{b}\right\rVert$$.

Example. Let $$\overrightarrow{u}=[2,3,0]^T$$, $$\overrightarrow{v}=[0,0,2]^T$$, and $$W=\operatorname{Span} \{\overrightarrow{u},\overrightarrow{v}\}$$. For $$\overrightarrow{y}=[1,0,1]^T$$, find the point on $$W$$ closest to $$\overrightarrow{y}$$ (the best approximation to $$\overrightarrow{y}$$ by a vector of $$W$$) and find the distance between $$\overrightarrow{y}$$ and $$W$$.

Solution. The point on $$W$$ closest to $$\overrightarrow{y}$$ is $$\operatorname{proj}_W \overrightarrow{y}=\frac{1}{13}[4,6,13]^T \in W$$ (show steps). The distance between $$\overrightarrow{y}$$ and $$W$$ is $$\left\lVert\overrightarrow{y}-\operatorname{proj}_W \overrightarrow{y}\right\rVert =\left\lVert\frac{1}{13}[9,-6,0]^T\right\rVert=\frac{\sqrt{117}}{13}$$.
To find $$\operatorname{proj}_W \overrightarrow{y}$$ in an alternative way, note that $$\left\lbrace \frac{\overrightarrow{u}}{\left\lVert\overrightarrow{u}\right\rVert}\; \frac{\overrightarrow{v}}{\left\lVert\overrightarrow{v}\right\rVert} \right\rbrace$$ is an orthonormal basis of $$W$$. Let $$U=\left[\frac{\overrightarrow{u}}{\left\lVert\overrightarrow{u}\right\rVert}\; \frac{\overrightarrow{v}}{\left\lVert\overrightarrow{v}\right\rVert} \right] =\left[\begin{array}{cc} \frac{2}{\sqrt{13}}&0\\ \frac{3}{\sqrt{13}}&0\\ 0&1\end{array}\right]$$. Then $\operatorname{proj}_W \overrightarrow{y}=UU^Ty= \left[\begin{array}{cc} \frac{2}{\sqrt{13}}&0\\ \frac{3}{\sqrt{13}}&0\\ 0&1\end{array}\right] \left[\begin{array}{ccc} \frac{2}{\sqrt{13}}&\frac{3}{\sqrt{13}}&0\\ 0&0&1\end{array}\right] \left[\begin{array}{r}1\\0\\1\end{array} \right] =\left[\begin{array}{rr} \frac{2}{\sqrt{13}}&0\\ \frac{3}{\sqrt{13}}&0\\ 0&1\end{array}\right] \left[\begin{array}{c} \frac{2}{\sqrt{13}}\\1\end{array} \right] =\frac{1}{13}\left[\begin{array}{c}4\\6\\13\end{array} \right].$

Last edited