Linear Algebra Home

Orthogonal Projections

    


Theorem.(Orthogonal Decomposition Theorem) Let \(W\) be a subspace of \(\mathbb R^n\) and \(\overrightarrow{y}\in \mathbb R^n\). Then \[\overrightarrow{y}=\overrightarrow{w}+\overrightarrow{z}\] for unique vectors \(\overrightarrow{w}\in W\) and \(\overrightarrow{z}\in W^{\perp}\). Moreover, if \(\{\overrightarrow{w_1},\overrightarrow{w_2},\ldots,\overrightarrow{w_k}\}\) is an orthogonal basis of \(W\), then \[\overrightarrow{w}=\frac{\overrightarrow{y}\cdot \overrightarrow{w_1}}{\overrightarrow{w_1} \cdot \overrightarrow{w_1}}\overrightarrow{w_1}+ \frac{\overrightarrow{y}\cdot \overrightarrow{w_2}}{\overrightarrow{w_2} \cdot \overrightarrow{w_2}}\overrightarrow{w_2}+\cdots+ \frac{\overrightarrow{y}\cdot \overrightarrow{w_k}}{\overrightarrow{w_k} \cdot \overrightarrow{w_k}}\overrightarrow{w_k} \text{ and } \overrightarrow{z}=\overrightarrow{y}-\overrightarrow{w}.\]

Suppose \(\{\overrightarrow{w_1},\overrightarrow{w_2},\ldots,\overrightarrow{w_k}\}\) is an orthogonal basis of \(W\). Then \[\overrightarrow{w}=\frac{\overrightarrow{y}\cdot \overrightarrow{w_1}}{\overrightarrow{w_1} \cdot \overrightarrow{w_1}}\overrightarrow{w_1}+ \frac{\overrightarrow{y}\cdot \overrightarrow{w_2}}{\overrightarrow{w_2} \cdot \overrightarrow{w_2}}\overrightarrow{w_2}+\cdots+ \frac{\overrightarrow{y}\cdot \overrightarrow{w_k}}{\overrightarrow{w_k} \cdot \overrightarrow{w_k}}\overrightarrow{w_k} \in \operatorname{Span} \{\overrightarrow{w_1},\overrightarrow{w_2},\ldots,\overrightarrow{w_k}\}=W.\] Let \(\overrightarrow{z}=\overrightarrow{y}-\overrightarrow{w}\). We show that \(\overrightarrow{z}=\overrightarrow{y}-\overrightarrow{w}\in W^{\perp}\). For \(i=1,2,\ldots,k\), \[\begin{align*} \overrightarrow{z} \cdot \overrightarrow{w_i}&= (\overrightarrow{y}-\overrightarrow{w}) \cdot \overrightarrow{w_i}\\ &= \overrightarrow{y} \cdot \overrightarrow{w_i}-\overrightarrow{w} \cdot \overrightarrow{w_i}\\ &= \overrightarrow{y} \cdot \overrightarrow{w_i}-\left( \frac{\overrightarrow{y}\cdot \overrightarrow{w_1}}{\overrightarrow{w_1} \cdot \overrightarrow{w_1}}\overrightarrow{w_1}+ \frac{\overrightarrow{y}\cdot \overrightarrow{w_2}}{\overrightarrow{w_2} \cdot \overrightarrow{w_2}}\overrightarrow{w_2}+\cdots+ \frac{\overrightarrow{y}\cdot \overrightarrow{w_k}}{\overrightarrow{w_k} \cdot \overrightarrow{w_k}}\overrightarrow{w_k} \right) \cdot \overrightarrow{w_i}\\ &= \overrightarrow{y} \cdot \overrightarrow{w_i}-\left(0+\cdots+0+ \frac{\overrightarrow{y}\cdot \overrightarrow{w_i}}{\overrightarrow{w_i} \cdot \overrightarrow{w_i}}\overrightarrow{w_i}\cdot \overrightarrow{w_i}+0+\cdots+0 \right)\\ &= 0. \end{align*}\] Since \(\overrightarrow{z} \cdot \overrightarrow{w_i}=0\) for \(i=1,2,\ldots,k\), \(\overrightarrow{z} \cdot \overrightarrow{w}=0\) for all \(\overrightarrow{w}\in W=\operatorname{Span} \{\overrightarrow{w_1},\overrightarrow{w_2},\ldots,\overrightarrow{w_k}\}\) and consequently \(\overrightarrow{z}\in W^{\perp}\). To show the uniqueness of the decomposition \(\overrightarrow{y}=\overrightarrow{w}+\overrightarrow{z}\), let \(\overrightarrow{y}=\overrightarrow{w}'+\overrightarrow{z}'\) for some \(\overrightarrow{w}' \in W\) and \(\overrightarrow{z}' \in W^{\perp}\). Then \[\begin{array}{rrl} &\overrightarrow{0} = &\overrightarrow{y}-\overrightarrow{y}=(\overrightarrow{w}+\overrightarrow{z})-(\overrightarrow{w}'+\overrightarrow{z}')\\ \implies & \overrightarrow{w}'-\overrightarrow{w} =& \overrightarrow{z}-\overrightarrow{z}' \in W\cap W^{\perp}=\{\overrightarrow{0} \}\\ \implies & \overrightarrow{w}'=\overrightarrow{w}, & \overrightarrow{z}'=\overrightarrow{z}. \end{array}\]

Definition. Let \(W\) be a subspace of \(\mathbb R^n\). Each vector \(\overrightarrow{y}\in \mathbb R^n\) can be uniquely written as \(\overrightarrow{y}=\overrightarrow{w}+\overrightarrow{z}\) where \(\overrightarrow{w}\in W\) and \(\overrightarrow{z}\in W^{\perp}\). The unique vector \(\overrightarrow{w}\in W\) is called the orthogonal projection of \(\overrightarrow{y}\) onto \(W\) and it is denoted by \(\operatorname{proj}_W \overrightarrow{y}\).

Example.

  1. Let \(\overrightarrow{w}=[2,1]^T\) and \(W=\operatorname{Span} \{\overrightarrow{w}\}\). For \(\overrightarrow{y}=[2,3]^T\), find \(\operatorname{proj}_W \overrightarrow{y}\) and the orthogonal decomposition of \(\overrightarrow{y}\) with respect to \(W\).

    Solution. \(\operatorname{proj}_W \overrightarrow{y}=\frac{\overrightarrow{y}\cdot \overrightarrow{w}}{\overrightarrow{w} \cdot \overrightarrow{w}}\overrightarrow{w}= \frac{7}{5}[2,1]^T \in W\) and \(\overrightarrow{y}-\operatorname{proj}_W \overrightarrow{y}=\frac{1}{5}[-4,8]^T \in W^{\perp}\). The orthogonal decomposition of \(\overrightarrow{y}\) with respect to \(W\) is \[\overrightarrow{y}=[2,3]^T=\frac{7}{5}[2,1]^T+\frac{1}{5}[-4,8]^T.\]

  2. Let \(\overrightarrow{w_1}=\left[\begin{array}{r}2\\3\\0\end{array} \right]\), \(\overrightarrow{w_2}=\left[\begin{array}{r}0\\0\\2\end{array} \right]\), and \(W=\operatorname{Span} \{\overrightarrow{w_1},\overrightarrow{w_2}\}\). For \(\overrightarrow{y}=\left[\begin{array}{r}1\\0\\1\end{array} \right]\), find \(\operatorname{proj}_W \overrightarrow{y}\) and the orthogonal decomposition of \(\overrightarrow{y}\) with respect to \(W\).

    Solution. \[\begin{align*} \operatorname{proj}_W \overrightarrow{y}&=\frac{\overrightarrow{y}\cdot \overrightarrow{w_1}}{\overrightarrow{w_1} \cdot \overrightarrow{w_1}}\overrightarrow{w_1} +\frac{\overrightarrow{y}\cdot \overrightarrow{w_2}}{\overrightarrow{w_2} \cdot \overrightarrow{w_2}}\overrightarrow{w_2}\\ &=\frac{2}{13}\left[\begin{array}{r}2\\3\\0\end{array} \right] +\frac{2}{4}\left[\begin{array}{r}0\\0\\2\end{array} \right]\\ &=\frac{1}{13}\left[\begin{array}{r}4\\6\\13\end{array} \right]\in W,\\ \overrightarrow{y}-\operatorname{proj}_W \overrightarrow{y}&=\frac{1}{13}\left[\begin{array}{r}9\\-6\\0\end{array} \right] \in W^{\perp}. \end{align*}\] The orthogonal decomposition of \(\overrightarrow{y}\) with respect to \(W\) is \[\overrightarrow{y}=\left[\begin{array}{r}1\\0\\1\end{array} \right] =\frac{1}{13}\left[\begin{array}{r}4\\6\\13\end{array} \right]+\frac{1}{13}\left[\begin{array}{r}9\\-6\\0\end{array} \right].\]

Corollary. Let \(W\) be a subspace of \(\mathbb R^n\) with an orthonormal basis \(\{\overrightarrow{w_1},\overrightarrow{w_2},\ldots,\overrightarrow{w_k}\}\). Let \(U=[\overrightarrow{w_1}\; \overrightarrow{w_2}\; \cdots \;\overrightarrow{w_k}]\). Then for each \(\overrightarrow{y}\in \mathbb R^n\), \[\operatorname{proj}_W \overrightarrow{y}=UU^Ty= (\overrightarrow{y}\cdot \overrightarrow{w_1}) \overrightarrow{w_1} +(\overrightarrow{y}\cdot \overrightarrow{w_2}) \overrightarrow{w_2} +\cdots+ (\overrightarrow{y}\cdot \overrightarrow{w_k}) \overrightarrow{w_k}.\]

\[U^T\overrightarrow{y}=\left[\begin{array}{c}\overrightarrow{w_1}^T\\\overrightarrow{w_2}^T\\ \vdots\\\overrightarrow{w_k}^T \end{array} \right] \overrightarrow{y} =\left[\begin{array}{c}\overrightarrow{w_1}^T\overrightarrow{y} \\\overrightarrow{w_2}^T\overrightarrow{y} \\ \vdots\\\overrightarrow{w_k}^T\overrightarrow{y} \end{array} \right] =\left[\begin{array}{c}\overrightarrow{w_1}\cdot \overrightarrow{y}\\\overrightarrow{w_2}\cdot \overrightarrow{y}\\ \vdots\\\overrightarrow{w_k}\cdot \overrightarrow{y} \end{array} \right].\] \[UU^T\overrightarrow{y} =[\overrightarrow{w_1}\; \overrightarrow{w_2}\; \cdots \;\overrightarrow{w_k}] \left[\begin{array}{c}\overrightarrow{w_1}\cdot \overrightarrow{y}\\\overrightarrow{w_2}\cdot \overrightarrow{y}\\ \vdots\\\overrightarrow{w_k}\cdot \overrightarrow{y} \end{array} \right] =(\overrightarrow{y}\cdot \overrightarrow{w_1}) \overrightarrow{w_1} +(\overrightarrow{y}\cdot \overrightarrow{w_2}) \overrightarrow{w_2} +\cdots+ (\overrightarrow{y}\cdot \overrightarrow{w_k}) \overrightarrow{w_k} =\operatorname{proj}_W \overrightarrow{y}.\]

Remark. Recall that for an \(m\times n\) real matrix \(A\), \(A\overrightarrow{x}=\overrightarrow{b}\) has a solution if and only if \(\overrightarrow{b} \in \operatorname{CS}\left(A\right)\). So \(A\overrightarrow{x}=\overrightarrow{b}\) has no solution if and only if \(\overrightarrow{b} \notin \operatorname{CS}\left(A\right)\). We find \(\overrightarrow{w}\in \operatorname{CS}\left(A\right)\) that is closest to \(\overrightarrow{b}\), i.e., the best approximation to \(\overrightarrow{b}\) by a vector \(\overrightarrow{w}\in \operatorname{CS}\left(A\right)\).

Theorem.(Best Approximation Theorem) Let \(W\) be a subspace of \(\mathbb R^n\) and \(\overrightarrow{b}\in \mathbb R^n\). Then \[\min_{\overrightarrow{w}\in W}\left\lVert\overrightarrow{b}-\overrightarrow{w}\right\rVert =\left\lVert\overrightarrow{b}-\operatorname{proj}_W \overrightarrow{b}\right\rVert.\]

It suffices to show that \(\left\lVert\overrightarrow{b}-\operatorname{proj}_W \overrightarrow{b}\right\rVert < \left\lVert\overrightarrow{b}- \overrightarrow{w}\right\rVert\) for all \(\overrightarrow{w}\in W\) when \(\overrightarrow{w}\neq \operatorname{proj}_W \overrightarrow{b}\). Let \(\overrightarrow{w}\in W\) and \(\overrightarrow{w}\neq \operatorname{proj}_W \overrightarrow{b}\). Then \(\overrightarrow{0}\neq \operatorname{proj}_W \overrightarrow{b}-\overrightarrow{w}\in W\). Since \(\operatorname{proj}_W \overrightarrow{b} \in W\), \(\overrightarrow{b}-\operatorname{proj}_W \overrightarrow{b} \in W^{\perp}\) by the orthogonal decomposition. Then \[(\overrightarrow{b}-\operatorname{proj}_W \overrightarrow{b})\cdot (\operatorname{proj}_W \overrightarrow{b}-\overrightarrow{w})=0.\] By Pythagorean theorem, \[\begin{align*} & \left\lVert(\overrightarrow{b}-\operatorname{proj}_W \overrightarrow{b})+ (\operatorname{proj}_W \overrightarrow{b}-\overrightarrow{w})\right\rVert^2 =\left\lVert\overrightarrow{b}-\operatorname{proj}_W \overrightarrow{b}\right\rVert^2 +\left\lVert\operatorname{proj}_W \overrightarrow{b}-\overrightarrow{w}\right\rVert^2\\ \implies & \left\lVert\overrightarrow{b}-\overrightarrow{w}\right\rVert^2= \left\lVert\overrightarrow{b}-\operatorname{proj}_W \overrightarrow{b}\right\rVert^2 +\left\lVert\operatorname{proj}_W \overrightarrow{b}-\overrightarrow{w}\right\rVert^2 > \left\lVert\overrightarrow{b}-\operatorname{proj}_W \overrightarrow{b}\right\rVert^2 \end{align*}\] because \(\operatorname{proj}_W \overrightarrow{b}-\overrightarrow{w}\neq \overrightarrow{0}\). Thus \(\left\lVert\overrightarrow{b}- \overrightarrow{w}\right\rVert > \left\lVert\overrightarrow{b}-\operatorname{proj}_W \overrightarrow{b}\right\rVert\).

Example. Let \(\overrightarrow{u}=[2,3,0]^T\), \(\overrightarrow{v}=[0,0,2]^T\), and \(W=\operatorname{Span} \{\overrightarrow{u},\overrightarrow{v}\}\). For \(\overrightarrow{y}=[1,0,1]^T\), find the point on \(W\) closest to \(\overrightarrow{y}\) (the best approximation to \(\overrightarrow{y}\) by a vector of \(W\)) and find the distance between \(\overrightarrow{y}\) and \(W\).

Solution. The point on \(W\) closest to \(\overrightarrow{y}\) is \(\operatorname{proj}_W \overrightarrow{y}=\frac{1}{13}[4,6,13]^T \in W\) (show steps). The distance between \(\overrightarrow{y}\) and \(W\) is \(\left\lVert\overrightarrow{y}-\operatorname{proj}_W \overrightarrow{y}\right\rVert =\left\lVert\frac{1}{13}[9,-6,0]^T\right\rVert=\frac{\sqrt{117}}{13}\).
To find \(\operatorname{proj}_W \overrightarrow{y}\) in an alternative way, note that \(\left\lbrace \frac{\overrightarrow{u}}{\left\lVert\overrightarrow{u}\right\rVert}\; \frac{\overrightarrow{v}}{\left\lVert\overrightarrow{v}\right\rVert} \right\rbrace\) is an orthonormal basis of \(W\). Let \(U=\left[\frac{\overrightarrow{u}}{\left\lVert\overrightarrow{u}\right\rVert}\; \frac{\overrightarrow{v}}{\left\lVert\overrightarrow{v}\right\rVert} \right] =\left[\begin{array}{cc} \frac{2}{\sqrt{13}}&0\\ \frac{3}{\sqrt{13}}&0\\ 0&1\end{array}\right]\). Then \[\operatorname{proj}_W \overrightarrow{y}=UU^Ty= \left[\begin{array}{cc} \frac{2}{\sqrt{13}}&0\\ \frac{3}{\sqrt{13}}&0\\ 0&1\end{array}\right] \left[\begin{array}{ccc} \frac{2}{\sqrt{13}}&\frac{3}{\sqrt{13}}&0\\ 0&0&1\end{array}\right] \left[\begin{array}{r}1\\0\\1\end{array} \right] =\left[\begin{array}{rr} \frac{2}{\sqrt{13}}&0\\ \frac{3}{\sqrt{13}}&0\\ 0&1\end{array}\right] \left[\begin{array}{c} \frac{2}{\sqrt{13}}\\1\end{array} \right] =\frac{1}{13}\left[\begin{array}{c}4\\6\\13\end{array} \right].\]


Last edited