Proof:

The solution x to the least-squares problem (11) satisfies

$$Ax=\mbox{proj}_{\r(A)}b.$$

Therefore, by the projection theorem, b-Ax must be orthogonal to $\r(A)$, and hence

$$(Ay)\cdot(b-Ax)=0\ \mbox{for all}\y\in{\bf {\rm R}}^n.$$

But

$$(Ay)\cdot(b-Ax)=y\cdot\left(A^Tb-A^TAx\right),$$

and

$$y\cdot\left(A^Tb-A^TAx\right)=0\ \mbox{for all}\y\in{\bf {\rm R}}^n$$

holds if and only if A^Tb-A^TAx=0. Since A has full rank, A^TA is invertible, so the unique solution x is given by

$$x=\left(A^TA\right)^{-1}A^Tb.$$

QED

Corollary 4.3   Suppose $A\in{\bf {\rm R}}^{m\times n}$, where $m\ge n$, has full rank and $b\in{\bf {\rm R}}^m$. Then the orthogonal projection of b onto $\r(A)$ is

$$A\left(A^TA\right)^{-1}A^Tb$$

and the projection of b onto

$$\r(A)^{\bot}={\cal N}(A^T)$$

is

$$b-A\left(A^TA\right)^{-1}A^Tb.$$