The Fundamental Theorem of Linear Algebra

The following theorem, which I present without proof, is one of the most important results from linear algebra.

Theorem 2.1 (The projection theorem)   Suppose V is any inner product space (that is, vector space with an inner product) and W is a finite-dimensional subspace of V. Given any $v\in V$, there exists a unique vector $w\in W$ closest to v. In other words, there is a unique solution to

$$\min_{z\in W}\Vert v-z\Vert.$$

Moreover, this closest vector w is characterized by the following orthogonality condition:

$$(v-w,z)=0\ \mbox{for all}\z\in W.$$ (5)

In the above theorem, (x,y) denotes the inner product of two vectors $x,y\in V$. The vector w is called the orthogonal projection of v onto W, or the best approximation to v from W. It is sometimes denoted by $w=\mbox{proj}_Wv$.

Given any matrix $A\in{\bf {\rm R}}^{m\times n}$, the range (or column space) of A is defined by

$$\r(A)=\left\{Ax\,:\,x\in{\bf {\rm R}}^n\right\}$$

and the null space (or kernel) of A is

$${\cal N}(A)=\left\{x\in{\bf {\rm R}}^n\,:\,Ax=0\right\}.$$

I wish to discuss the relationships that exist between the ranges and null spaces of A and A^T. The following concept is crucial.

Definition 2.2   Suppose S is a nonempty subset of ${\bf {\rm R}}^n$. The orthogonal complement of S is the set

$$S^{\bot}=\left\{y\in{\bf {\rm R}}^n\,:\,x\cdot y=0\ \forall\ x\in S\right\}.$$

The basic properties of orthogonal complements are summarized in the following theorem.

Theorem 2.3   Suppose S is a nonempty subset of ${\bf {\rm R}}^n$. Then

1.

$S^{\bot}$ is a subspace of ${\bf {\rm R}}^n$.

2.

$\left(S^{\bot}\right)^{\bot}=\mbox{span}(S)$, that is, $\left(S^{\bot}\right)^{\bot}$ is the smallest subspace containing S. In particular, if S is a subspace of ${\bf {\rm R}}^n$, then $\left(S^{\bot}\right)^{\bot}=S$.

3.

If S is a subspace of ${\bf {\rm R}}^n$, then ${\bf {\rm R}}^n=S\oplus S^{\bot}$, that is, every $x\in{\bf {\rm R}}^n$ can be written uniquely as x=y+z, where $y\in S$ and $z\in S^{\bot}$.