Any \(m\times n\) matrix \(A\) can be viewed as representing, in some bases, a linear transformation that maps vectors in \(\mathbb{R}^n\) to vectors in \(\mathbb{R}^m\).
The four fundamental subspaces of the matrix \(A\), with rank \(r\), are:
The column space, denoted as \(C(A)\), is the space spanned by the columns of the matrix \(A\) taken as vectors: \[C(A) = \{ Av \mid v\in \mathbb{R}^n \}\] Its dimension is \(r\).
The nullspace, denoted as \(N(A)\), is the set of vectors that are mapped to the zero vector by the matrix: \[N(A) = \{ v\in \mathbb{R}^n \mid Av = 0 \}\] It’s dimension is \(n-r\).
The row space, denoted as \(C(A^T)\) is the space spanned by the rows of \(A\). It is the same as the column space of \(A^T\). It’s dimension is also \(r\).
The left nullspace. denoted as \(N(A^T)\), is the nullspace of \(A^T\). It’s dimension is \(m-r\).
Two vector spaces \(U\) and \(V\) are said to be orthogonal and denoted \(U \perp V\), if for every pair of vectors \(u\in U\) and \(v\in V\), \(u^Tv = 0\).
The row space and the nullspace are subspaces of the domain set \(\mathbb{R}^n\) of \(A\) and are orthogonal: \(N(A) \perp C(A^T)\).
The column space and the left nullspace are the subspaces of the co-domain set \(\mathbb{R}^m\) of the matrix \(A\) and are orthogonal: \(C(A)\perp N(A^T)\).
Row reduce \(A\) to an echelon form \(U\). Then:
The non-zero rows of \(U\) are a basis for the row space of \(A\).
The columns of \(A\) that correspond to the pivot columns of \(U\) form a basis for the column space of \(A\).
The variables corresponding to the non-pivot columns are the free variables. The basis for the nullspace of \(A\) is obtained by vectors which are obtained by, in turn, assigning each free variable the value 1 and the other free variables the value 0.
The projection of vector \(b\) along the vector \(a\) is given by \(\left(\dfrac{a^Tb}{a^Ta}\right) a\).
The projection matrix \(\mathbb{P}\) associated with vector \(a\) is given by \[\mathbb{P} = \dfrac{aa^T}{a^Ta}\]
The projection of \(b\) along \(a\) can then be found as \(\mathbb{P} b\).
The projection matrix associated with a matrix \(A\) is \[\mathbb{P} = A(A^TA)^{-1}A^T\]
In case, \(A^TA\) is not invertible, we need to replace \((A^TA)^{-1}\) with the pseudo inverse \((A^TA)^\dagger \).
For a matrix \(A\), its projection matrix \(\mathbb{P}\) projects any vector \(b\) into the column space of \(A\), i.e. \(\mathbb{P}b \in C(A)\) and can be thought of as the best approximation of \(b\) in the column space of \(A\).
The projection matrix \(\mathbb{P}\)
is symmetric, that is, \(\mathbb{P}^T = \mathbb{P}\),
is idempotent, that is, \(\mathbb{P}^2 = \mathbb{P}\).
Conversely, any matrix satisfying the above two properties can be taken to be a projection matrix.
For any projection matrix \(\mathbb{P}\), the matrix \((I-\mathbb{P})\) is an orthogonal projection matrix. If \(\mathbb{P}\) projects a vector \(b\) onto the \(C(A)\), then \((I-\mathbb{P})\) projects onto the subspace orthogonal to column space of \(A\), that is, the left nullspace \(N(A^T)\).
The matrix equation \(Ax=b\) is solvable if and only if \(b \in C(A)\).
In case \(b\notin C(A)\), we can still find an approximate solution \(\hat{x}\) that minimizes the squared norm \(\Vert Ax-b\Vert^2\).
This least square solution can be obtained by solving the normal equations: \[A^TA\hat{x} = A^Tb\] and is given by \[\hat{x} = (A^TA)^{-1}A^T b\]
Finding the least square solution is equivalent to solving the equation \[A\hat{x} = \mathbb{P} b\] where \(\mathbb{P}\) is the projection matrix of \(A\).