We will now illustrate what the SVD Theorem tells us about matrix-vector multiplication (linear transformations) by examining the case where \(A \in \R^{2 \times 2} \text{.}\) Let \(A = U \Sigma V^T \) be its SVD. (Notice that all matrices are now real valued, and hence \(V^H = V^T \text{.}\)) Partition
Since \(U \) and \(V \) are unitary matrices, \(\{ u_0, u_1 \} \) and \(\{ v_0, v_1 \} \) form orthonormal bases for the range and domain of \(A \text{,}\) respectively:
\(\R^{2} \text{:}\) Domain of \(A \text{:}\)
\(\R^{2} \text{:}\) Range (codomain) of \(A \text{:}\)
and similarly \(A v_1 = \sigma_1 u_1 \text{.}\) This motivates the pictures in Figure 2.3.2.1.
\(\R^{2} \text{:}\) Domain of \(A \text{:}\)
\(\R^{2} \text{:}\) Range (codomain) of \(A \text{:}\)
\(\R^{2} \text{:}\) Domain of \(A \text{:}\)
\(\R^{2} \text{:}\) Range (codomain) of \(A \text{:}\)
Figure2.3.2.1. Illustration of how orthonormal vectors \(v_0 \) and \(v_1 \) are transformed by matrix \(A = U \Sigma V \text{.}\)
Next, let us look at how \(A \) transforms any vector with (Euclidean) unit length. Notice that \(x =
\left( \begin{array}{c}
\chi_0 \\ \chi_1
\end{array} \right)\) means that
\begin{equation*}
x = \chi_0 e_0 + \chi_1 e_1 ,
\end{equation*}
where \(e_0 \) and \(e_1 \) are the unit basis vectors. Thus, \(\chi_0 \) and \(\chi_1 \) are the coefficients when \(x \) is expressed using \(e_0 \) and \(e_1 \) as basis. However, we can also express \(x \) in the basis given by \(v_0 \) and \(v_1 \text{:}\)
This is illustrated by the following picture, which also captures the fact that the unit ball is mapped to an oval with major axis equal to \(\sigma_0 = \| A \|_2 \) and minor axis equal to \(\sigma_1 \text{,}\) as illustrated in Figure 2.3.2.1 (bottom).
Finally, we show the same insights for general vector \(x \) (not necessarily of unit length):
\(\R^{2} \text{:}\) Domain of \(A \text{:}\)
\(\R^{2} \text{:}\) Range (codomain) of \(A \text{:}\)
Another observation is that if one picks the right basis for the domain and codomain, then the computation \(A x \) simplifies to a matrix multiplication with a diagonal matrix. Let us again illustrate this for nonsingular \(A \in \R^{2 \times 2} \) with
Now, if we chose to express \(y \) using \(u_0 \) and \(u_1 \) as the basis and express \(x \) using \(v_0 \) and \(v_1 \) as the basis, then
\begin{equation*}
\begin{array}{rcl}
\begin{array}[t]{c}
\underbrace{U U^T} \\
I
\end{array}
y
\amp=\amp
U
\begin{array}[t]{c}
\underbrace{U^T y} \\
\widehat y
\end{array}
=
( u_0^T y ) u_0 + ( u_1^T y ) u_1 \\
\amp=\amp
\left(\begin{array}{c | c}
u_0 \amp u_1
\end{array} \right)
\left( \begin{array}{c}
u_0^T y \\ \hline
u_1^T y
\end{array}\right)
=
U
\begin{array}[t]{c}
\underbrace{
\left( \begin{array}{c}
\widehat \psi_0 \\ \hline
\widehat \psi_1
\end{array}
\right)}\\
\widehat y
\end{array}
\\
\begin{array}[t]{c}
\underbrace{V V^T} \\
I
\end{array}
x
\amp=\amp
V
\begin{array}[t]{c}
\underbrace{V^T x} \\
\widehat x
\end{array}
=
( v_0^T x ) v_0 + ( v_1^T x ) v_1 \\
\amp=\amp
\left(\begin{array}{c | c}
v_0 \amp v_1
\end{array} \right)
\left( \begin{array}{c}
v_0^T x \\ \hline
v_1^T x
\end{array}\right)
=
V
\begin{array}[t]{c}
\underbrace{
\left( \begin{array}{c}
\widehat \chi_0 \\ \hline
\widehat \chi_1.
\end{array}
\right)} \\
\widehat x
\end{array}.
\end{array}
\end{equation*}
If \(y = A x \) then
\begin{equation*}
U
\begin{array}[t]{c}
\underbrace{
U^T y
} \\
\widehat y
\end{array}
=
\begin{array}[t]{c}
\underbrace{
U \Sigma V^T x
} \\
A x
\end{array}
= U \Sigma \widehat x
\end{equation*}
so that
\begin{equation*}
\widehat y = \Sigma \widehat x
\end{equation*}
The above discussion shows that if one transforms the input vector \(x \) and output vector \(y \) into the right bases, then the computation \(y := A x \) can be computed with a diagonal matrix instead: \(\widehat y := \Sigma \widehat x \text{.}\) Also, solving \(A x = y \) for \(x \) can be computed by multiplying with the inverse of the diagonal matrix: \(\widehat x := \Sigma^{-1} \widehat y \text{.}\)
These observations generalize to \(A \in \C^{m \times
n}\text{:}\) If
\begin{equation*}
y = A x
\end{equation*}
then
\begin{equation*}
U^H y = U^H A
\begin{array}[t]{c}
\underbrace{V V^H }\\
I
\end{array}
x
\end{equation*}
so that
\begin{equation*}
\begin{array}[t]{c}
\underbrace{U^H y}\\
\widehat y
\end{array}
= \Sigma
\begin{array}[t]{c}
\underbrace{V^H x}\\
\widehat x
\end{array}
\end{equation*}
