I will first start with spectral decomposition and then move on to SVD.

Note: All the images used in the page are from the playlist by Visual Kernel[1]. Highly recommend that you check out his videos.

To truly understand this stuff, I’d have to first understand the following: 1. Symmetric matrix 2. Matrix Decomposition 3. Eigen Vectors 4. Orthogonal eigen vectors.

A symmetric matrix is where a[i,j] = a[j,i] where i != j. A rectangular matrix is never symmetrical.

A transpose of a matrix gives us the same matrix but with the rows and columns interchanged. If b is the transpose of a then a[i,j] = b[j, i]

The interesting thing about symmetric matrices is that the transpose of a symmetric matrix is the same as the original one.

Also, the transpose of an orthogonal matrix gives us its inverse.

Eigen Vectors

Given a matrix A, imagine it as a transformation. There exists a set of vectors upon which when we apply the transform, it just gets scaled and does not change direction. These special vectors whose directions from the origin do not get changed when A is applied on them are called eigen vectors. Every such vector also has an eigen value, the factor by which it was streched or squished during a transform.

While every other vector deviates from its initial direction, the eigen vectors stay on their original line even after applying the transform.

The Eigen vectors Symmetric matrices

The eigen vectors of symmetric matrices are orthogonal. i.e they form an angle of 90 degrees from each other.

Spectral Decomposition

S = QλQ^T

A symmetric matrix S can be expressed as the product of 3 matrices Q, lambda and Q transposed. Where: 1. Q is an orthogonal matrix (rotation) 2. Lambda is a diagonal matrix (streching)

The equation holds when Q is a matrix comprising of the eigen vectors of S.

1. Q rotates the vector to a basis made up of the eigen vectors of S
2. lambda scales it
3. Q transpose rotates it back to it’s original basis i.e the usual x and y axis.

The problem with spectral decomposition is that it works only for symmetric matrices.

Rectangular Matrices

A matrix of shape (m, n) can be thought of as a transform from an n dimensional space to an m dimensional space.

The Dimension Eraser

It’s a rectangular matrix which can be used to remove a specific dimension from a vector. In the example given below, we “erase out” the z axis i.e the last axis. Notice how the first 2 columns of the matrix is basically an identity matrix.

It takes a 3d matrix as input and returns a 2d matrix with the x and the y value preserved.

The Dimension Adder

It takes a 2d matrix as input and returns a 3d matrix with the x and the y value preserved with a new z axis where z=0

Resources

1 - Visual Kernel’s video series on matrices