NeuralPath // AI & ML Learning Hub

The Core Thesis: A matrix is not just a passive spreadsheet of numbers—it is an active machine that transforms space. Every time you multiply a vector by a matrix ( $\mathbf{y} = \mathbf{A}\mathbf{x}$ ), you are stretching, rotating, or projecting the entire coordinate grid to extract new features from raw data.

What is a Linear Transformation?

In algebra, a function $f(x)$ takes a scalar number as input and outputs another number. In Linear Algebra, a Transformation $T(\mathbf{x})$ is a function that takes an entire input vector $\mathbf{x}$ and maps it to a new output vector $\mathbf{y}$ :

T(\mathbf{x}) = \mathbf{A}\mathbf{x} = \mathbf{y}

Why is it called Linear? Visually, a transformation is strictly linear if and only if it obeys two geometric rules:

Rule 1: Grid Lines Remain Parallel

All straight lines in space stay straight after the transformation—they never curve or bend. Furthermore, parallel grid lines remain parallel and evenly spaced.

T(\mathbf{u} + \mathbf{v}) = T(\mathbf{u}) + T(\mathbf{v})

Rule 2: The Origin Remains Fixed

The center of the coordinate space $(0, 0)$ never shifts. If you pass a zero vector into a matrix multiplication, you always get the zero vector back.

T(c\mathbf{u}) = cT(\mathbf{u}) \quad \text{and} \quad \mathbf{A}\mathbf{0} = \mathbf{0}

AI Connection (Why Neural Networks Need Bias $\mathbf{b}$ ): Because pure matrix multiplication $\mathbf{W}\mathbf{x}$ locks the origin at $(0,0)$ , a neural network layer adds a bias vector $\mathbf{b}$ to create an Affine Transformation ( $\mathbf{y} = \mathbf{W}\mathbf{x} + \mathbf{b}$ ), allowing the decision boundary to shift away from the origin.

The Secret of Basis Vectors: Reading Any Matrix Instantly

How can you look at a $2 \times 2$ matrix like $\begin{bmatrix} 3 & 1 \\ 0 & 2 \end{bmatrix}$ and immediately see what it does to space? The secret lies in the Standard Basis Vectors:

Horizontal Unit Vector (

\hat{i}

)

Points 1 unit right along the X-axis

\hat{i} = \begin{bmatrix} 1 \\ 0 \end{bmatrix}

Vertical Unit Vector (

\hat{j}

)

Points 1 unit up along the Y-axis

\hat{j} = \begin{bmatrix} 0 \\ 1 \end{bmatrix}

Every vector $\mathbf{x} = \begin{bmatrix} x \\ y \end{bmatrix}$ is just a sum of scaled basis vectors: $x\hat{i} + y\hat{j}$ . When you multiply a matrix $\mathbf{A} = \begin{bmatrix} a & b \\ c & d \end{bmatrix}$ by $\mathbf{x}$ , look at what happens to the columns:

\begin{bmatrix} a & b \\ c & d \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} = x \begin{bmatrix} a \\ c \end{bmatrix} + y \begin{bmatrix} b \\ d \end{bmatrix}

Column 1 $\begin{bmatrix} a \\ c \end{bmatrix}$ is literally where the X-axis basis vector $\hat{i} = [1, 0]^T$ lands!

Column 2 $\begin{bmatrix} b \\ d \end{bmatrix}$ is literally where the Y-axis basis vector $\hat{j} = [0, 1]^T$ lands!

⚡ Knowledge Check

A 2D linear transformation moves the horizontal basis vector $\hat{i} = [1, 0]^T$ to $[2, -1]^T$ , and moves the vertical basis vector $\hat{j} = [0, 1]^T$ to $[0, 4]^T$ . Without doing any algebra, what is the $2 \times 2$ transformation matrix $\mathbf{A}$ ?

A) First col [2, -1], Second col [0, 4]▼

✓ Correct!The columns of any transformation matrix are simply the landing coordinates of the basis vectors

\hat{i}

and

\hat{j}

, giving

\mathbf{A} = \begin{bmatrix} 2 & 0 \\ -1 & 4 \end{bmatrix}

B) First row [2, -1], Second row [0, 4]▼

✕ Incorrect.Transformed basis vectors always populate the columns of the matrix, not the rows.

The Visual Catalog of Geometric Matrix Transformations

By changing the entries of a matrix $\mathbf{A}$ , we can perform specific geometric operations on data points (such as image augmentation in Computer Vision or feature projection in Transformers):

1. Scaling (Stretch / Shrink)Diagonal Matrix

Stretches X by factor $s_x$ and Y by factor $s_y$ . Off-diagonal zeros mean axes do not mix.

\mathbf{A} = \begin{bmatrix} s_x & 0 \\ 0 & s_y \end{bmatrix} = \begin{bmatrix} 2 & 0 \\ 0 & 0.5 \end{bmatrix}

2. Shear (Slanting Space)Upper Triangular

Keeps $\hat{i}$ fixed at $[1,0]^T$ , but pushes $\hat{j}$ sideways to $[k, 1]^T$ , slanting the grid like italics.

\mathbf{A} = \begin{bmatrix} 1 & k \\ 0 & 1 \end{bmatrix} = \begin{bmatrix} 1 & 1.5 \\ 0 & 1 \end{bmatrix}

3. Counter-Clockwise RotationOrthogonal Matrix

Rotates every vector by angle $\theta$ around the origin without changing lengths (powers RoPE in LLMs!).

\mathbf{R}(\theta) = \begin{bmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{bmatrix}

4. Reflection & ProjectionRank-Deficient

Flips space across an axis or squashes 2D space onto a 1D line by zeroing out a dimension.

\mathbf{P}_{\text{x-axis}} = \begin{bmatrix} 1 & 0 \\ 0 & 0 \end{bmatrix}

Two Ways to Compute Matrix Multiplication

When multiplying an $(m \times k)$ matrix $\mathbf{A}$ by a $(k \times n)$ matrix $\mathbf{B}$ to produce $\mathbf{C}$ of shape $(m \times n)$ , mathematicians and AI engineers use two complementary mental models:

1. The Row-by-Column Dot Product View

Each individual cell $c_{i,j}$ is computed by taking the dot product of Row $i$ from $\mathbf{A}$ and Column $j$ from $\mathbf{B}$ .

c_{i,j} = \sum_{p=1}^{k} a_{i,p} b_{p,j}

Best for: Computing single entries by hand and checking feature-to-neuron similarity scores.

2. The Column Combination View

Each column of $\mathbf{C}$ is a linear combination of the columns of $\mathbf{A}$ , weighted by the numbers in that column of $\mathbf{B}$ .

\mathbf{A}\mathbf{b}_j = b_{1,j}\mathbf{a}_{:,1} + b_{2,j}\mathbf{a}_{:,2} + \dots + b_{k,j}\mathbf{a}_{:,k}

Best for: Understanding how Attention blends Value vectors and how Low-Rank Adaptation (LoRA) works.

Step-by-Step Worked Example (Both Views Yield the Same Answer):

Multiply

\mathbf{A} = \begin{bmatrix} 1 & 3 \\ 2 & 4 \end{bmatrix}

by vector

\mathbf{x} = \begin{bmatrix} 5 \\ 2 \end{bmatrix}

Row Dot-Product Method

\begin{bmatrix} (1\cdot 5 + 3\cdot 2) \\ (2\cdot 5 + 4\cdot 2) \end{bmatrix} = \begin{bmatrix} 5 + 6 \\ 10 + 8 \end{bmatrix} = \begin{bmatrix} 11 \\ 18 \end{bmatrix}

Column Combination Method

5 \begin{bmatrix} 1 \\ 2 \end{bmatrix} + 2 \begin{bmatrix} 3 \\ 4 \end{bmatrix} = \begin{bmatrix} 5 \\ 10 \end{bmatrix} + \begin{bmatrix} 6 \\ 8 \end{bmatrix} = \begin{bmatrix} 11 \\ 18 \end{bmatrix}

Matrix Multiplication as Function Composition

Why do we multiply two matrices $\mathbf{B}$ and $\mathbf{A}$ together? Because if matrix $\mathbf{A}$ applies one transformation (say, a Rotation) and matrix $\mathbf{B}$ applies a second transformation (say, a Shear), then the product matrix $\mathbf{C} = \mathbf{B}\mathbf{A}$ is a single combined transformation that performs both in sequence!

Right-to-Left Execution Order (Just Like Nested Functions)

\mathbf{y} = \mathbf{B}(\mathbf{A}\mathbf{x}) = (\mathbf{B}\mathbf{A})\mathbf{x}

Because $\mathbf{x}$ is on the right, Matrix $\mathbf{A}$ acts on $\mathbf{x}$ FIRST, and Matrix $\mathbf{B}$ acts on the result SECOND.

This immediately explains why matrix multiplication is NOT commutative ( $\mathbf{B}\mathbf{A} \neq \mathbf{A}\mathbf{B}$ ): rotating an image by $90^\circ$ and then shearing it produces a completely different geometric result than shearing it first and rotating it second. However, it IS associative: $(\mathbf{C}\mathbf{B})\mathbf{A} = \mathbf{C}(\mathbf{B}\mathbf{A})$ .

⚡ Knowledge Check

Suppose you stack 3 linear neural network layers $\mathbf{W}_1$ , $\mathbf{W}_2$ , and $\mathbf{W}_3$ directly after one another WITHOUT any non-linear activation functions (like ReLU) in between: $\mathbf{y} = \mathbf{W}_3(\mathbf{W}_2(\mathbf{W}_1\mathbf{x}))$ . What happens mathematically?

A) All 3 layers collapse into a single matrix W_total▼

✓ Correct!By associativity,

\mathbf{W}_3(\mathbf{W}_2(\mathbf{W}_1\mathbf{x})) = (\mathbf{W}_3\mathbf{W}_2\mathbf{W}_1)\mathbf{x} = \mathbf{W}_{\text{single}}\mathbf{x}

. Without non-linear activations (ReLU/GELU) between matrices, a 100-layer deep network has the exact same expressive power as a 1-layer linear model!

B) It learns complex non-linear curves▼

✕ Incorrect.Multiplying linear matrices together always produces another strictly linear matrix. You must insert non-linear activation functions between matrix multiplications to bend space and learn curves.

Dimensionality Changes: Expanding and Compressing Space

Matrices do not have to be square! A non-square matrix $\mathbf{W}$ of shape $(m \times n)$ maps vectors from $n$ -dimensional input space ( $\mathbb{R}^n$ ) into $m$ -dimensional output space ( $\mathbb{R}^m$ ):

Tall Matrix (

m &gt; n

): Up-Projection

Maps low-dimensional inputs into a higher-dimensional feature space. In Transformer Feed-Forward networks, hidden vectors of size $768$ are multiplied by a $(3072 \times 768)$ matrix to expand into $3072$ dimensions.

Shape: (3072, 768) · (768, 1) = (3072, 1)

Wide Matrix (

m &lt; n

): Down-Projection

Compresses high-dimensional vectors down into a lower-dimensional bottleneck (used in Autoencoders, PCA, and Multi-Head Attention query projections).

Shape: (64, 768) · (768, 1) = (64, 1)

Determinants, Rank, and Invertibility

When a square matrix $\mathbf{A}$ transforms space, how much does the area (or volume) of a region scale? That scaling factor is called the Determinant, written $\det(\mathbf{A})$ :

\det\left(\begin{bmatrix} a & b \\ c & d \end{bmatrix}\right) = ad - bc

\det(\mathbf{A}) \neq 0

(Full Rank)

Space is stretched or rotated without being squashed flat. The transformation is reversible via the Inverse Matrix $\mathbf{A}^{-1}$ ( $\mathbf{A}^{-1}\mathbf{A} = \mathbf{I}$ ).

\det(\mathbf{A}) &lt; 0

(Flipped Space)

The orientation of space is inverted (like flipping a sheet of paper over), while areas scale by $|\det(\mathbf{A})|$ .

\det(\mathbf{A}) = 0

(Singular)

Columns are linearly dependent! Space is squashed into a lower dimension (e.g., 2D plane collapsed onto a 1D line). No inverse exists; information is permanently lost.

⚡ Knowledge Check

Consider the matrix $\mathbf{A} = \begin{bmatrix} 2 & 4 \\ 1 & 2 \end{bmatrix}$ . Notice that Column 2 is simply $2 \times$ Column 1. What is $\det(\mathbf{A})$ , and can you recover an input vector $\mathbf{x}$ after multiplying $\mathbf{y} = \mathbf{A}\mathbf{x}$ ?

A) det(A) = 0; Impossible to invert (Information Lost)▼

✓ Correct!

\det(\mathbf{A}) = (2)(2) - (4)(1) = 0

. Because both basis vectors land on the exact same line, the entire 2D plane is crushed onto a 1D line. Infinitely many inputs

\mathbf{x}

map to the same output

\mathbf{y}

, so

\mathbf{A}^{-1}

does not exist.

B) det(A) = 8; Fully invertible via A^-1▼

✕ Incorrect.Remember the formula is

ad - bc = (2 \times 2) - (4 \times 1) = 0

, not

ad + bc

. Whenever one column is a multiple of another, the determinant is zero and the matrix is non-invertible (singular).

Visual Explanation: Neural Layers as Space Warping

What is a Deep Neural Network actually doing geometrically? Each layer applies a Linear Transformation ( $\mathbf{W}$ ) to rotate and stretch the data space, shifts it with a Bias ( $\mathbf{b}$ ), and folds it with a Non-Linear Activation ( $\sigma$ ) until tangled classes become cleanly separable by a flat hyperplane:

Modern AI Mechanism	Linear Transformation Formula	Geometric Purpose
Dense Layer (nn.Linear)	y = x @ W.T + b	Projects $d_{\text{in}}$ features into $d_{\text{out}}$ learned representations.
Transformer Q, K, V Projections	Q = X @ W_Q, K = X @ W_K, V = X @ W_V	Maps a single token embedding into 3 distinct semantic subspaces (Query, Key, Value).
Rotary Position Embedding (RoPE)	q_rot = R(m * theta) @ q	Rotates token vectors in 2D planes by an angle proportional to sequence position $m$ .
LoRA (Low-Rank Adaptation)	W_new = W_frozen + (B @ A)	Compresses weight updates through a tiny rank- $r$ bottleneck ( $d \to r \to d$ ) to fine-tune LLMs cheaply.

Python Implementation: Geometric Transformations & Projections

Below is a complete NumPy implementation showing how to rotate vectors in 2D space, check matrix invertibility via the determinant, and project high-dimensional embeddings into a lower subspace:

linear_transformations.pyPython 3 · NumPy

import numpy as np # 1. Geometric 2D Rotation Matrix (Rotate 90 degrees counter-clockwise)theta = np.radians(90)R = np.array([    [np.cos(theta), -np.sin(theta)],    [np.sin(theta),  np.cos(theta)]]) v = np.array([3.0, 0.0])                    # Vector pointing along X-axisv_rotated = R @ v                           # Apply linear transformationprint("Rotated Vector:", np.round(v_rotated, 2))  # Output: [0. 3.] (Now on Y-axis!) # 2. Determinant & Matrix Inverse CheckA = np.array([[2.0, 1.0], [1.0, 3.0]])det_A = np.linalg.det(A)print("Determinant of A:", round(det_A, 2))       # Output: 5.0 (Invertible!) A_inv = np.linalg.inv(A)                    # Undo transformation: A_inv @ (A @ v) == vrecovered_v = A_inv @ (A @ v)print("Recovered Original Vector:", recovered_v)  # Output: [3. 0.] # 3. Dimensionality Projection (Transformer Query Projection)# Batch of 4 tokens, each a 512-D embedding -> Project down to 64-D headX_tokens = np.random.randn(4, 512)W_Query = np.random.randn(512, 64) * 0.02Q_projected = X_tokens @ W_Query            # Shape: (4, 512) @ (512, 64) -> (4, 64)print("Projected Query Shape:", Q_projected.shape)

Pro Tip (Convention Difference): In math textbooks, a single vector $\mathbf{x}$ is a column of shape $(d_{\text{in}}, 1)$ , so we write $\mathbf{y} = \mathbf{W}\mathbf{x}$ . In PyTorch and NumPy, a batch of data $\mathbf{X}$ stores samples as rows of shape $(\text{Batch}, d_{\text{in}})$ , so we multiply on the right: Y = X @ W.T!

Key Points

✓Every matrix

\mathbf{A}

represents a linear transformation that maps input vectors to output vectors while keeping grid lines straight, parallel, and evenly spaced.

✓The columns of any transformation matrix tell you the exact coordinates where the basis unit vectors (

\hat{i}, \hat{j}, \dots

) land after transformation.

✓Multiplying two matrices (

\mathbf{C} = \mathbf{B}\mathbf{A}

) composes their geometric transformations from right to left:

\mathbf{A}

transforms space first, followed by

\mathbf{B}

✓Non-square matrices

(m \times n)

change the dimensionality of data, projecting vectors between

\mathbb{R}^n

and

\mathbb{R}^m

(powering embeddings, bottlenecks, and Attention heads).

✓The determinant

\det(\mathbf{A})

measures how much a square matrix scales area or volume; if

\det(\mathbf{A}) = 0

, space collapses into a lower dimension and information is irreversibly lost.

✓Deep neural networks alternate between linear matrix transformations (

\mathbf{W}\mathbf{x} + \mathbf{b}

) and non-linear activations (ReLU/GELU) so they do not collapse into a single linear matrix.

Common Mistakes

✕ Reading matrix transformations from left to right instead of right to left.

In the expression $\mathbf{C}\mathbf{B}\mathbf{A}\mathbf{x}$ , matrix $\mathbf{A}$ multiplies $\mathbf{x}$ first, then $\mathbf{B}$ , and finally $\mathbf{C}$ . Reversing this order produces an entirely different geometric transformation.

✕ Stacking linear layers without non-linear activation functions.

Because matrix multiplication is associative, stacking $\mathbf{W}_2(\mathbf{W}_1\mathbf{x})$ without an activation function in between simply collapses into a single matrix $\mathbf{W}_{\text{combined}}\mathbf{x}$ , preventing the network from learning non-linear patterns.

✕ Calling np.linalg.inv(A) directly in production ML code.

Explicitly inverting large matrices is slow ( $\mathcal{O}(n^3)$ ) and numerically unstable when $\det(\mathbf{A})$ is close to zero. Always use linear system solvers like np.linalg.solve(A, b) or gradient-based optimization instead.

✕ Confusing mathematical column-vector notation (W @ x) with batched row-vector code (X @ W.T).

Textbooks treat a single sample as a column vector $(d, 1)$ , whereas PyTorch and NumPy stack batches as rows $(B, d)$ . Always check whether your weight matrix needs to be transposed when moving from paper equations to code.

The Big Picture

Static Spreadsheet View (Arithmetic)

Grid of Numbers → Tedious Row-by-Column Sums → Another Grid of Numbers

Geometric Transformation View (Deep Learning)

Input Vector Space → Rotate, Scale & Project via Matrix W → Separable Feature Space

The important conceptual shift is realizing that training a neural network—whether a simple classifier or a trillion-parameter Transformer—is literally the process of learning the exact matrix entries needed to warp messy, tangled input vectors into a clean geometric space where answers are easy to separate.

Remember: Whenever you see a weight matrix $\mathbf{W}$ inside a neural network diagram, do not just see a table of parameters. See a geometric lens that rotates, compresses, or expands high-dimensional space to spotlight the features that matter.

Matrix Multiplication and Linear Transformations

What is a Linear Transformation?

The Secret of Basis Vectors: Reading Any Matrix Instantly

The Visual Catalog of Geometric Matrix Transformations

Two Ways to Compute Matrix Multiplication

Step-by-Step Worked Example (Both Views Yield the Same Answer):

Matrix Multiplication as Function Composition

Dimensionality Changes: Expanding and Compressing Space

Determinants, Rank, and Invertibility

Visual Explanation: Neural Layers as Space Warping

Python Implementation: Geometric Transformations & Projections

Key Points

Common Mistakes

The Big Picture