Linear algebra with NumPy

Advanced

Introduction

Linear algebra is the mathematics of vectors and matrices, and it runs underneath data science: a linear regression is a matrix equation, a neural network layer is a matrix multiplication, and a recommendation system compares vectors with dot products. NumPy's @ operator and the np.linalg module make these one-line operations, as long as you know which product you are asking for.

How it works

Dot product. For two vectors of the same length, np.dot(u, v) (or u @ v) multiplies matching elements and adds the results: [1, 2, 3] · [4, 5, 6] = 4 + 10 + 18 = 32. It measures how much two vectors point the same way; it is 0 for perpendicular vectors. Vectors of different lengths raise a ValueError ("shapes ... not aligned").

Matrix multiplication. A @ B makes each entry (i, j) the dot product of row i of A with column j of B. The shape rule: (r, k) @ (k, c) gives (r, c); the inner sizes must match, otherwise NumPy raises a ValueError. This is not the same as A * B, which multiplies element by element:

A = np.array([[1, 2], [3, 4]])
B = np.array([[5, 6], [7, 8]])
print(A @ B)
# [[19 22]
#  [43 50]]
print(A * B)
# [[ 5 12]
#  [21 32]]

Order matters: A @ B and B @ A are usually different.

Identity, determinant and inverse. np.eye(n) is the n × n identity matrix: A @ np.eye(2) equals A. np.linalg.det(A) gives the determinant; for a 2 × 2 matrix [[a, b], [c, d]] it is ad − bc, here 1 × 4 − 2 × 3 = −2 (NumPy prints a float close to −2, with a tiny rounding error). np.linalg.inv(A) gives the inverse, which exists only when the determinant is not 0.

Solving equations. The system 2x + y = 3, x + 3y = 5 is A·x = b with A = [[2, 1], [1, 3]] and b = [3, 5]:

x = np.linalg.solve(np.array([[2, 1], [1, 3]]), np.array([3, 5]))
print(x)        # [0.8 1.4]

solve is faster and more accurate than inv(A) @ b. If A is singular (determinant 0), it raises LinAlgError: Singular matrix: the equations are either contradictory or repeat each other. For a 2 × 2 system, Cramer's rule gives the same answer by hand: x = (e·d − b·f) ÷ det and y = (a·f − e·c) ÷ det, for equations ax + by = e and cx + dy = f.

Example

# Linear algebra modelled with lists: dot product, matrix product and a 2 x 2 solve.
def dot(u, v):
    return sum(a * b for a, b in zip(u, v))


A = [[1, 2], [3, 4]]
B = [[5, 6], [7, 8]]
cols = list(zip(*B))
print("dot([1, 2, 3], [4, 5, 6]):", dot([1, 2, 3], [4, 5, 6]))
print("A @ B:", [[dot(row, col) for col in cols] for row in A])

# Solve 2x + y = 3 and x + 3y = 5 with Cramer's rule.
a, b, c, d = 2, 1, 1, 3
e, f = 3, 5
det = a * d - b * c
print("det:", det, " x:", (e * d - b * f) / det, " y:", (a * f - e * c) / det)
Open in compiler

Line by line

  1. dot adds the products of matching elements.
  2. zip(*B) gives the columns of B, so each entry of A @ B is a row of A dotted with a column of B.
  3. Cramer's rule solves the 2 × 2 system using the determinant.

Expected result

dot([1, 2, 3], [4, 5, 6]): 32
A @ B: [[19, 22], [43, 50]]
det: 5  x: 0.8  y: 1.4

In the real world

Least-squares regression solves the matrix equation (XᵀX)β = Xᵀy. Each layer of a neural network computes X @ W + b. Search engines and recommenders compare items by dot products of their vectors, and 3-D graphics rotate and project every point with matrix multiplications.

Common mistakes

  • Using * when @ was meant, which silently gives element-wise products for square matrices.
  • Getting the order wrong: A @ B is not B @ A.
  • Mismatched inner sizes, such as (2, 3) @ (2, 3), which raises an error.
  • Computing inv(A) @ b instead of solve, which is slower and less accurate.

Best practices

  • Write the shapes next to every matrix expression: (r, k) @ (k, c) → (r, c).
  • Use np.linalg.solve for systems and check the determinant or catch LinAlgError for singular cases.
  • Compare float results with np.allclose, not ==.

Summary

The dot product adds products of matching elements. A @ B multiplies matrices row by column, with (r, k) @ (k, c) giving (r, c), unlike the element-wise *. np.eye gives the identity, det and inv describe and invert square matrices, and np.linalg.solve solves A·x = b unless A is singular.

Quick check

1. What is the shape of (3, 4) @ (4, 2)?
2. What is np.dot([1, 2], [3, 4])?
3. For square matrices A and B, A * B computes…
4. np.linalg.solve raises LinAlgError when…

Key takeaways

  • * multiplies element by element; @ is matrix multiplication.
  • (r, k) @ (k, c) gives (r, c); the inner sizes must match.
  • Use np.linalg.solve(A, b) rather than computing the inverse.

Practice problems for this lesson

My notes

Write down what you want to remember from this lesson. Notes are saved on this device as you type, and appear on your dashboard.