Every machine learning model, from a simple linear regression to a transformer, is built on a handful of vector operations. Two of them appear almost everywhere: the dot product and the norm. If these ideas feel abstract, this guide turns them into small calculations you can follow by hand. You will see what each operation means geometrically, where machine learning uses it, and how to compute it in NumPy.
What is a vector?
A vector is an ordered list of numbers, for example x = (3, -4). In machine learning a vector usually represents one data point. Each number is a feature, such as the size of a house, the brightness of a pixel or how often a word appears in a document.
You can also picture a vector as an arrow from the origin to a point in space. With two numbers the space is a flat plane, with three it is 3D, and with a thousand numbers it is a space we cannot draw but can still calculate in. Every rule in this guide works the same way in any number of dimensions.
The dot product
The dot product multiplies matching elements of two vectors and adds the results. The answer is a single number, not a vector.
a · b = a₁b₁ + a₂b₂ + … + aₙbₙ
Example with a = (1, 2, 3) and b = (4, 5, 6):
1×4 + 2×5 + 3×6 = 4 + 10 + 18 = 32
Both vectors must have the same length. If they do not, the operation is undefined.
A dot product is a weighted sum
The easiest way to understand the dot product in machine learning is to see it as a weighted sum. Suppose a house is described by three features: size in units of 100 m², number of rooms and age in years. Let x = (1.2, 3, 10). A simple pricing model has weights w = (300, 20, -5) and a bias b = 50.
The prediction is the dot product plus the bias:
w · x + b = 300×1.2 + 20×3 + (-5)×10 + 50 = 360 + 60 - 50 + 50 = 420
Each weight says how much its feature matters and in which direction. A bigger house and more rooms raise the price, while age lowers it. Training a linear model means finding the weights that make these predictions as accurate as possible.
What the number tells you
The dot product also equals ‖a‖ · ‖b‖ · cos(θ), where θ is the angle between the two vectors. That gives you a quick way to read its sign:
- Positive: the vectors point in roughly the same direction (angle below 90°).
- Zero: the vectors are perpendicular, also called orthogonal.
- Negative: the vectors point in roughly opposite directions (angle above 90°).
For example, (1, 0) · (0, 1) = 0, so the two coordinate axes are perpendicular.
Cosine similarity
Divide the dot product by the lengths of both vectors and you get the cosine of the angle:
cos(θ) = (a · b) / (‖a‖ · ‖b‖)
For our vectors this is 32 / (√14 × √77) ≈ 0.975. A value close to 1 means the vectors point almost the same way, 0 means they are unrelated, and -1 means they are opposite. Search engines, recommendation systems and embedding models use cosine similarity to measure how close two items are, because it compares direction and ignores length.
Unit vectors and normalization
A unit vector has length 1. You can turn any vector into a unit vector by dividing it by its L2 norm. For x = (3, -4) the norm is 5, so the unit vector is (0.6, -0.8). Check it: 0.6² + (-0.8)² = 0.36 + 0.64 = 1.
This matters because once two vectors are normalized, their cosine similarity is just their dot product. Many embedding systems store normalized vectors for exactly this reason: it makes similarity search a single fast multiplication.
Norms: measuring the size of a vector
A norm turns a vector into one non-negative number that describes its length. The two you will meet most often are the L1 and L2 norms.
L1 norm (Manhattan distance)
Add the absolute values of all elements:
‖x‖₁ = |x₁| + |x₂| + … + |xₙ|
For x = (3, -4): |3| + |-4| = 7. Imagine walking through a city grid where you can only move along streets: 3 blocks across and 4 blocks up, 7 blocks in total.
L2 norm (Euclidean distance)
Square every element, add the squares, then take the square root:
‖x‖₂ = √(x₁² + x₂² + … + xₙ²)
For x = (3, -4): √(9 + 16) = √25 = 5. This is the straight-line distance from the origin, the Pythagorean theorem extended to any number of dimensions.
Why the two numbers differ
L1 counts every element equally. L2 squares the elements first, so large values weigh much more than small ones. Here is how they compare on a few vectors:
| Vector | L1 norm | L2 norm |
|---|---|---|
| (3, -4) | 7 | 5 |
| (5, 5) | 10 | 7.07 |
| (1, 1, 1, 1) | 4 | 2 |
| (10, 0, 0) | 10 | 10 |
For the same vector, L2 is never larger than L1. The two are equal only when a single element is non-zero.
How the two ideas connect
A vector dotted with itself gives its squared L2 norm: x · x = ‖x‖₂². For (3, -4) that is 9 + 16 = 25, and the square root is 5. So the norm is built from the dot product.
Dot products inside matrix multiplication
Matrix multiplication looks complicated until you notice that every entry of the result is one dot product: the row of the first matrix dotted with the column of the second.
A = [[1, 2], B = [[5, 6],
[3, 4]] [7, 8]]
AB[0][0] = (1, 2) · (5, 7) = 1×5 + 2×7 = 19
AB[0][1] = (1, 2) · (6, 8) = 1×6 + 2×8 = 22
AB[1][0] = (3, 4) · (5, 7) = 3×5 + 4×7 = 43
AB[1][1] = (3, 4) · (6, 8) = 3×6 + 4×8 = 50
So AB = [[19, 22], [43, 50]]. A neural network layer is exactly this operation: many dot products between input vectors and weight vectors, computed at once on a GPU.
Where machine learning uses them
1. Predictions
A linear model predicts with a dot product: ŷ = w · x + b. A single neuron does the same thing and then applies an activation function. When you hear that a model has billions of parameters, most of those parameters sit inside weight matrices that are multiplied with inputs through dot products.
2. Loss functions
- MAE (mean absolute error) follows the L1 idea. It treats all errors equally.
- MSE (mean squared error) follows the L2 idea. It punishes large errors much harder.
If the errors are 1, 1, 1 and 10, then MAE = 13 / 4 = 3.25, while MSE = 103 / 4 = 25.75. The single large error dominates MSE. Use MAE when your data contains outliers you do not want the model to chase, and MSE when large mistakes are genuinely more costly.
3. Regularization
Regularization adds a penalty on the weights w to the loss, so the model stays simple and overfits less.
- L1 (Lasso) adds λ‖w‖₁. It tends to push some weights to exactly zero, which works as automatic feature selection.
- L2 (Ridge) adds λ‖w‖₂². It shrinks all weights smoothly but rarely makes them exactly zero. In neural networks it is often called weight decay.
Take w = (3, -4, 0.5) and λ = 0.1. The L1 penalty is 0.1 × 7.5 = 0.75. The L2 penalty is 0.1 × 25.25 = 2.525. Notice how L2 is dominated by the two large weights, while the small weight 0.5 barely matters.
Why does L1 produce exact zeros? The pull toward zero from an L1 penalty has constant strength no matter how small the weight is, so small weights get pushed all the way to zero. The pull from an L2 penalty is proportional to the weight, so it fades as the weight shrinks and the weight approaches zero without reaching it.
Rule of thumb: start with L2. Reach for L1 when you have many features and expect most of them to be useless, or when you need a sparse model that is easy to explain. Elastic Net combines both penalties.
4. Distance and similarity
k-nearest neighbors and clustering rely on distances between data points. L2 is the default. L1 is sometimes more robust on high-dimensional or sparse data. For text and embeddings, cosine similarity is usually the better choice because it ignores vector length.
Try it in NumPy
import numpy as np
a = np.array([1, 2, 3])
b = np.array([4, 5, 6])
print(np.dot(a, b)) # 32
print(a @ b) # 32, same thing
print(a * b) # [ 4 10 18] element-wise, NOT a dot product
x = np.array([3, -4])
print(np.linalg.norm(x, 1)) # 7.0 (L1)
print(np.linalg.norm(x, 2)) # 5.0 (L2)
unit = x / np.linalg.norm(x)
print(unit) # [ 0.6 -0.8]
cos = a @ b / (np.linalg.norm(a) * np.linalg.norm(b))
print(round(cos, 4)) # 0.9746
A = np.array([[1, 2], [3, 4]])
B = np.array([[5, 6], [7, 8]])
print(A @ B) # [[19 22] [43 50]]
Common mistakes
- Using
a * binstead ofa @ b. The first multiplies element by element and returns a vector. The second returns one number. - Mixing up vector lengths. A dot product needs two vectors of the same length.
- Forgetting to normalize before comparing. Cosine similarity handles it for you, but a raw dot product grows with vector length.
- Regularizing without scaling features. If one feature is measured in thousands and another in fractions, the penalty treats them unfairly. Standardize your features first.
- Treating L1 and L2 as interchangeable. They give different numbers and different model behavior, so always compare values computed with the same norm.
Frequently asked questions
Is the dot product the same as multiplying two numbers? For single numbers, yes. For vectors it multiplies matching positions and then sums, so the result is one number.
Can a norm be negative? No. A norm measures length, so it is always zero or positive, and it is zero only for the zero vector.
Which norm does "the norm" usually mean? When no subscript is written, most people mean the L2 norm.
Why do embeddings use cosine similarity? Direction carries the meaning in an embedding, while length often does not, so comparing angles works better than comparing raw distances.
Quick checklist
- Dot product: multiply matching elements and sum. The result is a number.
- A zero dot product means the vectors are perpendicular.
- The L1 norm sums absolute values. The L2 norm is the straight-line length.
- L2 punishes large values more. L1 is more robust and encourages sparsity.
- Default to L2, and switch to L1 for feature selection or outlier-heavy data.
- Matrix multiplication is a grid of dot products.