Le neurone artificiel

Linear regression outputs any number at all. Most useful questions are yes/no instead: is this email spam, is this scan suspicious, should the car brake? To answer those we glue a step onto the end of the line.

Weighted sum, then non-linearity — that is the atom of every neural network in existence. Stack millions of them and you get GPT. The atom itself never changes.

In z = Σwᵢxᵢ + b, the bias shifts the decision boundary away from the origin. Think of a smoke detector: it does not alarm on the first smoked particle, it waits for the signal to cross a threshold. The bias sets that threshold.

• Large negative bias → needs strong evidence before firing (conservative neuron).
• Large positive bias → fires easily (trigger-happy neuron).
• No bias at all → every decision boundary is forced through the origin, a needless restriction.

The step function looks perfect for decisions, but its gradient is zero everywhere, so gradient descent has nothing to follow. The 1958-era fix was to smooth the step: the sigmoid σ(z) = 1/(1+e⁻ᶻ) is a differentiable ramp from 0 to 1.

A single neuron draws one straight decision boundary. AND and OR are linearly separable: a single line cuts them cleanly. XOR is not — no straight line can separate its positives from its negatives.

The fix is the Multi-Layer Perceptron: stack neurons into layers. Stacking only helps because of the activation function from Card 3 — compose two linear functions and you get another linear function, but compose two ReLUs and you get a bendable piecewise surface that can carve out any shape.

type Neuron = { weights: number[]; bias: number };

const relu = (z: number) => Math.max(0, z);

export function forward(neuron: Neuron, inputs: number[], activate = relu): number {
  if (inputs.length !== neuron.weights.length) {
    throw new Error('shape mismatch: one weight per input');
  }
  let z = neuron.bias;
  for (let i = 0; i < inputs.length; i++) {
    z += neuron.weights[i]! * inputs[i]!;
  }
  return activate(z);
}

A layer is many neurons reading the same inputs. Arrange the weights as a matrix and the loop becomes a single matrix multiply: Z = X·W + b. This is exactly the operation GPUs were built to accelerate — millions of tiny weighted sums per millisecond.

• Inputs → weights → sum → bias → activation → output.
• The same five-step shape repeats in every layer of every modern network.
• Weights are learned; the activation function is a design choice you make.
• Non-linearity between layers is what makes depth pay off.