A security camera used to be a tape recorder with a lens. Somebody had to watch the footage, or more often, nobody did until something went wrong. Modern systems are different. They add a layer of software that watches every frame, labels what it sees, follows people and vehicles across scenes, and raises an alert when something looks off.

That software is almost entirely deep learning. This guide explains how it works, piece by piece, without the sales language. It also covers where these systems fail, because the failures matter at least as much as the features.

The basic pipeline

Most AI video analytics follow the same general sequence:

Capture. A camera produces a stream of frames, typically 15 to 30 per second.
Detection. A model finds objects of interest in each frame: people, cars, bags, animals.
Tracking. The system links detections across frames, so it knows that the person in frame 100 is the same one in frame 101.
Recognition or classification. Optional extra steps identify attributes (a red jacket, a delivery van) or, in some systems, who someone is.
Rules and alerts. Business logic decides what to do: count people, flag a vehicle in a restricted zone, notify an operator.

Each step uses a different kind of model, and each has its own ways of going wrong.

Step one: object detection

Detection answers a simple question: what is in this image, and where? The output is a set of boxes, each with a label and a confidence score.

The workhorse here is the convolutional neural network, or CNN. A CNN scans an image with small filters that respond to patterns. Early layers pick up edges and textures. Deeper layers combine those into shapes, and deeper ones still into things like wheels, faces and torsos. After training on huge numbers of labeled images, the network learns which combinations mean "person" and which mean "bicycle."

Detectors come in two broad families. Two-stage designs first propose regions that might contain something, then classify each one. They're often more accurate but slower. Single-stage designs, such as the YOLO family, predict boxes and labels in one pass, which makes them fast enough to run in real time on modest hardware. Newer transformer-based detectors have joined the mix and are competitive on accuracy.

For surveillance, speed matters a lot. A system watching hundreds of cameras can't spend a second per frame.

Step two: tracking

Detection works frame by frame and has no memory. If you want to count how many different people walked through a door, or measure how long someone stood in an area, you need to connect detections over time.

That's multi-object tracking. A classic approach predicts where each tracked object should appear in the next frame, often using a motion model such as a Kalman filter, then matches predictions to new detections. Modern trackers add appearance features, small numerical fingerprints of how each object looks, so they can keep the right identity when two people cross paths or when someone is briefly hidden behind a pillar.

Trackers still make mistakes. Crowded scenes, occlusion, similar clothing and low frame rates all lead to identity switches, where the system quietly swaps two people. That's a small glitch if you're counting foot traffic. It's a serious one if the output is used to claim a particular person did something.

Step three: re-identification across cameras

A single camera sees only part of a building or street. Person re-identification, or re-ID, tries to recognize the same individual across different cameras, even with different angles and lighting.

Re-ID models turn a cropped image of a person into an embedding, a list of numbers that captures appearance such as clothing, build and carried items. Two crops of the same person should produce embeddings that sit close together. Different people should land far apart.

Because re-ID often relies on clothing, it's fragile in a specific way. Someone who changes their jacket can vanish from the system, and two strangers in similar outfits can get merged. It's best seen as a lead generator for a human to check, not as proof.

Step four: face recognition

Face recognition deserves separate treatment because it's the most sensitive piece, both technically and socially.

The pipeline has its own stages. First, a face detector finds faces in the frame. Then the face is aligned and cropped. A deep network converts it into an embedding, usually a vector of a few hundred numbers. Finally, that embedding is compared with a stored set of known faces using a distance measure. If the distance falls below a chosen threshold, the system declares a match.

That threshold is where a lot of trouble hides. Set it loose, and you catch more true matches but also more false ones. Set it strict, and you reduce false matches but miss genuine ones. There's no setting that eliminates both errors.

Two different jobs often get confused:

Verification (one-to-one): "Is this person who they claim to be?" Think of unlocking a phone. It's comparatively easy and the error costs are usually low.
Identification (one-to-many): "Who is this person, out of everyone in the database?" This is much harder, and error rates rise as the database grows, since every added face is another chance for a false match.

Step five: anomaly detection

Another popular claim is that AI can spot "suspicious behavior." In practice there are a few different things hiding under that phrase.

Some systems detect defined events, such as a person entering a zone, a vehicle stopping where it shouldn't, a crowd forming, or a fall. These are rule-based on top of detection and tracking, and they can work reasonably well.

Others use unsupervised anomaly detection. The model learns what normal footage looks like for a given camera, such as typical movement patterns and speeds, then flags what deviates. This catches the unexpected, but "unusual" and "dangerous" are not the same thing. A delivery person using an odd route, a child running, or a street performer can all be statistically strange and perfectly harmless.

Systems that claim to infer intent or emotion from body language rest on much shakier science. Be skeptical of anything that goes beyond describing what is observable.

Where does the computing happen?

Two architectures are common. In cloud processing, cameras send video to remote servers that run the models. This allows large, powerful models but uses bandwidth and raises privacy and security questions about where footage goes. In edge processing, the analysis runs on the camera itself or on a nearby box. It's faster, cheaper to scale, and keeps raw video local, but the models must be small and efficient.

Many real deployments mix the two: lightweight detection on the edge, heavier analysis in the cloud only when needed.

Where these systems fail

Accuracy numbers on a vendor's slide are measured under good conditions. Real life is harsher.

Image quality. Low light, glare, rain, motion blur, compression and long distances all degrade performance, sometimes sharply. Faces captured at an angle or partly covered are much harder.

Dataset bias. A model learns from its training data. If certain groups, environments or camera types were underrepresented, performance can be uneven. Independent evaluations of face recognition, including large benchmark programs run by the U.S. National Institute of Standards and Technology, have found that error rates can differ across demographic groups, with the size of the gap varying a lot between algorithms. Newer systems generally do better, but the variation is a reason to test any specific product on realistic data.

The base-rate problem. This one surprises people. Imagine a system that is 99.9 percent accurate at not falsely flagging an innocent person, and it scans 100,000 faces a day looking for a handful of people on a watchlist. That tiny error rate still produces around a hundred false alerts every day. If real targets are rare, most alerts will be wrong, even with an excellent model.

Automation bias. Operators tend to trust a confident-looking machine. A "92 percent match" label can push a person to accept a result they would otherwise question.

Adversarial tricks and drift. Camera angles change, seasons change, people change how they dress. Models trained on yesterday's conditions slowly lose accuracy unless they're monitored and updated.

Privacy, law and the human side

Surveillance isn't only a technical topic, and rules differ widely by country. Some places restrict or ban certain uses, especially real-time face identification in public spaces. The EU's AI Act, for instance, sets strict limits on remote biometric identification, with narrow exceptions. Several U.S. states and cities regulate biometric data or police use of face recognition. Check current local law rather than relying on a summary.

Beyond legality, there are practical questions any organization should answer before deploying such a system:

What exactly is the purpose, and is video analytics the least intrusive way to achieve it?
Who can see the data, how long is it kept, and how is it secured?
How are errors caught, and who is accountable when a wrong match affects a real person?
Can people find out they're being analyzed, and can they contest a result?

Good practice treats AI output as a prompt for human review, never as a verdict.

What it's actually good at

None of this means the technology is useless. Counting people and vehicles, measuring queues, detecting a fall in an elderly-care setting, noticing smoke or a blocked fire exit, or searching hours of footage for "a red truck at the gate" are all real, valuable uses that don't require identifying anyone. The less a system tries to know about individuals, the safer and more reliable it tends to be.

The short version

AI video surveillance is a stack of deep learning models: detectors that find things, trackers that follow them, embedding models that compare appearances or faces, and rules that turn it all into alerts. Each layer is impressive and each is fallible. The most important question isn't "how smart is the model?" It's "what happens when it's wrong, and who pays for that?"