Self-Host vs API Cost Calculator

An API bill is variable: zero requests cost zero dollars, and the bill grows in a straight line with tokens. A self-hosted deployment is fixed: the GPU is rented by the hour whether it answers one request or a million, so cost per token falls as utilization rises. There is no universally cheaper option — only a cheaper option at a given volume.

That asymmetry is the entire analysis. Below the crossover an API is cheaper because you pay only for what you use; above it, hardware wins because the marginal cost of one more token is nearly zero.

Two people can rent the same H100 at the same hourly price and get wildly different costs per token. The difference is utilization — the share of rented hours actually doing inference. A card busy 20% of the month costs five times more per token than one kept busy, because the bill is identical and only the denominator moved.

This is why self-hosting rarely wins for traffic with a daily peak. Renting for the peak means paying for an idle machine most of the night. Batch work — overnight re-indexing, dataset labelling, synthetic data — is the opposite: it runs flat out, so its utilization approaches 100%.

Throughput is the one input this calculator refuses to guess, because it varies by an order of magnitude between a stock single-stream setup and a properly batched engine. A 7B model can answer one chat stream at a few dozen tokens per second and serve hundreds in aggregate once requests are batched. Both numbers are true; they answer different questions.

Three things move it: the engine (vLLM, TensorRT-LLM and llama.cpp trade off differently), the batch size the workload can sustain, and the model itself — a sparse mixture-of-experts model decodes far faster than a dense model of the same total size. That is why the VRAM calculator reports a bandwidth ceiling and warns that real throughput lands 50–80% below it.

• APIs win below the crossover, for spiky traffic, for prototypes, for anything with a compliance budget instead of an ops team, and whenever the frontier model you need is one you cannot host at all.
• Self-hosting wins at steady high volume, on batch pipelines, when the data cannot leave your network, when you need a specific fine-tune you own, and when you already employ someone who can run a serving stack.
• Both win in the hybrid pattern most teams settle on: a small self-hosted model for the bulk of cheap classification and extraction, with an API for the rare hard request.

A word on quality: the API model and the open model are almost never the same model. A frontier model and a self-hosted 7B are not substitutes for every task, so price the gap deliberately instead of assuming the cheap option is good enough.

The GPU hourly rate is the visible part. The invisible part is a person: someone has to pick a serving engine, tune batching, watch for out-of-memory crashes at 3 a.m. and keep the thing patched. That is a real salary, and it does not shrink because the instance is small.

• Engineering time — the largest hidden line item, and what the extra-cost field exists for.
• Storage and egress — model weights are tens of gigabytes, and moving them between regions is not free.
• Idle capacity — you provision for the peak and pay for it through the trough.
• Cold starts — a serverless GPU that loads 15 GB of weights before answering is not comparable to a warm API.
• Rate limits — an API that throttles you at the wrong moment has a cost too, just not on the invoice.
• Privacy and control — sometimes worth paying for, and sometimes the entire reason for the exercise.