Cloud GPU rental for LLM hosting runs from roughly $0.3–$0.5 per hour for an RTX 4090 to $2–$3 per hour for an H100, with monthly committed rates typically 30–50% lower. The right choice depends less on raw speed than on whether the model plus its KV-cache fits in VRAM: a 7B model fits on one RTX 4090, a 70B model effectively requires multiple A100s or H100s. Below those thresholds, renting by the hour is cheap; above them, memory capacity dictates the bill.
Why GPU Choice Dominates LLM Hosting Costs
Hosting a large language model (LLM) means running inference: the model takes a prompt, processes it, and generates tokens one at a time. Almost all of that work happens in matrix multiplications inside the GPU, so the GPU is the component that sets both the speed and the price of the service. CPU, RAM, and disk matter for loading the model once, but they are rounding errors next to the accelerator's hourly rate.
Two GPU properties drive cost. The first is memory capacity (VRAM, the GPU's own high-bandwidth memory), because the model's weights must sit in VRAM for fast access. The second is memory bandwidth, the rate at which the GPU can read those weights. During token generation the GPU re-reads the weights for every token, so a model that fits but has low bandwidth produces tokens slowly, and you pay for idle GPU time while it grinds.
The practical consequence is a step function rather than a smooth curve. A model either fits on a given GPU or it doesn't; if it doesn't, you rent two or more GPUs, and the hourly cost multiplies. Choosing the GPU is therefore the single decision that moves the monthly bill by the largest factor, typically far more than provider choice or instance tuning.
H100 vs A100 vs RTX 4090: Specs and VRAM Limits
The H100 is NVIDIA's current data-center flagship, the A100 is its widely deployed predecessor, and the RTX 4090 is a consumer graphics card. All three run the same CUDA software stack, but they differ sharply in memory. The H100 comes in 80GB versions (and larger multi-die variants); the A100 comes in 40GB and 80GB versions; the RTX 4090 has 24GB. Data-center cards also support high-speed interconnects (NVLink) that let multiple GPUs share memory traffic efficiently, while consumer cards communicate over the slower PCIe bus.
Bandwidth follows the same hierarchy: H100 is the fastest, A100 next, RTX 4090 lowest of the three for its class, though still substantial. In practice this means the 4090 is excellent value for small models and short contexts, while the A100 and H100 pull ahead on large models and long prompts, where every token requires streaming tens of gigabytes of weights and cache.
A rough memory rule of thumb for weights alone, assuming 16-bit precision (2 bytes per parameter):
VRAM_weights_GB ≈ params_B × 2
7B → ~14 GB
13B → ~26 GB
70B → ~140 GB
Quantization (storing weights in 8 or 4 bits) cuts these figures roughly by half or three-quarters, which is why a 13B model can run on a 24GB RTX 4090 in 4-bit form. But weights are only part of the story: the KV-cache (see section 4) adds memory that grows with context length and concurrent users, and it is often what pushes a model over the edge.
Cloud Rental Price Ranges per Hour and per Month
Cloud GPU pricing varies by provider, contract length, and whether you rent a whole machine or a slice. The ranges below are estimates derived from publicly listed on-demand rates across major GPU clouds. Treat them as planning numbers, not quotes.
| GPU | VRAM | Typical on-demand $/hr | Typical $/month (730 hrs) | Spot / committed discount |
|---|---|---|---|---|
| RTX 4090 | 24 GB | $0.30–$0.80 | ~$220–$580 | 30–60% off spot |
| A100 40 GB | 40 GB | $1.00–$2.00 | ~$730–$1,460 | 30–50% off |
| A100 80 GB | 80 GB | $1.50–$2.50 | ~$1,100–$1,825 | 30–50% off |
| H100 80 GB | 80 GB | $2.00–$4.00 | ~$1,460–$2,920 | 30–60% off |
A monthly figure is simply hourly × 730 (hours in a month). Most providers bill per second or per minute, so short experiments cost far less than a full month.
Cost per Token: Matching GPU to Model Size and KV-Cache
To compare GPUs fairly, compute cost per million output tokens:
cost_per_1M_tokens = (hourly_price / tokens_per_hour) * 1_000_000
tokens_per_hour ≈ throughput_tokens_per_sec * 3600
Throughput depends on model size, batch size, and quantization. For a single user (batch size 1), a rough rule of thumb for a 7B model in 16-bit is 30–60 tokens/sec on an RTX 4090, 50–90 on an A100, and 80–150 on an H100. Larger batches raise total throughput substantially.
Worked example (7B model, 16-bit, batch 1): assume 60 tok/s on an RTX 4090 at $0.50/hr, and 120 tok/s on an H100 at $3.00/hr.
4090: 60 * 3600 = 216,000 tok/hr → $0.50 / 216,000 * 1e6 ≈ $2.31 per 1M tokens
H100: 120 * 3600 = 432,000 tok/hr → $3.00 / 432,000 * 1e6 ≈ $6.94 per 1M tokens
Here the 4090 wins on raw cost per token for a small model, but its 24 GB cap limits batch size and context length. For a 70B model in 16-bit (~140 GB of weights), a single 80 GB card cannot hold it without quantization or multi-GPU sharding.
KV-cache (key-value cache: stored attention states that grow with context length and batch) consumes memory beyond the weights. Estimate it with:
kv_bytes ≈ 2 * layers * heads * head_dim * seq_len * batch * bytes_per_value
Use this checklist before renting:
- Model weights fit in VRAM at chosen precision (16-bit ≈ 2 bytes/param; 8-bit ≈ 1 byte).
- KV-cache headroom: reserve 20–40% of VRAM for context and batch.
- Quantization (e.g., 4-bit) fits the model but may lower quality—test first.
- Compare cost per 1M tokens against API pricing for your expected volume.
Frequently Asked Questions
How much does it cost to rent an H100, A100 or RTX 4090 per hour in the cloud?
Expect roughly $2–$4 per hour for an H100, $1–$2.50 for an A100, and $0.30–$0.80 for an RTX 4090, though these are estimates that swing widely with provider, region, contract length and whether you pick on-demand or spot pricing. Monthly reservations usually discount the hourly rate by 30–50% compared with on-demand. Treat any single figure as a starting point and verify against current provider listings, since GPU rental prices move often.
Which GPU should I rent to host a 7B, 13B or 70B LLM?
A 7B model in 16-bit precision needs about 14 GB just for weights, so an RTX 4090's 24 GB fits it comfortably; a 13B model lands near 26 GB and generally needs an A100 40 GB or quantized weights. A 70B model needs roughly 140 GB in 16-bit, which means multi-GPU setups such as two or more A100 80 GB or H100 80 GB cards, or aggressive quantization on fewer. Always add headroom for KV-cache and activations on top of the weight estimate.
Is an RTX 4090 enough for LLM inference, and where does it fall short?
Yes, an RTX 4090 handles inference for 7B–13B models well and is the cheapest per hour of the three. It falls short on VRAM (24 GB caps model size and context length), lacks the high-bandwidth HBM memory and NVLink interconnect that make multi-GPU scaling on A100/H100 practical, and is usually offered as a single consumer card rather than in large clusters. For long contexts, large batches or 70B models, it becomes the bottleneck.
How does KV-cache size affect GPU memory and rental cost during inference?
The KV-cache stores attention keys and values for every token already processed, so it grows linearly with context length and batch size and competes with model weights for VRAM. A rough estimate is:
KV bytes ≈ 2 × layers × heads × head_dim × seq_len × batch × bytes_per_value
Longer contexts or more concurrent users can push you onto a larger or additional GPU, directly raising hourly rental cost even when the model itself would fit.
When is renting cloud GPUs cheaper than paying per token via an LLM API?
Renting wins when your sustained utilization is high: if a rented GPU costs a few dollars per hour and serves enough tokens per hour to undercut the API's per-token price, owning the workload financially makes sense. It also wins when you need data privacy, custom or fine-tuned models, or guaranteed latency that APIs don't offer. For bursty, low-volume or experimental traffic, per-token APIs are usually cheaper because you pay only for what you use and avoid idle GPU hours.