LLM VRAM calculator

How much VRAM an LLM needs

The weights decide whether a model loads. The KV cache decides whether it can serve your workload: a model that fits with 4K tokens of context can run out of memory at 128K, or at 16 users. The calculator prices both, using each model's attention layout from its config.json.

Weights: parameters times bytes

A model with 70 billion parameters stored in BF16 takes 140 GB (130 GiB). Quantized formats store fewer bits per weight plus small scaling factors, and the bits below include those scales.

GPU quantization tools usually leave the input embeddings and the output layer in 16 bits, so the calculator prices those at 2 bytes for every non-GGUF format. llama.cpp quantizes them too. When a model's official checkpoint is already quantized, such as DeepSeek-V4.1-Flash with FP4 experts, "As published" uses the size of the released files.

Mixture-of-experts (MoE) models do not get a discount. A token only passes through a few experts, but any token can pick any expert, so all of them stay in memory. gpt-oss-120b uses 5.1B parameters per token and still needs its full 60.8 GiB checkpoint loaded.

KV cache: grows with context and users

While it generates, the model keeps the keys and values of every earlier token so it does not recompute them. For standard attention the size is:

KV bytes = layers × 2 × KV heads × head dim × bytes per element × tokens × sequences

Llama 3.3 70B has 80 layers and 8 KV heads of dimension 128, so each token costs 320 KiB in BF16. FP8 halves that. Newer models shrink it by changing the attention itself, and the presets model each design separately:

• Latent attention (MLA). DeepSeek-V3.1, Kimi K2 and GLM-5.3 cache one 576-value vector per token per layer instead of full keys and values: 68.6 KiB per token in BF16 across 61 layers.
• Hybrid attention. Qwen3.8 keeps a normal KV cache on one layer in four. The others are Gated DeltaNet layers that hold a fixed-size state per sequence, so Qwen3.8-Flash-Next needs only 24.8 KiB per token. GLM-5.3-Flash pairs 11 latent-attention layers with 34 KDA layers in the same way.
• Sliding windows. gpt-oss and Gemma keep only the last 128 or 1,024 tokens on their local-attention layers, so those layers stop growing.
• Published figures. DeepSeek-V4.1-Flash compresses its global cache to 890 bytes per token in FP4. The calculator uses that number as published.

Recurrent state is per sequence and does not grow with context. It does multiply with concurrency, and engines keep extra copies for prefix caching: our GLM-5.3-Flash replicas reserve 28 state slots for four running requests, which became the bottleneck before the KV cache did.

Activations, runtime and the memory cap

Two smaller terms complete the estimate. Activations are the temporary tensors for the prompt chunk being processed:

activations = prefill chunk × (4 × hidden size + 2 × FFN width ÷ TP) × 2 bytes

Runtime covers the CUDA context, library workspaces and CUDA graphs. The vLLM / SGLang preset uses 2 GiB per GPU and 8,192-token chunks; the llama.cpp / Ollama preset uses 0.75 GiB and 512-token micro-batches. Both are editable under Overhead settings.

Engines do not use all of a GPU. vLLM claims a fixed share set by --gpu-memory-utilization: 0.90 up to v0.19 and 0.92 from v0.20. The calculator uses 0.90 for vLLM and SGLang and 0.95 for llama.cpp. Macs use the share macOS gives the GPU, which Apple puts at 21 GB of a 32 GB machine and 48 GB of a 64 GB one.

We checked the vLLM preset against Nanbeige4.3-33B on one RTX PRO 6000, where vLLM reported 62, 33 and 21 GiB of weights and room for 108K, 499K and 623K cached tokens in BF16, FP8 and NVFP4. The calculator predicts the weights and the cache room within 3% for all three. Those checks run as unit tests whenever the presets change.

Splitting a model across GPUs

Tensor parallelism (TP) splits every layer across the GPUs in one server. Each GPU holds 1/TP of the weights and of the KV heads, plus its own runtime memory. Two details change the answer:

• Once TP is larger than the number of KV heads, heads are copied, not split. vLLM gives every GPU at least one, so Qwen3.8-Flash-Next (2 KV heads) uses as much KV memory per GPU at TP4 as at TP2.
• Latent-attention caches are copied to every GPU in plain tensor parallel, so adding GPUs does not shrink them. Data-parallel attention, which we use for GLM-5.3 on 8x H200, avoids the copies.

vLLM also requires the number of attention heads to divide evenly by TP, so the calculator skips sizes that do not. Above eight GPUs a model needs more than one server, with pipeline or expert parallelism, and the calculator gives only a rough GPU count.

VRAM requirements for popular models

Total memory for one 32K-token conversation with a BF16 KV cache and vLLM-style overhead, and the smallest NVIDIA setup that holds it on GPUs that run the format natively (fewest GPUs, then the smallest card). Computed by the calculator above.

What the calculator does not model

• Speculative decoding. Draft models and multi-token-prediction heads add their own weights and cache.
• Extra recurrent-state slots that hybrid models keep for prefix caching (see the GLM-5.3-Flash note above).
• Offloading weights or cache to system RAM or SSD, which lets a model run on less VRAM at a large speed cost.
• Vision encoders' activations. Their weights are included when the official checkpoint ships them.
• Fine-tuning. Training needs gradients and optimizer state, several times the inference figure.

Model numbers come from each model's official config.json on Hugging Face and the parameter count its repository reports, checked on 5 November 2026. GPU capacities are the totals nvidia-smi reports (for example 97,887 MiB for the RTX PRO 6000 and 81,559 MiB for the H100), which differ from the GB on the box. Every preset links its source under "How this was calculated".

Frequently asked questions