LLM VRAM & Capacity Calculator_

Plan GPU memory for LLM inference: weights (shared + experts), KV cache, activations and runtime overhead — across quantization formats (FP16 → NVFP4), single rigs and multi-node clusters with tensor / pipeline parallelism. Every number derives from published formulas; the full derivation with your values plugged in is at the bottom.

PagedAttention KV mathGQA + MLA aware NVFP4 = 4.5 bitMegatron TP/PP sharding chunked-prefill activationsno pro tier — it's all just math

1Model

2Workload

Advanced runtime knobs
max batched tokens / step (chunked prefill)
vLLM gpu_memory_utilization, default 0.90
CUDA context, cudagraphs, allocator
NCCL buffers — applied when GPUs > 1

3Deployment

Connection between GPUs — higher bandwidth = better TP scaling

Performance & Memory Results

—
VRAM
OKAY
—
—
—
—
—

Memory Allocation

Breakdown (per GPU)

ComponentPer GPUTotal

Topology

Checks & warnings

The math, with your numbers plugged inevery value above derives from these

Model weights (shared + expert)bits-per-weight × parameter count

Weight memory is exact: parameters × effective bits/weight ÷ 8. Block-scaled formats carry their scale overhead: NVFP4 = 16 × FP4 (E2M1) + one FP8 (E4M3) scale per 16-block → (16×4+8)/16 = 4.5 bits/weight (NVIDIA spec); MXFP4 (OCP MX) = 32-block E8M0 scale → 4.25; INT4 AWQ/GPTQ g128 ≈ 4.25. For MoE, all experts stay resident: expert params = total − shared, where shared (attention + embeddings + dense parts) = active params − active-expert FFN (3 matrices × hidden × active FFN width × layers).

KV cacheKwon et al. 2023 (PagedAttention), arXiv:2309.06180 · DeepSeek-V2, arXiv:2405.04434 (MLA)

Each token of each active sequence stores one K and one V vector per layer. With grouped-query attention only n_kv heads are cached — this is exactly what calculators that ignore GQA and context growth get wrong:

KV bytes/token = 2 × n_layers × n_kv_heads × head_dim × bytes(kv dtype)

For MLA (DeepSeek), K and V compress into one shared latent per token per layer: bytes/token = n_layers × (kv_lora_rank + qk_rope_head_dim) × bytes — no factor 2, no per-head multiplier. PagedAttention keeps fragmentation under ~4%, so KV = users × context is a sound planning bound.

Activations (inference working set)Korthikanti et al. 2022, arXiv:2205.05198

Inference stores no gradients, so activation memory is the transient working set of the widest layer — not training's sbh(34+5as/h). Dominant live tensors: gate+up FFN projections (2 × FFN width) plus residual/attention streams (≈2 × hidden). Engines cap tokens in flight via chunked prefill, so the batch dimension is max(prefill_chunk, decode_batch) — not users × seq_len. TP shards these across ranks.

Tensor & pipeline parallel shardingShoeybi et al. 2019, arXiv:1909.08053 · Narayanan et al. 2021, arXiv:2104.04473

Megatron TP splits every attention/FFN matrix across t ranks; PP gives each stage ⌈L/p⌉ layers. KV shards by KV head across TP — but only up to n_kv ranks: if t > n_kv, KV heads are replicated (as vLLM / TensorRT-LLM do), so the KV divisor is min(t, n_kv) × p. MLA's latent is not head-sharded — it replicates across TP and divides by PP only.

Capacity: max concurrent users & max contextbudget algebra over the terms above

The engine may use utilization × VRAM (vLLM default 0.90). What remains after weights + activations + overheads is the KV budget:

Speed, TTFT & interconnectroofline (bandwidth-bound decode, compute-bound prefill) · Megatron all-reduce volume

Decode is memory-bandwidth-bound: each token reads the active-weight shard + the sequence's KV from HBM. Prefill is compute-bound: ≈ 2 × active params FLOPs per token (the 2PN rule). TP adds 2 ring all-reduces per layer per token (volume 2·(t−1)/t·h·2B each) over the interconnect; PP adds one hidden-vector hop per stage boundary. Real engines hit 50–80% of these bounds — treat as upper limits.