GPU VRAM Calculator for LLMs

The calculator sums four consumers of GPU memory: model weights, the KV cache, a working set, and framework overhead. Weights equal the parameter count times the bytes per weight (2 for FP16, 1 for INT8, 0.5 for INT4). For inference it adds a KV cache sized from your layers, hidden size, context length, and batch, plus a small activation buffer. It then adds a CUDA context allowance and a fragmentation margin, and reports the total in binary gigabytes.

A 7 billion parameter model in FP16 stores about 13 GB of weights (7 billion times 2 bytes). With a 4,096 token context, a single sequence, and a typical 32 layer, 4,096 hidden architecture, the KV cache adds roughly 2 GB, activations another 2 to 3 GB, and overhead about 3 GB, for around 20 GB total. Quantizing to INT4 cuts the weights to about 3.5 GB, which is why an 8 to 12 GB card can run a quantized 7B model.

Quantization sets how many bytes each weight occupies. FP16 and BF16 use 2 bytes per weight, INT8 uses 1 byte, and INT4 uses 0.5 bytes, so moving from FP16 to INT4 cuts the weight memory to a quarter. The KV cache stays in FP16 in most runtimes, so it does not shrink with weight quantization, which is why long contexts can still dominate memory even on a heavily quantized model.

The KV cache stores one Key and one Value vector per layer for every token in the context window so the model does not recompute attention each step. Its size is 2 times layers times hidden size times 2 bytes times context length times batch size. Because it scales linearly with context length and batch size, doubling either one doubles the cache, and very long contexts can need more memory than the weights themselves.

Full fine-tuning with mixed-precision Adam keeps far more than the weights in memory. The standard footprint is about 16 bytes per parameter: FP32 master weights, FP32 momentum, FP32 variance, plus FP16 weights and gradients. That is roughly eight times the FP16 weight size, so a 7B model can need well over 100 GB. LoRA and QLoRA freeze the base weights and train small adapters instead, which is why they fit on a single consumer card.

Treat the result as a planning estimate, not an exact figure. Real usage shifts with the runtime (llama.cpp, vLLM, Transformers), the attention kernel, paged or quantized KV caches, and the exact weight format, which the model approximates with fixed allowances. Leave a margin of 10 to 20 percent over the estimate, and prefer a card with headroom rather than one that the calculator says only just fits.