GPU calculator
Which GPU for my model?
Pick a model, its precision, context and concurrency. See the GPU memory it needs, line by line, and the GPUs that hold it.
Calculator
Needs 19.5 GiB of GPU memory. High confidence
| Model weights | 8.03B parameters × 2 bytes (16-bit (FP16 or BF16)) | 15.0 GiB |
|---|---|---|
| KV cache | 2 × 32 layers × 8 KV heads × 128 head dim × 2 bytes × 4,096 tokens × 1 concurrent | 0.5 GiB |
| Overhead margin | 20 % of the figures above (activations, allocator, quantization scales) | 3.1 GiB |
| CUDA runtime | 1 GiB per GPU (CUDA context and library workspaces) | 1.0 GiB |
| Total | 19.5 GiB |
| GPU memory | 1 GPU | 2 GPUs | 4 GPUs | 8 GPUs |
|---|---|---|---|---|
| 24 GB per GPU | Fits | Fits on 1 | Fits on 1 | Fits on 1 |
| 48 GB per GPU | Fits | Fits on 1 | Fits on 1 | Fits on 1 |
| 80 GB per GPU | Fits | Fits on 1 | Fits on 1 | Fits on 1 |
Every request may use the full context. KV cache in 16-bit.
Memory is calculated from each model's published facts: weights (parameters × bytes per parameter), KV cache (layers × KV heads × head dimension × context × requests), a 20 % margin and 1 GiB of CUDA runtime. It is an estimate, not a measurement: frameworks reserve memory differently, speed is not estimated, and image models count their weights only.