Pick a GPU for your model
Describe your workload, get the machines it fits, cheapest first. How the GPU memory is computed, what the result covers and where it stops.
In the console, open Catalog, then Launch by workload. Say what you want to do, which model, and your budget and preferences. You get the machines your workload fits, cheapest first, each with the reason it fits.
The answer is a calculation, the same every time for the same inputs. Nothing is booked until you press Launch on a machine.
The three steps
- What you want to do: run a model, fine-tune a model, generate images, or open a notebook.
- Which model: pick it from the list, then its precision (16-bit, 8-bit or 4-bit), the context per request and how many requests run at once. For a notebook without a model, enter the GPU memory you need.
- Budget and preferences: a budget, a maximum price per hour, the tier (Secure, Community or either) and the countries the machine must run in.
EUcovers every member state of the European Union.
Each model shows its parameter count, its license and a link to the facts it was sized from.
How the GPU memory is computed
For a language model, the memory needed is the sum of four lines:
| Line | Formula |
|---|---|
| Model weights | parameters × bytes per parameter: 2 in 16-bit, 1 in 8-bit, 0.5 in 4-bit |
| KV cache | 2 × layers × KV heads × head dimension × 2 bytes × context × requests at once |
| Overhead margin | 20 % of the two lines above |
| CUDA runtime | 1 GiB per GPU |
Every figure comes from the model’s published configuration: parameters, layers, KV heads, head dimension, longest context. The result shows each line with its formula filled in.
Example: Llama 3.1 8B in 16-bit
With a 4,096-token context and one request at a time:
| Line | Calculation | GiB |
|---|---|---|
| Model weights | 8.03 billion × 2 bytes | 14.96 |
| KV cache | 2 × 32 × 8 × 128 × 2 bytes × 4,096 × 1 | 0.50 |
| Overhead margin | 20 % of 15.46 | 3.09 |
| CUDA runtime | 1.00 | |
| Total | 19.55 |
A GPU with 24 GiB holds it. A 16 GiB GPU does not. In 4-bit the same model needs about 6.1 GiB.
The KV cache grows with the context and with the number of requests: at 128,000 tokens for one request, it alone takes 15.6 GiB. Size the context to what your requests use.
Fine-tuning
Fine-tune a model sizes LoRA training. The base model stays frozen and small adapters are trained:
- LoRA on a 16-bit or 8-bit base: the base weights, plus rank 16 adapters on the attention projections at 16 bytes per trainable parameter (weight, gradient and AdamW states).
- QLoRA on a 4-bit base: the base in 4-bit NormalFloat, 4.127 bits per parameter with double quantization, plus the same adapters.
Activations are added for the sequence length and batch size you enter, with gradient checkpointing on. The context field becomes Sequence length and the requests field becomes Batch size.
The formulas come from published work: LoRA, QLoRA, ZeRO for optimizer states, and activation recomputation.
Image generation
An image pipeline loads several networks: the denoiser, its text encoders and an autoencoder. The weights of all of them are counted exactly, then the margin and the runtime. The margin is meant for one image at the model’s native resolution; larger images and batches need more. That is why image results carry low confidence.
Speech
For Whisper, the decoder’s KV cache is sized on its 30-second window, for each audio stream at once.
How a machine is checked
A machine fits when its GPU memory holds the total:
- On one GPU, when one is enough.
- Across several GPUs of the same machine, for a language model too large for one: the weights and the KV cache are split evenly with tensor parallelism. That needs the model’s attention heads to divide evenly by the number of GPUs.
- Image models, speech models and fine-tuning are checked on one GPU.
The GPU memory of a Secure machine comes from the data center or the manufacturer’s datasheet. On a Community machine it is the figure the provider’s machine reports, not verified by Node Lodge.
When a fact is missing (the machine does not report its GPU memory, or a model fact is not published), the machine is left out. No machine is recommended on a guess.
Reading the result
- GPU memory needed: the total, line by line, with the assumptions behind it.
- Recommended machines: the machines that fit, cheapest first. Open a row for Why it fits and its Limits: where the memory figure comes from, a thin margin, a booking limit, an interruptible machine.
- Budget covers: how long your budget runs on that machine at its rate.
- Confidence: high, medium or low. High for a language model in 16-bit on a machine whose memory comes from the data center. Lower when the weights are quantized, when the model is split across GPUs, when the figure comes from a provider’s report, when the margin is thin, or for image generation and fine-tuning, which depend more on settings.
- Alternatives: machines that fit but break a preference (above your price, other tier), and the same model at a lower precision when it opens more machines or a cheaper one.
- Constraints applied: everything the answer took into account.
Machines outside the countries you chose are never offered, not even as alternatives. A machine whose location is not known does not count as inside them.
Limits
- Speed is not estimated. Tokens per second or images per minute appear only once they are measured on the machine.
- Fit is a calculation. Frameworks differ: an inference server may reserve more memory than it needs, a training script may keep extra copies. If you change the framework’s settings, keep the margin in mind.
- Machines available now. Only online Community machines and in-stock Secure machines are checked.
- Community machines run on the provider’s own hardware. The provider can read the session’s memory. See Secure and Community.