An OpenAI-compatible endpoint with vLLM.

Run an open-weight model behind an OpenAI-compatible API, for evaluation, batch generation or a private endpoint during a working session.

For whom

• Teams evaluating models before committing to a deployment.
• Engineers generating synthetic data or running offline evals.
• Anyone who needs a private model endpoint for a few hours.

Which GPU

Size by VRAM. Weights in 16-bit take about 2 GB per billion parameters, plus room for the KV cache.

Model size	16-bit	8-bit or 4-bit quantized
7B to 8B	24 GB minimum, 48 GB comfortable	16 GB
13B to 14B	48 GB	24 GB
30B to 34B	80 GB	48 GB
70B	Several GPUs (tensor parallel)	80 GB

Long contexts and many concurrent requests need more KV cache: pick the next size up.

Steps

1. Book a machine with enough VRAM.

2. Connect, then start the server inside tmux so it survives disconnects.

3. Follow the vLLM recipe.

4. From your computer, open a tunnel:

   ssh -N -L 8000:localhost:8000 ubuntu@203.0.113.42
   

5. Call http://localhost:8000/v1 with any OpenAI-compatible client.

Tips

• The API is reachable only through your SSH tunnel. Port 8000 is closed to the internet.
• Gated models (Llama, Gemma) need export HF_TOKEN=... before vllm serve.
• Model downloads land in ~/.cache/huggingface. They are deleted with the machine.
• On multi-GPU offers, add --tensor-parallel-size with the GPU count.
• Benchmark throughput in the first minutes, then end the session if the GPU is the wrong size.