vLLM

Serve an open-weight model behind an OpenAI-compatible API.

GPU: 24 GB VRAM or more for the 7B model below. See LLM inference for sizing.

1. Check the GPU

nvidia-smi

The header shows the driver and the highest CUDA version it supports.

2. Install

curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto

--torch-backend=auto selects the PyTorch build that matches the installed driver.

3. Serve

Start the server inside tmux so it survives disconnects:

tmux new -s vllm
source .venv/bin/activate
vllm serve Qwen/Qwen2.5-7B-Instruct --max-model-len 8192

Detach with Ctrl+b d. The first start downloads the weights to ~/.cache/huggingface. The server is ready when it logs that it is listening on port 8000.

Options worth knowing:

Flag	Use
--tensor-parallel-size 2	Split the model across 2 GPUs on multi-GPU offers
--max-model-len 8192	Cap context length to fit the KV cache in VRAM
--gpu-memory-utilization 0.90	Share of VRAM vLLM may use
--api-key <secret>	Require a bearer token on every request

Gated models (Llama, Gemma) need a Hugging Face token:

export HF_TOKEN=hf_...

4. Query from the machine

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-7B-Instruct",
    "messages": [{"role": "user", "content": "Say hello in five words."}],
    "max_tokens": 32
  }'

5. Query from your computer

Only SSH is open inbound. Forward port 8000:

ssh -N -L 8000:localhost:8000 ubuntu@203.0.113.42

Then point any OpenAI-compatible client at http://localhost:8000/v1:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
reply = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[{"role": "user", "content": "Say hello in five words."}],
)
print(reply.choices[0].message.content)

Set api_key to your secret if you started the server with --api-key.

6. Finish

Copy anything you need, then select End session. Downloaded weights are deleted with the machine.

Reference: vLLM documentation.