# Models at launch

> Pick a Hugging Face model when you launch a Node Lodge Secure machine and find its files on the machine, downloaded in the background.

Pick a model when you launch a Node Lodge Secure machine. The session is ready as soon as the machine is, and the model downloads in the background. Use the machine while it downloads. The session is billed from the moment it is ready, download included.

## Pick a model

In **Launch**, choose a Node Lodge Secure machine. In the **Environment** step, enter the model in **Model (optional)**:

- Use the name as Hugging Face lists it: `org/name`, like `Qwen/Qwen3-8B`, for the safetensors weights.
- Add `:VERSION` for one GGUF file of the repository: `org/name:VERSION`, like `unsloth/Qwen3-8B-GGUF:Q4_K_M`. See [GGUF versions](#gguf-versions).
- Accept the license of the model. The launch carries your acceptance.
- Check **Serve with** to get an OpenAI-compatible endpoint on the model. See [Serve the model](#serve-the-model).
- Leave the field empty to launch without a model.

With the API, send `model` (`id` and `licenseAccepted: true`) in `POST /v1/sessions`. With the Python SDK, pass `model="Qwen/Qwen3-8B"` to `sessions.launch`. With the TypeScript SDK, pass `model: { id, licenseAccepted: true }`. With the command line, add `--model <org/name[:VERSION]> --accept-license` to `nodelodge launch`. See [API reference](https://nodelodge.com/docs/api-reference) and [CLI and SDK](https://nodelodge.com/docs/cli-and-sdk).

`POST /v1/models/lookup` checks a model before you launch: its revision, its size, its download time and, for a GGUF version, its `format` and `file`.

## GGUF versions

A GGUF repository holds one file per version (quantization), like `Qwen3-8B-Q4_K_M.gguf` and `Qwen3-8B-Q8_0.gguf`. Write the version after the name, `org/name:VERSION`, and the machine downloads that file alone:

- The version matches the end of the file name, after a `-` or a `.`, upper or lower case alike: `Q4_K_M` picks `Qwen3-8B-Q4_K_M.gguf`.
- When several files end with it, like `Q4_K_M` and `UD-Q4_K_M`, the shortest name wins. Write `UD-Q4_K_M` for the other one.
- A file split in parts, like `-00001-of-00002.gguf`, downloads with all its parts.
- Each file is downloaded in parallel and checked against its SHA-256.

A launch with a version the repository does not have is refused before anything starts, with the versions it has:

```text
This repository has no Q3_K_S version. Versions in this repository: Q4_K_M, Q6_K, Q8_0.
```

A repository without GGUF files is refused the same way:

```text
This repository has no GGUF files. Enter it without :Q4_K_M to download its safetensors weights.
```

The estimated size and download time, the budget check and the disk space all count that one file.

## Budget and download time

The launch form shows the estimated download time of the model. A launch is refused when your budget or maximum duration does not cover the estimate with a margin. The message says what to raise, with the amount that covers the estimate at the rate of the machine. For example, for `Qwen/Qwen3-32B`:

```text
Your budget covers about 20 minutes. Downloading Qwen/Qwen3-32B and starting the machine takes about 29 minutes. Raise the budget to at least US$X.XX, or launch without a model.
```

The real duration depends on the load of the network. Once the session runs, its page shows the time left, measured from the download itself.

## Your Hugging Face token

Save a Hugging Face read token once. Node Lodge uses it to look up and resolve every model you launch, public models included. A gated or private model needs it.

1. In the console, open **Settings**, then **Integrations**.
2. Paste a read token in **Hugging Face** and choose **Save token**. Hugging Face must accept it. With two-factor authentication on, enter a fresh code; without it, enter your **Account password**.
3. You receive an e-mail each time a token is saved, replaced or removed.

- A read token is enough.
- For a gated model, accept its license on Hugging Face with the account of the token first.
- The token is stored encrypted and never shown again: the page shows its last 4 characters and the date it was saved. **Replace** and **Remove** are on the same page. Sessions already started keep theirs.
- It does not appear in your session details.

For a gated or private model, the token is put on the machine: in `$HF_HOME/token`, readable by the `ubuntu` user only, and set as `HF_TOKEN` in every login shell. `hf`, vLLM and transformers find it without `hf auth login`. A machine launched with a public model gets no token.

To upload to the Hub from your machines, turn on **Also put my token on every machine I launch (for uploads to the Hub)** in **Integrations**. It is off by default. Every machine you launch with a model then gets the token, and anyone who can open these machines can read it. Turning it on asks for a fresh two-factor code, or your password without two-factor. **On machines** shows the setting: **Gated and private models only** or **Every machine you launch with a model**. With the API, send `shareOnMachines` in `PUT /v1/integrations/hugging-face`, or `{ "shareOnMachines": true }` in `PATCH /v1/integrations/hugging-face`.

To use another token for one launch, send it as `modelToken` in `POST /v1/sessions`. With the Python SDK, pass `model_token` to `sessions.launch`; with the TypeScript SDK, pass `modelToken`. It takes the place of the saved token for that launch only, and it is put on the machine whatever the model. Without a saved token, the launch form asks for one under **Hugging Face token (gated and private models)**, for that launch. `POST /v1/models/lookup` takes a `token` the same way; lookups with a `token` are limited to 10 per minute per organization.

With an API key (CLI, SDKs, MCP server, GitHub Action), launches use your saved token only when the key was created with **Launches use my saved Hugging Face token** checked, in **API keys**. Anyone holding such a key can read the token on the machines it launches. With any other key, send the token with the launch (`modelToken`) or create a key with the box checked.

## What is loaded

For `org/name`: the weights in safetensors format, with the files that go with them: configuration, tokenizer, and license. Other formats of the same model, like GGUF or ONNX, are left out. A model published without safetensors weights is refused at launch: pick a GGUF version with `org/name:VERSION` when the repository has one.

For `org/name:VERSION`: the one GGUF file of that version, or all its parts.

The revision loaded is pinned when you launch, so a model updated on Hugging Face later does not change under your session. The session page shows the model and where it is on the machine.

## Where the files are

On the machine, the model is in the Hugging Face cache:

- `/data/huggingface` when the machine has a data disk,
- `~/.cache/huggingface` otherwise.

`HF_HOME` is set for every user and every shell from the first connection, so tools find the cache without further setup. The **Model** row of the session page shows the full path, with a copy button.

For safetensors weights, the session's `model.localPath` is the snapshot folder of the model. Load it by name. Nothing downloads again:

```bash
vllm serve Qwen/Qwen3-8B
```

```python
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B")
```

## Run a GGUF file

For a GGUF version, `model.localPath` is the GGUF file itself, for example:

```text
/data/huggingface/hub/models--unsloth--Qwen3-8B-GGUF/snapshots/<revision>/Qwen3-8B-Q4_K_M.gguf
```

`model.format` is `gguf` and `model.id` is `unsloth/Qwen3-8B-GGUF:Q4_K_M`. Copy the path from the **Model** row of the session page into `MODEL_PATH`, then give it to llama.cpp:

```bash
nodelodge-model wait && llama-server -m "$MODEL_PATH" --host 127.0.0.1 --port 8080
```

Or to Ollama, with a `Modelfile` that holds one line, `FROM` and the path:

```bash
echo "FROM $MODEL_PATH" > Modelfile
ollama create qwen3-8b -f Modelfile
```

For a file split in parts, the path is its first part. llama.cpp loads the other parts from the same folder.

## Serve the model

Check **Serve with** under the model, and a server starts on it once it is downloaded, behind an OpenAI-compatible endpoint with its own key:

- **vLLM** for `org/name` (safetensors weights). Requests name the model `org/name`, like `Qwen/Qwen3-8B`.
- **Ollama** for `org/name:VERSION` (one GGUF file). The file is imported under a short name, like `qwen3-8b:q4_k_m` for `unsloth/Qwen3-8B-GGUF:Q4_K_M`. Ollama takes a GGUF file in one part: a version split in parts is refused at launch.

With the API, send `serveWith` (`vllm` or `ollama`) in `POST /v1/sessions`, with a `model` written to match. With the command line, add `--serve` to `--model`. With the SDKs, pass `serveWith: 'vllm'` (TypeScript) or `serve_with="vllm"` (Python). With the MCP server, pass `serve: true`. See [CLI and SDK](https://nodelodge.com/docs/cli-and-sdk) and [MCP server](https://nodelodge.com/docs/mcp).

Serve with runs on sessions through the Node Lodge relay. On other sessions, the launch is refused:

```text
Serve with runs on sessions through the Node Lodge relay. Launch without it, then start the server over SSH.
```

### The endpoint

The server listens on the machine only. Its endpoint is published at the address of the session on the relay:

```text
https://<session>.<relay domain>/v1
```

- A port may follow the domain: copy the base URL as Node Lodge gives it.
- TLS runs all the way to the machine. The relay sees neither the key nor the requests.
- Each session has its own key, starting with `nls_`. Send it as `Authorization: Bearer <key>`. A request without a valid key gets `401`.
- Paths under `/v1/` are answered. Any other path is refused.
- The server and the endpoint run as a dedicated system user, `nodelodge-serve`, with no shell, never as root.
- The relay limits the open connections and the new connections per second of each session origin.
- The endpoint stops with the session, including when its budget is reached.

Read the endpoint:

- In the console, on the session page, the **OpenAI-compatible endpoint** panel shows **Base URL**, **Model**, **API key** (masked, with **Copy key** and **New key**) and a curl example.
- With the API, `GET /v1/sessions/{id}/access` returns `inference`: `baseUrl`, `apiKey`, `model`, `engine` and `status`.
- With the command line, `nodelodge endpoint <session-id>` prints the base URL, the model and the key.
- With the MCP server, `get_session` returns `endpoint`.

### Server status

The session gives the status of the server in `serve.status`:

| Status | Meaning |
|---|---|
| `waiting` | The model is still downloading. |
| `starting` | The server installs and starts. A first start takes a few minutes: vLLM installs on the machine. |
| `ready` | The endpoint answers. |
| `failed` | The server stopped. `failure` gives the reason. |

| `failure` | Meaning |
|---|---|
| `model_not_loaded` | The model was not downloaded, so the server did not start. See [When the model was not loaded](#when-the-model-was-not-loaded). |
| `install_failed` | The server could not be installed on the machine. |
| `start_failed` | The server did not start on this model. |
| `session_ended` | The session ended. |

After a failure, connect over SSH and start a server yourself on the model. See [Without Serve with](#without-serve-with).

### Send a request

Set `BASE_URL` to the base URL and `NODELODGE_ENDPOINT_KEY` to the key, then:

```bash
curl "$BASE_URL/chat/completions" \
  -H "Authorization: Bearer $NODELODGE_ENDPOINT_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen3-8B", "messages": [{"role": "user", "content": "Hello"}]}'
```

Any OpenAI client works. With the OpenAI Python library:

```python
import os

from openai import OpenAI

client = OpenAI(base_url=os.environ["BASE_URL"], api_key=os.environ["NODELODGE_ENDPOINT_KEY"])
reply = client.chat.completions.create(
    model="Qwen/Qwen3-8B",
    messages=[{"role": "user", "content": "Hello"}],
)
print(reply.choices[0].message.content)
```

With Ollama, name the short model, like `qwen3-8b:q4_k_m`. The key is a secret: keep it out of shared files, addresses and logs.

### Get a new key

Choose **New key** in the panel, or call `POST /v1/sessions/{id}/serve-key` (`sessions.newServeKey(id)` in TypeScript, `sessions.new_serve_key(id)` in Python). The call returns the new key. The previous key stops within 15 seconds, and its open connections close.

### Without Serve with

Nothing starts on the model. Connect with `ssh` and start the server yourself, on safetensors weights:

```bash
nodelodge-model wait && vllm serve Qwen/Qwen3-8B
```

On a GGUF file, with Ollama or llama.cpp: see [Run a GGUF file](#run-a-gguf-file).

## Disk space

The machine needs room for the weights (for a GGUF version, its file) plus a margin. When it does not have enough, the launch is refused before anything starts, with the space the model needs and the space the machine has for models. Pick a machine with more disk, or a smaller model.

## Follow the download

The **Model** row of the session page shows the progress and the time left. On the machine:

- `nodelodge-model status` prints the percentage downloaded and the size of the model.
- `nodelodge-model wait` returns when the model is loaded, with exit code 0. Any other exit code means the model was not loaded.
- A message at the opening of a shell tells you where the download stands.

Run `nodelodge-model wait` before you load the model or the GGUF file, so the server starts on complete files:

```bash
nodelodge-model wait && vllm serve Qwen/Qwen3-8B
```

## Restarts and interruptions

A restart of the machine resumes the download. To resume a download that stopped, run:

```bash
sudo nodelodge-model resume
```

Files already verified stay in place and are not downloaded again.

## When the model was not loaded

When the download does not finish, the session keeps running and its page says **The model was not loaded.** It gives the reason, for example:

- Hugging Face did not serve the files.
- The download took more than 6 hours.
- A downloaded file did not match its checksum.
- The disk ran out of space.
- The download failed.
- The download made no progress for 10 minutes.

The page shows the command that downloads the model. When the session is already running, you also receive an e-mail with the reason and the command. Open a terminal on the machine and run it, or run `sudo nodelodge-model resume`. The files land in the same cache.

For a gated or private model, the command signs you in to Hugging Face before it downloads: paste your read token when it asks. For a GGUF version, the command names its file: `hf download org/name <file> --revision <revision>`. `sudo nodelodge-model resume` uses the token in `$HF_HOME/token`. When that file is missing, sign in first, then resume:

```bash
/opt/nodelodge/jupyter/bin/pip install -q -U huggingface_hub && /opt/nodelodge/jupyter/bin/hf auth login
sudo nodelodge-model resume
```

The warning shows while the session runs. A session that ended or did not start has no machine to download to.

## E-mails

You receive an e-mail when the model is ready, unless it arrives within 5 minutes of the session being ready: the console shows it. When the session is already running, you also receive an e-mail when the model could not be downloaded. Turn either off in **Notification settings**. See [Notifications](https://nodelodge.com/docs/notifications).

## Community machines

Community environments that download a model do it at first launch, under your name, into `/workspace`. Models downloaded by Node Lodge in the background are on Node Lodge Secure machines. See [Secure vs Community](https://nodelodge.com/docs/secure-vs-community).