LoRA fine-tuning

Supervised fine-tuning with LoRA adapters, using the trl command line. No training script to write.

GPU: 24 GB for the 0.5B smoke test and QLoRA on 7B, 48 GB for LoRA on 7B. See Fine-tuning for sizing.

1. Install

curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install torch --torch-backend=auto
uv pip install "trl[peft]" bitsandbytes

Check the stack:

trl env

2. Smoke test

A short run on a small model checks memory and throughput:

trl sft \
  --model_name_or_path Qwen/Qwen2.5-0.5B \
  --dataset_name trl-lib/Capybara \
  --use_peft \
  --max_steps 50 \
  --output_dir runs/smoke

3. Train

Run the real job inside tmux:

tmux new -s train
source .venv/bin/activate
trl sft \
  --model_name_or_path Qwen/Qwen2.5-7B \
  --dataset_name trl-lib/Capybara \
  --use_peft \
  --lora_r 16 \
  --lora_alpha 32 \
  --lora_target_modules q_proj k_proj v_proj o_proj gate_proj up_proj down_proj \
  --learning_rate 2.0e-4 \
  --num_train_epochs 1 \
  --per_device_train_batch_size 2 \
  --gradient_accumulation_steps 8 \
  --save_steps 200 \
  --output_dir runs/qwen7b-lora

Detach with Ctrl+b d. Reattach with tmux attach -t train.

On 24 GB, load the base model in 4-bit (QLoRA): add --load_in_4bit.

Your own data: a JSONL file with a messages column (chat format) or a text column.

rsync -avz ./train.jsonl ubuntu@203.0.113.42:~/data/

Point trl at it with a config file instead of --dataset_name:

# sft.yaml
model_name_or_path: Qwen/Qwen2.5-7B
datasets:
  - path: json
    data_files: /home/ubuntu/data/train.jsonl
use_peft: true
output_dir: runs/qwen7b-lora

trl sft --config sft.yaml

Command-line flags override values from the file. A private Hub dataset works too: pass its name to --dataset_name.

Gated base models (Llama, Gemma) need export HF_TOKEN=hf_... before training.

4. Multi-GPU

On multi-GPU offers, pass the process count:

trl sft ... --num_processes 2

5. Copy the adapter

The adapter is saved in the output directory:

rsync -avz ubuntu@203.0.113.42:~/runs/qwen7b-lora/ ./qwen7b-lora/

Then select End session. Checkpoints, caches and the base model are deleted with the machine.

Reference: TRL CLI, PEFT integration.