Local Training

Fine-tune a supported base model with LoRA, compare it with the base, and serve the adapter on your own NVIDIA GPU.

This guide covers the adapter engine. The commands run on your machine without a TT access token and write datasets, run state, reports, and model artifacts there. Add runtime.gpu settings to delegate training and CUDA evaluation to an existing EC2 instance in your AWS account; see Use your AWS GPU for setup. Serving remains local.

Supported models and tasks

The adapter runtime supports:

Text supervised fine-tuning (SFT) with LoRA adapters through PEFT
Qwen/Qwen3.5-2B (pinned to snapshot 15852e8c16360a2fea060d615a32b45270f8a8fc)
nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 on hosts with enough GPU memory, including DGX Spark
meta-models/Muse-Glimmer-30B for text training, with its vision weights frozen and unused
Exact-match or selected JSON-field evaluation
OpenAI-compatible local serving

Nemotron uses activation checkpointing and applies LoRA to shared attention and Mamba projections. Training and serving in this guide require Linux and an NVIDIA CUDA GPU. Evaluation can also run on CPU. Run tt hardware to check the models against your host's memory and installed runtime.

Requirements

Node.js 22.19+
uv
A Linux host with a compatible NVIDIA GPU
Enough disk space for the model cache and run artifacts

curl -fsSL https://tunedtensor.com/install.sh | sh
tt --version
tt hardware
tt status
uv --version
nvidia-smi

Create a project

mkdir support-adapter
cd support-adapter
tt init \
  --name "Support Adapter" \
  --model Qwen/Qwen3.5-2B \
  --profile spark

This creates only tunedtensor.json, including when using --profile spark. Set system_prompt to the instructions the model should follow, and replace both placeholder examples with different, representative inputs and expected answers.

Check the project and run

tt doctor tunedtensor.json
tt validate tunedtensor.json
tt models prefetch tunedtensor.json
tt models verify-base tunedtensor.json
tt pipeline run --spec tunedtensor.json

Keep optional runtime overrides and evaluation policy in tunedtensor.json. Commands read these settings from the spec; omitted settings use the runner defaults.

Inspect and verify

tt runs list
tt runs get <run-id>
tt runs events <run-id>
tt runs report <run-id>
tt models verify local-<run-id>

The report compares the base and tuned model on the same held-out prompts. Use it to check whether the adapter improves the tasks in your evaluation set.

Use a chat JSONL dataset

For larger datasets, add dataset_prebuilt to tunedtensor.json:

{
  "dataset_prebuilt": {
    "training": "data/train.jsonl",
    "validation": "data/validation.jsonl",
    "format": "chat_jsonl"
  }
}

Each line contains a messages array with an optional system message, one user message, and one assistant answer. Message content must be text. Use a distinct validation or test file for held-out evaluation.

Check for regressions

Add a separate evaluation dataset under evaluation in tunedtensor.json for capabilities you want the model to retain. Activation requires this regression check to pass:

{
  "evaluation": {
    "generalRegression": {
      "dataset": "evals/general.jsonl",
      "maxScoreDrop": 0.03,
      "maxPassRateDrop": 0.05
    }
  }
}

Activate and serve

tt models activate local-<run-id>
tt models active
tt serve active --spec tunedtensor.json

# Restore the previous adapter or protected base
tt models rollback

Activation requires a completed run, a verified model artifact, and a passing regression result. Include the regression dataset before training the adapter you want to activate. tt serve active requires an activated adapter. You can also serve a verified adapter directly with tt models serve local-<run-id>.

Serving discovers tunedtensor.json in the current directory for runtime settings, or uses the file selected with --spec. Adapter serving uses the instructions saved with the training run by default. If you pass --spec, its base model and instructions must match the trained adapter. To serve a specific verified adapter:

tt models serve local-<run-id> \
  --host 127.0.0.1 \
  --port 8000

The default bind is localhost. A non-loopback bind requires --allow-remote and an API key supplied through --api-key-env. To serve the base model with your project's instructions, pass --spec tunedtensor.json to tt serve base.