Skip to main content

LLM Fine-Tuning (QLoRA)

Custom chat and instruction models via QLoRA / PEFT

Overview​

  • Active template ID: llm-qlora-finetune (QLoRA, lifecycle=ready)
  • Type: Large Language Model fine-tuning
  • Method: QLoRA — 4-bit quantized base + trainable LoRA adapters
  • Hardware: GPU
  • Best for: Custom chatbots, instruction following, domain assistants

QLoRA fine-tunes a quantized base model by training small LoRA adapters, so you can adapt a multi-billion-parameter model on a single GPU. llm-qlora-finetune is the tested, active template; llm-qlora-peft (PEFT) and llm-merge-adapter (adapter merging) are companion templates.

About llama-3b-finetune

A dedicated llama-3b-finetune preset exists in the catalog but is currently not active. For Llama-family (and other) LLM fine-tuning today, use the active llm-qlora-finetune template and select your base model there. Always confirm availability against the live catalog: GET /api/builder/v1/inference/models?category=deep_learning.

When to Use​

✅ Perfect for:

  • Custom chatbots and domain-specific assistants
  • Instruction fine-tuning and Q&A systems
  • Adapting tone/format to your data

❌ Not ideal for:

  • Simple text classification (use BERT)
  • Very small datasets (a few hundred examples or fewer)

Quick Start​

from colabhive import ColabHive

client = ColabHive(api_key="...", account_id="...")

# Upload an instruction/chat dataset (JSONL)
dataset = client.datasets.upload(name="instructions", file="./train.jsonl")

# Fine-tune with QLoRA
job = client.training.create(
    model="llm-qlora-finetune",
    dataset_id=dataset.id,
    hyperparameters={
        "epochs": 3,
        "micro_batch_size": 4,
        "learning_rate": 2e-4,
        "sequence_len": 512,
        "lora_r": 8,
        "lora_alpha": 16,
    },
)
job.wait()
print(job.get_metrics())

# Register the fine-tuned model as an inference endpoint
endpoint = client.training.register_for_inference(
    run_id=job.id,
    name="my-llm-custom",
    description="Fine-tuned instruction model (QLoRA)",
    visibility="account",
)

# Run inference
result = client.endpoints.infer(
    endpoint_id=endpoint.endpoint_id,
    input_data={
        "messages": [
            {"role": "user", "content": "Explain quantum computing in simple terms."}
        ]
    },
)
print(result)

Dataset Format​

JSONL (instruction)​

{"instruction": "Summarize this text:", "input": "Long text...", "output": "Summary..."}
{"instruction": "Translate to Spanish:", "input": "Hello world", "output": "Hola mundo"}

JSONL (chat)​

{"messages": [{"role": "user", "content": "Hi"}, {"role": "assistant", "content": "Hello!"}]}

Requirements​

  • A few hundred high-quality examples minimum; a few thousand is better.
  • Diverse instructions and clean, consistent responses.

Hyperparameters​

ParameterDefaultRangeDescription
base_modelQwen/Qwen2.5-Coder-7B-Instruct—Hugging Face repo ID of the base model
epochsnull1-10Training epochs (null = 3)
micro_batch_size11-8Per-device micro-batch size
gradient_accumulation_steps41-32Effective batch = micro × accumulation
learning_rate2e-41e-5 to 5e-4LoRA learning rate
sequence_len8192512-32768Max tokens per sample (input + output)
lora_r164-64LoRA rank (higher = more capacity)
lora_alpha328-32LoRA scaling factor

QLoRA Explained​

QLoRA = Quantized Low-Rank Adaptation.

  • 4-bit quantization of the frozen base model cuts memory dramatically.
  • LoRA trains only a tiny fraction of parameters (the adapters).
  • Result: fine-tune large models on a single GPU instead of a multi-GPU cluster.

Tips​

  1. Quality over quantity: a small set of high-quality examples beats a large noisy one.
  2. Diverse formats: train on varied instruction/chat shapes for robustness.
  3. System prompts: include them in training for consistent behavior.
  4. Longer context: raise sequence_len, lower micro_batch_size, and add gradient_accumulation_steps to compensate.
  5. Evaluate on held-out instructions before publishing.