Inference Lifecycle

Understanding how ColabHive handles an inference request helps you optimize latency and predict behavior on the first call to a model.

Model states

Every model on a node moves through these states:

cold → downloading_model → loading_model → warm → running → idle → evicted

State	Description	Latency impact
cold	Model not on the node. Needs download + load.	Highest (first request)
downloading_model	Weights downloading from Hugging Face / S3.	Progress reported as %
loading_model	Weights loading into GPU memory.	Seconds to tens of seconds
warm	Model resident in GPU memory.	Sub-second
running	Actively processing a request.	—
idle	Warm but no active requests. May be evicted.	Sub-second
evicted	Removed from GPU to free VRAM for other models.	Back to cold

Request flow

You → Builder Gateway → Orchestrator → Node (WebSocket) → Container → Model

1. Builder Gateway (api.colabhive.com)

Authenticates the request and forwards it to the orchestrator.

2. Orchestrator

Selects the best node based on:

• Model warmth — prefers nodes where the model is already loaded.
• VRAM availability — checks free GPU memory.
• Load balancing — distributes across nodes.

3. Node Runtime

Receives the task over WebSocket and routes it to the right container:

Model type	Container image	Path
LLMs (vLLM / transformers)	inference-vllm	Warm path — persistent container
Specialists (embeddings, OCR, STT, …)	inference-specialists	Warm path — persistent container
Generative (image, audio, video, speech)	inference-generative	Warm path — persistent container
Tools (web-fetch, web-scrape, …)	tools-base	Cold path — one-shot container

Images are always referenced by version, never by a floating latest tag.

4. Container execution

• Warm path — the container stays running between requests. The model is loaded once and reused.
• Cold path — a container is created per request and destroyed afterward.

Cold-start behavior

When you request a model that isn't warm on any node:

1. The orchestrator dispatches to the node with the most free VRAM.
2. The node downloads the weights from the Hugging Face Hub or S3 (for trained models).
3. The node creates a container with the appropriate image.
4. The container loads the model into GPU memory.
5. Inference executes and returns the result.

Cold-start time depends on model size and node hardware — small specialists warm in seconds, while large LLMs can take several minutes to download and load. Subsequent requests hit the warm model and respond in well under a second.

Monitoring a cold start

Poll the task status endpoint to watch progress:

GET /api/builder/v1/tasks/{task_id}

{
  "status": "running",
  "status_detail": {
    "status": "downloading_model",
    "progress_pct": 45,
    "metadata": {
      "bytes_downloaded": 3221225472,
      "bytes_total": 7155210240,
      "download_speed_mbps": 85,
      "eta_seconds": 46
    }
  }
}

Possible status_detail.status values during a cold start:

• downloading_model — weights downloading (with progress %)
• loading_model — loading into GPU memory
• warming_up — container starting
• running — inference executing

Warm profiles

ColabHive automatically keeps popular models warm to minimize cold starts:

1. Tracks usage — models with recent requests get priority.
2. Respects a VRAM budget — a fraction of each node's memory is reserved so on-demand models can still cold-start; the warm set never fills all of VRAM.
3. Replicates smartly — popular models are warm on several nodes, rare ones on one or two.
4. Evicts idle models — when an on-demand request needs more VRAM than is free, the least-recently-used warm model is evicted to make room.

Checking readiness before you call

You can check readiness before making an inference request — useful for agents and pipelines that want to pick the fastest option:

curl "https://api.colabhive.com/api/builder/v1/inference/models?include_readiness=true" \
  -H "X-API-Key: $COLABHIVE_API_KEY"

Each model in the response includes readiness (warm, cached, or cold), warm_nodes, and an estimated latency.

import httpx

BASE = "https://api.colabhive.com/api/builder/v1"
HEADERS = {"X-API-Key": KEY}

models = httpx.get(f"{BASE}/inference/models?include_readiness=true", headers=HEADERS).json()

warm_chat = [m for m in models if m.get("readiness") == "warm" and m.get("task_type") == "chat"]
print("Warm chat models:", [m["model_name"] for m in warm_chat])

See Inference API — Model Readiness and Agent Integration — Intelligent Routing.

Synchronous vs. asynchronous

POST /endpoints/{id}/infer is synchronous by default: the server blocks up to sync_timeout_s (default 30 seconds, max 300) and returns the result directly. If the task has not finished within that window, the call returns its task_id with the task's current state instead (queued with its reason_code, or running); if the task ended while the call waited, the call returns that outcome.

For anything that might hit a cold start, dispatch asynchronously and poll:

from colabhive import ColabHive

client = ColabHive(api_key="hive_...", account_id="...")

resp = client.endpoints.infer(
    "qwen-2.5-7b-instruct-public",
    {"messages": [{"role": "user", "content": "Hello"}]},
    sync=False,  # return immediately with a task_id
)

task = client.endpoints.get_task(resp["task_id"])
while task["status"] not in ("completed", "succeeded", "failed"):
    detail = task.get("status_detail") or {}
    if detail.get("status") == "downloading_model":
        print(f"Downloading model... {detail.get('progress_pct', 0)}%")
    task = client.endpoints.get_task(resp["task_id"])

print(task.get("result"))

While the task is queued, the response carries reason_code (why it is waiting, for example no_warm_replica) and deferred_since. Waiting for capacity never fails a task: it keeps its place, runs when capacity arrives and its result stays readable afterwards. If your client gives up at its own deadline, poll the same task_id later instead of resubmitting. Every field is described in Get Task Result.

Streaming: token-by-token SSE is available on /v1/chat/completions with "stream": true. There is no infer:stream route — /infer is the typed path for every model family and most of them have nothing to stream. Explicit async is POST /endpoints/{id}/infer:async (HTTP 202); sync=false plus polling remains supported.

Best practices

Minimize cold starts

• Dispatch async and poll for models that may be cold — the sync window is short.
• Pre-warm — send a small request before production traffic.
• Prefer specialists for small tasks — embeddings, OCR, and STT are small and warm quickly.

Optimize inference

• Batch multiple inputs into one request where the model supports it.
• Right-size the model — mid-size (7B–8B) LLMs usually give the best latency/quality trade-off.
• Watch VRAM — very large models may evict other warm models.