Open-Weight Models in Production: A Migration Checklist for AI Teams

The demo usually works. The bill arrives later, a model update changes tokenizer behavior, or a license question appears in procurement. That is the point where “we can run an open-weight model” stops being an architecture decision and becomes an operations project.

I would not begin with a leaderboard. Begin with the failure you are trying to remove: provider lock-in, a data-residency constraint, predictable throughput, or a model that must run beside your own tools. Then test the whole path—license, weights, serving, evaluation, and rollback—before moving traffic.

Key takeaway

• Open weights reduce one dependency; they do not remove GPU, serving, upgrade, or compliance work.
• Evaluate the model and the serving stack together on your own task set.
• Keep a hosted fallback until latency, cost, and quality are measured over real traffic.
• Pin model revisions, tokenizer revisions, quantization, and server settings as one release unit.

What “open” needs to mean in your checklist

“Open-weight” is not a license category. Record four separate answers:

Question	Evidence to collect	Why it changes the decision
Can we download the weights?	Model repository and access terms	Gated access can change deployment timing.
Can we use the weights commercially?	License text and acceptable-use policy	A public checkpoint may still have restrictions.
Can we modify or quantize them?	License plus provider notes	Quantization and fine-tuning may create a derivative.
Can we redistribute the result?	License, model card, and package terms	Internal serving is different from shipping a model.

The Hugging Face model index is useful for discovery, but its ranking is not a license decision. Keep the model card and license URL in the release record.

Figure 1. A model index helps find candidates; the repository’s license and model card decide whether a candidate fits your use case.

Measure the serving path, not just tokens per second

A model that looks cheap on paper can lose after GPU memory, batching, cold starts, and operational time are included. Build a small matrix with:

• time to first token and tokens per second at your target concurrency;
• prompt lengths that match your real requests;
• structured-output or tool-call success rate;
• failure behavior when the context is near the limit;
• cost per successful task, not cost per generated token.

Run the same prompts through the hosted baseline and the self-hosted candidate. Record the model revision, quantization, GPU type, runtime version, batch settings, and tokenizer. Otherwise the comparison cannot be repeated after an upgrade.

Choose a serving layer deliberately

Projects such as vLLM can provide a production-oriented inference server, but a server is not a complete platform. You still need authentication, quotas, request tracing, autoscaling, health checks, and a policy for oversized prompts.

Figure 2. The inference server is one layer in the deployment; application policy and observability remain yours.

Worth flagging: optimizing throughput before measuring queue time is a common trap. A high tokens-per-second number can coexist with a poor p95 experience if requests wait behind a large batch.

Keep the application contract stable

Your application should call a stable internal contract while the model route remains configurable. Keep model-specific behavior in an adapter:

MODEL = {
    "name": "provider/model-revision",
    "tokenizer": "provider/tokenizer-revision",
    "quantization": "bf16",
    "max_context": 32768,
}

def evaluate_response(response: dict) -> bool:
    return bool(response.get("text")) and response.get("finish_reason") in {"stop", "tool_calls"}

The point is not this exact configuration. It is the release boundary: changing weights or quantization should trigger the same evaluation and rollback process as changing application code.

Plan for the reasons self-hosting fails

Most migration plans undercount the boring failures:

1. Memory pressure: a longer prompt or concurrent batch causes OOM. Add admission limits and a clear 429 path.
2. Quality drift: a new quantization changes tool-call formatting. Keep golden tasks and schema-validation tests.
3. Capacity gaps: a single GPU is unavailable. Route to a hosted fallback instead of silently queueing forever.
4. Security gaps: downloaded artifacts and logs contain sensitive prompts. Scan images, restrict egress, and define retention.

The hosted fallback is not an admission of defeat. It is how you separate a controlled experiment from an outage.

Where SandBase fits

SandBase can supply the hosted model/API side of a routing plan while your team evaluates an open-weight path on infrastructure it controls. The boundary matters: SandBase does not make a third-party checkpoint self-hosted, and self-hosting does not remove the need for a stable API contract.

If your workflow needs more than an LLM—search, social data, image, or video APIs—keep those calls behind their own adapters. Start with the unified API guide and use the Docs quickstart for the current authentication and request path.

Figure 3. Keep the hosted fallback’s authentication and request contract explicit while the self-hosted path is evaluated.

Production sign-off

• License and redistribution terms reviewed by the owner who approves them.
• Model, tokenizer, quantization, runtime, and server settings pinned.
• Task-level quality and tool-call tests pass against the hosted baseline.
• p50/p95 latency, queue time, error rate, and cost per successful task are recorded.
• Rollback and hosted fallback have been exercised, not merely documented.
• Logs, downloaded artifacts, and egress policy have an explicit owner.

The answer may still be “keep the hosted model.” That is a valid result when the operational work costs more than the dependency it removes.

Sources

• Hugging Face text-generation models
• vLLM documentation
• SandBase Docs