Deploying Hy4 preview: the capacity checklist for a 770B MoE

Hy4 preview’s open weights make self-hosting possible, but a 770B total-parameter, 49B-active MoE cannot be sized from the active count alone. Weight storage, quantization, KV cache, expert routing, interconnect, and upgrade operations all belong in the capacity model.

Key takeaway

Open weights provide control, but a 770B MoE still requires a serious capacity and networking plan.

Benchmark quantization, concurrency, long context, cold starts, and rollback—not just one warm request.

Isolate tool execution and restrict outbound access; model weights do not remove security duties.

Establish a hosted quality baseline before committing to the operational burden of self-hosting.

Total parameters determine how much weight must remain available; active parameters influence per-token compute; context length drives potential KV-cache growth. A 49B active count does not mean only 49B parameters need to be stored, and a 1M window does not mean every request should use it.

Before deployment, verify the official weight format, license, and supported runtimes. Benchmark quantized variants on a fixed task, then load-test KV cache at the target concurrency and context length. Measure tail latency, communication overhead, and quality—not just peak throughput. Keep model configuration and rollback artifacts immutable.

Hosted APIs may still be cheaper for early evaluation or bursty traffic. Include GPU depreciation, power, bandwidth, on-call time, upgrades, and observability when comparing against TokenHub, OpenRouter, or a gateway such as SandBase. Confirm live catalog availability before claiming direct integration.

Production nodes also need least-privilege network access, data isolation, approval for tool calls, and monitoring for first-token latency, generation throughput, queue time, cache use, and retries. Open weights change deployment control; they do not remove security responsibility.

Bottom line: establish a quality and cost baseline through a hosted API, then decide whether the operational burden of a 770B MoE is justified by your data boundary and sustained workload.

A capacity worksheet for the first pilot

List the target precision, replica count, maximum context, concurrency, and service-level objective before choosing hardware. Weight memory is only the first line. Add KV-cache headroom, expert-routing communication, activation buffers, operating-system overhead, and room for rolling upgrades. A pilot that works for one warm request can still fail when several users request long contexts at once.

The test plan should include cold start, warm steady state, burst traffic, and a deliberately cancelled request. Measure p50 and p95 time to first token, decode throughput, queue time, error rate, and quality on the same fixture. Repeat after quantization and after a runtime upgrade. Keep the unquantized or reference run available as a quality anchor.

Operational boundaries matter

Treat model weights as production software. Pin the repository commit and runtime versions, scan the downloaded artifacts, restrict outbound network access, and make logs redact prompts that contain private data. Add health checks that exercise a tiny deterministic request, not just process liveness. A failed expert shard or stale cache should produce an actionable alert and a safe rollback.

For agent workloads, isolate tool execution from inference workers. The model may propose shell commands, network calls, or file writes, but an approval layer should decide which actions are allowed. This keeps the flexibility of open weights without turning a model-serving node into an unbounded automation host.

Finally, model the exit path. Decide how to drain traffic, preserve in-flight requests, and restore the previous model if quality or latency regresses. Keep a small hosted fallback for incident response, but test it before an outage. Document who can approve a weight upgrade, who owns the benchmark fixture, and which logs are retained for post-incident review. The value of self-hosting is control; that control is only real when the team can operate, audit, and roll back the service under pressure.