NVIDIA Switchyard: Run Coding Agents on Any LLM

The awkward part of changing the model behind a coding agent is rarely the model. It is the wire protocol. Claude Code expects Anthropic-shaped messages, Codex speaks OpenAI’s Responses API, and a local vLLM server usually exposes Chat Completions. Point one at the wrong endpoint and the failure can look like a bad model: missing tool calls, broken streaming, or context that disappears between turns.

NVIDIA NeMo Switchyard puts a translation and routing layer in that gap. It can launch Claude Code, Codex, or OpenClaw against NVIDIA NIM, vLLM, Ollama, OpenRouter, or another OpenAI-compatible backend without changing the agent itself.

TL;DR

• Switchyard is an open-source LLM proxy, protocol translator, and router.
• Its most useful trick is running an existing coding-agent client against a backend that uses a different API format.
• Routing profiles can send easy work to a cheap model and escalate harder requests.
• It adds another production component, so timeout budgets, credential boundaries, and translation tests matter.
• Use it when client compatibility or model portability matters more than having the smallest possible stack.

The problem is three APIs pretending to be one

OpenAI Chat Completions, the OpenAI Responses API, and Anthropic Messages all carry conversations, tools, and model output. They do not represent them the same way. Tool identifiers, streaming events, reasoning fields, stop reasons, and error objects differ.

That difference gets painful with coding agents because the conversation is not plain text. A single task can contain hundreds of tool calls and streaming events. A proxy that only renames messages to input will appear to work until the first parallel tool call or retry.

The official Switchyard repository documents translation between OpenAI Chat, Anthropic Messages, and OpenAI Responses. Source: NVIDIA NeMo.

Switchyard treats translation as a first-class stage. Its documented processing chain is:

request components -> LLM backend -> response components -> translation engine

That fixed shape is a good design decision. Authentication, routing, buffering, and logging can change independently, while every valid chain still makes exactly one backend call before translating the response for the client.

A working Claude Code launch

The public package is named nemo-switchyard, while imports and the command use switchyard. That naming mismatch is the first small trap.

python -m venv .venv
source .venv/bin/activate
python -m pip install "nemo-switchyard[cli,server]"

switchyard launch claude

The launcher starts the proxy, configures the selected client for that proxy, and opens the agent. Switchyard also provides launchers for Codex and OpenClaw:

switchyard launch codex
switchyard launch openclaw

For a persistent configuration, explicitly register a target rather than placing a secret in a routing file:

switchyard configure \
  --target provider \
  --provider openrouter \
  --api-key "$OPENROUTER_API_KEY" \
  --base-url https://openrouter.ai/api/v1 \
  --no-tui \
  --no-model-discovery

The project deliberately requires --api-key when writing a provider configuration for non-interactive use. A routing bundle describes where traffic goes; it should not become a casually committed secret store.

Switchyard includes dedicated launch paths for Claude Code, Codex, and OpenClaw rather than asking users to manually rewrite each client’s configuration.

Routing profiles are more useful than round robin

Sending every request to a single alternative model proves compatibility, but routing is where the proxy starts earning its keep. Switchyard supports single-model passthrough, random routing, classifier-based routing, signal-driven escalation, and custom routers.

A profile can expose one virtual model name such as smart, then choose a backend internally. The exact profile keys depend on the installed release, so validate configuration against the repository’s current examples before deploying. Publishing a plausible-looking YAML block here would be less useful than being explicit about that moving boundary.

The operational pattern is stable even when keys change:

1. The coding agent calls one virtual model.
2. A request-side component extracts routing signals.
3. The selected backend receives the translated request.
4. Failed or evicted requests can escalate to a stronger target.
5. Statistics record latency, token use, and estimated cost by route.

This is better than naïve round robin for agent work. Coding turns are not interchangeable. Repository search and short formatting edits can run on a cheaper model; architectural changes and tool-recovery turns deserve the stronger one.

Switchyard vs LiteLLM vs provider-native routing

Switchyard overlaps with model gateways, but its coding-agent launchers and protocol translation make the intended use unusually specific.

Decision	Switchyard	LiteLLM	Provider-native router
Run Claude Code/Codex on another backend	Best fit	Possible with configuration	Usually not the goal
OpenAI, Anthropic, Responses translation	Core feature	Broad gateway support	Limited to provider ecosystem
Local vLLM/Ollama targets	Yes	Yes	Usually no
Custom routing logic	Python profiles and components	Callbacks and routing policies	Product-specific rules
Managed operations	Self-hosted	Self-hosted or managed proxy	Yes
Smallest operational burden	No	No	Yes

Use Switchyard when the client is non-negotiable but the model is not. Use LiteLLM when many applications need one general gateway. Use a cloud router when your models already live in that cloud and you do not want to own another proxy.

For a broader gateway comparison, read LiteLLM vs OpenRouter. If the proxy will execute agent-generated code, pair it with the isolation controls in our AI sandbox comparison.

Production failure modes

The happy path takes minutes. The failures show up after the agent has been running for an hour.

Streaming translation can break clients subtly

A missing terminal event can leave a client spinner running forever even though the backend finished. Preserve event order, tool-call IDs, finish reasons, and error semantics in integration tests. A plain curl smoke test does not cover this.

The proxy consumes part of the timeout budget

Classification, translation, and retry logic add latency before and after inference. Set separate timeouts for route selection, backend calls, and streaming idle periods. One global 120-second timeout makes diagnosis needlessly hard.

Context limits need route-aware handling

The weak and strong models may have different context windows. Switchyard’s chain can react to context-window errors, but the safe policy is to calculate headroom before routing. Retrying a 200K-token request against three 128K models only turns one predictable error into three expensive ones.

Logs contain source code

Request statistics are useful, but full payload logs may contain proprietary code, secrets printed by tools, and user data. Keep metrics by default; gate payload capture behind an explicit debug mode and short retention.

Routing is only production-ready when route choice, latency, tokens, failures, and fallback behavior are observable together.

Where SandBase fits

Switchyard controls how an agent reaches a model. It does not isolate the shell commands or code that the agent runs. Those are different trust boundaries.

SandBase can provide the model/API layer and isolated execution environment behind an agent workflow, while Switchyard handles client compatibility and routing. A practical deployment keeps three policies separate:

• Model policy: which backend receives each request.
• Execution policy: which files, network destinations, and processes agent tools may access.
• Approval policy: which mutations require a human decision.

Collapsing all three into a prompt is convenient until the first malicious repository instruction reaches a tool call.

Verdict

Switchyard is compelling because it solves an unglamorous compatibility problem without pretending the APIs are identical. The dedicated coding-agent launchers reduce setup friction, while the typed chain and routing profiles leave room for serious deployments.

I would use it for internal evaluation, model migration, and teams that want Claude Code or Codex UX with self-hosted inference. I would not put it between every application and every model by default. A proxy is another service to patch, observe, and keep out of the critical path when it adds no value.

FAQ

Is NVIDIA Switchyard an LLM?

No. Switchyard is a proxy, translation, and routing layer that sends requests to LLM backends.

Can Switchyard run Claude Code with a local model?

Yes, if the local backend is reachable through a supported OpenAI-compatible target such as vLLM or Ollama and the model can handle the agent’s tool-use workload.

Does Switchyard replace LiteLLM?

Not universally. Switchyard is especially attractive for coding-agent launchers and cross-protocol translation; LiteLLM is a broader general-purpose model gateway.

Does protocol compatibility guarantee good agent performance?

No. Translation makes the request valid, but the selected model still needs reliable tool calling, instruction following, enough context, and suitable coding ability.

Should Switchyard store provider API keys in routing YAML?

No. Keep credentials in environment variables or a secret manager and treat routing configuration as non-secret deployment code.