Google Model Routing vs LiteLLM vs OpenRouter

For a broader decision that also includes multimodal and policy-focused gateways, see Best OpenRouter Alternatives: Managed & Self-Hosted.

The first model gateway configuration usually looks harmless: one virtual name, two backends, and a fallback. Six months later it has per-team budgets, retry rules, regional restrictions, prompt caching, three API dialects, and a dashboard nobody fully trusts.

Google Cloud’s August 2026 model-routing preview offers a managed escape from running that gateway yourself. It accepts OpenAI-compatible requests at API Gateway, transcodes them, and routes to models hosted through Vertex AI Model Garden. That sounds close to LiteLLM and OpenRouter. The overlap is real, but the ownership model is completely different.

TL;DR

• Google Cloud API Gateway is the cleanest choice when model traffic already belongs in Google Cloud and rules-based routing is enough.
• LiteLLM offers the most control and provider reach, but your team owns upgrades, state, scaling, and routing regressions.
• OpenRouter is the fastest path to many models and providers with managed performance-aware routing, at the cost of another commercial data and billing layer.
• “OpenAI-compatible” does not mean every provider feature survives transcoding or fallback.
• Pick the governance boundary first; routing algorithms are the second decision.

What Google actually launched

Google Cloud API Gateway model routing entered Public Preview in August 2026. It provides a serverless ingress that accepts OpenAI-compatible prompts and routes them to specific models in Vertex AI Model Garden, including Gemini, Anthropic Claude, and OpenAI OSS-GPT models hosted in Google’s environment.

Google positions API Gateway as a lightweight, managed LLM ingress for rules-based routing, rate limits, and token tracking. Source: Google Developers Blog.

Two limits are easy to miss.

First, this is not a marketplace that forwards traffic to every model vendor’s public API. The documented targets are models available through Vertex AI Model Garden. Second, Google’s launch emphasizes rules defined in OpenAPI configuration, not a magical router that measures semantic difficulty and always picks the best model.

That narrower scope is useful. It keeps IAM, audit logs, quota, network perimeter, and billing inside a cloud platform many enterprises already govern.

Three products, three ownership models

Question	Google API Gateway	LiteLLM	OpenRouter
Who operates the gateway?	Google Cloud	Your team or LiteLLM service	OpenRouter
Model reach	Vertex AI Model Garden	Broad direct-provider support	Broad marketplace catalog
Credential model	Google IAM and cloud config	Your provider keys	OpenRouter credits or BYOK
Routing style	OpenAPI rules, managed gateway	Configurable algorithms and hooks	Managed provider/model selection
Data plane location	Google Cloud	Your infrastructure	OpenRouter plus selected provider
Custom code	Limited gateway policy	Deep Python/plugin control	Request-level policy fields
Best fit	GCP-governed workloads	Platform teams needing control	Teams optimizing speed and catalog reach

The table explains more than a feature checklist. With Google, you delegate operations but accept the Vertex boundary. With LiteLLM, you retain policy and data-plane control but inherit a production service. With OpenRouter, you delegate both provider integration and much of routing intelligence.

Google Cloud: boring in the useful way

API Gateway already handles authentication, rate limiting, deployment, and traffic policy for ordinary APIs. Model routing extends that familiar control plane instead of introducing a separate open-source proxy stack.

The attractive architecture is:

application
  -> Google Cloud API Gateway (OpenAI-compatible endpoint)
  -> routing rule + request transcoding
  -> Vertex AI Model Garden deployment

This is a good fit when security teams already understand Google IAM, VPC Service Controls, Cloud Logging, and organization policy. A platform team can publish one endpoint while model owners change backend mappings without touching application code.

The trade-off appears the moment a required provider is not in the supported Google-hosted set, or when routing needs live quality signals. API Gateway can be combined with Google’s Agent Gateway and broader Apigee controls, but each layer adds policy surface and cost. Public Preview also means availability, limits, and supported fields can change before general availability.

The release notes describe OpenAI-compatible input, in-flight transcoding, and routing to Vertex AI Model Garden. Source: Google Cloud release notes.

LiteLLM: control has an on-call cost

LiteLLM exposes a unified proxy and SDK across many providers. Its router supports strategies such as simple shuffle, least-busy, usage-based, latency-based, and cost-based routing, alongside cooldowns, retries, context-window fallbacks, virtual keys, team policy, and custom hooks.

That flexibility is the reason to choose it. It is also why configuration needs the same discipline as application code.

A production deployment should pin a version and test these cases before upgrade:

• streaming tool calls through the primary route;
• failover after a partial stream;
• signed or provider-specific reasoning blocks across conversation turns;
• context-window fallback without losing system instructions;
• budget and rate-limit behavior at key, team, and global levels;
• provider error mapping and retry amplification.

Provider normalization has a hard edge. Some fields can be dropped when a target does not support them; response-side state cannot always be translated safely. Anthropic signed thinking blocks are a good example: switching the next turn to an unrelated provider is not equivalent to a stateless text fallback.

LiteLLM offers the deepest routing control of the three, including per-key and per-team policy. Source: LiteLLM router settings.

For an existing detailed comparison of self-hosted gateway trade-offs, see LiteLLM vs OpenRouter.

OpenRouter: marketplace routing, not your proxy

OpenRouter combines a unified API, consolidated billing, multiple upstream providers per model, and automatic fallback. Request policy can constrain provider order, data collection, zero-data-retention endpoints, quantization, maximum price, latency, and throughput.

Its strongest feature is information your own new proxy does not have. OpenRouter observes provider availability and recent performance across shared traffic, then can route around degraded capacity. Rebuilding that signal from one company’s request volume is difficult.

The cost is a new intermediary. Data handling depends on both OpenRouter policy and the ultimately selected provider. BYOK changes billing and ordering behavior rather than removing OpenRouter from the request path. Teams should capture the served model and provider in response metadata, because fallback can change latency, price, capability, and jurisdiction.

OpenRouter’s fallback defaults also deserve scrutiny. Documentation says rate limits, downtime, moderation refusal, and even context-length validation can move a request to another model. That may improve availability, but a moderation fallback is a product decision, not merely infrastructure recovery.

Fallback is not always safe

The common gateway diagram draws a neat arrow from Model A to Model B. Real conversations carry state.

Failure	Safe default	Why
Connection fails before response	Retry equivalent deployment	No output reached the client
429 before response	Same model in another region/provider	Capability remains stable
Partial stream then disconnect	Do not silently replay	Duplicate output or tool calls possible
Context window exceeded	Compact or reject explicitly	Smaller context may lose instructions
Content policy refusal	Return policy result by default	Another model changes safety semantics
Tool call already executed	Resume with idempotency record	Retrying may mutate twice

This table should become gateway tests. A generic retries: 3 is dangerous for agent traffic because tool calls produce side effects. Assign an idempotency key before the model call, persist tool execution status, and distinguish “no bytes returned” from “stream interrupted after action.”

Cost comparisons need the whole bill

Do not compare only token prices.

For Google API Gateway, include gateway requests, logging, networking, and Vertex model charges. For LiteLLM, add compute, Redis or database state, observability, engineering time, and incident ownership. For OpenRouter, account for credit-purchase or BYOK fees, the served provider price, and any enterprise requirements.

The cheapest router on a spreadsheet can be expensive if it destroys prompt-cache locality. OpenRouter documents sticky routing for supported caches. A self-hosted LiteLLM deployment can implement affinity. Google routing rules can keep a workload on a known deployment. Measure cache-read savings and fallback frequency together.

A practical selection rule

Choose Google Cloud API Gateway when:

• workloads and governance already live in GCP;
• the required models are available through Vertex AI;
• managed rules, IAM, quotas, and logs matter more than custom algorithms.

Choose LiteLLM when:

• provider portability and self-hosted data flow are mandatory;
• your platform team can operate and test a gateway;
• routing requires custom policy, hooks, or internal cost signals.

Choose OpenRouter when:

• broad model access and rapid experimentation are the priority;
• managed provider health and fallback are valuable;
• the additional intermediary and billing model pass security review.

SandBase can sit behind or alongside the routing layer when applications also need isolated agent execution and external APIs. Keep model selection separate from tool authorization: choosing a cheap model must not silently grant it a broader execution environment.

Verdict

Google’s model routing is not a LiteLLM replacement for every provider. It is a managed, intentionally narrower answer for teams that want a stable OpenAI-compatible ingress into Google-hosted models. OpenRouter wins on immediate catalog and shared routing intelligence. LiteLLM wins on control.

My default would be Google for a GCP-centered enterprise, OpenRouter for a small team validating model-market fit, and LiteLLM only when the organization has a real platform owner. “We can run the container” is not the same as owning the gateway.

FAQ

Is Google Cloud model routing generally available?

No. Google announced the API Gateway feature in Public Preview in August 2026.

Can Google API Gateway route to any external model API?

The launch documentation focuses on supported foundation models available through Vertex AI Model Garden, not arbitrary public provider endpoints.

Is LiteLLM free to operate?

The open-source software can be self-hosted, but production compute, state, observability, upgrades, and on-call work still have a cost.

Does OpenRouter always use the model I requested?

It uses the requested model unless model fallback or router configuration permits alternatives. Provider routing may still choose among multiple providers serving that model.

Should agents retry failed model calls automatically?

Only when the system knows no side effect or partial output was committed. Agent tool calls require idempotency and execution-state tracking before automatic retry.