Google Cloud API Gateway Model Routing in Production

Google Cloud API Gateway model routing gives engineering teams one OpenAI-compatible endpoint for selected Gemini, Claude, and OpenAI GPT models hosted in Agent Platform Model Garden. The useful part is not merely hiding provider URLs: the gateway can select a backend from the request's model value and transcode the request in flight. The dangerous part is equally important—an unknown model name silently reaches the configured default unless your client rejects it first.

This guide builds a small router, deploys it as a separate gateway, tests both routing branches, and adds the controls that the basic setup needs before production traffic reaches it.

Table of contents#

• What Google Cloud API Gateway model routing does
• Prerequisites before deployment
• Build the OpenAPI model router
• Deploy a separate router gateway
• Test rules and default fallback
• Prevent accidental fallback in the client
• Observe routing without model-level logs
• Roll out model routing safely
• Know when the preview is the wrong fit
• Frequently asked questions

What Google Cloud API Gateway model routing does#

Google announced model routing for API Gateway in Public Preview on August 4, 2026. A client sends an OpenAI-compatible JSON request to a POST operation. The gateway reads the body’s model string, compares it with rules in an OpenAPI 3.x document, chooses a configured backend, translates the payload to that backend’s native prediction schema, and returns the response.

That is deterministic string routing, not semantic routing. The preview does not inspect prompt meaning, estimate difficulty, optimize price, or fail over after a model error. If model matches a rule, the rule wins. If it does not, defaultModel wins.

Concern	What the preview provides	What your application still owns
Client interface	OpenAI-compatible JSON	Request validation and SDK adaptation
Selection	Exact model-string rules	Policy for choosing the string
Provider translation	In-flight transcoding	Feature compatibility tests
Upstream access	Gateway service account	Least-privilege IAM and client authentication
Visibility	Gateway logs and aggregate metrics	Per-model attribution and cost correlation
Failure handling	Router error categories	Retries, circuit breaking, and fallback policy

This makes the service a good managed ingress for a controlled model catalog. It is not a replacement for an application-level policy engine. If an agent decides which model to call, keep that decision explicit and auditable, just as you would for tool access in a programmatic tool-calling runtime.

Prerequisites before deployment#

Four constraints should determine the architecture before you write YAML.

First, the operator creating API configs and gateways needs roles/apigateway.admin. The service account attached to the API config needs roles/aiplatform.user so the gateway can call Agent Platform models. These are different trust boundaries: the human or CI principal provisions the gateway, while the runtime service account reaches the models.

Second, every backend referenced by one router must use the exact same scheme and hostname. You can use the global aiplatform.googleapis.com endpoint or one regional hostname, but you cannot combine hosts inside one router. The backend paths may differ.

Third, model routing is not an in-place toggle. Google’s current model-routing overview says a normal gateway cannot be converted to a routing gateway, and a routing gateway cannot have routing removed. Create a new API config and a new gateway when changing modes. API Gateway configs are immutable, so treat every routing change as a versioned deployment.

Fourth, a routing specification cannot mix routed and ordinary operations. Do not put a conventional x-google-backend endpoint beside a model-routed endpoint in the same OpenAPI document. Keep health checks, business APIs, and model ingress in separate gateway definitions.

Enable the required services and grant the runtime role:

export PROJECT_ID="your-project-id"
export GATEWAY_SA="model-router@${PROJECT_ID}.iam.gserviceaccount.com"

gcloud services enable \
  apigateway.googleapis.com \
  servicemanagement.googleapis.com \
  servicecontrol.googleapis.com \
  aiplatform.googleapis.com \
  --project="${PROJECT_ID}"

gcloud projects add-iam-policy-binding "${PROJECT_ID}" \
  --member="serviceAccount:${GATEWAY_SA}" \
  --role="roles/aiplatform.user"

Create the service account first if it does not exist. Avoid using a broad default service account for production; a dedicated identity makes permissions and audit events easier to reason about.

Build the OpenAPI model router#

The following OpenAPI 3.0.3 document defines one operation. Gemini is the required default, while an incoming model value of claude-opus-4-7 selects Claude. Both backend addresses share https://aiplatform.googleapis.com, and both use CONSTANT_ADDRESS, as required.

openapi: 3.0.3
info:
  title: model-router
  version: 1.0.0

x-google-api-management:
  backends:
    gemini-35-flash-lite:
      address: >-
        https://aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/global/publishers/google/models/gemini-3.5-flash-lite:generateContent
      deadline: 60.0
      pathTranslation: CONSTANT_ADDRESS
    claude-opus-47:
      address: >-
        https://aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/global/publishers/anthropic/models/claude-opus-4-7:rawPredict
      deadline: 60.0
      pathTranslation: CONSTANT_ADDRESS
  ai:
    models:
      routing:
        routers:
          production-router:
            defaultModel:
              backend: gemini-35-flash-lite
              targetModel: google/gemini-3.5-flash-lite
            rules:
              - model: claude-opus-4-7
                backend: claude-opus-47
                targetModel: anthropic/claude-opus-4-7

servers:
  - url: https://gateway.example.invalid

paths:
  /v1/chat/completions:
    post:
      operationId: createChatCompletion
      x-google-model-router: production-router
      responses:
        "200":
          description: Successful model response

Replace PROJECT_ID before deployment. The backend aliases such as claude-opus-47 are local YAML keys. targetModel is different: it must have the exact <provider>/<model-id> form, and the provider must be google, openai, or anthropic.

The selector has another provider-specific edge case. For Gemini and Claude routes, the incoming alias can be a convenient client-facing string mapped by a rule. For the OpenAI-compatible Agent Platform endpoint, Google documents that the selector forwarded upstream must itself be a valid publisher model identifier. A loose alias such as gpt-oss can produce 400 Malformed publisher model. Use the identifier accepted by the hosted endpoint and cover it in an integration test.

The router extension belongs on the post operation. It cannot appear at the document root or path level, and it cannot coexist with x-google-backend on the same operation. The complete validation rules and current example are in Google’s configuration guide.

Deploy a separate router gateway#

Save the specification as model-router.yaml, replace the placeholder, then create a versioned API config and a new gateway:

export API_ID="model-router"
export CONFIG_ID="model-router-20260807"
export GATEWAY_ID="model-router-v1"
export GATEWAY_LOCATION="us-central1"

sed "s/PROJECT_ID/${PROJECT_ID}/g" model-router.yaml > model-router.rendered.yaml

gcloud api-gateway api-configs create "${CONFIG_ID}" \
  --api="${API_ID}" \
  --openapi-spec=model-router.rendered.yaml \
  --backend-auth-service-account="${GATEWAY_SA}" \
  --project="${PROJECT_ID}"

gcloud api-gateway gateways create "${GATEWAY_ID}" \
  --api="${API_ID}" \
  --api-config="${CONFIG_ID}" \
  --location="${GATEWAY_LOCATION}" \
  --project="${PROJECT_ID}"

Wait until the gateway reports ACTIVE, then retrieve its final hostname:

export GATEWAY_HOST="$(
  gcloud api-gateway gateways describe "${GATEWAY_ID}" \
    --location="${GATEWAY_LOCATION}" \
    --project="${PROJECT_ID}" \
    --format='value(defaultHostname)'
)"

printf '%s
' "${GATEWAY_HOST}"

During the preview, that hostname ends in run.app. Do not persist a hostname obtained while the gateway is still being created; Google warns that it may not be final.

Test rules and default fallback#

Test the explicit branch with an authentication token appropriate for the client-to-gateway scheme you configured:

curl --fail-with-body \
  "https://${GATEWAY_HOST}/v1/chat/completions" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-opus-4-7",
    "messages": [
      {"role": "user", "content": "Explain recursion in one sentence."}
    ]
  }'

Then deliberately send an unknown selector:

curl --fail-with-body \
  "https://${GATEWAY_HOST}/v1/chat/completions" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "unknown-model",
    "messages": [
      {"role": "user", "content": "Return one short sentence."}
    ]
  }'

The second request does not prove that validation works. It proves that fallback works: unknown-model goes to the Gemini default. That can preserve availability, but it can also hide a typo, route regulated data to an unintended model, change latency, or change cost.

Also test the request shape against every target. “OpenAI-compatible” is an ingress contract, not a promise that every provider supports identical tools, structured-output options, stop behavior, token limits, or streaming event shapes after transcoding. For agents, a model swap can also change tool-selection behavior; the boundary and evaluation approach in the context engineering guide still applies.

Prevent accidental fallback in the client#

Because a default is mandatory, strict systems should reject unknown selectors before the gateway. The preview has an additional documented bug: a request missing model can be processed instead of rejected. Client validation closes both gaps.

const routes = new Set([
  "gemini-3.5-flash-lite",
  "claude-opus-4-7",
]);

export async function routedChat({ gatewayUrl, token, model, messages }) {
  if (typeof model !== "string" || !routes.has(model)) {
    throw new TypeError(`Unsupported model selector: ${String(model)}`);
  }
  if (!Array.isArray(messages) || messages.length === 0) {
    throw new TypeError("messages must be a non-empty array");
  }

  const response = await fetch(`${gatewayUrl}/v1/chat/completions`, {
    method: "POST",
    headers: {
      authorization: `Bearer ${token}`,
      "content-type": "application/json",
    },
    body: JSON.stringify({ model, messages }),
    signal: AbortSignal.timeout(65_000),
  });

  if (!response.ok) {
    const detail = await response.text();
    throw new Error(`Model gateway ${response.status}: ${detail.slice(0, 500)}`);
  }

  return response.json();
}

Keep the allowlist in the same configuration source that defines application policy. A model name should enter the router only after an intentional code or configuration review. If the caller is an AI agent, do not let unconstrained model output become the routing selector. This is the same least-authority principle used when securing remote MCP servers.

If fallback is a business requirement, make it explicit in your application telemetry: record the requested selector, policy version, gateway config ID, status, and latency. Never log raw prompts by default; they may contain secrets or personal data.

Observe routing without model-level logs#

Every routed request produces an API Gateway request log at:

projects/PROJECT_ID/logs/apigateway.googleapis.com%2Frequests

The entry includes request URL, status, latency, API, config, method, backend hostname, and responseDetails. Router failures use categories including model_router_application_error, model_router_timeout, model_router_upstream_error, and model_router_unavailable.

The preview’s important observability gap is per-request model attribution. All targets in a router share a hostname, so backendRequest.hostname cannot reveal which model handled the call. Google suggests single-rule test endpoints for an unambiguous test path or inspecting model-specific fields in the response. Neither is a complete production attribution system.

Build dashboards around what is actually available:

• request volume and response class from apigateway.googleapis.com/proxy/request_count;
• latency distributions from a log-based metric on httpRequest.latency;
• router error counts grouped by responseDetails;
• application-side counts grouped by validated selector and policy version;
• cost data reconciled separately with the Agent Platform billing dimensions available to your project.

Do not claim a per-model success rate from gateway logs alone until structured routing-decision logs exist. The response’s model field can help with debugging because the router echoes the configured targetModel, but logging it requires application instrumentation.

Roll out model routing safely#

A practical rollout treats the new gateway as a separate dependency, not a transparent URL swap.

1. Validate configuration in CI. Parse the YAML, reject placeholder project IDs, enforce HTTPS, confirm a shared host per router, and compare model rules with the application allowlist.
2. Deploy a new config and gateway. Use immutable, date-stamped config IDs. Record the generated hostname and config ID in deployment metadata.
3. Run a compatibility matrix. For every target, cover ordinary chat, maximum expected payload, SSE response streaming, timeouts, malformed bodies, missing model, and an unknown selector. Add tool or structured-output cases only if your application uses them.
4. Canary by application configuration. Send a small, non-critical traffic slice to the new hostname. Compare status, latency, output validity, usage, and cost with the current path.
5. Separate retries from model fallback. The gateway’s default handles an unmatched selector; it is not an upstream-error failover. Retry only idempotent requests, use a bounded attempt count, and decide separately whether a different model is an acceptable recovery.
6. Keep rollback boring. Preserve the old client endpoint and credentials until the canary meets its acceptance thresholds. Rolling back should mean changing the application’s gateway URL, not editing a live router.

Cold starts can affect the first request after a gateway scales to zero. Measure warm and cold paths separately instead of blending them into one average. The gateway permits timeouts up to 3,600 seconds, including long-lived SSE responses, but a large ceiling is not a sensible client timeout. Set a deadline based on the user journey and cancel abandoned requests.

Know when the preview is the wrong fit#

Use the managed router when you want a Google Cloud–hosted ingress, a small explicit model catalog, OpenAI-compatible text requests, public Agent Platform MaaS endpoints, and low operational overhead.

Choose another architecture when any of these are mandatory:

• VPC Service Controls or Private Service Connect;
• providers or private endpoints on different hosts;
• semantic, price-aware, latency-aware, or load-aware selection;
• automatic fallback after an upstream model failure;
• request streaming, gRPC, WebSockets, or Gemini Live;
• non-text modalities outside the Public Preview contract;
• precise per-request target attribution entirely inside gateway logs;
• one OpenAPI document mixing ordinary backends and model-routed operations.

Response streaming over server-sent events is supported, but that does not make the service a realtime bidirectional gateway. For voice or live multimodal sessions, use an endpoint designed for the required transport.

Frequently asked questions#

Does API Gateway choose a model from the prompt’s meaning?#

No. During Public Preview, it routes exclusively by the exact model value in the OpenAI-compatible JSON body. Semantic classification and policy decisions must happen before the request reaches the router.

What happens when the model name does not match a rule?#

The router sends the request to its required defaultModel. Production clients should validate selectors and emit an event for every rejected or intentionally defaulted request.

Can one router target Gemini, Claude, and OpenAI models?#

Yes, for supported pre-deployed MaaS models in Agent Platform Model Garden, provided all referenced backends use the identical URL scheme and hostname. Endpoint paths can differ.

Can an existing API Gateway be upgraded in place?#

No. A gateway deployed without model routing cannot be switched to routing, and a routing gateway cannot have the feature removed. Deploy a new API config and gateway, then move client traffic deliberately.

Does the router support realtime or bidirectional streaming?#

It supports response streaming with server-sent events. It does not support request-side streaming, gRPC, WebSockets, or Gemini Live during Public Preview.

Conclusion#

Google Cloud API Gateway model routing is a useful managed edge for a deliberate, small model catalog. Its production value comes from combining the OpenAPI router with strict client allowlists, separate immutable deployments, provider compatibility tests, bounded retries, and application-side attribution. Treat the default as a controlled fallback—not permission for unknown model names to pass silently—and the preview can simplify ingress without hiding the policy decisions your application still needs to own.