Begin with measurement
Use Observe mode or a low-volume gateway project to establish request volume, model mix, cost, latency, errors, and workload patterns before changing production routing.Keep environments separate
Use separate projects and Relay keys for development, staging, and production when they need independent credentials, limits, traffic, or reports. Store each key once and rotate it through the application’s secret-management process.Change one routing boundary at a time
1
Define candidates
Include only callable, routable models that meet the workload’s capability
and policy requirements.
2
Set a fallback
Choose a model that can complete the request when automatic routing cannot.
3
Replay decisions
Estimate distribution and cost on historical traffic.
4
Evaluate quality
Use shadow comparison or a quality evaluation on representative traffic.
5
Increase traffic gradually
Start with a bounded project or key and monitor routing reasons, errors,
cost, quality, and latency.
Preserve one execution authority
When Relay is the gateway, Relay executes the provider call. In route-only or existing-gateway mode, your infrastructure owns provider credentials, retries, timeouts, streaming, and final fallback. Avoid splitting those responsibilities ambiguously across two systems.Budget evaluation work
Shadow calls, quality evaluations, and custom-router training can create additional provider spend. Set their budgets independently of live request limits.Choose an integration mode
Decide how much of the request path Relay should own.