Providers & Routing
Provider Configuration
The providers section tells the gateway where to send model requests and which credential to use for each one. Each upstream endpoint is a target. Start with one target, make sure that it answers a request, and then add routing and fallback.
For a guided first request, use the quickstart. To change a running gateway, use the config-first workflow.
Syntax
The smallest providers section has one target with an ID, a provider alias, a model, and a credential reference:
pack:
name: config-providers-providers-1
version: 1.0.0
enabled: true
providers:
targets:
- id: primary
provider: openai
model: gpt-4o-mini
secret_key_ref:
env: OPENAI_API_KEY
policies:
chain:
- audit-logger
policy:
audit-logger: {}Use a model that your provider account can access.
Verify the first target
- Export
OPENAI_API_KEYin the gateway terminal. - Run
verdictan policy lint --file policy-config.yaml. - Run
verdictan gateway check --config policy-config.yaml. The result must show one active target. - Apply the file to a test agent and send the quickstart request with
"model": "gpt-4o-mini". - Find the selected provider and model in the response and the runtime event.
The configuration checks do not prove that the provider accepts the key or the model. Only a request proves that.
Fields
Section keys
| Field | Type | Default | Description |
|---|---|---|---|
targets | array | Provider targets. At least one is required. | |
routing | object | strategy: ordered | Route strategy and target filters |
retry_policy | object | max_retries: 3 | Retries against the same target before the gateway tries the next one |
circuit_breaker | object | enabled: true | Removes a target from the pool after consecutive failures |
zero_completion_insurance | object | enabled: true | Tries the next eligible target when a buffered chat completion is empty |
model_groups | array | Virtual model names that map to a list of target IDs | |
pipelines | array | Virtual model names that run more than one target in sequence or in parallel | |
logging | object | redact_message_bodies: false | Redaction of message bodies and API keys in traces and events |
Target fields
Each entry in providers.targets is one upstream endpoint. Cloud fields such as azure_deployment, aws_region, bedrock_model_family, gcp_project, and gcp_region are in Cloud Provider Configuration.
| Field | Type | Default | Description |
|---|---|---|---|
id | string | Unique target ID. Required. | |
provider | string | Provider alias. Required. The alias supplies default base_url, api_key_header, and path_template values. | |
model | string | Model that the target serves. Required for pipelines and most requests. | |
base_url | string | Alias default | Upstream base URL |
secret_key_ref | object | Credential reference with exactly one of env or store | |
api_key_header | string | Authorization | Header that carries the credential |
api_key_prefix | string | Bearer | Text before the credential value |
path_template | string | Alias default | Request path on the upstream host |
headers | map of strings | Extra request headers for this target | |
timeout_seconds | integer | 30 | Request timeout |
stream_timeout_seconds | integer | timeout_seconds | Timeout for streamed requests |
max_context_tokens | integer | Context limit for middle-out compression and pre-call checks | |
description | string | Text that semantic routing compares with the request | |
weight | number | Target weight for weighted_round_robin | |
region | string | Region label that routing.require_region compares | |
data_residency | object | Region list for this endpoint. See Data residency. | |
quantizations | array of strings | Quantization levels: fp32, fp16, bf16, int8, or int4 | |
data_policy | object | Declared data-handling terms for data-routing-policy | |
zdr | boolean | false | Short form of data_policy.zero_data_retention: true. If the full field is false, loading fails. |
pricing | object | Prices per million tokens: input_price_per_million, cached_input_price_per_million, output_price_per_million | |
models | array | Nested models, each with its own aliases, pricing, and escalation_routing | |
escalation_routing | object | Exactly one of team_id or user_id. A model-level value overrides the target value. | |
health_probe | object | Readiness probe with endpoint, interval_seconds (default 30), and timeout_ms (default 2000) | |
oauth2 | object | OAuth2 client credentials flow for the upstream bearer token | |
provider_type | string | From the alias | Provider type when the alias is ambiguous |
format | string | From the alias | Wire format. See Wire formats. |
The gateway also reads timeout_ms and stream_timeout_ms, but lint rejects them. Use the _seconds fields. It also reads a required flag that lint rejects. Use gateway check to find missing credentials before startup.
Routing fields
| Field | Type | Default | Description |
|---|---|---|---|
strategy | string | ordered | Route strategy. See Route strategies. |
order | array of strings | Target order | Explicit target order. Each ID must exist in targets. |
allow_fallbacks | boolean | true | When false, the gateway tries only the first eligible target. |
only | array of strings | Only these targets are eligible. Do not combine with ignore. | |
ignore | array of strings | These targets are not eligible. Do not combine with only. | |
require_region | string | Region that a target must declare. If no target matches, the request is blocked. | |
require_quantizations | array of strings | A target must declare at least one of these values. | |
semantic_embedding_provider | string | Cache embedding provider | Target ID that calculates embeddings for semantic routing |
semantic_similarity_threshold | number | 0.0 | Minimum similarity before semantic routing moves a target to the front |
enable_pre_call_checks | boolean | false | Estimates prompt tokens and skips targets whose max_context_tokens is too small |
measurement_window_seconds | integer | 300 | Sample window for lowest_latency and highest_throughput |
min_sample_count | integer | 5 | Samples that a target needs before the strategy ranks it |
exploration_ratio | number | 0.1 | Probability of an exploratory pick in sampled strategies |
warmup_ratio | number | 0.1 | Probability of moving an under-sampled target to the front |
Retry policy fields
| Field | Type | Default | Description |
|---|---|---|---|
max_retries | integer | 3 | Retries against the same target for transport and body read failures |
per_trigger | map | Retry limit for one trigger: rate_limit or server_error | |
backoff.strategy | string | exponential | exponential or fixed |
backoff.base_ms | integer | 500 | First wait for exponential. The wait doubles on each attempt. |
backoff.delay_ms | integer | 1000 | Wait for fixed |
Circuit breaker fields
| Field | Type | Default | Description |
|---|---|---|---|
enabled | boolean | true | Turns the circuit breaker on or off |
consecutive_failure_threshold | integer | 5 | Failures in a row before the circuit opens |
cooldown_seconds | integer | 30 | Time open before the circuit becomes half-open |
half_open_successes | integer | 1 | Successes while half-open before the circuit closes |
Route strategies
| Strategy | How the gateway picks a target |
|---|---|
ordered | In the configured order. The first healthy target wins. |
round_robin | Each target in turn |
weighted_round_robin | In turn, in proportion to each weight |
random | At random |
simple_shuffle | Shuffles the targets, then uses them in that order |
lowest_latency | The lowest observed P50 time to first token among sampled targets |
highest_throughput | The highest observed P50 throughput among sampled targets |
least_connections | The fewest active requests |
least_busy | The lowest load at selection time |
usage_based | The lowest total use |
semantic | The closest match between the request and the target description |
semantic changes the order only when the gateway can resolve an embedding target and at least one target has a description. If every similarity is below semantic_similarity_threshold, the order stays the same.
Provider aliases
The gateway knows more than 100 provider aliases. An alias supplies base_url, api_key_header, and path_template, but never a credential. The variable names below are examples. A target reads only the variable that its secret_key_ref names. Before a rollout, read the provider page under Integrations for its status and a test request.
Provider aliases and example credential variables
| Provider | Aliases | Example credential |
|---|---|---|
| OpenAI | openai | VERDICTAN_OPENAI_API_KEY |
| Anthropic | anthropic, claude | VERDICTAN_ANTHROPIC_API_KEY |
| Google AI Studio | google, gemini | VERDICTAN_GOOGLE_AI_STUDIO_API_KEY |
| Google Vertex AI | vertex-ai, vertex, google-vertex | Service account or OAuth2 |
| Azure OpenAI | azure-openai, azure | VERDICTAN_AZURE_OPENAI_API_KEY |
| AWS Bedrock | aws-bedrock, bedrock | SigV4. See AWS credentials. |
| Groq | groq | VERDICTAN_GROQ_API_KEY |
| Mistral | mistral | VERDICTAN_MISTRAL_API_KEY |
| DeepSeek | deepseek | VERDICTAN_DEEPSEEK_API_KEY |
| Together AI | togetherai, together | VERDICTAN_TOGETHER_API_KEY |
| Fireworks AI | fireworks, fireworks-ai | VERDICTAN_FIREWORKS_API_KEY |
| Cerebras | cerebras | VERDICTAN_CEREBRAS_API_KEY |
| Perplexity | perplexity | VERDICTAN_PERPLEXITY_API_KEY |
| OpenRouter | openrouter | VERDICTAN_OPENROUTER_API_KEY |
| GitHub Models | github, github-models | VERDICTAN_GITHUB_API_KEY |
| Cohere | cohere | VERDICTAN_COHERE_API_KEY |
| HuggingFace | huggingface, hf | VERDICTAN_HUGGINGFACE_API_KEY |
| Replicate | replicate | VERDICTAN_REPLICATE_API_KEY |
| Databricks | databricks | OAuth2 or VERDICTAN_DATABRICKS_API_KEY |
| Snowflake Cortex | snowflake-cortex | OAuth2 |
| Cloudflare AI | cloudflare-ai | VERDICTAN_CLOUDFLARE_AI_API_KEY |
| Cloudflare AI Gateway | cloudflare-gateway | The credential of the underlying provider, for example VERDICTAN_OPENAI_API_KEY |
| Alibaba or Qwen | alibaba, qwen, dashscope | VERDICTAN_QWEN_API_KEY |
| SambaNova | sambanova | VERDICTAN_SAMBANOVA_API_KEY |
| xAI (Grok) | xai, grok | VERDICTAN_XAI_API_KEY |
| Vercel AI Gateway | vercel, vercel-ai | VERDICTAN_VERCEL_API_KEY |
Wire formats
The gateway translates between these wire formats. It infers the format from a known alias. Set format when the provider page tells you to. A translator does not guarantee that every provider field works.
| Format | Providers |
|---|---|
openai | OpenAI, Azure, Groq, Mistral, Together, Fireworks, and most OpenAI-compatible endpoints |
anthropic | Anthropic, Claude |
cohere | Cohere |
huggingface | HuggingFace Inference Endpoints |
replicate | Replicate |
watsonx | IBM watsonx |
google-gemini | Google Gemini (AI Studio and Vertex) |
Examples
Credential resolution
A target that needs a key is active only when its secret_key_ref resolves. AWS Bedrock, Google Vertex AI, and OAuth2 targets use their own authentication methods.
pack:
name: config-providers-credentials
version: 1.0.0
enabled: true
providers:
routing:
strategy: ordered
targets:
- id: openai
provider: openai
model: your-openai-model
secret_key_ref:
env: VERDICTAN_OPENAI_API_KEY
- id: azure
provider: azure
provider_type: azure-openai
format: openai
model: your-azure-openai-model
base_url: https://replace-with-resource-name.openai.azure.com
secret_key_ref:
env: VERDICTAN_AZURE_OPENAI_API_KEY
azure_api_version: your-supported-api-version
azure_deployment: your-azure-deployment
policies:
chain:
- audit-logger
policy:
audit-logger: {}With ordered routing, a failure on openai sends the request to azure. To keep requests on the first target, set providers.routing.allow_fallbacks: false or pin the request to one provider.
Data policies
Record the data terms from your provider contract on each target. data-routing-policy uses these declarations to filter targets. Keep the contract evidence in its own workflow.
pack:
name: config-providers-providers-4
version: 1.0.0
enabled: true
providers:
targets:
- id: openai-zdr
provider: openai
model: your-openai-model
secret_key_ref:
env: VERDICTAN_OPENAI_API_KEY
data_policy:
zero_data_retention: true
training_opt_out: true
retention_days: 0
- id: azure-standard
provider: azure
provider_type: azure-openai
format: openai
model: your-azure-openai-model
base_url: https://replace-with-resource-name.openai.azure.com
secret_key_ref:
env: VERDICTAN_AZURE_OPENAI_API_KEY
azure_api_version: your-supported-api-version
azure_deployment: your-azure-deployment
data_policy:
zero_data_retention: false
training_opt_out: true
retention_days: 30
policies:
chain:
- audit-logger
policy:
audit-logger: {}Data Routing Policy lists every data_policy field.
Token pricing
A pricing block gives the gateway prices for spend tracking and routing scores. Take input_price_per_million, cached_input_price_per_million, and output_price_per_million from your provider contract. Do not copy prices from an example.
Nested models
One target can serve more than one model. Each model can have its own aliases, prices, and escalation route:
pack:
name: config-providers-providers-6
version: 1.0.0
enabled: true
providers:
targets:
- id: openai-prod
provider: openai
base_url: https://api.openai.com
secret_key_ref:
env: VERDICTAN_OPENAI_API_KEY
models:
- model_id: your-openai-model
aliases: [fast]
- model_id: your-second-openai-model
aliases: [reasoning]
escalation_routing:
team_id: 11111111-1111-4111-8111-111111111111
policies:
chain:
- audit-logger
policy:
audit-logger: {}When models is present, the gateway ignores the top-level model of the target.
Escalation route override
Send the escalations of one target to a team or a user. Set exactly one of team_id or user_id:
pack:
name: config-providers-providers-7
version: 1.0.0
enabled: true
providers:
targets:
- id: healthcare-openai
provider: openai
model: your-openai-model
secret_key_ref:
env: VERDICTAN_OPENAI_API_KEY
escalation_routing:
team_id: 11111111-1111-4111-8111-111111111111
policies:
chain:
- audit-logger
policy:
audit-logger: {}Health probes
A health probe runs in the background and sets the health status of the target:
pack:
name: config-providers-providers-8
version: 1.0.0
enabled: true
providers:
targets:
- id: openai-prod
provider: openai
model: your-openai-model
secret_key_ref:
env: VERDICTAN_OPENAI_API_KEY
health_probe:
endpoint: https://health.example.com/ready
interval_seconds: 30
timeout_ms: 2000
policies:
chain:
- audit-logger
policy:
audit-logger: {}The probe sends a plain GET, without the target credential or custom headers. Point endpoint at a readiness route that you control and that returns 2xx without authentication. The default, {base_url}/health, works only if the upstream has a public health endpoint.
OAuth2 authentication
Use oauth2 for providers that take an OAuth2 bearer token, such as Databricks, Snowflake Cortex, and custom endpoints:
pack:
name: config-providers-providers-9
version: 1.0.0
enabled: true
providers:
targets:
- id: databricks-prod
provider: databricks
model: your-databricks-model
base_url: https://my-workspace.databricks.net
oauth2:
grant_type: client_credentials
token_endpoint: https://accounts.example.com/oauth2/token
client_id: verdictan-gateway
client_secret_env: VERDICTAN_DATABRICKS_CLIENT_SECRET
scopes: [all-apis]
policies:
chain:
- audit-logger
policy:
audit-logger: {}Cloud provider fields
Azure deployment names, Bedrock signing, and Vertex project and region fields are in Cloud Provider Configuration. The provider page of each cloud has a complete credential and request example.
Region and quantization filtering
providers:
routing:
strategy: ordered
require_region: eu
require_quantizations:
- fp16
- int8
targets:
- id: openai-primary
provider: openai
model: your-openai-model
region: eu
quantizations: [fp16, int8]
secret_key_ref:
env: VERDICTAN_OPENAI_API_KEYrequire_region fails closed. If no target matches the region, the gateway blocks the request. It does not fall back to another region.
Data residency
A data_residency block lists the regions that a target can serve, taken from its provider contract. These values are your declarations. They do not move an endpoint or prove where it runs.
providers:
routing:
strategy: ordered
require_region: eu-west
targets:
- id: openai-eu
provider: openai
model: your-openai-model
secret_key_ref:
env: VERDICTAN_OPENAI_API_KEY
data_residency:
regions: [eu-west, eu-central]
data_center_locations: [Frankfurt]
sovereignty_compliant: true
- id: openai-us
provider: openai
model: your-openai-model
secret_key_ref:
env: VERDICTAN_OPENAI_API_KEY
data_residency:
regions: [us-east]Here, require_region: eu-west keeps openai-eu and excludes openai-us.
| Field | Type | Description |
|---|---|---|
regions | array of strings | Regions that this endpoint can serve. The gateway compares them with require_region and ignores case. At least one is required. |
data_center_locations | array of strings | Declared data center locations. Kept for audit review. Not used for routing. |
sovereignty_compliant | boolean | Declared sovereignty attestation. Kept for audit review. Not used for routing. |
When a target has data_residency, the gateway uses its regions and ignores the region field of the same target.
Data residency and zero-retention routing are separate checks. One selects targets by region and the other by declared retention terms. A target must pass every active check.
Target filtering
providers:
routing:
strategy: round_robin
only:
- openai-primary
targets:
- id: openai-primary
provider: openai
model: your-openai-model
secret_key_ref:
env: VERDICTAN_OPENAI_API_KEYContext window compression
Set max_context_tokens on a target to turn on middle-out compression for requests that are longer than that limit.
Zero-completion handling
For buffered POST /v1/chat/completions responses, the gateway checks for an empty completion by default. When it finds one, it tries the next eligible target. Turn this off if a second provider call is not approved:
zero_completion_insurance:
enabled: falseModel groups
A model group gives targets a stable name for clients. Add a group next to the targets of the starter example:
model_groups:
- name: fast-models
aliases: [fast, quick]
targets: [primary]A request with "model": "fast" now goes to primary. Each target ID in a group must exist in providers.targets.
For a fallback group, define a second group and set fallback_group to its name. Its targets follow the targets of the first group, and the usual provider attempt rules apply. See Provider Control Features for the fields.
Provider pipelines
A pipeline puts several targets behind one virtual model name. mode: sequence passes the output of one step to the next. mode: fan_out asks several models the same question and combines the answers.
pack:
name: config-providers-providers-25
version: 1.0.0
enabled: true
providers:
pipelines:
- name: draft-and-review
aliases: [reviewed-answer]
mode: sequence
steps:
- name: draft
target: writer
- name: review
target: reviewer
instruction: Review and improve the draft.
input_mode: replace
inject_as: user
targets:
- id: writer
provider: openai
model: your-openai-model
secret_key_ref:
env: VERDICTAN_OPENAI_API_KEY
- id: reviewer
provider: azure
provider_type: azure-openai
format: openai
model: your-azure-openai-model
base_url: https://replace-with-resource-name.openai.azure.com
secret_key_ref:
env: VERDICTAN_AZURE_OPENAI_API_KEY
azure_api_version: your-supported-api-version
azure_deployment: your-azure-deployment
policies:
chain:
- audit-logger
policy:
audit-logger: {}How pipelines work:
- The request
modelselects a pipeline, as it does formodel_groups. - Each step must use a target with a
model. For two models from one provider, declare two targets. - In a sequence,
input_mode: append(the default) adds the previous output to the original request.input_mode: replaceuses the previous output instead of the original request. inject_assets the role of the injected output:user(default),assistant, orsystem.fan_outruns each step on the original request.aggregation: concat(default) joins the outputs in order.aggregation: first_successreturns the first output without an error.- Pipelines support
POST /v1/chat/completionsandPOST /v1/responses, not embeddings or moderations. - The gateway rejects a pin of a pipeline model with
X-Verdictan-ProviderorX-Verdictan-Model.
Circuit breaker
circuit_breaker:
enabled: true
consecutive_failure_threshold: 5
cooldown_seconds: 30
half_open_successes: 1Retry policy
retry_policy:
max_retries: 3
per_trigger:
rate_limit: 5
backoff:
strategy: "exponential" # exponential | fixed
base_ms: 500These retries use the same target before the gateway tries the next one. per_trigger sets limits for rate_limit and server_error. Transport and body read failures use max_retries.
Logging controls
logging:
redact_message_bodies: trueWith redact_message_bodies: true, the gateway writes [REDACTED] instead of message content in traces and event payloads. redact_api_keys is true by default.
Complete provider example
pack:
name: openai-and-azure-routing
version: 1.0.0
enabled: true
providers:
targets:
- id: openai-primary
provider: openai
model: your-openai-model
secret_key_ref:
env: VERDICTAN_OPENAI_API_KEY
- id: azure-openai-fallback
provider: azure-openai
model: your-azure-openai-model
base_url: https://replace-with-resource-name.openai.azure.com
secret_key_ref:
env: VERDICTAN_AZURE_OPENAI_API_KEY
azure_deployment: your-azure-deployment
azure_api_version: your-supported-api-version
routing:
strategy: ordered
circuit_breaker:
enabled: true
consecutive_failure_threshold: 5
cooldown_seconds: 30
retry_policy:
max_retries: 3
backoff:
strategy: exponential
base_ms: 500
policies:
chain:
- prompt-injection
- pii-detector
- quality-scorer
policy:
prompt-injection:
response:
action: block
pii-detector:
action: redact
quality-scorer:
thresholds:
min_aggregate: 0.8retry_policy retries the same target. When its retries are used up, the gateway tries the next eligible target.
Limits and failure behavior
Provider attempts and fallback
The gateway orders the eligible targets and retries each one according to retry_policy before it moves to the next. The first successful response ends the loop.
A transport error always moves the request to the next target. An HTTP response moves it only when its class is in the trigger list:
rate_limit: HTTP429server_error: HTTP408,500,502,503, and504, and transport timeouts
An HTTP 401, or a 403 that is not a content filter block, also moves the request to the next target. This happens one time for each target.
The gateway returns every other upstream response to the client, including content filter and context window errors. These rules apply to every provider.
To use only the first eligible target, set providers.routing.allow_fallbacks: false or pin the request to a provider. Test each request family, because a streamed response cannot move to another target after output starts.
Load errors
providers.targetsmust have at least one target.- A target that needs a credential and has
required: truestops the gateway at startup if itssecret_key_refdoes not resolve. Withrequired: false, the gateway leaves the target out of routing. routing.onlyandrouting.ignoretogether are a load error.- An ID in
routing.orderthat is not intargetsis a load error. - Quantization labels must be
fp32,fp16,bf16,int8, orint4. Lint rejects the runtime-onlyfp8. The gateway rejects the schema-onlyawq,gptq, andggufat load time. - The gateway rejects the removed fields
providers.routing.upstream_fallback_policyandproviders.ab_test.
Region blocks
- If
require_regionexcludes every target and at least one target hasdata_residency, the gateway returns HTTP451with error codeno_compliant_provider. - If no target has
data_residency, the gateway comparesrequire_regionwith eachregion. No match returns HTTP403with error codeno_eligible_provider. verdictan policy lintreports arequire_regionvalue that no target supports.
Next steps
- Cloud Provider Configuration: cloud-specific target fields.
- Provider Control Features: model group fallback, health monitoring, and more.
- Routes and Consumer Groups: select policy chains for paths and client groups.