Skip to main content

Providers & Routing

Provider Configuration

The providers section tells the gateway where to send model requests and which credential to use for each one. Each upstream endpoint is a target. Start with one target, make sure that it answers a request, and then add routing and fallback.

For a guided first request, use the quickstart. To change a running gateway, use the config-first workflow.

Syntax​

The smallest providers section has one target with an ID, a provider alias, a model, and a credential reference:

policy-config.yaml
pack:
  name: config-providers-providers-1
  version: 1.0.0
  enabled: true
providers:
  targets:
  - id: primary
    provider: openai
    model: gpt-4o-mini
    secret_key_ref:
      env: OPENAI_API_KEY
policies:
  chain:
  - audit-logger
policy:
  audit-logger: {}

Use a model that your provider account can access.

Verify the first target​

  1. Export OPENAI_API_KEY in the gateway terminal.
  2. Run verdictan policy lint --file policy-config.yaml.
  3. Run verdictan gateway check --config policy-config.yaml. The result must show one active target.
  4. Apply the file to a test agent and send the quickstart request with "model": "gpt-4o-mini".
  5. Find the selected provider and model in the response and the runtime event.

The configuration checks do not prove that the provider accepts the key or the model. Only a request proves that.

Fields​

Section keys​

FieldTypeDefaultDescription
targetsarrayProvider targets. At least one is required.
routingobjectstrategy: orderedRoute strategy and target filters
retry_policyobjectmax_retries: 3Retries against the same target before the gateway tries the next one
circuit_breakerobjectenabled: trueRemoves a target from the pool after consecutive failures
zero_completion_insuranceobjectenabled: trueTries the next eligible target when a buffered chat completion is empty
model_groupsarrayVirtual model names that map to a list of target IDs
pipelinesarrayVirtual model names that run more than one target in sequence or in parallel
loggingobjectredact_message_bodies: falseRedaction of message bodies and API keys in traces and events

Target fields​

Each entry in providers.targets is one upstream endpoint. Cloud fields such as azure_deployment, aws_region, bedrock_model_family, gcp_project, and gcp_region are in Cloud Provider Configuration.

FieldTypeDefaultDescription
idstringUnique target ID. Required.
providerstringProvider alias. Required. The alias supplies default base_url, api_key_header, and path_template values.
modelstringModel that the target serves. Required for pipelines and most requests.
base_urlstringAlias defaultUpstream base URL
secret_key_refobjectCredential reference with exactly one of env or store
api_key_headerstringAuthorizationHeader that carries the credential
api_key_prefixstringBearer Text before the credential value
path_templatestringAlias defaultRequest path on the upstream host
headersmap of stringsExtra request headers for this target
timeout_secondsinteger30Request timeout
stream_timeout_secondsintegertimeout_secondsTimeout for streamed requests
max_context_tokensintegerContext limit for middle-out compression and pre-call checks
descriptionstringText that semantic routing compares with the request
weightnumberTarget weight for weighted_round_robin
regionstringRegion label that routing.require_region compares
data_residencyobjectRegion list for this endpoint. See Data residency.
quantizationsarray of stringsQuantization levels: fp32, fp16, bf16, int8, or int4
data_policyobjectDeclared data-handling terms for data-routing-policy
zdrbooleanfalseShort form of data_policy.zero_data_retention: true. If the full field is false, loading fails.
pricingobjectPrices per million tokens: input_price_per_million, cached_input_price_per_million, output_price_per_million
modelsarrayNested models, each with its own aliases, pricing, and escalation_routing
escalation_routingobjectExactly one of team_id or user_id. A model-level value overrides the target value.
health_probeobjectReadiness probe with endpoint, interval_seconds (default 30), and timeout_ms (default 2000)
oauth2objectOAuth2 client credentials flow for the upstream bearer token
provider_typestringFrom the aliasProvider type when the alias is ambiguous
formatstringFrom the aliasWire format. See Wire formats.

The gateway also reads timeout_ms and stream_timeout_ms, but lint rejects them. Use the _seconds fields. It also reads a required flag that lint rejects. Use gateway check to find missing credentials before startup.

Routing fields​

FieldTypeDefaultDescription
strategystringorderedRoute strategy. See Route strategies.
orderarray of stringsTarget orderExplicit target order. Each ID must exist in targets.
allow_fallbacksbooleantrueWhen false, the gateway tries only the first eligible target.
onlyarray of stringsOnly these targets are eligible. Do not combine with ignore.
ignorearray of stringsThese targets are not eligible. Do not combine with only.
require_regionstringRegion that a target must declare. If no target matches, the request is blocked.
require_quantizationsarray of stringsA target must declare at least one of these values.
semantic_embedding_providerstringCache embedding providerTarget ID that calculates embeddings for semantic routing
semantic_similarity_thresholdnumber0.0Minimum similarity before semantic routing moves a target to the front
enable_pre_call_checksbooleanfalseEstimates prompt tokens and skips targets whose max_context_tokens is too small
measurement_window_secondsinteger300Sample window for lowest_latency and highest_throughput
min_sample_countinteger5Samples that a target needs before the strategy ranks it
exploration_rationumber0.1Probability of an exploratory pick in sampled strategies
warmup_rationumber0.1Probability of moving an under-sampled target to the front

Retry policy fields​

FieldTypeDefaultDescription
max_retriesinteger3Retries against the same target for transport and body read failures
per_triggermapRetry limit for one trigger: rate_limit or server_error
backoff.strategystringexponentialexponential or fixed
backoff.base_msinteger500First wait for exponential. The wait doubles on each attempt.
backoff.delay_msinteger1000Wait for fixed

Circuit breaker fields​

FieldTypeDefaultDescription
enabledbooleantrueTurns the circuit breaker on or off
consecutive_failure_thresholdinteger5Failures in a row before the circuit opens
cooldown_secondsinteger30Time open before the circuit becomes half-open
half_open_successesinteger1Successes while half-open before the circuit closes

Route strategies​

StrategyHow the gateway picks a target
orderedIn the configured order. The first healthy target wins.
round_robinEach target in turn
weighted_round_robinIn turn, in proportion to each weight
randomAt random
simple_shuffleShuffles the targets, then uses them in that order
lowest_latencyThe lowest observed P50 time to first token among sampled targets
highest_throughputThe highest observed P50 throughput among sampled targets
least_connectionsThe fewest active requests
least_busyThe lowest load at selection time
usage_basedThe lowest total use
semanticThe closest match between the request and the target description

semantic changes the order only when the gateway can resolve an embedding target and at least one target has a description. If every similarity is below semantic_similarity_threshold, the order stays the same.

Provider aliases​

The gateway knows more than 100 provider aliases. An alias supplies base_url, api_key_header, and path_template, but never a credential. The variable names below are examples. A target reads only the variable that its secret_key_ref names. Before a rollout, read the provider page under Integrations for its status and a test request.

Provider aliases and example credential variables
ProviderAliasesExample credential
OpenAIopenaiVERDICTAN_OPENAI_API_KEY
Anthropicanthropic, claudeVERDICTAN_ANTHROPIC_API_KEY
Google AI Studiogoogle, geminiVERDICTAN_GOOGLE_AI_STUDIO_API_KEY
Google Vertex AIvertex-ai, vertex, google-vertexService account or OAuth2
Azure OpenAIazure-openai, azureVERDICTAN_AZURE_OPENAI_API_KEY
AWS Bedrockaws-bedrock, bedrockSigV4. See AWS credentials.
GroqgroqVERDICTAN_GROQ_API_KEY
MistralmistralVERDICTAN_MISTRAL_API_KEY
DeepSeekdeepseekVERDICTAN_DEEPSEEK_API_KEY
Together AItogetherai, togetherVERDICTAN_TOGETHER_API_KEY
Fireworks AIfireworks, fireworks-aiVERDICTAN_FIREWORKS_API_KEY
CerebrascerebrasVERDICTAN_CEREBRAS_API_KEY
PerplexityperplexityVERDICTAN_PERPLEXITY_API_KEY
OpenRouteropenrouterVERDICTAN_OPENROUTER_API_KEY
GitHub Modelsgithub, github-modelsVERDICTAN_GITHUB_API_KEY
CoherecohereVERDICTAN_COHERE_API_KEY
HuggingFacehuggingface, hfVERDICTAN_HUGGINGFACE_API_KEY
ReplicatereplicateVERDICTAN_REPLICATE_API_KEY
DatabricksdatabricksOAuth2 or VERDICTAN_DATABRICKS_API_KEY
Snowflake Cortexsnowflake-cortexOAuth2
Cloudflare AIcloudflare-aiVERDICTAN_CLOUDFLARE_AI_API_KEY
Cloudflare AI Gatewaycloudflare-gatewayThe credential of the underlying provider, for example VERDICTAN_OPENAI_API_KEY
Alibaba or Qwenalibaba, qwen, dashscopeVERDICTAN_QWEN_API_KEY
SambaNovasambanovaVERDICTAN_SAMBANOVA_API_KEY
xAI (Grok)xai, grokVERDICTAN_XAI_API_KEY
Vercel AI Gatewayvercel, vercel-aiVERDICTAN_VERCEL_API_KEY

Wire formats​

The gateway translates between these wire formats. It infers the format from a known alias. Set format when the provider page tells you to. A translator does not guarantee that every provider field works.

FormatProviders
openaiOpenAI, Azure, Groq, Mistral, Together, Fireworks, and most OpenAI-compatible endpoints
anthropicAnthropic, Claude
cohereCohere
huggingfaceHuggingFace Inference Endpoints
replicateReplicate
watsonxIBM watsonx
google-geminiGoogle Gemini (AI Studio and Vertex)

Examples​

Credential resolution​

A target that needs a key is active only when its secret_key_ref resolves. AWS Bedrock, Google Vertex AI, and OAuth2 targets use their own authentication methods.

policy-config.yaml
pack:
  name: config-providers-credentials
  version: 1.0.0
  enabled: true
providers:
  routing:
    strategy: ordered
  targets:
  - id: openai
    provider: openai
    model: your-openai-model
    secret_key_ref:
      env: VERDICTAN_OPENAI_API_KEY
  - id: azure
    provider: azure
    provider_type: azure-openai
    format: openai
    model: your-azure-openai-model
    base_url: https://replace-with-resource-name.openai.azure.com
    secret_key_ref:
      env: VERDICTAN_AZURE_OPENAI_API_KEY
    azure_api_version: your-supported-api-version
    azure_deployment: your-azure-deployment
policies:
  chain:
  - audit-logger
policy:
  audit-logger: {}

With ordered routing, a failure on openai sends the request to azure. To keep requests on the first target, set providers.routing.allow_fallbacks: false or pin the request to one provider.

Data policies​

Record the data terms from your provider contract on each target. data-routing-policy uses these declarations to filter targets. Keep the contract evidence in its own workflow.

policy-config.yaml
pack:
  name: config-providers-providers-4
  version: 1.0.0
  enabled: true
providers:
  targets:
  - id: openai-zdr
    provider: openai
    model: your-openai-model
    secret_key_ref:
      env: VERDICTAN_OPENAI_API_KEY
    data_policy:
      zero_data_retention: true
      training_opt_out: true
      retention_days: 0
  - id: azure-standard
    provider: azure
    provider_type: azure-openai
    format: openai
    model: your-azure-openai-model
    base_url: https://replace-with-resource-name.openai.azure.com
    secret_key_ref:
      env: VERDICTAN_AZURE_OPENAI_API_KEY
    azure_api_version: your-supported-api-version
    azure_deployment: your-azure-deployment
    data_policy:
      zero_data_retention: false
      training_opt_out: true
      retention_days: 30
policies:
  chain:
  - audit-logger
policy:
  audit-logger: {}

Data Routing Policy lists every data_policy field.

Token pricing​

A pricing block gives the gateway prices for spend tracking and routing scores. Take input_price_per_million, cached_input_price_per_million, and output_price_per_million from your provider contract. Do not copy prices from an example.

Nested models​

One target can serve more than one model. Each model can have its own aliases, prices, and escalation route:

policy-config.yaml
pack:
  name: config-providers-providers-6
  version: 1.0.0
  enabled: true
providers:
  targets:
  - id: openai-prod
    provider: openai
    base_url: https://api.openai.com
    secret_key_ref:
      env: VERDICTAN_OPENAI_API_KEY
    models:
      - model_id: your-openai-model
        aliases: [fast]
      - model_id: your-second-openai-model
        aliases: [reasoning]
        escalation_routing:
          team_id: 11111111-1111-4111-8111-111111111111
policies:
  chain:
  - audit-logger
policy:
  audit-logger: {}

When models is present, the gateway ignores the top-level model of the target.

Escalation route override​

Send the escalations of one target to a team or a user. Set exactly one of team_id or user_id:

policy-config.yaml
pack:
  name: config-providers-providers-7
  version: 1.0.0
  enabled: true
providers:
  targets:
  - id: healthcare-openai
    provider: openai
    model: your-openai-model
    secret_key_ref:
      env: VERDICTAN_OPENAI_API_KEY
    escalation_routing:
      team_id: 11111111-1111-4111-8111-111111111111
policies:
  chain:
  - audit-logger
policy:
  audit-logger: {}

Health probes​

A health probe runs in the background and sets the health status of the target:

policy-config.yaml
pack:
  name: config-providers-providers-8
  version: 1.0.0
  enabled: true
providers:
  targets:
  - id: openai-prod
    provider: openai
    model: your-openai-model
    secret_key_ref:
      env: VERDICTAN_OPENAI_API_KEY
    health_probe:
      endpoint: https://health.example.com/ready
      interval_seconds: 30
      timeout_ms: 2000
policies:
  chain:
  - audit-logger
policy:
  audit-logger: {}

The probe sends a plain GET, without the target credential or custom headers. Point endpoint at a readiness route that you control and that returns 2xx without authentication. The default, {base_url}/health, works only if the upstream has a public health endpoint.

OAuth2 authentication​

Use oauth2 for providers that take an OAuth2 bearer token, such as Databricks, Snowflake Cortex, and custom endpoints:

policy-config.yaml
pack:
  name: config-providers-providers-9
  version: 1.0.0
  enabled: true
providers:
  targets:
  - id: databricks-prod
    provider: databricks
    model: your-databricks-model
    base_url: https://my-workspace.databricks.net
    oauth2:
      grant_type: client_credentials
      token_endpoint: https://accounts.example.com/oauth2/token
      client_id: verdictan-gateway
      client_secret_env: VERDICTAN_DATABRICKS_CLIENT_SECRET
      scopes: [all-apis]
policies:
  chain:
  - audit-logger
policy:
  audit-logger: {}

Cloud provider fields​

Azure deployment names, Bedrock signing, and Vertex project and region fields are in Cloud Provider Configuration. The provider page of each cloud has a complete credential and request example.

Region and quantization filtering​

Inside policy-config.yaml
providers:
  routing:
    strategy: ordered
    require_region: eu
    require_quantizations:
    - fp16
    - int8
  targets:
  - id: openai-primary
    provider: openai
    model: your-openai-model
    region: eu
    quantizations: [fp16, int8]
    secret_key_ref:
      env: VERDICTAN_OPENAI_API_KEY

require_region fails closed. If no target matches the region, the gateway blocks the request. It does not fall back to another region.

Data residency​

A data_residency block lists the regions that a target can serve, taken from its provider contract. These values are your declarations. They do not move an endpoint or prove where it runs.

Inside policy-config.yaml
providers:
  routing:
    strategy: ordered
    require_region: eu-west
  targets:
  - id: openai-eu
    provider: openai
    model: your-openai-model
    secret_key_ref:
      env: VERDICTAN_OPENAI_API_KEY
    data_residency:
      regions: [eu-west, eu-central]
      data_center_locations: [Frankfurt]
      sovereignty_compliant: true
  - id: openai-us
    provider: openai
    model: your-openai-model
    secret_key_ref:
      env: VERDICTAN_OPENAI_API_KEY
    data_residency:
      regions: [us-east]

Here, require_region: eu-west keeps openai-eu and excludes openai-us.

FieldTypeDescription
regionsarray of stringsRegions that this endpoint can serve. The gateway compares them with require_region and ignores case. At least one is required.
data_center_locationsarray of stringsDeclared data center locations. Kept for audit review. Not used for routing.
sovereignty_compliantbooleanDeclared sovereignty attestation. Kept for audit review. Not used for routing.

When a target has data_residency, the gateway uses its regions and ignores the region field of the same target.

Data residency and zero-retention routing are separate checks. One selects targets by region and the other by declared retention terms. A target must pass every active check.

Target filtering​

Inside policy-config.yaml
providers:
  routing:
    strategy: round_robin
    only:
    - openai-primary
  targets:
  - id: openai-primary
    provider: openai
    model: your-openai-model
    secret_key_ref:
      env: VERDICTAN_OPENAI_API_KEY

Context window compression​

Set max_context_tokens on a target to turn on middle-out compression for requests that are longer than that limit.

Zero-completion handling​

For buffered POST /v1/chat/completions responses, the gateway checks for an empty completion by default. When it finds one, it tries the next eligible target. Turn this off if a second provider call is not approved:

Inside providers
zero_completion_insurance:
  enabled: false

Model groups​

A model group gives targets a stable name for clients. Add a group next to the targets of the starter example:

Inside providers
model_groups:
  - name: fast-models
    aliases: [fast, quick]
    targets: [primary]

A request with "model": "fast" now goes to primary. Each target ID in a group must exist in providers.targets.

For a fallback group, define a second group and set fallback_group to its name. Its targets follow the targets of the first group, and the usual provider attempt rules apply. See Provider Control Features for the fields.

Provider pipelines​

A pipeline puts several targets behind one virtual model name. mode: sequence passes the output of one step to the next. mode: fan_out asks several models the same question and combines the answers.

policy-config.yaml
pack:
  name: config-providers-providers-25
  version: 1.0.0
  enabled: true
providers:
  pipelines:
  - name: draft-and-review
    aliases: [reviewed-answer]
    mode: sequence
    steps:
    - name: draft
      target: writer
    - name: review
      target: reviewer
      instruction: Review and improve the draft.
      input_mode: replace
      inject_as: user
  targets:
  - id: writer
    provider: openai
    model: your-openai-model
    secret_key_ref:
      env: VERDICTAN_OPENAI_API_KEY
  - id: reviewer
    provider: azure
    provider_type: azure-openai
    format: openai
    model: your-azure-openai-model
    base_url: https://replace-with-resource-name.openai.azure.com
    secret_key_ref:
      env: VERDICTAN_AZURE_OPENAI_API_KEY
    azure_api_version: your-supported-api-version
    azure_deployment: your-azure-deployment
policies:
  chain:
  - audit-logger
policy:
  audit-logger: {}

How pipelines work:

  • The request model selects a pipeline, as it does for model_groups.
  • Each step must use a target with a model. For two models from one provider, declare two targets.
  • In a sequence, input_mode: append (the default) adds the previous output to the original request. input_mode: replace uses the previous output instead of the original request.
  • inject_as sets the role of the injected output: user (default), assistant, or system.
  • fan_out runs each step on the original request. aggregation: concat (default) joins the outputs in order. aggregation: first_success returns the first output without an error.
  • Pipelines support POST /v1/chat/completions and POST /v1/responses, not embeddings or moderations.
  • The gateway rejects a pin of a pipeline model with X-Verdictan-Provider or X-Verdictan-Model.

Circuit breaker​

Inside providers
circuit_breaker:
  enabled: true
  consecutive_failure_threshold: 5
  cooldown_seconds: 30
  half_open_successes: 1

Retry policy​

Inside providers
retry_policy:
  max_retries: 3
  per_trigger:
    rate_limit: 5
  backoff:
    strategy: "exponential"        # exponential | fixed
    base_ms: 500

These retries use the same target before the gateway tries the next one. per_trigger sets limits for rate_limit and server_error. Transport and body read failures use max_retries.

Logging controls​

Inside providers
logging:
  redact_message_bodies: true

With redact_message_bodies: true, the gateway writes [REDACTED] instead of message content in traces and event payloads. redact_api_keys is true by default.

Complete provider example​

policy-config.yaml
pack:
  name: openai-and-azure-routing
  version: 1.0.0
  enabled: true
providers:
  targets:
  - id: openai-primary
    provider: openai
    model: your-openai-model
    secret_key_ref:
      env: VERDICTAN_OPENAI_API_KEY
  - id: azure-openai-fallback
    provider: azure-openai
    model: your-azure-openai-model
    base_url: https://replace-with-resource-name.openai.azure.com
    secret_key_ref:
      env: VERDICTAN_AZURE_OPENAI_API_KEY
    azure_deployment: your-azure-deployment
    azure_api_version: your-supported-api-version
  routing:
    strategy: ordered
  circuit_breaker:
    enabled: true
    consecutive_failure_threshold: 5
    cooldown_seconds: 30
  retry_policy:
    max_retries: 3
    backoff:
      strategy: exponential
      base_ms: 500
policies:
  chain:
  - prompt-injection
  - pii-detector
  - quality-scorer
policy:
  prompt-injection:
    response:
      action: block
  pii-detector:
    action: redact
  quality-scorer:
    thresholds:
      min_aggregate: 0.8

retry_policy retries the same target. When its retries are used up, the gateway tries the next eligible target.

Limits and failure behavior​

Provider attempts and fallback​

The gateway orders the eligible targets and retries each one according to retry_policy before it moves to the next. The first successful response ends the loop.

A transport error always moves the request to the next target. An HTTP response moves it only when its class is in the trigger list:

  • rate_limit: HTTP 429
  • server_error: HTTP 408, 500, 502, 503, and 504, and transport timeouts

An HTTP 401, or a 403 that is not a content filter block, also moves the request to the next target. This happens one time for each target.

The gateway returns every other upstream response to the client, including content filter and context window errors. These rules apply to every provider.

To use only the first eligible target, set providers.routing.allow_fallbacks: false or pin the request to a provider. Test each request family, because a streamed response cannot move to another target after output starts.

Load errors​

  • providers.targets must have at least one target.
  • A target that needs a credential and has required: true stops the gateway at startup if its secret_key_ref does not resolve. With required: false, the gateway leaves the target out of routing.
  • routing.only and routing.ignore together are a load error.
  • An ID in routing.order that is not in targets is a load error.
  • Quantization labels must be fp32, fp16, bf16, int8, or int4. Lint rejects the runtime-only fp8. The gateway rejects the schema-only awq, gptq, and gguf at load time.
  • The gateway rejects the removed fields providers.routing.upstream_fallback_policy and providers.ab_test.

Region blocks​

  • If require_region excludes every target and at least one target has data_residency, the gateway returns HTTP 451 with error code no_compliant_provider.
  • If no target has data_residency, the gateway compares require_region with each region. No match returns HTTP 403 with error code no_eligible_provider.
  • verdictan policy lint reports a require_region value that no target supports.

Next steps​