Runtime Configuration

These optional sections of policy-config.yaml control what the gateway records in History, how it caches responses, what GET /v1/models returns, and a few other runtime behaviors. Change one at a time, so that you can see the effect of each change.

For endpoint formats and streaming support, see Runtime Request Families.

Start with request history

Add this section to a working configuration from the config-first workflow. It records request metadata without prompt or response bodies:

Inside policy-config.yaml

history:
  capture:
    enabled: true
    mode: metadata_only
    include_blocked: true

Send one allowed request and one blocked request through your test agent. Both show in History, with metadata and no message bodies.

The gateway does not wait for History writes, so a write can fail while the request succeeds. If an entry is missing, examine the gateway logs.

History

history.capture decides what the gateway writes to History.

Field	Type	Default	Description
capture.enabled	boolean	false	Turns History writes on or off
capture.mode	string	metadata_only	raw, redacted, metadata_only, or disabled
capture.include_blocked	boolean	false	Also records blocked requests

Without a history section, a connected gateway records metadata only. If you add a history.capture block without enabled, capture is off.

The schema also lists capture.fail_open and capture.retention_days. The gateway does not read them.

Metadata-only history

Inside policy-config.yaml

history:
  capture:
    enabled: true
    mode: metadata_only
    include_blocked: true

Other logging and capture settings work independently of this mode.

History off

Inside policy-config.yaml

history:
  capture:
    enabled: false

History write failures

A failed History write adds a log entry. It does not block or fail the request.

Response cache

cache turns on an exact or semantic response cache. Environment variables select where the cache is stored and for how long.

Field	Type	Default	Description
mode	string	exact	exact or semantic
similarity_threshold	number	0.85	Minimum cosine similarity for a semantic hit
embedding_provider	string		ID of the provider target that calculates embeddings. Required for semantic mode.
default_on	boolean	true	When false, the cache applies only to requests with the header x-verdictan-cache: on.

Exact cache

The exact cache hashes the whole request body. An identical request gets the stored response.

Inside policy-config.yaml

cache:
  mode: exact
  default_on: true

Semantic cache

The semantic cache embeds the request and returns a stored response for an earlier request that is similar enough.

Inside policy-config.yaml

cache:
  mode: semantic
  similarity_threshold: 0.85
  embedding_provider: replace-with-embedding-target-id
  default_on: true

embedding_provider must name a target that serves an embedding model. Naming a chat target does not make it produce embeddings.

Cache backend environment

Set VERDICTAN_LLM_CACHE_BACKEND on the gateway process. VERDICTAN_LLM_CACHE_TTL_SECS sets the time to live for every backend. The default is 14 days.

Backend	Required variables	Optional variables
memory
filesystem		VERDICTAN_LLM_CACHE_DIR, VERDICTAN_LLM_CACHE_MAX_BYTES
redis or valkey	VERDICTAN_LLM_CACHE_REDIS_URL	VERDICTAN_LLM_CACHE_REDIS_KEY_PREFIX
s3	VERDICTAN_LLM_CACHE_S3_BUCKET, VERDICTAN_LLM_CACHE_S3_REGION, VERDICTAN_LLM_CACHE_S3_ACCESS_KEY_ID, VERDICTAN_LLM_CACHE_S3_SECRET_ACCESS_KEY	VERDICTAN_LLM_CACHE_S3_ENDPOINT, VERDICTAN_LLM_CACHE_S3_PREFIX, VERDICTAN_LLM_CACHE_S3_FORCE_PATH_STYLE
gcs	VERDICTAN_LLM_CACHE_GCS_BUCKET, VERDICTAN_LLM_CACHE_GCS_ACCESS_KEY_ID, VERDICTAN_LLM_CACHE_GCS_SECRET_ACCESS_KEY	VERDICTAN_LLM_CACHE_GCS_REGION, VERDICTAN_LLM_CACHE_GCS_ENDPOINT, VERDICTAN_LLM_CACHE_GCS_PREFIX, VERDICTAN_LLM_CACHE_GCS_FORCE_PATH_STYLE
qdrant	VERDICTAN_LLM_CACHE_QDRANT_URL	VERDICTAN_LLM_CACHE_QDRANT_COLLECTION, VERDICTAN_LLM_CACHE_QDRANT_API_KEY

If the backend is unset or auto, the gateway uses memory when VERDICTAN_ENV is development or test. In every other environment, auto stops the gateway at startup. Set a backend name in production.

Redis cache backend

export VERDICTAN_LLM_CACHE_BACKEND=redis
export VERDICTAN_LLM_CACHE_TTL_SECS=86400
export VERDICTAN_LLM_CACHE_REDIS_URL=redis://localhost:6379/1

verdictan cache shows and clears cache entries.

Cache limits

• Only non-streaming Chat Completions and Responses requests use the cache. Streaming and other request families do not.
• Semantic lookup and storage need embedding_provider to resolve and the embedding call to succeed. Otherwise, the gateway skips the semantic path, and the exact path still works.
• Semantic mode works with memory, filesystem, redis, valkey, and qdrant. s3 and gcs support the exact cache only.
• The GCS backend uses interoperability access keys.
• Redis, Valkey, S3, GCS, and Qdrant need a CLI build with the distributed feature. Without it, or without a required variable, the cache fails at startup.

Models endpoint

models controls the public GET /v1/models endpoint.

Field	Type	Default	Description
disabled	boolean	false	When true, GET /v1/models returns HTTP 404.
include_disabled	boolean	false	Also lists disabled nested models

Moderation endpoint

moderation configures the backend of POST /v1/moderations. It does not moderate chat traffic. For a check that blocks requests in the chain, use External Moderation.

Field	Type	Default	Description
provider	string		openai or azure. Required.
secret_key_ref	object		Environment credential reference with env. Required for OpenAI and Azure.
endpoint	string	Provider default	Moderation URL. Required for Azure. Optional for OpenAI.
categories	array of strings	All	Moderates only these categories
threshold	number	0.5	Confidence at which a result is flagged

Azure moderation

Inside policy-config.yaml

moderation:
  provider: azure
  secret_key_ref:
    env: VERDICTAN_AZURE_CONTENT_SAFETY_KEY
  endpoint: https://my-resource.cognitiveservices.azure.com
  categories:
    - violence
    - hate
  threshold: 0.5

Moderation fail-open path

warning

Do not use POST /v1/moderations as a safety boundary that fails closed. If the credential or Azure endpoint is missing, or the upstream call fails or returns an invalid response, the endpoint returns an unflagged result with HTTP 200. The moderation section has no fail_closed field.

Auto provider

auto adds a virtual model that picks a configured target by cost and latency.

Field	Type	Default	Description
enabled	boolean	true at runtime	Turns the virtual provider on or off
name	string	auto	Model name that clients send to use it
description	string		Text for the virtual model in GET /v1/models
routing.cost_weight	number	0.5	Weight of price in the score
routing.latency_weight	number	0.5	Weight of latency in the score
routing.max_price_per_1m_tokens	number		Targets with a higher configured price are not eligible

The gateway turns the auto provider on when the auto section or its enabled field is missing, but the schema shows false as the default. Set enabled explicitly so that both agree. Any key other than the ones above is a load error.

Auto provider limits

• If max_price_per_1m_tokens excludes every target, routing returns no_eligible_provider.
• The auto provider pins the request to one target and does not try another after a provider failure.
• When auto.enabled is true, auto.name must not equal the name of a model group or pipeline. A conflict is a load error.

Default agent

agents.runtime.default_agent_id is the agent ID that the gateway adds to events and traces when a caller sends no agent context.

Inside policy-config.yaml

agents:
  runtime:
    default_agent_id: support-agent

Provider credentials for a connected gateway

A connected gateway uses its runtime token to talk to Verdictan, and secret_key_ref to find provider keys. Use env for a key in the local environment and store for a key saved in Verdictan:

policy-config.yaml

pack:
  name: config-runtime-providers-14
  version: 1.0.0
  enabled: true
providers:
  targets:
    - id: openai-chatgpt-shared
      provider: openai
      model: your-openai-model
      base_url: https://api.openai.com
      secret_key_ref:
        store: OPENAI_API_KEY
policies:
  chain:
    - audit-logger
policy:
  audit-logger: {}

Complete runtime example

Start from the provider example and add the History block. Add an exact cache only after the first request succeeds, and use verdictan cache to watch the cache.

Next steps

• Provider Configuration: the targets that the cache and the auto provider use.
• Environment Variable Use: set cache and credential variables for the gateway.
• Test Configuration: write tests for your configuration.