Skip to main content

Runtime configuration

These optional sections of policy-config.yaml control what the gateway records in History, how it caches responses, what GET /v1/models returns, and a few other runtime behaviors. Change one at a time, so that you can see the effect of each change.

For endpoint formats and streaming support, see Runtime request families.

Start with request history​

Add this section to a working configuration from the config-first workflow. It records request metadata without prompt or response bodies:

Inside policy-config.yaml
history:
  capture:
    enabled: true
    mode: metadata_only
    include_blocked: true

Send one allowed request and one blocked request through your test agent. Both show in History, with metadata and no message bodies.

By default, the gateway does not wait for History writes, so a write can fail while the request succeeds. If an entry is missing, examine the gateway logs. To make the request wait for its History write, set capture.fail_open: false. See History write failures.

History​

history.capture decides what the gateway writes to History.

FieldTypeDefaultDescription
capture.enabledbooleanfalseTurns History writes on or off
capture.modestringmetadata_onlyraw, redacted, metadata_only, or disabled
capture.include_blockedbooleanfalseAlso records blocked requests
capture.retention_daysintegernoneDays that History keeps each entry, from 1 to 2555
capture.fail_openbooleantruefalse makes a failed History write fail the request

Without a history section, a connected gateway records metadata only. If you add a history.capture block without enabled, capture is off.

The gateway sends retention_days with each History write only when you set it. Without the field, History keeps its existing retention. The schema shows a default of 30, but the gateway does not send that default. A retention_days value outside 1 to 2555, or a fail_open value that is not a boolean, stops the configuration from loading.

Metadata-only history​

Inside policy-config.yaml
history:
  capture:
    enabled: true
    mode: metadata_only
    include_blocked: true

Other logging and capture settings work independently of this mode.

History off​

Inside policy-config.yaml
history:
  capture:
    enabled: false

History write failures​

With the default fail_open: true, a failed History write adds a log entry. It does not block or fail the request.

With fail_open: false, the request waits for its History write. These events are failures:

  • The History session or entry write returns a non-2xx status or a network error.
  • The History write queue is full (128 writes in progress).
  • A connected gateway has no organization ID for the write.

The result depends on how the gateway returns the response:

ResponseWhen the History write fails
Buffered POST /v1/chat/completions or POST /v1/responsesThe gateway returns HTTP 503 with error code history_writeback_failed. The provider response is not sent.
Streamed chat completionThe gateway holds data: [DONE] until the write finishes. If the write fails, it sends an error frame with code history_writeback_failed, then data: [DONE].
Streamed Responses requestThe gateway holds response.completed, response.incomplete, or response.failed until the write finishes. If the write fails, it sends an error event with code history_writeback_failed instead.

A streamed client already has the content frames when the write fails. Treat a stream that ends with history_writeback_failed as a failed request.

A blocked or escalated response contains no provider content, so it does not wait for History. POST /v1/messages and WebSocket sessions do not write History entries.

Inside policy-config.yaml
history:
  capture:
    enabled: true
    mode: metadata_only
    retention_days: 90
    fail_open: false

Response cache​

cache turns on an exact or semantic response cache. Environment variables select where the cache is stored and for how long.

FieldTypeDefaultDescription
modestringexactexact or semantic
similarity_thresholdnumber0.85Minimum cosine similarity for a semantic hit
embedding_providerstringID of the provider target that calculates embeddings. Required for semantic mode.
default_onbooleantrueWhen false, the cache applies only to requests with the header x-verdictan-cache: on.

Exact cache​

The exact cache hashes the whole request body. An identical request gets the stored response.

Inside policy-config.yaml
cache:
  mode: exact
  default_on: true

Semantic cache​

The semantic cache embeds the request and returns a stored response for an earlier request that is similar enough.

Inside policy-config.yaml
cache:
  mode: semantic
  similarity_threshold: 0.85
  embedding_provider: replace-with-embedding-target-id
  default_on: true

embedding_provider must name a target that serves an embedding model. Naming a chat target does not make it produce embeddings.

Cache backend environment​

Set VERDICTAN_LLM_CACHE_BACKEND on the gateway process. VERDICTAN_LLM_CACHE_TTL_SECS sets the time to live for every backend. The default is 14 days.

BackendRequired variablesOptional variables
memory
filesystemVERDICTAN_LLM_CACHE_DIR, VERDICTAN_LLM_CACHE_MAX_BYTES
redis or valkeyVERDICTAN_LLM_CACHE_REDIS_URLVERDICTAN_LLM_CACHE_REDIS_KEY_PREFIX
s3VERDICTAN_LLM_CACHE_S3_BUCKET, VERDICTAN_LLM_CACHE_S3_REGION, VERDICTAN_LLM_CACHE_S3_ACCESS_KEY_ID, VERDICTAN_LLM_CACHE_S3_SECRET_ACCESS_KEYVERDICTAN_LLM_CACHE_S3_ENDPOINT, VERDICTAN_LLM_CACHE_S3_PREFIX, VERDICTAN_LLM_CACHE_S3_FORCE_PATH_STYLE
gcsVERDICTAN_LLM_CACHE_GCS_BUCKET, VERDICTAN_LLM_CACHE_GCS_ACCESS_KEY_ID, VERDICTAN_LLM_CACHE_GCS_SECRET_ACCESS_KEYVERDICTAN_LLM_CACHE_GCS_REGION, VERDICTAN_LLM_CACHE_GCS_ENDPOINT, VERDICTAN_LLM_CACHE_GCS_PREFIX, VERDICTAN_LLM_CACHE_GCS_FORCE_PATH_STYLE
qdrantVERDICTAN_LLM_CACHE_QDRANT_URLVERDICTAN_LLM_CACHE_QDRANT_COLLECTION, VERDICTAN_LLM_CACHE_QDRANT_API_KEY

If the backend is unset or auto, the gateway uses memory when VERDICTAN_ENV is development or test. In every other environment, auto stops the gateway at startup. Set a backend name in production.

Redis cache backend​

export VERDICTAN_LLM_CACHE_BACKEND=redis
export VERDICTAN_LLM_CACHE_TTL_SECS=86400
export VERDICTAN_LLM_CACHE_REDIS_URL=redis://localhost:6379/1

verdictan cache shows and clears cache entries.

Cache limits​

  • Only non-streaming Chat Completions and Responses requests use the cache. Streaming and other request families do not.
  • Semantic lookup and storage need embedding_provider to resolve and the embedding call to succeed. Otherwise, the gateway skips the semantic path, and the exact path still works.
  • Semantic mode works with memory, filesystem, redis, valkey, and qdrant. s3 and gcs support the exact cache only.
  • The GCS backend uses interoperability access keys.
  • Redis, Valkey, S3, GCS, and Qdrant need a CLI build with the distributed feature. Without it, or without a required variable, the cache fails at startup.

Models endpoint​

models controls the public GET /v1/models endpoint.

FieldTypeDefaultDescription
disabledbooleanfalseWhen true, GET /v1/models returns HTTP 404.
include_disabledbooleanfalseAlso lists disabled nested models

Moderation endpoint​

moderation configures the backend of POST /v1/moderations. It does not moderate chat traffic. For a check that blocks requests in the chain, use External moderation.

FieldTypeDefaultDescription
providerstringopenai or azure. Required.
secret_key_refobjectEnvironment credential reference with env. Required for OpenAI and Azure.
endpointstringProvider defaultModeration URL. Required for Azure. Optional for OpenAI.
categoriesarray of stringsAllModerates only these categories
thresholdnumber0.5Confidence at which a result is flagged

Azure moderation​

Inside policy-config.yaml
moderation:
  provider: azure
  secret_key_ref:
    env: VERDICTAN_AZURE_CONTENT_SAFETY_KEY
  endpoint: https://my-resource.cognitiveservices.azure.com
  categories:
    - violence
    - hate
  threshold: 0.5

Moderation fail-open path​

warning

Do not use POST /v1/moderations as a safety boundary that fails closed. If the credential or Azure endpoint is missing, or the upstream call fails or returns an invalid response, the endpoint returns an unflagged result with HTTP 200. The moderation section has no fail_closed field.

Auto provider​

auto adds a virtual model that picks a configured target by cost and latency.

FieldTypeDefaultDescription
enabledbooleantrue at runtimeTurns the virtual provider on or off
namestringautoModel name that clients send to use it
descriptionstringText for the virtual model in GET /v1/models
routing.cost_weightnumber0.5Weight of price in the score
routing.latency_weightnumber0.5Weight of latency in the score
routing.max_price_per_1m_tokensnumberTargets with a higher configured price are not eligible

The gateway turns the auto provider on when the auto section or its enabled field is missing, but the schema shows false as the default. Set enabled explicitly so that both agree. Any key other than the ones above is a load error.

Auto provider limits​

  • If max_price_per_1m_tokens excludes every target, routing returns no_eligible_provider.
  • The auto provider pins the request to one target and does not try another after a provider failure.
  • When auto.enabled is true, auto.name must not equal the name of a model group or pipeline. A conflict is a load error.

Default agent​

agents.runtime.default_agent_id is the agent ID that the gateway adds to events and traces when a caller sends no agent context.

Inside policy-config.yaml
agents:
  runtime:
    default_agent_id: support-agent

Provider credentials for a connected gateway​

A connected gateway uses its gateway token to talk to Verdictan, and secret_key_ref to find provider keys. Use env for a key in the local environment and store for a key saved in Verdictan:

policy-config.yaml
pack:
  name: config-runtime-providers-14
  version: 1.0.0
  enabled: true
providers:
  targets:
    - id: openai-chatgpt-shared
      provider: openai
      model: your-openai-model
      base_url: https://api.openai.com
      secret_key_ref:
        store: OPENAI_API_KEY
policies:
  chain:
    - audit-logger
policy:
  audit-logger: {}

Complete runtime example​

Start from the provider example and add the History block. Add an exact cache only after the first request succeeds, and use verdictan cache to watch the cache.

Next steps​