Rate Limit Configuration

Rate limits protect the gateway and your provider account from bursts of traffic. You can limit requests for the whole gateway, for each client IP, and for each user, limit the tokens in responses, and limit the size of requests and responses.

The counters work in different places:

Limit	Counts	Shared between gateway instances
global_rate_limit	Requests to the public proxy handlers	Yes, with distributed_rate_limit
ip_rate_limit	Requests from each client IP	No
user_rate_limit	Requests from each user header value	No
token_rate_limit	Tokens in buffered Chat Completions responses	No
size_limits	Bytes in a request or buffered response	Not a counter

For a request limit on one consumer group, see Routes and Consumer Groups.

Try a request limit

Add this block to the configuration of a test agent. It allows two requests in each fixed one-minute window. Do not use this limit on a shared production agent.

Inside policy-config.yaml

global_rate_limit:
  max_requests: 2
  window_seconds: 60

Apply it with the config-first workflow. Send the quickstart request three times in one minute. The third request returns HTTP 429 with a Retry-After header. If a new window starts between two calls, try again.

Global rate limit

One counter for all public proxy handlers of the gateway. Health, configuration, and admin endpoints do not count.

Field	Type	Description
max_requests	integer	Requests allowed in each window. Required.
window_seconds	integer	Length of the fixed window. Required.

Global rate limit behavior

• The window is fixed and aligned to the epoch. The count starts at zero in each new window.
• A request over the limit gets HTTP 429 Too Many Requests with a Retry-After header.

Rate limit for each IP address

A separate counter for each client IP address.

Field	Type	Default	Description
max_requests	integer		Requests allowed for each IP in each window. Required.
window_seconds	integer		Length of the window. Required.
trusted_proxy_cidrs	array of strings	[]	IPv4 or IPv6 networks of proxies that can add X-Forwarded-For

Gateway behind a reverse proxy

Behind nginx or a load balancer, every request comes from the proxy address. List each trusted proxy network so that the gateway can find the real client. The gateway starts at the socket peer and reads X-Forwarded-For from right to left while each hop is in a listed network.

Inside policy-config.yaml

# Gateway behind one nginx reverse proxy
ip_rate_limit:
  max_requests: 50
  window_seconds: 60
  trusted_proxy_cidrs:
    - 192.0.2.10/32  # Replace with the exact trusted proxy address

Per-user rate limit

A separate counter for each user, identified by a request header.

Field	Type	Default	Description
max_requests	integer		Requests allowed for each user in each window. Required.
window_seconds	integer		Length of the window. Required.
header_names	array of strings	["x-user-id"]	Headers that carry the user identity. The first non-empty value counts.

Per-user rate limit behavior

• A request without a matching header is not counted. The gateway does not put it in an unknown group.
• The caller controls these headers. If the limit must apply to every request, have a trusted proxy at the gateway boundary remove the caller's value and set its own.

Token rate limit

A moving-window limit on the tokens in buffered Chat Completions responses. It is not a prepaid reservation or a cost budget.

Field	Type	Default	Description
max_tokens	integer		Tokens allowed in each window. Required.
window_seconds	integer		Length of the moving window. Required.
scope	string	global	global, per_key, or per_ip

Token limit scopes

Define token_rate_limit only once in a file, with one scope:

Inside policy-config.yaml

token_rate_limit:
  max_tokens: 100000
  window_seconds: 3600
  scope: global

Scope	One budget for
global	All counted traffic on this gateway
per_key	All requests with the same API key
per_ip	All requests from the same socket peer address

verdictan policy lint reports an error for scope: per_key (no api-key-policy or auth-check in chain) that you cannot clear, because neither name is a policy control. verdictan config validate and the gateway accept the scope. Make sure that the Authorization header reaches the gateway, because the key comes from it.

Token rate limit behavior

• The limiter uses six moving sub-windows for buffered POST /v1/chat/completions traffic.
• After a successful response, it records usage.total_tokens from the JSON body.
• When no budget is left, the next Chat Completions request gets HTTP 429 with error code token_rate_limit_exceeded.
• Streaming responses and other request families are not checked or counted.
• scope: per_ip uses the socket peer address, not X-Forwarded-For or trusted_proxy_cidrs. Behind a reverse proxy, all callers share one budget.

Size limits

Field	Type	Default	Description
max_body_bytes	integer	No limit	Largest request body
max_header_bytes	integer	No limit	Largest total header size
max_url_bytes	integer	No limit	Longest URL
max_response_bytes	integer	16777216	Largest buffered response body. The gateway lowers a larger value to 16 MiB.

Size limit behavior

• The request checks run on POST /v1/chat/completions only, in this order: body, headers, URL. A request over a limit gets HTTP 413 Payload Too Large with the header X-Rejected-By: size_limit.
• Do not rely on the request size limits to protect other routes.
• max_response_bytes applies to every buffered Chat Completions and Responses path. An oversized upstream response returns HTTP 502 Bad Gateway with error code response_size_exceeded.
• A streaming response has already sent its headers, so the gateway sends a final SSE error event and stops the stream.

Distributed rate limit

distributed_rate_limit stores the global request counter in Redis or Valkey, so that all gateway instances share it. Configure global_rate_limit too, because it sets the limit and window of the shared counter.

Field	Type	Default	Description
backend	string		redis or valkey. Both use the Redis protocol. Required.
url_env	string	VERDICTAN_LLM_CACHE_REDIS_URL	Environment variable that holds the connection URL

The gateway also accepts these two fields in a distributed object inside global_rate_limit, ip_rate_limit, or token_rate_limit. Keep them in distributed_rate_limit so that the setting is in one place.

Distributed rate limit behavior

• The CLI build must include distributed-state support. Otherwise, the gateway does not start.
• Connected and multi-node profiles that need shared state do not start when the URL is missing or the connection fails.
• If the backend goes away, dependent requests and /readyz return HTTP 503 with reason dependency.distributed_state_unavailable.
• A process-local fallback exists only for a one-node, self-hosted development profile. It is not a recovery path for a distributed deployment.

Complete rate limit example

This example needs Redis or Valkey and a CLI build with distributed-state support. Replace the proxy address and the model.

policy-config.yaml

pack:
  name: "rate-limited-gateway"
  version: "1.0.0"
  enabled: true

# Global ceiling
global_rate_limit:
  max_requests: 5000
  window_seconds: 60

# Per-IP protection
ip_rate_limit:
  max_requests: 100
  window_seconds: 60
  trusted_proxy_cidrs:
    - 192.0.2.10/32  # Replace with the exact trusted proxy address

# Per-user fairness
user_rate_limit:
  max_requests: 30
  window_seconds: 60
  header_names: ["x-user-id", "x-consumer-id"]

# Token budget
token_rate_limit:
  max_tokens: 1000000
  window_seconds: 3600
  scope: "global"

# Request size protection
size_limits:
  max_body_bytes: 2097152      # 2 MB
  max_response_bytes: 15728640 # 15 MB

# Shared global request counter
distributed_rate_limit:
  backend: "valkey"
  url_env: "VERDICTAN_LLM_CACHE_REDIS_URL"

providers:
  targets:
    - id: "openai-prod"
      provider: "openai"
      model: "your-openai-model"
      secret_key_ref:
        env: "VERDICTAN_OPENAI_API_KEY"

policies:
  chain:
    - "audit-logger"

Before you start the gateway, set VERDICTAN_LLM_CACHE_REDIS_URL to the Redis or Valkey connection string.

Next steps

• Routes and Consumer Groups: request limits for one consumer group.
• Security and Network Configuration: IP allowlists, CORS, and TLS.
• Provider Configuration: retries and circuit breakers on the provider side.