Streaming and SSE

Streaming lets an application show an answer when it receives each piece. Enable it with "stream": true. Verdictan uses Server-Sent Events (SSE) to deliver those pieces.

Some policies must examine the complete response before delivery. In that case, Verdictan waits for the complete response. It applies the policies, then sends the result. For a blocking output policy, the gateway examines the text while it arrives and sends the text that cannot start a blocked term.

Send a streaming request

Start with a completed quickstart request. Use the same central agent URL, model, and application token. Select a model that supports streaming.

Set VERDICTAN_BASE_URL to the central URL without /v1. For a direct local test, use http://127.0.0.1:41002.

export VERDICTAN_BASE_URL="https://<your-published-hostname>"
export VERDICTAN_CLIENT_TOKEN="replace-with-client-token"

curl --fail-with-body --no-buffer "$VERDICTAN_BASE_URL/v1/chat/completions" \
  -H "Authorization: Bearer ${VERDICTAN_CLIENT_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "replace-with-streaming-model",
    "stream": true,
    "messages": [{"role": "user", "content": "Reply with one short sentence."}]
  }'

--no-buffer turns off cURL output buffering. cURL prints each SSE frame when it receives it. A completed Chat Completions stream ends with data: [DONE].

To keep the headers and the body for an investigation, add --fail-with-body and -D stream-headers.txt. After the stream ends, use your account CLI login to read the recent decision events:

verdictan events tail --since 10m --json

Compare the streaming_mode in the event with the mode specified for the policy chain.

Routes that stream

Streaming is available on the public text-family routes:

• POST /v1/chat/completions
• POST /v1/responses
• POST /v1/messages

Each route runs the same access, input, tool, output, and audit stages as a buffered request. Runtime Request Families lists the capability limits of each route.

Streaming modes

The gateway selects one of three modes for each streaming request. The decision event records the mode in streaming_mode.

Mode	Trigger	Client receives
passthrough	No control examines the output	Each provider frame when the gateway receives it
buffered_policy	An output control examines the output	Chat Completions and Responses: each frame when the gateway receives it, except text that can start a blocked term. Messages: the output after the controls examine all of it.
buffered_redaction	An output redaction control is active	One frame with the redacted content

In passthrough mode, the gateway translates frames only for supported streaming pairs.

These controls select buffered_policy: citation-verifier, mnpi-filter, financial-compliance, healthcare-compliance, legal-privilege, upl-filter, bias-monitor, quality-scorer, human-oversight, safety-filter, itar-ear-filter, entity-list-filter, dual-use-filter, response-rewriter, agent-firewall, and student-privacy when action is block.

These controls select buffered_redaction: pii-detector and hipaa-phi-detector when action is not off, dlp-filter, student-privacy, and dual-use-filter when action is redact, and case-privacy always.

In buffered_policy mode on Chat Completions and Responses, agent-firewall keeps all frames until it has examined the tool calls. With gateway 0.1.26 or later, bias-monitor keeps all frames until the stream ends, and human-oversight with action: escalate stops the stream at the first output text. The other controls do not stop the frames. See Output blocks in a stream and Output escalations in a stream.

In buffered_redaction mode on /v1/chat/completions, the single frame is a Chat Completions object with the model name verdictan-buffered-stream. On /v1/responses, it is one response object with the complete output_text. The stream then ends with data: [DONE]. A client that measures first-token latency sees the complete output at one time.

The gateway reads provider streams that use LF, CRLF, or CR line endings. The line endings do not change the policy, redaction, or usage results.

Output blocks in a stream

With gateway 0.1.26 or later, these controls examine streamed Chat Completions, Responses, and Messages output while it arrives:

• safety-filter. With action: escalate, a match stops the stream as an escalation.
• itar-ear-filter and entity-list-filter
• dual-use-filter, except with action: redact
• student-privacy with action: block
• bias-monitor with action: block. The gateway holds all output until the stream ends, and the error event has bias.detected.

The term match ignores the case of ASCII letters. The gateway does these steps on Chat Completions and Responses:

1. It sends the text to the client when it receives the text.
2. When the end of the text can be the start of a blocked term, it holds back the frames with that text.
3. When the next text shows that no blocked term follows, it sends the held frames. It also sends them when the stream ends.
4. When a blocked term matches, it discards the held frames and sends an error event.

Thus no part of a blocked term gets to the client. Text before the blocked term can already be with the client. On Messages, the gateway keeps the output until the controls have examined all of it.

Route	Error event after a block
/v1/chat/completions	A data: event with an error object and the code content_policy_violation, then data: [DONE]
/v1/responses	A final event: error frame. Its sequence_number is one more than the last event that the client received.
/v1/messages	An Anthropic event: error frame

The decision event records the verdict block and the output result of the control. For student-privacy, the reason code is student_privacy.block.

student-privacy has two limits on a stream:

• A stream that ends with student number or studentid is not blocked. Only that phrase gets to the client, with no identifier after it.
• With two or more spaces between student and id, the text student can get to the client before the block.

With gateway 0.1.26 or later, hipaa-phi-detector with action: block also blocks streamed Chat Completions, Responses, and Messages output that has PHI. This stream is in buffered_redaction mode, so the gateway sends no text before it has examined the complete output. The error event has the reason code of the first PHI, for example phi.mrn_detected. See HIPAA PHI Detector.

Usage after a block

After the error event, the gateway closes the client response. Then it reads the remaining provider stream for a maximum of 30 seconds. It does not send this output to the client.

The gateway uses the usage that the provider reports to complete the usage authorization. Without a usage authorization, the gateway records that usage as spend. If the provider reports no usage in that time, the gateway records an estimate with usage_estimated: true.

Output escalations in a stream

With gateway 0.1.26 or later, safety-filter with action: escalate, human-oversight with action: escalate, and bias-monitor with action: escalate (the default) escalate streamed Chat Completions, Responses, and Messages output. The stream stops with the error event of the route. The error message is Streaming response escalated by policy: <reason code>. The verdictan object has the verdict ESCALATE, the reason code, and an escalation with the ID esc_<request_id> and status queued.

The controls stop the stream at different points:

• safety-filter uses the same holdback as a block. No part of the matched term gets to the client. On Chat Completions and Responses, text before the term can already be with the client.
• human-oversight escalates at the first output text. The client gets no output text.
• bias-monitor holds all streamed output until the stream ends. If the score reaches the threshold, the client gets no output text. Otherwise, the gateway sends all the output when the stream ends.

The decision event records the verdict escalate, the reason code, and the escalation routing of the target. After an escalation, the gateway reads the remaining provider stream for a maximum of 30 seconds and records the usage that the provider reports, as in Usage after a block.

A block or escalation stops the policy chain. The output result of the control that stops the stream has chain_stopped: true, and its not_evaluated list names the later output controls of the chain that did not examine the output.

Redaction results in a stream

The gateway sends the response headers before the first frame. Thus the headers show the admission decision. The x-verdictan-verdict header of a redacted stream can show ALLOW, and the stream does not have x-verdictan-response-redacted.

The decision event records the final result after the stream ends. With gateway 0.1.26 or later, a redacted Chat Completions, Responses, or Messages stream has these values in its decision event:

• The verdict redact and the reason code redact.applied.
• An output result that names the redacting policy, with the reason code of that policy. See Output redaction reason codes.
• The result details "streaming": true and "streaming_mode": "buffered_redaction".

Input controls always run before the provider request. Provider fallback can select a different target before the stream starts. After the gateway has sent frames to the client, it does not start the stream again on a different target.

Minimal provider config

This config streams through one OpenAI target with only the audit control. The mode is passthrough. Replace replace-with-streaming-model with a model from model discovery that supports streaming.

pack:
  name: streaming-sse
  version: 0.1.0
  enabled: true
providers:
  targets:
  - id: openai
    provider: openai
    model: replace-with-streaming-model
    secret_key_ref:
      env: VERDICTAN_OPENAI_API_KEY
policies:
  chain:
  - audit-logger
policy:
  audit-logger: {}

Failure behavior

Symptom	Cause	Action
401 or 403 before the first frame	Authentication, authorization, or a policy rejected the request	Read the error code. Examine the application token, its access, and the policy decision.
One JSON body, no SSE frames	The request has no "stream": true, or the model does not support streaming	Add the field. Select a streaming model from model discovery
Frames, then a terminal policy error	An output control found a blocked term. The client can have the text before the term, but no part of the term.	Read the policy result in the decision event. See Output blocks in a stream and Handle a policy block
A terminal policy error with the verdict ESCALATE	An output control escalated the response	Find the escalation in Inbox. See Output escalations in a stream and Resolve an Escalation
A JSON error body instead of a stream on /v1/messages	The request failed before the stream started	Read error.code in the Anthropic error format
One frame with model verdictan-buffered-stream	The chain contains an output redaction control	This is the correct behavior in buffered_redaction mode. Remove the control only if the policy allows it
x-verdictan-verdict: ALLOW on a redacted stream	The headers show the admission decision	Read the final verdict in the decision event. See Redaction results in a stream
Frames, then a network error	The provider stream ended	The gateway does not start the stream again on a different provider
No matching decision event	The event sink, organization, region, or time filter does not match	Examine verdictan events tail filters before you conclude that the request bypassed the gateway

Next steps

• Runtime Request Families: find the capability limits of each streaming route.
• Format Translation: see which provider stream formats the gateway translates.
• verdictan gateway run: start the gateway that serves the stream.
• Provider Configuration: configure the provider target and its credential reference.