Gateway overhead
This page gives the time that the Verdictan gateway adds to a request. The numbers come from the benchmark in the bench/ directory of the Verdictan CLI source. You can run the same benchmark on your hardware.
What the benchmark measures
The benchmark measures overhead. Overhead is the time that the gateway adds between your application and the model provider.
The benchmark does not use a real model provider. A mock provider on the same computer sends each response after a fixed time. The difference includes gateway work and local scheduling variation. It excludes network delay to a remote control plane.
Method
The benchmark does these steps for each policy chain:
- It starts a mock OpenAI-compatible provider. The provider sends a response after 20 ms. A streamed response starts after 20 ms and has 16 chunks, 5 ms apart.
- It starts a mock Verdictan control plane. The mock control plane answers token validation, usage authorization, and event delivery immediately.
- It starts
verdictan gateway runand waits until/readyzshows that the gateway is ready. - It sends each set of requests directly to the mock provider. This result is the baseline.
- It sends the same requests through the gateway.
- It calculates the overhead as the gateway percentile minus the baseline percentile.
The benchmark uses these fixed settings:
| Setting | Value |
|---|---|
| Warm-up requests in each run | 25 |
| Measured requests in each run | 250 |
| Concurrency | 1, 10, and 50 requests at the same time |
| Request body | A system message and a user message of approximately 1 KiB |
| Payload seed | 20261023 |
| Cache | Each prompt is different, so the cache does not answer a request |
Each scenario has three accepted runs. The reported value is the median of the three run values. The one-minute load must stay below 16 at the recorded start and end of each scenario. The comparison excludes any run with failed requests or incomplete samples.
The benchmark uses three policy chains:
| Chain | Policies |
|---|---|
| No policies | None. The gateway authenticates, routes, and records events. |
| Starter chain | prompt-injection and pii-detector, with default settings |
| Finance chain | The finance starter template: seven policies. The benchmark removes its Valkey rate limits. |
Results
These results measure build b35050794 on Linux with an AMD Ryzen 9 7950X3D and 32 logical CPUs.
The same executable runs on two local XFS filesystems with the same durability settings.
The original disk is a Samsung SSD 990 PRO 4TB. The faster disk is a KIOXIA-EXCERIA PLUS G3 SSD.
These results describe the tested configurations. They do not compare these products under all workloads.
The mock provider and control plane run on the same computer. Each value is the median of three accepted runs. Overhead values are in milliseconds. Throughput counts completed requests each second and excludes the final event-drain wait. Overhead percentile values can decrease because each value subtracts results from separate request sets.
| Chain | Mode | Concurrency | Original disk p50 / p95 / p99 | Original disk requests/s | Faster disk p50 / p95 / p99 | Faster disk requests/s |
|---|---|---|---|---|---|---|
| No policies | non-stream | 1 | 30.1 / 36.4 / 39.1 | 19.5 | 17.5 / 21.5 / 24.0 | 25.9 |
| No policies | non-stream | 10 | 87.3 / 109.0 / 130.8 | 90.8 | 71.3 / 87.6 / 97.7 | 106.0 |
| No policies | non-stream | 50 | 335.3 / 415.8 / 457.5 | 136.1 | 338.4 / 398.8 / 423.7 | 134.3 |
| No policies | stream | 1 | 30.3 / 36.1 / 47.1 | 7.8 | 18.6 / 20.8 / 21.9 | 8.6 |
| No policies | stream | 10 | 73.7 / 88.6 / 89.5 | 57.9 | 72.6 / 87.5 / 88.8 | 58.8 |
| No policies | stream | 50 | 313.3 / 340.2 / 373.3 | 118.4 | 303.6 / 367.1 / 375.4 | 119.8 |
| Starter | non-stream | 1 | 34.0 / 42.7 / 47.7 | 18.0 | 20.1 / 25.4 / 27.4 | 24.2 |
| Starter | non-stream | 10 | 108.4 / 134.1 / 148.6 | 76.7 | 99.6 / 133.6 / 150.1 | 80.1 |
| Starter | non-stream | 50 | 455.1 / 559.1 / 627.9 | 99.1 | 457.8 / 573.0 / 634.5 | 99.5 |
| Starter | stream | 1 | 33.5 / 44.0 / 48.9 | 7.6 | 21.5 / 24.6 / 26.0 | 8.4 |
| Starter | stream | 10 | 102.5 / 125.9 / 128.9 | 50.4 | 96.4 / 115.4 / 132.1 | 52.6 |
| Starter | stream | 50 | 452.4 / 528.1 / 551.0 | 87.8 | 434.9 / 466.1 / 463.7 | 93.6 |
| Finance | non-stream | 1 | 61.0 / 87.2 / 189.0 | 11.6 | 28.0 / 32.2 / 35.5 | 20.5 |
| Finance | non-stream | 10 | 161.3 / 193.3 / 214.2 | 54.2 | 151.0 / 186.8 / 197.6 | 56.9 |
| Finance | non-stream | 50 | 722.0 / 890.3 / 1058.9 | 65.0 | 721.2 / 896.2 / 1073.0 | 65.9 |
| Finance | stream | 1 | 48.0 / 63.1 / 77.5 | 6.8 | 25.4 / 29.6 / 34.7 | 8.2 |
| Finance | stream | 10 | 126.8 / 154.8 / 173.5 | 43.9 | 112.2 / 147.9 / 159.0 | 47.0 |
| Finance | stream | 50 | 606.3 / 702.8 / 738.9 | 68.6 | 635.6 / 705.5 / 705.9 | 67.5 |
Time to first byte
Time to first byte (TTFB) measures when the client receives the first response bytes. These values show TTFB overhead with one request at a time.
| Chain | Original disk TTFB p50 / p95 / p99 (ms) | Faster disk TTFB p50 / p95 / p99 (ms) |
|---|---|---|
| No policies | 13.8 / 16.8 / 21.2 | 9.4 / 12.2 / 13.6 |
| Starter | 93.7 / 98.6 / 103.3 | 89.4 / 92.8 / 95.0 |
| Finance | 100.2 / 106.0 / 113.6 | 92.9 / 97.0 / 99.3 |
The starter chain checks response PII before it exposes output to the client. This check requires the complete response. The finance chain also checks response content for MNPI, financial compliance, and quality. The gateway buffers output when these checks require it. A chain without response checks can send chunks as they arrive.
The full comparison records all three runs, their load readings, and the before/after measurements. See the benchmark results.
Limits of these results
- Shared test computer. Accepted scenarios have recorded one-minute load below 16 at both endpoints. This does not guarantee an idle computer.
- Control-plane time. The benchmark records about 12 calls per request, including background event delivery. Admission calls add network delay in remote deployments.
- Provider time. The benchmark uses a fixed mock provider. These results do not predict latency with a real model provider.
- Optional models. The benchmark does not install the optional local models. Policies that use them can take more time.
- Disk. Requests wait for durable audit writes. Faster storage reduces this wait, but serialization and policy checks still limit throughput.
- Durability. Both configurations use the same sync boundaries. The comparison does not test hardware behavior during power loss.
Run the benchmark
Run the benchmark from a checkout of the Verdictan CLI source:
make -C cli bench VERDICTAN_BIN=/path/to/verdictanThe command writes bench/RESULTS.md and bench/results.json. The results include the commit, the gateway version and checksum, the Node.js version, the CPU, the memory, and the system load. If other workloads load the computer during the run, RESULTS.md shows a warning.
To do a short test of the setup, add BENCH_ARGS=--quick. If ports 8840 to 8842 are in use, set BENCH_PORT_BASE to a different first port.
To compare storage, select an existing directory on another durable filesystem:
make -C cli bench VERDICTAN_BIN=/path/to/verdictan \
BENCH_ARGS="--out /path/to/run-1 --state-root /path/on/durable/storage"Keep the executable and other settings the same. Repeat each configuration three times with separate output directories. The harness removes only its own temporary gateway directory.