Replicate

Run a Replicate text-generation model through your Verdictan gateway. Your application sends a Chat Completions request. The gateway creates a prediction, waits for it, and returns the text output as a chat reply.

This works only with models that take a text prompt input and return text output. Check the model's input schema on Replicate before you start.

Before you start

• A working gateway from the quickstart.
• A Replicate API token, and one model in owner/model form with a text prompt input and text output.
• The gateway startup command, and a separate application token for test requests.

Configure the provider

provider_type and format select the native Replicate translation. Keep them as shown.

policy-config.yaml

pack:
  name: replicate-integration
  version: 1.0.0
  enabled: true
policies:
  chain:
    - prompt-injection
providers:
  targets:
    - id: replicate-primary
      provider: replicate
      provider_type: replicate
      format: replicate
      model: "replace-with-owner/model"
      base_url: https://api.replicate.com
      secret_key_ref:
        env: VERDICTAN_REPLICATE_API_KEY

The gateway posts to /v1/models/{model}/predictions with Authorization: Token <key>, using the key from VERDICTAN_REPLICATE_API_KEY. It also sends Prefer: wait, so Replicate holds the connection until the prediction ends or its wait period runs out.

Start and verify

You can deploy this Replicate configuration with CLI 0.1.14 or later. Check In sync, then use the request below to verify the selected model and Replicate token.

With the quickstart in Docker, follow the container steps instead of steps 1 to 3.

1. In the gateway terminal, set the token and lint the file:

   export VERDICTAN_REPLICATE_API_KEY="<provider-api-key>"
   verdictan policy lint --file policy-config.yaml

2. Restart the test gateway with the Replicate token in its environment.

3. Deploy the file to the test agent in Configurations. If that agent already runs pack.version 1.0.0, increase the version first. Then wait for the gateway log line connected read model refresh applied updated config with your version as config_version. Try the change on a test gateway has the steps.

4. In a second terminal, send a non-streaming request for the configured model:

   export VERDICTAN_CLIENT_TOKEN="<application-token>"
   curl --fail-with-body http://127.0.0.1:41002/v1/chat/completions \
     -H "Authorization: Bearer ${VERDICTAN_CLIENT_TOKEN}" \
     -H "Content-Type: application/json" \
     -d '{"model":"replace-with-owner/model","stream":false,"messages":[{"role":"user","content":"Reply with Hello."}]}'

choices[0].message.content must contain text. The request appears in History when capture is on, or in the delivered events.

warning

Empty content usually means the prediction did not finish within the wait period, or the model returns a different output schema. The gateway does not poll for results. Don't retry in a loop: each attempt can start a new billable prediction.

How the request is translated

Sent to Replicate	From the request
input.prompt	The last user message
input.system_prompt	All system messages, joined
input.max_new_tokens	max_tokens or max_completion_tokens
input.temperature	temperature
input.top_p	top_p

Other request fields and earlier turns are not sent. The reply is read from a text or text-array output.

Request	Support
Chat Completions, stream: false	Yes, for models on the /v1/models/{owner}/{model}/predictions endpoint
Streaming, Messages, legacy Completions	Not tested
Tools, image input, embeddings, Responses, response_format	No
Version-specific or deployment endpoints	No

Failure behavior

Problem	What to check
401 from the gateway	The application token is active, and the token is set on the gateway host.
Provider authentication error	VERDICTAN_REPLICATE_API_KEY is set in the gateway process. Restart the gateway.
Model not found	The model is in owner/model form and matches the target.
200 with empty content	The prediction did not finish, or the model's output is not text.
Provider 422	The model does not accept the input fields in the table.
Provider 429 or another upstream error	The response body and your Replicate limits. See Error Handling.

Limits

• Model names, prices, rate limits, and data terms are in the Replicate documentation.
• A listener on a non-loopback address needs TLS and network controls.

Next steps

• Format Translation: how the gateway translates between provider formats.
• Provider Configuration: target fields, routing, and fallback.