Skip to main content

Model Providers

Cohere

Call Cohere chat models through your Verdictan gateway. Your application sends OpenAI Chat Completions or Responses requests with its application token. The gateway translates each request to the Cohere Chat v2 API and calls Cohere with your Cohere key.

Streaming and non-streaming requests both work. See Supported requests.

Before you start​

  • A working gateway from the quickstart.
  • A Cohere API key, and one chat model ID that the key can use.
  • The gateway startup command, and a separate application token for test requests.

Configure the provider​

policy-config.yaml
pack:
  name: cohere-integration
  version: 1.0.0
  enabled: true
policies:
  chain:
    - prompt-injection
providers:
  targets:
    - id: cohere-chat
      provider: cohere
      model: "replace-with-cohere-chat-model-id"
      base_url: https://api.cohere.com
      secret_key_ref:
        env: VERDICTAN_COHERE_API_KEY

Use the exact value provider: cohere. The gateway rejects aliases such as cohere-chat. base_url is required and must start with https://. If you use a host that contains cohere.ai, also set provider_type: cohere and format: cohere, or verdictan policy lint reports an error.

The gateway sends each request to /v2/chat on that host, with the key from VERDICTAN_COHERE_API_KEY as a Bearer token.

Start and verify​

For terminal deployment, use CLI 0.1.14 or later with this file. After In sync, test both streaming and non-streaming requests if your application uses them.

With the quickstart in Docker, follow the container steps instead of steps 1 to 3.

  1. In the gateway terminal, set the key, then check the file. policy lint checks the policy pack, and config validate checks the provider target fields.

    export VERDICTAN_COHERE_API_KEY="<provider-api-key>"
verdictan policy lint --file policy-config.yaml
verdictan config validate --file policy-config.yaml

  2. Restart the test gateway with the key in its environment.

  3. Open Configurations in the Console. Create or edit the configuration, import the file, and select Next. Under Deploy to agent, select the test agent. Select Deploy and confirm the gateway count. Wait about 30 seconds for the configuration pull. Send the request below and check the reply.

  4. In a second terminal, send a streaming request:

    export VERDICTAN_CLIENT_TOKEN="<application-token>"
curl -N --fail-with-body http://127.0.0.1:41002/v1/chat/completions \
  -H "Authorization: Bearer ${VERDICTAN_CLIENT_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"model":"replace-with-cohere-chat-model-id","stream":true,"messages":[{"role":"user","content":"Reply with Hello."}]}'

The reply arrives as data: lines. Each line holds a chat.completion.chunk object with text in choices[0].delta.content, and the stream ends with data: [DONE]. The text proves the whole path: application token, Cohere key, and model access. The request appears in History when capture is on, or in the delivered events.

Supported requests​

RequestSupport
Chat Completions with stream: trueYes. Text and tool calls stream back as Chat Completions chunks.
Responses with stream: trueYes. Events stream back in the Responses format.
Chat Completions or Responses without streamingYes. The reply comes back as one Chat Completions or Responses object, with token usage.
ToolsYes. Function tools and tool_choice are forwarded.
response_format with json_object or json_schemaYes, forwarded unchanged. A schema with strict: true is rejected.
Messages, embeddings, image input, parallel_tool_calls: true, strict toolsNo. The gateway rejects the request with HTTP 422 before it calls Cohere.

Both supported request families go through your policy chain.

The gateway builds a new Cohere request from these fields: model, messages (or Responses input), stream, max_tokens or max_completion_tokens, temperature, top_p, presence_penalty, frequency_penalty, seed, response_format, tools, and tool_choice. It renames stop to stop_sequences. It drops all other fields.

Assistant tool_calls stay in the message history, and tool messages become Cohere document content. When tool_choice is required or names a function, the gateway also sets strict_tools: true.

Failure behavior​

ProblemResult or cause
Lint reports provider 'cohere-chat' must use exact provider id 'cohere'provider is an alias. Set it to cohere.
Lint reports base_url looks like Cohere API but 'format' and 'provider_type' are not setThe host contains cohere.ai. Add provider_type: cohere and format: cohere.
config validate reports cohere runtime requires an https:// base_urlUse an https:// base URL.
The gateway reports Provider target 'cohere-chat' is inactive: environment variable 'VERDICTAN_COHERE_API_KEY' is not setSet the key in the gateway process and restart it.
401 from the gatewayThe application token is not active.
400 with provider_request_invalidThe body uses the Cohere v1 shape (message and chat_history). Send Chat Completions or Responses requests.
422 with capability.unsupported_familyThe request uses a family that this provider does not support, such as /v1/messages or /v1/embeddings.
422 with runtime.structured_output.strict_mode_unsupportedA tool or the json_schema sets strict: true.
Cohere returns an error statusThe client gets the same status with code upstream_provider_error and the message Upstream provider returned HTTP <status>. The Cohere error body is not returned. Check the key, the model, and your Cohere limits. If another target is eligible, some errors move the request to that target first.

Limits​

  • Model IDs, prices, rate limits, and request fields are in the Cohere Chat v2 reference.
  • finish_reason contains the value that Cohere returns, not an OpenAI value.
  • The gateway does not return Cohere citations.
  • If the Cohere call fails with a rate limit, a server error, an authentication error, or a transport error, the gateway tries the next eligible target. Provider attempts and fallback has the rules.
  • A listener on a non-loopback address needs TLS and network controls.

Next steps​