Hugging Face Inference

Call a Hugging Face text-generation model on HF Inference through your Verdictan gateway. Your application sends a Chat Completions request. The gateway flattens the messages into one prompt, calls the model, and returns the generated text as a chat reply.

This adapter covers text generation only. It is not a general adapter for other Hugging Face tasks.

Before you start

• A working gateway from the quickstart.
• A Hugging Face access token, and one text-generation model that HF Inference serves, in organization/model form.
• The gateway startup command, and a separate application token for test requests.

Configure the provider

provider_type and format select the native Hugging Face translation. Keep them as shown.

policy-config.yaml

pack:
  name: huggingface-integration
  version: 1.0.0
  enabled: true
policies:
  chain:
    - prompt-injection
providers:
  targets:
    - id: huggingface-primary
      provider: huggingface
      provider_type: huggingface
      format: huggingface
      model: "replace-with-organization/model"
      base_url: https://router.huggingface.co/hf-inference
      secret_key_ref:
        env: VERDICTAN_HUGGINGFACE_API_KEY

The gateway sends requests to base_url plus /models/{model}, with the token from VERDICTAN_HUGGINGFACE_API_KEY as a Bearer token.

Start and verify

You can deploy the Hugging Face target from a file with CLI 0.1.14 or later. After In sync, send the non-streaming request below to check the endpoint and token.

With the quickstart in Docker, follow the container steps instead of steps 1 to 3.

1. In the gateway terminal, set the token and lint the file:

   export VERDICTAN_HUGGINGFACE_API_KEY="<provider-api-key>"
   verdictan policy lint --file policy-config.yaml

2. Restart the test gateway with the Hugging Face token in its environment.

3. Deploy the file to the test agent in Configurations. If that agent already runs pack.version 1.0.0, increase the version first. Then wait for the gateway log line connected read model refresh applied updated config with your version as config_version. Try the change on a test gateway has the steps.

4. In a second terminal, send a non-streaming request for the configured model:

   export VERDICTAN_CLIENT_TOKEN="<application-token>"
   curl --fail-with-body http://127.0.0.1:41002/v1/chat/completions \
     -H "Authorization: Bearer ${VERDICTAN_CLIENT_TOKEN}" \
     -H "Content-Type: application/json" \
     -d '{"model":"replace-with-organization/model","stream":false,"messages":[{"role":"user","content":"Reply with Hello."}]}'

choices[0].message.content must contain text. An empty string means the check failed, even with HTTP 200: the model probably does not return generated_text. The request appears in History when capture is on, or in the delivered events.

Supported requests

Request	Support
Chat Completions, stream: false	Yes, for text-generation models that return generated_text
Streaming, Messages, embeddings, other HF tasks	Not tested with this format
Tools, image input, Responses, response_format	No

The gateway sends only these fields to the model:

Sent to Hugging Face	From the request
inputs	All messages, flattened into one prompt
parameters.max_new_tokens	max_tokens or max_completion_tokens
parameters.temperature	temperature
parameters.top_p	top_p
options.wait_for_model	Always true

Failure behavior

Problem	What to check
401 from the gateway	The application token is active, and the token is set on the gateway host.
Provider authentication error	VERDICTAN_HUGGINGFACE_API_KEY is set in the gateway process. Restart the gateway.
Model not found	HF Inference serves the model, and the ID is in organization/model form.
200 with empty content	The model is not a text-generation model, or it returns another response shape.
capability.unsupported_family	The request is Chat Completions.
Provider 429 or another upstream error	The response body and your Hugging Face limits. See Error Handling.

Limits

• Model names, prices, rate limits, and data terms are in the HF Inference documentation.
• Custom endpoint schemas and non-text-generation tasks do not work with this target.
• A listener on a non-loopback address needs TLS and network controls.

Next steps

• Format Translation: how the gateway translates between provider formats.
• Provider Configuration: target fields, routing, and fallback.