Skip to main content

Cloud Platforms

Amazon SageMaker Model Serving

Call your own SageMaker real-time endpoint through Verdictan. The gateway signs each request with AWS credentials from its own environment, and your application only ever holds a Verdictan application token.

Your inference container must accept and return OpenAI-compatible JSON. Verdictan forwards the OpenAI body unchanged and does not adapt it to other model schemas.

Before you start​

  • A working gateway from the quickstart.
  • A SageMaker real-time endpoint whose container reads and writes the OpenAI format.
  • AWS credentials allowed to call sagemaker:InvokeEndpoint on that endpoint.
  • The gateway startup command, and a separate application token for test requests.

Configure the provider​

model is the endpoint name. base_url is the SageMaker runtime host of the endpoint's region, with https://, and aws_region must name the same region.

policy-config.yaml
pack:
  name: sagemaker-integration
  version: 1.0.0
  enabled: true
policies:
  chain:
    - prompt-injection
providers:
  targets:
    - id: sagemaker-primary
      provider: sagemaker
      model: "replace-with-endpoint-name"
      base_url: https://runtime.sagemaker.us-east-1.amazonaws.com
      aws_region: us-east-1

There is no secret_key_ref. The gateway signs each request with AWS Signature Version 4 on the gateway host. It does not use secret_key_ref or api_key_prefix for this provider. It gets AWS credentials in this order:

  1. The variables that the optional target fields aws_access_key_env and aws_secret_key_env name. The optional aws_session_token_env field names a session token variable. Set the first two fields together.
  2. AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY from the gateway environment, with AWS_SESSION_TOKEN when it is set.
  3. If neither of the two variables is set, the AWS SDK default credential chain for aws_region. This includes an instance role. Set aws_profile on the target to use a named profile.

When the target sets the credential fields, use names with the VERDICTAN_ prefix, such as VERDICTAN_SAGEMAKER_ACCESS_KEY_ID.

Start and verify​

Use the Console procedure below or CLI 0.1.14 or later to deploy this endpoint configuration. Check In sync. Then send the non-streaming request to verify the endpoint and AWS credentials.

With the quickstart in Docker, follow the container steps instead of steps 1 to 3.

  1. In the gateway terminal, set the AWS credentials (and AWS_SESSION_TOKEN for temporary ones), then lint:

    export AWS_ACCESS_KEY_ID="replace-with-access-key-id"
export AWS_SECRET_ACCESS_KEY="replace-with-secret-access-key"
verdictan policy lint --file policy-config.yaml

  2. Restart the test gateway with the AWS credentials in its environment.

  3. Deploy the file to the test agent in Configurations. If that agent already runs pack.version 1.0.0, increase the version first. Then wait for the gateway log line connected read model refresh applied updated config with your version as config_version. Try the change on a test gateway has the steps.

  4. In another terminal, send a non-streaming request whose model is the endpoint name:

    export VERDICTAN_CLIENT_TOKEN="replace-with-different-client-token"
curl --fail-with-body http://127.0.0.1:41002/v1/chat/completions \
  -H "Authorization: Bearer ${VERDICTAN_CLIENT_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"model":"replace-with-endpoint-name","stream":false,"messages":[{"role":"user","content":"Reply with Hello."}]}'

choices[0].message.content contains a reply. If it fails, tell AWS authorization errors apart from errors that the container returns.

Supported requests​

RequestSupport
Chat CompletionsYes, sent to /endpoints/{endpoint-name}/invocations with the OpenAI body
MessagesTranslated to Chat Completions
EmbeddingsYes, if the container accepts the embeddings schema. The policy chain does not run on embeddings.
StreamingNo. Keep stream: false. The SageMaker streaming API is not implemented.
Tool callsNot verified
Responses, image input, structured output, parallel tool callsNo

Failure behavior​

SituationResult
model or base_url is emptyLint stops with an error naming the key.
base_url is not https://The configuration is rejected.
aws_region is emptyThe gateway starts, but each request fails with sagemaker requires explicit aws_region in provider config.
Only one of aws_access_key_env and aws_secret_key_env is setThe configuration is rejected.
A variable that a credential field names is not setThe gateway starts, but each request fails with which is not set or empty.
Only one of AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY is setThe gateway starts, but each request fails with found only one of AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.
No AWS credentials are foundThe gateway starts, but each request fails with AWS credential resolution failed.
No sagemaker:InvokeEndpoint permissionSageMaker returns an authorization error, which the gateway passes on.
The request model is not the endpoint nameThe gateway does not select the target.
SageMaker returns 4xx or 5xxThe gateway passes the status on.
The runtime host is unreachableThe gateway returns 502 with upstream_error.

Limits​

  • Prices, regional hosts, and quotas are in the SageMaker real-time inference documentation.
  • The base_url host and aws_region must be the same region: the gateway signs for aws_region and sends to the base_url host.
  • The container owns the request and response schema.
  • If the SageMaker call fails with a rate limit, a server error, an authentication error, or a transport error, the gateway tries the next eligible target. Provider attempts and fallback has the rules.
  • A listener on a non-loopback address needs TLS and network controls.

Next steps​