Content Extractor

content-extractor records the URLs in a request and can fetch their text from hosts that you allow. The URLs and text go into the decision details, as evidence of what the request linked to.

It does not add the fetched text to the prompt, and later controls in the chain do not examine it. If the model must use a source, fetch it in your application and put it in the request.

Configuration

Start by recording URLs without any network request:

policy-config.yaml

pack:
  name: record-request-urls
  version: 1.0.0
  enabled: true
policies:
  chain:
    - content-extractor
policy:
  content-extractor:
    fetch_urls: false

Check it offline with one test case:

Add to policy-config.yaml

testing:
  suites:
    - name: content-extractor
      cases:
        - name: records-url
          input: {messages: [{role: user, content: "Read https://example.com/guide please"}]}
          expected: {verdict: allow, reason_code: ok}

verdictan policy test --json passes, and the nested result lists https://example.com/guide in details.urls with no fetched text.

To fetch content, set fetch_urls: true and list the exact hosts in allow_hosts. The host must serve public HTTPS text, without redirects or compression. See Examples.

Fields

Field	Type	Default	Description
allow_hosts	list of strings	[]	Exact host names that can be fetched. Case does not matter. No wildcards. An empty list fails every fetch with host_not_allowlisted.
timeout_ms	integer	2000	Timeout for the connection, the read, and the whole extraction. At most 10000. Lint rejects 0.
max_bytes	integer	65536	Bytes kept from one response body, at most 1 MiB. A longer body is cut without an error.
fetch_urls	boolean	true	false only records URLs
action_on_error	warn or block	warn	warn allows the request when a fetch fails. block blocks it.

Lint warns when fetch_urls is true and allow_hosts is empty.

How it works

1. The gateway joins the text of all request messages.
2. The control finds up to eight http:// or https:// URLs. With more than eight, it stops with url_limit_exceeded and fetches nothing.
3. With fetch_urls: false or no URL, it allows with content-extractor.ok.
4. Each URL must use https, have a host in allow_hosts, have no user name or password, and resolve only to public addresses.
5. It fetches up to four URLs at a time with GET, follows no redirects, and accepts only an uncompressed 2xx body.
6. It keeps the first max_bytes of each body, strips HTML tags, and stores the text in details.extracted_text in URL order.
7. The first failure stops the extraction. The reason goes into details.blocked_reason, and the reason code is content-extractor.<reason>.
8. A failure blocks with action_on_error: block. Everything else is allowed.

The details contain urls, extracted_text, and blocked_reason.

Phase and verdicts

Item	Value
Stages	pre-request (default), pre-response
Request families	Chat Completions, Completions, Responses, Messages
Verdicts	allow, block
Reason codes	content-extractor.ok, content-extractor.<blocked_reason>

Blocked reason	Cause
url_limit_exceeded	More than eight URLs
invalid_url	The URL does not parse or has no usable port
url_userinfo_blocked	The URL has a user name or password
https_required	The URL uses http://
unsupported_url_scheme	The scheme is not http or https
missing_host	The URL has no host
host_not_allowlisted	The host is not in allow_hosts
dns_resolution_failed	The host does not resolve
private_address_blocked	The host resolves to a loopback, private, link-local, or other restricted address
redirect_blocked	The server returned 3xx
http_status_error	The server returned a status other than 2xx
unsupported_content_encoding	The body is compressed
aggregate_body_limit_exceeded	All bodies together are larger than 4 MiB
timeout	The connection, read, or whole extraction took longer than timeout_ms
fetch_failed	Connection or request error
body_read_failed	Error while reading the body
invalid_utf8	The body is not UTF-8 text
client_build_failed	The gateway could not create its HTTP client

Examples

Fetch from files.example.com, and block the request if the fetch fails:

Inside policy-config.yaml

policy:
  content-extractor:
    allow_hosts:
      - files.example.com
    timeout_ms: 10000
    max_bytes: 524288
    fetch_urls: true
    action_on_error: block

Limits and failure behavior

• Only https:// URLs are fetched. With VERDICTAN_ENV=development, the gateway also fetches localhost and loopback URLs over HTTP.
• A block returns HTTP 400 with error code content_policy_violation. With the default warn, a failed fetch only shows as the reason code of an allow.
• Later controls, such as prompt-injection or dlp-filter, do not examine details.extracted_text.
• The decision details can hold up to max_bytes of text per URL. Restrict access to decision events if the linked documents are confidential.
• Fixed runtime limits: eight URLs, four parallel fetches, 10 seconds in total, 1 MiB per body, and 4 MiB for all bodies.
• In pre-response, the control reads URLs from the model response with the same rules.
• Lint rejects an unknown action_on_error and unknown fields. A block with a wrong type that bypasses lint uses all defaults, so every fetch fails with host_not_allowlisted.

Next steps

• Document Analyzer: documents inside the request body.
• Investigate a Blocked Request: read the blocked_reason of a failed fetch.