Quality Scorer

quality-scorer checks a finished answer before your application receives it. It can require a minimum length, score the answer with local benchmarks, and run your assertions. A failed answer is blocked or replaced with a fixed message.

It runs on Chat Completions and Responses with stream: false. A good score does not prove that the facts in an answer are correct.

Configuration

Start with one measurable rule. Add it to a test gateway with the config-first workflow:

policy-config.yaml

pack:
  name: minimum-answer-length
  version: 1.0.0
  enabled: true
policies:
  chain:
    - quality-scorer
policy:
  quality-scorer:
    min_output_chars: 20
    failure_action:
      action: block

Compare a short and a long answer

Test the rule offline with two JSON test files that give the provider answer in upstream_response:

tests/short.json

{"name": "short answer blocks",
 "input": {"messages": [{"role": "user", "content": "Is the report ready?"}],
           "upstream_response": {"choices": [{"message": {"role": "assistant", "content": "OK"}}]}},
 "expected": {"verdict": "block", "reason_code": "quality.too_short"}}

tests/long.json

{"name": "long answer passes",
 "input": {"messages": [{"role": "user", "content": "Is the report ready?"}],
           "upstream_response": {"choices": [{"message": {"role": "assistant", "content": "The requested report is ready for review."}}]}},
 "expected": {"verdict": "allow", "reason_code": "ok"}}

Run verdictan policy test --json. Both pass. Through a gateway, the short answer returns HTTP 400 with quality.too_short in the decision. Another control can still block an answer that passes this rule, so read the nested policy results to find the failing check.

To return a fixed message instead of an error, use:

Inside policy.quality-scorer

failure_action:
  action: fallback
  fallback_message: "The answer did not meet the minimum length."

The gateway then returns HTTP 200 with that message and reason code quality.fallback. It does not call the model again.

When the basic rule works, add quality assertions for JSON, required phrases, or a rubric.

Fields

Field	Type	Default	Description
mode	string	buffer	buffer scores the complete response. stream_through returns allow with quality.deferred and does not score.
min_output_chars	integer	0	Fails an answer with fewer characters. 0 turns it off.
min_sentences	integer	0	Fails an answer with fewer sentences. 0 turns it off.
benchmarks	object	{}	Switches: ragas_faithfulness, ragas_relevancy, bleu_score, nli_entailment, coherence, completeness
bleu_reference	string		Reference text for bleu_score when the request has no context documents
assertions	array	[]	Assertions. See the fields below.
thresholds	object	{}	Minimum scores: min_aggregate, min_faithfulness, min_relevancy, min_bleu, min_accuracy, min_coherence, min_completeness. A threshold turns on its benchmark if the benchmark is not set.
weights	object	{}	Weights in the aggregate: faithfulness, relevancy, bleu, accuracy, coherence, completeness. An enabled benchmark without a weight uses 1.0.
failure_action.action	string	block	block, fallback, or retry. Only fallback changes the response.
failure_action.fallback_message	string	Built-in text	Replacement text with action: fallback. The default is I apologize, but I cannot provide a sufficiently accurate response to this query.
pass_policy.strategy	string	all	all, quorum, or weighted_average, for assertions with mode: enforce and severity: critical
pass_policy.quorum	number	0.5	Share of those assertions that must pass with quorum
pass_policy.threshold	number	0.5	Minimum weighted score with weighted_average
industry_profiles.<industry>	object	{}	Threshold overrides for requests with verdictan.industry set to this name
judge	object		Optional judge model call. Its result appears in the details and does not change the verdict.

Assertion fields

Field	Type	Default	Description
type	string		Assertion type. Required. An unsupported type shows as an assertion error.
name	string	The type	Name in the details
enabled	boolean	true	false skips the assertion
threshold	number	Depends on type	Minimum score that passes
weight	number	1.0	Weight with weighted_average
mode	string	enforce	enforce, audit, or shadow. Only enforce can fail an answer.
severity	string	critical	critical, warning, or info. Only critical can fail an answer.
config	object	{}	Options of the type, such as value for contains or rubric for llm-rubric

Quality Assertion Configuration lists the types and their options.

How it works

1. The control reads the answer from choices[0].message.content, and the query and context documents from the request.
2. An answer shorter than min_output_chars records quality.too_short. Fewer sentences than min_sentences records quality.too_few_sentences.
3. If the request has verdictan.industry and industry_profiles has that name, the profile thresholds replace the top-level ones.
4. Each enabled benchmark is scored with local heuristics. If the process environment has VERDICTAN_NLI_ENDPOINT, nli_entailment calls that endpoint with a 5-second timeout.
5. Each enabled assertion is evaluated and scored.
6. The aggregate combines the weighted benchmark and assertion scores.
7. Each score below its threshold records a failure, such as quality.faithfulness_below_threshold or quality.aggregate_below_threshold.
8. pass_policy is applied to the enforced, critical assertions. A failure records the names of the failed assertions.
9. No failure: allow with ok.
10. A failure with fallback: the gateway replaces the answer with fallback_message, and the verdict is allow with quality.fallback.
11. A failure with any other action: block, with the comma-separated failures as the reason code, such as quality.too_short,quality.aggregate_below_threshold.

The details contain output_chars, sentence_count, metrics, thresholds, weights, has_context, assertions, failures, pass_policy_result, and judge. Scores in the details are percentages.

Phase and verdicts

Item	Value
Phase	output
Enforced path	Chat Completions and Responses with stream: false
Verdicts	allow, block
Reason codes	ok, quality.fallback, quality.deferred, and one or more of quality.too_short, quality.too_few_sentences, quality.<metric>_below_threshold, quality.aggregate_below_threshold, or an assertion name

A block returns HTTP 400 with error code content_policy_violation and a verdictan.decision object with verdict BLOCK. A fallback returns HTTP 200 with the replacement text and quality.fallback in verdictan.decision.

Limits and failure behavior

• With stream: true, the gateway buffers the response but does not score it.
• failure_action.action: retry does not retry. It blocks. Lint accepts max_retries, but the runtime ignores it.
• Benchmarks are local heuristics, not model evaluations. Measure the thresholds on your own traffic before you block on them.
• thresholds.min_aggregate without benchmarks or assertions turns on ragas_relevancy and completeness to calculate an aggregate.
• ragas_faithfulness needs context documents in the request. Without them, there is no faithfulness score, and min_faithfulness cannot fail. Without context, bleu_score uses bleu_reference.
• Audit and shadow assertion results stay available in the details but never block.
• The judge adds latency and provider cost to every response it evaluates.
• Lint also accepts industry, mock_scoring, providers, targets, and regression_monitoring. This page does not cover them.

Next steps

• Quality Assertion Configuration: assertion types and options.
• Citation Verifier: check that the answer cites the supplied context.
• Flagged Review Configuration: a second model reviews each answer.