Evaluator library

Every evaluator returns a pass or a fail. A fail carries one named failure_mode. Import evaluators from sengol.evaluators, or list them by class name in sengol.yaml and ControlBooks. List them, filtered by policy, from the CLI:

sengol eval list --policy PIPEDA

NAME                       VERSION  CATEGORY       POLICIES                    FAILURE_MODE
-------------------------  -------  -------------  --------------------------  ---------------------------
PIIEvaluator               1.0.0    DETERMINISTIC  OSFI_E23,FINTRAC_6F,PIPEDA  PII_DETECTED
PIPEDAIdentifierEvaluator  1.0.0    DETERMINISTIC  PIPEDA                      PIPEDA_IDENTIFIER_DISCLOSED
PIILeakEvaluator           1.0.0    LLM            OSFI_E23,PIPEDA             PII_LEAK_DETECTED

A bare class name in sengol.yaml or a ControlBook builds the evaluator with default arguments. In sengol.yaml you can pass arguments with - name: KeywordEvaluator and an args: mapping. With no phrases, RequiredDisclosureEvaluator fails every input.

Deterministic

No model and no network. They run first, in order. A CRITICAL failure skips the LLM evaluators that follow.

Evaluator	Arguments	Failure mode	Severity
`PIIEvaluator`	`judge=None` (optional second opinion)	`PII_DETECTED`: SIN (Luhn-checked), SSN, card number, email, Canadian postal code, date of birth	CRITICAL
`PIPEDAIdentifierEvaluator`	none	`PIPEDA_IDENTIFIER_DISCLOSED`	CRITICAL
`RequiredDisclosureEvaluator`	`required_phrases`, `case_sensitive=False`	`REQUIRED_DISCLOSURE_MISSING`	CRITICAL
`FINTRACTypologyEvaluator`	none	`AML_TYPOLOGY_INDICATOR_UNFLAGGED`	CRITICAL
`ForbiddenToolEvaluator`	`forbidden` (tool names)	`FORBIDDEN_TOOL_INVOKED`	CRITICAL
`MCPToolIntegrity`	none	`MCP_TOOL_DEFINITION_CHANGED`	CRITICAL
`UnauthorizedDelegation`	none	`UNAUTHORIZED_DELEGATION`	CRITICAL
`ExactMatchEvaluator`	`case_sensitive=False`	`EXACT_MATCH_FAILED` (also when `expected` is missing)	HIGH
`ResponseLengthEvaluator`	`min_length=0`, `max_length=10000` (characters)	`RESPONSE_TOO_SHORT`, `RESPONSE_TOO_LONG`	HIGH
`KeywordEvaluator`	`required_keywords`, `forbidden_keywords`	`REQUIRED_KEYWORD_MISSING`, `FORBIDDEN_KEYWORD_PRESENT`	HIGH
`RegexEvaluator`	`patterns`, `require_all=True`	`REGEX_PATTERN_NOT_MATCHED`	HIGH
`SchemaValidationEvaluator`	`schema` (JSON Schema)	`SCHEMA_VALIDATION_FAILED`	HIGH
`DriftMonitorPresent`	none; reads `metadata["drift_monitor_enabled"]`	`DRIFT_MONITOR_NOT_CONFIGURED`	HIGH
`ToolCallMatchEvaluator`	`expected` (tool names), `mode="exact"`	`TOOL_CALL_MISMATCH`	HIGH
`ToolArgSchemaEvaluator`	`schemas` (per tool)	`TOOL_ARG_SCHEMA_VIOLATION`	HIGH

The tool, MCP and delegation evaluators take trajectory or delegation inputs, not a plain prompt and response.

LLM judges

Each takes judge= (see Add an LLM judge). Without a judge, the evaluator scores LLM_EVALUATOR_ERROR. A judge that exceeds its severity timeout scores JUDGE_TIMEOUT.

Evaluator	Failure mode	Needs
`FaithfulnessEvaluator`	`FAITHFULNESS_FAIL`	`context`
`HallucinationEvaluator`	`HALLUCINATION_DETECTED`
`GroundednessEvaluator`	`GROUNDEDNESS_FAIL`	`context`, and a judge that answers in JSON
`RelevanceEvaluator`	`RELEVANCE_FAIL`
`TaskAdherenceEvaluator`	`TASK_ADHERENCE_FAIL`
`IntentResolutionEvaluator`	`INTENT_UNRESOLVED`
`RefusalEvaluator`	`REFUSAL_MISMATCH`
`ToxicityEvaluator`	`TOXICITY_DETECTED`
`ExplainabilityEvaluator`	`NO_EXPLANATION_PROVIDED`
`AdversarialRobustnessEvaluator`	`ADVERSARIAL_MANIPULATION_SUCCEEDED`
`GDPRDataSubjectRightsEvaluator`	`GDPR_DATA_SUBJECT_RIGHT_MISHANDLED`
`PIILeakEvaluator`	`PII_LEAK_DETECTED`	Deprecated: use `PIIEvaluator`
`RedTeamEvaluator`	`RED_TEAM_PROBE_BREACH`	`sengol[redteam]`

In our tests with OpenAICompatibleJudgeClient(model="gpt-4o-mini"), the judge answered GroundednessEvaluator in prose, which it scores as LLM_EVALUATOR_ERROR. FaithfulnessEvaluator and HallucinationEvaluator worked with the same client.

EnsembleEvaluator combines several judges and reports ENSEMBLE_MEMBER_FAILED.

Statistical model evaluators

For traditional ML models. Install sengol[model]: AUCEvaluator, PSIEvaluator, ABEvaluator, ChampionChallengerEvaluator, FairnessEvaluator, OutcomeMonitorEvaluator. They take a ModelEvalInput (features, prediction, ground truth) instead of a prompt and response.

Writing your own

Subclass DeterministicEvaluator or LLMEvaluator and declare a failure_mode:

custom_evaluator.py

import asyncio
from typing import ClassVar

from sengol.build.suite import EvalSuite
from sengol.core.evaluator import DeterministicEvaluator
from sengol.core.types import EvalScore, LLMEvalInput
from sengol.governance.audit_store import NullAuditStore


class NoGuaranteesEvaluator(DeterministicEvaluator):
    name: ClassVar[str] = "NoGuaranteesEvaluator"
    failure_mode: ClassVar[str] = "RETURN_GUARANTEED"
    severity: ClassVar[str] = "CRITICAL"

    def evaluate(self, input: LLMEvalInput) -> EvalScore:
        promised = "guaranteed return" in input.response.lower()
        return EvalScore(
            evaluator=self.name,
            passed=not promised,
            failure_mode=self.failure_mode if promised else None,
            reason="Response promises a guaranteed return." if promised else "",
        )


suite = EvalSuite([NoGuaranteesEvaluator()], audit_store=NullAuditStore(), tenant_id="local")
result, _ = asyncio.run(suite.run(LLMEvalInput(
    agent_id="fund-advisor", agent_version="1.0.0",
    prompt="Is Fund X safe?", response="Fund X has a guaranteed return of 8%.",
)))
print([(s.evaluator, s.passed, s.failure_mode) for s in result.scores])

python custom_evaluator.py

[('NoGuaranteesEvaluator', False, 'RETURN_GUARANTEED')]

Get started

Journeys

Concepts

Guides

Reference

Evaluator library

Deterministic

LLM judges

Statistical model evaluators

Writing your own

​Deterministic

​LLM judges

​Statistical model evaluators

​Writing your own

Deterministic

LLM judges

Statistical model evaluators

Writing your own