> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sengol.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluator library

> Every built-in evaluator, what it checks, and the failure mode it reports.

Every evaluator returns a pass or a fail. A fail carries one named
`failure_mode`. Import evaluators from `sengol.evaluators`, or list them by
class name in `sengol.yaml` and ControlBooks.

List them, filtered by policy, from the CLI:

```bash theme={null}
sengol eval list --policy PIPEDA
```

```text theme={null}
NAME                       VERSION  CATEGORY       POLICIES                    FAILURE_MODE
-------------------------  -------  -------------  --------------------------  ---------------------------
PIIEvaluator               1.0.0    DETERMINISTIC  OSFI_E23,FINTRAC_6F,PIPEDA  PII_DETECTED
PIPEDAIdentifierEvaluator  1.0.0    DETERMINISTIC  PIPEDA                      PIPEDA_IDENTIFIER_DISCLOSED
PIILeakEvaluator           1.0.0    LLM            OSFI_E23,PIPEDA             PII_LEAK_DETECTED
```

> [!NOTE]
>
> A bare class name in `sengol.yaml` or a ControlBook builds the evaluator
> with **default arguments**. In `sengol.yaml` you can pass arguments with
> `- name: KeywordEvaluator` and an `args:` mapping. With no phrases,
> `RequiredDisclosureEvaluator` fails every input.

## Deterministic

No model and no network. They run first, in order. A CRITICAL failure skips
the LLM evaluators that follow.

| Evaluator | Arguments | Failure mode | Severity |
| - | - | - | - |
| `PIIEvaluator` | `judge=None` (optional second opinion) | `PII_DETECTED`: SIN (Luhn-checked), SSN, card number, email, Canadian postal code, date of birth | CRITICAL |
| `PIPEDAIdentifierEvaluator` | none | `PIPEDA_IDENTIFIER_DISCLOSED` | CRITICAL |
| `RequiredDisclosureEvaluator` | `required_phrases`, `case_sensitive=False` | `REQUIRED_DISCLOSURE_MISSING` | CRITICAL |
| `FINTRACTypologyEvaluator` | none | `AML_TYPOLOGY_INDICATOR_UNFLAGGED` | CRITICAL |
| `ForbiddenToolEvaluator` | `forbidden` (tool names) | `FORBIDDEN_TOOL_INVOKED` | CRITICAL |
| `MCPToolIntegrity` | none | `MCP_TOOL_DEFINITION_CHANGED` | CRITICAL |
| `UnauthorizedDelegation` | none | `UNAUTHORIZED_DELEGATION` | CRITICAL |
| `ExactMatchEvaluator` | `case_sensitive=False` | `EXACT_MATCH_FAILED` (also when `expected` is missing) | HIGH |
| `ResponseLengthEvaluator` | `min_length=0`, `max_length=10000` (characters) | `RESPONSE_TOO_SHORT`, `RESPONSE_TOO_LONG` | HIGH |
| `KeywordEvaluator` | `required_keywords`, `forbidden_keywords` | `REQUIRED_KEYWORD_MISSING`, `FORBIDDEN_KEYWORD_PRESENT` | HIGH |
| `RegexEvaluator` | `patterns`, `require_all=True` | `REGEX_PATTERN_NOT_MATCHED` | HIGH |
| `SchemaValidationEvaluator` | `schema` (JSON Schema) | `SCHEMA_VALIDATION_FAILED` | HIGH |
| `DriftMonitorPresent` | none; reads `metadata["drift_monitor_enabled"]` | `DRIFT_MONITOR_NOT_CONFIGURED` | HIGH |
| `ToolCallMatchEvaluator` | `expected` (tool names), `mode="exact"` | `TOOL_CALL_MISMATCH` | HIGH |
| `ToolArgSchemaEvaluator` | `schemas` (per tool) | `TOOL_ARG_SCHEMA_VIOLATION` | HIGH |

The tool, MCP and delegation evaluators take trajectory or delegation
inputs, not a plain prompt and response.

## LLM judges

Each takes `judge=` (see [Add an LLM judge](https://docs.sengol.io/journeys/ai-developer.md#3-add-an-llm-judge)).
Without a judge, the evaluator scores `LLM_EVALUATOR_ERROR`. A judge that
exceeds its severity timeout scores `JUDGE_TIMEOUT`.

| Evaluator | Failure mode | Needs |
| - | - | - |
| `FaithfulnessEvaluator` | `FAITHFULNESS_FAIL` | `context` |
| `HallucinationEvaluator` | `HALLUCINATION_DETECTED` | |
| `GroundednessEvaluator` | `GROUNDEDNESS_FAIL` | `context`, and a judge that answers in JSON |
| `RelevanceEvaluator` | `RELEVANCE_FAIL` | |
| `TaskAdherenceEvaluator` | `TASK_ADHERENCE_FAIL` | |
| `IntentResolutionEvaluator` | `INTENT_UNRESOLVED` | |
| `RefusalEvaluator` | `REFUSAL_MISMATCH` | |
| `ToxicityEvaluator` | `TOXICITY_DETECTED` | |
| `ExplainabilityEvaluator` | `NO_EXPLANATION_PROVIDED` | |
| `AdversarialRobustnessEvaluator` | `ADVERSARIAL_MANIPULATION_SUCCEEDED` | |
| `GDPRDataSubjectRightsEvaluator` | `GDPR_DATA_SUBJECT_RIGHT_MISHANDLED` | |
| `PIILeakEvaluator` | `PII_LEAK_DETECTED` | Deprecated: use `PIIEvaluator` |
| `RedTeamEvaluator` | `RED_TEAM_PROBE_BREACH` | `sengol[redteam]` |

> [!TIP]
>
> In our tests with `OpenAICompatibleJudgeClient(model="gpt-4o-mini")`, the
> judge answered `GroundednessEvaluator` in prose, which it scores as
> `LLM_EVALUATOR_ERROR`. `FaithfulnessEvaluator` and
> `HallucinationEvaluator` worked with the same client.

`EnsembleEvaluator` combines several judges and reports
`ENSEMBLE_MEMBER_FAILED`.

## Statistical model evaluators

For traditional ML models. Install `sengol[model]`:
`AUCEvaluator`, `PSIEvaluator`, `ABEvaluator`, `ChampionChallengerEvaluator`,
`FairnessEvaluator`, `OutcomeMonitorEvaluator`. They take a
`ModelEvalInput` (features, prediction, ground truth) instead of a prompt
and response.

## Writing your own

Subclass `DeterministicEvaluator` or `LLMEvaluator` and declare a
`failure_mode`:

```python custom_evaluator.py theme={null}
import asyncio
from typing import ClassVar

from sengol.build.suite import EvalSuite
from sengol.core.evaluator import DeterministicEvaluator
from sengol.core.types import EvalScore, LLMEvalInput
from sengol.governance.audit_store import NullAuditStore


class NoGuaranteesEvaluator(DeterministicEvaluator):
    name: ClassVar[str] = "NoGuaranteesEvaluator"
    failure_mode: ClassVar[str] = "RETURN_GUARANTEED"
    severity: ClassVar[str] = "CRITICAL"

    def evaluate(self, input: LLMEvalInput) -> EvalScore:
        promised = "guaranteed return" in input.response.lower()
        return EvalScore(
            evaluator=self.name,
            passed=not promised,
            failure_mode=self.failure_mode if promised else None,
            reason="Response promises a guaranteed return." if promised else "",
        )


suite = EvalSuite([NoGuaranteesEvaluator()], audit_store=NullAuditStore(), tenant_id="local")
result, _ = asyncio.run(suite.run(LLMEvalInput(
    agent_id="fund-advisor", agent_version="1.0.0",
    prompt="Is Fund X safe?", response="Fund X has a guaranteed return of 8%.",
)))
print([(s.evaluator, s.passed, s.failure_mode) for s in result.scores])
```

```bash theme={null}
python custom_evaluator.py
```

```text theme={null}
[('NoGuaranteesEvaluator', False, 'RETURN_GUARANTEED')]
```


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.