Gate releases in CI

sengol gate runs your evaluators over a dataset, signs one evaluation-run record, and tells CI whether the agent passed. The run record carries a signed result for each row.

1. Write a golden dataset

One JSON object per line. Each row needs prompt (or input) and at least one of response or expected. All rows must name the same agent and version.

golden.jsonl

{"prompt": "Management fee on Fund X?", "response": "0.42%", "expected": "0.42%", "agent_id": "fund-advisor", "agent_version": "1.1.0"}
{"prompt": "Management fee on Fund Y?", "response": "0.55%", "expected": "0.55%", "agent_id": "fund-advisor", "agent_version": "1.1.0"}
{"prompt": "Who manages Fund X?", "response": "Acme Asset Management", "expected": "Acme Asset Management", "agent_id": "fund-advisor", "agent_version": "1.1.0"}

Optional keys: policies, context_docs (grounding text for LLM judges), id, and metadata.

2. Configure the suite

A dataset run must not contain a sample_input block:

sengol.yaml

agent:
  agent_id: fund-advisor
  agent_version: 1.1.0

suite:
  name: release-gate
  evaluators:
    - ExactMatchEvaluator
    - PIIEvaluator

Evaluators are listed by class name and built with default settings. To pass constructor arguments, use - name: <Class> with an args: mapping. Each name may appear once. To attach LLM judges and thresholds, point suite.controlbook at a ControlBook.

3. Run it locally

sengol gate --config sengol.yaml --dataset-path golden.jsonl \
  --fail-on-gate-failure --output junit > eval-results.xml

run_id=01a10cb1-... gate_passed=True pass_rate=100.00% (3/3)

The first run registers the dataset by its SHA-256. Later runs with the same content reuse that registration. Add a regression and the gate fails:

echo '{"prompt": "Fee on Fund Z?", "response": "1.10%", "expected": "0.90%", "agent_id": "fund-advisor", "agent_version": "1.1.0"}' >> golden.jsonl

sengol gate --config sengol.yaml --dataset-path golden.jsonl --fail-on-gate-failure

run_id=01a10cb2-... gate_passed=False pass_rate=75.00% (3/4)

How the verdict is decided

With a ControlBook, its thresholds decide.
Without one, the pass rate must be at least 95%.

Exit codes

Code	Meaning
`0`	Gate passed. Also returned on a failed gate unless you pass `--fail-on-gate-failure`.
`1`	Gate failed, with `--fail-on-gate-failure` or `--promote`
`2`	Bad config, dataset, or unknown evaluator
`3`	An evaluator raised an exception
`4`	The appliance refused the run, or a case could not be signed
`5`	Upload retries ran out. Resume within 24 hours with `sengol eval resume <run_id>`

Useful flags

Flag	Effect
`--output junit`	JUnit XML on stdout
`--output github-summary`	Markdown table in `$GITHUB_STEP_SUMMARY`, plus a sticky PR comment
`--output azure-pipelines`	Azure DevOps logging commands
`--repeat N --min-pass M`	Run LLM evaluators N times; M must pass
`--promote`	On pass, write a promotion record to the production chain

4. Run it in GitHub Actions

Use sengol-io/sengol-github-action. It installs sengol with pip, runs the gate, and posts a sticky comment on the pull request.

.github/workflows/sengol.yml

name: sengol
on: pull_request

jobs:
  gate:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write   # sticky PR comment
      id-token: write        # OIDC exchange instead of a stored token
    steps:
      - uses: actions/checkout@v4
      - uses: sengol-io/sengol-github-action@2.0.0
        with:
          config: sengol.yaml
          dataset-path: golden.jsonl
        env:
          SENGOL_API_URL: ${{ vars.SENGOL_API_URL }}
          SENGOL_TENANT_ID: ${{ vars.SENGOL_TENANT_ID }}
          SENGOL_API_TOKEN: ${{ secrets.SENGOL_API_TOKEN }}  # fallback if OIDC is unavailable

The runner must reach your appliance and a package index that serves sengol. Set PIP_INDEX_URL or PIP_EXTRA_INDEX_URL if you host it privately.
fail-on-gate-failure defaults to true in the action.
Outputs: passed ("true"/"false"), score, report_url, regression.
OIDC: the appliance must set SENGOL_GITHUB_OIDC_AUDIENCE (for example sengol). Without it the exchange returns 503 and the action falls back to SENGOL_API_TOKEN. Every exchange is logged with the repository, ref, commit and run id.
Allowlist (from the 2026-10-07 sengol release): set SENGOL_CI_OIDC_ALLOWED_REPOSITORIES to the repositories that may exchange, for example acme/service or acme/*. Unset or empty admits none, and the exchange returns 401. Earlier releases ignore the variable.
LLM judges from a ControlBook need ANTHROPIC_API_KEY in the step env. Without it they are skipped.

Other CI systems

The gate is a CLI command. In any CI:

Install sengol.
Set SENGOL_API_URL, SENGOL_TENANT_ID and SENGOL_API_TOKEN.
Run sengol gate --config sengol.yaml --dataset-path golden.jsonl --fail-on-gate-failure --output junit > eval-results.xml.
Publish eval-results.xml as a test report.

For OIDC from other CI providers, the appliance lists trusted issuers in SENGOL_CI_OIDC_ISSUERS and the job sets SENGOL_CI_OIDC_TOKEN_URL. The repository allowlist applies to these issuers too, and an unprefixed entry covers GitHub Actions only. Add each repository or project as <issuer>=<repository>, for example https://issuer.example=group/project. An issuer whose token has no repository claim is matched on the token’s sub, exactly. Without an entry the exchange returns 401.

Get started

Journeys

Concepts

Guides

Reference

Gate releases in CI

1. Write a golden dataset

2. Configure the suite

3. Run it locally

How the verdict is decided

Exit codes

Useful flags

4. Run it in GitHub Actions

Other CI systems

​1. Write a golden dataset

​2. Configure the suite

​3. Run it locally

​How the verdict is decided

​Exit codes

​Useful flags

​4. Run it in GitHub Actions

​Other CI systems

1. Write a golden dataset

2. Configure the suite

3. Run it locally

How the verdict is decided

Exit codes

Useful flags

4. Run it in GitHub Actions

Other CI systems