> ## Documentation Index > Fetch the complete documentation index at: https://docs.sengol.io/llms.txt > Use this file to discover all available pages before exploring further. # Gate releases in CI > Run a golden dataset through the gate on every pull request and fail the build on a regression. `sengol gate` runs your evaluators over a dataset, signs one evaluation-run record, and tells CI whether the agent passed. The run record carries a signed result for each row. ## 1. Write a golden dataset One JSON object per line. Each row needs `prompt` (or `input`) and at least one of `response` or `expected`. All rows must name the same agent and version. ```json golden.jsonl theme={null} {"prompt": "Management fee on Fund X?", "response": "0.42%", "expected": "0.42%", "agent_id": "fund-advisor", "agent_version": "1.1.0"} {"prompt": "Management fee on Fund Y?", "response": "0.55%", "expected": "0.55%", "agent_id": "fund-advisor", "agent_version": "1.1.0"} {"prompt": "Who manages Fund X?", "response": "Acme Asset Management", "expected": "Acme Asset Management", "agent_id": "fund-advisor", "agent_version": "1.1.0"} ``` Optional keys: `policies`, `context_docs` (grounding text for LLM judges), `id`, and `metadata`. ## 2. Configure the suite A dataset run must not contain a `sample_input` block: ```yaml sengol.yaml theme={null} agent: agent_id: fund-advisor agent_version: 1.1.0 suite: name: release-gate evaluators: - ExactMatchEvaluator - PIIEvaluator ``` Evaluators are listed by class name and built with default settings. To pass constructor arguments, use `- name: ` with an `args:` mapping. Each name may appear once. To attach LLM judges and thresholds, point `suite.controlbook` at a [ControlBook](https://docs.sengol.io/concepts/controlbook.md). ## 3. Run it locally ```bash theme={null} sengol gate --config sengol.yaml --dataset-path golden.jsonl \ --fail-on-gate-failure --output junit > eval-results.xml ``` ```text theme={null} run_id=01a10cb1-... gate_passed=True pass_rate=100.00% (3/3) ``` The first run registers the dataset by its SHA-256. Later runs with the same content reuse that registration. Add a regression and the gate fails: ```bash theme={null} echo '{"prompt": "Fee on Fund Z?", "response": "1.10%", "expected": "0.90%", "agent_id": "fund-advisor", "agent_version": "1.1.0"}' >> golden.jsonl ``` ```bash theme={null} sengol gate --config sengol.yaml --dataset-path golden.jsonl --fail-on-gate-failure ``` ```text theme={null} run_id=01a10cb2-... gate_passed=False pass_rate=75.00% (3/4) ``` ### How the verdict is decided * **With a ControlBook**, its thresholds decide. * **Without one**, the pass rate must be at least 95%. ### Exit codes | Code | Meaning | | - | - | | `0` | Gate passed. **Also returned on a failed gate unless you pass `--fail-on-gate-failure`.** | | `1` | Gate failed, with `--fail-on-gate-failure` or `--promote` | | `2` | Bad config, dataset, or unknown evaluator | | `3` | An evaluator raised an exception | | `4` | The appliance refused the run, or a case could not be signed | | `5` | Upload retries ran out. Resume within 24 hours with `sengol eval resume ` | ### Useful flags | Flag | Effect | | - | - | | `--output junit` | JUnit XML on stdout | | `--output github-summary` | Markdown table in `$GITHUB_STEP_SUMMARY`, plus a sticky PR comment | | `--output azure-pipelines` | Azure DevOps logging commands | | `--repeat N --min-pass M` | Run LLM evaluators N times; M must pass | | `--promote` | On pass, write a promotion record to the production chain | ## 4. Run it in GitHub Actions Use [`sengol-io/sengol-github-action`](https://github.com/sengol-io/sengol-github-action). It installs `sengol` with `pip`, runs the gate, and posts a sticky comment on the pull request. ```yaml .github/workflows/sengol.yml theme={null} name: sengol on: pull_request jobs: gate: runs-on: ubuntu-latest permissions: contents: read pull-requests: write # sticky PR comment id-token: write # OIDC exchange instead of a stored token steps: - uses: actions/checkout@v4 - uses: sengol-io/sengol-github-action@2.0.0 with: config: sengol.yaml dataset-path: golden.jsonl env: SENGOL_API_URL: ${{ vars.SENGOL_API_URL }} SENGOL_TENANT_ID: ${{ vars.SENGOL_TENANT_ID }} SENGOL_API_TOKEN: ${{ secrets.SENGOL_API_TOKEN }} # fallback if OIDC is unavailable ``` * The runner must reach your appliance and a package index that serves `sengol`. Set `PIP_INDEX_URL` or `PIP_EXTRA_INDEX_URL` if you host it privately. * `fail-on-gate-failure` defaults to `true` in the action. * Outputs: `passed` (`"true"`/`"false"`), `score`, `report_url`, `regression`. * **OIDC:** the appliance must set `SENGOL_GITHUB_OIDC_AUDIENCE` (for example `sengol`). Without it the exchange returns `503` and the action falls back to `SENGOL_API_TOKEN`. Every exchange is logged with the repository, ref, commit and run id. * **Allowlist (from the 2026-10-07 sengol release):** set `SENGOL_CI_OIDC_ALLOWED_REPOSITORIES` to the repositories that may exchange, for example `acme/service` or `acme/*`. Unset or empty admits none, and the exchange returns `401`. Earlier releases ignore the variable. * LLM judges from a ControlBook need `ANTHROPIC_API_KEY` in the step env. Without it they are skipped. ## Other CI systems The gate is a CLI command. In any CI: 1. Install `sengol`. 2. Set `SENGOL_API_URL`, `SENGOL_TENANT_ID` and `SENGOL_API_TOKEN`. 3. Run `sengol gate --config sengol.yaml --dataset-path golden.jsonl --fail-on-gate-failure --output junit > eval-results.xml`. 4. Publish `eval-results.xml` as a test report. For OIDC from other CI providers, the appliance lists trusted issuers in `SENGOL_CI_OIDC_ISSUERS` and the job sets `SENGOL_CI_OIDC_TOKEN_URL`. The repository allowlist applies to these issuers too, and an unprefixed entry covers GitHub Actions only. Add each repository or project as `=`, for example `https://issuer.example=group/project`. An issuer whose token has no repository claim is matched on the token's `sub`, exactly. Without an entry the exchange returns `401`. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.