Sifflet

Sifflet is a hosted data-observability service. The Sifflet tool (nistor-tool-sifflet, tool id sifflet) connects Nistor to a Sifflet tenant, so that the monitors you already run appear beside your lineage, runs and incidents.

The package carries the Alpha classifier and its README states no other status (see the overview).

What needs an Enterprise module#

The pull from Sifflet is not in the open-source tool. The ingester that reads rules, rule runs and asset health, mirrors them as checks, and opens and closes Nistor incidents from them, is part of the Data quality module (Nistor Enterprise). So is the migration provider that proposes Nistor checks from Sifflet rules, and the action that records a verdict.

Nothing in this repository starts a Sifflet rule run. The tool only reads from Sifflet, except for the verdict write-back below.

Create the connection#

Create a connection of type sifflet in the web shell (Connections, /admin/connections) or with POST /api/v1/connections:

The test sends one authenticated request, GET /api/v1/rules?page=0&itemsPerPage=1, and waits up to 8 seconds. It answers healthy only when Sifflet returns the rules listing. Everything else is unhealthy, with the cause: a refused token (401 or 403), a wrong address (404, or a reply that is not a Sifflet tenant), throttling (429), a certificate that is not trusted, a host that does not resolve, or a timeout. The probe never answers degraded.

Where credentials are read today#

Today only the test reads the connection. The ingest, the rule reads and the write-back read these environment variables:

Set them on the backend and jobs roles. The reference Terraform root does it for you when you set sifflet_base_url: it creates the Secret Manager secret sifflet-api-token-<env>, gives the backend and jobs accounts access to it, and sets SIFFLET_BASE_URL and SIFFLET_API_TOKEN on both roles. You add the token yourself, so that it never appears in a variable file or in the state:

gcloud secrets versions add sifflet-api-token-prod --data-file=<token-file>

An empty sifflet_base_url switches the integration off: no secret, no variable, no ingest. The root does not wire a write token.

What the ingester stores#

With the Data quality module installed, the daily ingest fills five tables in the tool_sifflet schema: rules, the datasets each rule watches, asset health, rule runs, and an ingest log. A dataset is matched to a graph node by its Sifflet bigquery: address, which becomes a BigqueryTable node, so a failing monitor shows on the table it watches. The tool is written for a BigQuery warehouse. The ingest also:

• re-checks every open Sifflet incident against the rule's current state, closes the ones that recovered, and flags rules that stopped reporting (SIFFLET_INCIDENT_SWEEP_ENABLED);
• projects the owners that Sifflet assigns to an asset into the catalog's ownership tables (SIFFLET_OWNER_PROJECTION_ENABLED), which also needs the Catalog module.

The open-source routes add one estimate of their own. Sifflet runs its rules in its own Google Cloud project, so Nistor sees only a lightweight read event for them, with no billed bytes. GET /api/v1/sifflet/cost-estimate multiplies the number of reads by the median billed bytes of reads on the same tables and by the BigQuery on-demand price. It is labelled an estimate. Nistor finds Sifflet's reads by the account that matches SIFFLET_BQ_PRINCIPAL_PATTERN.

Write verdicts back#

When a person qualifies a Sifflet-mirrored point in Nistor, as a real error or as acceptable, Nistor can send the verdict to the incident in Sifflet so that its anomaly detection learns from it. This is off by default (SIFFLET_WRITE_ENABLED=false).

• It needs an Editor or Admin token, in SIFFLET_WRITE_API_TOKEN or, if that is empty, in SIFFLET_API_TOKEN. A Viewer token is refused by Sifflet's write endpoints.
• It calls POST /api/v1/incidents/{issue}/qualification with one of FALSE_POSITIVE, EXPECTED, FIXED or QUALIFIED.
• It never undoes the Nistor verdict. If the call fails, the verdict and the closing of the Nistor incident stand.
• The code notes that the exact path and body have not been confirmed against a tenant with a write-capable token. Test it on a non-production tenant first.

Permissions on the Sifflet side#

In Nistor, reading Sifflet data needs dataquality:view, and the on-demand actions (sync-mirror, reconcile-runs, compute-metric-values) need dataquality:write.

Settings#