Incidents

An incident is a durable record that something failed: a rule's action, a compilation, a data contract, or a promise about freshness. It carries a severity, the nodes it affects, a history of what happened to it, and who is working on it. Nistor opens and closes most incidents itself. A person accepts, acknowledges, assigns, snoozes or resolves them.

This page covers how incidents open, how related failures are grouped, how Nistor suggests what caused one, and how a destructive operation can be undone. It builds on Rules and scheduling, because most incidents start with a rule's run, and on Lineage and impact, because grouping and blast radius follow the graph.

What an incident holds#

Nistor advises the severity once, when the incident opens. It weighs the environment (production weighs most), the kind of failure (a permission gap weighs less than a failed run), how critical the node is (its criticality: or tier: label), how many nodes sit downstream, and, a little, how often the run was retried. An incident raised inside a maintenance window is marked as not counting towards time to acknowledge and time to resolve.

How incidents open#

Enterprise modules declare kinds of their own: data_quality from the Data quality module, budget_exceeded and cost_anomaly from FinOps, and ai_feedback_spike from the AI module. GET /api/v1/incidents/kinds lists the kinds registered in your deployment.

Many detectors are off by default, because a detector that is wrong for your estate can open thousands of incidents at once. Where a detector has a dry-run endpoint, run it before you turn the detector on: it reports what the detector would open and writes nothing. Those endpoints accept only service-account callers. The settings are environment variables of the backend.

A rule's freshness_expectation has no flag of its own: the freshness evaluator opens its delivery_slo incident whenever the deadline passes without a fresh target.

One incident per failing thing#

With the generic lifecycle on (the default), an action_run incident is identified by the action type and the node. The first failed run opens it. Each further failure of the same pair adds to failure_count instead of opening a second incident. A later success closes it, whichever part of Nistor saw that success: a refresh that the scheduler ran and a refresh that a poller saw happen outside Nistor close the same incident.

Some failures never open one. A run of a rule in tuning mode (shadow_mode), and a test run, open none. When one Dataform invocation fails, only the actions that really failed carry the incident. An action that Dataform cancelled because a sibling failed, or because the invocation timed out, is collateral and does not.

Class and tier#

Nistor sorts each incident once, when it opens, and stores the result:

page means someone should be woken, ticket means it belongs in the backlog, and log means record it only. The tier is advice. Your alert policies decide who is notified.

Grouping around a root cause#

Three mechanisms keep one cause from showing up as fifty incidents.

Read-time groups. The console collapses incidents that share a dedup_fingerprint into one group with a member count. A critical incident is never merged into a group.

Root-cause clusters. Nistor normalises the failure text of an incident and hashes it into an error_signature. The normalisation replaces identifiers, timestamps, dates and numbers with placeholders and, when a failure message joins one reason per table, keeps only the first reason. With the source_project, the signature groups the open and acknowledged incidents that share a cause. GET /api/v1/incidents/{id}/root-cause returns the cluster size, the oldest incident in it and the others. The incident pane shows a "Root Cause cluster" card when the cluster has more than one member.

Folding under an upstream incident. When an ancestor within four hops of a node already has an open incident, Nistor folds the node's new incident under it: the same failure, seen one step later. The child is recorded with a root_incident_id, and the console shows it under its root. When the root closes, the link is cleared and the child shows on its own again. Lineage is not causality, so some incidents never fold: a configuration incident or an extraction failure is never caused upstream, and a data incident (data_quality, data_contract or delivery_slo) only explains other data incidents.

What changed?#

An incident pane can show the changes that happened before it, ranked by how likely each is to be the cause. The console calls the panel "What changed?" and the API calls the feature attribution. Nistor ranks candidates. It does not state a cause.

The feature is off by default. Set DATA_GIT_BLAME_ENABLED to true; until you do, every route answers 404. Reading it needs the attribution:view permission, which the viewer role template includes.

For an incident, Nistor takes the failing nodes and their ancestors (up to ATTRIBUTION_MAX_ANCESTOR_DEPTH, 6 hops) and collects the changes in the window before the incident (ATTRIBUTION_WINDOW_DAYS, 14 days) from four sources: node definition changes, deployments (the Git commit a node was deployed from), rule changes, and GitOps syncs. It scores each change:

A score of 0.65 or more is high confidence, 0.40 or more is medium, and the rest is low. The routes, under /api/v1/attribution, are GET /incidents/{id} (the ranked changes), GET /commits/{sha} (the incidents that opened downstream of a deploy, within window_days) and GET /timeline?node_id= (one node's changes, incidents and cost; the cost needs the FinOps module). ATTRIBUTION_CACHE_TTL_SECONDS (60) caches each answer.

Working an incident#

Reading incidents needs no special permission. Changing one needs alerts:manage, which the oncall-sre and tenant-admin role templates include and data-engineer does not.

• Accept and decline. A machine-created incident starts in the triage stage. Accepting it moves it to active and starts the clock for time to acknowledge. Declining it closes it as not an incident, and it does not count. An incident that recovers by itself while still in triage is normally declined by the system rather than resolved, for the same reason.
• Acknowledge, resolve, close, reopen. These move status. Each is recorded in the incident's history.
• Snooze. Quiet one incident, or a whole group.
• Mark as expected. Flags the incident as a known issue and saves a suppression rule for its source. With INCIDENT_SUPPRESSION_GATE_ENABLED on (it is off by default), an incident that an active permanent rule matches is never created, and the failure is counted on the rule (suppressed_count) instead.
• Maintenance windows. A window with alert suppression turns the incidents that would open inside it into expected records. Windows are managed under /api/v1/maintenance_windows; changing one needs maintenance:manage.

Unacknowledged incidents that have a team are nudged in the team's channel, after 48 hours by default (INCIDENT_ESCALATION_HOURS_CRITICAL, INCIDENT_ESCALATION_HOURS_HIGH). INCIDENT_ESCALATION_FALLBACK_CHANNEL_ID adds a second channel after INCIDENT_ESCALATION_FALLBACK_AFTER_NUDGES (2) nudges. In a deployment from the reference Terraform, a scheduled janitor (POST /api/v1/alerts/check-incident-aging) closes configuration incidents that have not failed for 7 days and any incident that has not failed for 30 days (a snoozed incident is exempt from the 30-day close), and it alerts on incidents older than 14 days. An incident that keeps failing is never closed this way.

Undo#

Some operations destroy something. Nistor routes them through reversible operations, a time-boxed, audited undo. The operation runs in five steps: capture a recovery artifact, ask for approval if the operation requires it, perform the change, record it, and later restore from the artifact if you ask. Each operation type has a reverser that knows how to capture, restore and purge its artifact. A recorded operation is in one of these states: pending_approval, executed, restored, expired, execute_failed or restore_failed. A failed restore keeps its artifact and can be tried again while the time box holds. When the time box lapses, a sweep purges the artifact and marks the operation expired.

In this repository, deleting an action run is the operation the backend guards itself. The other reversers serve callers that carry out proposals through the same guard, such as the AI module's investigation agent and the Advisor module. None of the reversers in this repository requires approval; a reverser can require it always, or above a threshold it sets. Approval is a two-person rule: the approver must differ from the requester, and a pending request expires after 24 hours.

Reading the log needs reversible:view, restoring needs reversible:restore, and approving a gated operation needs reversible:approve. Of the role templates, only tenant-admin (which holds every permission) includes them, so an administrator grants them to anyone else. The log is at GET /api/v1/reversible-operations/, a restore is POST /api/v1/reversible-operations/{op_id}/restore (409 if it was already restored, 410 if the time box has lapsed), and the approval gate is under /api/v1/operation-approvals. The kill switch is the backend environment variable REVERSIBLE_OPS_ENABLED. It is on by default; set it to false and a guarded operation runs with no capture and no record.

What needs what#

The modules are part of Nistor Enterprise. For the settings, see Configuration.