Rules and scheduling

A rule tells Nistor to act: for these assets, when something happens or at a set time, and if some conditions hold, run an action. Rules keep pipelines running without a person starting each job. Most runs start with a rule, and so do most incidents (see Incidents).

Scheduling is a capability named scheduler. An administrator can disable it with PATCH /api/v1/capabilities/scheduler and {"enabled": false}. While it is disabled, the backend refuses the rule, calendar, on-hold, maintenance-window, backfill and chain routes. Disabling takes effect within about 30 seconds; a capability that was disabled when the backend started comes back only after a restart. Runs, incidents, alerts and node impact analysis stay, because Nistor also records runs of work it did not schedule, such as a Dataform run started from outside.

Anatomy of a rule#

A rule has four blocks, read in this order:

Two optional blocks cover lateness: failsafe runs the rule if its trigger never came, and freshness_expectation tells a person when the target is late. Beyond the blocks, a rule has a name, a description, tags, a priority from 1 to 100 (1 by default) and a status.

• Status. An active rule runs for real. A test rule is evaluated, but its action is only simulated: it opens no incident. inactive, paused and archived rules do not fire. Nistor sets external itself, on read-only copies of rules that another tool owns.
• Selectors. node_ids and display_names are glob patterns (fnmatch); a pattern that also looks like a regular expression is tried as one. pause_node_ids patterns are globs only. {ENV} in a selector becomes the environment's name (dev, test or prod) when the rule is evaluated.
• Priority. When several rules match one node, most executors run only the highest-priority rule per executor type. then.target.highest_priority_only overrides that for one rule. The custom_python executor and the platform job executor run every matching rule by default.
• Tuning mode. shadow_mode: true keeps a rule running and recording its runs, but stops its runs from opening incidents. The console calls it tuning mode.

Example#

This rule rebuilds a daily sales mart once the tables it reads have settled, retries a failed run, and flags the mart if it is not fresh by 08:00 on weekdays. Its target is a node of the fictional acme estate in services/backend/nistor_backend/demo/data.json.

- name: Refresh the daily sales mart after the warehouse loads
  code_id: marts/mart-sales-daily        # the rule's stable identity when it is kept as a file
  description: Rebuilds mart_sales_daily once its source tables have settled.
  priority: 5
  status: active
  status_by_env:                         # file-only: the status per environment
    dev: paused
  tags:
    - marts
    - sales
  applies_to:
    node_type: DataformAction
    node_ids:
      - DataformAction_acme-mart-{ENV}.marts.mart_sales_daily
  when:
    events:
      - origin: ancestors                # events on the nodes this one reads from
        ancestor_depth: direct_materialized
        types:                           # matched exactly; list every write type you care about
          - BIGQUERY_INSERT
          - BIGQUERY_MERGE
    wait_for_all_ancestors: true         # fire once every source has produced a new event
    wait_for_all_ancestors_timeout_seconds: 5400
    wait_for_all_ancestors_timeout_alert: true
  if:
    quiet_period_seconds: 600            # wait for 10 quiet minutes before running
    max_runtime_seconds: 7200
    run_limit:
      max: 3
      period: day
  then:
    target:
      same_as_applies_to: true
      scope: node
    do:
      kind: single
      action:
        type: dataform_run
        parameters:
          release: projects/acme-mart-{ENV}/locations/europe-west1/repositories/acme-marts/releaseConfigs/daily
          retry:
            mode: exponential
            max_retries: 3
  failsafe:                              # acts on machines: run the rule if no trigger came
    enabled: true
    deadline_cron: "30 7 * * *"
    time_zone: Europe/London
  freshness_expectation:                 # acts on people: open an incident if still late
    days: weekdays
    before: "08:00"
    time_zone: Europe/London
    incident_severity: high

The acme names are fictional; use your own project and release configuration. The example passes nistor rules validate, which checks it against the rule schema for dev, test and prod.

What fires a rule#

Events#

An event listener sits under when.events. Its fields:

Event types include the BigQuery write types (BIGQUERY_INSERT, BIGQUERY_UPDATE, BIGQUERY_DELETE, BIGQUERY_MERGE, BIGQUERY_TRUNCATE, BIGQUERY_CREATE, BIGQUERY_DROP, BIGQUERY_WRITE and BIGQUERY_COPY), the completion of a Nistor-run action (ACTION_COMPLETED, or its alias DATAFORM_RUN_COMPLETED), the action lifecycle (ACTION_STARTED, ACTION_SUCCEEDED, ACTION_FAILED, ACTION_CANCELLED, ACTION_TO_RETRY, ACTION_RETRIED_SUCCEEDED and ACTION_RETRIED_FAILED), DQ_FAILURE and COMPILATION_FAILURE.

Audit-log events from your monitored projects arrive on Pub/Sub at the input-logs role. It matches them to nodes in the graph and forwards an event to the scheduler only when a rule listens for it. The scheduler evaluates the rule, applies its if block and dispatches the action.

wait_for_all_ancestors holds the rule until every direct ancestor has produced a new event since the last run. wait_for_all_ancestors_timeout_seconds makes it fire anyway after that long, with whatever is ready, and wait_for_all_ancestors_timeout_alert opens an incident naming the ancestors that were still missing.

Cron#

when.schedule runs a rule on time:

A rule can have events and a schedule at once.

How a firing runs#

A firing that has to wait, for a quiet period, a lock held by another run or a cron time, is stored as an on-hold action with a due time. A timer calls the scheduler role once a minute, and the scheduler claims the due actions and dispatches them. In production that timer is Cloud Scheduler, and the reference Terraform creates the jobs. The Docker Compose stack in the Quickstart has no such timer: its seeded cron rules are armed, but nothing fires them. You can still run a rule by hand there.

Conditions#

Actions#

then.target says what to act on. By default (same_as_applies_to: true) it is the matched nodes. Otherwise you select other nodes with node_type, node_ids, display_names or tags. scope is node, children, descendants, ancestors or all, relative to the matched node. group: true makes one invocation for all targets. exclusive: true stops other rules of the same executor from cascading onto the target nodes.

then.do is either one action (kind: single) or a chain (kind: chain) of steps. A step has a key, a type, optional parameters and node_ids, the steps it runs after, and on_failure (abort, the default, or continue).

A failed action is retried according to parameters.retry:

A rule's value overrides the executor's, and the defaults apply when neither sets a field. A value that does not validate is rejected when you save the rule. retry_number: 3 is the older spelling of max_retries: 3.

Executors in this repository#

An executor is a package that runs one kind of action. The scheduler finds installed executors through the nistor.executors entry-point group; GET /api/v1/rules_scheduling/executor-types lists them.

Nistor also ships an executor that runs its own maintenance jobs (ingests, polls, clean-up sweeps) as rules. Only holders of config:manage see those rules or can write one. Enterprise modules add executors of their own, such as data_quality_check from the Data quality module. See Writing a tool to add one.

Running an action by hand, and saving a rule that runs it, needs the executor's permission where it has one: actions:run_dataform, actions:run_microstrategy (both MicroStrategy actions), actions:run_bigquery_copy or actions:run_bigquery_export_gcs. The data-engineer role template holds the Dataform and BigQuery ones; actions:run_microstrategy is in no template but tenant-admin.

When a rule does not fire#

The two lateness blocks answer different questions, and you can use both.

Rules as code#

Keep rules in a repository and let CI send them to Nistor. A rules file is YAML or JSON. It holds a list of rules (YAML also accepts one rule). Every rule needs a code_id, and code_id and name must be unique across all files.

nistor rules validate --dir rules/
nistor rules sync --env prod --dir rules/ --dry-run
nistor rules sync --env prod --dir rules/ --delete-orphaned --source-repo your-group/your-rules

• nistor rules validate needs no API. It parses the files and checks duplicates and the schema for dev, test and prod. It also flags any key the schema does not know, which the backend would silently drop, so a typo such as quiet_period_secnds fails here.
• nistor rules sync takes --env dev|test|prod and --files or --dir. Useful options are --dry-run, --validate (server-side validation first), --require-rules-format (refuse a backend that would drop fields), --output-json and --continue-on-error. --delete-orphaned removes rules that are no longer in the files, scoped to --source-repo when you pass it. It needs NISTOR_API_URL and NISTOR_API_TOKEN, or the global options --api-url and --token placed before rules.
• nistor rules list shows the rules stored in the API.

See CLI for installing the command.

A synced rule is locked: its source is code, and the API refuses to edit or delete it (HTTP 403, "code-managed and locked"). Change the file instead. An emergency unlock exists: POST /api/v1/rules_scheduling/{id}/unlock. Every write to a rule through the API, unlock included, needs actions:trigger, which the data-engineer role template includes. Creating a rule, or an edit that can make it run, also needs the executor's permission above.

status_by_env lets one file set a different status per environment; a sync resolves it for the --env you pass. If you delete an optional block such as if from a rule in a file, the next sync clears it in Nistor: the sync sends an explicit empty value for every optional block that the file omits.

Rule Lab#

Rule Lab (/scheduler/rule-lab in the console) shows what a draft rule would do before you save it. Paste a rule as JSON, or load an existing one. Nothing is stored until you save. The page shows:

• Validation: schema errors, and whether the rule matches one that is already stored, in which case saving updates it.
• Blast radius: the nodes the rule would match, and for each the outcome of the priority contest with the rules that already hold it: uncontested, wins, loses or shares, and which rules compete. A rule that selects on tags alone never joins the contest, and the page says so. This is not the impact analysis of a node, described in Lineage and impact.
• Replay: when the rule would have fired over the last 7, 14, 30, 60 or 90 days, for up to three nodes, next to what ran. The page picks the busiest nodes the rule would win first.
• Cost and run-rate: the measured spend, over the same window, of the nodes the rule would win. It is spend already incurred, not a forecast, and it needs the FinOps module. Without it, the nodes are listed unpriced.

An array of rules opens a batch table. Validation, blast radius, replay and saving need actions:trigger. Save appears only when the rule is valid.

Alerts#

Alerts tell people. A channel is a destination. A policy says which events go to which channels.

Cards sent to gchat_space and gchat_dm channels carry Acknowledge, Snooze and Resolve buttons only when CHAT_SURFACE_GCHAT_AUDIENCE is set. The route that receives those clicks belongs to the AI module.

A policy has these fields:

Times in cards and emails are shown in ALERT_DISPLAY_TIMEZONE (Europe/Brussels by default); an unknown name falls back to UTC.

Keep channels and policies as code the same way as rules: nistor alerts sync --env prod --dir alerts/ reads alert_channels.json, alert_policies.json and email_settings.json. Entries carry a code_id, and a policy names its channels with channel_code_ids. A ${VARIABLE} anywhere in a file is replaced by that environment variable of the sync job, so a webhook URL stays out of Git. {ENV} becomes the environment name.

What needs what#

The modules are part of Nistor Enterprise. The flags are environment variables of the backend; see Configuration.