Integrations

An integration is a tool: one package for one external system. It reports what it observes (assets, lineage, runs, events, freshness, check results, health), and some tools also act on the system from rules. Tools never import each other or a plugin. See Concepts for the layers and Writing a tool to add one.

Seven integrations ship in this repository. Each has a page of its own.

The integrations#

Every package but SAP HANA's also carries the Python classifier Development Status :: 3 - Alpha, including the two whose README says Production; the SAP HANA package carries no status classifier. This table shows what each README says, and the classifier is the lower bar.

What needs an Enterprise module#

Each integration page says plainly what a pure open-source install does and what does not work without an Enterprise module (Nistor Enterprise).

Install and activate#

Install the packages#

Install the CLI and core, then one package for each tool you use:

pip install nistor-core nistor-cli nistor-tool-gcp nistor-tool-gitlab

The Google Cloud package has extras for the Google clients it uses: bigquery, storage, dataform, dataplex, resourcemanager, recommender, monitoring, cloudsql and all. The MicroStrategy package has the extra sdk (see MicroStrategy).

The runtime image is not built by hand. It is built from a lock of exactly the packages a deployment runs.

Activate the tools#

Installing a package is not enough. A deployment's distribution file lists the tools that are active, and a tool that is installed but not listed is not loaded:

[activation]
tools = ["gcp", "gitlab", "odcs"]
plugins = []

[packages]
extras = { nistor-tool-gcp = ["dataplex"] }

The open-source distribution (distributions/oss/distribution.toml) activates all of them: castor, dummy, gcp, gitlab, google_workspace, microstrategy, odcs, sap_hana and sifflet. A deployment of your own lists the ones it wants. The example deployment repository shows the files and the commands: nistor dist lock writes the lock, and nistor dist build builds the runtime image from it. The image carries its activation set in /etc/nistor/distribution.json.

A run from source, with NISTOR_DEV=1 and no such file, activates everything that is installed. See the Quickstart.

Create a connection#

A connection is a configured instance of a tool's connection type: some configuration and a reference to its secrets. Each integration except SAP HANA has a connection type, and its id is the tool's (gcp, microstrategy, sifflet, castor, gitlab, google_workspace, odcs).

• Where: the Connections page of the web shell (/admin/connections), or POST /api/v1/connections with name, tool_type, config, secrets and optionally is_default. Listing and reading need connections:view, and creating, changing, testing and rotating need connections:manage.
• Secrets: you send a secret value once. Nistor writes it to Secret Manager when it can reach it, or, in a local run, keeps it as NISTOR_CONN_SECRET__<id> in the process environment, and stores only a reference. The connection API lists a secret's key and age, not its value. Rotating a secret needs the connection's name as confirmation.
• Test: runs the tool's health probe, which never raises. A scheduled sweep repeats it every cadence_minutes (default 60; 0 means manual only; otherwise 15 to 1440, and an interval under 30 needs an administrator).
• Default: one connection of a tool type can be the default. A rule or node that names none uses it.

See Configuration for the rest.

A connection is not always what a tool reads. Several tools still read environment variables for their work, and use the connection only for the test. Set both:

Tell the compiler which tools to run#

The compiler runs the tools that the environment's compiler configuration lists, each with its own settings. Three tools have a discovery step: gcp, microstrategy and gitlab. Save the configuration with nistor compiler sync or PUT /api/v1/compiler-config/ (needs compiler:trigger):

[
  {
    "environment": "prod",
    "tools": [
      { "id": "gcp", "config": { "connection_id": "<connection-uuid>", "dataprojects": ["acme-data-prod"] } }
    ]
  }
]

The backend also reads this list to decide which tools are active in an environment, for the GitLab features. The other tools need no entry. Each page shows its own entry.

Pollers#

Many integrations also run pollers. A tick poller starts itself on its host at its interval: Dataform runs and MicroStrategy jobs started outside Nistor are polled every 60 seconds, and a GitOps pull follows its connection's interval. A batch poller runs when a scheduling rule of Nistor's own fires it, or when someone uses "Run now". Nothing in this repository creates those rules. The Google Workspace poll, the MicroStrategy Platform Analytics read and the Sifflet ingest are batch pollers; the platform-jobs plugin shows the rule shape.

Executors that ship without a tool#

Some actions need no external system, so they are not part of a tool. run_chain runs several actions in order, and it is part of the core. custom_python runs trusted Python in a sandbox, from the package nistor-plugin-custom-python. The dummy tool's dummy_run succeeds, fails or waits for up to a minute, so that you can try a rule without touching a system. A fourth, in nistor-plugin-platform-jobs, runs Nistor's own maintenance jobs.

Read next#

• Write a rule that runs one of these actions.
• Lineage and impact for what the graph does with the nodes these tools report.
• Writing a tool to integrate a system that is not listed.