Lineage and impact — Nistor Docs

Lineage is the part of the graph that says how data moves: which table feeds which table, which column is computed from which columns, which report reads which cube. Nistor uses it to answer two questions. Where does this come from (upstream)? What depends on this (downstream)? The second question is impact analysis; the console calls it "Blast Radius".

Concepts explains how tools report what they observe and how the graph is built. This page goes deeper on the edges and on what Nistor computes from them.

Two levels#

Level	The graph records	You use it to
Asset	For each node, its ancestors and descendants. Processing nodes, such as Dataform actions, sit between the tables they read and the tables they write.	Trace a table back to its sources, or a report back to its cubes and tables.
Column	For each column of a node, its source columns and, when the SQL shows one, the expression that derives it.	Find which source columns a figure comes from, and which columns a change to one column reaches.

The compiler role builds the graph from what each tool discovers, and the backend serves the last graph it loaded. A change in a source system appears in lineage once the compiler has picked it up.

Where lineage comes from#

Tool	Reports
Google Cloud	Table lineage from view definitions and Dataform compilation results, column lineage parsed from SQL, and links between services (a Pub/Sub subscription to the service it calls, for example).
MicroStrategy	The relations between its objects (cubes, reports, dashboards, documents) and the tables behind them, and column lineage for cubes and reports, parsed from their SQL.

Column lineage is parsed from SQL, so it is only as complete as the parser's reading of that SQL. A column with no traced source has has_lineage set to false.

Read lineage#

A node's page in the console has a Lineage tab. The same data is available from the API, under /api/v1. Reading it needs a signed-in user and no other permission.

Route	Returns
`GET /dependencies/ancestors/{id}`, `GET /dependencies/descendants/{id}`	Every upstream or downstream node, with the processing nodes that connect them. Query parameters: `max_depth`, `exclude_patterns` (regular expressions on node ids), `include_scheduler_deps`.
`GET /dependencies/first_ancestors/{id}`, `GET /dependencies/first_descendants/{id}`	Direct neighbours only.
`GET /lineage/columns?node_id=`	Every column of a node, with a one-line derivation summary and its source columns.
`GET /lineage/column_field_group/{node_id}?column_name=`	The columns in other tables that hold the same field (see below).
`GET /lineage/erd/{node_id}`	The table and its direct neighbours as an entity-relationship bundle: columns, table edges and column edges.

Over MCP, nistor_get_lineage returns upstream and downstream lineage to an assistant; see MCP server.

Same-field columns#

A field is often copied across tables (SELECT customer_id) or renamed (customer_id AS cust_id). Nistor walks the column edges that are pure copies and groups the columns that hold the same field. A copy has exactly one source column and no operator.

With SAME_FIELD_RELAXED_PASSTHROUGH_ENABLED (on by default), wrappers that keep the value also count as copies: TRIM, CAST, UPPER, LOWER, COALESCE and their relatives. SAME_FIELD_PASSTHROUGH_KINDS selects which: all by default, or a comma-separated subset of whitespace, cast, case_fold and coalesce. A group holds at most 50 columns unless you pass max_group_size, up to 500.

Impact analysis#

GET /api/v1/nodes/{node_id}/impact walks the graph downstream from a node, breadth first, and reports what it reaches.

Parameter	Default	Meaning
`max_depth`	3 (1 to 20)	How many edges to follow.
`exclude_patterns`	none	Regular expressions on node ids. A matching node is left out, and the walk does not continue through it.
`include_scheduler_deps`	`false`	Also follow the links that rules create: a node reaches the nodes that the rules it triggers act on.
`since`	none	A timestamp. Adds write evidence from the lineage ledger (see Row-level evidence).

The response has:

Field	Contents
`affected_nodes`	Every downstream node, with its type, its distance from the origin (`depth`) and its console URL.
`rules`	The scheduling rules that target an affected node: executor type, priority, status, and which affected nodes they target.
`dq_checks`	The rules on affected nodes that run the `data_quality_check` executor. That executor comes with the Data quality module, so without it the list is empty.
`scheduled_actions`	Actions already queued against an affected node.
`summary`	The four counts.

Impact analysis follows edges in the graph and nothing else. A consumer that Nistor has no edge to, such as a notebook or a service that reads a table directly, is not in the answer.

Over MCP, nistor_impact_analysis calls this route.

Impact tier#

Nistor classifies a blast radius by the number of downstream nodes: critical from 20, high from 10, medium from 3, and low below that. The count says how wide a failure reaches, not how important the nodes are.

GET /api/v1/dashboard/impact-summary returns the tier for every ongoing incident at once: the union of what each incident's affected nodes reach, the tier, and up to three downstream consumers, dashboards first. The console's incident views read it. The answer is cached for 60 seconds.

Impact of a rule#

GET /api/v1/rules/{rule_id}/impact starts from the nodes a rule resolves to and merges what they reach. It adds the users subscribed to the affected nodes and a 30-day cost estimate per node. The cost estimates need the FinOps module and are empty without it. Rule impact belongs to the scheduling capability (see Rules and scheduling); node impact is always available.

Row-level evidence#

Edges describe the structure of your pipelines. Two optional features add evidence of what actually happened. Both are off by default.

LINEAGE_LEDGER_ENABLED (default false) makes the backend record a ledger entry for each BigQuery write event it receives: inserts, merges, creates, updates, deletes, loads and streaming writes. An entry holds the destination table, the writer, the job, the source tables, the row counts and the shape of the write (an append, a merge, a full rewrite and so on). With the ledger on and since set, impact analysis adds to each affected node the first write after since, the number of writes, the rows written, and confirmed_from_origin, which is true when a write lists the origin node among its sources.

The ledger separates a measured zero from a missing measurement. A BigQuery table reports writes_since: 0 only when the window starts after the ledger's earliest entry. Any other node, or an older window, reports nothing, and you must not read that as "nothing was written".

The row-trace routes under /api/v1/lineage need the lineage:row_trace permission. Of the role templates only tenant-admin, which holds every permission, includes it, so an administrator grants it to anyone else.

Route	Returns
`row_trace`	The ledger's write history of a BigQuery table and, recursively, of the sources that fed each write.
`row_journey`	The ledger write that most likely produced one row's current state, matched from the row's `SYS_*` audit values, and the writes upstream of it.
`row_preview`	Up to 100 rows of a BigQuery table, read through the BigQuery API with the node's connection (no query job). Columns with a policy tag are left out.
`provenance_coverage`	Per BigQuery project, how many tables carry `SYS_*` provenance columns (a load id, an ingestion reference, a row-key hash, an update timestamp).

row_trace can also look up one row by a key column that you name, and return that row's SYS_* audit columns. That lookup runs a BigQuery query, so it needs ROW_TRACE_BQ_PROBES_ENABLED (default false) and a backend identity that can read the table (roles/bigquery.dataViewer). It refuses a query whose dry-run estimate is above ROW_TRACE_MAX_SCAN_BYTES (30 GB by default).

What needs what#

Capability	Needs
Asset and column lineage, node impact, same-field groups, the entity-relationship bundle	Open source, always on.
Rule impact	The `scheduler` capability.
Primary-key and personal-data flags on the bundle's columns	The Catalog module. Without it, no column is flagged.
Table-to-dashboard links from Castor on the lineage view	The Catalog module, which stores the links. Nistor draws them as a separate, badged overlay, because Castor's matching is heuristic. Set `LINEAGE_CASTOR_LINKS_ENABLED` to `false` to turn the overlay off.
Data-quality checks in an impact answer	The Data quality module.
Cost estimates in a rule impact answer	The FinOps module.
Write evidence and row trace	`LINEAGE_LEDGER_ENABLED`, off by default. The row lookup also needs `ROW_TRACE_BQ_PROBES_ENABLED`, off by default.

The modules are part of Nistor Enterprise. The settings are environment variables of the backend; see Configuration.