Data Trust
Monitored assets, freshness, SLOs and error budgets, data contracts, the Trust Score and status pages.
Data Trust answers one question for the people who use your data: can I rely on this table, cube or report right now? You choose the assets that matter and say how fresh each must be. Nistor then tracks their health, their freshness, the error budget they burn and the contracts they must honour, and sums it up in a Trust Score.
Data Trust is open source. Two of its signals come from Enterprise modules: data-quality results (the Data quality module) and business domains (the Catalog module). The sections below say where, and What needs what lists them.
Monitored assets#
A monitored asset points at a node of the graph (node_id) and carries what you expect of it:
| Field | Meaning |
|---|---|
freshness_sla_hours |
How old the data may get before it is stale. 24 by default. |
delivery_deadline, deadline_grace_minutes |
Optional: a time of day (HH:MM, read as UTC) by which the asset must be fresh, and the slack after it. |
health_mode |
How health is judged: auto (from runs and incidents, the default), mstr (from the MicroStrategy API) or manual (set through the API). |
owner |
Who answers for it. |
expected_termination |
Set it when an asset is paused or retired on purpose. Its staleness then counts as expected: it burns no budget and opens no heartbeat or delivery-SLO incident. |
Add assets in the console under Data Trust, or with POST /api/v1/data-trust/assets. Importing an ODCS contract also
creates one (see ODCS data contracts).
A health group gathers assets, for a team or a product. Each member is watched alone (self) or with the nodes
upstream of it (ancestors, optionally bounded by a depth), so a group turns red when something it depends on breaks. A
group is public by default; a private group is visible to its creator only.
Health and freshness#
Every 30 minutes, Nistor takes a health snapshot of each active asset. In auto mode it reads the asset's last 20 runs
of the past seven days and the incidents open on its node. The latest successful run is the last refresh, and the asset
is fresh while that refresh is younger than freshness_sla_hours.
| Status | When |
|---|---|
critical |
An incident is open on the node, the data is staler than the SLA, or the last run failed. |
warning |
Fewer than 80% of the recent runs succeeded. |
unknown |
Nistor has no runs or checks to judge by. |
healthy |
None of the above. |
Data-quality failures never change the status. The console shows them beside it, and they lower the Trust Score instead.
Separately, every five minutes, a freshness evaluator judges each asset from its latest freshness observation: a tool's
reading of the source's last-modified time, or a successful run. The asset is at risk from 75% of its SLA and in breach
when the lag reaches it. A breach sends freshness_sla_breach, and its end freshness_sla_resolved, to the
alert policies that match. It can also hold the rules downstream that use
hold_while_upstream_stale. The same evaluator judges the freshness_expectation blocks of rules (see
Rules and scheduling).
Data Weather looks ahead. Every 30 minutes it fits a trend to each asset's arrival times over the last 30 days (it
needs at least five successful runs) and estimates whether the next load will miss the delivery deadline or, without
one, the SLA. From a 50% chance of a miss it sends predicted_freshness_breach, as a critical alert from 90%. It opens a
predicted_freshness incident only with FRESHNESS_FORECAST_CREATE_INCIDENTS on (off by default).
FRESHNESS_FORECAST_ENABLED turns the forecast off.
SLOs and error budgets#
Each asset has a freshness SLO: the share of time it should be fresh, over a rolling window. Unless you set your own
with PUT /api/v1/data-trust/assets/{id}/slo, it is:
| Setting | Default |
|---|---|
target |
0.99 |
compute_window_days |
30 |
alerts |
A critical window when 2% of the budget burns in 1 hour, and a warning window when 5% burns in 6 hours. |
The error budget is the time the asset may be stale: 1% of the window at a 0.99 target. The burn rate compares what is spent with what is allowed:
burn rate = (stale snapshots / all snapshots) / (1 - target)
At a burn rate of 1 the budget lasts exactly the window; above 1 it runs out early. Spending 2% of a 30-day budget in an
hour is a burn rate of about 14.4. Snapshots come every 30 minutes, so that is the resolution. A snapshot taken inside an
alert-suppression maintenance window, or of an asset marked expected_termination, counts as fresh.
GET /api/v1/data-trust/assets/{id}/burn returns the burn over 1, 6, 24 and 720 hours (windows= changes them). A
node's page in the console shows its SLO and error budget.
A burning window does not open an incident by itself. With DELIVERY_SLO_INCIDENTS_ENABLED on (off by default), Nistor
opens one delivery_slo incident per asset when the shortest alert window has fired for three consecutive snapshots
(DELIVERY_SLO_SUSTAIN_CYCLES), or when the delivery deadline is missed past its grace; a missed deadline is critical.
A clean snapshot resolves it. See Incidents.
Data contracts#
A data contract states what one or more assets promise, dimension by dimension. It has a version and a status
(draft, active or deprecated); only active contracts are checked. Each dimension appears at most once:
| Dimension | Breached when | Measured from |
|---|---|---|
freshness |
The last refresh is older than sla_hours. |
The health snapshots. |
schema |
A column in expected_columns is missing or, with allow_extra_columns: false, an unexpected column appears. |
The node's columns in the graph. |
volume |
The latest row count is below min_rows, or moved more than max_delta_pct percent from the previous one. |
Row-count metrics from the Data quality module. |
quality |
Fewer than min_pass_rate of the data-quality runs in the window passed. |
Data-quality runs from the Data quality module. |
Every 30 minutes, Nistor checks each pair of asset and dimension. A failing pair opens a breach, a data_contract
incident and a contract_breach alert. When the pair passes again, the breach and the incident close and a
contract_resolved alert goes out. A pair that cannot be measured is skipped: it opens nothing and closes nothing.
Without the Data quality module, the volume and quality dimensions have no data, so they never breach.
A contract's compliance is the share of measured pairs that pass: GET /api/v1/data-contracts/{id}/compliance.
Changing a contract's dimensions cuts a new version, and GET /api/v1/data-contracts/{id}/versions lists them. Write
contracts in the console or through the API, or import them in the Open Data Contract Standard (see
ODCS data contracts).
Trust Score#
The Trust Score is a number from 0 to 100, for one asset, a health group or the whole platform. The backend computes it, so every screen shows the same number. For one asset, from its latest snapshot:
| Part | Effect |
|---|---|
| Status | 60% of the score: 100 when healthy, 65 for a warning, 50 when unknown, 0 when critical. |
| Recent success rate | 40% of the score. |
| Staleness | Minus 5 points for each SLA-length beyond the SLA, up to 30. |
| Failing data-quality checks | Minus up to 40 points, in proportion to the share of checks failing. Zero for an asset without checks, and so without the Data quality module. |
A group's score is the mean of its assets' scores, less up to 15 points for open critical incidents on its nodes. The platform's score is the same over every active asset. Bands: red below 60, amber from 60, green from 80.
GET /api/v1/data-trust/trust-score?scope=platform returns it; use scope=workspace (a health group) or scope=asset,
with scope_id, for the others. GET /api/v1/data-trust/score-history returns the platform's daily scores.
Status pages#
A status page shows a health group, or a business domain, the way a service status page does: an overall level
(operational, degraded, down or unknown), the level of each asset, daily uptime, the incidents open on its
assets, and its unresolved freshness and contract breaches. Uptime comes from the health snapshots: a warning counts as
half, and an unknown snapshot is left out. A page covers 90 days unless you pass window_days.
Status pages are in the console under Data Trust, Status, and in the API under /api/v1/data-trust/status-page/
(health-groups, health-group/{id} and domain/{term_id}). They are for signed-in users with datatrust:view:
sharing a page means sending its link to a colleague who has that permission. A domain is a term of the business
glossary, so domain pages need the Catalog module; without it they answer 501.
Permissions#
The Data Trust pages need datatrust:view, which the viewer role template, and so every template, includes. Changing
assets, groups, SLOs and contracts needs datatrust:edit. Of the templates, only tenant-admin (which holds every
permission) includes it, so an administrator grants it to anyone else. A group's alert policies need alerts:manage.
What needs what#
| Feature | Needs |
|---|---|
| Monitored assets, health groups, health and freshness, Data Weather, SLOs and burn rates, the Trust Score, health-group status pages | Open source. |
Contracts with freshness and schema dimensions |
Open source. |
Contracts with volume and quality dimensions |
The Data quality module, which measures them. |
| The data-quality penalty in the Trust Score, and an asset's data-quality breakdown | The Data quality module. |
| Status pages for business domains | The Catalog module. |
delivery_slo incidents from burn or deadlines |
DELIVERY_SLO_INCIDENTS_ENABLED, off by default. |
predicted_freshness incidents |
FRESHNESS_FORECAST_CREATE_INCIDENTS, off by default. |
The modules are part of Nistor Enterprise. The settings are environment variables of the backend; see Configuration.