ODCS data contracts

The ODCS tool (nistor-tool-odcs, tool id odcs) lets Nistor speak the Open Data Contract Standard, version 3.1.0. It validates a contract against the standard's JSON Schema, imports a contract that a producer wrote, exports the live state of a node as a contract, and syncs a repository of contracts from CI. It talks to no outside system: a contract is a YAML document you hand to Nistor. The classifier calls the package Alpha (see the overview).

What needs an Enterprise module#

Validation, import and export work in a pure open-source install. What a contract is projected onto depends on which other parts are present:

An export has the node's live columns (from its table schema), its freshness SLA and its stored contract. Descriptions from the catalog and attached checks appear in it only with those modules.

Write a contract#

A contract is a YAML document with apiVersion: v3.1.0, kind: DataContract, an id, a version and a status. This one describes a BigQuery table:

apiVersion: v3.1.0
kind: DataContract
id: acme-orders
name: acme_orders
version: 1.0.0
status: active
domain: sales
description:
  purpose: One row per order, for finance reporting.
team:
  - username: data-team@example.com
    role: owner
schema:
  - name: orders
    physicalName: orders
    physicalType: table
    properties:
      - name: order_id
        physicalType: STRING
        logicalType: string
        required: true
        primaryKey: true
      - name: amount
        physicalType: NUMERIC
        logicalType: number
        required: true
      - name: ordered_at
        physicalType: TIMESTAMP
        logicalType: date
        required: true
    quality:
      - type: library
        rule: rowCount
        dimension: completeness
        mustBeGreaterOrEqualTo: 1000
slaProperties:
  - property: latency
    value: 24
    unit: hours
    element: orders.ordered_at
customProperties:
  - property: nodeId
    value: BigqueryTable_acme-data-prod.sales.orders

Nistor reads the contract like this:

• The node. The nodeId custom property (or node_id) binds the contract to a graph node. You can also pass the node in the import request, which wins. A contract with no node is stored but projected onto nothing. Only the first object of schema is projected; the others are kept in the stored document.
• Columns. The properties of the first object. required: true makes a column non-nullable. classification and criticalDataElement become tags.
• SLA. A latency property becomes the freshness target in hours. frequency and timeOfAvailability are recorded. retention is kept in the document and not enforced.
• Dimensions. The contract above produces four on its Data Trust contract: schema (the three columns, extra columns allowed), freshness (24 hours), volume (at least 1000 rows, from the rowCount rule) and quality (a pass rate of 95 per cent, when the contract has any executable rule).
• Relationships are parsed and kept in the stored document. They are not written as links.

Data Trust evaluates a contract every 30 minutes. A dimension that breaches opens a contract breach and an incident, and a recovery resolves them. A dimension with nothing to measure is left out of the compliance score. See Incidents.

Validate, import and export#

The API is under /api/v1/odcs. Reading needs datatrust:view, and importing and deleting need datatrust:edit.

# Validate: returns the errors and a summary
curl -X POST "$NISTOR_API_URL/api/v1/odcs/validate" \
  -H "Authorization: Bearer $NISTOR_API_TOKEN" -H "Content-Type: application/json" \
  -d "$(jq -n --rawfile y orders.odcs.yaml '{yaml_content: $y}')"

# Import: a document that fails validation is refused with 422
curl -X POST "$NISTOR_API_URL/api/v1/odcs/import" \
  -H "Authorization: Bearer $NISTOR_API_TOKEN" -H "Content-Type: application/json" \
  -d "$(jq -n --rawfile y orders.odcs.yaml '{yaml_content: $y}')"

# Export a node's live state as a contract
curl -H "Authorization: Bearer $NISTOR_API_TOKEN" \
  "$NISTOR_API_URL/api/v1/odcs/node/BigqueryTable_acme-data-prod.sales.orders/export?download=false"

An import of an unchanged document is a no-op, because the stored row is keyed on a hash of its content. A document is limited to ODCS_MAX_YAML_BYTES (2,000,000). The export merges the stored contract under the live state, so keys Nistor does not model (pricing, roles, retention, relationships) survive a round trip, and the live state wins where they overlap.

Other routes: GET /contracts, GET /contracts/{id}, GET /by-node/{node_id}, DELETE /contracts/{id} and GET /health.

Contracts as code#

Keep the contracts in a repository and push them from CI:

nistor odcs sync --env prod --contracts-dir contracts/ --source-repo acme/data-contracts --delete-orphaned

nistor odcs sync sends the files to POST /api/v1/odcs/contracts/sync. For each one it replaces {ENV} with the environment, validates the document, and imports it as a code-managed, locked contract. A file that fails is reported and the batch goes on.

Use {ENV} inside a contract where an identifier differs between environments, as in the node id of a project named acme-data-{ENV}.

The connection#

A connection of type odcs describes the repository that holds the contracts:

The secret is token, and nothing reads it. The test checks that the address and the branch are well formed and says so: "repository not contacted (CI push model)". Nothing else reads the connection; the sync route reads the settings below.

Switch the feature off#

The feature is the capability odcs. It is on unless an administrator turns it off, with PATCH /api/v1/capabilities/odcs and a body of {"enabled": false}. Turning it off removes /api/v1/odcs.

Settings#