Google Cloud

The Google Cloud tool is Nistor's reference integration. It reads the Google Cloud services of the projects you monitor, builds assets and lineage from them, follows what happens in those projects through their audit logs, and runs Dataform workflows, Data Fusion pipelines and BigQuery copies and exports from rules.

Package nistor-tool-gcp, tool id gcp, connection type gcp. Its README calls it "Production"; its package classifier says Alpha (see the overview).

What needs an Enterprise module#

A pure open-source install discovers resources, builds lineage, ingests audit-log events, probes freshness and runs the four actions below. These parts need an Enterprise module (Nistor Enterprise):

The open-source code declares the catalog and billing_export blocks and does not read them. For Recommender, it provides only the client.

What it collects#

The tool declares 25 node types. A compile reads each service of each project you list (see Tell the compiler what to scan).

The sheets that a BigQuery external table reads become GoogleSheet nodes. That node type belongs to the Google Workspace tool.

Lineage is built at compile time, from SQL and from the links between services:

• Table lineage comes from view definitions and from the actions of Dataform compilation results.
• Column lineage is parsed from SQL: Dataform actions, BigQuery view SQL and source files.
• Links between services are inferred: a Pub/Sub push subscription to the Cloud Run service or function it calls, a Cloud Scheduler job to its target, an Airflow task to the BigQuery tables its operator names, Cloud Run and Compute Engine to Cloud SQL, Vertex AI to its BigQuery and Cloud Storage artefacts, a workload to its container image.
• A Cloud Function's reads and writes combine its source code with the BigQuery and Cloud Storage events seen for its service account.
• With LINEAGE_LEDGER_ENABLED=true (off by default), the backend also keeps a ledger of observed writes with the tables each BigQuery job read.

See Lineage and impact for how the graph uses them.

Events reach Nistor through Cloud Logging. The monitored_project Terraform module (below) creates eleven log sinks in each monitored project and routes them to the platform's Pub/Sub topic, where the input-logs role reads them:

• Cloud Storage object create, read and delete;
• BigQuery job completions, child jobs, table reads, table data changes and table metadata changes, and the BigQuery Storage Write API;
• Cloud Run and Cloud Functions deploys and deletes;
• Data Fusion pipeline start and end lines.

The BigQuery events record the bytes billed per job.

Runs that did not start from Nistor are ingested too: Dataform workflow invocations (polled every 60 seconds by default, for the repositories in the projects a connection lists in monitored_projects) and Data Fusion pipeline runs (from the log sink and from a backfill that scans the connection's ingestion_projects).

Freshness probes read the last modification of a BigQuery table (with tables.get, which runs no job and bills nothing), of a linked dataset table and of a Cloud Storage file selection.

Before you start#

You need, for each project to monitor:

• the APIs of the services you want read, enabled in that project. For Compute Engine, Vertex AI, Cloud SQL and Dataform a compile checks first and skips a project whose API is off. For the other services a disabled API is an error, which raises an extraction incident;
• a service account for the connection (the "connection service account"), with the roles in Permissions;
• for each Nistor role that reads or acts on Google resources, the right to impersonate that account (see Identity).

The platform modules create nothing in a monitored project. The monitored_project module does, and you instantiate it once per project.

Create the connection#

Create a connection of type gcp in the web shell (Connections, /admin/connections, which needs connections:manage) or through the API:

curl -X POST "$NISTOR_API_URL/api/v1/connections" \
  -H "Authorization: Bearer $NISTOR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "acme-estate",
    "tool_type": "gcp",
    "is_default": true,
    "config": {
      "billing_project": "acme-nistor-prod",
      "identity": {
        "kind": "impersonation",
        "service_account": "nistor-gcp-estate@acme-nistor-prod.iam.gserviceaccount.com"
      },
      "monitored_projects": ["acme-data-prod", "acme-ingest-prod"],
      "locations": ["europe-west1", "EU"]
    }
  }'

Then test it. The test builds the credentials and refreshes them once, which reaches Google (IAM Credentials, for an impersonated account) without a billable call. The probe answers healthy or unhealthy, never degraded. See Configuration for connections in general.

Fields#

The only secret is service_account_json, and only for identity.kind service_account_json.

Identity#

The last two kinds are refused on a deployed role, because they would run as the runtime's own identity. A role is deployed on Cloud Run (a service or a job), or wherever NISTOR_DEPLOYED_ROLE is set to anything but an explicit no (0, false, no or empty). Set NISTOR_DEPLOYED_ROLE=1 on any other Google runtime, such as GKE or Compute Engine.

Every Google call of the tool runs as the connection of its unit of work. Nothing falls back to the runtime's identity. A unit of work resolves its connection in one of these ways: the run's connection, the connection of the rule or node, the poller's own connection, the one named in the compile request, or, for work that knows only a project, the connection that lists that project in monitored_projects. If several list it, the default connection wins. If none does, the work fails.

Tell the compiler what to scan#

A compile reads the projects and Dataform repositories listed in the gcp entry of the environment's compiler configuration. The entry names the connection it runs as. Save the configuration with nistor compiler sync (see CLI):

[
  {
    "environment": "prod",
    "tools": [
      {
        "id": "gcp",
        "config": {
          "connection_id": "<connection-uuid>",
          "dataprojects": ["acme-data-prod", "acme-ingest-prod"],
          "dataform": [
            "projects/acme-data-prod/locations/europe-west1/repositories/acme-dwh"
          ],
          "object_types": { "vertex_ai": false, "compute_engine": false },
          "project_object_types": { "acme-ingest-prod": { "datafusion": false } }
        }
      }
    ]
  }
]

• object_types switches a service off for every project, and project_object_types for one project. The services are bigquery, pubsub, cloudrun, cloudfunctions, secrets, cloud_storage, cloud_scheduler, cloud_composer, cloud_sql, bigtable, compute_engine, vertex_ai, datafusion and artifact_registry. All are on by default. Switch off a service you do not use, so that a missing role does not raise an extraction incident on every compile.
• Artifact Registry is read in the dataprojects, in the registry projects named by the images of your Cloud Run services and functions, and in any project you list in artifact_registry_projects.
• You can add one gcp entry per connection.
• The reference Terraform schedules the compile with cron_compilation (default 0 2 * * *).

Run Google Cloud actions from rules#

Four executors ship with the tool. A rule names one in then.do.action.type. See Rules and scheduling for the rest of a rule.

- name: Rebuild acme-dwh when a source changes
  priority: 3
  status: active
  code_id: dwh/acme-source-change
  applies_to:
    node_type: DataformAction
    tags:
      - dwh_rebuild
  when:
    events:
      - origin: ancestors
        types: data_updated
        ancestor_depth: direct
    wait_for_all_ancestors: true
  if:
    quiet_period_seconds: 300
    max_estimated_cost_usd: 5
  then:
    target:
      same_as_applies_to: true
      scope: node
    do:
      kind: single
      action:
        type: dataform_run
        parameters:
          release: projects/acme-data-prod/locations/europe-west1/repositories/acme-dwh/releaseConfigs/daily

Check the file with nistor rules validate --files <file>, then push it with nistor rules sync.

Things to know before you rely on the actions:

• bigquery_copy overwrites its destination by default (WRITE_TRUNCATE). It derives the destination project by replacing the -<source_environment> suffix of the source project with the target environment, or from the project patterns of the environment mappings (nistor env-mapping). A table that maps to nothing fails the run.
• bigquery_export_gcs writes to the bucket you name. Check the destination before you enable the rule.
• dataform_run is the only executor that can dry-run its SQL. If the rule sets if.max_estimated_cost_usd, Nistor estimates the cost first, and blocks the run, recorded as failed, when the estimate is above the limit. For the other executors that limit has nothing to estimate.
• An interrupted Dataform dispatch is failed, not started again, because Dataform offers no way to find the invocation that the dispatch started. A Data Fusion dispatch is found again after a crash.
• Nistor can check, before a rule runs, that the connection holds the permissions in the last table below. The check for bigquery_copy reads parameters.target_table, and the check for bigquery_export_gcs reads parameters.gcs_uri. The executors read other parameters (above), so for a rule written as above the check reports unknown.

Permissions#

Nistor reads a project as the connection service account. Grant it the roles below in each monitored project. The monitored_project Terraform module can bind the roles marked "yes" in the Terraform default column (set grant_tool_roles = true; the deployer then needs resourcemanager.projects.setIamPolicy). By default it binds nothing, and you grant the roles yourself:

SA=nistor-gcp-estate@acme-nistor-prod.iam.gserviceaccount.com
gcloud projects add-iam-policy-binding acme-data-prod \
  --member="serviceAccount:$SA" --role="roles/dataform.viewer" --condition=None

Read roles#

The default roles read metadata only: none reads a secret payload, a table row or an object. roles/bigquery.dataViewer does read table data, which is why it is optional. The Terraform default list has no BigQuery role, so grant them yourself in each monitored project, and roles/bigquery.jobUser in the billing project.

Roles for optional features#

Permissions the actions need#

Before a rule runs, Nistor can ask Google which of these the connection holds (testIamPermissions):

The runtime side#

With impersonation, the service account of each Nistor role that reads or acts on Google resources must be allowed to create access tokens for the connection service account: roles/iam.serviceAccountTokenCreator on the connection service account. The Terraform in this repository does not grant it. The runtime accounts hold no role in the monitored projects.

Terraform for the monitored projects#

packages/integrations/gcp/terraform/monitored_project holds everything the tool creates in a monitored project:

• one log sink for each of the eleven default filters, routed to the platform's log topic;
• optionally, Cloud Storage object notifications for the buckets you list, which add the size, content type and generation that audit logs lack;
• optionally, the default read roles, bound to the connection service account.

The reference root wires it from three inputs. enable_input_logs is true by default and the module needs it:

monitored_projects                  = ["acme-data-prod", "acme-ingest-prod"]
gcp_tool_connection_service_account = "nistor-gcp-estate@acme-nistor-prod.iam.gserviceaccount.com"
monitored_project_buckets           = { "acme-data-prod" = ["acme-landing"] }

The publish grants on the platform's topics are in the runtime project, so the platform modules make them. The module's README lists its inputs and how to move a deployed root onto it. For the rest of the install, see Install on GCP.

Settings#

The tool has no application-wide settings: what it reaches is the configuration of each connection. Two environment variables matter: