Google Workspace

Spreadsheets often feed real pipelines. The Google Workspace tool (nistor-tool-google-workspace, tool id google_workspace) turns a change to a Drive file, whether a Sheet, a Doc or a folder, into an event on the timeline of that file, beside the events of BigQuery and the other tools. It only reads. The README and the classifier call it Alpha (see the overview).

Everything on this page is open source: no Enterprise module (Nistor Enterprise) is involved.

What it collects#

Each change becomes a node event and a run record. The node type depends on the file:

The GoogleSheet type is also the node that the Google Cloud compile builds for a sheet that a BigQuery external table reads, so a change to such a sheet appears on a node that already sits in lineage.

For each change Nistor writes:

• an input-log event of type WORKSPACE_FILE_CHANGE;
• a run record, with origin gworkspace_external, so that the change shows on the node's activity. Set GWORKSPACE_EMIT_ACTION_RUN=false to write only the event.

The record is keyed on the file and its revision, so a message delivered twice does not become two changes. Native Docs and Sheets carry no head revision, so for them the key is the modification time. People who follow the node are notified after the change is committed.

If Nistor cannot read a file's details (a 403: the file is not shared with the service account, or delegation is not granted), it still ingests the change from the event, and raises an alert and an incident that name the file and the class of error.

Drive file ids carry no environment. Every Workspace node takes the compile's environment, unless you set a fixed one with GWORKSPACE_NODE_ENV or pin a file to an environment in the environment mappings.

Two ways to receive changes#

You can use either, or both. They write the same records.

Push#

1. Enable the APIs workspaceevents.googleapis.com, drive.googleapis.com, sheets.googleapis.com and docs.googleapis.com in the project.
2. In the reference Terraform root, set enable_workspace_events = true, gworkspace_dwd_subject, gworkspace_sa_key_secret_id, and the workspace_events role service account. The module creates the topic, a pull subscription with a dead-letter topic (ten attempts), the workspace-events Cloud Run service, and the roles/pubsub.publisher grant on the topic for Google's Workspace Events service agent (service-<project-number>@gcp-sa-workspaceevents.iam.gserviceaccount.com).
3. Set GWORKSPACE_TARGET_RESOURCES to the Drive resources to follow, for example //drive.googleapis.com/drives/<shared-drive-id>, on the workspace-events role and on the backend, which renews the subscriptions. The module takes the value as its target_resources input, which the reference root does not pass: add it there. The backend also needs WORKSPACE_EVENTS_TOPIC_NAME, the name of the module's topic.
4. Renew the subscriptions before they expire (about seven days for Drive; GWORKSPACE_SUBSCRIPTION_TTL_SECONDS asks for 604800). Fire the workspace_renew_subscriptions platform job from a rule, or call POST /api/v1/workspace-events/internal/renew-subscriptions from a scheduler. The module has its own daily Cloud Scheduler job, but it is created only when the module's target_resources input is set, which the reference root does not pass.

GET /api/v1/workspace-events/status shows the topic, subscription, targets and whether a delegated user is set.

Pull#

1. Set GWORKSPACE_POLL_TARGETS on the backend (in the reference root, through role_extra_env_vars) to a comma-separated list: shared-drive ids, or my-drive for the delegated user's own Drive.
2. Fire the workspace_poll_changes poller from a scheduling rule, every five minutes for example, on the backend. A batch poller runs only when a rule fires it, and nothing in this repository creates that rule for you: the platform-jobs plugin shows the rule shape.

The first poll of a target records only future changes. Set GWORKSPACE_POLL_BACKFILL=true to walk the whole feed once. Each target keeps its own page-token cursor, so a failing target never moves its cursor and never stops the others. A change that fails to ingest three times in a row is quarantined, so that one file cannot hold a cursor for ever, and the run shows ChangeQuarantined. A later edit to that file is tried again from the start.

Credentials#

The tool reads Drive as a service account that impersonates a Workspace user (domain-wide delegation).

1. Create a service account and a key for it. Keep the key in a Secret Manager secret in the runtime project. You create the secret container and add the version yourself: the key never goes through Terraform.

2. In the Workspace admin console, authorise the service account's client id for these scopes:

   Scope	Used for
   https://www.googleapis.com/auth/drive.readonly	Push and pull
   https://www.googleapis.com/auth/spreadsheets.readonly	Push and pull
   https://www.googleapis.com/auth/documents.readonly	Push and pull
   https://www.googleapis.com/auth/workspace.events	Push: creating and renewing subscriptions
   https://www.googleapis.com/auth/admin.directory.user.readonly	The connection test only

3. Set the delegated user and the secret on the roles that read Drive (backend, jobs, workspace-events): GWORKSPACE_DWD_SUBJECT and GWORKSPACE_SA_KEY_SECRET_ID (a bare secret id, not a path). The reference root sets both when you pass gworkspace_dwd_subject and gworkspace_sa_key_secret_id, and grants each role's account roles/secretmanager.secretAccessor on that one secret.

On a deployed role (Cloud Run, or where NISTOR_DEPLOYED_ROLE is set) a service-account key is required: ambient Application Default Credentials are refused, so that the runtime's own identity can never stand in for the Workspace identity. They remain usable on a developer machine. The package README also allows sharing each file with the service account in place of delegation.

The connection#

You can create a connection of type google_workspace in the web shell (Connections, /admin/connections):

The secret is service_account_json.

Today only the test reads the connection. The push worker, the subscription renewal and the poller read GWORKSPACE_DWD_SUBJECT and GWORKSPACE_SA_KEY_SECRET_ID.

The test asks the Admin Directory for one user (users.list), as the delegated user, with the key of the connection:

healthy means only that the Directory call worked. A grant that carries the Directory scope and not the Drive scopes still tests healthy, and ingestion then fails.

Settings#

The tool also has a request limit per process, NISTOR_TOOL_LIMIT_GOOGLE_WORKSPACE.