GitLab

The GitLab tool (nistor-tool-gitlab, tool id gitlab) owns everything in Nistor that calls the GitLab API. It does four separate things, and you can use any of them without the others:

The package README calls the tool Alpha, as does its classifier (see the overview).

What needs an Enterprise module#

None of the four features does. One section of the merge-request comment depends on a module: "DQ checks touched" lists rules whose action is a data-quality check, which come from the Data quality module (Nistor Enterprise). Without it, that section reads "none".

What the features share#

CI push needs nothing here. The other features need these three things:

• The tool must be active in the environment. Add {"id": "gitlab"} to the environment's compiler configuration (below). Pull sync, the merge-request comment, rule-request merge requests and the test of a GitOps connection all refuse while the tool is not listed there. The list is re-read every minute.
• A default Git host. Set DEPLOYMENTS_DEFAULT_GIT_HOST, or leave it empty and Nistor uses the host of RULE_REQUEST_GITLAB_URL. Nistor sends a GitLab token only to that host, whatever repository address a connection names.
• Tokens as secret names. A GitOps connection and the rule-request flow name a Secret Manager secret, by its name and not its path. The backend's service account needs roles/secretmanager.secretAccessor on it.

The reference Terraform root has no input for the settings on this page. Set the plain ones on the backend role through role_extra_env_vars. GITLAB_DISCOVERY_TOKEN is itself a token and must reach the compiler as a secret reference, which that input cannot express.

Make the tool active and find repositories#

Save a compiler configuration with a gitlab entry (see CLI):

[
  {
    "environment": "prod",
    "tools": [
      {
        "id": "gitlab",
        "config": {
          "groups": ["acme/data"],
          "discover_known_groups": true,
          "include_subgroups": false,
          "include_archived": false
        }
      }
    ]
  }
]

A compile builds GitlabProject nodes from two scopes:

• Known repositories. The repositories the backend already knows, on your GitLab host: GitOps connections, CI deployment reports, deployment provenance (including every Dataform remote) and the repository that owns a config-as-code rule. These need no token.
• Discovery. Only with GITLAB_DISCOVERY_TOKEN set: the projects of every group in groups, and, with discover_known_groups (default true), the parent group of every known repository. include_subgroups and include_archived default to false.

If GitLab or the backend is down during a compile, the previous nodes are kept. A group that Nistor derived itself and that the token cannot see is skipped quietly. A group you listed that fails is reported.

A repository is linked to what it deploys when the graph is read (as "deployed by"), and never as lineage.

The gitlab connection#

You can create a connection of type gitlab in the web shell (Connections, /admin/connections):

The test calls GET {base_url}/api/v4/version with the token and is healthy only when the reply is a version document. Nothing else reads this connection yet: discovery and the API clients use the settings on this page. A GitOps connection (below) is a different thing, with its own token.

Rules from a repository#

A repository keeps its rules as top-level *.json, *.yaml or *.yml files in one directory, each file a list of rules. There are two ways to get them into Nistor, and one repository uses exactly one:

Pull mode, on a GitOps connection:

• Create the connection on the GitOps page of the web shell (/gitops) in the mode labelled "Pull". The API value of that mode still carries the product's former name, so use the page.
• Its fields: repo_url, repo_project_path, branch_mapping (default dev to dev, test to test, prod to prod), rules_directory (default rules/), gitlab_token_secret (a secret name), and poll_interval_minutes (default 5, from 1 to 1440).
• Each backend syncs only the branch mapped to its own environment, and at most one branch per environment.
• A sync reads one commit, all or nothing: if any file fails to parse or validate, nothing is written. A head that did not change is a no change. Rules that came from this repository and are no longer in it are removed.
• A repository that is pulled refuses CI reports: remove its sync job, or switch the connection to CI push.
• Creating a pull connection, or changing its repository, branch mapping, directory or token, needs config:manage. Other changes to GitOps connections need connections:manage.

Check a rule file before you push it:

nistor rules validate --dir rules/
nistor rules sync --env prod --dir rules/ --source-repo acme/rules --delete-orphaned

Merge-request impact comment#

The bot reads a merge request and posts one comment saying what it touches. To switch it on:

1. Set GITOPS_MR_BOT_ENABLED=true on the backend.

2. Create a GitOps connection for the repository and set its mr_impact_enabled to true, with gitlab_token_secret naming the secret that holds the token.

3. Create a webhook endpoint for the connection. The response carries the endpoint's secret once, and its path:

   curl -X POST "$NISTOR_API_URL/api/v1/webhooks/" \
     -H "Authorization: Bearer $NISTOR_API_TOKEN" \
     -H "Content-Type: application/json" \
     -d '{"slug": "acme-rules-mr", "name": "acme rules merge requests",
          "webhook_type": "gitlab_mr", "gitops_connection_id": "<connection-uuid>"}'
   

4. In the GitLab project, add a webhook that points at https://<your-nistor-host>/api/webhooks/acme-rules-mr, with the secret as its secret token, for merge request events. Nistor checks the X-Gitlab-Token header against it.

The receiver route has no sign-in of its own: it checks the token and nothing else, so your proxy must let GitLab reach /api/webhooks/. While the bot is off, the endpoint answers 404 exactly as an unknown path does.

The bot analyses a merge request when it is opened, reopened, or updated with new commits, and not when it is closed, merged or approved. It runs inside the backend, two analyses at a time, and does the same head commit only once. It maps the changed files to nodes by three sources: Dataform files, Terraform files and rule files. The comment, which it edits in place on later pushes, has these sections:

• affected nodes and the size of their blast radius (the changed nodes and what is downstream);
• scheduling rules that match them;
• data-quality checks that match them (see above);
• open incidents on them;
• an estimated BigQuery cost delta, which this release does not compute ("not available in this release");
• files that matched no node, with their count.

To see what the bot would post without posting it, call POST /api/v1/gitops/mr-preview with gitops_connection_id and mr_iid (needs gitops:view).

Rule-request merge requests#

In the web shell a person can request a rule. When a reviewer approves the request, Nistor opens a merge request in your rules repository that adds the rule to a file. If the approval fails, the request stays unapproved.

The environment variables GITLAB_TOKEN and GITOPS_GITLAB_TOKEN, when set, take precedence over the secret. They are meant for development and test.

The file is chosen from the action of the rule, and the file names are fixed in the code:

The code accepts seven file names in all, and refuses any other: data_insight_rules.json, data_mart.json, data_quality_rules.json, dwh_rules.json, export_rules.json, historization_rules.json and microstrategy_rules.json. The file is created if it does not exist, and must be a JSON array of rules if it does.

Nistor gives the rule a code_id of the form <prefix>/<slug>, commits to a new branch with the message feat(rules): add <code_id> (Nistor rule request), and opens the merge request. If an approval is retried, Nistor reuses the open or merged merge request of that request instead of opening a second one.

Permissions#

In Nistor, GitOps connections and webhook endpoints need connections:manage (pull connections also config:manage), a merge-request preview needs gitops:view, and replacing the compiler configuration needs compiler:trigger.

Settings#