Install on Kubernetes

What you install#

One Helm chart, oci://registry.gitlab.com/nistor-io/nistor/charts/nistor (signed with cosign, same version as nistor; the chart's README lists every value), into one namespace:

• the backend, scheduler, jobs, compiler, input-logs, runner and frontend Deployments (mcp and workspace-events are optional);
• the migrate Job, the compile CronJob and one CronJob for each enabled cronjob cron of the role catalogue;
• the objects between them: one projected service-account token per call edge (audience nistor:<namespace>:<role>), NetworkPolicies that allow only the catalogue's call graph, PodDisruptionBudgets and autoscalers.

Cloud Run stays supported: deployments/gcp is unchanged. One release lives in one namespace; a Nistor instance is a namespace.

Before you start#

• Kubernetes with NetworkPolicy enforcement and service-account issuer discovery (GKE: Dataplane V2; kind 0.24 or later). The Gateway objects need the Gateway API.
• PostgreSQL 17 that the namespace can reach, with a database and a user; the password in a Secret of the namespace.
• Google Pub/Sub and Cloud Storage: in this release the message bus and the object store are Google's on every profile (pull subscriptions only, so nothing calls into the cluster). Other stores are planned for a later release.
• No persistent secret store outside Google. Where Secret Manager is not configured or not reachable (the generic profile, kind, any cluster outside Google), the backend's secret writes answer 503: creating, updating or rotating a connection with secret values stores nothing. A connection without secret values is not affected. A portable secret store is planned for a later release.
• The images of a release lock (nistor dist lock, nistor dist build), pushed to a registry the cluster pulls from and given to the chart by digest.
• helm 3.16, kubectl, cosign 2.

The database connection budget#

The chart does not cap connections; check this sum against the instance's max_connections before you raise any maximum. Per pod, a role holds the engines the code bounds: for each worker its pools (size plus overflow), the backend's idempotency pool, a background engine of twice BACKGROUND_TASKS_MAX_CONCURRENT connections, and on the backend the one connection its auth-cache listener holds open, times the workers; the scheduler and input-logs pods add up to 8 for the leader-elected crons while cycles overlap. The budget a release needs is the sum, over the roles that reach the database, of that figure times roles.<role>.replicas.max. With the defaults:

Keep at least 20 more connections free for what the code does not bound:

• the catalog engine (one connection per catalog request in flight) and the orchestrator engine (one per running chain or backfill run);
• the tick pollers that bypass the background semaphore (the Google Cloud and MicroStrategy tools' ones), which open two more connections per backend process while their tool is active;
• the migrate Job and the data-quality rebuild Job (parallelism 8), which take transient connections.

A stock non-production instance (max_connections 100) fits the defaults with that reserve.

The candidate track (candidate.enabled) is not in that sum and does not fit the reserve. It adds one pod of every role, 59 connections at the defaults (backend 13, scheduler 20, jobs 14, input-logs 8, runner 4), so with the stable track at its maximum the worst case is 139. On a max_connections 100 instance a candidate release is therefore a documented limit: the figures are worst cases (every pool and overflow slot in use, every cron cycle overlapping), so run the candidate with little traffic, on a stable track that is below its maximum, and watch pg_stat_activity; or use a larger instance. With deployments/gke in prod (max_connections 400) the stable track at its maximum and a candidate fit.

To raise a maximum, raise it together with the instance's limit: roles.<role>.replicas.max, and in roles.<role>.env UVICORN_WORKERS, DB_POOL_SIZE, DB_MAX_OVERFLOW and, on the backend, IDEMPOTENCY_DB_POOL_SIZE and IDEMPOTENCY_DB_MAX_OVERFLOW (2 and 8 in the code; the chart sets 1 and 2). Enabling workspace-events adds its own pools. With deployments/gke, Cloud SQL's max_connections is 100 outside prod and 400 in prod, and the helm_values output sets the maxima for the tier.

GKE with deployments/gke#

Terraform creates (or reuses, cluster.create = false) a VPC-native cluster with Workload Identity, Dataplane V2, Gateway API, the Secret Manager add-on, a DNS control-plane endpoint and GKE's binding protection (no RBAC binding may grant system:anonymous, system:unauthenticated or system:authenticated; a reused cluster keeps its own setting, so turn it on there yourself); one Google service account per role bound to its Kubernetes service account; Cloud SQL on a private IP; the pull subscriptions with the push subscriptions' delivery settings; the buckets; the Gateway's address and certificates. It does not install the chart. A new cluster's first apply runs twice (-var kubernetes_objects=false, then without). terraform output -json helm_values > helm-values.json is the values file of the section "Values". Point the DNS names at gateway_address.

A slow on-hold-actions message is redelivered later than under push: the pull client extends a message's lease up to 600 s while the subscription's own acknowledgement deadline stays 60 s.

Bring your own cluster#

Set global.platform: generic, and provide:

• the database Secret (database.passwordSecret) and database.host;
• oidc.issuer: your API server's --service-account-issuer (kubectl get --raw /.well-known/openid-configuration);
• the subscriptions (bus.subscriptions) and topics (bus.topics), the buckets (COMPILATION_BUCKET, set once in global.env), GOOGLE_CLOUD_PROJECT, and whatever Pub/Sub and Cloud Storage need to authenticate, through global.env;
• networkPolicy.extraEgress for anything the policies do not already allow. By default they allow DNS, the call graph, the database, the API server and HTTPS to any address; with networkPolicy.restrictInternet the Google APIs need a rule of yours (see "Security notes").

Values#

The values reference is in the chart's README, generated from values.schema.json. Always set global.image.*, global.frontendImage.*, global.environment, database.*, bus.subscriptions, bus.topics, oidc.issuer, and in roles.backend.env the settings the backend needs, among them BOOTSTRAP_ADMIN_EMAIL (the first admin: a person who signs in through IAP). GKE adds -f values-gke.yaml and the helm_values file.

A setting that must not sit in the values (an identifier your platform issues, a licence or API key) is read from a Kubernetes Secret key instead: create the Secret in the release's namespace, then list it in global.envFromSecrets (every pod, the Jobs and the migrate hook included) or roles.<role>.envFromSecrets (one role), as {name, secretName, key}. The chart renders a secretKeyRef, so only the Secret's name and key reach the manifests. A pod does not start while the Secret or key is missing, unless the entry says optional: true, and a change to the Secret reaches the pods at their next restart (kubectl rollout restart), not before. No profile sets an entry by default; see the chart's README for the rules.

The mcp role stays off outside global.platform: gke: with NISTOR_MCP_BACKEND_AUDIENCE set it asks the Google metadata server for a token, and a code change (listed under "Planned for a later release") has to come first.

A standby install beside Cloud Run takes this in its values:

bus: {delivery: push, subscriptions: {}}
crons: {mode: http, cronjobs: []}
jobs: {compile: {enabled: false}}

The chart then pulls nothing, runs no in-process cron and renders no CronJob, so Cloud Scheduler alone drives the crons. With crons.cronjobs and jobs.compile left at their defaults the CronJobs would fire beside Cloud Scheduler.

Install and upgrade with the install template#

Set TARGET=kubernetes with:

• K8S_NAMESPACE, and HELM_RELEASE (nistor);
• K8S_CLUSTER: gke with GKE_CLUSTER and GKE_LOCATION, or kubeconfig with a File variable KUBECONFIG;
• GKE_PROJECT_ID and GKE_PROJECT_NUMBER when the cluster is in another project than the runtime's;
• CHART_REF and CHART_VERSION (with CHART_COSIGN_IDENTITY and CHART_COSIGN_ISSUER, the chart signer), or CHART_DIR;
• K8S_PROFILE (generic or gke), HELM_VALUES_JSON (deployments/gke's helm_values) and/or HELM_VALUES_FILES, HELM_EXTRA_SET;
• K8S_PROBE_IMAGE, a curl image by digest that the smoke, watch and record stages read through.

The pipeline's resolve step needs a deployments/gke root. With TARGET=kubernetes the deployment's runtime root (RUNTIME_TF) must pin deployments/gke?ref=<commit>, the nistor commit the release lock was built from. A root that pins deployments/gcp stops it with exit 2, and a root pinning another commit, or none, with exit 3, which ALLOW_RUNTIME_PIN_DRIFT=1 turns into a warning. A cluster you bring yourself (K8S_CLUSTER=kubeconfig) without a deployments/gke root runs the template's pipeline only with ALLOW_RUNTIME_PIN_DRIFT=1 and some RUNTIME_TF file (that file must not pin deployments/gcp): resolve then only warns, and nothing checks the root against the lock. Otherwise install and upgrade it by hand with Helm (below).

The pipeline's cluster identity needs create, get, list, watch and delete on batch/jobs in the namespace and get on the CronJob compile; deployments/gke grants both (ClusterRole admin in the namespace), and on another cluster you bind them to the KUBECONFIG identity. A stage that starts a Job checks this first and stops with exit 2.

Each release runs these stages:

1. migrate: first the expand/contract gap check (exit 3 refuses the release), then the chart's migrate Job migrate-<12 hex> on an upgrade. On a first install there is no release yet, so the stage leaves the migration to the deploy stage, which installs with jobs.migrate.hook=true: the chart's pre-install hook, which brings its own environment and ServiceAccount.
2. deploy: a <role>-candidate track on the new digest beside the stable one.
3. probe: the chart's probe Job probe-<track>-<12 hex>, run inside the cluster against every role's readiness path and the backend's /health/installed.
4. canary: the Services select both tracks and the Gateway sends 10 percent to the candidate.
5. smoke.
6. promote: the stable track moves to the new digest and the candidates go. On GKE with IAP, promote also sets the backend-service audience once the Gateway has reconciled.
7. watch: readiness, restarts and the Gateway's 5xx rate.
8. record: the deployment record deployment.env with helm_revision_before and helm_revision_after.

TARGET=kubernetes refuses MARK_TESTED=1. The kind end-to-end in "Try it offline" runs the same stages.

Install by hand with Helm#

Verify the chart's signature first:

cosign verify registry.gitlab.com/nistor-io/nistor/charts/nistor@sha256:<chart digest> \
  --certificate-identity https://gitlab.com/nistor-io/nistor//.gitlab-ci.yml@refs/tags/v<version> \
  --certificate-oidc-issuer https://gitlab.com

helm pull oci://registry.gitlab.com/nistor-io/nistor/charts/nistor --version <version> prints the chart's digest. The identity is the release pipeline of the tag v<version>, as for the images (supply chain).

then install:

helm install nistor oci://registry.gitlab.com/nistor-io/nistor/charts/nistor --version <version> \
  -n <namespace> -f <your values file> \
  --set-string global.image.digest=sha256:<runtime digest> \
  --set-string global.frontendImage.digest=sha256:<shell digest> \
  --set jobs.migrate.hook=true

The migrate Job then runs as a pre-install and pre-upgrade hook, before the Deployments. Upgrade with helm upgrade and the same arguments. By hand there is no candidate track and no canary: the Deployments roll with Kubernetes' own rolling update, so read Upgrading first.

On GKE also pass -f values-gke.yaml (the file at the root of the chart: helm pull ... --untar) and the helm_values file. IAP's audience exists only after the Gateway has reconciled: read the ids of the two frontend backend services (gcloud compute backend-services list), then run helm upgrade again with the audiences of the frontend and the frontend-candidate, comma-separated. On the command line Helm splits on a comma, so escape it:

--set-string 'gke.backendServiceAudience=/projects/<number>/global/backendServices/<id>\,/projects/<number>/global/backendServices/<id>'

or put them in a values file, where the value is one string:

gke:
  backendServiceAudience: /projects/<number>/global/backendServices/<id>,/projects/<number>/global/backendServices/<id>

Until then browser logins are refused.

After the first install: let the roles write#

A role that calls the backend is a machine principal, identified by its subject, system:serviceaccount:<namespace>:<service account>. Without an iam_user row it may only read: the backend refuses its writes (403), as it does for an unregistered Google service account on Cloud Run. Once, after the first install, register each caller that writes (scheduler, jobs, compiler, which is also the compile CronJob's account, input-logs, runner, dq-rebuild, and mcp and workspace-events when enabled) from inside the backend pod:

kubectl -n <namespace> exec deploy/backend -- nistor iam register-service system:serviceaccount:<namespace>:scheduler

The command is idempotent, never makes an admin, and takes --permission <code> (repeatable) to grant a permission code as well. A registered service account is admin-equivalent infrastructure, governed by the chart's trust lists rather than by permission codes. Subjects are matched exactly: the same service account name in another namespace is refused. BOOTSTRAP_ADMIN_EMAIL stays a person's e-mail, never a service account. Never register backend: the chart leaves its subject read-only on purpose, so that a bare loopback cannot write as the backend. The frontend (transport only) and the pipeline's probe account (a fixed read-only pipeline principal) are never registered either. The kind end-to-end registers the scheduler this way (scripts/ci/k8s_e2e.sh).

Rollback#

The pipeline drops the candidate track, or runs helm rollback to the revision it recorded (helm_revision_before in deployment.env); a failed first install is uninstalled. By hand: helm rollback nistor <revision>. Rollback and uninstall leave the ServiceAccount migrate in place (Helm's resource policy keep), so a rollback to the first install's revision does not take the migrate Job's account away; delete it with the namespace.

The schema is never rolled back. Releases ship expands before contracts (the migrate stage refuses the opposite), so the previous code runs on the migrated schema.

Security notes#

• Identity. On Kubernetes the backend never trusts a plaintext IAP header: TRUST_IAP_HEADER_FALLBACK is false and a boot guard refuses it. Calls between roles carry a projected token with one audience per edge; the receiver accepts only RS256 and ES256, a matching issuer and audience, and a subject on its allow-list.
• The machine route. On GKE the Gateway has a second route to the backend's /api/ for machine callers outside the cluster. It removes these identity headers from the request: X-Goog-IAP-JWT-Assertion, X-Goog-Authenticated-User-Email, X-Goog-Authenticated-User-Id, X-Serverless-Authorization, X-User-Email, X-Forwarded-User, X-Nistor-Agent-Token, X-Nistor-Acting-For, X-Nistor-MCP-Tool and X-Nistor-MCP-Call-Id. Webhook callers on the machine hostname keep their own signature headers (X-Gitlab-Token, X-Webhook-Signature and the like): the sender authenticates itself with them. An MCP gateway that needs tool attribution calls the backend in-cluster, because the route removes X-Nistor-MCP-Tool.
• Workload Identity trusts the namespace name: whoever can create the namespace inherits the Google identities bound to it. Only the people who install Nistor may create its namespace.
• The backend can start any Job in its namespace. Its Role <release>-job-launcher grants create on batch/jobs, and RBAC cannot narrow create by a Job's image or service account, so a compromised backend pod could start a Job as any service account of the namespace. The chart cannot limit that; an admission policy of your own is the only limit. A shared cluster can enforce one with an admission controller; a single-tenant install ships none.
• NetworkPolicies. Every pod is denied all traffic except the call graph's edges, DNS, its database, the API server and, unless you set networkPolicy.restrictInternet, HTTPS to any address. restrictInternet needs networkPolicy.kubeApi.cidrs (on a cluster of your own, the API server's addresses): without them the API-server rule would open 443 and 6443 to every destination, so the chart refuses to render. Give the Google APIs a rule through networkPolicy.extraEgress. On the gke profile the policies also allow the metadata server that answers Workload Identity (169.254.169.254 on 80 and 8080, 169.254.169.252 on 987 and 988).
• Pods run as uid 10001, non-root, with a read-only root filesystem and the runtime default seccomp profile; images are referenced by digest only; the database password lives in a Secret, never in values.
• A registered service account is admin-equivalent (see "After the first install"): register only the roles above.

Known limits on Kubernetes#

• A long AI turn can lose its tool calls. A Kubernetes token does not carry the user the request acts for, so the backend refuses an acting-for header that arrives with one, and the AI loopback falls back to the forwarded browser headers; a turn that outlives the browser's assertion has its tool calls refused, and unattended routines get no tools. A per-user token is planned for a later release.
• The bigquery-streaming-monitor cron runs on the gke profile only.

Troubleshooting#

• Probes answer 400: an image older than the health-route exemption. The chart's probes send Host: <role>.
• A role refuses to start because POSTGRES_PORT is tcp://...: a Service named postgres injects it when service links are on. The chart turns service links off, so this means a pod that does not come from the chart.
• Candidate pods Pending: the namespace quota must leave room for one candidate per role.
• Browser logins refused on GKE right after the first install: the IAP audiences are set by promote once the Gateway has created both frontend backend services; a later release sets them if it had not yet.
• A role's writes answer 403 (the scheduler cannot run actions, for example): its subject is not registered. Run nistor iam register-service for it (see "After the first install").
• Creating, updating or rotating a connection with secret values answers 503 "Secret store unavailable": the instance has no persistent secret store (Secret Manager); a portable secret store is planned for a later release.
• too many clients already: the sum of the roles' pools and cron connections exceeds max_connections. Lower roles.<role>.replicas.max, or raise the instance's limit (see "The database connection budget").
• helm template refuses a values name: the chart sets or forbids it (the deployed signal, the trust lists). Set it through the value the chart offers for it.

Try it offline#

scripts/ci/k8s_e2e.sh installs, upgrades, releases and rolls back on kind with Postgres, the Pub/Sub emulator and fake-gcs-server, with no Google account: a first install, an upgrade that takes over the migrate hook's ServiceAccount, a release through the candidate track, a broken candidate rolled back, and a rollback to the first revision. It needs docker, kubectl and the nistor CLI on PATH, and the memory of a kind node running every role the open-source image serves: that image has no Data Quality runner, so its kind profile sets roles.runner.enabled: false. When a step fails once the cluster exists, it prints the namespace's pods, events and the descriptions and logs of the pods that are not ready before it removes the cluster. KIND_CLUSTER, KIND_REGISTRY_PORT and KIND_NAMESPACE move it off its defaults (nistor-e2e, 5001, nistor-e2e); --keep leaves the cluster running and writes its kubeconfig to KIND_STATE_DIR when set. The runtime image is built for linux/amd64 only, so the run needs a Docker daemon on a linux/amd64 host. A local Docker daemon keeps the registry and kind's API server on this machine's loopback; for a daemon on another host (a CI job's docker:dind service), set KIND_DOCKER_HOST_ADDRESS to the address its published ports are reached at, and the run publishes both there and forwards the registry to this machine's loopback for image signing. In this repository's CI the manual job k8s-e2e-kind runs it on main that way. On a host of another architecture (an arm64 laptop, for example) it stops at once; KIND_ALLOW_FOREIGN_ARCH=1 skips that check but does not enable emulation, so the run then fails at the first image pull.

Planned for a later release#

A Postgres-backed bus, S3 and Azure Blob object stores, a portable secret store, generic OIDC login, and the EKS and AKS profiles. Also on the list:

• the mcp role off Google: today, with NISTOR_MCP_BACKEND_AUDIENCE set it asks the Google metadata server for a token;
• Kubernetes acting-for: the backend mints a token per user through the TokenRequest API, the AI loopback sends it, and the receiver checks its subject and audience.