Troubleshooting

Each entry starts with the message as Nistor prints it. Search this page for a few words of what you see. The messages are in the logs of the service that failed (Cloud Logging on Google Cloud, docker compose logs <service> locally).

The backend does not start#

SECURITY: BOOTSTRAP_ADMIN_EMAIL is not set in environment '<env>'. … refusing to boot.#

• Cause. A backend on Cloud Run, or in test or prod, needs a first administrator. Without one, nobody could ever administer the instance.
• Fix. Set the root's bootstrap_admin_email at plan time (TF_VAR_bootstrap_admin_email=<your e-mail>), then apply.
• Confirm. The new revision becomes ready, and that account signs in as an administrator.

SECURITY: AUTO_REGISTER_EMAIL_DOMAINS is not set in environment '<env>'. …#

• Cause. In test and prod, the backend refuses to register sign-ins from any domain.
• Fix. Set auto_register_email_domains to your organisation's domains, comma-separated ("example.com,example.org").
• Confirm. The revision becomes ready. A first sign-in from one of those domains creates a user with no permissions.

SECURITY: PROXY_IAP_AUDIENCE is not set in environment '<env>'. … refusing to boot.#

• Cause. trust_iap_header_fallback is false, so the backend must verify the IAP assertion, but it was given no audience to verify it against.
• Fix. Use the reference root, which passes the frontend's IAP audience to the backend. If you have more than one frontend in front of one backend, list them all (the backend module's extra_proxy_iap_audiences). Or set trust_iap_header_fallback = true again while you fix it.

SECURITY: neither CORS_CHROME_EXTENSION_ID nor CORS_ORIGIN_REGEX is set in non-dev environment '<env>'. …#

• Cause. In prod, the backend needs to know which browser extension, if any, may call it.
• Fix. The backend module sets CORS_ORIGIN_REGEX. If you replaced the backend's environment, set CORS_CHROME_EXTENSION_ID to your Chrome extension's id, or set CORS_ORIGIN_REGEX (role_extra_env_vars).

A role is not ready: /ready answers 503 with "reason": "distribution_unreadable"#

• Cause. The image's /etc/nistor/distribution.json is missing or invalid, so the role cannot tell which tools and plugins it should activate. It refuses to serve rather than run with the core alone.
• Fix. Run an image built by nistor dist build, unchanged. Outside an image, set NISTOR_DEV=1 for local work only.

DATABASE SCHEMA IS BEHIND THE CODE for chain <chain>: … — run: nistor migrate.#

• Cause. The services run a newer release than the database: the migrations did not run. /ready answers 503 with the reason schema_behind.
• Fix. Execute the migrate job (the root's output migrate_job_name), or docker compose up migrate locally.
• Confirm. The log line is gone at the next start, and /ready answers 200.

Migrations refuse to run#

REFUSING TO MIGRATE A LIVE DATABASE#

• Cause. nistor migrate was run outside the migrate job, against a deployed database (a Cloud SQL socket, or the database name every deployed instance uses). This is a guard against migrating production from a laptop.
• Fix. Use the migrate job. If you really mean to migrate that database from where you are, set NISTOR_ALLOW_LIVE_MIGRATION=1 for that one command.

Refused: a newer release migrated this database (… holds … revisions this release does not ship). Run the newer release (its image, its nistor migrate); a release never migrates back.#

• Cause. You are running an older release's migrate against a database that a newer release has already migrated, for example by deploying the older release again to roll back.
• Fix. Migrations only move forward, so redeploying an older release is not a rollback. Keep the newer schema and either run the newer release again, or send the traffic back to the previous revisions without running the older migrate. To return to the older schema, restore the backup taken before the upgrade, together with the previous images (see Upgrading and Backup and restore).

another migrate holds the migrate lock on this database (waited lock_timeout): …#

• Cause. Two migrations started at once. Only one may run.
• Fix. Wait for the other one to finish, then run again. If its host is gone, the database drops its session within about 90 seconds. Otherwise end that session with pg_terminate_backend.

❌ Refused: release <release> would apply <chain>:<revision> before an earlier release ran the expand revisions they need …#

• Cause. nistor migrate --check-gap (exit code 3) found that the release removes or moves something in the database before any release you ran added its replacement. This happens when you skip a release.
• Fix. Upgrade through the release that carries those expand revisions first, then to the target. See Upgrading.

no database: set DATABASE_URL or POSTGRES_SERVER, POSTGRES_DB, …#

• Cause. The process has no database settings.
• Fix. Set DATABASE_URL, or all of POSTGRES_SERVER, POSTGRES_DB, POSTGRES_USER and POSTGRES_PASSWORD. The message lists the ones that are missing.

Signing in and API calls#

IAP shows a 403 page before Nistor loads#

• Cause. You hold no IAP access on the frontend service.
• Fix. An administrator grants you roles/iap.httpsResourceAccessor (see Identity and access).

API calls answer 401, and the frontend logs REJECT /api: no IAP identity header#

• Cause. IAP is not in front of the frontend service, so requests carry no identity.
• Fix. Turn IAP on for the frontend (the root's apply does this), then check it with gcloud iap web get-iam-policy --resource-type=cloud-run --service=<frontend> ….

Signed in, but every page says a permission is missing#

• Cause. New users are registered with no permissions.
• Fix. An administrator gives the user permissions, or applies a role template, under /api/v1/iam or in the web shell's user administration (Platform → IAM Admin, see First sign-in).

Pages say they failed to load, and API calls answer 503 Read replica is not configured#

• Cause. The read-only endpoints that the web shell's pages use read from a read replica. When no replica is set (READ_REPLICA_POSTGRES_SERVER empty), they answer 503 rather than load the primary database.
• Fix. The compose files set READ_REPLICA_FALLBACK_TO_PRIMARY=true from 1.0.0-rc.1. If you run your own stack, point READ_REPLICA_POSTGRES_SERVER (and the other READ_REPLICA_* settings) at a replica or, without one (a local stack, a small environment), set READ_REPLICA_FALLBACK_TO_PRIMARY=true on the backend so that those endpoints read the primary.

Tools and connections#

A connection test answers unhealthy#

• Cause. The tool could not reach its service, or the credentials were refused. The detail of the test names which.
• Fix. See the tool's page under Integrations: each one lists the fields, the permissions it needs and what its test checks.

The MicroStrategy SDK (mstrio-py) is not installed: MicroStrategy discovery needs it. …#

• Cause. The public image does not ship MicroStrategy's SDK.
• Fix. Build your own image with it. See Optional extras.

Infrastructure#

A deploy fails on a Cloud Run quota#

• Cause. The reference root's defaults (8 vCPU per compiler instance, 8Gi per backend instance, old and new revisions side by side during a deploy) can exceed a new project's default Cloud Run quotas.
• Fix. Lower compiler_max_instances and backend_memory, or ask Google for more quota. See Scaling.

Containers of the local stack exit with code 137#

• Cause. Docker ran out of memory. The local stack runs PostgreSQL, five backend roles and the web shell together.
• Fix. Give Docker more memory (Docker Desktop: Settings → Resources), then run docker compose up again. docker stats shows how much memory each container of the stack uses.

Enterprise modules#

Licence messages of the Enterprise modules are covered in the Enterprise edition's troubleshooting page.