AI key management

Terraform wiring that lets an ai:admin operator set and rotate the OpenAI and Gemini API keys from the AI admin Keys tab, without a redeploy. Keys are stored encrypted at rest in the app DB (DB-first, with the env keys as fallback), so nothing breaks before adoption. One flag gates everything and it defaults to false, so terraform plan is a no-op until an operator opts in:

This is the same additive, code-gated pattern already used for Sifflet (sifflet_base_url), Castor (castor_api_url), and Agent Governance (enable_agent_governance): a disabled flag means the secret + IAM resource are never created (count = 0) and the env var is never emitted.

Why an encryption key is needed#

The gateway picks a cipher for DB-stored keys in packages/integrations/ai_assistant/.../key_cipher.py::resolve_key_cipher(), in order: Cloud KMS (AI_KMS_KEY_REF) then app-level Fernet (AI_KEY_ENCRYPTION_SECRET) then None. With neither set, the resolver returns None: GET /api/v1/ai/providers reports key_management_available: false, the Keys tab shows the "key management unavailable" notice with the set/rotate buttons disabled, and the assistant keeps using the AI_OPENAI_API_KEY / AI_GEMINI_API_KEY env keys. Flipping this flag supplies the Fernet tier, so cipher_mode becomes "local" and the UI unlocks.

Rollout sequence#

With generate_secrets = true (a first install), there is no hand step. module.core writes the Fernet key's first version itself (32 random bytes, URL-safe base64, as Fernet requires) and its secret-id output waits for that version and the reader grant, so one apply creates the secret, the version and the backend and jobs revisions in order: a tenant that turns the feature on needs no step 1 or 2 below. The key is then in the Terraform state (marked sensitive): the state bucket is as secret as the key. Turn generate_secrets on only at the first install; on a deployment whose key was added by hand, the generated version would become latest and every stored provider key would become undecryptable.

The steps below are the hand path, for generate_secrets = false (the default, today's deployments):

Run this against the deploying root (the deployment repository's terraform/), once per environment. All three tfvars (dev, test, prod) now carry enable_ai_key_encryption = true, so do steps 1 and 2 for an environment before the flag reaches a full apply for that environment.

1. Provision the (empty) secret first, two-phase apply. AI_KEY_ENCRYPTION_SECRET is a Secret Manager reference with version = "latest", which Cloud Run resolves at revision-deploy time. If the secret creation and the backend/jobs env wiring land in the same apply, the secret has zero versions when Cloud Run deploys the new revision and the deploy fails (a real outage risk on an already-running backend). Create only the secret + IAM first, with a targeted apply:

   terraform apply -target=module.core -var-file=dev.tfvars
   

   module.backend / module.jobs are not dependencies of module.core, so they are untouched at this point.

2. Populate the secret (out-of-band, so with generate_secrets = false the cleartext never enters tfvars or state). The value MUST be a Fernet key, exactly 32 url-safe base64 bytes. Generate it with Fernet.generate_key(), NOT openssl rand (an arbitrary secret fails Fernet(...) construction, the cipher degrades to None, and the Keys tab stays locked):

   python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())" \
     | gcloud secrets versions add ai-key-encryption-secret-dev \
         --project=example-dataplatform-dev --data-file=-
   

3. Full apply / redeploy. Now that a version exists, apply without -target (or let CI apply normally):

   terraform apply -var-file=dev.tfvars
   

   This deploys new backend and jobs service revisions carrying the now-resolvable AI_KEY_ENCRYPTION_SECRET secret ref.

4. Verify (see below), then repeat 1 to 3 for test and prod, swapping the -var-file, the secret name suffix, and the --project:

   env	secret name	project
   dev	ai-key-encryption-secret-dev	example-dataplatform-dev
   test	ai-key-encryption-secret-test	example-dataplatform-test
   prod	ai-key-encryption-secret-prod	example-dataplatform-prod

Verify#

After the full apply for an environment:

• GET /api/v1/ai/providers (as an ai:admin) returns key_management_available: true and cipher_mode: "local".
• In the AI admin Keys tab, the "unavailable" notice is gone and the Set / Replace buttons for OpenAI and Gemini are enabled. Saving a key flips that provider's status from "Configured via env" to "Configured via DB" (encrypted at rest, only last4 shown).

Rollback#

Set enable_ai_key_encryption = false for the environment and re-apply. The env var and the count-gated secret + IAM are removed; any keys already stored in the DB stay encrypted but unused, and the gateway falls back to the AI_OPENAI_API_KEY / AI_GEMINI_API_KEY env keys. Rotating the Fernet key (adding a new secret version) makes previously stored DB keys undecryptable, so re-enter them from the UI after a rotation.

KMS alternative (not used here)#

Setting AI_KMS_KEY_REF to a Cloud KMS crypto-key resource id takes precedence over the Fernet key (cipher_mode: "kms"), giving envelope encryption instead of an app-held key. That path needs nistor_tool_gcp installed in the image and a provisioned KMS key ring / crypto key, and is out of scope for this flag-driven Fernet rollout.

Operator notes#

• Both backend and jobs get the key. The env var is emitted from local.backend_extra_env_vars, which feeds module.backend AND module.jobs. The gateway resolves DB keys from whichever service runs it (the API on the backend, the investigation worker / routine driver on the jobs service), so both need the secret.
• Set it in your deployment repository. deployments/gcp/ is the reference composition: it deploys nothing by itself; a deployment repository sources it pinned to a commit or tag (//deployments/gcp?ref=<tag>) and CI applies that repository's terraform/. Set enable_ai_key_encryption = true in that repository's per-environment tfvars.
• Ordering vs CI. Once the flag is true in an environment's tfvars, a full apply that runs before the matching secret version exists will fail the backend/jobs deploy. Populate each environment's secret (steps 1 and 2) before that environment's full apply / CI run.