Monitoring and alerts — Nistor Docs

The monitoring module defines 16 Cloud Monitoring alert policies and the log-based metrics they read. Each policy is named Nistor - <what> - <environment>.

Turn them on#

The reference root creates the monitoring module only when both of these are set:

enable_monitoring = true (the default), and
notification_channel_id is a Cloud Monitoring notification channel. It is empty by default, and while it is empty no alert policy is created.

Create the channel first (e-mail, a chat webhook or a paging service), then pass its id:

gcloud beta monitoring channels create --display-name="Nistor on-call" \
  --type=email --channel-labels=email_address=ops@example.com --project=<project>
gcloud beta monitoring channels list --project=<project> --format="value(name)"

notification_channel_id = "projects/<project>/notificationChannels/<id>"

Three policies the reference root does not create#

The module creates these three only from inputs that the reference root does not pass. With the root as shipped, they do not exist:

Policy	Created when
Cloud SQL connection saturation	`cloud_sql_connection_name` is set
Compiler Job Failed	`compiler_job_name` is set
Compiled Graph Stale	`compiler_job_name` is set and `enable_compiled_graph_stale_alert` is `true`

To have them, pass those inputs in your copy of the root's module "monitoring" block: the database's connection name (the root's output db_connection_name) and the compile job's name (the compiler module's compile_job_name). Leave enable_compiled_graph_stale_alert off in an environment whose scheduled compile runs less than once an hour.

The policies#

Policy (`Nistor - … - <env>`)	Fires when	First action
Backend Service Errors	Error logs from the backend persist for 15 minutes	Read the backend's error logs in Cloud Logging; look for a failing dependency (database, a tool's API)
Backend 5xx Response Rate	The backend answers 5xx in every minute for 15 minutes, or more than 200 times in 5 minutes	Check the latest revision; if the errors started with a deploy, roll the images back
Backend Event Loop Stalled	The backend's event loop is blocked more than 25% of the time for 3 minutes	Look for slow synchronous work in the logs around that time; check the backend's CPU and memory
Backend Dropped Scheduling Work	Rules are dropped from the active set, leaving nodes unscheduled, above an hourly threshold	Check the rules named in the backend logs, and the compiled graph they target
Backend Incident Write Rejected	An incident could not be written because the database refused it	Read the backend log entry for the rejected write; it is a bug to report with that entry
PA Read Poll Failures	MicroStrategy's usage read fails on every hourly run for 3 hours	Check the MicroStrategy connection's credentials and that the usage cube is reachable
Scheduler Service Errors	Error logs from the scheduler persist for 5 minutes	Read the scheduler's logs; check that the backend answers
Compiler Service HTTP Errors	The compiler answers with a server error	Read the compiler's logs for the failing request
Input-Logs Service Errors	Error logs from input-logs persist for 5 minutes	Read its logs; check the log sink and its subscription
`<name>` Subscription Backlog	A Pub/Sub subscription's oldest unacknowledged message is older than 300 seconds (one policy per subscription: Events Scheduler, and Log Sink when input-logs is on)	Check that the consuming service is up and not failing on one message
Cloud Scheduler Job Failures	Scheduled jobs keep failing over 5 minutes	Find the failing job in Cloud Scheduler, then read the target service's logs
`<service>` Memory Pressure	A service's p99 memory stays above 85% for 5 minutes (one policy per service)	Raise that service's memory, or find what grew
Cloud Run OOM kills	One service is killed for running out of memory 3 times or more in 15 minutes	Raise that service's memory; see Sizing
Cloud SQL connection saturation	The database has more than 70% of `max_connections` in use for 2 minutes	Lower the services' pool sizes or instance counts, or move to a bigger database tier
Compiler Job Failed	The scheduled compile has 2 or more failed attempts within 2 hours	Read the compile job's logs; the last good graph stays in use meanwhile
Compiled Graph Stale	No scheduled compile has published a graph for 3 hours	Check that the compile job runs, and that it succeeds

The thresholds are the module's defaults. Several of them are inputs of the monitoring module (memory_utilization_threshold_ratio, memory_utilization_duration_seconds, cloud_sql_max_connections, cloud_sql_saturation_threshold_ratio, cloud_sql_saturation_duration_seconds).

Health endpoints#

Every service answers GET /health (it is running) and GET /ready (it can serve). GET /health/installed lists the core version, the image's lock digest and every installed tool and plugin with its status. It answers 503 when an activated tool or plugin is not ok. GET /health/installed/status is the short form: ok or not_ok and the lock digest. Cloud Run's run.invoker permission still applies to every one of them.