Monitoring

Metrics, logs, dashboards and alerts for your install.

With monitoring.enabled the helmfile installs kube-prometheus-stack (Prometheus, Alertmanager, Grafana, node-exporter, kube-state-metrics) and, with monitoring.logs, Loki and Alloy. Grafana is at https://<domains.grafana>; the admin password is in monitoring/grafana-admin.

​

What is scraped

Source	How	Used for
Nodes, pods, kube-state-metrics	kube-prometheus-stack defaults	Capacity, restarts, OOM kills
metrics-server	Kubernetes metrics API (not Prometheus)	HPAs of user apps; billing CPU/memory
Traefik	ServiceMonitor, router and service labels	Requests, latency, bytes per app
CloudNativePG	PodMonitor atoll-db	Replication lag, WAL archiving, connections
cert-manager	ServiceMonitor	Certificate expiry
Prometheus picks up every ServiceMonitor and PodMonitor in the cluster
(serviceMonitorSelectorNilUsesHelmValues: false).

​

Who can reach it

NetworkPolicies in monitoring (chart charts/network-policies) admit only Grafana, Prometheus, Alertmanager and Alloy to the monitoring pods, the ingress controller to Grafana, and atoll-server to Prometheus (billing). Loki runs without multi-tenancy (auth_enabled: false), which is fine only because nothing outside the monitoring components can reach it; app and build pods are also stopped by their own egress policies. Grafana is the one entry point, behind its admin login.

​

Billing data

Billing counts the CPU and memory running app pods request (their size, see Scaling) and reads bandwidth from Prometheus at ATOLL_BILLING_PROMETHEUS_URL. Apps that scale to zero are served through their w--<app> Service, which billing maps to the same project. Traefik labels its metrics with the Kubernetes service of each app Ingress, for example:

sum by (service) (increase(traefik_service_responses_bytes_total{service=~"atoll-apps-.*"}[1h]))
sum by (service) (increase(traefik_service_requests_bytes_total{service=~"atoll-apps-.*"}[1h]))

service looks like atoll-apps-<project>--<module>-80@kubernetes. Web apps are served by atoll-server, which reports their bytes itself.

​

App metrics

The panel’s Metrics tab (GET /api/projects/<project>/metrics?module=&preview=&range=1h|24h|7d|30d, members) reads Prometheus at ATOLL_METRICS_PROMETHEUS_URL (unset: ATOLL_BILLING_PROMETHEUS_URL; neither: the tab says metrics are unavailable). For module shop--api in atoll-apps, with a window W of the range’s step but at least 2m:

Metric	Source	PromQL
CPU (cores)	cAdvisor	sum(max by (pod) (rate(container_cpu_usage_seconds_total{namespace="atoll-apps",pod=~"shop--api-[a-z0-9]+-[a-z0-9]+",container="app"}[W])))
Memory (bytes)	cAdvisor	the same with avg_over_time(container_memory_working_set_bytes{…}[W])
Replicas	kube-state-metrics	max(max_over_time(kube_deployment_status_replicas_available{namespace="atoll-apps",deployment="shop--api"}[W]))
Requests/s	Traefik	sum(rate(traefik_service_requests_total{service=~S}[W]))
p50 / p95 latency	Traefik	histogram_quantile(0.95, sum by (le) (rate(traefik_service_request_duration_seconds_bucket{service=~S}[W])))
Error rate	Traefik	(sum(rate(…{service=~S,code=~"5.."}[W])) or vector(0)) / sum(rate(…{service=~S}[W]))

S matches the module’s Service, its wake Service (w--shop--api, while it scales to zero) and the custom-domain Services leading to it, e.g. atoll-apps-(w--)?shop--api-[a-z0-9]+@kubernetes|atoll-domains-shop--domain-…. Traefik’s latency buckets are set finer than its defaults in values/traefik.yaml.gotmpl for useful percentiles. Answers are cached on the server (15 seconds for the last hour, longer for longer ranges).

​

Logs

Alloy tails every pod through the Kubernetes API and ships to Loki with labels namespace, pod, container and atoll_project (from the atoll.app/project label of app and build pods). In Grafana → Explore → Loki:

{namespace="atoll"} |= "ERROR"
{namespace="atoll-apps", atoll_project="hello"}
{namespace="atoll-builds"} |= "error:"

​

Alerts worth adding

kube-prometheus-stack ships general Kubernetes alerts. Add PrometheusRules for the platform:

Alert	Expression (sketch)
Server down	kube_deployment_status_replicas_available{namespace="atoll",deployment="atoll-server"} < 1
Postgres not archiving	cnpg_pg_stat_archiver_seconds_since_last_archival > 600
Postgres replica lag	cnpg_pg_replication_lag > 60
Certificate expiring	certmanager_certificate_expiration_timestamp_seconds - time() < 7*24*3600
Builds waiting for capacity	sum(kube_pod_status_phase{namespace="atoll-builds",phase="Pending"}) > 0 for 10m
Storage filling	kubelet_volume_stats_available_bytes / kubelet_volume_stats_capacity_bytes < 0.15

Route Alertmanager to email or chat through kube-prometheus-stack values (alertmanager.config) in your site file’s release overrides.

​

Without the stack

Turning monitoring.enabled off skips Prometheus, Grafana and Loki. metrics-server stays (HPAs need it), but billing has no bandwidth source then.