Docker monitoring stack

Includes

• victoria-metrics (Prometheus-compatible TSDB & scraper on :9090)
• vmalert (Prometheus alert & recording rule evaluator)
• grafana
• alertmanager
• loki
• node-exporter
• blackbox-exporter
• cadvisor
• pve-exporter (Prometheus Proxmox VE exporter on :9221)

Access logs

Traefik sends OTLP access logs to Loki on port 3100; Grafana provisions Loki as a datasource and loads the Traefik access analytics dashboard.

Web interfaces

Services are routed through Traefik using *.${HOST_SUFFIX}:

• Grafana: http://grafana.${HOST_SUFFIX}
• VictoriaMetrics & vmalert UI proxy: http://prometheus.${HOST_SUFFIX} (vmalert UI proxied under /vmalert)
• Alertmanager: http://alertmanager.${HOST_SUFFIX}

Prerequisites

• docker
• docker compose
• traefik with network

Steps

1. Create .env file

   GRAFANA_PASSWORD=SECURE_PASSWORD
   HOST_SUFFIX=xxx.local
   TZ=Europe/Paris
   TELEGRAM_BOT_TOKEN=123456:ABC-DEF
   TELEGRAM_CHAT_ID=-123456789
   PVE_TOKEN_VALUE=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
   
   # Optional: Grafana OAuth / SSO
   PUBLIC_HOST=example.com
   OPENID_URL=https://sso.example.com
   GRAFANA_CLIENT_SECRET=sso-client-secret
   

2. Deploy the stack

   docker compose up -d

Telegram notifications

The HTML template shows firing/resolved counts, severity, target labels, a short description and local timestamps. It displays up to six alerts within a 3,400-byte budget, followed by an omitted-alert count and a link to Alertmanager. Long fields are shortened before HTML escaping so markup stays valid. The assembled message uses safeHtml to preserve its formatting without escaping the fields twice.

Validate template changes in HTML mode, as used by the Telegram receiver, without sending notifications:

python3 scripts/test_telegram_template.py

After pulling template changes on the monitoring host, reload Alertmanager:

docker compose kill -s SIGHUP alertmanager

After modifying scrape configurations or alert rules, restart VictoriaMetrics and vmalert:

docker compose restart prometheus vmalert

Upstream alert rules

The fifteen imported rule files in conf/prometheus/alerts/ come from samber/awesome-prometheus-alerts (commit 822af1e59938844e9f32c9c2ff7cbe63e8438655, 2026-10-02).

Service	Local file under conf/prometheus/alerts/	Upstream path under dist/rules/	Alerts
ArgoCD	argocd.yml	argocd/embedded-exporter.yml	9
Blackbox	blackbox-exporter.yml	blackbox/blackbox-exporter.yml	10
cert-manager	cert-manager.yml	cert-manager/embedded-exporter.yml	4
Cilium	cilium.yml	cilium/embedded-exporter.yml	31
Prometheus	embedded-exporter.yml	prometheus-self-monitoring/embedded-exporter.yml	36
etcd	etcd.yml	etcd/embedded-exporter.yml	19
cAdvisor	google-cadvisor.yml	docker-containers/google-cadvisor.yml	10
Kubernetes	kubestate-exporter.yml	kubernetes/kubestate-exporter.yml	46
Loki	loki.yml	loki/embedded-exporter.yml	6
MySQL	mysqld-exporter.yml	mysql/mysqld-exporter.yml	18
Host and hardware	node-exporter.yml	host-and-hardware/node-exporter.yml	41
PostgreSQL	postgres-exporter.yml	postgresql/postgres-exporter.yml	25
Proxmox VE	proxmox-ve.yml	proxmox-ve/prometheus-pve-exporter.yml	9
SMART devices	smartctl-exporter.yml	s.m.a.r.t-device-monitoring/smartctl-exporter.yml	9
Traefik	traefik.yml	traefik/embedded-exporter-v2.yml	6

The existing alerts/*.yml configuration loads these files automatically. Traefik uses the upstream v2 metric family, rather than legacy v1 metrics. Each set requires its service/exporter metrics to be scraped; importing rules alone does not configure scrape targets. Validate all loaded rules with:

promtool check config conf/prometheus/prometheus.yml
promtool check rules conf/prometheus/alerts/*.yml

Keep these local cAdvisor overrides when refreshing upstream rules:

• ContainerKilled groups container_last_seen{id!="/",pod=""} by instance, name, container, and pod so recreating a Docker container does not alert while its replacement is running.
• ContainerHighMemoryUsage, ContainerLowMemoryUsage, and ContainerHighLowChangeCpuUsage group by instance, container, and pod.
• CadvisorFilesystemInodesUsage replaces upstream ContainerVolumeUsage, using host filesystem series, positive inode totals, and excluding synthetic none and overlay devices, including Talos overlays.
• ContainerHighThrottleRate keeps the duration guard and ten-minute hold described below.

Keep these local PostgreSQL and SMART overrides when refreshing upstream rules:

• PostgresqlNotEnoughConnections groups by instance, job, and datname, excludes temporary prisma_migrate_shadow_db_.* databases, and uses severity: info for low-traffic homelab databases.
• PostgresqlCommitRateLow excludes prisma_migrate_shadow_db_.* databases and uses severity: info.
• PostgresqlReplicationRoleChanged checks max_over_time(pg_replication_is_replica[1m]) != min_over_time(pg_replication_is_replica[1m]) instead of changes(...) > 0 for compatibility with VictoriaMetrics and Prometheus.
• SmartMediaErrors in smartctl-exporter.yml checks increase(smartctl_device_media_errors[1h]) > 0 so lifetime historical NVMe media error counts do not fire indefinitely unless new media errors occur.

Keep these local Proxmox VE, Cilium, and Traefik overrides when refreshing upstream rules:

• PveVm/ctDown excludes stopped or on-demand desktop guests (id!~"qemu/(100|102|103)" for Arch Linux, macOS, and Windows VMs).
• PveHighMemoryUsage scopes memory usage checks to PVE host nodes (id=~"node/.*"), preventing guest VM memory allocations from triggering host memory alerts.
• PveGuestNotBackedUp uses severity: info so unbacked development or ephemeral guests do not trigger Telegram alert noise.
• CiliumAgentHighDropRate groups by (instance, node, pod, reason) and excludes benign dropped traffic reasons (Policy denied, VLAN traffic disallowed by VLAN filter, Unsupported L3 protocol, and Service backend not found).
• TraefikServiceDown checks max by (instance, job, service) (traefik_service_server_up) == 0.
• TraefikHighHttp4xxErrorRateService groups by (instance, job, service), requires a request rate above 0.1 req/s, holds for 5m, and uses severity: info.

conf/alertmanager/alertmanager.tmpl routes PrometheusAlertmanagerE2eDeadManSwitch and severity="info" alerts to the null receiver so they remain visible in Prometheus/Alertmanager without sending Telegram noise or requiring permanent silences.

Keep locally authored rule files, including talos-nodes.yml, separate from upstream imports.

Preserve the local HostContextSwitchingHigh history guard in node-exporter.yml when refreshing host rules. It requires a counter sample from a day ago before comparing the 15-minute and daily rates. This avoids false spikes from newly scraped nodes while retaining detection after the baseline is established.

promtool test rules scripts/test_context_switching_rules.yml

cert-manager metrics

The cert-manager job scrapes 192.168.0.8:30101. The controller is pinned to talos-worker-core; a separate NodePort Service uses local endpoints to preserve the scraper IP, and an ingress policy permits only 192.168.0.6/32 on metrics port 9402. Controller egress, webhook and cainjector traffic are unchanged. If the core VM's Proxmox firewall filters NodePorts, permit TCP 30101 from the monitoring VM only. Deploy the ArgoCD changes before restarting VictoriaMetrics (docker compose restart prometheus), then verify up{job="cert-manager"} == 1 and the certificate readiness metrics.

Proxmox VE metrics

Proxmox VE metrics are collected via prometheus-pve-exporter scraping the PVE API at 192.168.0.3 using credentials defined in the environment:

• PVE_USER: monitoring@pve
• PVE_TOKEN_NAME: exporter
• PVE_TOKEN_VALUE: API token secret (configured in .env)
• PVE_VERIFY_SSL: false

The exporter listens internally on port 9221 and handles requests routed by VictoriaMetrics via the pve scrape job (/pve?target=192.168.0.3).

Cluster and edge services metrics

The scrape configuration monitors Talos cluster services via exposed NodePorts and internal endpoints:

• Traefik: scrapes metrics on 192.168.0.8:30102.
• ArgoCD: scrapes server and repo-server metrics on 192.168.0.8:30103 and 30104.
• Cilium: scrapes Cilium agents on Talos nodes (192.168.0.7:9962, 192.168.0.8:9962, 192.168.0.9:9962) and operator on 192.168.0.8:9963.
• etcd: scrapes etcd on talos-master-0 (192.168.0.7:2381).
• Loki & vmalert: scraped directly via internal Docker container networking (loki:3100, vmalert:8880).

CPU throttling alerts

ContainerHighThrottleRate requires more than 25% of scheduling periods to be throttled and more than 100 ms of accumulated throttled time per second, sustained for 10 minutes. Throttled time measures scheduler delays, not application latency. This filters frequent tiny bursts; review application latency before changing CPU limits. Missing throttled-duration metrics cannot satisfy the alert.

Validate the rule and its eight regression scenarios with promtool:

promtool check rules conf/prometheus/alerts/google-cadvisor.yml
promtool test rules scripts/test_cadvisor_rules.yml

Talos node memory and OOM history

The node-exporter ArgoCD application runs on all three Talos nodes, including the control plane and Freebox worker. It reads host /proc and /sys through read-only mounts, without host networking, host PID access or a host root mount. The NodePort service uses port 30100 and externalTrafficPolicy: Local so each node address reports its own metrics. Its ingress policy allows only the monitoring VM, 192.168.0.6.

The node_talos scrape job attaches the Kubernetes node name. Existing HostOutOfMemory and HostOomKillDetected rules cover these targets; additional rules detect sustained memory PSI stalls and missing memory/OOM/PSI collectors. Host OOM counters include kills within container memory limits as well as host memory exhaustion; correlate them with pod OOM events before assigning a cause.

Use these recorded series for history:

talos_node:memory_available:ratio
talos_node:memory_stall:ratio5m
talos_node:oom_kills:increase1h

Validate the new recording and alert rules:

promtool check rules conf/prometheus/alerts/talos-nodes.yml
promtool test rules scripts/test_talos_node_rules.yml

Deploy the exporter application first and verify its DaemonSet is ready on all three nodes. Then update and restart VictoriaMetrics on the monitoring host (docker compose restart prometheus). Verify up{job="node_talos"} and node_scrape_collector_success{job="node_talos"}, confirm node_uname_info matches each target's node label, and check all three nodes expose node_vmstat_oom_kill and node_pressure_memory_waiting_seconds_total. History starts with the first successful scrape; past incidents cannot be filled in retroactively.