Docker monitoring stack

Includes

• victoria-metrics (Prometheus-compatible TSDB & scraper on :9090)
• vmalert (Prometheus alert & recording rule evaluator)
• grafana
• alertmanager
• VictoriaLogs (logs, 7-day retention)
• node-exporter
• blackbox-exporter
• cadvisor
• pve-exporter (Prometheus Proxmox VE exporter on :9221)

Access logs

Traefik sends OTLP access logs to VictoriaLogs on LAN port 3100 (container port 9428). The Kubernetes victoria-logs-collector Application in argocd-apps collects container logs on every Talos node, with persistent checkpoints and a 600 MiB retry buffer per node. Grafana installs the VictoriaLogs datasource plugin and loads the Traefik access analytics dashboard. Logs remain searchable after a Job or Pod is deleted.

VictoriaLogs stores new data in ./data/victorialogs, with 7-day retention and a 10 GiB retention target (oldest daily partitions are removed when exceeded). It stops ingestion below 1 GiB of free disk. The HTTP listener is published only on the monitoring VM LAN address, 192.168.0.6:3100, with no public ingress. The HTTP API has no built-in authentication; retain the VM firewall restriction to trusted LAN/cluster sources. Loki history is not imported or deleted.

Open the Kubernetes Logs dashboard (/d/kubernetes-logs) for namespace, node, workload type/name, optional pod and container selectors, message search and the latest 1,000 log lines. Workload ownership is resolved using historical kube_pod_owner, kube_replicaset_owner and kube_job_owner metrics in VictoriaMetrics, including the Deployment → ReplicaSet → Pod and CronJob → Job → Pod relationships. The dashboard includes a Newsbot shortcut and also works for deleted pods. Rust tracing messages in fields.message are mapped at collection time; the dashboard also recovers this field when displaying older placeholder entries. Log lines show namespace/pod, container and the available HTTP method, path, status, latency and request ID. Expand Détails structurés — JSON complet for complete structured records (up to 200 lines), or expand any log line.

Alternatively, search in Grafana Explore using the VictoriaLogs datasource:

kubernetes.pod_namespace:=misc kubernetes.pod_name:~"newsbot-cronjob-.*"

Deployment order: stop the old Loki container (docker stop loki) to free LAN port 3100, start VictoriaLogs and update Grafana, then sync the collector and Traefik Applications. The old Loki data directory is retained.

Web interfaces

Services are routed through Traefik using *.${HOST_SUFFIX}:

• Grafana: http://grafana.${HOST_SUFFIX}
• VictoriaMetrics & vmalert UI proxy: http://prometheus.${HOST_SUFFIX} (vmalert UI proxied under /vmalert)
• Alertmanager: http://alertmanager.${HOST_SUFFIX}

Container memory limits

Every running container on the monitoring VM has a Docker cgroup memory limit. memswap_limit equals mem_limit, so the containers cannot use swap. These are ceilings, not reserved allocations. The combined limits are 1,328 MiB, leaving about 143 MiB of the VM's 1,471 MiB usable RAM for the host. Normal usage should remain below the combined ceilings; review these budgets if workloads grow.

Container	Memory limit
VictoriaMetrics (prometheus)	384 MiB
Grafana	320 MiB
VictoriaLogs	192 MiB
cAdvisor	96 MiB
vmalert	64 MiB
Alertmanager	64 MiB
Proxmox exporter	64 MiB
Traefik (separate Compose project)	64 MiB
Blackbox exporter	48 MiB
Node exporter	32 MiB

VictoriaMetrics uses -memory.allowedBytes=64MiB to size its internal caches, GOMEMLIMIT=128MiB as a soft Go runtime memory target, and a 32 MiB per-query memory limit with at most two concurrent queries. The cache setting is not a total process memory limit: mmap caches and other allocations also need headroom under the 384 MiB Docker hard limit. Queries exceeding their memory budget are rejected instead of being allowed to grow without a per-query limit. VictoriaLogs uses a 64 MiB cache budget. Grafana's Go runtime uses GOMEMLIMIT=128MiB as a soft GC target below its Docker hard limit.

Traefik's base Compose file is managed separately on the VM. Install the tracked resource override beside it, then apply it without pulling a different image:

cp conf/traefik/docker-compose.override.yml ../traefik/docker-compose.override.yml
docker compose -f ../traefik/docker-compose.yml -f ../traefik/docker-compose.override.yml config -q
docker compose -f ../traefik/docker-compose.yml -f ../traefik/docker-compose.override.yml up -d --pull never traefik

For the monitoring services, use docker compose up -d --pull never. Verify the effective values with docker inspect (HostConfig.Memory and HostConfig.MemorySwap), and check health, restart counts and State.OOMKilled after deployment.

Prerequisites

• docker
• docker compose
• traefik with network

Steps

1. Create .env file

   GRAFANA_PASSWORD=SECURE_PASSWORD
   HOST_SUFFIX=xxx.local
   TZ=Europe/Paris
   TELEGRAM_BOT_TOKEN=123456:ABC-DEF
   TELEGRAM_CHAT_ID=-123456789
   PVE_TOKEN_VALUE=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
   
   # Optional: Grafana OAuth / SSO
   PUBLIC_HOST=example.com
   OPENID_URL=https://sso.example.com
   GRAFANA_CLIENT_SECRET=sso-client-secret
   

2. Deploy the stack

   docker compose up -d

Telegram notifications

The HTML template shows firing/resolved counts, severity, target labels, a short description and local timestamps. It displays up to six alerts within a 3,400-byte budget, followed by an omitted-alert count and a link to Alertmanager. Long fields are shortened before HTML escaping so markup stays valid. The assembled message uses safeHtml to preserve its formatting without escaping the fields twice.

Validate template changes in HTML mode, as used by the Telegram receiver, without sending notifications:

python3 scripts/test_telegram_template.py

After pulling template changes on the monitoring host, reload Alertmanager:

docker compose kill -s SIGHUP alertmanager

After modifying scrape configurations or alert rules, restart VictoriaMetrics and vmalert:

docker compose restart prometheus vmalert

Upstream alert rules

The fourteen imported rule files in conf/prometheus/alerts/ come from samber/awesome-prometheus-alerts (commit 822af1e59938844e9f32c9c2ff7cbe63e8438655, 2026-10-02).

Service	Local file under conf/prometheus/alerts/	Upstream path under dist/rules/	Alerts
ArgoCD	argocd.yml	argocd/embedded-exporter.yml	9
Blackbox	blackbox-exporter.yml	blackbox/blackbox-exporter.yml	10
cert-manager	cert-manager.yml	cert-manager/embedded-exporter.yml	4
Cilium	cilium.yml	cilium/embedded-exporter.yml	31
Prometheus	embedded-exporter.yml	prometheus-self-monitoring/embedded-exporter.yml	36
etcd	etcd.yml	etcd/embedded-exporter.yml	19
cAdvisor	google-cadvisor.yml	docker-containers/google-cadvisor.yml	10
Kubernetes	kubestate-exporter.yml	kubernetes/kubestate-exporter.yml	46

| MySQL | mysqld-exporter.yml | mysql/mysqld-exporter.yml | 18 | | Host and hardware | node-exporter.yml | host-and-hardware/node-exporter.yml | 41 | | PostgreSQL | postgres-exporter.yml | postgresql/postgres-exporter.yml | 25 | | Proxmox VE | proxmox-ve.yml | proxmox-ve/prometheus-pve-exporter.yml | 9 | | SMART devices | smartctl-exporter.yml | s.m.a.r.t-device-monitoring/smartctl-exporter.yml | 9 | | Traefik | traefik.yml | traefik/embedded-exporter-v2.yml | 6 |

The existing alerts/*.yml configuration loads these files automatically. Traefik uses the upstream v2 metric family, rather than legacy v1 metrics. Each set requires its service/exporter metrics to be scraped; importing rules alone does not configure scrape targets. Validate all loaded rules with:

promtool check config conf/prometheus/prometheus.yml
promtool check rules conf/prometheus/alerts/*.yml

Keep these local cAdvisor overrides when refreshing upstream rules:

• ContainerKilled groups container_last_seen{id!="/",pod=""} by instance, name, container, and pod so recreating a Docker container does not alert while its replacement is running.
• ContainerHighMemoryUsage, ContainerLowMemoryUsage, and ContainerHighLowChangeCpuUsage group by instance, container, and pod.
• CadvisorFilesystemInodesUsage replaces upstream ContainerVolumeUsage, using host filesystem series, positive inode totals, and excluding synthetic none and overlay devices, including Talos overlays.
• ContainerHighThrottleRate keeps the duration guard and ten-minute hold described below.

Keep these local PostgreSQL and SMART overrides when refreshing upstream rules:

• PostgresqlNotEnoughConnections groups by instance, job, and datname, excludes temporary prisma_migrate_shadow_db_.* databases, and uses severity: info for low-traffic homelab databases.
• PostgresqlCommitRateLow excludes prisma_migrate_shadow_db_.* databases and uses severity: info.
• PostgresqlConfigurationChanged copies each metric name into a setting label before comparing it with its value five minutes earlier. This preserves setting identity with VictoriaMetrics, whose binary operators drop metric names. Regression check: promtool test rules scripts/test_postgres_rules.yml.
• PostgresqlReplicationRoleChanged checks max_over_time(pg_replication_is_replica[1m]) != min_over_time(pg_replication_is_replica[1m]) instead of changes(...) > 0 for compatibility with VictoriaMetrics and Prometheus.
• SmartMediaErrors in smartctl-exporter.yml checks increase(smartctl_device_media_errors[1h]) > 0 so lifetime historical NVMe media error counts do not fire indefinitely unless new media errors occur.

Keep these local Proxmox VE, Cilium, and Traefik overrides when refreshing upstream rules:

• PveVm/ctDown excludes stopped or on-demand desktop guests (id!~"qemu/(100|102|103)" for Arch Linux, macOS, and Windows VMs).
• PveHighMemoryUsage scopes memory usage checks to PVE host nodes (id=~"node/.*"), preventing guest VM memory allocations from triggering host memory alerts.
• PveGuestNotBackedUp uses severity: info so unbacked development or ephemeral guests do not trigger Telegram alert noise.
• CiliumAgentHighDropRate groups by (instance, node, pod, reason) and excludes benign dropped traffic reasons (Policy denied, VLAN traffic disallowed by VLAN filter, Unsupported L3 protocol, and Service backend not found).
• TraefikServiceDown checks max by (instance, job, service) (traefik_service_server_up) == 0.
• TraefikHighHttp4xxErrorRateService groups by (instance, job, service), requires a request rate above 0.1 req/s, holds for 5m, and uses severity: info.

conf/alertmanager/alertmanager.tmpl routes PrometheusAlertmanagerE2eDeadManSwitch and severity="info" alerts to the null receiver so they remain visible in Prometheus/Alertmanager without sending Telegram noise or requiring permanent silences.

Keep locally authored rule files, including talos-nodes.yml, separate from upstream imports.

Preserve the local HostContextSwitchingHigh history guard in node-exporter.yml when refreshing host rules. It requires a counter sample from a day ago before comparing the 15-minute and daily rates. This avoids false spikes from newly scraped nodes while retaining detection after the baseline is established.

promtool test rules scripts/test_context_switching_rules.yml

cert-manager metrics

The cert-manager job scrapes 192.168.0.8:30101. The controller is pinned to talos-worker-core; a separate NodePort Service uses local endpoints to preserve the scraper IP, and an ingress policy permits only 192.168.0.6/32 on metrics port 9402. Controller egress, webhook and cainjector traffic are unchanged. If the core VM's Proxmox firewall filters NodePorts, permit TCP 30101 from the monitoring VM only. Deploy the ArgoCD changes before restarting VictoriaMetrics (docker compose restart prometheus), then verify up{job="cert-manager"} == 1 and the certificate readiness metrics.

Proxmox VE metrics

Proxmox VE metrics are collected via prometheus-pve-exporter scraping the PVE API at 192.168.0.3 using credentials defined in the environment:

• PVE_USER: monitoring@pve
• PVE_TOKEN_NAME: exporter
• PVE_TOKEN_VALUE: API token secret (configured in .env)
• PVE_VERIFY_SSL: false

The exporter listens internally on port 9221 and handles requests routed by VictoriaMetrics via the pve scrape job (/pve?target=192.168.0.3).

Cluster and edge services metrics

The scrape configuration monitors Talos cluster services via exposed NodePorts and internal endpoints:

• Traefik: scrapes metrics on 192.168.0.8:30102.
• ArgoCD: scrapes server and repo-server metrics on 192.168.0.8:30103 and 30104.
• Cilium: scrapes Cilium agents on Talos nodes (192.168.0.7:9962, 192.168.0.8:9962, 192.168.0.9:9962) and operator on 192.168.0.8:9963.
• etcd: scrapes etcd on talos-master-0 (192.168.0.7:2381).
• VictoriaLogs & vmalert: scraped directly via internal Docker container networking (victorialogs:9428, vmalert:8880).
• VictoriaLogs local health/ingestion/disk rules live in alerts/victorialogs.yml.

CPU throttling alerts

ContainerHighThrottleRate requires more than 25% of scheduling periods to be throttled and more than 100 ms of accumulated throttled time per second, sustained for 10 minutes. Throttled time measures scheduler delays, not application latency. This filters frequent tiny bursts; review application latency before changing CPU limits. Missing throttled-duration metrics cannot satisfy the alert.

Validate the rule and its eight regression scenarios with promtool:

promtool check rules conf/prometheus/alerts/google-cadvisor.yml
promtool test rules scripts/test_cadvisor_rules.yml

Talos node memory and OOM history

The node-exporter ArgoCD application runs on all three Talos nodes, including the control plane and Freebox worker. It reads host /proc and /sys through read-only mounts, without host networking, host PID access or a host root mount. The NodePort service uses port 30100 and externalTrafficPolicy: Local so each node address reports its own metrics. Its ingress policy allows only the monitoring VM, 192.168.0.6.

The node_talos scrape job attaches the Kubernetes node name. Existing HostOutOfMemory and HostOomKillDetected rules cover these targets; additional rules detect sustained memory PSI stalls and missing memory/OOM/PSI collectors. Host OOM counters include kills within container memory limits as well as host memory exhaustion; correlate them with pod OOM events before assigning a cause.

Use these recorded series for history:

talos_node:memory_available:ratio
talos_node:memory_stall:ratio5m
talos_node:oom_kills:increase1h

Validate the new recording and alert rules:

promtool check rules conf/prometheus/alerts/talos-nodes.yml
promtool test rules scripts/test_talos_node_rules.yml

Deploy the exporter application first and verify its DaemonSet is ready on all three nodes. Then update and restart VictoriaMetrics on the monitoring host (docker compose restart prometheus). Verify up{job="node_talos"} and node_scrape_collector_success{job="node_talos"}, confirm node_uname_info matches each target's node label, and check all three nodes expose node_vmstat_oom_kill and node_pressure_memory_waiting_seconds_total. History starts with the first successful scrape; past incidents cannot be filled in retroactively.

Freebox Wi-Fi info labels

The global scrape relabeling rule applies only to freebox_wifi_ap_info{job="freebox"}. It retains job, instance, ap_id, ap_name, ap_band, and ap_state, removing hardware capability flags that otherwise exceed VictoriaMetrics' 40-label ingestion limit. Other Freebox metrics retain their labels. The conditional if rule uses VictoriaMetrics syntax; validate the scrape configuration with VictoriaMetrics -dryRun.