ns8-metrics: false certificate alerts from stale Traefik metrics

[stephdl]

Steps to reproduce

• Have a node certificate covering several names, e.g. posta.example.com, node3.example.net, proxy3.example.net
• Let it be reissued with a different CN but the same name set, here posta.example.com to node3.example.net
• Do not restart Traefik
• Wait until the old certificate has less than 28 days left

Expected behavior

No alert. The only certificate in acme.json is valid for about 90 more days, and the TLS certificates page shows it as valid.

Actual behavior

CertExpiringSoon fires, then CertExpiringCritical, then CertExpired, which never clears.

Traefik keeps exposing traefik_tls_certs_not_after for a certificate it has replaced, until it restarts. Upstream: traefik/traefik#8606.

rules.d/tlscert.yml aggregates with max by (cn, sans), which masks a stale series only while CN and SANs both stay the same. On a CN change the stale series forms its own group and alerts alone.

Seen on a production cluster, two nodes:

2026-10-15  traefik4  posta.example.com  node3.example.net,posta.example.com,proxy3.example.net  511458900122312166490624809264886668471982
2026-12-14  traefik4  node3.example.net  node3.example.net,posta.example.com,proxy3.example.net  519048561576165143401225526064484491625112

Serial …471982 is gone from acme.json, per api-cli run module/traefik4/list-certificates. Same on traefik8, CN moved from proxy6.example.net to node6.example.net. About twenty other stale series sit on those nodes, silent only because their CN did not change.

Same expression drops the node and module_id labels. Annotations read "TLS certificate on Node expires in …", and alert-proxy builds cert-expiring-soon:<cn>:node:unknown.

Workaround

Restart the Traefik instance, the alert clears after the 5m for window.

runagent -m traefik4 systemctl --user restart traefik

It drops HTTPS for a few seconds, so not an option on shared production nodes, and it comes back on the next CN change.

Possible fixes

Short term, aggregate as max by (node, module_id, sans). Both stale series carry the same sans string as their replacement, so they get masked, and the labels come back. No restart needed. Still breaks if the name set changes, or if a certificate is deleted with no replacement.

Longer term, source the expiry from acme.json instead of the Traefik gauge. ns8-traefik already parses it in cert_helpers.list_internal_certificates, and certificate-exporter.path already fires on every change.

Components

• ns8-metrics
• ns8-traefik

See also

• tls_certs_not_after metric shows old renewed serials traefik/traefik#8606

see mattermost talk https://mattermost.nethesis.it/nethesis/pl/xyaw4zw1d38jfcq7at4ijdei5r