Thirteen days of silent TLS failure, and the probe that reported healthy the whole time 

Certificate renewal had been failing for thirteen days. Seventeen days of validity left. No alert had fired, because the alert was working correctly and measuring the wrong thing.

The probe reported seventy-five days remaining. It was not lying. It was reading a different certificate than the one about to expire.

What the probe was looking at

The setup is ordinary. Kubernetes, cert-manager issuing certificates via DNS-01, a CDN in front terminating TLS at the edge, and a blackbox exporter checking certificate expiry the way every guide tells you to: dial the public hostname, read probe_ssl_earliest_cert_expiry, alert when it gets low.

That works right up until something sits between the prober and the origin.

When a CDN terminates TLS, the certificate presented on the public hostname is the CDN's. It renews on the CDN's schedule, entirely independently of yours, and it is basically always healthy. The certificate your origin serves, the one your ingress actually needs, is behind that termination and the probe never sees it.

So the probe measures the edge and reports on the edge. The dashboard is green. The origin is quietly running down.

The probe was not broken. It was structurally incapable of observing the thing it was named after. That is a worse failure than a broken probe, because a broken probe eventually stops reporting and someone notices the gap. This one reported confidently, in the right units, on the right dashboard, forever.

Why thirteen days

DNS-01 renewal had started failing. Renewal failures are not user-visible: the existing certificate keeps working until it expires, so nothing degrades, nothing 500s, no user complains. The only signal is the countdown, and the countdown was being read from the wrong certificate.

Thirteen days is just how long it took me to look at the cert directly for an unrelated reason. Seventeen days of runway left. It would have gone all the way to zero and taken every service on that ingress down at once, on whatever day that landed, with no warning at all.

The fix

Move certificate truth to where the certificate lives.

Cert-manager already knows. It publishes the expiry of every Certificate resource it manages, from the actual secret in the cluster, with no network path and nothing in between to terminate anything. Alert on that.

The external probe stays, but it is honest about its job now: it checks that the public endpoint is reachable and serving TLS at all. It is a reachability check. It is not, and never was, a certificate expiry check.

Two signals, two questions:

• Is the origin's certificate running out? Ask the thing that issues it.
• Can the outside world reach us over TLS? Ask from outside.

Conflating those is the bug, and the CDN is what makes them look like the same question.

The general shape

This is not really about certificates. It is about the distance between what a metric is named and what it measures.

Every layer you put in front of a system, a CDN, a load balancer, a service mesh, a proxy, terminates something and re-originates it. Anything downstream of that termination is now measuring the terminator, not the origin. Response time measured at the edge includes the edge's cache hits. Status codes at the edge include the edge's error pages. Certificate expiry at the edge is the edge's certificate.

The check I now run on any probe before trusting it: what would have to be broken for this to go red, and is that the same thing I am afraid of? If the honest answer is no, the probe is decoration.

For this one, the answer was: the CDN would have to fail to renew its certificate. Cloudflare failing to renew its own certificates is not the risk I was trying to cover.

What I kept

Thirty-three alert rules across the estate, every one of them backtested against real history before being armed, because an alert that would have fired eleven times last month for nothing will be muted by week two and then it is not an alert, it is a habit of ignoring a channel.

Severity routing, so the pager is reserved for things that are actually on fire and everything else goes somewhere you read on purpose. Inhibit rules, so one root cause produces one page instead of thirty.

And now, one fewer probe that was measuring a stranger's certificate and calling it mine.