It Happened on a Tuesday
A production API goes dark. No deployment happened. No config change was pushed. The on-call engineer is staring at a wall of 503s and the Slack thread is already twenty messages deep before anyone types the words nobody wants to type: when did the cert last renew?
It's expired. The certificate on the load balancer — the one that was supposedly managed by that automation script Steve set up two years ago — quietly expired at 2:14 AM and nobody noticed until clients started screaming. Steve left the company eight months ago.
This exact scenario plays out, in various forms, at organizations of every size. And here's what drives me nuts about it: TLS certificate expiration is one of the most preventable failure modes in modern infrastructure. We've had the tooling to fully automate this for years. And yet teams still get burned, regularly, because they treat certificate management as a "set it and forget it" problem when it's actually an ongoing operational discipline.
Why This Problem Is Worse Than You Think
Let's be honest about the scope here. Most organizations have a sprawling, partially-documented certificate inventory spread across cloud load balancers, internal Kubernetes clusters, legacy appliances, third-party SaaS integrations, and the occasional rogue server someone stood up in 2019 and forgot about. You might have hundreds of certs. Large enterprises have thousands. And unlike a misconfigured firewall rule that causes obvious breakage immediately, an expiring certificate is a slow-fuse problem — it works perfectly fine right up until the moment it doesn't.
The attack surface angle gets underappreciated too. Expired certificates are embarrassing, yes. But the real PKI risk isn't expiration — it's silent compromise. A certificate that's been issued to the wrong entity, or signed by an untrusted intermediate, or whose private key has been exfiltrated from a poorly-secured secrets store. Expiration monitoring is the table stakes. The more sophisticated practitioners are thinking about certificate transparency logs, key material protection, and revocation infrastructure that actually works.
Hot take: most organizations' revocation infrastructure is essentially fictional. Certificate Revocation Lists (CRLs) are often multi-megabyte files hosted on infrastructure that nobody monitors for availability. OCSP stapling is misconfigured or disabled on half the servers I audit. If a private key gets compromised tomorrow, how confident are you that clients will actually stop trusting that certificate within a reasonable timeframe? If the answer is "not very," you've got a paper-thin revocation story.
The ACME Protocol Changed Everything (And Some Teams Still Haven't Noticed)
Before Let's Encrypt and the ACME protocol, certificate issuance was a manual, expensive, friction-heavy process. You'd generate a CSR, submit it through some web portal, wait hours or days for validation, download a bundle, figure out which intermediate to concatenate, and then calendar a reminder for 13 months from now. It was awful, and the awfulness is why so many teams built ad-hoc scripts that half-worked and then became tribal knowledge.
ACME — the Automatic Certificate Management Environment protocol, standardized as RFC 8555 — made fully automated certificate lifecycle management genuinely achievable for everyone. The certbot client was the first widespread implementation most people encountered, but the ecosystem has matured significantly. acme.sh is a shell-only alternative with broad DNS provider support. cert-manager for Kubernetes has become the de facto standard for container workloads, integrating directly with Let's Encrypt, ZeroSSL, and internal CAs like HashiCorp Vault.
Here's the thing though: ACME automation is not the same as certificate management. I've seen teams deploy cert-manager, celebrate that certs auto-renew, and then completely miss the part where their cert-manager deployment itself is in an unhealthy state. The CertificateRequest objects are failing silently. The renewal is two days away. Nobody is watching the Kubernetes events. Automation without observability is just deferred panic.
The 90-day validity window that Let's Encrypt popularized — and that the CA/Browser Forum has been pushing to shrink further, with proposals now on the table to go down to 47 days — is intentional design. Shorter validity forces automation. If you can't automate 90-day renewal, you're going to be manually rotating certificates every six weeks, which is an operational nightmare. The industry is essentially mandating that you build proper automation or suffer the consequences. And I think that's exactly right.
The Kubernetes Cert-Manager Trap
Since we're in the weeds on cert-manager: it's excellent tooling, genuinely, but teams treat it like magic and then get surprised when reality intervenes.
A scenario I've seen more than once. Team deploys cert-manager via Helm. They create an Issuer or ClusterIssuer pointing at Let's Encrypt's production ACME endpoint. They annotate their Ingress resources with cert-manager.io/cluster-issuer. Certificates get issued, everything looks great. Six months later, the Let's Encrypt account credentials used in the Issuer configuration have been rotated or the ACME account was somehow invalidated. Or the DNS-01 challenge provider credentials — maybe AWS Route53 API keys — have been rotated as part of a routine secret rotation and nobody updated the cert-manager secret. Now certificate renewal fails. But the existing certificates are still valid for another 30 days. Nobody notices until expiration is imminent.
The fix isn't complicated, but it requires intentionality. You need to be actively monitoring the Certificate resource status in Kubernetes, not just checking that the Ingress is serving traffic. kubectl get certificates -A should be part of your operational runbook. Better yet, export the certmanager_certificate_expiration_seconds metric from cert-manager's Prometheus endpoint and build an alert that fires when any certificate is within 30 days of expiry. That alert should wake someone up at 3 AM if necessary — because the alternative is getting woken up by a production outage instead.
Internal PKI: The Quiet Disaster
Public-facing certificate management gets most of the attention because the failures are visible and embarrassing. But internal PKI is where the real operational landmines live.
Internal CAs — whether you're running Microsoft Active Directory Certificate Services, a HashiCorp Vault PKI secrets engine, or something home-rolled with openssl and a lot of optimism — typically have longer-lived certificates, less monitoring, and significantly higher blast radius when something goes wrong. The root CA certificate expiring is a different category of catastrophe than a single service cert expiring. When your root expires, everything signed by it stops being trusted simultaneously. I have personally witnessed this happen to an organization's internal CA and the resulting trust chain breakage took down authentication infrastructure for half the company for most of a day.
The openssl x509 -in cert.pem -noout -dates command is burned into my muscle memory at this point. But running that manually is not a certificate management strategy. If you're running internal PKI with ADCS, you should absolutely have a monitoring solution checking your CA's own certificates — both the root and any issuing CAs — with long advance warning windows. I'd say 180 days minimum for root CA alerts, 90 days for issuing CAs. These are not things you can renew quickly. Renewing a root CA requires careful planning, coordinated trust store updates across your entire environment, and usually a change window. Give yourself time.
HashiCorp Vault's PKI secrets engine is excellent for internal certificate automation, but it comes with its own footgun: the Vault cluster's own TLS certificate. I've seen Vault clusters where the Vault server certificate expires and suddenly nothing can authenticate to Vault anymore — including the systems that would normally auto-renew certificates from Vault. Circular dependency, lights out. Don't let this happen to you. Monitor Vault's own TLS configuration separately from the certificates it issues.
What Your Inventory Actually Needs to Look Like
Unpopular opinion: most certificate inventory spreadsheets are useless. By the time you've finished populating it, half the information is wrong and the team that owns it has already moved on to the next fire. A static inventory is not the answer.
What you actually need is continuous discovery. Tools like Censys and Shodan can tell you what certificates are visible on your public-facing infrastructure — which is useful but incomplete. For comprehensive internal visibility, something like certspotter monitoring Certificate Transparency logs for your domains gives you an audit trail of every publicly-trusted certificate ever issued for your domain space. This is not just useful for inventory; it's a legitimate security control. If someone issues a certificate for api.yourdomain.com through a CA you don't recognize — because of a BGP hijack, a compromised CA, or social engineering — Certificate Transparency gives you visibility into that. The catch is that you have to be monitoring it, not just knowing it exists.
For the internal inventory problem, the honest answer is that your secrets management system — Vault, AWS Secrets Manager, Azure Key Vault — should be the authoritative source of truth for certificates, and your rotation automation should be built around it. Certificates that live outside your secrets management infrastructure are certificates you will eventually lose track of. That's not hyperbole, that's just how organizational entropy works.
The Key Material Problem Nobody Wants to Talk About
We've been talking mostly about certificate lifecycle, but let's get uncomfortable for a second and talk about private keys.
A certificate is only as trustworthy as the security of its corresponding private key. And the private key hygiene I see in real-world environments ranges from "pretty good" to "actively terrifying." Keys stored in plaintext in Git repositories — yes, in 2026, still happening. Keys baked into Docker images. Keys copied to developer laptops for "local testing" and never removed. Keys in /etc/ssl/private/ with permissions that are technically correct but on a server that twenty people have SSH access to.
Key material should never leave the system that generated it if you can help it. The ideal architecture is that the private key is generated on the system that will use it, a CSR is submitted to the CA, and the signed certificate comes back — the key itself never traverses a network or gets stored anywhere except the local system and possibly an encrypted backup. Tools like certbot operate this way by default. cert-manager in Kubernetes stores keys as Kubernetes Secrets, which is better than plaintext files but still requires you to think carefully about RBAC — a Secret accessible to any pod in the cluster is not adequately protected key material.
For high-value services, consider HSM-backed keys via PKCS#11 interfaces, or at minimum use your cloud provider's managed certificate services — AWS Certificate Manager, Google-managed SSL certificates — where the private key is managed by the provider and you literally cannot extract it. There's a real argument that "you can't extract it" is a feature, not a limitation.
The Clock Is Already Ticking on Something in Your Environment
I'll leave you with this: right now, in your infrastructure, there is almost certainly a certificate that is going to expire before anyone on your team notices. Maybe it's the wildcard cert on the legacy load balancer that predates your ACME automation. Maybe it's the client certificate your monitoring system uses to authenticate to an internal API. Maybe it's the code signing certificate the build pipeline uses. Maybe it's your internal CA issuing certificate.
It's not a question of if you have a lurking expiration problem. It's a question of whether you find it before it finds you.
The organizations that handle this well aren't necessarily the ones with the fanciest tooling. They're the ones that made certificate management boring — automated, monitored, alerting, with clear ownership for every certificate in the estate. Boring is good. Boring means it's not your problem this week.
The ones that get burned are the ones that treated automation as a one-time project rather than an ongoing operational commitment. Who deployed cert-manager and closed the ticket. Who set up a monitoring dashboard that nobody looks at. Who have a runbook for certificate renewal that hasn't been tested since it was written.
So here's the question: when's the last time someone on your team actually audited your certificate inventory? Not updated the spreadsheet — audited it. Verified that the automation is working. Checked that the alerts fire. Confirmed that renewal actually happened, not just that renewal was scheduled to happen. If the answer is "longer than I'd like to admit," you already know what you need to do this week.
