Short-lived :4003 node leaves + renewal as the passive revocation backstop

LANDED (2026-07). Shipped as proposed: short-lived node leaves with a 7-day TTL, half-life renewal, and hourly renewal checks (b5acf9db0). This is the passive revocation backstop the proposal describes.

The inter-node mTLS plane on :4003 authenticates any client whose leaf chains to the cluster CA. Two facts about the current PKI make credential revocation weak:

1. Node leaves live ~1 year. internal/modules/cluster/cert_provider.go:131 defines nodeValidity = 1 * 365 * 24 * time.Hour // 1y per task spec, and generateNodeCertificate stamps it directly: NotAfter: now.Add(nodeValidity) (cert_provider.go:617). The CA itself is caValidity = 10 * 365 * 24 * time.Hour (cert_provider.go:130, RSA-4096 at :127); leaves are RSA-2048 (:128). NotBefore is set to bare now (cert_provider.go:584/616) — no backdating margin.

2. There is no CRL/OCSP. The CA is minted with KeyUsage: …| x509.KeyUsageCRLSign (cert_provider.go:548) but nothing ever issues or serves a CRL, and there is no OCSP responder. Once a leaf is signed, the only thing that stops it authenticating is expiry — a year away.

The recently-shipped membership gate (internal_api_membership.go) is the first real mitigation. It mounts a chi middleware after InternalAuthMiddleware that, for cert-authenticated peers, extracts the peer leaf’s identity (CommonName + DNS SANs via peerNodeIdentities, internal_api_membership.go:177; localhost excluded) and rejects with X403044INTNMB when the live node registry no longer lists it (authorize, :115). Its deliberate limits:

• Only catches formally-removed nodes. It checks membership in NodeRepo.List(ctx, false) (excludes REMOVED tombstones, internal_api_membership.go:150). A leaked/copied-but-not-removed leaf still belongs to a live row, so it passes.
• Depends on registry reachability. A read fault before any good read fails open (ok==false → return nil, :126); after a good read it serves the last-good set. Membership is cached for internalMembershipTTL = 5s (:54).
• Gated on cluster size > 1. if rows <= 1 { return nil } (:128) — genesis and single-node installs are exempt, so on a one-node box the gate is inert.

So the gate closes the “removed node keeps talking” hole but does nothing for a leaked private key whose node is still a member. With a 1-year leaf, that stolen key is a control-plane credential for up to a year with no revocation lever.

The rotation machinery already exists — it is just tuned for long leaves. internal/subsystems/impl/internal_cert_rotation.go runs a ModeAny (every node, leader and follower) reconcile loop:

• DefaultInternalCertCheckInterval = 12h (:17) — “a node leaf is minted with months of life, so two checks a day is plenty” (comment :14-16).
• DefaultInternalCertRenewBefore = 30 * 24 * time.Hour (30d, :23).
• CheckOnce reads the live leaf’s NotAfter via the leafNotAfter seam, returns early when notAfter.Sub(s.now()) > s.renewBefore (internal_cert_rotation.go:127), otherwise renew → install, every error path warn-and-continue so a failed renew never drops :4003.

The leaf hot-reloads without a restart: the listener serves its cert through the GetCertificate handshake callback (internal_api.go:327/343 → internalLeaf.get, internal_leaf_holder.go:53), and install writes the pair atomically (cluster.InstallNodeLeaf, cert_provider.go:747) then republishes via impl.SetInternalLeaf (internal_leaf_holder.go:65). The renew dispatcher (internal_cert_rotation_wire.go:43, buildInternalCertRenew) picks per call: a leader self-signs in-process off the CA key it holds (localSign, :86); a follower POSTs the leader’s /internal/certs/rotate over the mTLS peer transport (peerRotate, :106), which re-signs off the caller’s mTLS CN (handleCertRotate, certs_internal_router.go:120). The wire comment states the load-bearing assumption: renewal “fires 30d BEFORE expiry while the old leaf still validates” (internal_cert_rotation_wire.go:10-11).

The trust anchor (ClientCAs/RootCAs) is a separate holder that re-reads ca.pem on a 30s TTL (internal_trust_holder.go:34); the CA is 10y and is NOT rotated in this design.

Turn the existing rotation loop into a real revocation mechanism by making node leaves short-lived (SPIFFE-style) and renewing continuously, so a leaked-but-not-removed leaf self-expires in days instead of a year. The membership gate stays as the immediate reject for removed nodes; short-TTL is the passive backstop for leaked creds.

• nodeValidity = 7 * 24 * time.Hour (7 days). CA caValidity unchanged (10y).
• Add a clock-skew backdate in generateNodeCertificate: introduce leafBackdate = 1 * time.Hour and set NotBefore: now.Add(-leafBackdate) (cert_provider.go:616). At 1y this was irrelevant; at 7d a follower whose clock trails the leader by minutes would otherwise reject a just-minted leaf as not-yet-valid.

Renewal interval must be << TTL, with margin for warn-and-continue retries:

• DefaultInternalCertRenewBefore = TTL/2 = 84h (~3.5d). Renew at half-life, the standard short-lived-cert cadence: after renewal a leaf always has ≥ TTL/2 of life, so a node has to be unreachable for the full half-life before its leaf even enters the danger zone.
• DefaultInternalCertCheckInterval = 1h. With renewBefore = 84h that is ~84 renewal attempts inside the window — each transient “leader unreachable” tick is a warn-and-retry (internal_cert_rotation.go:130-133) with enormous slack. Cadence invariant: interval << (TTL − renewBefore) − maxClockSkew, i.e. 1h << 84h − ~minutes. Holds comfortably.

Update the now-false comments at internal_cert_rotation.go:14-16 and :20-23 (“months of life”, “30 days”).

A forward-only TTL change alone would leave an existing 1-year leaf untouched for ~362 days (its remaining life > renewBefore, so CheckOnce never fires). Add a second renew trigger keyed on remaining life exceeding the new max TTL:

• Add field maxLeafTTL time.Duration to internalCertRotation, defaulted to nodeValidity (7d) in newInternalCertRotation.
• In CheckOnce, renew when either notAfter.Sub(now) <= renewBefore (normal) or notAfter.Sub(now) > maxLeafTTL (leaf minted under the old long-TTL regime). leafNotAfter already returns only NotAfter, so this uses remaining life — no need for NotBefore. A freshly minted 7d leaf has remaining ≈ 7d, not > 7d, so the accelerator does not self-loop.

This replaces a legacy 1y leaf on the first tick after upgrade, then settles into the half-life cadence.

• internal/modules/cluster/cert_provider.go — nodeValidity, leafBackdate, NotBefore.
• internal/subsystems/impl/internal_cert_rotation.go — the two default constants, maxLeafTTL field + the over-long trigger in CheckOnce, comment fixes.
• No change to internal_leaf_holder.go, internal_trust_holder.go, internal_api_membership.go, internal_cert_rotation_wire.go, certs_internal_router.go — the mint/renew/hot-reload plumbing and the membership gate are reused verbatim.

• CRL needs a signed revocation list to be generated on revoke, distributed to every node, and consulted on every handshake — plus a serving endpoint and a freshness policy. On a self-hosted appliance that is new distribution infra and a new “stale CRL = fail open or fail closed?” decision.
• OCSP needs a highly-available responder reachable from every node during every handshake — a new single point of failure on the very :4003 plane we are trying to harden, plus stapling to avoid the responder round-trip.
• Short-TTL needs zero new infra: it reuses the CA key the leader already holds and the rotation loop that already runs. Revocation becomes “stop renewing” — a leaked leaf is dead within one TTL with no list to publish and no responder to serve. The cost is more signing (cheap, RSA-2048, half-life cadence) and a hard dependency on renewal actually running (addressed below).

The renew trigger is the on-disk leafNotAfter read (NodeLeafNotAfter, certs_internal_router.go:252), driven by the loop’s immediate-sweep-then-ticker base (reconcile_loop.go:72-96). If a node is offline past its leaf TTL, on wake the immediate CheckOnce sees notAfter.Sub(now) < 0 < renewBefore and renews. Two cases:

• Leader / single node: localSign mints in-process off the CA key it holds (internal_cert_rotation_wire.go:86-92) — no mTLS, so an expired local leaf never blocks its own renewal. Single-node and the leader can always self-heal.
• Follower whose leaf already expired: peerRotate dials the leader over the mTLS peer transport presenting its expired on-disk leaf (internal_cert_rotation_wire.go:106-124). The leader’s TLS handshake rejects an expired client cert → renewal cannot complete → deadlock. A leaf that needs mTLS to renew cannot renew once it has expired.

Mitigation, layered:

1. Half-life cadence makes this rare. A follower must be continuously offline for the full TTL (7d) before its leaf dies; renewal starts at 3.5d remaining with hourly retries. Normal reboots/flaps never reach it.
2. Recovery is re-enroll, not renew. A follower past TTL must re-run the enrollment path (/internal/enroll), which is cert-less and explicitly exempt from both InternalAuthMiddleware and the membership gate (internal_api_membership.go:89-94). This is the same bootstrap the join flow already uses; document it as the operator recovery for a node offline longer than the leaf TTL. (An automatic “expired local leaf → route to enroll instead of rotate” branch in peerRotate is a possible follow-up but is OUT OF SCOPE here — it needs a bootstrap token available at renew time.)

The deadlock is therefore bounded to the pathological offline-longer-than-TTL follower, and the TTL is chosen so that window is operationally rare and has a known manual recovery.

Forward-only, no data migration. The change is a mint-time TTL, a renewal interval, and an in-memory accelerator trigger — nothing on-disk or in dqlite changes format. The certs/node.pem/node-key.pem layout, the CA row, the sealed CA key, and ca.pem are all untouched.

• Read-compat / dual-format window: existing long-lived (1y) leaves remain valid and keep serving handshakes after upgrade — a 7d verifier happily accepts a leaf with 1y remaining. There is no flag day.
• Accelerated convergence: the §3 over-long trigger re-mints each legacy leaf to 7d on the first CheckOnce after the daemon restarts into the new build, so the fleet converges to short leaves within one check interval rather than drifting for a year.
• Ordering: the rotation subsystem depends on internal_api (priority 45, deps: ["internal_api"], internal_cert_rotation.go:157-158) and the renew seams are published by wireCluster before the subsystem Starts (SetInternalCertRenewSource, internal_cert_rotation_wire.go:144). No new ordering constraints.
• Single-node verification: confirm the leader path renews a 7d leaf before expiry with interval=1h/renewBefore=84h, and that the accelerator replaces a synthetic 1y leaf on the first tick — leader localSign is in-process so this cannot deadlock.
• Follower verification: confirm a follower renews via peerRotate while its old leaf is still valid (renewal fires at 3.5d remaining, cert authenticates), and confirm the hot-reload lands on :4003 with no restart (GetCertificate reads the new pointer).
• Multi-node / no bootstrap deadlock: renewal must not require the very mTLS it secures in a way that can wedge a healthy node. Because renewal fires at half-life while the leaf still validates, and because the leader path is in-process, a running cluster never deadlocks; only a follower offline past the full TTL does, and its recovery is re-enroll (above).

• Renewal loop stalls → mass expiry. With 1y leaves a stalled loop was invisible for a year; at 7d it is a fleet outage in a week. Mitigation: half-life renewal gives 3.5d of daily-alarmed slack; CheckOnce already warn-logs every failure (X200044CRTCHK/X200948CRTRLD). Add an alarm when a live leaf’s remaining life drops below renewBefore/2 (renewal is failing) so the stall is loud well before expiry.
• Clock skew rejects fresh leaves. Mitigated by the NotBefore backdate (§1) and by renewBefore (84h) dwarfing any plausible node drift.
• Signing load on the leader. Half-life cadence over N nodes is a handful of RSA-2048 signs per node per 3.5d — negligible; SignNodeCert’s PersistLocal:false renew path mints fresh each time (the on-disk reuse in loadExistingNodeCert/certCoversIPs is gated on PersistLocal, cert_provider.go:441, so rotation never hands back a stale leaf).
• Follower offline > TTL. Bounded, manual re-enroll recovery (above).
• OUT OF SCOPE: CA rotation / CA TTL (stays 10y); an automatic enroll-on-expired-leaf branch in peerRotate; per-request CRL/OCSP; changing the membership gate’s fail-open matrix or its rows > 1 gate. The membership gate remains the immediate-reject complement; short-TTL is only the passive backstop.

Unit (internal_cert_rotation_test.go, fakes for leafNotAfter/renew/install):

• CheckOnce renews when notAfter - now <= renewBefore and no-ops above it (existing behavior, re-verify with the new constants).
• Accelerator: CheckOnce renews when remaining life > maxLeafTTL (legacy 1y leaf) and does NOT renew a freshly minted 7d leaf (remaining ≈ maxLeafTTL, not >), i.e. no self-loop.
• Expired leaf (notAfter in the past) triggers renew.
• Warn-and-continue: renew/install errors do not panic and leave the loop running.

Unit (cert_provider_test.go):

• generateNodeCertificate stamps NotAfter ≈ now + 7d and NotBefore ≈ now − leafBackdate; the leaf still chains to the CA (tls.X509KeyPair / verify against the CA pool).

Integration:

• Single node: run the rotation loop with a compressed clock (inject now); assert a 7d leaf is renewed before expiry and hot-reloaded (GetCertificate serves the new NotAfter).
• Follower: with a valid old leaf, peerRotate against a fake leader /internal/certs/rotate returns a fresh pair that installs and reloads.
• Migration: seed a synthetic 1y leaf on disk, start the loop, assert it is re-minted to 7d on the first tick.
• Deadlock path: a follower with an expired leaf → peerRotate handshake is rejected by the leader; assert the loop warns-and-continues (no crash) and that the re-enroll path (/internal/enroll, cert-less) succeeds.
• Complement with membership gate: a removed node is rejected immediately by X403044INTNMB regardless of leaf validity; a leaked-but-not-removed leaf is accepted until it expires (≤ 7d), proving short-TTL is the backstop.

App/e2e: a multi-node bring-up survives a leaf half-life with no :4003 reachability regression (ping/event broadcast stay green across a renewal).

CRL published to dqlite + consulted per handshake. The CA already carries KeyUsageCRLSign (cert_provider.go:548), and dqlite gives free replication of a revocation list. But it needs a revoke command, a per-handshake CRL check in the TLS verify callback, and a fail-open-vs-fail-closed policy when the list is stale — and it still leaves a revoked-but-unexpired leaf valid until the next node observes the list. Short-TTL achieves the same “revoked cred stops working” outcome with a bounded, automatic horizon and zero new verify-path code or distribution semantics, and it composes cleanly with the existing rows > 1 membership gate (immediate reject) rather than duplicating it. CRL would be worth revisiting only if we needed sub-TTL revocation latency for leaked-but-not-removed creds; the membership gate already covers the removed-node case at ≤ 5s.