Cluster-CA succession — dual-trust rollover for the node-mTLS root

SUPERSEDED (2026-07-02). Cluster-CA rotation shipped with a DIFFERENT design than this proposal: a single name="cluster" row whose cert_pem holds a new‖old union bundle (first block = signing generation), CAS row updates, generation-addressed vault key paths, and a kv-backed resumable phase machine with an explicit --rollback verb — NOT the cluster-pending/cluster-retiring row family, generation column, or atomic row-name swap proposed below. See docs/superpowers/specs/2026-07-02-ca-rotation-design.md and internal/modules/cluster/ca_rotation.go. Do not implement from this doc.

Two residuals here remain live and are folded into the spec’s follow-ups: the pre-sign-flip barrier-wait (step 2 below) and the algorithm upgrade (--algo, RSA→P-256) plan with its SKI/AKI generalization. The min-hold retirement floor was considered and REJECTED for the shipped design: the per-node AKI verification gate plus the registry-membership gate (X403044INTNMB) already bound the stale-leaf window, and a time floor would add days to every rotation for nothing. The rest of this document is retained as the survey of the pre-rotation state and the rationale for dual-trust-over-a-bundle vs a hard cutover.

The node-mTLS cluster CA is minted exactly once and can never be replaced. Everything downstream (node leaves, the :4003 trust pool) is built to renew leaves under the one immortal root — there is no mechanism to roll the root itself.

• The CA is a 10-year RSA-4096 self-signed root, no pathlen. internal/modules/cluster/cert_provider.go:127-131 sets caRSABits = 4096, caValidity = 10 * 365 * 24h. generateClusterCA (cert_provider.go:517-562) builds the template at cert_provider.go:538-551: IsCA:true, KeyUsage: CertSign|CRLSign, self-signed (AuthorityKeyId = SKI), no MaxPathLen/BasicConstraints pathlen asserted, and the key algorithm is hardcoded rsa.GenerateKey (cert_provider.go:518). The comment at cert_provider.go:515-516 states this is the SOLE minter and “must never grow a second caller” — i.e. minting a second CA today would be a bug.

• Minting is a one-time genesis singleton. EnsureClusterCA (cert_provider.go:152-187) is READ-ONLY and never mints; a missing row is the hard alarm X500048CACUNP (cert_provider.go:169), not a re-mint trigger. ProvisionClusterCAGenesis (cert_provider.go:237) is the only path that calls generateClusterCA, and it returns the existing row without minting whenever one is present (cert_provider.go:290-300). The cluster_ca_genesis subsystem (internal/subsystems/impl/cluster_ca_genesis.go, priority 7, ModeAny, deps {ent_client, vault}) drives it on a reconcile loop that only ever converges on the one row — reconcileOnce at cluster_ca_genesis.go:105-139.

• One row, one vault key, keyed by the reserved name cluster. The CA lives in the single certificates table (ent/schema/certificate.go:25-75): columns name (UNIQUE index, certificate.go:71-75), is_ca, cert_pem, key_ref, issuer, provenance, fingerprint, not_after. The cluster row is name="cluster", issuer="" (self-signed), provenance="autogen" (internal/modules/cluster/cluster_ca_store.go:50-59). Its private key is sealed in the protected vault ca/ section at the exact un-ref-able path ca/cluster/root-key (cert_provider.go:55-62) and loaded only at sign-time (cert_provider.go:405-412). WriteClusterCARow is INSERT … ON CONFLICT(name) DO NOTHING (cluster_ca_store.go:182-185): first-writer-wins on the name UNIQUE index, so the singleton cannot fork. An identity guard at cluster_ca_store.go:155-157 refuses any row at name="cluster" whose issuer != "" or provenance != "autogen". The row leaves not_after NULL (cluster_ca_store.go:181).

• The trust holder can accept a re-issued CA but nothing mints one. internal/subsystems/impl/internal_trust_holder.go re-reads ca.pem on a 30s TTL (internalTrustReloadTTL, line 34) precisely because the “CA is 10y and is not rotated” assumption is false (its own doc, lines 20-27). Crucially the pool is built with pool.AppendCertsFromPEM(caPEM) (internal_trust_holder.go:80), which parses every PEM block in the file — so a ca.pem containing two CA certs already yields a two-anchor trust pool for free. But the comment at internal_trust_holder.go:104-105 states the node “trusts exactly one cluster CA at a time,” and the only writer of ca.pem is materializeCACache (cert_provider.go:197-214), which writes the single name="cluster" row’s cert. So the mechanism to trust two anchors exists; the source only ever produces one.

• The renewal loop only rotates leaves, never the root. internal/subsystems/impl/internal_cert_rotation.go (12h check, DefaultInternalCertRenewBefore = 30d, lines 17-23) renews the :4003 leaf off whatever name="cluster" currently is: leader self-signs, follower POSTs the leader’s /internal/certs/rotate (internal/daemon/internal_cert_rotation_wire.go:43-53).

• The operator surface hard-rejects CA rotation. certs_router.go (/api/method/shc.cluster.certs.rotate, type due|cluster-ca|nebula-ca|mtls) rejects type="cluster-ca" with X500046CRTCAU — “not supported on this endpoint … rotation is daemon owned” (certs_router.go:241-259) — and rejects the due path once the CA is actually within 30d of expiry (certs_router.go:343-349). There is no working cluster-CA rotation path anywhere. A CA approaching its 10y expiry, a compromised CA key, or a desired algorithm upgrade (RSA→P-256) are all dead ends today.

Contrast the Nebula mesh CA, which does rotate: RotateNebulaCA (internal/modules/cluster/mesh_orchestrator_impl.go:612) regenerates the CA, archives the old pair under previous-<ts>/, re-signs every peer’s host cert, and pushes the new bundle via /internal/mesh/install. That is a hard synchronous cutover — viable only because a nebula reload is coordinated per-node. The :4003 x509 leaves renew lazily on a 12h loop, so a hard cutover would orphan any node not yet re-issued the instant the old anchor leaves the pool. That is exactly why the cluster CA needs a dual-trust window instead.

A manual, operator-triggered rollover ceremony with a dual-trust window, driven by a new daemon-owned orchestrator that mirrors RotateNebulaCA in shape but adds the overlap window x509 lazy renewal requires. Ceremony state is fully derivable from the certificate rows (the row names are the state machine), so a leader crash mid-ceremony resumes idempotently.

Keep name="cluster" as the invariant meaning “the active signer” — every existing read path (EnsureClusterCA, SignNodeCert, cluster_ca_genesis) stays correct unchanged. Add two reserved sibling names and a generation:

• name="cluster" — the active CA (the signer). Highest generation.
• name="cluster-pending" — a freshly minted CA staged for activation (exists only during the propagation phase; at most one).
• name="cluster-retiring" — the previous active CA kept in the trust bundle during the overlap window (at most one).

Add one nullable column to ent/schema/certificate.go: field.Int("generation").Optional().Nillable() (NULL / absent ⇒ generation 1, the legacy row). The vault key path becomes generation-suffixed: ca/cluster/root-key@<gen>, with the legacy path ca/cluster/root-key read-compat as @1. This guarantees a new CA’s key never clobbers the live signing key before the atomic swap commits.

New store methods in cluster_ca_store.go (the store still owns only cluster-family rows — invariant preserved):

• ReadClusterCAFamily(ctx) ([]ClusterCARow, error) — returns the active + pending + retiring rows (query is_ca=true AND issuer="" AND provenance="autogen" AND name IN ('cluster','cluster-pending','cluster-retiring')), each carrying Name, Generation, RootCertPEM, RootKeyRef, Fingerprint, NotAfter.
• WritePendingClusterCARow(ctx, gen, certPEM, keyRef, fp) — same INSERT … ON CONFLICT(name='cluster-pending') DO NOTHING first-writer-wins contract as WriteClusterCARow (cluster_ca_store.go:167-202).
• ActivateClusterCA(ctx, newGen) — the atomic swap, run in one dqlite write txn: UPDATE certificates SET name='cluster-retiring' WHERE name='cluster' then UPDATE … SET name='cluster' WHERE name='cluster-pending'. Both hit the name UNIQUE index; the txn is the linearization point.
• RetireClusterCA(ctx) — DELETE … WHERE name='cluster-retiring'.

The ReadClusterCARow identity guard (cluster_ca_store.go:155-157) is unchanged and still gates name="cluster"; cluster-pending/cluster-retiring carry the same issuer=""/provenance="autogen" shape and pass the same guard when read via the family query.

Rewrite materializeCACache (cert_provider.go:197-214) to source from ReadClusterCAFamily and write the concatenation of every family row’s cert_pem (deduped by fingerprint via the existing clusterCAFingerprint, cluster_ca_store.go:218-221) into ca.pem, still write-if-different by hashing the full rendered bundle. No change to internal_trust_holder.go — it already appends every block (internal_trust_holder.go:80), so the two-cert bundle produces a two-anchor ClientCAs/RootCAs pool. SignNodeCert continues to sign off name="cluster" alone (the active CA) — leaves chain to exactly one anchor; the bundle only widens verification.

Re-issue is driven by the existing renewal machinery, not a new leaf pipeline. internalCertRotation (internal_cert_rotation.go) already re-signs the :4003 leaf off name="cluster"; once the swap makes the new CA active, the next renewal signs under it automatically. To make it prompt rather than wait up to 12h/30d, the ceremony fans out the existing on-demand hook: RotateInternalCertNow (internal_cert_rotation.go:255) is already wired to the public certs.rotate-now route (certs_router.go:136-156) and published to peers via SetInternalCertRotateNow (internal_cert_rotation_wire.go:152). The orchestrator calls it on every node (leader in-process, followers over the existing mTLS peer transport used by peerRotate, internal_cert_rotation_wire.go:106-124). Short leaf TTLs (see the sibling proposal below) make even a missed node self-heal at its next renewal, so the re-issue window is naturally bounded.

The old CA (name="cluster-retiring") may be dropped from the bundle only when both hold:

1. No live leaf chains to it. Extend the per-node cert inventory (certs_internal_router.go CertEntry, currently reports Subject at certs_internal_router.go:202-207 but not issuer) with an issuer_fingerprint field = clusterCAFingerprint of the leaf’s issuer / the CA whose SKI matches the leaf AKI. The gate walks every registry member via the existing certs fan-out (certs_router.go:174-205) and requires every reachable node’s active leaf to carry the new CA’s fingerprint.
2. The min-hold floor has elapsed: now > activated_at + nodeValidity (nodeValidity = 1y, cert_provider.go:131), so any leaf that could have been signed by the old CA before the swap has expired even if a node was unreachable for the whole window. activated_at = the updated_at stamp of the swap on the cluster-retiring row.

If any member is unreachable (reuse the #113 reachability probe posture, docs/proposals/113-removal-quorum-fix-v2.md), retirement refuses with a typed code unless the operator passes --force and the min-hold floor has elapsed.

Replace the hard rejection at certs_router.go:241-259 with a real kickoff, and add a rollover verb family (new daemon-owned ClusterCAOrchestrator, sibling to the mesh orchestrator):

• shc cluster cert rollover begin [--algo rsa4096|p256] [--reason compromise|upgrade|expiry] → mint gen N+1, seal at ca/cluster/root-key@<N+1>, write cluster-pending, materialize the dual bundle everywhere, wait for all-trust, swap, fan out re-issue.
• shc cluster cert rollover status → current generation, window state (pending/active/retiring rows present), per-node leaf issuer fingerprint, min-hold remaining.
• shc cluster cert rollover retire [--force] → gated drop of the old CA.

Each maps to a new /api/method/shc.cluster.certs.rollover.{begin,status,retire} route alongside the existing certs.* endpoints.

--algo p256 requires generalizing three RSA-hardcoded spots off *rsa.PublicKey onto crypto.Signer/crypto.PublicKey: generateClusterCA (cert_provider.go:518), the leaf AKI derivation which casts the parent to *rsa.PublicKey (cert_provider.go:604-608), and computeRFC5280SubjectKeyID which hashes MarshalPKCS1PublicKey (cert_provider.go:810-818, RSA-only). For ECDSA the SKI must be RFC 5280 method 1 over the subjectPublicKey BIT STRING, which diverges from the Python parity path (cert_provider.go:529-536) — because a P-256 cluster CA is a clean-break Go-only artifact (no Python side re-issues it), the parity constraint does not bind the new CA, but the change must not regress the RSA SKI algorithm the existing chain depends on.

The certificates table holds three CA levels (ent/schema/certificate.go:13-24); this proposal targets exactly one:

• Cluster CA (name="cluster", self-signed, :4003 node-mTLS) — in scope, the full ceremony above.
• Global app root (name="global-root", self-signed; global_ca_genesis, internal/subsystems/impl/global_ca_genesis.go) — the same dual-trust machinery applies, but its blast radius includes every tenant intermediate (issuer="global-root") and every app leaf beneath them, so rolling it means re-issuing all intermediates then all app leaves. Sketched, phase 2 — reuse the row-family + dual-bundle pattern, out of scope for v1.
• Tenant intermediate (name="<tenant>", issuer="global-root") — rolling one is a per-tenant re-mint signed by the current global root plus re-issue of that tenant’s leaves; simpler and orthogonal. Out of scope here.

Existing installs need zero migration to keep running: the new generation column is nullable (absent ⇒ gen 1), the legacy vault path ca/cluster/root-key is read-compat as @1, and materializeCACache over a single-member family produces the identical one-cert ca.pem it writes today. A cluster that never runs a rollover is byte-for-byte unchanged.

The rollover itself is a multi-node ceremony with strict ordering so no node is ever orphaned. Trust the new CA everywhere before issuing any leaf under it; retire the old only after every node has rotated:

1. Mint + stage (leader). Mint gen N+1, seal key at ca/cluster/root-key@<N+1>, WritePendingClusterCARow. Idempotent: a re-run that finds cluster-pending present adopts it.
2. Propagate dual-trust. Every node’s cluster_ca_genesis reconcile loop (cluster_ca_genesis.go:105-139, already ModeAny on a 30s tick) re-materializes ca.pem from the family → now two anchors. Barrier: the orchestrator polls every registry member’s cert inventory and waits until all report both fingerprints in their trust bundle. Only then proceed. This is the load-bearing ordering guarantee — a leaf under the new CA before this barrier would fail mTLS on a not-yet-updated peer.
3. Activate (atomic swap). ActivateClusterCA renames cluster→ cluster-retiring and cluster-pending→cluster in one dqlite txn on the leader; the rename replicates to followers. The new CA is now the signer; both remain trusted.
4. Re-issue. Fan out RotateInternalCertNow to every node (internal_cert_rotation.go:255, followers via the mTLS peer transport). Each re-signs its :4003 leaf under the new active CA.
5. Retire (gated). When the retirement gate passes (all reachable leaves on the new CA and now > activated_at + nodeValidity), RetireClusterCA deletes cluster-retiring; the next materialize collapses ca.pem to the new anchor only. The old vault key @N is left sealed (audit) or scrubbed.

Node offline during rollover. A node down for steps 2–4 catches up on rejoin purely from replicated rows: its materialize loop reads the current family (both anchors during the window, or new-only after retirement) and its internal_cert_rotation loop re-issues its leaf under the active CA. The retirement gate protects the offline node: it refuses to drop the old CA while any member is unreachable (unless --force + min-hold elapsed), so a rejoining node still finds the old anchor in the bundle and its old leaf keeps validating until it re-issues. If the old CA was force-retired and its old leaf no longer validates, the node re-enrolls — bounded tight by short leaf TTLs.

Leader crash mid-ceremony. Every step is idempotent off the row state: cluster-pending present but not activated ⇒ resume at the barrier; cluster-retiring present ⇒ resume at re-issue/retire. No ceremony state lives outside dqlite.

Tie-in — short-lived leaves. This ceremony’s overlap window is bounded by the leaf TTL: the old CA can be retired once all leaves signed under it have expired. Today nodeValidity = 1y (cert_provider.go:131) makes the min-hold floor a year. The sibling proposal docs/proposals/pki-short-lived-leaves-and-revocation.md (short-TTL :4003 leaves + revocation) shrinks that floor to hours/days, making rollover retirement fast and making a compromised-CA rollover practically effective. These two proposals are complementary — succession relies on short leaves to keep the dual-trust window short.

• Split-trust orphan (a leaf under the new CA reaches a peer that doesn’t yet trust it → tls: bad certificate). Mitigated by the step-2 barrier: activation is gated on all nodes reporting both anchors.
• Premature retirement stranding a node still on the old CA. Mitigated by the two-part retirement gate (issuer-fingerprint check on every reachable member + the activated_at + nodeValidity floor) and the unreachable-member refusal.
• Vault key clobber before the swap commits. Mitigated by generation-suffixed key paths (ca/cluster/root-key@<gen>) — the new key never overwrites the live signer’s key; the pairing guard (cert_provider.go:414-423) still verifies cert↔key generation match at every sign.
• Leader crash mid-ceremony wedging the cluster. Mitigated by making row names the entire state machine — every step resumes idempotently, mirroring the existing first-writer-wins genesis idempotency (cert_provider.go:283-300).
• SKI parity regression when generalizing key algorithms. Mitigated by keeping the RSA path (computeRFC5280SubjectKeyID, cert_provider.go:810-818) bit-identical and adding the ECDSA SKI as a new branch, gated behind --algo p256.
• Out of scope: automatic/scheduled rollover (v1 is manual/operator-triggered only); global-root and tenant-intermediate rollover (cluster CA only); CRL/OCSP revocation (owned by the sibling short-lived-leaves proposal).

• Unit — mint/algo. generateClusterCA parameterized on algorithm: RSA-4096 and P-256 each produce a valid self-signed CA; SKI matches RFC 5280 method 1 for each key type; the RSA SKI is unchanged from the current byte output (golden).
• Unit — store. WritePendingClusterCARow first-writer-wins under a concurrent race (mirror cluster_ca_store_test.go); ActivateClusterCA swap is atomic and leaves exactly one name="cluster" + one name="cluster-retiring"; ReadClusterCAFamily returns the right members in each phase; the identity guard still rejects a foreign row at any reserved name.
• Unit — trust bundle. materializeCACache writes a 2-cert bundle during the window and a 1-cert bundle after retirement, write-if-different over the full bundle hash; internalTrustHolder.reload accepts the 2-anchor bundle and a leaf signed by either CA verifies against the resulting pool (extends internal_trust_holder_test.go).
• Unit — retirement gate. Refuses when any inventory leaf still carries the old CA’s issuer fingerprint; refuses when a member is unreachable; refuses before the min-hold floor; passes only when all three clear (or --force + floor elapsed).
• Integration — sign path. After activation, SignNodeCert signs under the new CA; an old-CA leaf still validates during the window and is rejected after retirement.
• Integration — resume. Kill the orchestrator with cluster-pending present, then with cluster-retiring present; confirm a re-run resumes at the correct step and reaches a consistent end state.
• e2e (proxmox multi-node). Full ceremony end-to-end: verify the step-2 barrier blocks activation until all nodes trust both anchors; verify inter-node ping succeeds across mixed-CA leaves during the window; kill a node during steps 2–4 and confirm it catches up on rejoin and re-issues before retirement; confirm a compromise-mode rollover completes fast when paired with short leaf TTLs.
• App test. A dual-trust-window inter-node broadcast (health/event) succeeds when some nodes hold old-CA leaves and others hold new-CA leaves.

• Hard synchronous cutover (the RotateNebulaCA pattern, mesh_orchestrator_impl.go:612): regenerate, archive old, re-sign all peers, push in one pass — no overlap window. Loses because :4003 leaves renew lazily on a 12h/30d loop (internal_cert_rotation.go:17-23), so the instant the old anchor leaves the trust bundle any node not yet re-issued is orphaned. Nebula gets away with it only because its reload is coordinated per-node; x509 leaf renewal is not.
• Cross-signed transitional CA (new CA cert also signed by the old key, served alongside leaves so chains bridge both roots): standard web PKI practice, but the trust holder builds anchor pools from whole-cert AppendCertsFromPEM (internal_trust_holder.go:80), not intermediate chain-building, and Go leaf verification would need the cross-cert distributed with every leaf. Since we control both ends of every mTLS handshake, a plain two-anchor bundle achieves the same overlap with far less machinery — so cross-signing loses on complexity.