Vault envelope encryption: replace per-row PBKDF2 with a cached KEK + per-row DEK

LANDED (2026-07). Shipped: envelope encryption with an Argon2id-derived cached KEK + per-row AES-256-GCM DEKs and a dual-format migration (f73e4ae78), plus the restart-orphan / post-catch-up KEK-reconcile follow-ups (3cba9a041, P1i e19ec4761). Live in internal/modules/vault/verifier.go (v2 envelope verifier) + the crypto path.

The local (dqlite/sqlite) vault provider derives a fresh Fernet key per encrypt and per decrypt by running PBKDF2-HMAC-SHA256 at 480 000 rounds. The round count is fixed in code:

• internal/modules/vault/crypto.go:38 — var cryptoPBKDF2Iter = 480000
• internal/modules/vault/crypto.go:104-108 — deriveFernetKey calls pbkdf2.Key(password, salt, cryptoPBKDF2Iter, cryptoKeyLen, sha256.New) and base64-encodes the 32-byte output into the Fernet signing+encryption halves.

Crucially the KDF runs on every operation, keyed by the row’s own salt:

• Encrypt generates a fresh 16-byte salt per call and derives from it (crypto.go:69-81, salt at :70-73, derive at :74).
• Decrypt reads the salt embedded in the stored blob and re-derives (crypto.go:84-99, derive at :94).

Because the salt is per-row and embedded, there is no key caching possible: every Get and every Set pays a full 480 000-round PBKDF2. The provider hot path proves this — each read/write is exactly one KDF:

• internal/modules/vault/providers/local/provider.go:64 — Get → p.crypto.Decrypt(ciphertext)
• internal/modules/vault/providers/local/provider.go:78 — Set → p.crypto.Encrypt(value)

The cost is acknowledged in the code itself: crypto.go:36-37 — “each Encrypt/Decrypt would otherwise burn ~1–2s on PBKDF2 alone, and the suite calls Decrypt many times per test.” The test suite only stays tractable because crypto_testseam_test.go:15 lowers the count to 1000; production runs the full 480 000. An install that resolves N autogen refs, or a vault list-style read fan-out, pays N × ~1–2 s.

On-disk / wire format (per the header at crypto.go:4-14 and constants at :40-48):

base64std( salt(16) || fernet_token )
fernet_token = 0x80 || ts(8) || iv(16) || AES-128-CBC(ciphertext) || HMAC-SHA256(32)

Storage: the ciphertext string lives in the vault.value column, typed field.Text("value") (ent/schema/vault.go:31); rows are unique on (key, tenant_name, stack_name, environment, deployment, app_name) (ent/schema/vault.go:46). The provider upserts the ciphertext verbatim (provider.go:87-111).

Unlock / key lifecycle:

• The CryptoBox interface is minimal: Encrypt(string)(string,error) / Decrypt(string)(string,error) (internal/modules/vault/factory.go:84-87).
• The bootstrap memoryPasswordManager derives a *VaultCrypto from the master password on Unlock and hands it to the factory (internal/subsystems/impl/vault.go:276-287, box built at :277); it also retains the raw password in memory for the backup export default (vault.go:259, MasterPasswordForExport at :309-316).
• Service.Unlock gates on a verifier before adopting a box: it reads the persisted verifier and rejects a wrong password on an initialised cluster (internal/modules/vault/service.go:242-245), then plumbs s.manager.CryptoBox() into the factory (service.go:257-258).
• The verifier is itself a Fernet blob: buildVerifier = box.Encrypt(magic) and verifyPassword = box.Decrypt(stored) == magic (internal/modules/vault/verifier.go:50-56, :61-71). It is stored in the dqlite-replicated vault_config table under key master_verifier (verifier.go:39-44, :148-171). So verification also pays a 480k PBKDF2.

Rotation re-encrypts the entire vault: rotateLocked dry-decrypts every row with the old box then re-encrypts every row with the new box inside one ent transaction (internal/modules/vault/rotate.go:161-219, re-encrypt at :202-218). Cost is 2 × 480k-PBKDF2 × row-count.

Backup relevance: the export/import bundle re-encrypts values under a user-supplied (or master-default) password using the same VaultCrypto (transfer.go:177 export box, :214 exportBox.Encrypt, :308 import box, :321 importBox.Decrypt), tagged ExportFormatVersion = 1 (transfer.go:42, gate at :294-295). The inline encrypted vault bundle that rides in a BackupManifest (secrets.vlt) is produced through this same path.

Net: PBKDF2 was chosen for Python parity, but 480k rounds on the read path is the wrong tool — the master password is unlocked once and held in memory, yet we re-derive from it on every single row touch.

Move to envelope encryption: derive one key-encryption key (KEK) per unlock, cache it in memory, and protect each row with a random per-row data-encryption key (DEK) that is wrapped by the KEK and stored alongside the payload. Reads become a cheap AES-GCM unwrap + decrypt — no KDF per row.

• KEK KDF: Argon2id via golang.org/x/crypto/argon2 (already vendored under golang.org/x/crypto v0.50.0 — no new dependency). argon2.IDKey(password, kekSalt, time=3, memory=64*1024, threads=4, keyLen=32). Runs once per unlock, not per row.
• KEK salt: a random 16-byte salt persisted in vault_config under a new key kek_salt (dqlite-replicated, same table/pattern as master_verifier). Generated on first unlock of an uninitialised cluster; read (never regenerated) thereafter.
• DEK: 32 random bytes per row.
• Payload + wrap cipher: AES-256-GCM for both wrapping the DEK and encrypting the payload (authenticated; replaces the AES-128-CBC + separate HMAC-SHA256 Fernet construction).

The vault.value column stays Text. New rows get an ASCII prefix so old and new are unambiguously distinguishable without decoding:

"v2:" || base64std( blob )
blob = 0x01                       # envelope format version byte
     || dekNonce(12) || wrappedDEK(32 ct + 16 GCM tag)
     || payloadNonce(12) || payloadCiphertext(+16 GCM tag)

Detection is robust: legacy values are pure base64std (alphabet A-Za-z0-9+/=, which never contains :), so a leading "v2:" can never collide with a legacy blob. wrappedDEK = AES-256-GCM(KEK, dekNonce, DEK); payloadCiphertext = AES-256-GCM(DEK, payloadNonce, plaintext, aad). Optional AAD = key || scope binds a ciphertext to its row and defeats cross-row ciphertext swaps; make it opt-in behind the format byte so it can be added without another version bump.

The KEK-salt is not stored per row (it is one cluster-wide value in vault_config), so wrappedDEK needs no salt — the unwrap uses the cached KEK directly.

In internal/modules/vault/crypto.go:

• Add type EnvelopeBox struct { kek []byte; legacy *VaultCrypto } implementing the existing CryptoBox interface (factory.go:84-87) — no interface change, so provider.go, factory.go, and service.go:257 wiring are untouched.
• func DeriveKEK(password string, salt []byte) []byte — Argon2id, the only KDF call on unlock.
• EnvelopeBox.Encrypt(plaintext string) (string, error) — mint DEK, GCM-wrap, GCM-encrypt payload, emit "v2:" + base64(blob).
• EnvelopeBox.Decrypt(ciphertext string) (string, error) — dual-path: if the value has the "v2:" prefix, unwrap DEK with kek and decrypt (fast, no KDF); otherwise fall back to legacy.Decrypt (the current PBKDF2 Fernet path) so pre-migration rows still read. legacy is a *VaultCrypto built from the same master password, so both paths are always available while unlocked.
• Keep VaultCrypto intact (legacy reads, the export/import bundle, and the legacy verifier).

In internal/subsystems/impl/vault.go and internal/modules/vault/service.go:

• The KEK salt lives in the DB, which the memoryPasswordManager cannot reach (verifier.go notes “the manager has no DB handle”). Service.Unlock already owns DB-backed verifier lifecycle (service.go:242-267), so it also owns the KEK salt: after the verifier gate passes, read-or-create kek_salt, call DeriveKEK, build the EnvelopeBox{kek, legacy: NewVaultCrypto(password)}, and set it on the factory (reusing the existing SetCryptoBox plumb at service.go:257-258). Add a small PasswordManager.SetCryptoBox(CryptoBox) so CryptoBox() (used by the rotate probe at rotate.go:138) returns the same envelope box. The manager keeps holding the raw password for MasterPasswordForExport (vault.go:309) unchanged.

Extend the verifier to the same "v2:" scheme (verifier.go):

• buildVerifierV2(kek, magic) = "v2:" + base64(AES-256-GCM(kek, nonce, magic)).
• verifyPasswordV2(stored, kek) = GCM-decrypt back to masterVerifierMagic.
• Unlock resolution order: read master_verifier + kek_salt.
  • v2 verifier present → derive KEK, GCM-verify. Match ⇒ correct password (this KEK derivation is the one-per-unlock cost).
  • legacy verifier present → verifyPassword (PBKDF2, one-time), and on success adopt v2: persist kek_salt, rewrite master_verifier as v2.
  • neither present (fresh/upgraded) → generate kek_salt, derive KEK, write v2 verifier (adopt-first-password, preserving today’s behaviour at service.go:263-273).

rotateLocked (rotate.go:127-219) gains a fast path: for a v2 row, decrypt only the wrapped DEK with the old KEK and re-wrap it with the new KEK — the payload ciphertext is never touched. A rotation over an all-v2 vault becomes N tiny GCM ops instead of 2N × 480k-PBKDF2 + full re-encrypt. Legacy rows encountered during rekey are decrypted (PBKDF2) and re-written as v2 under the new KEK, folding migration into rotation. The verifier flip and single-tx atomicity (rotate.go:189-219) are preserved.

This changes stored data on a replicated store, so ordering is load-bearing.

Read-compat dual format. EnvelopeBox.Decrypt reads both formats from day one (prefix detection above). No flag day for reads: a freshly-upgraded node transparently reads legacy rows and any v2 rows written by peers.

Lazy re-wrap on write. Provider.Set always writes v2 once envelope writes are enabled (see epoch gate). Any row that is updated migrates itself. No bulk pass required for correctness.

Explicit migrate pass. Add a doctor/CLI action (extend the existing internal/modules/vault/doctor.go row-iteration) that walks every row, decrypts via the dual-path box, and re-writes as v2. Idempotent — a row already "v2:" is decrypted-and-rewritten (or skipped) with no semantic change. Runs online.

Multi-node / mixed-version cluster — the danger. A node running old code cannot decode a "v2:" value: base64 decode of "v2:..." fails or the Fernet parse rejects it (crypto.go:85-91), so it surfaces a clean decrypt error, never silent wrong plaintext. But that means v2 writes must not begin until every node can read them. Gate envelope writes behind a cluster epoch:

• Add a vault_config key crypto_format (values 1 legacy, 2 envelope).
• New code reads both regardless. New code only writes v2 when crypto_format >= 2.
• An operator (or an all-nodes-upgraded health check) flips crypto_format to 2 once the rolling upgrade completes; the flip replicates via dqlite. Until then, upgraded nodes keep writing legacy Fernet — fully readable by not-yet- upgraded peers.
• kek_salt and the v2 verifier can be written earlier (they are read by new code only, and the legacy verifier stays authoritative for old nodes) — but simplest is to adopt them at the same epoch flip.

Data-loss guardrails.

• kek_salt is critical replicated state: once any v2 row exists, the salt must never be regenerated (a new salt ⇒ a different KEK ⇒ every v2 row undecryptable). Read-or-create must be read-first with a hard guard: never generate a fresh salt if v2 rows or a v2 verifier already exist. It replicates and is captured by dqlite backups exactly like master_verifier.
• Migration writes are per-row upserts through the same unique-tuple path (provider.go), so a crash mid-pass leaves a mix of v1/v2 rows — both readable — never a corrupt row.

Backup bundle stays on v1. The export/import wire format (ExportFormatVersion = 1, transfer.go:42) and the inline BackupManifest secrets bundle are produced by decrypting from storage and re-encrypting under the export password with VaultCrypto (transfer.go:214, :321). That path is deliberately out of scope: it keeps emitting v1 Fernet so existing backups — and backups taken by mixed-version clusters — remain decryptable by any SHC version. Import continues to decrypt v1 and store via Provider.Set, which writes v2 once the epoch is flipped. No ExportFormatVersion bump.

• Mixed-version cluster reads v2 it can’t decode. Mitigated by the crypto_format write-epoch gate (writes stay v1 until all nodes upgraded) and by decode failing closed with a clear error, never silent corruption.
• KEK-salt loss / regeneration. Read-first with a “never regenerate if v2 data exists” guard; salt replicated via dqlite and included in DB backups.
• Format-detection false positive. Impossible in the chosen scheme: legacy blobs are pure base64 (no :), v2 is prefixed "v2:". A unit test asserts a legacy blob never matches the v2 detector.
• Argon2 memory cost on constrained nodes. 64 MiB × threads once per unlock is bounded and infrequent; params are constants and tunable. The per-row cost (the actual hot path) drops to microseconds.
• Downgrade after migration. Rolling back to pre-envelope code after crypto_format = 2 leaves unreadable v2 rows. Documented as a one-way epoch; rollback requires a crypto_format = 1 re-migrate pass (decrypt v2, re-write v1) before downgrading — feasible because new code holds both boxes.
• Out of scope: the export/import bundle and BackupManifest inline vault format; external providers (aws/gcp/vault/etc. under providers/), which never touch VaultCrypto; the CryptoBox interface shape (unchanged).

Unit (internal/modules/vault):

• Envelope round-trip: Encrypt→Decrypt for empty, small, and >1 AES-block payloads.
• Dual-format decrypt: an EnvelopeBox decrypts both a legacy VaultCrypto blob and a v2 blob under the same password; cross-format tamper (flip a GCM tag byte) fails closed.
• Detection: a corpus of legacy base64 blobs never matches the "v2:" detector.
• Verifier: (a) v2 build/verify round-trip; (b) legacy verifier + correct password unlocks and adopts v2 (kek_salt + v2 verifier persisted); (c) wrong password still rejects on both formats (mirror verifier_test.go).
• KEK-salt guard: unlock never regenerates kek_salt when v2 rows/verifier exist; fresh cluster generates exactly once.
• Benchmark: BenchmarkEnvelopeDecrypt vs the current PBKDF2 decrypt, demonstrating no KDF on the read path (extend crypto_test.go).

Integration:

• Rotate over a populated v2 vault re-wraps DEKs only (assert payload ciphertext bytes unchanged, DEK-wrap bytes changed) — new assertions on rotate.go path; existing rotate tests still pass for legacy rows.
• Migrate pass: seed a vault with legacy rows, run the doctor migrate action, assert all rows become "v2:", values decrypt identically, and a second run is a no-op.
• Epoch gate: with crypto_format = 1, Provider.Set writes legacy; after flip to 2, Set writes v2; both remain readable.
• Mixed-version simulation: decode a v2 blob with the legacy-only VaultCrypto and assert a clean error (no panic, no wrong plaintext).

App / e2e:

• Install an app resolving multiple ?autogen refs on a populated vault and assert the read fan-out no longer serialises on PBKDF2 (wall-clock regression guard).
• Backup→restore across the format change: capture a BackupManifest on a v2 vault, restore, assert secrets decrypt (bundle still v1) and land as v2 rows.

Just lower the PBKDF2 round count / cache the derived Fernet key per unlock. Caching is impossible in the current design because the salt is per-row and embedded in the blob (crypto.go:70, :92) — the key is a function of the row’s salt, not just the password, so there is no single unlock-scoped key to cache. Making the salt cluster-wide to enable caching is most of the envelope work but yields a strictly weaker result: it forfeits fast rekey (still a full re-encrypt of every payload on rotation) and keeps the AES-128-CBC+HMAC Fernet construction instead of authenticated AES-256-GCM. Envelope encryption gets the read-path win and cheap rekey for the same migration cost, so the intermediate step loses.