Portal Terraform runner: secret injection and the run ledger (design)
Status: built, with owner rulings on 2026-08-15 and 2026-11-01. Customer credentials and harvested secrets use Frappe Password fields. State lives in a separate B2 object bucket, encrypted per cluster with AES-256-GCM; only its data key lives in a Password field.
vault_passworddefaults togenerated(class B). Sections 1-4 retain the original design discussion; §4b and §5.1 state the binding custody and mode rulings. This elaborates §6 and §10.5-10.6 of2026-08-14-portal-architecture.md.
1. The three classes of secret
Section titled “1. The three classes of secret”A portal-run terraform apply touches three kinds of secret, and they have
different lifetimes, different owners, and must not share a path.
| class | example | who owns it | where it may appear |
|---|---|---|---|
| A: customer cloud credential | HCLOUD_TOKEN, AWS access/secret, Exoscale API key | the customer | process env of the runner, for the life of one run |
| B: cluster secret Terraform GENERATES | ssh_private_key (module output, sensitive = true), vault_password, the cluster CA material | nobody yet, born during apply | state, then harvested to OpenBao |
| C: cluster secret Terraform CONSUMES | ACME eab_hmac, registry auth | the portal mints it per-cluster | passed by reference, never by value |
Class B is why the ruling says custody of state is custody of the cluster:
terraform/modules/shc/*/outputs.tf exposes ssh_private_key as a sensitive
output, so the private key is in the state file by construction.
2. Injection, per class
Section titled “2. Injection, per class”Class A: environment only. Cloud credentials are exported into the
runner process environment (HCLOUD_TOKEN, AWS_ACCESS_KEY_ID, …) for the
duration of one run and never written to disk. Specifically: no .tfvars
file, no -var on the command line (argv is world-readable via /proc),
no Frappe DocType field, no bench log. The runner fetches them from OpenBao
at job start, holds them in memory, and the job’s process death is the
expiry. Where the provider supports short-lived credentials (AWS STS
AssumeRole), the runner prefers minting a scoped session over holding the
long-lived key.
Class C: ref+vault:// indirection, which the provider already speaks.
terraform/provider/internal/provider/cluster_resource.go:213 documents
eab_hmac as “(ref+vault:// ok)”, and config_masked_readback_ssh_test.go
pins the deliberate rule that a ref+vault:// pointer is never masked
(never_redact_value_prefixes); the reference stays legible in state and
plan output while the value only ever exists inside the cluster’s vault.
That is exactly the primitive the portal wants: the portal writes
ref+vault://cluster_vault_password?autogen into the plan, and the secret
itself is born in, and never leaves, the target cluster.
RULED 2026-11-01: vault_password is class B. It is generated during
apply and lands in state, as the owner’s 2026-08-16 ruling intended.
render.default_vault_password_mode is "generated"; "reference" remains
an implemented alternative. The separate encrypted bucket in §4b protects
state, and every run records which mode it used.
Class B: harvest then encrypt. After a successful apply the runner
reads the sensitive outputs, writes them into the customer’s OpenBao
compartment (this is what the portal hands the customer when they ask for
their SSH key), and the state itself is stored encrypted at rest in
OpenBao, never on the bench filesystem and never in the Frappe DB. State
is fetched per run, locked for the duration, written back encrypted, and the
plaintext working copy lives in a per-run tmpfs directory destroyed with
the job, including on failure and on kill.
Blast-radius rule. One OpenBao compartment per customer, so a portal app-server compromise does not yield every customer’s infrastructure. The runner’s own OpenBao token is scoped to the single customer path for the single job, minted at job start and revoked at job end.
3. The run ledger (what you review)
Section titled “3. The run ledger (what you review)”Every provisioning action is a Terraform Run record. This is the surface the owner asked for: “the logs of terraform so i can review them for issues.”
Each run carries: cluster, customer, cloud, action (plan/apply/destroy),
who or what triggered it, module + provider versions, start/end, exit status,
the resource-change counts, and the full captured stdout/stderr, streamed
live to the dashboard and retained after.
Redaction is a pipeline stage, not a hope. Three defenses, because any one of them alone fails:
TF_LOGis neverDEBUG/TRACEin the portal runner. Terraform’s debug logging prints provider request/response bodies; that is a bulk credential leak into the log store. DefaultTF_LOGunset; a support engineer may raise it only for a run explicitly markeddiagnostic = true, whose log is quarantined to a shorter retention and flagged in the UI.- Value-based scrubbing on the way in. The runner knows every class A
and class C secret it injected for that run; it scrubs those exact byte
sequences (and their base64/JSON-escaped forms) out of the stream before
the log is persisted. This catches the case Terraform’s own
(sensitive value)masking misses: a secret echoed by a provider that never marked it sensitive. - A guard test that proves it. A mutation-proven test injects a known canary secret into a fake run whose fake provider echoes it, and asserts the persisted log does not contain the canary. Removing the scrubber must red that test by name. Without this, the redactor rots silently.
State never enters the log. Not on error, not in a crash dump. A failed apply reports resource addresses and diagnostics, never the state document.
Retention and access: run records are customer-visible for their own clusters (that is the review surface), staff-visible under the same support-bundle consent modes ruled in §5: a one-off code or a standing grant, not ambient staff access to every customer’s infrastructure log.
4. Failure handling
Section titled “4. Failure handling”A failed apply must not leak billing. The runner’s failure path is
reap-on-failure (§10.6): on a non-zero apply the runner attempts
destroy of what that run created, and if the destroy also fails it opens a
leak record naming the cloud, the run, and the resource addresses, a
loud, tracked artifact rather than a silent orphan. Two independent findings
from the 2026-08-14 campaign motivate this: a Linode volume that billed
silently because product-created volumes carry no estate tag, and perfchaos
nodes that were invisible to the documented cleanup selector because the
campaign tag was written as a label value instead of a label key. The
portal must tag every provisioned resource in the house form, and the leak
record is the backstop when tagging or teardown fails.
Per the C1 amendment, the reaper is campaign insurance and is explicitly not the portal’s answer; the portal relies on the product teardown path plus this leak record.
4b. RULED 2026-08-15: Frappe password fields, not OpenBao (owner)
Section titled “4b. RULED 2026-08-15: Frappe password fields, not OpenBao (owner)”Owner ruling, verbatim: “use the frappe db to store the secrets using password field which i think is encrypted”. OpenBao is OUT for phase 1.
He is right that it is encrypted: a DocField of fieldtype Password is stored
in the __Auth table (not the doctype table), encrypted with Fernet
(AES-128-CBC + HMAC) keyed on the site’s encryption_key from
site_config.json, and read back only through get_password(). It never
appears in frappe.db.get_value, in a list view, in the API’s normal document
payload, or in the desk UI. That is genuinely better than a plain field and it
is the framework’s own supported answer.
What this ruling changes: §2’s class A (customer cloud credentials) and
class B (harvested SSH keys) live in Password fields on the customer’s
Frappe record. ref+vault:// (class C) is unaffected; those never touched
the portal anyway. The per-run injection rules in §2 stand unchanged: read via
get_password() at job start, into the process environment only, never a
tfvars file, never argv, never a log.
Three properties the framework does not give us, recorded so they are chosen rather than discovered:
- One key for the whole site.
encryption_keydecrypts every customer’s secrets, and it sits insite_config.jsonon the same host as the database. There is no per-customer separation; a single host compromise or a leaked backup pair (DB + site config) exposes every customer at once. Mitigation while staying on Frappe: keepencryption_keyout of the DB backup path, hold it in the container’s environment/secret mount rather than the on-disk file where the platform allows, and treat DB+config as a single custody boundary in the runbook. bench backupis now a secret export. Any routine backup, replica, or staging restore carries__Auth. Mitigation: the portal bench’s backups are encrypted at rest and access-controlled like the secrets they contain, and no prod→staging data copy is permitted without scrubbing__Auth. This must be written into the operational runbook, not assumed.- No lease, no revocation, no read audit.
get_password()leaves no trail. Mitigation: the §3 run ledger becomes the audit surface; every run records which customer’s credentials it read, so “which secrets did that job touch” stays answerable at the job level even though the framework cannot answer it at the secret level.
RULED 2026-11-01: state has its own custody boundary. Terraform state
leaves __Auth. It lives in its own bucket on the existing bitspur B2 account,
using the S3-compatible API. Each cluster gets a random 32-byte data key in
Terraform State.data_key, a Password field. AES-256-GCM encrypts each state
object with the cluster name as authenticated additional data; its key is
<prefix>/<cluster>.tfstate.enc. A DB backup holds keys without state, and the
bucket holds ciphertext without keys. Neither alone recovers the state.
There is no fallback to Password chunks. The runner refuses before taking a state lock or creating a workdir when the bucket configuration is missing, and names the missing configuration keys in the failed run’s redacted log. The old chunk DocType is removed by a migration patch. Credentials and harvested cluster secrets retain the 2026-08-15 Password-field ruling.
4c. BUILT 2026-08-15: the runner exists
Section titled “4c. BUILT 2026-08-15: the runner exists”Implemented at frappe/apps/marketplace/marketplace/provisioning/, with the
operating half at docs/guide/operations/portal-secret-custody.md. Everything
in §2, §3 and §4 above is code with a guard test behind it, and every §4b
mitigation is implemented rather than noted: custody of the site
encryption_key is stamped on each run row, scripts/portal-backup-guard.sh
refuses a prod→non-prod restore still carrying __Auth, and secrets_read on
each run row is the job-level read audit.
Two implementation notes worth carrying forward because they are not obvious from the design text:
- The provider was already published. 38 versions in the project’s generic
registry tracking tags to
0.0.56, plus the0.0.0-latestrolling channel, behind an immutability guard. There was never a publishing prerequisite; §10.6 of the architecture doc is stale on that point. What there IS is a choice, and the runner pins an exact release: the rolling channel is re-uploaded on every main push, so pinning it would let a bad push silently change what every customer provision does, with no release and nothing to roll back to. - State is encrypted per cluster in B2. The 2026-11-01 ruling replaces the Password-chunk implementation. The database keeps the data key and object metadata, never the state document.
5. Open questions for the owner
Section titled “5. Open questions for the owner”vault_password: RULED 2026-11-01, generated (class B). This makes explicit the owner’s 2026-08-16 ruling that the vault password lives in state.render.default_vault_password_modeis"generated"; the runner harvests it into a Password field after apply. State’s separate bucket and per-cluster encryption provide its custody boundary."reference"remains implemented as the alternative, and every run records its mode.- Does the customer ever see raw cloud credentials again after entry? (Write-only-then-opaque is the safer default; it costs a re-entry when a key rotates.)
- Log retention window, and whether a customer may delete their own run logs (auditability vs. their data rights).
- Concurrency: one run at a time per cluster is assumed (state locking demands it); confirm the queue is per-cluster, not global.
- Managed phase: when we hold the cloud account (§6), class A collapses into our own credential; the compartment-per-customer rule still applies to state and class B, but the injection story changes.