The portal↔cluster channel: protocol spec, 2026-08-16
Status: open design. The cluster side is in
modules/portal; the portal side is not in this repo. This is the wire contract the Build rungs implement against. It elaborates2026-08-14-portal-architecture.md§4 (ruled) and the marketplace doc’s §6 install path. Every (ruled) item is the owner’s; every other choice is argued here and is open to revision except where marked INVARIANT: those are the security properties the design exists to hold, and changing one is an owner ruling, not a refactor.
The cluster-side half of this spec ships in this branch (shc/modules/portal).
The vendor side is Frappe and is not in this repo.
0. The five invariants
Section titled “0. The five invariants”| # | Invariant | Enforced by |
|---|---|---|
| I1 | The cluster dials out. The portal never dials in. The portal holds no cluster address, no cluster credential, and no route into the estate. | No inbound route exists (§7.3). The portal is given a public key, never a secret. |
| I2 | An unregistered estate has no channel, no listener, no timer, and no attack surface from this feature. Refusal is the default and it is structural, not a config flag. | Subsystem.Start returns before creating anything when no registration row exists (§7.1). |
| I3 | v1 carries exactly two capabilities: status UP, install dispatch DOWN. Not “an envelope that happens to carry two verbs”. | Closed verb table with per-verb typed parameters; DisallowUnknownFields; a literal-list ceiling test (§4). |
| I4 | Execution is at-most-once. A reconnect, a redelivery, or a portal replay never installs twice. | Claim-before-effect ledger on a raft-replicated row (§6). |
| I5 | Every dispatch is a chained audit row before it has an effect, and never a per-tick row for the heartbeat. | §8, following ce@a0fef7f6. |
1. Transport: the choice and its failure mode
Section titled “1. Transport: the choice and its failure mode”The candidates
Section titled “The candidates”| Latency the owner feels | Behind NAT / captive firewall | Vendor cost @ 1000 estates | Verdict | |
|---|---|---|---|---|
| Persistent WSS carrying commands | best (~0) | WS upgrade is the first thing a transparent proxy breaks | 1000 sockets is fine on a Node sidecar, but the sidecar is Frappe’s socketio, the least-hardened process in the stack, and it would be holding the dispatch path | rejected: puts the trust boundary in the wrong process |
| Long-poll (hanging GET) on Frappe | good (~0) | excellent: an ordinary HTTPS request | fatal: a held request pins a gunicorn worker for the whole hold. Frappe ships single-digit workers. 1000 estates × 60 s holds needs 1000 concurrent workers | rejected on the vendor side |
| Periodic poll only | bad: the owner presses Install and waits up to a full interval | excellent | cheap, but the interval is the latency, so cheap and fast are opposites | rejected alone |
The choice: poll floor + an authority-free doorbell
Section titled “The choice: poll floor + an authority-free doorbell”Two mechanisms, deliberately unequal:
-
The sync loop (mandatory, the whole protocol). The agent makes a short, ordinary
POSTto the portal everyfloor_seconds(default 60 s, full-jitter). It carries the heartbeat up and receives a bounded batch of pending dispatches down. The Frappe worker is held for milliseconds. This is the only path with authority; everything the protocol does, it does here. -
The doorbell (optional accelerator, zero authority). The agent may hold an outbound WSS to the portal that carries nothing but a wake signal: no payload, no command, no identity claim it can act on. On a wake, the agent runs a sync early. That is all it can ever cause.
Why this shape and not one of the three above:
- The owner feels ~250–400 ms, because the doorbell collapses the poll interval to a round trip on the one occasion that matters (he pressed Install).
- The failure mode is latency, never function. If the WS upgrade is blocked by his pfSense, by a captive middlebox, or by the portal being restarted, the floor poll still delivers; the install lands within 60 s instead of within a second. Nothing is lost, no state diverges, no error is surfaced to the operator. State the failure mode plainly: a broken doorbell degrades an install from “instant” to “within a minute” and is otherwise invisible.
- The doorbell cannot be abused, by construction. The worst a spoofed, hijacked, or malicious doorbell can do is make the agent perform a sync it was going to perform anyway: an authenticated request to the pinned portal URL. It therefore needs no authentication of its own, and a compromise of the Node sidecar grants an attacker nothing. INVARIANT: the doorbell frame is never parsed for content beyond “wake”.
- Vendor cost @ 1000 estates: ~17 short requests/second at the floor, plus 1000 idle sockets on one Node process. Both are unremarkable.
Reconnect/backoff (both mechanisms): full-jitter exponential backoff,
min(base·2^n, cap) randomised over [0, that), base 2 s, cap 15 min. Jitter
is not a nicety: 1000 estates that back off in lockstep re-stampede a portal
that just came back. The floor poll’s own jitter is ±20 %.
Ordering. The sync is a request/response, not a stream: there is exactly one in flight per estate at a time, and the agent never runs two concurrently. That removes reordering from the protocol entirely.
2. Identity and authentication: both directions
Section titled “2. Identity and authentication: both directions”2.1 Cluster → portal: an estate-held private key, never a bearer token
Section titled “2.1 Cluster → portal: an estate-held private key, never a bearer token”At shc portal register <code>:
- The daemon mints an Ed25519 key pair in-process. The private half never leaves the estate and is never transmitted, not even at enrolment.
- It POSTs
{code, public_key, cluster_name, node_count, version}toPOST {portal}/api/method/portal.agent.enroll. - The portal validates the registration code, short-lived (15 min),
single-use, bound to the portal account that minted it in the UI, and
returns
{cluster_uid, key_bundle, channel_url, floor_seconds}.
Every later request is authenticated by signing the request, not by presenting a secret:
X-SHC-Cluster: <cluster_uid>X-SHC-Timestamp: <unix seconds>X-SHC-Nonce: <16 random bytes, base64>X-SHC-Signature: ed25519(agent_key, canonical) , base64canonical = "shc-portal-agent/1\n" + METHOD + "\n" + PATH + "\n" + timestamp + "\n" + nonce + "\n" + hex(sha256(body))Why signing and not a bearer token: this is the part that matters:
- A portal database compromise cannot impersonate any cluster, because the portal only ever holds public keys. The vendor cannot even accidentally leak a cluster credential; it does not have one. (I1.)
- Nothing secret ever appears in a proxy log, a Frappe error page, or a support bundle.
- Replay is bounded by
timestamp(±300 s) +nonce(portal-side seen-set for the same window).
Rotation. The agent re-keys on demand (shc portal rotate-key, owed) and on
a portal-signalled rekey_requested flag in the sync response: mint a new pair,
present it signed by the old key, portal swaps. Old key is invalid the moment
the swap acks. No downtime and no operator step.
Where it lives on the estate. The vault’s protected section, via
vault.Service.PutRaw("portal/agent-key") → row shc/ca/portal/agent-key under
the reserved __ca__ tenant sentinel. That section is structurally
unreachable by any ref+vault:// at any scope (ce modules/vault/paths.go),
so no tenant template, no job var, and no vault HTTP read can ever resolve the
agent key. It is also the estate-replicated encrypted store, which is what the
leader-mobility of the channel requires. (The section is spelled ca/ for
historical reasons; it is really “the daemon-direct, ref-unreachable section”.
An honest rename in ce is owed and is not this rung’s business.)
Consequence, stated because it is load-bearing: a locked vault means no
channel. The agent refuses with X503490PRTVLT, logs once, and retries; it
does not fall back to a weaker store.
2.2 Portal → cluster: the direction that actually matters
Section titled “2.2 Portal → cluster: the direction that actually matters”A vendor-hosted button causes software installation on customer infrastructure. The estate must be able to prove the instruction came from the vendor, and the proof must not be “TLS to a hostname”.
Two-level trust, cold root + warm issuer (the same pattern licensing already
uses, modules/license/pubkey.go):
- The binary embeds portal root public keys (primary + reserve). The private seeds are offline, out of band, and are used a handful of times in the product’s life.
- The root signs a key bundle:
{keys:[{kid, pub, purpose:"dispatch", not_after}], issued_at, expires_at}. The agent verifies the bundle against the embedded roots, pins it, and refreshes it on every sync. - Each dispatch is signed by a bundle key, referenced by
kid.
So the vendor’s day-to-day dispatch key is warm and rotatable without a fleet release, while the thing that would let an attacker install software on every customer estate is a key that lives offline.
Fail closed, on purpose. This branch shipped the root list EMPTY. With
no root compiled in, portal.VerifyDispatch and shc portal register both
refuse with X503485PRTNRT. The feature was inert until the vendor key
ceremony put real roots in the source. That was deliberate: a channel that
installs software must not be switch-on-able by accident, and inventing key
material to make a demo work would be worse than shipping it off.
★ THE CEREMONY WAS PERFORMED 2026-08-18 (§11.1, now closed). Two cold roots (
portal-root-primary-2026andportal-root-reserve-2026) are inmodules/portal/trust.goas public halves; the warm dispatch keyportal-dispatch-2026-08lives in the portal bench’s site config. The operational half isdocs/internal/portal-vendor-key-ceremony.md.★ AND THE CEREMONY ALONE WOULD NOT HAVE SWITCHED THE FEATURE ON. The portal signed
json.dumps(payload, sort_keys=True); the daemon rebuilds its message by re-marshalling the typed Go struct. Those are different bytes for every directive, not an edge case, so the day the roots landed the channel would have refused every install withX401479PRTSIG, which is indistinguishable, from either side, from a wrong key. Three further defects rode with it:protocolwas absent from the envelope (a closed decode rejects the whole thing),kidwas attached after signing so it was not covered, andsignaturewas omitted rather than emitted empty. All four are fixed, andtests/unit/portal/directive_signing_test.gonow drives the shipped Python signer against the shipped Go verifier so the two halves cannot drift again without a test going red.
2.3 What deregistration actually revokes
Section titled “2.3 What deregistration actually revokes”shc portal deregister performs four independent revocations, and each one
alone is sufficient:
- The agent key is deleted from the vault. The estate can no longer sign a sync, so it can no longer be handed a dispatch.
- The registration row is tombstoned (a sentinel, not a delete; an absent
row means “I do not know”, a tombstone means “this estate has released it”;
the licence delink ruling,
modules/license/distribution.go). The subsystem stops on the next tick and never restarts. - The portal is told (
portal.agent.deregister, signed with the key that is about to die), which marks the cluster revoked so its public key stops authenticating. - The master-realm federation edge is removed (§9), so vendor operator login into the estate’s master realm stops.
--local performs 1, 2 and 4 without the portal, and refuses to fail: an
estate must be able to sever unilaterally when the vendor is unreachable, has
gone away, or is the thing you are severing from. This is the self-hosted
premise; a deregistration that needs the vendor’s cooperation is not one.
3. The sync exchange
Section titled “3. The sync exchange”POST {portal}/api/method/portal.agent.sync, signed per §2.1.
// request: the heartbeat (UP){ "protocol": 1, "cluster_uid": "…", "cursor": "…", // last dispatch cursor this estate has settled "agent": { "version": "0.0.57", "edition": "shc" }, "posture": { // ruled §4: version, health, licence, inventory "healthy": true, "nodes": 3, "license_tier": "hobby", "license_state": "keyed" }, "catalog": [ // the marketplace's per-cluster truth overlay {"app": "nextcloud", "version": "30", "stack": "cloud", "environment": "prod"} ], "outcomes": [ // idempotent re-report; see §6 {"dispatch_id": "…", "state": "done", "operation_id": "…", "stack": "…"} ]}// response: the dispatch batch (DOWN){ "protocol": 1, "floor_seconds": 60, "key_bundle": { … , "signature": "…" }, // root-signed, §2.2 "rekey_requested": false, "dispatches": [ { …signed dispatch envelope, §4… } ], "cursor": "…"}posture and catalog are the “status” half of the ruled ceiling. Note that
status only ever goes up: there is no downward status verb, because the
portal asking a question would be a second capability. The portal learns the
estate’s state as a side effect of the estate speaking.
4. The dispatch envelope: and why the ceiling is structural
Section titled “4. The dispatch envelope: and why the ceiling is structural”{ "protocol": 1, "dispatch_id": "018f…", // portal-minted UUID, the idempotency key "cluster_uid": "…", // this estate, and no other "issued_at": "2026-08-16T12:00:00Z", "expires_at": "2026-08-16T12:10:00Z", "portal_actor": "clay@…", // WHO pressed Install, a vendor ASSERTION "verb": "install", "install": { … typed params … }, // exactly one params field, named by the verb "kid": "dispatch-2026a", "signature": "…" // ed25519 over the canonical bytes, minus signature}Five structural properties, each one a thing that has to be deleted for the ceiling to be lost, not merely a value that has to change:
- There is no
argsmap, anywhere. Each verb owns a named, typed parameter struct on the envelope. A new capability cannot be smuggled through an existing field because no existing field is open-ended. DisallowUnknownFieldson every decode. A portal that sends a field this binary does not know is refused, not tolerated. An unknown verb isX403480PRTVRB, not a silent skip.- The verb table is a closed Go map from
Verb→{typed params, executor}. Adding one requires a struct, an executor, a table entry, and a change to the literal list inTestV1CeilingIsExactlyInstall, whose failure message says in as many words that widening this set is a security decision requiring an owner ruling. - The install executor cannot be pointed at arbitrary code.
install.sourceis validated against the catalog app-name grammar only, never a git URL, never a local path, never an archive, all of whichshc installitself accepts. “Install dispatch” must not become “run what I host”. INVARIANT. - Envelope binding.
cluster_uidmismatch →X403487PRTAUD(a dispatch for estate A cannot be fed to estate B). Outside[issued_at, expires_at]→X410482PRTEXP(a captured dispatch cannot be replayed next week).
The v1 verb set:
install: install a catalog app onto this estateThat is the whole set. status is not in it, because status is the request body
going up.
5. The install parameters
Section titled “5. The install parameters”"install": { "source": "nextcloud", // catalog app name (or repo/app). NOT a URL/path. "version": "30", // optional; default "latest" "stack": "cloud", // optional; defaults from source "environment": "prod", // optional "tenant": "acme", // optional "tls": true, "gated": false, "public": false, "vars": {"admin.email": "…"} // string→string only; the marketplace preset}The executor translates this into the same daemon call the CLI makes
(POST /api/method/shc.stack.install) and follows the returned operation.
It gains nothing the CLI does not have: the dispatch path is a client of the
ordinary install path, never a privileged bypass of it. Every existing guard
(quota, licence, taxonomy, tenancy) applies unchanged, and refuses the dispatch
exactly as it would refuse the operator.
6. Idempotency: at-most-once execution
Section titled “6. Idempotency: at-most-once execution”The agent keeps a dispatch ledger: one raft-replicated kv row per
dispatch_id, namespace portal.dispatch.
receive dispatch ├─ verify signature, binding, window → refuse, record, report ├─ CLAIM: conditional insert of {state:"running"} ← THE WRITE HAPPENS FIRST │ └─ row already present? → REPLAY: do not execute; re-report the │ recorded outcome and move on. (X409481PRTDUP) ├─ execute └─ COMPLETE: update row {state:"done"|"failed", outcome}The claim is written before any effect, through raft, so it survives the crash that would otherwise make “did I already run this?” unanswerable. Only the leader runs the channel, so the claim is uncontended in practice; the conditional write is what makes it correct anyway during a leadership flip.
- Delivery is at-least-once (the portal may redeliver until the cursor advances).
- Execution is at-most-once (the ledger).
- Outcome reporting is idempotent (re-reporting a
donerow is a no-op).
Ledger retention is bounded: 90 days (the ce@a0fef7f6 lesson: a table
nobody prunes is a table that eventually costs you the estate). A dispatch that
arrives with an id older than the retained window is refused
(X410482PRTEXP), not executed: once the ledger can no longer prove the
dispatch has not run, refusing is the only safe answer. The dispatch’s own 10
minute expires_at means this can only ever be a portal bug or an attack, never
a legitimate slow delivery.
7. Refusal is the default: three named proofs
Section titled “7. Refusal is the default: three named proofs”An estate that never registered must be indistinguishable from one built without this feature.
| Property | Named test |
|---|---|
7.1 No goroutine, no socket, no timer, no HTTP client is created when no registration exists. Start returns nil having allocated nothing. | TestUnregisteredEstateStartsNoChannel |
| 7.2 The daemon exposes no route that can receive a dispatch. The portal router’s entire surface is enumerated and asserted to be read-only and local. | TestPortalRouterExposesNoInboundDispatchRoute |
| 7.3 Registration cannot happen without an operator running the verb with a code. No config key, no env var, no auto-enrol, no first-boot default. | TestRegistrationRequiresOperatorCode |
And the ceiling proof:
| 7.4 The downward verb set is exactly ["install"]. | TestV1CeilingIsExactlyInstall |
8. Audit: recorded the way the vault lane left the system
Section titled “8. Audit: recorded the way the vault lane left the system”ce@a0fef7f6 established the rule this spec follows: a mutation gets a
per-operation synchronous chained row, because its effect survives a crash and
so must its record; a read does not, because a per-read raft commit on a hot
path is how you get 295,822 commits and a login latency outage.
Applied here:
| Event | Code | Written how |
|---|---|---|
| registered with the portal | A200491PRTREG | synchronous chained audit row |
| deregistered | A200492PRTDRG | synchronous chained audit row |
| dispatch executed | A200493PRTDSP | synchronous chained audit row, written at claim time, before the effect |
| dispatch refused | A200494PRTRFS | synchronous chained audit row |
| channel state change (connected / degraded / stopped) | V200495PRTCHN | non-audit event, transitions only |
The heartbeat is never audited. A 60 s floor is 525,600 rows a year of five
constant fields, precisely the V200303VLTGET pathology, and precisely as
useless: it would carry no principal, no correlation, and one message. The
estate’s last-sync time is a replicated row the status verb reads, not an
audit trail. If a future auditor genuinely needs sync counts, the answer is the
windowed digest the vault lane built (V200307VLTRDG-style), not per-tick rows.
A200493PRTDSP carries portal_actor, the account the vendor says pressed
Install. Recorded honestly as a vendor assertion: the cryptographic actor
this estate can actually verify is the dispatch kid, and both are in the row.
An estate cannot verify a human it has never met; pretending otherwise in an
audit trail would be worse than saying so.
9. The master-realm federation carve-out
Section titled “9. The master-realm federation carve-out”Registration federates the vendor portal IdP into the cluster’s master realm
only (ruled). The sso entitlement, which gates external IdPs for user
realms, is untouched; free estates still have user-facing SSO off.
The guard exemption is deliberately the narrowest expressible thing
(keycloak/spi/.../FederationGuard.java):
realm.getName().equals("master") AND resourceType == IDENTITY_PROVIDER (never COMPONENT / user federation) AND alias.equals("shc-portal") (a compile-time constant, not config)All three must hold. It is not driven by an environment variable, because a variable is a widening surface: an exemption you can set from outside is an exemption an attacker can set from outside. One alias, one realm, one resource type, hard-coded.
10. Cluster-side surface (what this branch ships)
Section titled “10. Cluster-side surface (what this branch ships)”shc portal register <code> [--portal URL] mint the key, enrol, federate, startshc portal status registration, channel, last sync, ledgershc portal deregister [--local] revoke all four ways (§2.3)GET /api/method/shc.portal.status read-only, localshc portal is the chosen spelling of the doc’s (design: verb name):
- The noun-first subtree is the house shape (
shc support bundle,shc license activate,shc leader status). - A bare top-level
shc registersquats the namespace and reads as mandatory. Registration is “optional in every tier”; the verb should not imply otherwise. join/linkcollide with node formation (shc cluster enroll).- “register” is the owner’s own word: “the point of the portal is registering your clusters.”
deregisteroverunregister/leave, for symmetry with the verb it undoes.
Placement: shc only, never ce (ruled). ce carries none of this: no module, no route, no verb, no config key.
11. Owed, and explicitly not done here
Section titled “11. Owed, and explicitly not done here”-
The vendor key ceremony.DONE 2026-08-18. Two cold roots (primary + reserve) minted offline, public halves inmodules/portal/trust.go, a root-signed bundle naming the warm dispatch key installed in the portal bench’s site config. Runbook, rotation and the renewal date:docs/internal/portal-vendor-key-ceremony.md. The private seeds are out of band and appear in no repository, no CI variable and no estate. -
The portal (Frappe) half:
portal.agent.enroll,.sync,.deregister, the registration-code doctype, the dispatch queue, the doorbell sidecar.★ STILL THE BLOCKER, AND MEASURED 2026-08-18.
.synclanded asportal.agent.sessionand is live.portal.agent.enrollandportal.agent.deregisterdo not exist, measured against the live portal, which answersmodule 'portal.agent' has no attribute 'enroll'. So no estate can register, and therefore no signed directive can reach one no matter how correct the signing is. After the ceremony the Install button stops failing with “no trust root” and starts failing with “this cluster is not registered”. This is now the single thing standing between a store click and a running app. -
The federation executor. The script builder and the alias ship here; the apply/remove path needs an exported kcadm runner in ce (
auth.KeycloakAdminClient.runScriptis unexported today); a one-method ce change, not a portal feature in ce. -
shc portal rotate-key(§2.1 describes the flow; the verb is owed). -
Support-bundle grants over this channel, a third capability, and therefore an explicit ceiling change, not an extension. It gets its own ruling.