Skip to content
SHC Docs

The install request / approval state machine: BUILT 2026-08-16

Status: BUILT, behind the portal flag, with a named list of what is not reachable yet. This is the implementation record for 2026-08-16-portal-cluster-identity.md Ruling 1 and 2026-08-16-marketplace-entitlement-and-licensing.md §6. Both documents are design records; this one says what now exists in the tree, what it is guarded by, and (§5) precisely which claims about an end-to-end paid install would be false today.

THE RULING. Every paid install is a REQUEST. Policy decides whether a human sees it. A spending limit is an AUTO-APPROVAL RULE, not a second path. One state machine, one audit record, one grant point.


piecefile
the state machine, four policies, three traps, the grant pointfrappe/apps/portal/portal/install_request.py
the audit rowfrappe/apps/portal/portal/portal/doctype/install_request/
the approver’s limitfrappe/apps/portal/portal/portal/doctype/install_spending_policy/
hourly expiry sweepfrappe/apps/portal/portal/hooks.py (scheduler_events)
the wire contractmodules/portal/wire/wire.go: InstallRequest, InstallOutcomes, InstallDispatch.RequestID, the state/decision/period constants
goldens (mirrored both sides)tests/unit/modules/portal/testdata/install_{request,outcomes}.golden.json and byte copies in tests/unit/portal/testdata/
behaviour: 37 scenariostests/unit/portal/testdata/install_request_driver.py
harness + structural guardstests/unit/portal/install_request_test.go

There is no second code path. An auto-approved install and a human-approved install are the same Install Request row, differing only in decision (auto_approved_by_policy vs approved_by_human) and decided_by (the literal policy vs the portal account). The scenario every_paid_install_is_one_row_whether_policy_or_human_decided asserts both rows carry the same populated audit fields; the mutation that made an auto-approval record no decision turned exactly that scenario red.

1.1 The four policies, and one ruling made inside them

Section titled “1.1 The four policies, and one ruling made inside them”

Install Spending Policy, one row per cluster, set by the approver:

  1. per-request cap: a single request above it goes to a human.
  2. rolling budget: monthly, evaluated against committed recurring spend; plus a separate one-time allowance, because a purchase that happens once must not consume a budget that renews.
  3. app allowlist: apps that never need approval.
  4. requester scope: which tenant realms may be auto-approved.

★ The allowlist waives the HUMAN, never the BUDGET. This was a real fork. An allowlist that also waived the budget would reintroduce the recurring-spend trap through the back door: twelve allowlisted $20/mo apps under a $100/mo budget would each be individually “pre-approved” and land at $240/mo. Pinned by policy3_app_allowlist_waives_the_human_but_never_the_budget.

★ Requester scope gates auto-approval; it never auto-denies. An out-of-scope requester goes to the approver rather than being refused, because auto-denying hides the request from the only person who can say yes. Pinned by policy4_requester_scope_gates_auto_approval_and_never_auto_denies.

Defaults are deny. No policy row, a disabled policy, an empty realm list or a currency the policy is not denominated in all mean “a human sees it”. Nothing auto-approves by omission.


2. The three traps, and how each was proved

Section titled “2. The three traps, and how each was proved”

Every guard below was mutation-proved: break it, watch the NAMED scenario go red, restore, green.

(a) Recurring spend is not transaction spend

Section titled “(a) Recurring spend is not transaction spend”

The budget predicate sums monthly_equivalent_minor over every row already holding a reservation or a grant and adds this request’s monthly equivalent. A one-time purchase contributes zero to that sum and draws on the one-time allowance instead. An annual price is normalised to a monthly figure, rounded up; rounding is toward the budget’s protection.

The running total is recomputed from the ledger on every decision and is never a decremented counter: a counter drifts on cancellation, refund, downgrade and failed provisioning, and a drifted budget counter fails OPEN.

Mutation: evaluate the budget against amount_minor (the transaction reading) instead of the committed recurring total:

recurring_spend_consumes_the_budget_forever_not_once ->
AssertionError: auto-approved 12 of twelve $20/mo apps, want 5

Twelve individually-correct decisions, $240/mo. That is the trap, and it is the exact number the ruling predicted.

(b) The budget check and the grant must be atomic

Section titled “(b) The budget check and the grant must be atomic”

Every decision (policy or human) runs inside _lock_cluster, a SELECT ... FOR UPDATE on the Cluster row, taken before the ledger is read and held until the reservation is committed. The Cluster row is the lock because it always exists; a policy row may not, and you cannot lock a row that is absent.

The slow, failable part sits deliberately outside the lock: run_pending_grants acts against a reservation the ledger already holds and releases it (state failed) when the grant does not land.

This was tested with concurrency, not by inspection. The stub (tests/unit/portal/testdata/frappe_stub.py) implements for_update as a real row lock released on commit or rollback, and the scenario runs two threads that rendezvous at the ledger read.

Mutation: drop for_update=True from _lock_cluster:

concurrent_requests_cannot_both_pass_the_same_remaining_budget ->
AssertionError: concurrent outcomes ['approved', 'approved'], want one of each

Two $60/mo requests, a $100/mo budget, both approved. With the lock, one approves and one goes to the approver, and the ledger reads 6000 of 10000 minor units committed.

Precedent honoured: modules/cluster/enroll_capacity_race_test.go exists because the node capacity gate had exactly this shape.

(c) Requests expire, and the requester is told

Section titled “(c) Requests expire, and the requester is told”

expire_stale_requests (hourly, scheduler_events) moves unanswered pending rows to expired, a terminal state distinct from denied, because an approver who said nothing and an approver who said no are different facts and the requester is owed the difference.

The portal cannot notify the requester: it holds no email for a tenant realm user and must never hold one. So every terminal outcome, including expiry, is handed to the daemon through poll_install_outcomes, and keeps being handed back until ack_install_outcomes confirms delivery. A lost poll costs a duplicate notification rather than a silent expiry.

Expiry never touches a reservation: releasing an approved-but-ungranted request behind the approver’s back would silently undo a decision they made (expiry_never_reaps_a_reservation).

Mutations: (i) expire to denied, (ii) withhold expired rows from the poll:

unanswered_requests_expire_to_a_state_distinct_from_denied ->
AssertionError: state 'denied' after the TTL
the_requester_is_told_about_an_expiry_through_the_daemon ->
AssertionError: the expiry was not handed to the daemon; a silent expiry is the failure mode

Open, deliberately: the TTL default is 72 hours with a 720-hour ceiling, and an expired request does not auto-resubmit. That is marketplace-entitlement-and-licensing Q8, and the numbers are a placeholder a ruling should replace, not an answer.


3. The limit is the approver’s, and the cluster cannot reach it

Section titled “3. The limit is the approver’s, and the cluster cannot reach it”

★ A control the restricted party can flip is not a control. Here the restricted party is the cluster’s users, so the mirror of the L2 compliance seal applies: the limit lives on the portal, on the billing relationship, and the daemon never receives it, never stores it and cannot move it.

Three guards, all mutation-proved:

  • set_spending_policy / get_spending_policy are session-only. The guest surface guard fails if either becomes allow_guest, because allow_guest here means “reachable with a cluster token”.
  • Both are owner-checked against Cluster.account_owner, unconditionally, System Manager included, the same posture api.fleet already has, where the filter is the permission.
  • modules/portal/config_keys.go is scanned for the vocabulary of a cluster-side limit (spending, budget, per_request_cap, allowance, …). A daemon config key for any of it is a red test.

Mutation: allow_guest=True on set_spending_policy:

TestInstallRequestGuestSurfaceIsExactlyTheClusterVerbs:
endpoint set_spending_policy must be session-only; a limit the cluster can reach is not a limit

A human may approve past their own limit, and the row says who. The limit is an auto-approval rule, not a ceiling on the approver; that is the escape hatch the request path exists for. What they cannot do is make it invisible.

The approver’s words land in approver_note, a separate column from decision_reason. decision_reason keeps the predicate that sent the request to a human, because “over_per_request_cap, and the owner approved anyway” is a different fact from “the owner approved”, and overwriting it would lose why the request needed a human at all. Mutating the note back over the predicate turns a_human_may_approve_past_the_limit_and_the_row_says_who red.


4. Routing: two populations, neither enumerating the other

Section titled “4. Routing: two populations, neither enumerating the other”
requesterapprover
whotenant realm userportal account (Cluster.account_owner)
held by the portalan opaque requester_ref and a realm nametheir email
notified bythe daemonthe portal

The Install Request DocType has no email, name or user column, and submit_install_request refuses a requester.ref containing @. Both are guarded, and the schema guard is structural on purpose: no runtime test can prove a column that does not exist will never be filled.

Mutation: rename requester_realm to requester_email:

TestInstallRequestHoldsNoRequesterIdentity:
Install Request has a "requester_email" field; the portal must never
hold a tenant realm user's identity; the daemon notifies the requester

5. ★ SCOPE, HONESTLY: what is NOT reachable

Section titled “5. ★ SCOPE, HONESTLY: what is NOT reachable”

Nothing below is a claim about an end-to-end paid install working. It does not. These are the specific reasons.

modules/portal is still the ruled skeleton: shc portal register / deregister refuse X503145PTLNYI past the flag and Agent.Run refuses rather than connecting (modules/portal/agent.go:167). So the up-channel this state machine listens on has no client. The endpoints are exercised by the driver against the real credential path, not by a running daemon.

5.2 InstallDispatch.RequestID is carried, not honoured

Section titled “5.2 InstallDispatch.RequestID is carried, not honoured”

The correlation the sibling doc §1.5 recorded as missing now exists on the wire: InstallDispatch carries the cluster-minted RequestID, omitempty so a store-initiated dispatch puts the same bytes on the wire it always did.

★ Carrying it is not honouring it. Nothing in the daemon consumes the field. The property “the daemon honours an inbound install dispatch only if it matches a pending request it minted” remains RULED AND OWED, and it must not be described as operative. What changed is that it is now sayable: the id exists at both ends and is pinned by golden.

5.3 ★ The grant seam is unwired, and POST /record cannot carry a per-app grant as it stands

Section titled “5.3 ★ The grant seam is unwired, and POST /record cannot carry a per-app grant as it stands”

grant_backend is None. run_pending_grants therefore attempts nothing, reports {"wired": false, "attempted": 0}, and (pinned by the_grant_point_is_unwired_in_this_build_and_says_so) does not mark any row granted. A mutation that fabricated grants turned that scenario red.

Verified how brokerd/record.go keys idempotency, as instructed, and it does not key the way the field name suggests. Two different keys are at work:

  • The lookup key is the DOMAIN. findLicense queries metadata[domain]=<domain> with limit=1 (record.go:954). There is exactly one Keygen record per estate domain.
  • The invoice is metadata on that one record (metaKeyInvoice, record.go:65), and sameRecord compares the whole metadata document including it, which is what makes a replay of an unchanged invoice write nothing (record.go:571, record.go:1051). The file’s own header states it exactly: “Idempotency is structural, not bookkeeping: a record is keyed on metadata.domain.”

The consequence for the marketplace is concrete and blocking: posting a per-app grant to /record with a new invoice id would UPDATE the estate’s single tier record in place, swapping its policy, expiry, cores and invoice reference. It would not add an entitlement; it would overwrite the customer’s SHC licence with a marketplace line item. /record is a tier-recording endpoint, not a grant ledger, and sync_paid_invoices.py confirms the shape from the money side: it maps an invoice item code to an SHC tier (TIER_BY_ITEM) and posts {domain, tier, cores, expiry}. There is no app dimension anywhere in the pipeline.

So wiring grant_backend needs, in order:

  1. A per-app entitlement namespace (the entitlement doc §8 item 1, Q9): app codes must not collide with tier codes in one flat feature-key space.
  2. A place for a per-app grant to land that is not the estate’s single tier record: either a second Keygen record shape keyed on (domain, app), or per-app codes added to the existing record’s policy. Whichever is chosen is a broker change, and it is the decision that unblocks everything else.
  3. A marketplace item → app-entitlement mapping in ERPNext, the sibling of TIER_BY_ITEM.

When those land, grant_backend takes the row and returns the invoice id; the seam is already idempotent on the row (grant_ref present means skip, no call) and already releases the reservation on failure.

  • No portal UI. The endpoints exist; nothing renders them.
  • No per-app lapse behaviour: that is Q2, unruled.
  • No FX. A request in a currency the policy is not denominated in never auto-approves; it does not convert.
  • Minor units are hundredths, so zero- and three-decimal currencies (JPY, KWD) are not correctly representable. Amounts with more than two decimal places are refused rather than silently rounded.
  • Q7 (monthly vs annualised limits) is answered here provisionally: monthly-recurring plus a separate one-time allowance, annual normalised to monthly. That is an implementation choice made to have something testable, not a ruling.

Ruling 1 (request-and-approve): implemented as the substrate, with auto-approval as a policy short-circuit inside it, never a bypass. §6 of the entitlement doc: all three traps handled and tested, the limit portal-side. Sibling §2.1 routing: two populations, two notification channels, neither enumerating the other. “Money never touches crypto”: nothing in install_request.py signs anything or knows a Keygen credential; it records a decision and names an invoice. Placement: the whole surface is the Frappe portal bench app and the shc-proprietary wire package; ce carries none of it.