The install request / approval state machine: BUILT 2026-08-16
Status: BUILT, behind the portal flag, with a named list of what is not
reachable yet. This is the implementation record for
2026-08-16-portal-cluster-identity.md
Ruling 1 and
2026-08-16-marketplace-entitlement-and-licensing.md
§6. Both documents are design records; this one says what now exists in the
tree, what it is guarded by, and (§5) precisely which claims about an
end-to-end paid install would be false today.
THE RULING. Every paid install is a REQUEST. Policy decides whether a
human sees it. A spending limit is an AUTO-APPROVAL RULE, not a second
path. One state machine, one audit record, one grant point.
There is no second code path. An auto-approved install and a
human-approved install are the same Install Request row, differing only in
decision (auto_approved_by_policy vs approved_by_human) and
decided_by (the literal policy vs the portal account). The scenario
every_paid_install_is_one_row_whether_policy_or_human_decided asserts both
rows carry the same populated audit fields; the mutation that made an
auto-approval record no decision turned exactly that scenario red.
1.1 The four policies, and one ruling made inside them
Install Spending Policy, one row per cluster, set by the approver:
per-request cap: a single request above it goes to a human.
rolling budget: monthly, evaluated against committed recurring
spend; plus a separate one-time allowance, because a purchase that
happens once must not consume a budget that renews.
app allowlist: apps that never need approval.
requester scope: which tenant realms may be auto-approved.
★ The allowlist waives the HUMAN, never the BUDGET. This was a real
fork. An allowlist that also waived the budget would reintroduce the
recurring-spend trap through the back door: twelve allowlisted $20/mo apps
under a $100/mo budget would each be individually “pre-approved” and land at
$240/mo. Pinned by
policy3_app_allowlist_waives_the_human_but_never_the_budget.
★ Requester scope gates auto-approval; it never auto-denies. An
out-of-scope requester goes to the approver rather than being refused,
because auto-denying hides the request from the only person who can say yes.
Pinned by policy4_requester_scope_gates_auto_approval_and_never_auto_denies.
Defaults are deny. No policy row, a disabled policy, an empty realm list
or a currency the policy is not denominated in all mean “a human sees it”.
Nothing auto-approves by omission.
The budget predicate sums monthly_equivalent_minor over every row already
holding a reservation or a grant and adds this request’s monthly equivalent.
A one-time purchase contributes zero to that sum and draws on the one-time
allowance instead. An annual price is normalised to a monthly figure,
rounded up; rounding is toward the budget’s protection.
The running total is recomputed from the ledger on every decision and is
never a decremented counter: a counter drifts on cancellation, refund,
downgrade and failed provisioning, and a drifted budget counter fails OPEN.
Mutation: evaluate the budget against amount_minor (the transaction
reading) instead of the committed recurring total:
Every decision (policy or human) runs inside _lock_cluster, a
SELECT ... FOR UPDATE on the Cluster row, taken before the ledger is
read and held until the reservation is committed. The Cluster row is the
lock because it always exists; a policy row may not, and you cannot lock a
row that is absent.
The slow, failable part sits deliberately outside the lock:
run_pending_grants acts against a reservation the ledger already holds and
releases it (state failed) when the grant does not land.
This was tested with concurrency, not by inspection. The stub
(tests/unit/portal/testdata/frappe_stub.py) implements for_update as a
real row lock released on commit or rollback, and the scenario runs two
threads that rendezvous at the ledger read.
Mutation: drop for_update=True from _lock_cluster:
AssertionError: concurrent outcomes ['approved', 'approved'], want one of each
Two $60/mo requests, a $100/mo budget, both approved. With the lock, one
approves and one goes to the approver, and the ledger reads 6000 of 10000
minor units committed.
Precedent honoured: modules/cluster/enroll_capacity_race_test.go exists
because the node capacity gate had exactly this shape.
expire_stale_requests (hourly, scheduler_events) moves unanswered
pending rows to expired, a terminal state distinct from denied,
because an approver who said nothing and an approver who said no are
different facts and the requester is owed the difference.
The portal cannot notify the requester: it holds no email for a tenant realm
user and must never hold one. So every terminal outcome, including
expiry, is handed to the daemon through poll_install_outcomes, and keeps
being handed back until ack_install_outcomes confirms delivery. A lost
poll costs a duplicate notification rather than a silent expiry.
Expiry never touches a reservation: releasing an approved-but-ungranted
request behind the approver’s back would silently undo a decision they made
(expiry_never_reaps_a_reservation).
Mutations: (i) expire to denied, (ii) withhold expired rows from the
poll:
AssertionError: the expiry was not handed to the daemon; a silent expiry is the failure mode
Open, deliberately: the TTL default is 72 hours with a 720-hour ceiling,
and an expired request does not auto-resubmit. That is
marketplace-entitlement-and-licensingQ8, and the numbers are a
placeholder a ruling should replace, not an answer.
3. The limit is the approver’s, and the cluster cannot reach it
★ A control the restricted party can flip is not a control. Here the
restricted party is the cluster’s users, so the mirror of the L2
compliance seal applies: the limit lives on the portal, on the billing
relationship, and the daemon never receives it, never stores it and cannot
move it.
Three guards, all mutation-proved:
set_spending_policy / get_spending_policy are session-only. The
guest surface guard fails if either becomes allow_guest, because
allow_guest here means “reachable with a cluster token”.
Both are owner-checked against Cluster.account_owner, unconditionally,
System Manager included, the same posture api.fleet already has, where
the filter is the permission.
modules/portal/config_keys.go is scanned for the vocabulary of a
cluster-side limit (spending, budget, per_request_cap, allowance,
…). A daemon config key for any of it is a red test.
Mutation: allow_guest=True on set_spending_policy:
endpoint set_spending_policy must be session-only; a limit the cluster can reach is not a limit
A human may approve past their own limit, and the row says who. The
limit is an auto-approval rule, not a ceiling on the approver; that is the
escape hatch the request path exists for. What they cannot do is make it
invisible.
The approver’s words land in approver_note, a separate column from
decision_reason. decision_reason keeps the predicate that sent the
request to a human, because “over_per_request_cap, and the owner approved
anyway” is a different fact from “the owner approved”, and overwriting it
would lose why the request needed a human at all. Mutating the note back
over the predicate turns
a_human_may_approve_past_the_limit_and_the_row_says_who red.
4. Routing: two populations, neither enumerating the other
The Install Request DocType has no email, name or user column, and
submit_install_request refuses a requester.ref containing @. Both are
guarded, and the schema guard is structural on purpose: no runtime test can
prove a column that does not exist will never be filled.
Mutation: rename requester_realm to requester_email:
TestInstallRequestHoldsNoRequesterIdentity:
Install Request has a "requester_email" field; the portal must never
hold a tenant realm user's identity; the daemon notifies the requester
modules/portal is still the ruled skeleton: shc portal register /
deregister refuse X503145PTLNYI past the flag and Agent.Run refuses
rather than connecting (modules/portal/agent.go:167). So the up-channel
this state machine listens on has no client. The endpoints are exercised by
the driver against the real credential path, not by a running daemon.
5.2 InstallDispatch.RequestID is carried, not honoured
The correlation the sibling doc §1.5 recorded as missing now exists on the
wire: InstallDispatch carries the cluster-minted RequestID, omitempty
so a store-initiated dispatch puts the same bytes on the wire it always did.
★ Carrying it is not honouring it. Nothing in the daemon consumes the
field. The property “the daemon honours an inbound install dispatch only if
it matches a pending request it minted” remains RULED AND OWED, and it
must not be described as operative. What changed is that it is now
sayable: the id exists at both ends and is pinned by golden.
5.3 ★ The grant seam is unwired, and POST /record cannot carry a per-app grant as it stands
grant_backend is None. run_pending_grants therefore attempts nothing,
reports {"wired": false, "attempted": 0}, and (pinned by
the_grant_point_is_unwired_in_this_build_and_says_so) does not mark
any row granted. A mutation that fabricated grants turned that scenario red.
Verified how brokerd/record.go keys idempotency, as instructed, and it
does not key the way the field name suggests. Two different keys are at
work:
The lookup key is the DOMAIN. findLicense queries
metadata[domain]=<domain> with limit=1 (record.go:954). There is
exactly one Keygen record per estate domain.
The invoice is metadata on that one record (metaKeyInvoice,
record.go:65), and sameRecord compares the whole metadata document
including it, which is what makes a replay of an unchanged invoice write
nothing (record.go:571, record.go:1051). The file’s own header states
it exactly: “Idempotency is structural, not bookkeeping: a record is keyed
on metadata.domain.”
The consequence for the marketplace is concrete and blocking: posting a
per-app grant to /record with a new invoice id would UPDATE the estate’s
single tier record in place, swapping its policy, expiry, cores and
invoice reference. It would not add an entitlement; it would overwrite the
customer’s SHC licence with a marketplace line item. /record is a
tier-recording endpoint, not a grant ledger, and sync_paid_invoices.py
confirms the shape from the money side: it maps an invoice item code to an
SHC tier (TIER_BY_ITEM) and posts {domain, tier, cores, expiry}.
There is no app dimension anywhere in the pipeline.
So wiring grant_backend needs, in order:
A per-app entitlement namespace (the entitlement doc §8 item 1, Q9):
app codes must not collide with tier codes in one flat feature-key space.
A place for a per-app grant to land that is not the estate’s single
tier record: either a second Keygen record shape keyed on
(domain, app), or per-app codes added to the existing record’s policy.
Whichever is chosen is a broker change, and it is the decision that
unblocks everything else.
A marketplace item → app-entitlement mapping in ERPNext, the sibling
of TIER_BY_ITEM.
When those land, grant_backend takes the row and returns the invoice id;
the seam is already idempotent on the row (grant_ref present means skip,
no call) and already releases the reservation on failure.
No portal UI. The endpoints exist; nothing renders them.
No per-app lapse behaviour: that is Q2, unruled.
No FX. A request in a currency the policy is not denominated in never
auto-approves; it does not convert.
Minor units are hundredths, so zero- and three-decimal currencies (JPY,
KWD) are not correctly representable. Amounts with more than two decimal
places are refused rather than silently rounded.
Q7 (monthly vs annualised limits) is answered here provisionally:
monthly-recurring plus a separate one-time allowance, annual normalised to
monthly. That is an implementation choice made to have something testable,
not a ruling.
Ruling 1 (request-and-approve): implemented as the substrate, with
auto-approval as a policy short-circuit inside it, never a bypass. §6 of the
entitlement doc: all three traps handled and tested, the limit portal-side.
Sibling §2.1 routing: two populations, two notification channels, neither
enumerating the other. “Money never touches crypto”: nothing in
install_request.py signs anything or knows a Keygen credential; it records
a decision and names an invoice. Placement: the whole surface is the Frappe
portal bench app and the shc-proprietary wire package; ce carries none of it.