Retire the Frappe/Python-era API wire dialect
Status: open design. Step 1 shipped. The daemon and CLI flag-day, steps 2 and 3, are not done.
| Proposed | 2026-07-14 (sweep-legacy decision) |
| Step 1: provider dual-dialect shim | SHIPPED 2026-07-14, shc 880b07a3b |
| Step 2: daemon + CLI flag-day | owed |
| Step 3: drop the shim | owed, optional |
| Re-measured against | ce d054d79, shc 184dc7c99 (2026-08-07) |
This document was written on 2026-07-14 and its step 1 landed the same day, but
the document itself was never landed; it sat on an unmerged branch while eight
comments in shipped code cited it by path (pkg/apiclient/client.go and
pkg/apiclient/envelope_internal_test.go in both repos). This revision
lands it and re-measures every count against current main, because the CE/EE
split moved all of the code it describes and changed the sequencing conclusion.
What this is
Section titled “What this is”The daemon still speaks the two success envelopes the Python port inherited
from Frappe: {"message": …} on the /api/method/* surface (WriteMessage)
and {"data": …} on the /api/resource/* single-doc surface (WriteDoc),
both in ce core/api/respond.go. Errors ride a third Frappe shape:
{exc_type, code, message, http_status, _server_messages, exc}
(coded.Envelope, ce core/coded/transport.go, written by
ErrorHandler/WriteError). Every consumer is in-house: the shc CLI (plus
the x9s TUI) and the terraform provider. Under the no-backcompat doctrine this
is a clean kill-candidate, nothing external parses these bodies, but it
touches the daemon response layer AND every client decoder at once, and one of
those decoders is version-decoupled from the daemon. So it needs a migration
plan, not a blind kill.
The target already exists in-tree: the newer REST-style routes (/api/dns,
/api/user, /api/health, …) skip the envelope entirely (WriteJSON;
envNone in pkg/apiclient/client.go) and are the de-facto dialect this
proposal promotes to universal.
Where the code lives now (post-CE/EE split)
Section titled “Where the code lives now (post-CE/EE split)”The 2026-07-14 draft used the pre-split layout (internal/core/api/…,
one module, one repo). That layout is gone. Current homes:
| Thing | Repo | Path |
|---|---|---|
WriteMessage / WriteDoc / WriteJSON / envelope structs | ce | core/api/respond.go |
WriteError / ErrorHandler, coded.Envelope | ce | core/api/error_handler.go, core/coded/transport.go |
| Generated API client + the shim (CE route surface) | ce | pkg/apiclient/client.go |
| Generated API client + the shim (full/EE route surface) | shc | pkg/apiclient/client.go |
| The client the terraform provider actually imports | shc | gitlab.com/selfhosted-cloud/shc/pkg/apiclient |
| CLI / x9s decoders | both | per-module daemon_client.go, types.go |
internal/ no longer exists in either repo (0 files under it in shc), so every
internal/core/... and internal/modules/... path in the original draft should
be read with that prefix removed and the repo column above applied.
The surface (re-measured, non-test files, at the SHAs above)
Section titled “The surface (re-measured, non-test files, at the SHAs above)”Server writers: the wrapped surface now spans BOTH repos, because shc’s EE
modules call ce’s api package:
| writer | ce | shc | total |
|---|---|---|---|
WriteMessage → {"message": v} | 42 | 12 | 54 |
WriteDoc → {"data": v} | 2 | 1 | 3 |
WriteJSON → raw v (the target shape, already live) | 13 | 1 | 14 |
So 57 wrapped call sites, up from the 42 the draft measured; the surface grew, and a third of the growth is on the shc side, which did not exist as a separate repo when the draft was written.
Client decoders. Files carrying a json:"message" or json:"data" decode
tag, non-test: 90 in ce, 25 in shc. The CLI/x9s side still has no choke
point: the unwrap is hand-rolled per module in daemon_client.go files and
generic Message[T] / Data[T] / ListResponse[T] wrapper structs. The one
thing that IS centralized on the CLI side is the error envelope:
daemonHTTPErr → coded.FromEnvelope reconstructs the typed error in one
place.
Provider decode. The entire success-decode is the env switch in
invokeJSON selecting one of envNone | envMessage | envData; the entire
error-decode is DecodeError. Generated call counts:
| envMessage | envData | envNone | total | |
|---|---|---|---|---|
ce pkg/apiclient | 144 | 14 | 76 | 234 |
shc pkg/apiclient (what the provider imports) | 182 | 15 | 78 | 275 |
Two functions still cover every provider call, which is what made the shim cheap, and they are now duplicated across the two generated clients, so step 1 had to land in both. It did.
Two warts the map exposes, both worth fixing in the same sweep:
datais overloaded.docEnvelope.Datais a pure single-key wrapper, butListResponse.Datais a field of a list payload that also carriesnext_cursor/has_more/total_countsiblings. De-wrapping is therefore not “unwrap to the inner value” uniformly: a single doc collapses to the bare object, but a list must keep its pagination-bearing struct at top level. The target must be stated as “the response object, un-nested,” not “the value under.data.”- Double-wrapping. Several
approuter sites callWriteMessage(w, map[string]any{"data": data})→ the wire is{"message":{"data":…}}. This is the Frappe muddle at its worst and disappears the moment the message envelope is removed.
Target dialect (the rule)
Section titled “Target dialect (the rule)”Promote the shape that the envNone calls and 14 server routes already speak:
Success = the response object as the top-level JSON body. No
messagewrapper, nodatawrapper. Lists carry their items plus pagination as sibling fields of that object; single docs are the bare object. Failure = a single consistent error object. Recommended:{"error": {"code", "message", "details"}}, replacing the six-key Frappe error envelope (exc_type,exc,_server_messages,http_statusall drop;http_statusis already redundant with the HTTP status line, and the daemon-side warning path keeps its204 + X-SHC-Warning-Codeheader unchanged).
Server-side this means: delete WriteMessage, WriteDoc, messageEnvelope,
docEnvelope; every current caller becomes a WriteJSON. WriteJSON (already
funnelling encode-failures through the coded handler as X500938RSPENC)
becomes the sole success writer.
The error-envelope change is separable from the success-envelope change (its decode is two choke points, not 115 files) and MAY be sequenced as its own phase, or dropped entirely from this pass; see non-goals.
The coupling that decides sequencing (the crux)
Section titled “The coupling that decides sequencing (the crux)”The client families have different version relationships to the daemon, and this, not code size, dictates the migration. The split added a third one.
-
The CLI ships inside the same
.debas the daemon.shcandshcdare one artifact, built from one commit, installed together. An operator NEVER runsshcvX againstshcdvY where X≠Y; the deb makes it impossible. For the CLI boundary, a synchronized flag-day is safe by construction. -
The terraform provider is a separately versioned artifact. An operator pins a provider version in
required_providersand points that pinned plugin at an estate whose daemon runs whatever deb happens to be installed, upgraded on a different cadence. So provider vN genuinely talks to daemon vM, N≠M. This is the ONLY real cross-version wire surface in the system, and it is exactly where a naive flag-day breaks in both directions: estate upgraded first → old provider’senvMessageunwrap finds no.messageand every resource read errors; provider bumped first → new decoder chokes on the old daemon’s wrapper. -
NEW since the draft: ce and shc are two repos on one wire. The writers live in ce; 13 of the 57 wrapped call sites live in shc. A daemon flag-day is therefore a lockstep two-repo change, not a single commit. It lands ce-first, then shc with the ce pin bumped (the landing order this repo already mandates) and the shc pin bump is the only synchronization point where both halves are observable together. An shc that pins a flipped ce while still calling
api.WriteMessagesimply fails to compile (the symbol is gone), which is the desired guardrail: the break is a compile error, not a wire mismatch.
Verdict: flag-day the CLI (safe, same artifact), land the daemon flip ce-then-shc in one pin bump, and keep the provider boundary dialect-tolerant. That tolerance is now in place.
Migration
Section titled “Migration”Step 1: provider shim. DONE (shc 880b07a3b, 2026-07-14; mirrored in ce).
invokeJSON’s envMessage/envData arms route through UnwrapEnvelope(raw, key), which peeks the top-level object: if the wrapper key is present it
unwraps, otherwise it returns the body as-is. DecodeError gained the same
either-shape tolerance. The wrapper is preferred whenever present, so decode
is byte-identical against every daemon shipping today; the fallback cannot fire
until a flag-dayed daemon exists. Tolerance is scoped to the
envMessage/envData/DecodeError call sites only, never envNone, so a
genuine top-level message/data field on an already-raw response is never
routed through the peek. Covered by pkg/apiclient/envelope_internal_test.go
(both shapes, both keys) and end-to-end in client_test.go.
Step 2: daemon + CLI flag-day. OWED. Replace WriteMessage/WriteDoc at
all 57 sites with WriteJSON; delete the two envelope structs; update the CLI
and x9s decoders to read the un-nested body (delete the Message/Data
wrapper level from each daemon_client.go / types.go). Land ce first, then
shc with the pin bumped. Because shc ships with shcd, the CLI half lands in
one deb with no partial state. This is mechanical breadth, not depth.
Step 3: drop the shim. OWED, optional. Once the tolerant provider is the enforced minimum pin across all estates, delete the peek fallback so the decoder is single-dialect again. Not correctness; the shim is harmless if left.
This ordering respects the crux: at no point does a pinned provider meet a daemon whose dialect it cannot parse.
Not superseded by the x9s wire work
Section titled “Not superseded by the x9s wire work”The x9s effort (“catch shc-x9s up to the current wire: column roles and
relations”, e77c6ab08) modernizes a different wire: the k9s provider
protobuf that cmd/shc-x9s speaks to the k9s TUI host (ResourceType,
ListReq, Column.Role, Relations). That contract has no relationship to
the daemon’s HTTP success/error envelopes. x9s is a consumer of the HTTP
dialect described here (it decodes the same {"message": …} bodies the CLI
does) so the flag-day in step 2 includes its decoders. Nothing about the x9s
catch-up decided, replaced, or obsoleted this proposal.
Blast radius + test strategy
Section titled “Blast radius + test strategy”What breaks if done wrong: every CLI command (each daemon_client.go
decodes a specific envelope; a missed one silently returns a zero-value struct,
not an error, the nastiest failure mode, so decode changes must be
compile-enforced by deleting the wrapper type, not by making it optional) and
every provider resource/data-source. The double-wrap sites and the
pagination-bearing list structs are the two places a careless “unwrap .data”
loses data; the target rule (“un-nest the object,” not “read .data”) is
written to forbid exactly that.
- Unit, server:
respond_test.gocurrently PINS the envelopes (it assertsWriteMessagewraps in{"message": …}). Step 2 inverts it: assertWriteJSONis the only writer, plus a grep-guard for zero non-test hits onWriteMessage/WriteDocacross both repos; an shc-only scan would miss 42 of the 54 sites, and a ce-only scan would miss 12. - Unit, provider: table-test
UnwrapEnvelopeandDecodeErroragainst both a wrapped and a raw body for each envelope, the dual-dialect proof. Already in place from step 1; keep it through step 2 and delete it with the shim at step 3. - Unit, CLI: each touched
daemon_client.gogets a fixture of the new raw body; the compiler catches the rest because the wrapper field is deleted. - e2e/bats: the app suites exercise real
shccommands against a live daemon; they pass unchanged only if the flag-day deb is internally consistent, so they ARE the CLI cross-check. - Provider acceptance: run the suite TWICE during the transition: the tolerant provider against an old (wrapped) daemon AND against a new (raw) one. That is the executable form of the crux verdict.
Effort
Section titled “Effort”- Provider shim (step 1): S, done.
- CLI + server flag-day (step 2): M, mechanical but broad (57 server sites across two repos + ~115 decoder files); depth is trivial, breadth is the cost. Best done module-by-module with the compiler as the guardrail.
- Shim removal (step 3): S, optional.
- Error-envelope change: S if folded in (two choke points), independently sequenceable.
Non-goals
Section titled “Non-goals”- No route-path renames.
/api/method/*and/api/resource/*keep their paths even thoughmethod/resourceare Frappe-isms; renaming URLs is a separate, larger break (every generated path string, every provider endpoint, every bats invocation) with no wire-dialect payoff. Paths stay; only bodies change. - No auth change. Bearer/UDS/
ExtractAuth, the?token=query fallback, and the204 + X-SHC-Warning-Codewarning path are untouched. - No streaming change.
invokeStream(log tails, PEM/file downloads) already returns the raw body and never wrapped, out of scope. - The error-envelope swap is optional. It can ride this sweep or be deferred; the success-envelope removal stands on its own and carries the blast radius. Do not let the smaller, independent error change gate the larger one.
- No new response fields or pagination redesign. List responses keep their
existing
next_cursor/has_more/total_countshape; they merely stop being nested inside a wrapper.