Skip to content
SHC Docs

Communication patterns

Layer 2. Core. Core

The system is a CLI tool backed by a Go daemon (chi router) with a distributed task queue and node-to-node communication layer. The CLI communicates with the server over HTTP (Unix socket or TCP), and the server manages state in SQLite (raft-replicated by rqlite in cluster mode) with Docker as the container runtime.

There are seven distinct communication patterns used across the codebase.

Path convention. Every path on this page is ce-relative unless it is prefixed shc . The transports, the routers and the whole CLI command tree live in the ce module; there is no cmd/shc/commands/ directory in either repo; a module’s cobra surface is modules/<module>/commands/, and the CLI-side runtime wiring that those commands call into is cli/*_wiring.go. Paths were verified against ce main and shc main; verify a symbol before citing it from this page.


Pattern 1: CLI → Resource API → Service (synchronous read)

Section titled “Pattern 1: CLI → Resource API → Service (synchronous read)”

Standard REST-style CRUD on domain objects. The CLI blocks until the response comes back.

When to use: Read/list operations on first-class resources (stacks, nodes, apps, tenants, backups).

CLI (daemonGet) → GET /api/resource/Stack → listStacksHandler → Service.ListStacks() → DB query → response

shc ls additionally falls back to an on-disk lister when the daemon answers with an empty list and the resolved endpoint is this machine; CLI-side installs write state.json under the state home without touching the daemon’s Stack table, so an empty daemon response is indistinguishable from “nothing here”. Against a remote endpoint the daemon’s answer is surfaced verbatim, because local state would describe a different cluster.

CLI: modules/app/commands/stack_lifecycle.go (NewLsCommand):

var resp listEnvelope[StackResp]
err := daemonGet(ctx, "/api/resource/Stack", q, &resp)

Router: modules/app/router.go. Routes carry their full path and a handler closure over the already-resolved *Service; auth and the request-scoped Ctx are middleware applied at the mount point, not per-route:

r.Get("/api/resource/Stack", listStacksHandler(svc, stackRepoFn))

Service: modules/app/service.go:

func (s *Service) ListStacks(ctx context.Context, tenant, environment string, status DeploymentStatus) []StackModel

The handler reads its filters off the query string, collapses the tenant to the caller’s own unless they are a platform admin or a UDS-local user, calls the service method, and encodes the result. The CLI decodes the response body into the matching Go struct and prints it.

LayerFile
CLI commandmodules/<module>/commands/*.go
HTTP clientcore/daemonclient/client.go (Client.Get, Client.Post, …)
Routermodules/<module>/router.go
Servicemodules/<module>/service.go

Pattern 2: CLI → Method API → Service (synchronous action)

Section titled “Pattern 2: CLI → Method API → Service (synchronous action)”

RPC-style endpoints for operations that don’t fit REST semantics. Still synchronous from the CLI’s perspective.

When to use: Actions like config get/set, validation, or any operation that completes quickly and needs a result back.

CLI (ConfigPort.Get) → POST /api/method/shc.config.get → getHandler → Service.Get() → response

CLI: modules/config/commands/get.go. The cobra handler calls a typed ConfigPort, not an HTTP client:

result, err := port.Get(ctx, GetRequest{Key: key, Scope: scope})

commandAdapter (modules/config/register.go) implements that port. In the shc process there is no daemon-side *Service to resolve, so it falls through to remoteGet, the POST to /api/method/shc.config.get over the daemon UDS (modules/config/daemon_client.go). This port-with-fallback shape is the norm for module command surfaces; the raw client call is one layer below the command.

Router: modules/config/router.go:

r.Post("/api/method/shc.config.get", getHandler(svc))
AspectResource (/api/resource/...)Method (/api/method/...)
StyleREST (GET/POST/PUT/DELETE on nouns)RPC (POST to a function name)
ExamplesGET /api/resource/StackPOST /api/method/shc.config.get
Use forCRUD reads, entity lookupsActions, procedures, operations

There is no central method-router file: each module builds its own chi router (e.g. modules/config/router.go, modules/app/router.go) and registers it via registry.Router (core/registry/registry.go); the daemon flattens every registered router onto its listeners with MountPublicRoutes (subsystems/impl/unix_socket.go), so /api/method/... and /api/resource/... routes land on one mux.

The same flattening wraps every POST, PUT, PATCH and DELETE handler in api.DetachMutation (core/api/mutation.go): the handler runs on context.WithoutCancel of the request, keeping its values (user, tenant, actor), under a server deadline, api.DefaultMutationBudget (30 min) unless the route registers a longer one. A CLI that times out or is interrupted no longer cancels the change part way, and the daemon logs one mutation ended line per request with the route, status, duration, actor and whether the client had gone. The internal mux, the container broker and the Incus member proxy do the same.

A route that belongs to its caller says so next to its registration with api.RegisterMutationPolicy: Cancellable serves it on the request’s context as before (shc.stack.exec, backup verify, broker wait and attach), Conditional keeps it detached and takes the client’s cancellation back with api.ClientContext in the modes that need it (an internal RPC follow, attached exec, the doctor run’s wait), and Budget sets its deadline. TestMutationMountedGuard7298 (tests/_ce/subsystems/impl) fails on a mutating route without the wrapper and on any change to those lists.

Work that runs for minutes and wants progress is still an operation (Pattern 3); detaching only means the request’s own work finishes.


Pattern 3: CLI → Method API → Background Task + SSE (async operation)

Section titled “Pattern 3: CLI → Method API → Background Task + SSE (async operation)”

For long-running operations (install, update, uninstall, backup). The API returns immediately with an operation_id, the real work runs in a background goroutine, and the CLI watches progress via Server-Sent Events.

When to use: Any operation that takes more than a few seconds and the CLI needs to show progress.

sequenceDiagram
participant CLI
participant Server
CLI->>Server: POST /shc.stack.install
Note over Server: create operation in registry<br/>spawn goroutine for work
Server-->>CLI: AsyncOperationResponse (returns immediately)
CLI->>Server: GET /shc.operation.stream/{id} (blocking follow)
Note over Server: SSE endpoint subscribes to sseBroker
Note over Server: goroutine runs the install
Server-->>CLI: event: step (EmitStep progress)
Server-->>CLI: event: step (EmitStep progress)
Server-->>CLI: event: complete (EmitComplete result)
Note over CLI: streamOperation returns the terminal frame

Phase 1: CLI POSTs, then follows the operation over SSE

Section titled “Phase 1: CLI POSTs, then follows the operation over SSE”

CLI: modules/app/commands/install.go:

if err := daemonPostJSON(ctx, "/api/method/shc.stack.install", BuildInstallBody(args), &resp); err != nil {
return err
}

The POST returns an operation id; followAsyncOperation (modules/app/commands/operation_sse.go, rendering in operation_live.go) then streams /api/method/shc.operation.stream/{operation_id} until a terminal frame. (Only when the daemon is unreachable does the command fall back to an in-process compose deploy; a real daemon-side error surfaces as-is, so nothing gets deployed twice.)

onFrame := func(f SSEFrame) { r.OnFrame(stackForFrame(f), f) }
terminal, err := streamOperation(ctx, operationID, onFrame)

streamOperation GETs /api/method/shc.operation.stream/{operation_id} and hands every frame to onFrame until a terminal (complete / error) frame arrives, which it returns. It is a blocking read on the calling goroutine, not a background listener: the CLI has already POSTed by the time it starts, and a 50 ms backoff before the first connect keeps the GET from racing the daemon’s registration of the operation id.

Phase 2: API creates the operation, schedules background work

Section titled “Phase 2: API creates the operation, schedules background work”

Router: modules/app/router.go, installHandler. The handler decodes and validates the body, gates tenant access, registers the operation, spawns the deploy, and answers immediately:

store := operations.EnsureRuntime() // *core/operations.Store
opID := store.Create(OpKindInstall, installed.TenantName, installed.Name, body.OperationID)
go func() {
defer recoverDeployGoroutine(store, opID, svc, tenant, name, env, failEvent)
// … RunInstallWithObserver → ExecuteCompose, pushing store.EmitStep …
store.EmitComplete(opID, map[string]any{"success": true, /* … */})
}()
writeJSON(w, http.StatusOK, AsyncOperationResponse{OperationID: opID, Message: /* … */})

recoverDeployGoroutine is the panic guard: it mints coded.X500901OPFAIL with the captured stack, stamps the stack row FAILED, writes the failure audit row, and calls store.EmitError(opID, codedErr) so the CLI’s follow terminates on a real coded error instead of hanging on a channel nobody will close.

core/operations/store.go owns a per-operation buffered channel (64 frames by default) plus the operation’s state, and retains a terminated operation for 5 minutes so a late subscriber can still replay the terminal frame. The deploy observer pushes through three methods:

func (s *Store) EmitStep(operationID, step string, percent int, message string, snapshot any)
func (s *Store) EmitComplete(operationID string, result any)
func (s *Store) EmitError(operationID string, err error)

This is not the SSEBroker subsystem (subsystems/impl/sse_broker.go); that one is the events module’s browser-facing fan-out and has nothing to do with operation progress.

SSE endpoint: modules/backup/router.go:190 (operationStreamHandler; the shc.operation.stream route rides the backup router for historical reasons):

r.Get("/api/method/shc.operation.stream/{operation_id}", operationStreamHandler(resolveOp))

The handler resolves the operation against two backing stores, in order, because the product has two:

  1. operations.EnsureRuntime(): the core/operations.Store the app module’s install / update / uninstall write to.
  2. modules/operation.Registry, resolved off the daemon composition root, where the stack module’s start / stop / restart kicks register instead.

Either way it replays the terminal frame straight from state when the operation already finished (so a late-connecting client cannot hang on a channel that is already closed), otherwise subscribes and streams, with a 15-second heartbeat so a proxy does not idle the connection out between phases. The wire format is id: <sequence> / event: <type> / data: <json> / blank line.

ComponentFile
SSE follow (CLI side)modules/app/commands/operation_sse.go + operation_live.go
Operation store (app deploys)core/operations/store.go (EnsureRuntime)
Operation registry (stack start/stop/restart)modules/operation/registry.go
SSE endpointmodules/backup/router.go (shc.operation.stream)

Pattern 4: CLI → local daemon WebSocket → peer proxy (data plane)

Section titled “Pattern 4: CLI → local daemon WebSocket → peer proxy (data plane)”

For interactive streaming operations. The CLI opens a WebSocket to the local daemon; the daemon decides local-vs-remote and, when the target container lives on another node, proxies the socket to that node’s mTLS internal listener. The client never dials a remote node itself.

When to use: Shell access, port forwarding: anything bidirectional or long-lived against a specific container.

Earlier revisions of this page had the topology backwards, describing the CLI resolving node_address off /api/resource/Stack and then dialing ws://node:9443/internal/ws/exec with an X-SHC-Internal-Token of its own. Client-side WebSocket dials target the local daemon (ce modules/shell/local_session.go, ce cli/forward_wiring.go, and the x9s provider’s container-exec tunnel (shc pkg/x9sprov/stream.go)) over the UDS where one is reachable. The retired bubbletea HUD stream is gone. The internal listener is port 4003 (modules/cluster/types.go:457 InternalAPIPort), not 9443, and it is reached by the daemon, not the client.

sequenceDiagram
participant CLI
participant Local as Local daemon
participant Peer as Peer daemon (:4003, mTLS)
CLI->>Local: ws://shc-uds/api/method/shc.container.exec?…
Note over Local: bearer/UDS auth upstream, then<br/>CanAccessTenant + container ownership gate
Note over Local: NodeRouter: is the container local?
Local->>Peer: wss://host:4003/internal/ws/exec (proxy hop, only when remote)
Note over Peer: VerifyInternalAuth → docker exec
Peer-->>Local: stdout
Local-->>CLI: stdout
CLI->>Local: stdin

CLI: modules/shell/commands/shell.go builds the request and hands it to a ShellFunc runtime hook; cli/shell_wiring.go installs the CLI’s implementation, which calls shell.DialExec (modules/shell/local_session.go):

socketPath, baseURL, authHeader := daemonclient.ResolveEndpointWithAuth()
if socketPath != "" {
dialOpts.HTTPClient = daemonclient.UDSClient(socketPath, 0)
target = "ws://shc-uds/api/method/shc.container.exec?" + query.Encode()
} else {
// ws(s):// against the resolved base URL, with the bearer Authorization header
}
conn, _, err := websocket.Dial(dctx, target, dialOpts)

There is no metadata round-trip: the stack / service / tenant go in the query string and the daemon resolves the container.

modules/shell/router.go mounts three surfaces:

RouteRole
WS /api/method/shc.container.execpublic-facing proxy; the handler adds a CanAccessTenant + container-ownership gate on top of the mount-point auth
WS /internal/ws/execinter-node exec; proxies onward when node is not the local node
WS /internal/containers/{container_id}/execalways-local docker exec, no proxy hop

The local-vs-remote decision is the module’s NodeRouter (modules/shell/service.go); the remote hop is modules/shell/proxy.go, which dials wss://<host>:4003/internal/ws/exec through the daemon-wired MeshDialer (node client certificate). modules/logs/proxy.go and modules/portforward/proxy.go are the same shape.

Auth: core/api/auth.go, VerifyInternalAuth:

func VerifyInternalAuth(r *http.Request, opts InternalAuthOpts) error {
if r.TLS != nil && len(r.TLS.PeerCertificates) > 0 && len(r.TLS.VerifiedChains) > 0 {
return nil
}
if tokenMatches(r, opts.ExpectedToken) { // constant-time X-SHC-Internal-Token compare
return nil
}
return X401000UNAUTH()
}

Two credentials, and only two: a cluster-CA client certificate validated by the TLS handshake, or the shared X-SHC-Internal-Token. The caller’s source address is never consulted. The historical loopback-and-mesh-IP acceptance was deleted deliberately; the internal listener binds TLS-only in production, so treating an IP as authentication was a misbind away from an unauthenticated admin channel.

EndpointPurposeDialed by
/api/method/shc.container.execpublic entry for interactive execshc shell (local_session.go)
/internal/ws/execinter-node interactive execa peer daemon (modules/shell/proxy.go)
/internal/ws/forwardTCP port relaythe CLI over the UDS only (cli/forward_wiring.go), then peer-to-peer via modules/portforward/proxy.go
/internal/ws/logscontainer log streaminga peer daemon (modules/logs/proxy.go)

/internal/ws/logs has no client-side caller today. shc logs reaches the daemon over plain HTTP (GET /api/method/shc.logs.get, modules/logs/daemon_client.go), and its --follow path returns X500940LOGRUN("logs --follow is not supported by the CLI HTTP fallback") because the SetLogsRunner seam that would replace that fallback has zero callers in either repo. The endpoint itself is live and is what one daemon uses to stream another node’s container logs.


Cross-node function calls. An internal_rpc.Register registered function is invoked remotely via HTTP POST to /internal/rpc/{function_name} on the target node’s mTLS internal listener. No DB record is created. It’s a direct call with a result. Apart from the live streams and enrol, every node-to-node call is one of these: the functions, with their arguments and results, are listed in the generated internal RPC reference.

When to use: You need an answer from a specific node (or a set of nodes) and want it back immediately. Health checks, metrics collection, volume capture, cross-node stack operations.

Handlers share one signature (Handler func(ctx context.Context, args []any, kwargs map[string]any) (any, error)) and register by name (modules/metrics/router.go, daemon/services_logs_remote.go):

const NodeStatsRPC = "metrics.node_stats"
func init() {
internal_rpc.Register(NodeStatsRPC, nodeStatsHandler)
}
func nodeStatsHandler(ctx context.Context, args []any, kwargs map[string]any) (any, error) {
return NodeStatsResponse{}, nil
}

Register records the handler in the module-level registry: modules/internal_rpc/registry.go. Handlers that must not run on every node register with RegisterWithCapability so the receiver can refuse calls the local node cannot serve.

Every registered function is also described, in an rpc_reference.go beside its package’s Register calls:

func init() {
internal_rpc.Describe(internal_rpc.Function{
Name: NodeStatsRPC,
Summary: "This node's cached node and container resource stats.",
Result: internal_rpc.FieldsOf(NodeStatsResponse{}),
})
}

make docs/generate writes the reference from these entries, and two guards fail when a function is registered without one or an entry outlives its function.

The caller picks the target node and calls through internal_rpc.Call (or CallValue, CallNode), which posts over client-certificate mTLS to the internal listener:

out, err := internal_rpc.Call(ctx, peer, node.NodeAddr, "logs.snapshot", kwargs)

Nothing builds an /internal/rpc/ request by hand: Call carries the per-peer breaker, the busy backoff, the typed errors and the byte transfer, and a guard fails on a hand-posted call. Fan-out to several nodes is a loop at the call site. Scheduled work is not an RPC concern: crons register via registry.Crons and run under the cron-runner subsystem (subsystems/impl/cron_runner.go).

Receiver: modules/internal_rpc/router.go:

r.Post("/internal/rpc/{function_name}", handleRPCCall)

The receiver looks the handler up in the registry and invokes it with the decoded args/kwargs. What the handler answers comes back as HTTP 200: its result, or its failure as a typed {"error", "error_type"} payload the caller can re-raise. Three refusals made before the handler runs use the same HTTP 200 envelope: an unknown function name, a node already serving its limit of calls from this peer, and a binary argument that cannot be resolved, whether its reference is malformed or its transfer cannot be pulled. A request refused on its shape or its caller gets a real HTTP status: 403 on the wrong listener or for a missing capability, 400 with no function name or a body that cannot be read or parsed, 413 for a body over the size limit.

ComponentFile
Handler registry + registrationmodules/internal_rpc/registry.go
Receiver endpointmodules/internal_rpc/router.go
Caller sidemodules/internal_rpc/client.go
Function descriptionsmodules/internal_rpc/reference.go, each package’s rpc_reference.go

Pattern 6: Task System (event-driven queue)

Section titled “Pattern 6: Task System (event-driven queue)”

Persistent, retryable background tasks. Handlers are triggered by events or cron schedules. Each execution creates a TaskModel row in the database with status tracking and retry logic.

When to use: Fire-and-forget reactions to events. Config change handlers, retention cleanup, repo sync, notification delivery: anything that needs an audit trail or retry on failure.

Event-triggered:

func init() {
events.Register(V200100CFGCHG, backupConfigHandler)
events.Register(events.Run, handleJobRun)
events.RegisterWithOptions(V200701NETTNT, handleTenantNetworkCreate, events.Options{Dispatch: DispatchModeLeader})
}
func backupConfigHandler(ctx context.Context) error {
// Runs whenever config changes.
return nil
}
func handleJobRun(ctx context.Context, jobRunID int) error {
// Runs when a job run event fires.
return nil
}
func handleTenantNetworkCreate(ctx context.Context) error {
// Leader-only handler.
return nil
}

Cron-triggered:

func init() {
task.Cron(func() string { return config.MaintenanceCleanupInterval }, cleanupCron)
task.CronFixed("*/5 * * * *", retentionConfigWatchCron)
}
func cleanupCron(ctx context.Context) error {
// Hourly cleanup: orphaned containers, stale jobs.
return nil
}
func retentionConfigWatchCron(ctx context.Context) error {
// Fixed crontab schedule.
return nil
}

Manual enqueue:

func init() {
task.Register("deploy_stack", deployStack)
}
func deployStack(ctx context.Context, deploymentID, compose string) error {
// ...
return nil
}
// Later:
task.Enqueue("deploy_stack", deploymentID, compose)
task.EnqueueLeader("deploy_stack", deploymentID, compose)
task.EnqueueNode("deploy_stack", "node-02", deploymentID, compose)

When an event fires (e.g. via log.Event(V200100CFGCHG)), the queue’s DispatchEvent finds all registered handlers and enqueues them: subsystems/impl/queue.go:

func (q *Queue) DispatchEvent(code string, message string, payload map[string]any, ctx *Ctx) error {
handlers := q.eventHandlers[code]
for _, h := range handlers {
taskKwargs := map[string]any{
"event_code": code,
"event_message": message,
"event_payload": payload,
}
if h.dispatchMode == DispatchModeBroadcast {
go q.executeBroadcastHandler(h, taskKwargs)
} else {
q.Enqueue(h.name, "event", code, h.dispatchMode, taskKwargs)
}
}
return nil
}

Enqueue() creates a TaskModel row in the DB. The queue’s poll loop picks it up and executes the handler.

ModeBehavior
SingleEnqueue as DB row, any worker picks it up
LeaderEnqueue as DB row, only the leader node executes
BroadcastExecute immediately in-process on all nodes (no DB record)

task.Cron registers with the cron scheduler. When the schedule fires, it checks for already-running tasks with the same name (skip-if-running), then enqueues:

func cronEnqueue(c *Cron) error {
// Skip if already pending/running
var count int
db.QueryRow(`
SELECT COUNT(*) FROM task
WHERE name = ? AND status IN ('pending', 'running')
`, c.name).Scan(&count)
if count > 0 {
return nil
}
return q.Enqueue(c.name, "cron", "", c.schedule, nil)
}
ComponentFile
QueueSubsystem + TaskRegistrationsubsystems/impl/queue.go
Event/handler registrycore/registry/registry.go
Task model + workermodules/task/models.go (TaskModel), modules/task/worker.go; ent table in ent/schema/task.go
Task handlers (examples)modules/health/tasks.go, modules/network/tasks.go

File transfers go through the public API.

Upload: shc backup import <path> posts the archive body to POST /api/method/shc.backup.import via the daemon client (modules/backup/register.go; server side in modules/backup/restore_file.go).

Download: shc backup export streams GET /api/method/shc.backup.export/{backup_id} as chunked application/octet-stream (modules/backup/router.go, streaming service in modules/export/).


QuestionRPC (internal_rpc.Register)Task (queue-claimed handler)
Need result back?Yes: returns *RpcResultNo: fire and forget
Need retries?Transport-level onlyConfigurable MaxAttempts
Need audit trail?No DB recordTaskModel row persisted
Execution frequencyCan be very high (every 2s)Lower frequency (minutes/hours)
Cross-node?Yes: peer-client POST to any node; fan-out per call siteSingle/leader via queue, broadcast in-process
ExamplesHealth check, metrics, network opsConfig change reaction, repo sync, notifications
QuestionSSEWebSocket
DirectionServer → Client onlyBidirectional
Client sends data?NoYes (stdin, TCP packets, stop commands)
DurationUntil operation completesUntil user disconnects
Reconnectable?Yes (Last-Event-Id)No (new connection)
ExamplesInstall/update/uninstall/backup progressShell, port forward, HUD stream
QuestionResource APIMethod API
HTTP verbsGET, POST, PUT, DELETEPOST only
URL pattern/api/resource/{Type}/{id}/api/method/shc.{module}.{action}
SemanticsCRUD on domain objectsExecute an action/procedure
ExamplesList stacks, get node, delete backupInstall stack, create backup, set config

A few CLI commands skip the public API for good reasons:

CommandPatternReason
shc doctor runCLI → API, degraded local fallbackDaemon-first (DB/keycloak checks live there); with no daemon it runs a local diagnose-only pass so triage still works when the API is broken (--fix requires the daemon)
shc ctxCLI → local filesystemClient-side context; only calls API for validation
shc cluster initCLI → local bootstrapGenesis runs before any daemon exists; every other shc cluster * verb dials the daemon like the rest of the CLI
shc forwardCLI → local daemon over the UDS onlyThe forward tunnel is deliberately not exposed on the public API (cli/forward_wiring.go), so it needs a local daemon socket and refuses to run from a remote ctx login; the daemon owns the peer hop