Skip to content
SHC Docs

Unified event bus

Layer 2. Core. Core

The Unified Event Bus is the central event routing system in SHC. It receives all events from log.Event(), log.Error(), and log.Warn() and routes them to handlers, cross-node broadcast, and SSE streaming.

interactive: drag/click to explore

Previously, events flowed through multiple disconnected paths:

  • log.Event() → DB polling → task handlers
  • Docker events → SSE broadcast → clients
  • Separate notification dispatch

This fragmentation caused:

  • Inconsistent event handling
  • Complex debugging
  • No unified tracing
  • Different tracking mechanisms for different dispatch modes

A single in-memory event bus that:

  1. Receives all events via log.Event(), log.Error(), log.Warn()
  2. Routes to local handlers based on dispatch mode
  3. Broadcasts to other nodes via HTTP POST
  4. Publishes to SSE channels for client streaming
flowchart TB
LOG["log.Event() / log.Error() / log.Warn()"] --> BUS["EventBus (in-memory)<br/>event channel · consumer goroutine · handler registry"]
BUS --> LH["Local Handlers"]
BUS --> XN["Cross-Node<br/>RPC events.deliver"]
BUS --> SSE["SSE Broadcast<br/>(progress, audit, etc.)"]

Events are represented by the Event struct:

type Event struct {
// Identity
Code string `json:"code"` // A200100STKINS, X500907STKFAL
Message string `json:"message"` // Rendered human-readable message
Payload map[string]any `json:"payload"` // Event-specific data
// Tracing
TraceID string `json:"trace_id"` // 32 hex chars
SpanID string `json:"span_id"` // 16 hex chars
ParentSpanID string `json:"parent_span_id"` // Optional
// Scope
TenantName string `json:"tenant"` // Default: "default"
StackName string `json:"stack"` // Optional
Environment string `json:"environment"` // Optional
Actor string `json:"actor"` // Default: "shc"
// Metadata
Timestamp time.Time `json:"timestamp"`
SourceHostname string `json:"source_hostname"`
Level Level `json:"level"` // info, warning or error
// Flags
IsAudit bool `json:"is_audit"` // A-prefix
}

Wire format note: when serialized for the RPC bus or SSE, the scope fields are emitted as tenant / stack (without the _name suffix) for backward compatibility with external consumers. The Go struct uses TenantName / StackName internally to match ctx and the rest of the codebase.

Event codes follow a specific format:

PrefixMeaningExample
VVerbose/info eventV200300CTRSTR - container started
XCoded error or warning fanned onto the busX500907STKFAL - deployment operation failed
AAudit eventA200100STKINS - deployment installed

Format: {prefix}{http_status}{sequence}{mnemonic}

  • Prefix: 1 char (V/X/A)
  • HTTP status: 3 digits
  • Sequence: 3 digits
  • Mnemonic: 6+ chars

There is no dedicated warning prefix: warnings ride X2xxxxx codes published through the typed logger at level warning.

Events can trigger handlers in three dispatch modes:

Only one node executes the handler. Used for tasks that must run exactly once.

func init() {
events.Register("A200100STKINS", events.DispatchSingle, handleInstall)
}
func handleInstall(ctx context.Context, e *events.Event) error {
// Only one node executes this
return nil
}
  • Task is enqueued to database
  • First node to claim the task executes it
  • Status tracked in node_executions JSONB field

All nodes execute the handler. Used for cache invalidation, local state updates.

func init() {
events.Register("V200100CFGCHG", events.DispatchBroadcast, handleConfigChange)
}
func handleConfigChange(ctx context.Context, e *events.Event) error {
// All nodes execute this
reloadLocalConfig()
return nil
}
  • Executed immediately on receiving node
  • All nodes track execution in node_executions
  • Task is “complete” when all nodes finish

Only the leader node executes the handler. Used for cluster-wide coordination.

func init() {
events.Register("V200401BKPCMP", events.DispatchLeader, handleBackup)
}
func handleBackup(ctx context.Context, e *events.Event) error {
// Only leader executes this
return nil
}
  • Only executes if current node is leader
  • Enqueued to database for tracking
  • Falls back to SINGLE if no leader

Events are published via log.Event():

import "gitlab.com/bitspur/selfhosted-cloud/ce/core/log"
log.Event(ctx, events.A200100STKINS, map[string]any{
"stack_name": "postgres",
"version": "16.2",
})

This:

  1. Gets current context from ctx
  2. Creates an Event with tracing info
  3. Publishes to the event bus channel

The event bus consumer goroutine:

  1. Receives events from the channel
  2. Routes to registered handlers
  3. Broadcasts to other nodes (if local origin)
  4. Publishes to SSE channels

For each registered handler:

// BROADCAST: Execute immediately in goroutine
go executeHandler(handler, event, kwargs)
// LEADER: Execute if leader
if isLeader() {
enqueueTask(handler, event, kwargs, DispatchLeader)
}
// SINGLE: Enqueue for claim-based execution
enqueueTask(handler, event, kwargs, DispatchSingle)

Handlers receive context with event details:

func handleInstall(ctx context.Context, e *events.Event) error {
slog.InfoContext(ctx, "Handling event", "code", e.Code)
stack := e.Payload["stack"].(string)
return nil
}

Events are broadcast to other nodes with the events.deliver internal RPC function:

sequenceDiagram
participant A as Node A (origin)
participant B as Node B
participant C as Node C
Note over A: Event created
A->>B: RPC events.deliver
A->>C: RPC events.deliver
Note over B,C: Event received<br/>Published to local bus<br/>Handlers execute<br/>NOT re-broadcast

The NodeBroadcaster (modules/events/broadcaster.go) handles cross-node delivery. Per B12 Bug 2 FIX, delivery is no longer fire-and-forget: each peer has a per-peer in-memory retry queue with bounded exponential backoff (5 attempts, base 200ms, jitter, 30s wall budget). After exhaustion the shc_event_bus_peer_failures_total{peer=…} counter increments.

The event travels as Event.ToDict(), the same JSON envelope the SSE stream’s data: payload uses, including the sequence field once the originating node persists the row. It is passed as one JSON string in the event argument, next to source_hostname. The RPC layer reads a one-key $blob or $transfer map as a binary reference, and an event payload is free to hold one, so the event is never sent as a nested object.

Other nodes receive events through events.deliver, served by the one internal RPC dispatch (POST /internal/rpc/events.deliver) on the internal listener only. The function lives in modules/events/rpc.go and enforces the loop-prevention contract from B12 Bug 5:

  1. source_hostname is required. A delivery without it is refused with X400000BADREQ. The broadcaster always sets it.
  2. Self-loop drop. If source_hostname equals this node’s own SourceHostname, the call answers {delivered: false} without touching the bus, so the sender does not retry and a wrong peer address cannot amplify into a storm.
  3. Deliver, never persist. The envelope decodes into an Event and fans to in-process subscribers through DeliverPeer. Nothing is written to event: on a multi-node cluster that table is the raft-replicated database, so the originator’s row is already this node’s row, and its id stays the SSE resume cursor.
AnswerMeaning
{delivered: true}Delivered to local subs. Broadcaster acks.
{delivered: false}Self-loop dropped. Broadcaster acks (no retry).
X400000BADREQMissing source_hostname or malformed event. Counts as a failed attempt.
X500007EVTBUSBus not running. Broadcaster retries with backoff.

Events are published to SSE channels for client streaming:

ChannelDescription
operation:{tenant}:{operation_id}Operation progress (step, status, complete, error events)
events:{tenant}All events (docker, audit, etc.) for a tenant
errors:{tenant}Error events for a tenant
sequenceDiagram
participant Client
participant Server
Client->>Server: POST /api/method/shc.backup.create_async
Server-->>Client: 202 Accepted<br/>traceparent: 00-abc123...-01<br/>{"operation_id": "abc123..."}
Client->>Server: GET /api/method/shc.operation.stream/abc123...
Server-->>Client: SSE Events<br/>event: step<br/>data: {"step": "validating"}

SSE Event Types (operation:{tenant}:{operation_id} channel)

Section titled “SSE Event Types (operation:{tenant}:{operation_id} channel)”
Event TypeDescription
stepProgress update with step name, percentage, and message
container_statusContainer state changes
service_statusService replica counts and status
stack_statusOverall stack status changes
completeOperation completed successfully
errorOperation failed
{
"code": "A200100STKINS",
"message": "Stack installed successfully",
"payload": {"stack_name": "postgres", "version": "16.2"},
"timestamp": "2026-02-04T12:00:00.000Z",
"node": "node-1"
}

Tasks are tracked in the database with a node_executions JSONB field:

type NodeExecution struct {
NodeID string `json:"node_id"`
Status NodeExecutionStatus `json:"status"` // PENDING, RUNNING, COMPLETED, FAILED, SKIPPED
StartedAt *time.Time `json:"started_at"`
CompletedAt *time.Time `json:"completed_at"`
Error string `json:"error,omitempty"`
}

Overall task status is derived from node executions:

Dispatch ModeStatus Logic
SINGLEStatus = single execution’s status
LEADERStatus = leader execution’s status
BROADCASTALL must complete, ANY failure = failed

Handlers are registered via the events.Register function:

import "gitlab.com/bitspur/selfhosted-cloud/ce/modules/events"
func init() {
events.Register("A200100STKINS", events.DispatchSingle, handleStackInstalled)
}
func handleStackInstalled(ctx context.Context, e *events.Event) error {
stack := e.Payload["stack"].(string)
return notifyUsers(ctx, stack)
}

Multiple handlers can be registered for the same event code.

The event bus is managed as a service:

// In server startup
manager.Register(NewEventBusComponent(), 5)
// EventBusComponent delegates to eventBus.Start() / Stop()

The event bus starts early (priority 5) so other components can publish events during their startup.

After startup, the SSE broker and node broadcaster are wired into the event bus:

// In server.go
eventBus.SetSSEBroker(sseBroker)
eventBus.SetNodeBroadcaster(nodeBroadcaster)

If the event channel buffer is full (default: 10,000 events), new events are dropped:

select {
case bus.queue <- event:
// sent
default:
slog.Debug("event bus queue full, dropping event", "code", event.Code)
}

Design decision: Fire-and-forget. Events are ephemeral. Critical state uses polling.

A handler is isolated twice over: a returned error and a panic are both contained, and neither takes the bus down or skips the remaining handlers for that code. Both are handed to the service’s OnHandlerError seam, which the daemon wires to the log:

func (s *Service) invokeTaskHandler(ctx context.Context, reg TaskRegistration, e *Event) {
defer func() {
if r := recover(); r != nil {
if s.opts.OnHandlerError != nil {
s.opts.OnHandlerError(reg.handlerName(), e.Code, errFromPanic(r))
}
}
}()
if err := reg.Handler(ctx, *e, reg.Kwargs); err != nil {
if s.opts.OnHandlerError != nil {
s.opts.OnHandlerError(reg.handlerName(), e.Code, err)
}
}
}

A returned error travels as it was raised; the bus does not re-code it, so whatever the handler minted is what gets logged. A panic is projected to an error by errFromPanic: a panic value that already is an error is adopted through coded.FromExternal, and anything else becomes X500016EVTHND: event handler panicked {reason} {error}. That is the only code the bus itself mints on this path.

Cross-node broadcast failures are logged but not retried:

if err := bus.nodeBroadcaster.Broadcast(ctx, event); err != nil {
slog.Debug("event broadcast failed", "error", err)
}

The event bus has minimal configuration:

SettingDefaultDescription
Queue size10,000Maximum buffered events
MetricDescription
shc_event_bus_events_totalTotal events processed
shc_event_bus_events_droppedEvents dropped (queue full)
shc_event_bus_handlers_totalHandler registrations
shc_event_bus_handler_errorsHandler execution errors

Enable debug logging to see event flow:

Terminal window
# Dev: run the daemon with debug logging
SHC_DEBUG=true make dev
# Production: edit the systemd unit to add the env var, then restart
sudo systemctl edit shc # add: Environment=SHC_DEBUG=true
sudo systemctl restart shc
  1. Use appropriate dispatch modes

    • SINGLE for exactly-once execution
    • BROADCAST for local cache/state updates
    • LEADER for cluster coordination
  2. Keep handlers fast

    • Long-running work should spawn goroutines, not block
    • Return quickly from handlers
  3. Don’t rely on event delivery

    • Events are fire-and-forget
    • Critical state should use polling/DB queries
  4. Use operation_id for SSE progress

    • operation_id equals trace_id for the request
    • SSE clients subscribe to GET /api/method/shc.operation.stream/{operation_id}
  5. Handle errors gracefully

    • Handler errors are logged, not fatal
    • Check for nil when context might be missing
modules/events/
├── service.go # EventBus, Event, EventBusComponent
├── broadcaster.go # NodeBroadcaster for cross-node events
├── router.go # SSE stream handlers
├── rpc.go # events.deliver, the cross-node receive function
├── handlers.go # Handler registration, DispatchMode
└── models.go # Event struct, NodeExecution