Observability

SHC emits four signals operators can use to see what’s happening inside a running cluster:

1. HTTP health / status endpoints — quick liveness checks
2. Structured logs — every event the daemon emits
3. Metrics — internal gauges/counters/histograms, including node-exporter/cAdvisor-compatible series for containers
4. Traces — request-scoped spans across the API surface

All four are push-based (OTLP-HTTP) to an observability backend rather than scraped. The destination is configured once — or derived automatically from the system OpenObserve app — and SHC streams to it. See OTLP export configuration below.

Two health surfaces, on two different listeners — pick by who is asking:

The cluster status method on the public API listener (0.0.0.0:8000). Deliberately exempt from auth and the quorum-required middleware — it must answer even when the control plane is degraded. Returns the standard { "message": ... } envelope:

{
  "message": {
    "nodes": [ { "node": "shc-node-1", "state": "healthy", "cpu_percent": 12.4 } ],
    "deployments": [ { "stack": "keycloak", "state": "running" } ],
    "last_poll": "2026-07-19T00:00:00Z"
  }
}

Source: modules/health/router.go (GET and POST serve the same projection).

Being unauthenticated is by design, so treat it as public surface: restrict at the network layer (firewall the port, or front it with your load balancer) rather than expecting auth.

The node-liveness probe peer daemons poll, served on the internal API listener (:4003) — mTLS-only, so it is not reachable without a cluster-issued client certificate. Reports the node’s local state (healthy / degraded / unhealthy from disk + memory thresholds), uptime, leader flag, and system metrics. Source: modules/health/internal_router.go.

Point the check at the public listener — the internal :4003 listener rejects anything without a cluster client cert:

HealthCheck:
  Path: /api/method/shc.status.get
  Port: 8000
  Interval: 10s
  HealthyThreshold: 2
  UnhealthyThreshold: 3
  Timeout: 5s

SHC does not use bare stdlib log — it emits structured events through the log module (core/log/log.go), which fans records out to the console/journal AND (when OTLP export is active) to the OTel log bridge.

Every log record is a structured object with:

• timestamp
• severity — DEBUG, INFO, WARN, ERROR (there is no TRACE or FATAL tier; corelog.Fail logs at ERROR then exits)
• body — human message
• attributes — dict of contextual key/value pairs
• trace_id, span_id — if emitted inside a traced request
• resource — { tenant, stack, environment, service, container, node_id } as available

Three ways in:

Every error / warning event carries a code of the form X<HTTP><SEQ><MNEMONIC> (e.g. X500908HKFAIL). The code is stable across releases and documented in the error reference — if you see a code in a log or user-facing response, that’s the canonical place to look up what it means.

Every SHC module declares the metrics it emits via the metrics helper (core/otel/otel.go). Three primitives:

• Counter — monotonically increasing integer
• Gauge — point-in-time value
• Histogram — bucketed distribution; HistogramTimer times operations via a context manager

In addition to SHC-internal metrics, each node collects host and container stats and emits them under names that match node-exporter 1.x and cAdvisor exactly (modules/metrics/definitions.go, ce module). That means standard Grafana dashboards work without translation:

Full list: see prom_definitions.go.

Tested against Grafana dashboard 1860 (Node Exporter Full) and dashboard 893 (Docker Container).

SHC does not currently expose a /metrics scrape endpoint. All metrics are pushed (OTLP-HTTP) to the configured export destination — see OTLP export configuration. The /metrics path is reserved (exempt from auth in middleware) but no handler is installed.

If you need a scrape-style workflow, point the OTLP export at an OpenTelemetry Collector with a Prometheus exporter, or run a backend that accepts OTLP directly (OpenObserve, Mimir).

Beyond the tenant label every metric already carries, the collector subsystem emits per-tenant metrics under a distinct OTel instrumentation scope so dashboards can filter at the scope level rather than scanning labels:

Both representations are emitted simultaneously — the base set with the tenant label, and the tenant-scoped set without it. The scope attribute is materialised by OTel SDK when a different Meter() name is requested from the same MeterProvider, so single-tenant installs incur no overhead (no extra scope is created).

The fan-out covers three collectors: container, quota, and aggregate. Node-wide collectors (node, subsystem_state, scheduling, collect_node_tick) stay on the base scope — they have no tenant axis to differentiate.

Operators using Mimir / Tempo / OpenObserve can write queries like:

sum by (stack) (shc_container_running{scope_name="shc.tenant.acme"})

…to filter to one tenant without scanning the label cardinality of the whole cluster.

The tenant set refreshes every 30 ticks (≈60s default). A newly- created tenant starts emitting under its scope on the next refresh.

For implementation details and limitations (per-tenant Resource is NOT differentiated — that’s a separate workstream), see architecture/modules/metrics.md.

Every API request opens a span. Child spans wrap:

• Database queries
• Docker client calls
• Inter-node API calls (over mTLS)
• Hook / integration-phase job execution

Spans propagate trace_id / span_id across internal API boundaries via request headers so a single request can be traced across the leader and a worker node.

Traces are pushed to the same OTLP destination as logs and metrics.

Source: core/otel/otel.go (Tracer, Span, CurrentSpan).

All export is standard OTLP-HTTP from the OpenTelemetry SDK — there is no bespoke collector process and no custom record format. Each signal POSTs to the standard per-signal path under the configured base: <base>/v1/traces, <base>/v1/metrics, <base>/v1/logs.

Signals shipped today:

Not yet shipped: application/container log shipping (your app stacks’ stdout/stderr). Container metrics are shipped; use shc stack logs <stack> / docker logs for app logs.

When the system OpenObserve app is deployed (it is part of the default system-app set) and the cluster has a public host (shc init --host), the daemon auto-wires itself to it — endpoint https://<SHC_HOST>, path base /otel/api/default. Once the vault is unlocked, a leader reconcile pass persists this as the managed provider with the openobserve root credential attached, writing a boot-readable drop-in (/etc/shc/config.d/otel-managed.yaml, mode 0600 — it holds the Basic-auth credential) plus the global config rows, and emits X200506OTLPST: one daemon restart later, ingest is authenticated. No configuration required.

An explicit provider always overrides the managed default. Set it under otel: in your config (see config-files reference):

otel:
  provider: obs
  providers:
    obs:
      endpoint: https://observe.example.com
      url_path_base: ""          # empty = standard /v1/* paths
      insecure: false            # true = plaintext http:// (dev only)
      headers:
        Authorization: ref+vault://otel_token

Source of truth: default.yaml (the otel: block), core/otel/otel.go (exporter assembly).

Any OTLP-HTTP-compatible backend works (OpenObserve, Grafana Cloud, Honeycomb, an OpenTelemetry Collector, …). TLS verification uses the union of the host’s system roots and the cluster’s internal global root, so both public SaaS endpoints and the SHC-internal https://<SHC_HOST>/otel route verify without any insecure toggle.

If the OTLP endpoint is unreachable, the SDK’s batch exporters:

• Buffer records/spans in bounded in-memory queues and retry
• Drop data when a queue overflows — never block the daemon
• Surface export failures as WARN lines in the daemon log (deliberately routed outside the log bridge, so a failing log export cannot feed itself)

Losing telemetry never blocks daemon work — all export is asynchronous, off the hot paths.

For a real-time operator view that doesn’t require an observability backend:

shc hud

The HUD is a full-screen TUI that subscribes to the same event stream, renders node/stack/container health live, and surfaces errors as they happen. See architecture/modules/hud.

It’s the single best first-look tool when something seems wrong.

1. shc hud — visual overview, shows red where things are broken.
2. shc stack logs <stack> — raw container logs.
3. shc logs tail --filter stack=<stack> — daemon events for the stack (install, hook runs, healthcheck transitions).
4. shc -s <stack> events — lifecycle events (A200100STKINS, V500320STKERR, etc.).

shc logs search --filter node_id=<id> --since 1h
shc events --node <id>                   # live events from that node
shc events --audit --since <YYYY-MM-DD>  # audited actions in the window

All container metrics carry tenant, stack, environment labels. Query on those:

rate(container_cpu_usage_seconds_total{tenant="acme"}[5m])

• Disaster recovery — what a degraded control plane means and how to fix it
• Architecture: events flow — how lifecycle events propagate to subscribers
• Architecture: audit module — audit trail details
• Architecture: metrics module — metric collection internals
• OpenObserve app — turnkey OTLP target (the managed default)
• Error reference — what error codes mean