Metrics module

Source: modules/metrics/ and subsystems/impl/collectors.go.

The metrics module wires SHC’s OpenTelemetry metrics pipeline — counters, gauges, and histograms for HTTP requests, database queries, task queue depth, docker operations, per-stack quota usage, and cluster-wide rollups. Metrics export to an OTLP collector (OpenObserve by default; see collector in default.yaml).

The pipeline has three parts:

1. Stats cache (stats_cache.go) — in-memory per-container + per-node readings the docker subsystem pushes on every tick. Hot reads serve Status, Inspect, and HUD render paths under sync.RWMutex (≈37ns/op for single-key reads; see BenchmarkGetContainer).
2. Service (service.go) — the emit surface every other module calls via metrics.GetRuntimeService(). Wraps one otel.Meter and applies cardinality caps + diagnostic-cardinality dropping per B9 fix #3.
3. Tick cron + collectors (modules/metrics/collectors/ and the collectors subsystem at subsystems/impl/collectors.go) — periodic emitters that scan the cache + DB and call Service.RecordGauge / RecordCounter. The tick cron runs every 2s by default (timing.tick_interval).

Registered in priority order at Collectors.Start:

Leader-only collectors run only on the elected raft leader so followers don’t double-emit cluster-wide signals.

Per B9-observability.md Phase 3, every known non-default tenant gets its own scope-named Meter alongside the base set:

mp := daemonMeterProvider
baseMeter   := mp.Meter("shc")
acmeMeter   := mp.Meter("shc.tenant.acme")
mangoMeter  := mp.Meter("shc.tenant.mango")

Each per-tenant Meter backs a separate *metrics.Service and a tenant-filtered variant of the Container / Quota / Aggregate collectors (NewContainerCollectorScoped etc.). The scoped collector iterates the same source data but only emits rows whose TenantName matches; the emit goes through the tenant’s Meter so the OTLP instrumentation_scope.name attribute carries the tenant identifier.

Downstream OTLP receivers (Mimir, Tempo, OpenObserve, etc.) can filter metrics by scope.name directly:

shc_container_running{scope_name="shc.tenant.acme"}

…instead of relying on the tenant label, which is still emitted but joins less efficiently in large clusters. Single-tenant installs see no per-tenant scope — only the base shc Meter.

The collectors subsystem re-enumerates tenants every 30 ticks (≈60s by default). A new tenant created at runtime starts emitting under its scope on the next refresh; the per-tenant metrics.Service allocates its own instruments on demand.

Failures of per-tenant service construction (cardinality cap, etc.) log X503056COLPRT and skip that tenant for the cycle — the base collectors keep emitting normally.

Per-tenant Resource attributes (not just scope name) are NOT differentiated. OTel SDK ties Resource to MeterProvider, and the daemon initializes a single MeterProvider at boot. Per-tenant Resource would require a per-tenant CollectorConfig surface that doesn’t exist yet — it’s a separate workstream documented in the B9 follow-up.

modules/metrics/
├── service.go            # emit surface (RecordCounter/Gauge/Histogram)
├── stats_cache.go        # in-memory per-container + per-node readings
├── types.go              # ContainerCacheEntry, NodeStats
├── catalog.go            # metric definitions (catalog of names + labels)
├── collectors/
│   ├── container.go      # per-container gauges + tenant filter
│   ├── node.go           # per-node gauges
│   ├── quota.go          # per-scope quota gauges + tenant filter
│   ├── aggregate.go      # four-tier rollup + tenant filter
│   ├── subsystem_state.go
│   ├── scheduling.go
│   └── system_tick_source.go
└── ...

The collectors subsystem at subsystems/impl/collectors.go owns the tick cron + the per-tenant service wiring.