Metrics module
Layer 4. Primitives. Primitives Readings have no owned record.
Source: modules/metrics/ and subsystems/impl/collectors.go.
Responsibilities
Section titled “Responsibilities”The metrics module wires SHC’s OpenTelemetry metrics pipeline:
counters, gauges, and histograms for HTTP requests, database queries,
task queue depth, docker operations, per-stack quota usage, and
cluster-wide rollups. Metrics export to an OTLP collector
(OpenObserve by default; see collector in default.yaml).
The pipeline has three parts:
- Stats cache (
stats_cache.go): in-memory per-container + per-node readings the docker subsystem pushes on every tick. Hot reads serveStatus,Inspect, and HUD render paths undersync.RWMutex(≈37ns/op for single-key reads; seeBenchmarkGetContainer). - Service (
service.go): the emit surface every other module calls viametrics.GetRuntimeService(). Wraps oneotel.Meterand applies cardinality caps + diagnostic-cardinality dropping per B9 fix #3. - Tick cron + collectors (
modules/metrics/collectors/and thecollectorssubsystem atsubsystems/impl/collectors.go): periodic emitters that scan the cache + DB and callService.RecordGauge/RecordCounter. The tick cron runs every 2s by default (timing.tick_interval).
Collectors
Section titled “Collectors”Registered in priority order at Collectors.Start:
| Collector | Leader-only | Source |
|---|---|---|
collect_node_tick | no | writes the stats cache from the docker daemon |
node | no | per-node CPU/MEM/disk gauges |
container | no | per-container gauges (one entry per running container) |
subsystem_state | no | lifecycle state per subsystem |
quota | yes | per-(tenant, env, stack) quota readings |
aggregate | yes | four-tier rollup (service / stack / env / tenant) |
scheduling | no | capacity vs scheduled per node |
Leader-only collectors run only on the rqlite leader so followers don’t double-emit cluster-wide signals.
The disk axis is absent, not zero
Section titled “The disk axis is absent, not zero”shc_stack_disk_usage_bytes (and its environment and tenant siblings)
publish only when something measured the bytes. A stack’s disk is the
bytes its block volumes hold, as the nodes holding them last measured them
(storage.StackVolumeBytes, FZ-7801; see storage). A stack
with no measured volume publishes no gauge and shc.metrics.stack sends
disk_bytes: null for it; data a deployment keeps on the node disk is
quota’s to measure, not this gauge’s.
It used to publish a hard 0 instead, from a rollup field nothing ever
assigned. Measured on the corp estate 2026-09-07: all 91 rows of the
console’s estate read carried disk_bytes: 0, the springboard status
strip read “Disk 0 B”, and the disk glyph could never change colour; on
an estate holding hundreds of GB.
A consumer must therefore treat disk_bytes: null as “no reading” and
draw nothing, not ?? 0. A measured zero still travels as 0 and
still prints: absence and idle are different facts (SHC-566).
Per-tenant scope routing
Section titled “Per-tenant scope routing”Per B9-observability.md Phase 3, every known non-default tenant
gets its own scope-named Meter alongside the base set:
mp := daemonMeterProviderbaseMeter := mp.Meter("shc")acmeMeter := mp.Meter("shc.tenant.acme")mangoMeter := mp.Meter("shc.tenant.mango")Each per-tenant Meter backs a separate *metrics.Service and a
tenant-filtered variant of the Container / Quota / Aggregate
collectors (NewContainerCollectorScoped etc.). The scoped collector
iterates the same source data but only emits rows whose TenantName
matches; the emit goes through the tenant’s Meter so the OTLP
instrumentation_scope.name attribute carries the tenant identifier.
Operator impact
Section titled “Operator impact”Downstream OTLP receivers (Mimir, Tempo, OpenObserve, etc.) can
filter metrics by scope.name directly:
shc_container_running{scope_name="shc.tenant.acme"}…instead of relying on the tenant label, which is still emitted but
joins less efficiently in large clusters. Single-tenant installs see
no per-tenant scope, only the base shc Meter.
Tenant set refresh
Section titled “Tenant set refresh”The collectors subsystem re-enumerates tenants every 30 ticks
(≈60s by default). A new tenant created at runtime starts emitting
under its scope on the next refresh; the per-tenant metrics.Service
allocates its own instruments on demand.
Failures of per-tenant service construction (cardinality cap, etc.)
log X503056COLPRT and skip that tenant for the cycle; the base
collectors keep emitting normally.
Known limitation
Section titled “Known limitation”Per-tenant Resource attributes (not just scope name) are NOT
differentiated. OTel SDK ties Resource to MeterProvider, and the
daemon initializes a single MeterProvider at boot. Per-tenant
Resource would require a per-tenant CollectorConfig surface that
doesn’t exist yet; it’s a separate workstream documented in the
B9 follow-up.
Source layout
Section titled “Source layout”modules/metrics/├── service.go # emit surface (RecordCounter/Gauge/Histogram)├── stats_cache.go # in-memory per-container + per-node readings├── types.go # ContainerCacheEntry, NodeStats├── collectors/│ ├── container.go # per-container gauges + tenant filter│ ├── node.go # per-node gauges│ ├── quota.go # per-scope quota gauges + tenant filter│ ├── aggregate.go # four-tier rollup + tenant filter│ ├── subsystem_state.go│ ├── scheduling.go│ └── system_tick_source.go└── ...The collectors subsystem at
subsystems/impl/collectors.go
owns the tick cron + the per-tenant service wiring.