Skip to content
SHC Docs

Metrics

Two places hold numbers about the cluster. The daemon holds the last few seconds of every container’s CPU and memory, which shc status --resources prints. The system OpenObserve on obs01 holds everything the collectors shipped: host metrics, container metrics, container logs and the daemon’s own log lines, kept for days. The daemon ships there on its own, the managed default; otel.provider would name an otel.providers entry instead.

explained, not walked: the managed default ships here; a named provider needs a second OTLP destination
$ shc config set otel.providers.remote.endpoint https://otlp.example.com:4318
$ shc config set otel.provider remote

Three more rows shape what is shipped. otel.filter.quiet drops the container log lines that match a pattern before they leave the node, and otel.filter.rules takes a raw OTTL condition for what a pattern cannot say; both are scoped like any config row: -t prod quiets one tenant’s deployments and -s web -e prod one deployment, and a row set with no -t, -s, -e or --scope is global. otel.enabled false is the master switch: it takes the collector off every node.

explained, not walked: each of these would thin or stop the telemetry the rest of this page reads
$ shc config set otel.filter.quiet 'GET /healthz' --scope tenant -t prod
$ shc config set otel.filter.rules 'IsMatch(body, "^DEBUG")' --scope tenant -t prod
$ shc config set otel.enabled false
home01
$ shc status --resources
last polled 2026-10-09 07:03:04 UTC
NODE STATE CPU% MEM% DISK% LOAD
home01 healthy 31.8 37.5 17.9 2.63
obs01 healthy 6.1 6.6 2.5 0.17
all 285 queues green on 2 nodes
TENANT STACK ENVIRONMENT STATE SERVICES NODES LABELS NO HEALTHCHECK
master keycloak default healthy 1/1 home01 app=keycloak environment=default stack=keycloak tenant=master -
master openobserve default healthy 1/1 obs01 app=openobserve environment=default stack=openobserve tenant=master 1
master postgres default healthy 2/2 home01 app=postgres environment=default stack=postgres tenant=master -
master traefik default healthy 15/15 home01, obs01 app=traefik environment=default stack=traefik tenant=master -
prod cloud prod healthy 4/4 home01 app=nextcloud environment=prod stack=cloud tenant=prod 1
prod db prod healthy 2/2 home01 app=postgres environment=prod stack=db tenant=prod -
prod pgweb prod healthy 1/1 home01 app=/root/learn/pgweb environment=prod stack=pgweb tenant=prod -
prod placed prod healthy 1/1 home01 app=notes environment=prod stack=placed tenant=prod tier=learn -
prod store prod healthy 1/1 home01 app=/root/learn/versitygw environment=prod stack=store tenant=prod -
prod web prod healthy 1/1 home01 app=/root/learn/nginx environment=prod stack=web tenant=prod 1
STACK ENVIRONMENT CPU MEMORY MEASURED
keycloak default 0.43% 1.03 GiB 1/1
openobserve default 3.08% 636.6 MiB 1/1
postgres default 2.68% 164.0 MiB 2/2
traefik default 7.49% 853.4 MiB 15/15
cloud prod 0.81% 1.15 GiB 4/4
db prod 0.65% 89.5 MiB 2/2
pgweb prod 0.00% 8.5 MiB 1/1
placed prod 0.64% 77.6 MiB 1/1
store prod 0.05% 85.1 MiB 1/1
web prod 0.00% 3.0 MiB 1/1
Published ports
master/default/traefik traefik 80/tcp listen=node=@(service=traefik) nodes: home01 local, obs01 local
master/default/traefik traefik 443/tcp listen=node=@(service=traefik) nodes: home01 local, obs01 local
...
2 nodes, 10 deployments

OpenObserve listens on port 5080 inside its deployment. Forward that port to this machine and sign in with the admin user and the password page 00 set. shc forward holds the tunnel open until it is stopped, so run it in tmux.

home01
$ tmux new -d -s fwd shc forward -t master openobserve 5080=15080
home01
$ curl -s -u [email protected]:P@ssw0rd http://127.0.0.1:15080/healthz
{"status":"ok"}

A query is a SQL statement over a stream, sent as JSON. Metrics are one stream per series; this one asks the newest memory reading of web’s container. The time window is in microseconds since the epoch, wide enough to cover the walk.

~/learn/metrics-query.json
{
"query": {
"sql": "SELECT name, value FROM \"container_memory_usage_bytes\" WHERE name LIKE '%prod_prod_web%' ORDER BY _timestamp DESC",
"start_time": 1700000000000000,
"end_time": 2000000000000000,
"from": 0,
"size": 1
}
}
home01
$ curl -s -u [email protected]:P@ssw0rd --json @/root/learn/metrics-query.json 'http://127.0.0.1:15080/api/default/_search?type=metrics'
{"took":11,"took_detail":{"total":11,"cache_took":0,"file_list_took":0,"wait_in_queue":0,"idx_took":0,"search_took":9},"hits":[{"_timestamp":1791561157022238,"name":"prod_prod_web-nginx-r0-nginx-r0-1","value":3096576.0}],"total":1,"from":0,"size":1,"cached_ratio":0,"scan_size":1,"idx_scan_size":0,"scan_records":1630,"trace_id":"01a1215e57c97520be8f4f2e441bfc7a","is_partial":false,"result_cache_ratio":0,"work_group":"short","order_by":"desc","order_by_metadata":[["_timestamp","desc"]],"is_histogram_eligible":true,"peak_memory_usage":0.0}

Now make one change and read it three ways. Restart web; the daemon raises an event, nginx writes its startup lines, and the container’s memory reading starts again from a fresh process.

home01
$ shc restart web
restarting deployment "web"
✔ restarted deployment "web"
$ shc events ls -s web -T docker -n 2
2026-10-10T18:47:39Z home01 container prod_prod_web-nginx-r0-nginx-r0-1 exited with code 0, deployment prod/web/prod
2026-10-10T18:47:40Z home01 container prod_prod_web-nginx-r0-nginx-r0-1 started, deployment prod/web/prod
$ shc logs -s web -n 2
nginx-r0 | 2026/10/10 18:47:40 [notice] 1#1: start worker process 23
nginx-r0 | 2026/10/10 18:47:40 [notice] 1#1: start worker process 24

Container log lines land in the default logs stream, each stamped with the tenant, deployment and environment it came from. Ask for the newest line about web: it is the last startup line shc logs printed.

~/learn/logs-query.json
{
"query": {
"sql": "SELECT shc_tenant, shc_stack, body FROM \"default\" WHERE shc_stack = 'web' ORDER BY _timestamp DESC",
"start_time": 1700000000000000,
"end_time": 2000000000000000,
"from": 0,
"size": 1
}
}
home01
$ curl -s -u [email protected]:P@ssw0rd --json @/root/learn/logs-query.json 'http://127.0.0.1:15080/api/default/_search?type=logs'
{"took":22,"took_detail":{"total":22,"cache_took":0,"file_list_took":0,"wait_in_queue":0,"idx_took":0,"search_took":20},"hits":[{"_timestamp":1791561198675937,"shc_tenant":"prod","shc_stack":"web","body":"2026/10/09 15:53:18 [notice] 1#1: start worker process 23"}],"total":1,"from":0,"size":1,"cached_ratio":0,"scan_size":17,"idx_scan_size":0,"scan_records":22634,"trace_id":"01a1215e90827202aec8123a44243c8a","is_partial":false,"result_cache_ratio":0,"work_group":"short","order_by":"desc","order_by_metadata":[["_timestamp","desc"]],"is_histogram_eligible":true,"peak_memory_usage":2.0}

shc support bundle collects all of this into one archive for someone else to read: a dump of the telemetry store for the window --since names, capped at --max-records, the state snapshot, every in-scope deployment’s render with secrets left as references, the certificate inventory and the fabric state. Nothing is uploaded; --file is where it lands. --quick skips the telemetry dump.

home01
$ shc support bundle --since 1h --max-records 10000 --file /root/learn/bundle.tar.gz
✔ wrote support bundle "/root/learn/bundle.tar.gz", 56518 bytes
$ shc support bundle --quick --file /root/learn/bundle-quick.tar.gz
✔ wrote support bundle "/root/learn/bundle-quick.tar.gz", 56399 bytes
$ tar tzf /root/learn/bundle-quick.tar.gz
state/status.json
state/license.json
state/deployments.json
render/master/keycloak/default/compose.yaml
render/master/keycloak/default/manifest.json
render/master/openobserve/default/compose.yaml
render/master/openobserve/default/manifest.json
render/master/postgres/default/compose.yaml
render/master/postgres/default/manifest.json
render/master/traefik/default/compose.yaml
render/master/traefik/default/manifest.json
render/prod/cloud/prod/compose.yaml
render/prod/cloud/prod/manifest.json
render/prod/db/prod/compose.yaml
render/prod/db/prod/manifest.json
render/prod/pgweb/prod/compose.yaml
render/prod/pgweb/prod/manifest.json
render/prod/placed/prod/compose.yaml
render/prod/placed/prod/manifest.json
render/prod/store/prod/compose.yaml
render/prod/store/prod/manifest.json
render/prod/web/prod/compose.yaml
render/prod/web/prod/manifest.json
certs/inventory.json
fabric/wireguard.json
manifest.json

A container’s readings reach OpenObserve once a minute, so the reading taken right after the restart could still be the old process’s. Two minutes on, ask for web’s newest memory reading again.

home01
$ curl -s -u [email protected]:P@ssw0rd --json @/root/learn/metrics-query.json 'http://127.0.0.1:15080/api/default/_search?type=metrics'
{"took":10,"took_detail":{"total":10,"cache_took":0,"file_list_took":0,"wait_in_queue":0,"idx_took":0,"search_took":9},"hits":[{"_timestamp":1791561277018281,"name":"prod_prod_web-nginx-r0-nginx-r0-1","value":3072000.0}],"total":1,"from":0,"size":1,"cached_ratio":0,"scan_size":1,"idx_scan_size":0,"scan_records":1767,"trace_id":"01a1216085cf7ac180c7dd6612f0700b","is_partial":false,"result_cache_ratio":0,"work_group":"short","order_by":"desc","order_by_metadata":[["_timestamp","desc"]],"is_histogram_eligible":true,"peak_memory_usage":0.0}

The same restart is two docker events in shc events, two lines in shc logs, its startup line in the logs stream and a memory reading of the new process in OpenObserve.

An AI client reads the same cluster through shc mcp serve, which offers the daemon’s API as MCP tools on its standard input and output. The daemon’s own permissions apply to every call, and --read-only offers the reading tools alone: search, status, logs and the lists.

explained, not walked: the server speaks MCP on stdin and stdout until its client hangs up, so a shell has nothing to type at it
$ shc mcp serve --read-only

The admin password is the last thing to read here, because changing it is not a config change. OpenObserve reads modules.openobserve.vars.password once, when its store is made, so the row page 00 set no longer reaches the running store. shc system openobserve reset-admin-password does: it changes the password in the running store, then the vault record the collectors sign in with, then updates the deployment. --password names the new one; without it SHC makes one and prints it once. Typed on home01, the change is handed to obs01, which holds the store.

home01
$ shc system openobserve reset-admin-password --password N3w-P@ssw0rd
✔ changed the system openobserve admin password

The update replaced OpenObserve’s container, and the tunnel went to the old one. Open it again and ask for the list of log streams twice: the old password is refused and the new one signs in.

home01
$ tmux kill-session -t fwd
$ tmux new -d -s fwd shc forward -t master openobserve 5080=15080
home01
$ curl -s -o /dev/null -w '%{http_code}' -u [email protected]:N3w-P@ssw0rd 'http://127.0.0.1:15080/api/default/streams?type=logs'
200
$ curl -s -o /dev/null -w '%{http_code}' -u [email protected]:P@ssw0rd 'http://127.0.0.1:15080/api/default/streams?type=logs'
401

Put the example password back, so the pages after this one find the lab as page 00 left it, and close the tunnel.

home01
$ shc system openobserve reset-admin-password --password P@ssw0rd
✔ changed the system openobserve admin password
$ tmux kill-session -t fwd

This page leaves two query files and two archives under ~/learn on home01 and nothing else changed. The HUD tutorial starts from here.

Next: The HUD.