Underneath a node failing
Read this right after A node failing. Every command here reads; none changes the cluster.
The leader logs by node id. far01’s is in the first column here:
$ shc node info far01ID HOSTNAME RUNTIME AVAILABILITY INTERNAL ADDRESS UNDERLAY ADDRESS PUBLIC ADDRESS CLOUD LOCATION IS VOTER FABRIC GRANT FABRIC TENANT SLOTS FABRIC FULLEST BRIDGEnode-3845076fd5b8 far01 docker active 10.192.0.4:4003 65.21.55.73 65.21.55.73 learn learn/far false 10.128.64.0/20 2 of 31 master, 3 of 125 addressesThe leader’s journal for the last half hour, the lines about losing far01,
holding its deployments and getting it back. If far01 joined less than half
an hour ago, the window also holds a short unavailable and reachable pair
from just after the join, when home01’s daemon restarted to replicate the
cluster’s database; the ... stands for it.
$ journalctl --no-pager -u shc --since '30 min ago' -o cat -g 'X503014NODNAV|V200872FOVHLD|node reachable again'...time=2026-10-09T20:22:59.039Z level=WARN msg="node node-729da85a624c unavailable" code=X503014NODNAV http_status=503 node_id=node-729da85a624ctime=2026-10-09T20:23:26.772Z level=WARN msg="master/traefik/default on lost node node-729da85a624c is held: none: it stays assigned and starts when its node returns; traefik: local volume static" code=V200872FOVHLD deployment=master/traefik/default from=node-729da85a624c movability=pinned action=held detail="none: it stays assigned and starts when its node returns; traefik: local volume static"time=2026-10-09T20:23:26.772Z level=WARN msg="prod/placed/prod on lost node node-729da85a624c is held: none: it stays assigned and starts when its node returns; postgres: local volume pgdata" code=V200872FOVHLD deployment=prod/placed/prod from=node-729da85a624c movability=pinned action=held detail="none: it stays assigned and starts when its node returns; postgres: local volume pgdata"time=2026-10-09T20:23:43.094Z level=INFO msg="node reachable again" node=node-729da85a624c suppressed=21 was_unavailable_for=44.054416917sThe leader marked far01 unavailable when it stopped answering. A deployment on
a lost node moves only when its data can go with it. placed keeps its data
in a local volume on far01, and so do far01’s copy of traefik and any other
database the scheduler had put there, so moving one would start it empty on
another node. SHC held each where it was, and they ran again when far01 came
back.
Next: Render once on the leader.