Cordon, drain, undrain, uncordon
A node’s availability says whether it takes work. active takes it;
cordoned takes nothing new and keeps what it has; draining is being
emptied; drained is empty. Nothing moves by itself: every change below is
typed.
Cordon home02. A new install lands elsewhere; a1, a2 and a3 are untouched.
$ shc node cordon home02warning: node home02 is cordoned; this binds SHC's own placement only: it is excluded from install-time scoring and from drain relocation targets, but a compose deploy typed against it directly with `shc install` on that node or `--node home02` still lands there✔ cordoned node "home02"; nothing running on it was touched$ shc node lsNAME STATE CONDITION AVAIL CLOUD LOCATION ROOM BUILD STORE LEADER VOTERfar01 healthy ready active learn learn/far ok 0.1.47 replica no nohome01 healthy ready active learn learn/home ok 0.1.47 voter yes yeshome02 healthy cordoned cordoned learn learn/home ok 0.1.47 voter no yeshome03 healthy ready active learn learn/home ok 0.1.47 voter no yes$ shc install ~/learn/cluster/hello -s a6 --where '@learn/home' --label owner=learninstalling deployment "a6"[a6] Healthy 1/1✔ nginx healthy✔ installed deployment "a6"$ shc ls --where stack=a6STACK TENANT ENV APP STATUS NODEa6 prod prod /root/learn/cluster/hello running home03$ shc node uncordon home02✔ node "home02" is schedulable againDrain home03. Each deployment moves to the node its rules allow, one at a
time; a3 prefers rack=r1 but any node in learn/home will do. The node
ends drained.
$ shc node drain home03 --yeshome03 is a raft voter in slot 2 of 3 and a docker-runtime host. - draining does NOT change raft membership: home03 stays a configured voter and keeps counting toward quorum, so fault tolerance is unchanged: it is the workload that moves, not the vote - 2 deployments must relocate; the other docker-runtime nodes are far01, home01, home02 - traefik runs on every node, so its replica here stopswarning: drain of home03 relocated a3, a6 BRING-UP-FIRST: each mounts no volume, so its copy on the target was up and ready before the one on home03 stopped, and it was never down✔ drained node "home03"$ shc ls --where owner=learnSTACK TENANT ENV APP STATUS NODEa1 prod prod /root/learn/cluster/hello running home01a2 prod prod /root/learn/cluster/hello running far01a3 prod prod /root/learn/cluster/hello running home02a6 prod prod /root/learn/cluster/hello running home02$ shc node lsNAME STATE CONDITION AVAIL CLOUD LOCATION ROOM BUILD STORE LEADER VOTERfar01 healthy ready active learn learn/far ok 0.1.47 replica no nohome01 healthy ready active learn learn/home ok 0.1.47 voter yes yeshome02 healthy ready active learn learn/home ok 0.1.47 voter no yeshome03 healthy drained drained learn learn/home ok 0.1.47 voter no yesUncordon returns the node to service. Nothing moves back. a3 returns when its rule says so.
$ shc node uncordon home03✔ node "home03" is schedulable again$ shc update a3 --where node=home03 --yesrelocation plan for prod/a3/prod placement: home03 is the only node its rules, condition and capacity leave replicas: nginx r0 moves home02 -> home03 volumes: none durable data: 0 B to carry downtime: about 30s✔ relocated deployment "a3" to home03$ shc ls --where stack=a3STACK TENANT ENV APP STATUS NODEa3 prod prod /root/learn/cluster/hello running home03$ shc update a3 --where '@learn/home' --prefer rack=r1 --yesupdating deployment "a3"[a3] Healthy 1/1✔ nginx healthysource: /root/learn/cluster/hello, replayed from the deployment's persisted source (path)the rollout of prod/a3/prod to v1:sha256:78fdce16baee809f341064f02397eb1080475fa44fe37b5be8e18f3d45082456 finishedUndrain stops a drain and leaves the node cordoned. --timeout caps the
whole run in seconds.
$ shc node drain home02 --timeout 600 --yeshome02 is a raft voter in slot 3 of 3 and a docker-runtime host. - draining does NOT change raft membership: home02 stays a configured voter and keeps counting toward quorum, so fault tolerance is unchanged: it is the workload that moves, not the vote - 1 deployment must relocate; the other docker-runtime nodes are far01, home01, home03 - traefik runs on every node, so its replica here stopswarning: drain of home02 relocated a6 BRING-UP-FIRST: each mounts no volume, so its copy on the target was up and ready before the one on home02 stopped, and it was never down✔ drained node "home02"$ shc node undrain home02✔ the drain of node "home02" is stopped, 1 deployment had moved off it and 1 was stopped in place; the node is cordoned$ shc node lsNAME STATE CONDITION AVAIL CLOUD LOCATION ROOM BUILD STORE LEADER VOTERfar01 healthy ready active learn learn/far ok 0.1.47 replica no nohome01 healthy ready active learn learn/home ok 0.1.47 voter yes yeshome02 healthy cordoned cordoned learn learn/home ok 0.1.47 voter no yeshome03 healthy ready active learn learn/home ok 0.1.47 voter no yes$ shc node uncordon home02✔ node "home02" is schedulable again--plan asks where a drain would put each stack and changes nothing. It
prints a row per stack with what the drain would do, the node it would go
to and what blocks it. Ask it about far01.
$ shc node drain far01 --plan...a2 may only run on tier=edge, and far01 is the only such node, so a
drain of far01 would have nowhere to put it and would refuse. Move a2 home
first. The drain then has only traefik’s global replica, which stops in
place; uncordon starts it again, and a2 goes back to the edge.
$ shc update a2 --where '@learn/home' --yes...$ shc node drain far01 --yes...$ shc node ls...$ shc node uncordon far01✔ node "far01" is schedulable again$ shc update a2 --where tier=edge --yes...$ shc ls --where stack=a2...--force stops in place what no other node can take, a deployment pinned
to the node included, instead of refusing. a2 is pinned to the edge, so a
forced drain of far01 stops it there, and it stays stopped after the
uncordon.
$ shc node drain far01 --force --yes...$ shc ls --where stack=a2...$ shc node uncordon far01✔ node "far01" is schedulable again$ shc ls --where stack=a2...A maintenance window is a cordon with an announced return: the scheduler
does not fail over the node’s work while it is away inside the window.
--drain empties it first; --end lifts the window and leaves the node
where the drain put it, so uncordon it.
$ shc node maintenance home03 --window 15mwarning: node home03 is cordoned; this binds SHC's own placement only: it is excluded from install-time scoring and from drain relocation targets, but a compose deploy typed against it directly with `shc install` on that node or `--node home03` still lands there✔ node "home03" is in maintenance until 2026-11-06T17:46:12Z, availability cordoned: nothing is rescheduled from it before then$ shc node maintenance home03 --end✔ maintenance on node "home03" lifted, availability active$ shc node maintenance home03 --window 15m --drain --yeshome03 is a raft voter in slot 2 of 3 and a docker-runtime host. - draining does NOT change raft membership: home03 stays a configured voter and keeps counting toward quorum, so fault tolerance is unchanged: it is the workload that moves, not the vote - 2 deployments must relocate; the other docker-runtime nodes are far01, home01, home02 - traefik runs on every node, so its replica here stopswarning: drain of home03 relocated a3, a6 BRING-UP-FIRST: each mounts no volume, so its copy on the target was up and ready before the one on home03 stopped, and it was never down✔ drained node "home03"✔ node "home03" is in maintenance until 2026-11-06T17:46:45Z, availability drained: nothing is rescheduled from it before then$ shc node maintenance home03 --endwarning: node home03 is still drained: `shc node uncordon home03` returns it to service✔ maintenance on node "home03" lifted, availability drained$ shc node uncordon home03✔ node "home03" is schedulable againRun shc ls once more: a2 is still stopped where the forced drain left
it, the others are running, a3 and a6 where the last move put them, and
every node is active.
$ shc ls --where owner=learnSTACK TENANT ENV APP STATUS NODEa1 prod prod /root/learn/cluster/hello running home01a2 prod prod /root/learn/cluster/hello stopped far01a3 prod prod /root/learn/cluster/hello running home02a6 prod prod /root/learn/cluster/hello running home02$ shc node lsNAME STATE CONDITION AVAIL CLOUD LOCATION ROOM BUILD STORE LEADER VOTERfar01 healthy ready active learn learn/far ok 0.1.47 replica no nohome01 healthy ready active learn learn/home ok 0.1.47 voter yes yeshome02 healthy ready active learn learn/home ok 0.1.47 voter no yeshome03 healthy ready active learn learn/home ok 0.1.47 voter no yesKeep a1, a2, a3 and a6 for the next page.
Nothing moves back on its own after a drain. shc rebalance --plan prints
the moves that would even out the load, with each one’s explain, and
changes nothing; --apply would run them.
$ shc rebalance --plan...--apply runs the plan’s moves after a confirmation --yes skips. The
next pages count on the deployments staying where these pages put them, so
it is not typed.
$ shc rebalance --apply --yesNext: Nodes failing and rejoining.