Skip to content
SHC Docs

Upgrading SHC

There are two distinct upgrade paths, which you should not confuse:

What you’re upgradingUse
A deployment (an app installed on SHC)shc update <deployment>
SHC itself (the daemon binary / package)This page

This page is about upgrading SHC, the control plane, on one node or across a cluster, within one release line. For deployment updates, see the app update flow.


From the old line to the new line: no in-place upgrade

Section titled “From the old line to the new line: no in-place upgrade”

You can’t install the new line (dev, v0.1) over the old one (main and release/v0.0, the Swarm and Nebula line) and restart. The new line reset its migration history: no migration file is common to the two lines. Before any SQL runs, shcd reads its _shc_migrations ledger, finds the old line’s rows in no catalogue it ships, and refuses to boot with X409990MIGRST (“this store’s ledger records … no catalogue in this build ships …; reinstall from this release, nothing migrates the store”, ce core/db/migrate.go AssertLedgerCovered). That refusal is the design (the reset doctrine), not a bug to work around.

The path is a fresh cluster on the new release, then each app’s data brought across: through the app’s own export (a database dump, the app’s export feature), or through an old-line backup archive whose manifests were converted to the new line’s shape first. The new line never reads an old-line manifest as is. The bitspur estate’s move is planned that way, app by app (SHC-1061, the plan). Which line a branch belongs to is in release lines.


A reset line is a release whose store a cluster has to be formed from. v0.1.14 squashed the store’s migrations into one file again (SHC-3098) and hands the fabric out in /26 chunks. v0.1.21 squashed them again after the store’s convention pass (SHC-3446), and is the line today. A node records the line it was formed on as reset_line in node_bootstrap.json: shc init writes its build’s line, shc node join the leader’s.

Nothing upgrades a node across a reset line. An estate formed before v0.1.21 is re-formed, not upgraded. On a node formed before the line:

  • the package’s preinst refuses it before dpkg unpacks anything, so the installed package and its running daemon stay as they were and apt ends with an error;
  • shcd, started on it anyway (a reboot), stops before anything starts and exits 65, which shc.service does not restart.

Both print one line, X409992PRELIN, naming what the node holds from before the line: no reset_line or another one, a fabric chunk that is not a /26, or ledger rows of migrations the line squashed. shcd start-check asks the same question and changes nothing.

v0.1.28 squashed the store’s migrations into one file again without a reset, and v0.1.28 through v0.1.38 adopted a store from v0.1.21 through v0.1.27 the first time shcd started on them. Later builds don’t: a store that never started on one of those releases still records the files the squash replaced, and the deb and shcd refuse it with X409992PRELIN, which then says to start shcd once on a release from 0.1.28 through 0.1.38:

shc: [X409992PRELIN] this node was formed on this build's v0.1.21 reset line, its store's ledger records 1 migration that 0.1.28's squash replaced, 20261018013510_auto.sql; start shcd once on a release from 0.1.28 through 0.1.38, then upgrade (https://selfhostedcloud.org/guide/operations/upgrades/#reset-lines)

Take such a node to one of those releases first and start shcd there once, then upgrade. A store that also holds any other row from before the line, or a node whose bootstrap or fabric chunks are off the line, is still told to re-form.

To re-form, take each app’s data out first, through its own export or a backup the new line restores. Then form the cluster again on hosts that hold nothing of the old one: reinstall them, or stop and remove the old cluster’s containers and networks, since shc init refuses a host where the old fabric’s addresses are still up (X400450ADROVL). Install this release’s package there: its preinst refuses it on a host that still holds the old node’s node_bootstrap.json or store. Run shc init --force on the first node and shc node join --force <token> on every other one, install the apps and bring their data back.

Everything below is an upgrade within one line.


Within one line, a single-node upgrade is “install the new version on top of the old one and restart the daemon,” as long as the release leaves the store’s migration file as it was; one that changes it refuses the old store (see the store’s migration).

Terminal window
cd shc
git pull
make deb
sudo dpkg -i ../shc_*.deb # the package restarts the daemon itself
journalctl -u shc -f # confirm it comes back healthy

The package’s configure step restarts a running daemon once the operations it runs have finished (shcd roll-wait, 30 minutes at most), and starts the daemon of an initialised node that is stopped. A systemctl restart shc after it restarts the daemon a second time. If an operation still runs after 30 minutes, the package keeps the old daemon running and names the operation; restart shc yourself once it ends.

Terminal window
cd shc
git pull
make install
sudo systemctl restart shc
  1. Stops the daemon (SIGTERM → graceful shutdown of subsystems).
  2. New daemon process starts.
  3. shcd applies any pending migration from the files built into it (ce core/db/migrate.go); no atlas binary runs. In a cluster the store is replicated, so only the first node that starts on a new build finds a numbered migration to apply.
  4. Subsystems initialize in dependency order (see Daemon manager).
  5. Leader election runs; a new leader is elected if needed.
  6. Status/health reconciliation loops start; the daemon picks up running containers and resumes management.

The runtime keeps containers running during a brief daemon restart. The independent watchdog stops containers labelled shc.single_writer if the node’s lease expires. See the watchdog.


Nodes take a new build one at a time, so for the length of a roll some nodes run the new build and some the previous one. Within one line that works: every numbered migration is additive, so the previous build runs on the migrated store, and the deployments keep serving through it (see the store’s migration). Finish the roll; don’t leave a cluster on two builds.

Upgrade one node at a time, the nodes that don’t vote first and the leader last (shc cluster status names the leader), and wait for each to read healthy before moving on:

Terminal window
# On each node in turn
sudo systemctl stop shc # or `shc node drain` to move deployments off first
sudo dpkg -i shc_<version>_amd64.deb # the package starts shc again
systemctl is-active shc # active
shc cluster status # wait for "nodes: N healthy of N" and "quorum: healthy"
shc node ls # BUILD shows which nodes run the new build

dpkg -i starts the daemon itself: its configure step starts an initialised node’s daemon that is stopped, so a systemctl start shc after it changes nothing. Skip the systemctl stop and the package restarts the running daemon instead, once the operations it runs have finished. The first node that starts on a build carrying a numbered migration applies it, once, and every node after it finds it applied.

The terraform provider rolls in the same order when package.version changes, gates each node on shc node ls, quorum and shc doctor run, and stops at the first node that fails. See the provider reference.

Once every node runs the new build, the leader finishes each deployment’s stored render again with what the new build computes, and rolls only the replicas whose documents changed, a few deployments at a time. Nothing needs redeploying. shc doctor run shows it under app.refinish: the nodes it still waits for, what of a node the new leader cannot read yet (its runtime or its fabric address, for a minute or so after it starts), and any deployment it could not re-finish.

Two important concerns during rolling upgrade:

  1. Leader continuity. The leader may be the node you’re upgrading. Raft will elect a new leader automatically; in-flight leader-only operations (scheduled job ticks, retention pruning) may be paused until election settles (typically < 5s).
  2. Raft quorum. Don’t stop a majority at once. With 3 nodes, never have more than 1 offline. With 5 nodes, at most 2. The shc cluster status output shows quorum health.

Draining before upgrade (stacks migrate off)

Section titled “Draining before upgrade (stacks migrate off)”

For a cleaner upgrade where the node takes no user traffic while being restarted:

Terminal window
shc node drain <node-id>
# Wait for stacks to reschedule (or relocate them with `shc update --where`).
sudo systemctl stop shc
sudo dpkg -i shc_<version>_amd64.deb # the package starts shc again
shc node uncordon <node-id>

Drain asks SHC’s placement engine for eligible targets and reports each relocation. A deployment with no eligible target refuses the drain unless --force permits stopping it in place. Check the drain result before stopping shcd.

shc node uncordon returns the node to service from any state. Nothing moves back: the node takes work as capacity is needed. To stop a drain and keep the node closed, shc node undrain <node-id> leaves it cordoned and says how many deployments had already moved.


Until v1, SHC’s store is built by one generated migration file (SHC-2197), the ce module’s migrations/, shipped inside shcd with its atlas.sum and tracked in the _shc_migrations ledger. Behavior:

  • A fresh store is built from the file on the first start.
  • A store made from the same file starts as it is. Nothing applies.
  • A store made from another file is refused at start, before any SQL runs: X409990MIGRST when the store’s file is older than the build’s (“reinstall from this release, nothing migrates the store”), X409991MIGNEW when it is newer. Until a release ships the file, a schema change regenerates it, and that is no in-place upgrade: install the new build on a fresh cluster and bring each app’s data back from its backups.
  • Once a release ships the file it is frozen, and a schema change is a numbered migration on top of it. The first node that starts on the new build applies it once, as one atomic batch the store’s leader executes, and the cluster keeps working while the other nodes still run the previous build.
  • Every numbered migration is additive (a new table, a new nullable or defaulted column, a new index), so a node still on the previous build runs on the new schema for the length of a roll. Removing or reshaping what the previous build reads takes two releases: the first stops reading it, the second removes it and names the first in a -- shc:last-reader-removed-in vX.Y.Z line.
  • Down migrations do not exist. The loader skips .down.sql outright. Rollback strategy is “backup first, forward-only in production.”
  • In cluster mode DDL is raft-replicated, so the file the first node applies builds the store for every node. Wiping and rejoining a node replays it rather than escaping it.
  • A failure applying the file aborts daemon startup. The daemon fails fast rather than running against an inconsistent schema, and the log shows the error.
  • A node formed before the build’s reset line logs the one X409992PRELIN line and exits 65 before any subsystem starts, and the package’s preinst refuses it first (see Reset lines).
  • An older build on a store a newer build migrated logs the one X409991MIGNEW line and exits 78 before any subsystem starts, so the store engine, the vault and the container runtime never come up on it. shc.service does not restart it on that code: the node stays down until it runs the newer build.

Never upgrade a cluster past a failed start on one node.


For a production cluster, confirm the control-plane snapshot is current before a major upgrade. There is no operator verb that takes one on demand: the leader-only db_snapshot_cron writes a consistent dump of the control-plane database to {state}/snapshots/control-plane/ every 24 hours (raft-consistent in cluster mode, VACUUM INTO single-node), and that dump is the operator-level recovery artifact. Check its timestamp on the leader before you upgrade:

Terminal window
shc db snapshot ls # the newest nightly, and any taken on demand
shc db snapshot create # take one now, on the leader

If the newest snapshot is older than the change you are about to make, take one with shc db snapshot create before proceeding. Retention is maintenance.control_plane_snapshot_retention (default 7) for each kind on its own, so an on-demand snapshot never prunes a nightly. This is control-plane state, not app data; take stack backups separately. See the backup module for the full backup model and the backup/restore flow for stack-level backups.


  • Patch (x.y.Z): safe to roll forward in any order. No schema or wire-format changes.
  • Minor (x.Y.z): may add columns, endpoints, or events. Older clients still work; mixed-version clusters should not run for long.
  • Major (X.y.z): may break CLI/API/config compatibility. Read the release notes before upgrading.

Check the current version before and after:

Terminal window
shc version
shc node ls # the BUILD column is the shcd each node runs

See Rolling a daemon back for the full procedure. The summary:

  1. Decide whether the roll crossed a migration. git diff OLD..NEW -- go.mod go.sum in the shc checkout: empty means the ce pin did not move, the migration file did not change, and the roll is reversible by binary alone.
  2. If it did not: install the old deb node by node, primary last, waiting for each daemon to answer before the next. dpkg -i accepts an older version silently.
  3. If it did: restore the control-plane snapshot first, then install the old deb. There are no down-migrations, and none can be written after the fact.

Two things to know before you rely on any of this:

  • dpkg does not keep the deb it replaced. A release’s deb can be fetched again, from https://downloads.selfhostedcloud.org/shc/<ver>/ or with apt install shc=1:<ver> on a node that reads the Based apt repo. A snapshot build is in neither, so save the outgoing deb before you roll or your rollback artifact will not exist.
  • A daemon downgraded across a migration refuses to boot. Its ledger holds a migration file newer than any the older build ships, so shcd stops with X409991MIGNEW and exits 78 before any subsystem starts (ce core/db/ledger_newer.go, read from the node’s own copy of the store). That’s why step 3 restores the snapshot first: the snapshot’s ledger predates the migration.