Upgrading SHC

There are two distinct upgrade paths, which you should not confuse:

This page is about upgrading SHC — the control plane — on one node or across a cluster. For stack updates, see the app update flow.

For a single-node install, the upgrade is essentially “install the new version on top of the old one and restart the daemon.” SHC runs its database migrations on startup.

cd shc
git pull
make deb
sudo dpkg -i ../shc_*.deb
sudo systemctl restart shc
journalctl -u shc -f      # confirm it comes back healthy

cd shc
git pull
make install
sudo systemctl restart shc

1. Stops the daemon (SIGTERM → graceful shutdown of subsystems).
2. New daemon process starts.
3. atlas migrate apply runs against the control-plane database to apply any pending schema migrations.
4. Subsystems initialize in dependency order (see Daemon manager).
5. Leader election runs; a new leader is elected if needed.
6. Status/health reconciliation loops start; the daemon picks up running containers and resumes management.

Running containers are not affected by the daemon restart. Docker keeps containers running; SHC reattaches on restart via container labels.

Every node in the cluster should run the same SHC version. Version skew across nodes is not supported — the replicated database and the internal API assume matching wire/schema on all nodes.

Upgrade one node at a time, verifying each is healthy before moving on.

# On node 1
sudo systemctl stop shc       # or `shc node drain` to migrate stacks first
# install new version
sudo systemctl start shc
shc cluster status             # wait for HEALTHY before continuing

Two important concerns during rolling upgrade:

1. Leader continuity. The leader may be the node you’re upgrading. Raft will elect a new leader automatically; in-flight leader-only operations (scheduled job ticks, retention pruning) may be paused until election settles (typically < 5s).
2. Raft quorum. Don’t stop a majority at once. With 3 nodes, never have more than 1 offline. With 5 nodes, at most 2. The shc cluster status output shows quorum health.

For a cleaner upgrade where the node takes no user traffic while being restarted:

shc node drain <node-id>
# Wait for stacks to reschedule (or migrate them with `shc move`).
sudo systemctl stop shc
# ... install and start new version ...
sudo systemctl start shc
shc node undrain <node-id>

Drain sets the node’s schedulable flag to False; the scheduler stops placing new stacks there. Existing swarm-mode services reschedule automatically; compose-mode stacks pinned to this node stay put and become unavailable during the restart.

SHC uses Atlas for schema migrations. They run automatically on daemon start. Behavior:

• Forward migrations are applied idempotently on startup (the daemon invokes the embedded atlas runner against migrations/*.sql).
• Down migrations are not supported. Rollback strategy is “backup first, forward-only in production.”
• Migration failures abort daemon startup. The daemon fails fast rather than running against an inconsistent schema. Logs show the exact Atlas error.

If a migration fails on one node of a cluster:

1. Stop the new-version daemon there.
2. Restart the old version on that node.
3. Investigate why the migration failed (check migrations/ for the specific revision and atlas.sum for the integrity hash).
4. Fix the DB state manually or file a bug before retrying.

Never upgrade a cluster past a failed migration on one node.

For a production cluster, always take a backup of the control plane state before a major upgrade. The fastest way:

shc cluster backup-state --output /tmp/shc-pre-upgrade.bak

This captures the control-plane database, the cluster CA/certs, and the node config. It’s an operator-level recovery artifact, not an app backup. See the backup module for the full backup model and the backup/restore flow for stack-level backups.

• Patch (x.y.Z): safe to roll forward in any order. No schema or wire-format changes.
• Minor (x.Y.z): may add columns, endpoints, or events. Older clients still work; mixed-version clusters should not run for long.
• Major (X.y.z): may break CLI/API/config compatibility. Read the release notes before upgrading.

Check the current version before and after:

shc --version
shc cluster nodes      # shows the version reported by each node

If the new version misbehaves and you need to roll back:

1. Stop the new daemon on all nodes.
2. Restore the pre-upgrade backup (see above) if you took one and if migrations ran.
3. Install the old package version on each node.
4. Start the daemons in any order once the DB is back to the old schema.

If migrations ran and you don’t have the pre-upgrade backup, the Alembic down-migration may or may not work. File a bug with the maintainers before attempting anything destructive.

• App update flow — upgrading a stack, not SHC
• Daemon manager — what happens at startup
• Cluster module — cluster state and recovery
• Installation — pick an install method