Restoring the whole cluster
This page assumes the app ladder has run on this cluster, so it holds web, db, pgweb, placed, store and cloud. It backs them all up into archives outside the cluster, wipes every machine’s SHC state, builds a fresh cluster on the same four machines and recovers every app from the archives. The platform’s own deployments are not backed up: a fresh init makes them again.
The app ladder’s pages 01 to 07 left placed held to learn/home. Move it
to learn/far first, as that ladder’s node-failure page does, so the
restore below brings back a deployment whose rule keeps it to one node.
$ shc update placed --where location=learn/far --yes...$ shc ls...Back everything up
Section titled “Back everything up”cloud reads db and store through its plugs, and pgweb reads db. A backup
of one of them alone is a database without its app, or an app without its
database. --all-connected backs up the whole integration graph at one
point in time, each member into the repository, and writes one group
archive of all of them under /var/lib/shc/exports. The archive is in
the clear, so that a cluster with a different key can read it.
$ shc backup create cloud --all-connected...$ shc backup ls -s cloud --limit 1...Export the group as one archive: -s cloud takes cloud’s latest
backup, the row above, and --with-connected adds every member of its
group. /srv/handover is
outside SHC’s state, so the wipe below leaves it alone; on a real cluster
copy the directory off the machine.
$ shc backup export -s cloud --with-connected --file /srv/handover/cluster.bkg...placed and web stand alone. Seal each backup’s secrets under a password of your own, so that a cluster whose vault differs can open them, and write the latest one out as a file named for its deployment.
$ shc backup create placed --secrets-password P@ssw0rd...$ shc backup export -s placed --latest --file /srv/handover/placed.bak/srv/handover/placed.bak$ shc backup create web --secrets-password P@ssw0rd...$ shc backup export -s web --latest --file /srv/handover/web.bak/srv/handover/web.bak$ ls -1 /srv/handovera1.bakcluster.bkgplaced.bakweb.baka1.bak is the repositories page’s leftover; it stays, unused.
placed is limited to learn/far, and its archive carries that rule. A
recover lands a deployment on the node it is typed on, so placed’s
archive has to be on far01 too. Write it there as well. No tenant is
selected on far01, so the export names prod with -t.
$ mkdir -p /srv/handover$ shc backup export -s placed --latest -t prod --file /srv/handover/placed.bak/srv/handover/placed.bakThe app ladder’s app directories under ~/learn on home01 and the notes
app under /usr/share/shc/apps are files of yours, not SHC state. They
stay too.
Wipe the machines
Section titled “Wipe the machines”shc node leave --purge takes a node out of the cluster and destroys
every bit of SHC state on it: the database, the certificates, every
deployment’s containers, networks and volumes. Do the three joined nodes
first, then home01, which is by then a cluster of one.
$ shc node leave --purge --force --yes...$ shc node leave --purge --force --yes...$ shc node leave --purge --force --yes...$ shc node leave --purge --force --yes...A leave that fails part-way is picked up where it stopped with
--resume; none fails here, so it is not typed.
$ shc node leave --resume --yesNothing of the cluster is left on any machine. The archives are.
$ ls -1 /srv/handovera1.bakcluster.bkgplaced.bakweb.bakBuild the cluster again
Section titled “Build the cluster again”The same four nodes, the same names, locations and labels, the same vault
password, this time read from files only root can read. --force would
wipe a node that still held a cluster; this one holds none.
--modules-config names the platform modules, here the four the daemon
installs by default. On these two-CPU machines init skips openobserve, as
it did on the first init. placed’s rule names learn/far
and comes back with it, so far01 must be there again.
P@ssw0rdP@ssw0rd$ chmod 600 ~/vault-password ~/backup-password$ shc init --name home01 --cloud learn --location home --label tier=core --force --modules-config '{"postgres":{},"keycloak":{},"openobserve":{},"traefik":{}}' --vault-password-file ~/vault-password --backup-password-file ~/backup-password --yes...$ shc license install ~/shc-license.tokenlicence installedstate: keyedtier: pro...$ ssh home01 shc node token --ttl 30m --fields token -o text | shc node join - --name home02 --cloud learn --location home --control-plane✔ joined cluster as node "home02" at 192.168.232.12$ ssh home01 shc node token --ttl 30m --fields token -o text | shc node join - --name home03 --cloud learn --location home --control-plane✔ joined cluster as node "home03" at 192.168.232.13$ ssh home01 shc node token --ttl 30m --fields token -o text | shc node join - --name far01 --cloud learn --location far --label tier=edge✔ joined cluster as node "far01" at 192.168.232.14$ shc node lsNAME STATE CONDITION AVAIL CLOUD LOCATION ROOM BUILD STORE LEADER VOTERfar01 healthy ready active learn learn/far ok 0.1.47 replica no nohome01 healthy ready active learn learn/home ok 0.1.47 voter yes yeshome02 healthy ready active learn learn/home ok 0.1.47 voter no yeshome03 healthy ready active learn learn/home ok 0.1.47 voter no yes$ shc ls -t masterSTACK TENANT ENV APP STATUS NODESkeycloak master default keycloak running home01postgres master default postgres running home01traefik master default traefik running far01, home01, home02, home03The fresh cluster holds master alone, and home01’s context went with the
old cluster’s state. Add prod again and select it, so the recovered
deployments land in it. The catalogue apps come from the public repository,
which shc init registered again as shc. The backup repository is set
again too, at global scope as on the repositories page, so that the
recovered deployments can be backed up from here on.
$ shc tenant add prod --no-admin✔ created tenant "prod"warning: keycloak realm provisioning still in progress$ shc ctx -t prod...$ shc repo ls...$ shc config set backup.repo /var/backups/shc/repo --scope globalbackup.repo is a path: unless every node mounts the same storage at /var/backups/shc/repo, each of this cluster's 4 nodes keeps its own repository there. A backup is then on the node that took it alone and is lost with that nodeKEY VALUE SCOPEbackup.repo /var/backups/shc/repo globalRecover every app
Section titled “Recover every app”shc recover installs deployments from archives, with their variables,
their integrations and their data. The archives were sealed by the old
cluster’s vault, so give it that vault’s password, on the command line or
from the file. --dry-run prints the plan and does nothing: --rename
would bring a deployment back under another name, --override under
another ingress host, --skip-integration without one of its plugs, and
--runtime and -e name the runtime and environment it lands in, here
the ones it had. Read the plans, then recover as it was.
$ shc recover -b /srv/handover/web.bak --rename web=web2 --runtime docker --dry-run...$ shc recover -f /srv/handover/cluster.bkg --secrets-password P@ssw0rd --override cloud:hosts=cloud.learn.example --skip-integration s3 --dry-run...$ shc recover -f /srv/handover/cluster.bkg --secrets-password-file ~/vault-password --dry-run...$ shc recover -f /srv/handover/cluster.bkg --secrets-password-file ~/vault-password --yes...$ shc recover -b /srv/handover/web.bak --secrets-password P@ssw0rd --yes...placed’s archive carries its rule, where location=learn/far. A recover is
placed the way an install is, and home01, where it is typed, is not in
learn/far, so placed lands on far01, the node the rule keeps: the archive
goes there and its data replays there. The dry run names the node.
$ shc recover -b /srv/handover/placed.bak --secrets-password P@ssw0rd --dry-run...$ shc recover -b /srv/handover/placed.bak --secrets-password P@ssw0rd --yes...Three forms of shc recover are not typed on this ladder.
--consistency-id recovers a group straight from the repository, with
no archive, which a wiped cluster no longer has. --storage puts every
recovered deployment on one storage backend instead of the one each
backup recorded. -e names the environment the deployments land in.
$ shc recover --consistency-id 20261008T170012Z_5c1d3e4f5a6b7c8d --storage local -e prod --yesA control plane restored from an older backup meets nodes that hold a
newer peer table and refuse it; shc cluster confirm-restore makes the
restored one authoritative. This cluster was rebuilt, not restored, so it
is not typed.
$ shc cluster confirm-restore --yesRun shc ls until every deployment is running.
$ shc lsSTACK TENANT ENV APP STATUS NODEcloud prod prod nextcloud running home01db prod prod postgres running home01pgweb prod prod /root/learn/pgweb running home01placed prod prod notes running far01store prod prod /root/learn/versitygw running home01web prod prod /root/learn/nginx running home01...$ shc integration ls...placed is on far01 again, and the rule that keeps it there came back with it.
$ shc constraint explain -s placed...The data came with them. nginx serves its page, placed has its row, and Nextcloud has the file the app ladder wrote.
$ curl -s http://localhost:18080/<!doctype html><title>hello</title><p>Hello from SHC.</p>$ shc exec -s placed -S postgres -- psql -U postgres -tAc 'select body from notes'kept by the backup$ curl -s -u admin:P@ssw0rd http://localhost:8082/remote.php/dav/files/admin/hello.txtHello from SHC.The cluster is as the app ladder left it, on machines that held nothing an hour ago.
Next: The daemon’s settings.