Skip to content
SHC Docs

Losing a location

A location is lost when every node of one location label is gone and is not coming back: the site burned, the provider will not return the hardware, or you have stopped waiting. Its deployments come back from their off-site archives into a surviving location. You lose whatever changed since each deployment’s last archive, so the recovery point is the backup schedule.

This page assumes the control plane survived. If the lost location held a majority of the voters, shc cluster status says so, and Disaster recovery comes first: restore quorum, then come back here.

  • Every location label names an off-site repo, one that lives outside that location. A deployment’s archives go to its location’s repo unless a narrower scope sets backup.repo. shc label ls -v shows the BACKUP-REPO column; set a missing one with shc label set location=<provider>/<location> --backup-repo <repo>.
  • The mirror copies each of those repos to a second provider every night, at <backup/copy_repo>/locations/<provider>/<location> (a location with no provider sits under _). See 6b in Enable backups. bitspur’s second provider is Backblaze B2.
  • Every deployment is on a schedule. A deployment with no archive has no recovery.

1. Confirm the location is gone, and list what ran there

Section titled “1. Confirm the location is gone, and list what ran there”

shc ls lists only the environment $SHC_ENVIRONMENT or the context (shc ctx -e) selects, even with --all-tenants, so clear both before the inventory or the lost deployments of every other environment never reach the list. shc ctx shows the context; when it names an environment, reset it, and select the cluster again if it named one:

Terminal window
unset SHC_ENVIRONMENT
shc ctx
shc ctx --reset
shc ctx --cluster bitspur
Terminal window
shc node ls
shc cluster status
shc label show location=gorillaservers/ogden
shc ls --all-tenants > ogden-deployments.txt

Every node of the location reads lost in the CONDITION column. shc cluster status shows the voters against the control-plane fence; if fewer than a majority are healthy, stop and go to Disaster recovery.

shc label show names the location’s member nodes. The deployments to restore are the rows of shc ls --all-tenants whose node, or any entry of whose nodes, is one of those members. The label value’s own deployment list is not enough: it holds only the deployments a constraint, a store or a volume ties to the label, not everything that happened to run there.

Run every step from here on a survivor node. The recover is placed the way an install is and runs on the node it lands on, which need not be the one you run it from: the archives go there and the data replays there. Check the surviving location has room:

Terminal window
shc node status
shc label show location=hetzner/hel1

Room includes disk for the deployments’ images. A recover lands only on a node that can take the images it lacks, and refuses before it installs anything when no node can, naming the one that comes closest (step 5).

An app whose provider (its database, say) survived in another location is outside this page: leave it off, stop for it and escalate. Its recover runs the app’s install hooks against the live provider before the app’s own files are back, with no restore protection on that provider, and an app’s init can drop tables it does not recognise as its own.

A deployment on a block volume is outside this page as well. A recover keeps the volume the deployment already has, wherever it lands, and a cloud volume in the lost location attaches to no node anywhere else (Hetzner refuses a server in another location). List the block volumes:

Terminal window
shc volume ls --all-tenants -o json > ogden-volumes.txt

A row whose tenant, environment and stack match a deployment on the list means leave that deployment off, stop for it and escalate. Data on host binds and Docker named volumes is not a block volume, is not listed here, and comes back from the archive.

If the lost location held the estate’s only ingress entry point, relocate that first by changing where it may run. Its durable volumes died with the node, so it comes back from its latest backup; it re-points every managed DNS record at the new node and needs a DNS provider to do it. --where replaces the deployment’s whole expression, so a term naming the lost node does not stay behind and leave no node. The update prints its plan before it asks:

Terminal window
shc update -t master traefik -e default --where 'node=hel-02' --restore-from-backup

With an edge per location, the survivor’s edge already serves, and this step does not apply.

3. Take each deployment’s archive out of its repo

Section titled “3. Take each deployment’s archive out of its repo”

Restore only the deployments on the list from step 1, less those step 2 set aside. A deployment that runs in a surviving location is never restored: replaying an older archive over it would roll back its live data.

A deployment is a tenant, an environment and a stack, and the archive is chosen by all three. backup ls filters by stack only, and its table shows neither the environment nor the consistency id:

Terminal window
mkdir -p /var/lib/shc/recovery
shc backup ls -t <tenant> -s nextcloud -o json
shc backup ls -t <tenant> -s nextcloud-db -o json

Use the completed rows of that stack. An app and its database that are both on the list recover together, in one call, and only from archives that carry the same consistency id, or neither carries one: a scheduled group fire gives every member the same id, and the recover refuses inputs with two different ones. So take the newest app archive whose id also appears on a database archive, and that database archive with it, even when one of them has a newer archive alone: the pair is the recovery point.

A pair also has to share its tenant and environment. One recover call installs and restores every archive into the tenant named by -t and the -e it is given, so a prod app whose database lives in another environment, or another tenant, would get that database restored under the wrong deployment. The export below lets you check it from each archive’s own manifest.

If no pair shares an id, or the app’s database sits in another tenant or environment, stop and escalate; this page has no safe path for it. Recovering them one at a time is not one: the database’s restore protection ends when its own recover finishes, and the app’s recover then runs its install hooks before its files are back, and an app’s init can drop tables it does not recognise as its own.

Verify each chosen archive and export it to a file on this node. A backup id is looked up in the tenant named by -t.

backup verify reads an archive of any size: it streams, so a large archive costs time and egress, not memory. Its refusal is about the archive itself (no data, a missing subject, a container that does not add up), and only that is a reason to take an older one:

Terminal window
shc backup verify -t <tenant> 20260930T031500Z_645cafff7f65e1d6
shc backup export -t <tenant> -i 20260930T031500Z_645cafff7f65e1d6 -f /var/lib/shc/recovery/nextcloud.bak
shc backup verify -t <tenant> 20260930T031500Z_9d1e2f3a4b5c6d7e
shc backup export -t <tenant> -i 20260930T031500Z_9d1e2f3a4b5c6d7e -f /var/lib/shc/recovery/nextcloud-db.bak

The export reads the archive from the lost location’s off-site repo with that deployment’s settings.

Before recovering a pair, read both manifests. Each .bak is an ar archive (ar comes with binutils) whose manifest.json names the deployment it came from and the providers it was wired to:

Terminal window
ar p /var/lib/shc/recovery/nextcloud.bak manifest.json | jq '{tenant, environment, consistency, integrations: [.integrations[]? | {provider_stack_name, provider_tenant_name, provider_environment}]}'
ar p /var/lib/shc/recovery/nextcloud-db.bak manifest.json | jq '{tenant, environment, consistency}'

Both archives name the same tenant, environment and consistency (or neither has one), and the app’s integration with the database carries no provider_tenant_name or provider_environment: those appear only when the provider sat in another tenant or environment. Any difference, or a manifest you cannot read, means stop and escalate.

4. Replace the rules that name the lost location

Section titled “4. Replace the rules that name the lost location”

A recover is placed by the deployment’s rules, so a deployment whose placement rule names the lost location, or one of its nodes, cannot come back until that rule is replaced. List the rules that name the location and those that name each of its nodes:

Terminal window
shc constraint ls --label location=gorillaservers/ogden --all
shc constraint ls --label node=ogden-01 --all

Replace each deployment’s rule with the survivor’s location, naming the deployment by its tenant, stack and environment:

Terminal window
shc constraint set -t <tenant> -s nextcloud -e prod --where 'location=hetzner/hel1' --scope deployment --reason 'ogden lost 2026-09-30'

A rule set at tenant or cluster scope binds every deployment under it, and a narrower rule cannot loosen it: replace it at its own scope, with a where that keeps the survivor. shc constraint explain -t <tenant> -s nextcloud -e prod shows which scope set each rule.

Dry-run first, then recover, with the deployment’s environment. The pair from step 3 goes in one call, and the database comes up before the app. The dry run names the node the pair lands on (node_name, and node_reason for why):

Terminal window
shc recover -t <tenant> -b /var/lib/shc/recovery/nextcloud.bak -b /var/lib/shc/recovery/nextcloud-db.bak -e prod --dry-run
shc recover -t <tenant> -b /var/lib/shc/recovery/nextcloud.bak -b /var/lib/shc/recovery/nextcloud-db.bak -e prod --yes

A deployment with no provider at all recovers alone, the same way.

The recover is placed the way an install is: the deployments’ rules, each node’s condition, its cpu and memory, and its disk for the images the archived deployments run and it does not hold. Every deployment of one call lands on one node, so the app and its database come back together. It is never bound to the lost node. When the node it lands on is not the one you run it from, the archives go to that node streamed to its disk, and that node runs the install and the data replay; the operation’s progress names it. The deployment’s row still names the dead node and moves to the node it lands on. The deployment keeps its name. A deployment whose rules leave out the node it would land on is refused with X409173FNCRCV, naming each rule and the command that replaces it, and nothing is installed: go back to step 4 for it, and the dry run says the same.

A call no node has the disk for, keeping 2 GiB free, is refused with X507299RCVNRM, which names the node that comes closest, and nothing is pulled. Free disk there, or recover the deployments in separate calls when they have no provider between them.

A node or an image the check cannot measure, an incus node or an image from a registry that wants credentials, is recovered without it, and the recover says so on a warning line (X200297RCVRUM).

The recover replays each deployment’s ingress, and managed DNS records follow it to the new node. If that replay fails, the log says so and names the update that finishes it.

Three commands look like they fit and do not:

  • shc recover --consistency-id restores each member in place, which needs the dead node, and a scheduled fire’s group can include deployments that are still running elsewhere.
  • shc backup restore restores into a stack that already runs on this node.
  • shc update --where relocates a deployment off a dead node by reusing the volumes that survive it. A volume in the lost location cannot attach anywhere else, so the update is refused naming it, and --restore-from-backup restores only the deployment’s latest backup in its own repo.

6. Give the recovered deployments a backup home

Section titled “6. Give the recovered deployments a backup home”

They now back up to the survivor location’s repo, unless a narrower scope names one: a backup.repo set at the tenant, environment, stack or deployment wins over the location. Check those first, and unset or change any that names the lost location’s repo:

Terminal window
shc config get -t <tenant> backup.repo --scope deployment
shc config get -t <tenant> backup.repo --scope stack
shc config get -t <tenant> backup.repo --scope environment
shc config get -t <tenant> backup.repo --scope tenant
shc config get backup.repo --scope deployment -t <tenant>
shc config unset -t <tenant> backup.repo --scope deployment

The recover restored the deployment’s own settings from the archive, so a deployment-scope backup.repo is the likeliest to still name the lost repo.

Check the survivor’s location label names a repo, set it if not (outside that location), take the first archive in the new place and read its row:

Terminal window
shc label show location=hetzner/hel1
shc label set location=hetzner/hel1 --backup-repo s3:https://fsn1.your-objectstorage.com/backups-hel1
shc backup create -t <tenant> -s nextcloud -e prod
shc backup verify -t <tenant> 20260930T120000Z_1f0c9e2a7b3d4e5f
shc backup ls -t <tenant> -s nextcloud -o json

The location names the survivor’s repo. Until that archive exists, the only copy of this deployment’s data is the archive you recovered from.

Once every deployment on the list is recovered:

Terminal window
shc node forget ogden-01 --yes
shc node forget ogden-01 --force --yes

The first form refuses (X400017NODDEP) while rows still name the node. On a lost node the refusal names the second form and no drain: --force removes the node and leaves the rows naming it as they are. Keep the lost location’s fence. It carries the old repo, so the archives taken there stay listed and restorable, and the mirror keeps copying them.

When the location’s off-site repo is gone too

Section titled “When the location’s off-site repo is gone too”

Then the archives come from the second-provider mirror. The mirror is a file copy, so the repo password is the same; the credentials are the second provider’s, which the daemon only reads from backup.* settings. Set those first, at global scope, and at any narrower scope that overrides them for a deployment you are restoring:

Terminal window
shc config get backup.aws_access_key_id --scope stack
shc config set backup.aws_access_key_id <copy-key-id> --scope global
shc config set backup.aws_secret_access_key <copy-secret> --scope global

The keys are the vault’s backup/copy_aws_*. The relocate opens the mirror with the global key and these credentials.

Then point every row that names the lost repo at its mirror copy. The mirror runs nightly, so it lacks whatever was captured after its last run, and those archives went with the repo. --skip-missing moves only the rows whose snapshot the mirror holds and leaves the rest naming the lost repo; the dry run says how many it leaves:

Terminal window
shc backup relocate --from s3:https://ord.example.com/backups-ogden --to s3:https://s3.us-west-004.backblazeb2.com/shc-backup-copy/locations/gorillaservers/ogden --skip-missing --dry-run
shc backup relocate --from s3:https://ord.example.com/backups-ogden --to s3:https://s3.us-west-004.backblazeb2.com/shc-backup-copy/locations/gorillaservers/ogden --skip-missing --yes

In step 3, take the newest archive the relocate moved, the newest row whose location names the mirror: a newer row it left naming the lost repo cannot be read.

Run steps 3 to 5 while the keys are set. They switch every repo that inherits them, so put the originals back before the next scheduled fire.

Then carry on from step 6.

The two-site rig runs this page as scenario S10: it cuts a location, follows the steps, and the log goes on SHC-1164. tests/unit/cliverbs/location_loss_runbook_test.go checks that every command here resolves, every flag exists, and the recovery keeps to backup export and recover -b ….