Skip to content
SHC Docs

Groups and HA

This page writes a group: copies of a service that are not alike. A service scales by replicas that are all the same, but a database with a primary and replicas is not: copy 2 is not copy 3. A group is a {% members %} block whose body SHC renders once per member, each member with its own number, services and volumes. The app ladder’s Postgres members across locations installs the catalogue’s group. Write one.

A group earns its keep only across machines: members on one node all go down with it. This is the first page that needs more than home01, so join three more now.

Prepare three machines like home01: Debian 13, SHC installed, 2 CPUs and 8 GB of memory, a root shell. Name them home02 and home03, in location home, and far01, in location far. Root on each must be able to ssh home01 with a key and no prompt: a join reads its token from home01 that way.

Each node joins from its own shell. The join asks home01 for a fresh token over ssh and hands it to shc node join -, which reads it from standard input, so the token is never copied or typed. The join also names the node, its cloud and its location.

home02
$ ssh home01 shc node token --fields token -o text | shc node join - --name home02 --cloud learn --location home
✔ joined cluster as node "home02" at 192.168.231.12
home03
$ ssh home01 shc node token --fields token -o text | shc node join - --name home03 --cloud learn --location home
✔ joined cluster as node "home03" at 192.168.231.13
far01
$ ssh home01 shc node token --fields token -o text | shc node join - --name far01 --cloud learn --location far
✔ joined cluster as node "far01" at 192.168.231.14

The cluster takes a few minutes to settle. The first join moves home01’s store to the replicated one and restarts its daemon, and the two new home nodes become voters about a minute apart. Run these again until they read like this.

home01
$ shc node ls
NAME STATE CONDITION AVAIL CLOUD LOCATION ROOM BUILD STORE LEADER VOTER
far01 healthy ready active learn learn/far ok 0.1.47 replica no no
home01 healthy ready active learn learn/home ok 0.1.47 voter yes yes
home02 healthy ready active learn learn/home ok 0.1.47 voter no yes
home03 healthy ready active learn learn/home ok 0.1.47 voter no yes
$ shc cluster status
cluster name: shc-cluster
leader: home01
nodes: 4 healthy of 4
voters: 3
quorum: healthy, 2 voters required
leader-only work, recorded by home01 19s ago:
acme last swept 28s ago
app_refinish last swept 58s ago
backup_scheduler running 10m58s, no sweep yet
container_reconciler last swept 57s ago
mesh_snapshot_sign last swept 28s ago
operation_registry last swept 28s ago
proxy_reconcile last swept 56s ago
schedule_dispatcher last swept 28s ago
scheduler last swept 23s ago
service_vip_alloc last swept 23s ago

Three voters, so the cluster now survives losing one node. Nothing you installed moved: an app moves when a rule or a command says so. The vault’s cold start, which home01 has held alone since the vault page, can move now too. coldstart replant hands that duty to another node.

home01
$ shc vault coldstart replant --node home02
...
$ shc vault coldstart replant --node home01
...

scores is a Redis replica set: member 1 is the primary and every other member replicates it. Write it on home01 as a compose file.

~/learn/scores/app.yaml
name: scores
version: "1.0.0"
title: Scores
description: A Redis replica set and a witness
category: demo
license:
id: MIT
~/learn/scores/compose.yaml
x-shc-service-selectors:
kv: role=master:1
services:
{% members "kv", default=3, max=5 as m %}
kv-{{ m.n }}:
image: redis:7-alpine
command: ["sh", "-c", "if [ {{ m.n }} = 1 ]; then exec redis-server --appendonly yes; else exec redis-server --appendonly yes --replicaof kv-1 6379; fi"]
expose:
- "6379"
volumes:
- data-{{ m.n }}:/data
environment:
MEMBER_LOCATION: "{{ m.location }}"
x-shc-live-labels:
role:
test: ["CMD-SHELL", "redis-cli role | head -n 1"]
interval: 5s
timeout: 3s
{% endmembers %}
watcher:
image: redis:7-alpine
command: ["sh", "-c", "while true; do redis-cli -h kv role | head -n 1; sleep 10; done"]
x-shc-scale:
witness: kv
  • {% members "kv", default=3, max=5 as m %} is group kv, three members unless the install says otherwise and never more than five. m.n is the member’s number and m.location the location it was given.
  • data-{{ m.n }} is one volume per member, which SHC declares itself.
  • x-shc-live-labels runs redis-cli role in each member every 5 seconds and keeps the first line it prints as that member’s label role: master or slave. SHC does nothing with a label by itself. You read it, and so do a node selector and a service selector. A few label names are SHC’s own, service and node among them, and a live label that names one is refused.
  • x-shc-service-selectors, at the top of the file, is the names the deployment answers under besides its services’ own. Each is a name and a selector over the labels of the deployment’s replicas. kv: role=master:1 sends the name kv to the member whose label reads master. The :1 means exactly one: with no master, or with two, kv answers nothing rather than reach the wrong member. Without :1 a name goes to every member that matches, so kv-ro: role=slave would spread over the replicas.
  • x-shc-scale.witness asks for watcher on a node that holds no member of kv, so losing one node never takes a member and the witness together.

Every rule of the block is in Groups.

Install it with a count per location: --scale kv=2@home,1@far puts two members in home and one in far. A bare --scale kv=3 would set the count and leave the places to SHC.

home01
$ shc validate ~/learn/scores
✔ validated "/root/learn/scores"
$ shc install ~/learn/scores -s scores --scale kv=2@home,1@far
installing deployment "scores"
[scores] Healthy 4/4
✔ kv-1 healthy
✔ kv-2 healthy
✔ kv-3 healthy
✔ watcher healthy
✔ installed deployment "scores"

shc status -s scores shows the node each replica runs on, and its Groups section lists the group’s members with the location each was given and the member the service selector kv points at. kv-3 is on far01, the only node in far, the witness is on the home node that holds no member, and each member carries its role label: master on kv-1, slave on the others. Its Service selectors section shows the service selector kv with the replica it points at.

home01
$ shc status -s scores
...
prod scores prod kv-1-r0 home02 ready role=master
prod scores prod kv-2-r0 home03 ready role=slave
prod scores prod kv-3-r0 far01 ready role=slave
prod scores prod watcher-r0 home01 ready -
...
Groups
GROUP MEMBER LOCATION READY ALIAS SERVICES
kv 1 learn/home ready kv kv-1
kv 2 learn/home ready kv-2
kv 3 learn/far ready kv-3
...

Write on member 1 and read it from member 2 and through the service selector kv.

home01
$ shc exec -s scores -S kv-1 -- redis-cli set top 100
OK
$ shc exec -s scores -S kv-2 -- redis-cli get top
100
$ shc exec -s scores -S watcher -- redis-cli -h kv get top
100
$ shc logs watcher -s scores
...

Scale the group, never its services. A member keeps its number, and a member scaled away keeps its volume until it comes back. Put a file on kv-3’s volume, then scale the far member away.

home01
$ shc exec -s scores -S kv-3 -- touch /data/mark
$ shc update scores --scale kv=2@home --yes
...
home01
$ shc status -s scores
...
Groups
GROUP MEMBER LOCATION READY ALIAS SERVICES
kv 1 learn/home ready kv kv-1
kv 2 learn/home ready kv-2
...

Two members, both in home. Bring the far one back.

home01
$ shc update scores --scale kv=2@home,1@far --yes
...
home01
$ shc status -s scores
...
prod scores prod kv-3-r0-i1 far01 ready role=slave
...
$ shc exec -s scores -S kv-3 -- ls /data/mark
/data/mark

kv-3 is back on far01 with its number and its volume: the file is still there. The -i1 on its name says this is the replica’s second making.

Next: A drain and a disruption budget.