Postgres members across locations
The catalogue Postgres can run more than one member: one writes, the others
stream from it and can take over. Members are worth having when they sit on
different machines in more than one place, so that losing one machine or one
place leaves the others. home01 is one machine in one location, so this page
adds two: home02 in location home and far01 in location far.
Prepare two more Debian 13 machines the way home01 was prepared, with 2 CPUs,
8 GB of memory and SHC installed, named home02 and far01. They must reach
home01 and each other, and root on each must be able to ssh home01 with a
key and no prompt: a join reads its token from home01 that way.
Each node joins from its own shell. The join asks home01 for a fresh token
over ssh and hands it to shc node join -, which reads it from standard
input, so the token is never copied or typed. The join also names the node,
its cloud and its location; a join without them takes a random name and
whatever the machine’s cloud reports.
$ ssh home01 shc node token --fields token -o text | shc node join - --name home02 --cloud learn --location home✔ joined cluster as node "home02" at 91.99.184.50$ ssh home01 shc node token --fields token -o text | shc node join - --name far01 --cloud learn --location far✔ joined cluster as node "far01" at 65.21.55.73Run these commands again if a node is still vault-locked or not ready.
Alone, home01 kept the cluster’s database as one SQLite file. When the first node joins, SHC promotes that file into a replicated database and restarts home01’s daemon to do it, within a minute of the join. A command typed on home01 in that moment can fail; type it again.
$ shc node lsNAME STATE CONDITION AVAIL CLOUD LOCATION ROOM BUILD LEADER VOTERfar01 healthy ready active learn learn/far ok 0.1.0 no nohome01 healthy ready active learn learn/home ok 0.1.0 yes yeshome02 healthy ready active learn learn/home ok 0.1.0 no no$ shc cluster statuscluster name: shc-clusterleader: home01nodes: 3 healthy of 3voters: 1quorum: healthy, 1 voter requiredwarning: only 1 raft voter: losing one stops the control plane committing, and then every deployment with a single-writer volume stops its writer within about a minute, on the surviving nodes too, unless the volume is far, with no control-plane fence node in its reachwarning: control-plane fence has 2 members, fewer than three, so it cannot hold three voters until `shc node update <node> --control-plane` brings a node into it, and the voter count followsleader-only work, recorded by home01 17s ago: acme last swept 22s ago backup_scheduler running 2m22s, no sweep yet container_reconciler last swept 21s ago mesh_snapshot_sign last swept 22s ago operation_registry last swept 22s ago proxy_reconcile last swept 22s ago schedule_dispatcher last swept 22s ago scheduler last swept 17s ago service_vip_alloc last swept 17s agoThree nodes in two locations, and one voter. home01 and home02 are the control plane’s fence, and two fence members keep one voter, as the warnings say; the cluster ladder grows the fence to three. The deployments of the earlier pages stay on home01: a node joining moves nothing.
Install the group with --scale:
$ shc install postgres -s members --scale postgres=2@home,1@farinstalling deployment "members"[members] Healthy 6/6✔ etcd-1 healthy✔ etcd-2 healthy✔ etcd-3 healthy✔ postgres-1 healthy✔ postgres-2 healthy✔ postgres-3 healthy✔ installed deployment "members"$ shc lsSTACK TENANT ENV APP STATUS NODES...members prod prod postgres running far01, home01, home02...$ shc exec -s members -S postgres-1 -- psql -U postgres -d postgres -tAc 'select pg_is_in_recovery()'f$ shc exec -s members -S postgres-1 -- psql -U postgres -d postgres -tAc 'select application_name, state from pg_stat_replication order by 1'postgres-2|streamingpostgres-3|streaming$ shc exec -s members -S postgres-2 -- psql -U postgres -d postgres -tAc 'select pg_is_in_recovery()'t$ shc exec -s members -S postgres-3 -- psql -U postgres -d postgres -tAc 'select pg_is_in_recovery()'t--scale postgres=3 sets a count without choosing locations. Here
2@home,1@far places two members in learn/home and one in learn/far.
The member services are postgres-1, postgres-2 and postgres-3.
The service name postgres reaches the writer. shc ls lists the
deployment; shc status -s members lists its group, each member with its
location, readiness and services, and the member the postgres address
points at.
The members come from the catalogue’s compose.yaml. It writes a member’s
services once, between members and endmembers, and SHC repeats them for
each member:
$ grep -E '^\{% (end)?members|^ [a-z]+-\{\{ m.n \}\}:' /usr/lib/shc/apps/postgres/compose.yaml{% members "postgres", default=1, max=5 as m %} postgres-{{ m.n }}: etcd-{{ m.n }}:{% endmembers %}m.n is the member’s number, so member 2 is postgres-2 and etcd-2.
default=1 is why db has one member, and max=5 is the most --scale
can ask for. The etcd-N services run only from two members on. See
Groups.
Read Underneath the members group before going on; it reads what this page left running.
Next: A node failing.