Skip to content
SHC Docs

Database apps, HA by scale

Layer 6. Services. Services

New line, in development. This page describes SHC as it is on the dev branch, not the released product. See The new line.

Status: built (SHC-1789). Every engine is built: PostgreSQL (SHC-1793), MongoDB (SHC-1792), MySQL (SHC-1794), MariaDB (SHC-1795) and Redis (SHC-1796). So are the platform pieces: groups.<name> (SHC-1790), the writer service selector (SHC-1787, a service selector over a live label since SHC-2459), witness placement (SHC-1791) and one backup per group (SHC-1826). ClickHouse is decided: HA by scale is not offered (SHC-1798).

Clay, 2026-10-07: “If you only have a single replica it doesn’t deploy the proxy or HA stuff, but the moment you add a second or third replica it goes into HA mode, and can go back and forth. From the point of view of the person managing them they’re just scaling it up or down.” No sharding. High availability only.

So the person running a database types one thing:

Terminal window
shc update -s db --scale postgres=3
shc update -s db --scale postgres=1

A group with two or more members makes a deployment highly available. The drain budget reads the stored member groups (SHC-1789).

Each database app is a group: a {% members %} block with default=1. Member n is the service <engine>-<n> with its own volume <volume>-<n>.

MembersWhat runs
1One plain server. No manager, proxy, witness or arbiter. The replication settings that need a restart to change are already on, so a second member can join later.
2The engine’s own HA among the two members, plus a tiny third vote where the engine has one (see Two members).
3 or moreThe engine’s own HA among the members, with automatic failover by quorum.

A new member clones from the current writer. The engine does the copy (pg_basebackup, initial sync, clone, SST, full resync). SHC only starts the member.

A scale change restarts no member it keeps. Going from 3 to 5 adds members 4 and 5 and leaves the bytes and digests of members 1 to 3 alone (Groups). So a member’s data service may depend on the member count only at the one-to-two edge. Two engines need a restart there, because their HA layer has to own the server process: PostgreSQL (Patroni) and MariaDB (Galera). The others never restart member 1.

A scale-down keeps the writer and every committed row. Scaling down retires the highest numbers. Before SHC stops them, the app’s pre_update hook moves the writer to a surviving member (the lowest-numbered unless the engine’s priorities name another) and removes the retiring members from the engine’s own membership. Without that step a scale-down can leave no primary at all: the retired mongodb-replicaset going from 3 to 1 was the known case (SHC-1728).

HA pieces render from the member count. A manager sidecar, a witness or a sentinel set is written in the template under {% if groups.<name>|length >= 2 %}. groups.<name> is the group’s member list, readable anywhere in the template and in its hook jobs (Groups, SHC-1790).

One name reaches the writer. Consumers connect to one host name through the app’s socket, today <environment>.postgres.<stack>. At two or more members that name has to reach whichever member is the writer now. MongoDB drivers find the primary on their own, and Sentinel-aware Redis clients ask the sentinels. Most catalogue consumers take a single host, so the other engines need SHC to route the name. See The writer service selector.

With two voters, losing either one loses the majority, so two members on a consensus engine are less available than one unless a third vote exists. Each engine gets one of two answers:

  • A witness where the engine has one in its image: an etcd-only member for PostgreSQL, an arbiter for MongoDB, the sentinels for Redis. Failover is automatic at two. The witness renders only at two members and goes at three, except Redis’s sentinels, which are its failover at every count above one. It belongs on a third node (SHC-1791).
  • Member 1 keeps the data path where the engine has no witness: MySQL runs member 2 as an asynchronous standby, and MariaDB gives member 1 the larger Galera weight. Losing member 2 changes nothing. Losing member 1 needs one documented promote command. Automatic failover starts at three.
PieceWhyTicket
groups.<name> in the template and its jobs (built)Render HA pieces only at two or more members, and let a pre_update job see which members a scale-down retiresSHC-1790
A writer service selector (built)The group’s own name reaches only the member whose role live label says primary, and <group>-ro the replicasSHC-1787, SHC-2459
Witness placement (built)A witness prefers a node that holds no member of its group (x-shc-scale witness: <group>)SHC-1791
One-member backups (built)A converter’s from: reads one member per run: a ready standby, else the writer (Backups and restore)SHC-1826

Nothing else is new. Groups, member pods, member VIPs, per-member volumes, stable member numbers and pre_update gating already exist (Groups, Lifecycle hooks).

A service has one virtual IP over its ready replicas, and ready means running and healthy (ce modules/app/readiness.go). A standby is healthy, so a VIP cannot pick the primary by readiness without a second check. A health check that fails on standbys is not a fix: the node agent restarts a container that stays unhealthy (x-shc-health.restart_after, five minutes by default), and an install waits for every replica to read ready.

What is built (SHC-1787, SHC-2459): each member’s database service declares a role live label, a command SHC runs beside its health check whose first line is the member’s role, and the app declares the service selectors that read it at the top of its compose file, under x-shc-service-selectors. The group’s own name, which no service may take (X400096GRPDUP), is <engine>: role=primary:1: one VIP whose backend is the one ready member that says primary. With none, or with more than one, it has no backend and the connection is refused: it never reaches two writers. <engine>-ro: role=replica spreads over every ready member that says replica. At one member the command prints primary whenever the server takes connections, so postgres keeps meaning what it means now and no consumer changes. Groups has the syntax and the timing.

A service stays its replicas plus one VIP. Every other name is a service selector the app declares, a selector over its replicas’ labels.

What it asks of each engine’s app:

  • one role live label per member, on the database service, that prints primary on a single plain member too, and the service selectors in x-shc-service-selectors at the top level. SHC reads compose’s $$ escape in the command as one $;
  • the socket fronted by the group (service: <engine>, or a socket named after it), so the consumer’s host name is the service selector named for the group;
  • context: service jobs naming a member (service: <engine>-1) and dialling the group’s name, since the group has no container;
  • a pre_update hand-over on the way down: if the update drops the member that is the writer, the service selector is refused until the update has retired it.

MongoDB drivers find the primary on their own, and mongodb declares the check anyway (hello().isWritablePrimary), so a consumer that takes one host name reaches the primary too.

The other two answers, for the record: a router (HAProxy or the engine’s own) that is always present, which breaks “no proxy at one member”; or a router only at two or more with every consumer re-pointed at the edge, which restarts every consumer when the database crosses one to two.

Image facts were read from the pinned images on a test box on 2026-10-07.

EngineAppOne memberTwoThree or moreRestart at 1↔2Writer checkTicket
PostgreSQL 16ce modules/postgresplain server, wal_log_hints onPatroni + etcd, etcd witness, asynchronousPatroni + etcd, synchronous_mode: quorummember 1, oncePatroni GET /primarySHC-1793 (built)
MongoDB 8.0mongodbone-member replica setreplica set + arbiterreplica setnonehello().isWritablePrimarySHC-1792 (built)
MySQL 8.4 (Percona Server)mysqlGTID, binlog, plugins loadedasync standby, manual promoteGroup Replication, single primarynonePRIMARY and super_read_only = 0SHC-1794 (built)
MariaDB 12.3mariadbGalera prerequisites set, wsrep offGalera, member 1 weight 2Galeramember 1, oncesynced and lowest synced memberSHC-1795 (built)
Redis 8.10redisplain server, masterauth setreplica + three sentinelsreplicas + three sentinelsnonerole:masterSHC-1796 (built)
ClickHouseclickhouse, openreplay-chone servernot offerednot offeredn/an/aSHC-1798 (decided)

Built (SHC-1793). ce modules/postgres is the group postgres: member n is the service postgres-<n> on the volume pgdata-<n>, default 1, max 5. Every member runs the major-version check in front of its own server, on its own pgdata-<n>; there is no separate one-shot (SHC-2596).

One member is the plain server it always was: the image’s own entrypoint, with wal_log_hints on so a second member can rewind it later. No Patroni, no etcd, nothing from the app’s bin/.

From two members every member runs Patroni 4.1.3, which the Percona image ships with its etcd3 store and no etcd server, beside an etcd 3.6 sidecar etcd-<n> on the volume pgetcd-<n>. The etcd members are Patroni’s store, so consensus runs among the postgres members and never uses SHC’s own store. Patroni’s Raft store cannot run: pysyncobj is not in the image. The etcd image has no shell, so bin/ha.sh in the member adds etcd-<n> to the etcd cluster and writes the sidecar’s config on the volume they share. Patroni takes member 1’s data directory over in place, which restarts that one server at the one-to-two edge in each direction, and every other member clones it with pg_basebackup. At exactly two members postgres-witness, an etcd that holds no data, is the third vote. Two members replicate asynchronously and fail over only to a standby within 1 MiB of the leader. From three, synchronous_mode is quorum, so a failover promotes a standby that holds every committed transaction. The role label is primary where Patroni’s GET /primary answers and replica where GET /replica does from two members, and primary whenever pg_isready passes at one, so postgres and <environment>.postgres.<stack> reach the leader, postgres-ro the standbys, and consumers change nothing.

Each run at two or more members is a formation, with an id in etcd and in every data directory Patroni runs. Member 1 starts one when its data last ran as a plain server, on a first install, and after a restore. A member holding another formation’s data, or an unfinished copy, starts empty and clones again. A member that left the etcd cluster, because a handover took it out to move it or a scale-down retired it, and comes back with this formation’s data rejoins from that data: Patroni rewinds it to the leader, and it clones again only if it does not stream within five minutes. Member 1 never discards its data. No other member starts until member 1’s own running Patroni reports the formation it serves, so after a restore a member holding rows newer than the backup can never lead.

pre_update runs handover before the major-version ladder. When the update retires the leader it switches the lead to the lowest member that stays and waits for it, takes the retiring members out of etcd (and the witness past two), and going to one member pauses Patroni so the lead stays on member 1. A major-version upgrade is refused at two or more members: scale to one, upgrade, scale back. post_install, post_update and post_restore run settle, which sets the synchronous mode and the witness for the member count. Back at one member, member 1 restarts as a plain primary with its own postgresql.conf and pg_hba.conf, and Patroni’s replication slots go. A member that was a standby refuses to start as the only server, so a group never comes back on a copy that missed commits. Jobs that run in a container exec into postgres-1, and the ones that write find the leader first.

Backups follow the one-backup-per-group rule (SHC-1826): the converter for pgdata-1 names from: group=postgres, so each backup reads one member’s pgdata, the one member’s or from two members on the lowest ready standby’s, else the writer’s, and never etcd’s state. At two or more members quiesce runs in that member. On the writer it makes the server read-only and checkpoints it, as at one member. On a standby it pauses the replay and writes a restartpoint, so its files hold still while the writer keeps writing. Patroni resumes a paused replay on every pass of its loop, so every member runs Patroni through bin/patroni-hold.py, which leaves the replay paused while the hold lasts. The held member’s failover priority is 0 meanwhile, so a failover during the copy promotes another member. At two members there is no other, so a leader lost during the copy is replaced once the backup ends. resume releases both. A group backup takes no pause, so a writer’s copy, one member’s included, is of the running server. The backup’s converter starts every copy with ordinary crash recovery from the WAL it carries, which repairs whatever was written during the copy. pg_resetwal would drop that WAL, and on a standby’s copy every row replayed since its last restartpoint, so it runs only on a writer’s copy that cannot recover, and a standby’s copy that cannot recover is kept raw. A restore at any member count writes the copy into member 1, makes it the writer and starts a new formation, and the other members clone it. It also empties member 1’s pgetcd-1, as it empties every directory of a group’s lowest member that backup.yaml lists under neither include nor keep, so etcd starts over from the restored data instead of waiting for a quorum of members the restore emptied.

Measured in a plain-docker lab of the same scripts on a test box (2026-10-07), with a write loop running throughout. Going 1→2→3→2→3→1→3, and both scale-downs that first move the leader, lost no committed row. With the leader’s containers killed, Patroni promoted a standby in 24 s at three members and 25 s at two. A backup at three held member 2 while the writer acked 169 more rows, and with the leader killed during such a hold Patroni promoted member 3 in 28 s while member 2 stayed paused. The converter’s start on every copy, a live one-member copy and a held standby’s, kept every row of the copy. A restore at three, with members 2 and 3 emptied and started first, and a restore at one both came back with exactly the backup’s rows.

When the leader’s whole node goes, its etcd member goes with it. At three members a standby leads within 60 s of the node’s power-off, and the name postgres follows it with nothing typed. On a rig cell of Hetzner nodes (SHC-1793, 2026-10-07) that took 52 s and 42 s, counted from the power-off call, and the name reached the new leader 71 s and 58 s after it. At two members the same power-off was 26 s to the new leader and 41 s to the name. Patroni’s ttl of 15 s and loop_wait of 5 s are kept: shorter ones would fail over on a network blip.

The replica set is native and every change is online: rs.add, rs.remove, rs.stepDown. At one member the set has one member. A new member adds itself through the primary and initial-syncs. A member the set no longer lists, because a handover took it out to move it or a scale-down retired it, and that comes back with a copy of this same set (its replicaSetId) adds itself again and catches up from its own oplog; it starts empty and initial-syncs only when it is not a secondary within five minutes, or when its copy is of another set. Member 1 coming back this way joins like any other member and never initiates a second set. At two members mongodb-arbiter votes. The members var sets each member’s priority, votes, hidden and tags by location path (members: {hetzner/hel1: {priority: 2}, edge: {priority: 0, votes: 0}}, the deeper key winning field by field); a settle job applies it after every install, update and restore without recreating a member, and the handover keeps those priorities instead of resetting them. MongoDB moves the primary back to the highest-priority member once it has caught up: the engine following priorities the operator set, never an SHC home. Members are named in the set by the scoped name a consumer resolves, <environment>.mongodb-<n>.<stack>, so a driver that discovers the set reaches every member. Beside the arbiter a new member joins through rs.reconfigForPSASet, since mongod refuses an electable secondary in that set outright. The socket is fronted by the service selector named for the group, and a backup reads one member through from: group=mongodb on its converter, a ready secondary when there is one: quiesce and resume run in mongodb-{{ backup.member }}, and the quiesce takes the fsync lock on whichever member that is. A secondary more than 10 s behind the primary, or a member in any state but primary or secondary, refuses, so the backup reads the next member. mongodb-replicaset and librechat-db are retired into mongodb.

Percona Server 8.4 ships group_replication.so, mysql_clone.so and mysqlsh. Every setting that Group Replication needs and that only applies at startup is on from the first start, so no scale change restarts a member. Group Replication has no witness, and a two-member group stops writing when either member is lost, so two members run an asynchronous standby instead and three or more run Group Replication.

Built (SHC-1794). Member n is mysql-<n> on volume mysqldata-<n>, up to seven. Every member runs bin/member.sh with the same command line at every count: GTIDs and the binary log on, its number as server_id, both plugins loaded and Group Replication off at boot. It also hands mysqld the group’s local address at every start: the container’s own address on the interface its default route leaves by, which is the fabric (bin/address.sh, SHC-2084). A member’s name answers with its service’s address, which is on no interface in the container, and Group Replication refuses a local address its server does not hold. Nothing persists that address, so a member recreated at another one advertises the new one; the seeds, recovery and clone go to the members’ names. The app’s hooks run bin/ha.sh: converge after an install or an update, handover before an update.

MembersWhat runs
1A plain writable server. The replication account is dropped.
2Member 2 clones the writer (CLONE INSTANCE) and follows it asynchronously with GTID auto-position, super_read_only on. Losing member 2 changes nothing. Losing member 1 takes one command, shc job run promote -s <stack>, which applies what member 2 already received and makes it the writer. When member 1 is back, shc job run converge -s <stack> (or any update) makes it the standby, and refuses if it holds transactions member 2 never received.
3 or moreA single-primary Group Replication group, started on the writer. A new member clones the writer before it joins. A lost primary is expelled as soon as it is suspected, so a new one is elected in about ten seconds. A member cut off from the majority leaves after 30 seconds, and when every member stopped at once, the one holding every transaction starts the group again on its own.

Scale-down: handover moves the writer to the lowest kept member (group_replication_set_as_primary in a group; at two, member 1 must have applied everything first) and takes each retiring member out of replication, read-only, before SHC stops it. A member move runs settle once the member is back on its own volume, the same ha.sh converge with every member, and the member rejoins by incremental recovery from its own data. A first scale to three or more is refused while a table has no primary key or is not InnoDB, because Group Replication refuses writes to those tables.

Backups follow the one-backup-per-group rule (SHC-1826): the mysqldata-1 converter says from: group=mysql, so a backup reads one member, the lowest ready standby or else the writer, and member 1 at one member. The pre_backup job quiesce runs in that member alone: it stops a standby’s applier, then holds FLUSH TABLES WITH READ LOCK with super_read_only on one open connection until resume puts the member back as it was. The other members’ volumes stay out of the archive: a restore writes member 1, detach takes the others out of replication, the dump replays into member 1, and reattach has them clone it again.

Consumers reach the writer through the service selector named for the group (SHC-1787, SHC-2459): every member’s role live label is bin/role.sh, which prints primary while @@global.super_read_only = 0 and replica on an online group secondary or a running replica, and nothing on a member out of its group or a retired one. mysql: role=primary:1 and mysql-ro: role=replica read it, and the socket is named after the group, so <environment>.mysql.<stack> follows the primary through a failover. The vApp and exporter jobs dial mysql the same way.

Built (SHC-1795). The app is the group mariadb: member n is the service mariadb-<n> with the volume mariadbdata-<n>, one by default and seven at most. Each member starts through bin/member.sh, which decides how its server starts.

One member is a plain server, with no wsrep and no provider. binlog_format=ROW, innodb_autoinc_lock_mode=2 and InnoDB as the default engine are set at every count, so its data can always be replicated.

Two or more are a Galera cluster. Galera loads at server start, so member 1 restarts once when the count goes from one to two, and once on the way back. No other scale change restarts a member it keeps. At the edge member 1 starts a new formation from its own data with a fresh state uuid, so every other member copies member 1 whole by SST (mariabackup), a retired member that comes back with old data included. The copy runs as 'mysql'@'localhost' by unix_socket, so no password sits in the server’s flags or files. A joiner copies from the highest-numbered member first: member 1 is usually the writer, and a donor pauses commits briefly at the end of a copy.

Two members. The image ships no garbd, so there is no third vote. Member 1 weighs 2 while the primary component has two members and 1 otherwise. Losing member 2 leaves member 1 serving. Losing member 1 stops member 2 rather than risk a split. shc job run promote -s <stack> runs in member 2 and makes the running member that applied the most writes a primary component; member 1 joins it when it comes back.

One name for consumers. Each member’s role live label is bin/role.sh. It prints primary on a plain server, and in Galera on the lowest-numbered member that is synced in a primary component, and replica on every other synced member. mariadb: role=primary:1 and mariadb-ro: role=replica read it. mysql.socket.yaml says service: mariadb, so a consumer reaches the service selector named for the group and one member takes the writes, which keeps Galera free of certification conflicts. When the writer’s node goes, the service selector moves to the next synced member: 5 seconds in the lab. Member 1 takes the writes back once it rejoins and is synced.

Starting when nobody serves. Each member publishes its saved state on port 4566 inside the deployment, because a server that waits for a primary component answers no query. Member 1’s saved state names the formation; a member that holds another formation is a stale copy and joins. Within the formation, the member that left last (safe_to_bootstrap, written on a clean stop) starts it again. When nobody left last, because every member stopped at once or every member crashed, the member with the highest saved position starts it, the lowest number on a tie. It waits until every member whose name answers has published its state, since an absent member could hold more. A member’s name reaches it only while it reads ready, and in such a start nobody does yet, so while a member is synced it keeps every member’s fabric address beside its data (.shc-peers), and a member whose name publishes nothing is asked at that address (SHC-2176). An answer there counts only when it names that member, since a replica that moved leaves its old address to whoever gets it next. After a crash the position comes from InnoDB (mariadbd --wsrep-recover). A member whose name answers but never publishes holds the start up; shc job run promote -s <stack> then starts the formation from member 2’s data.

Scale-down. The pre_update job handover waits for member 1 to be synced, and when going to one member for it to have applied everything it received. A member move that takes member 1 out waits for the lowest member it keeps instead, and member 1 comes back by IST from its own data. SHC then stops the retired members and each leaves the cluster cleanly. Members get two minutes to stop, so a clean leave is recorded.

Backup and restore. The mariadbdata-1 converter in backup.yaml says from: group=mariadb, so a backup reads one member, as the backup rule asks: the lowest ready member that is not the writer, else the writer, and member 1 at one member. SHC pauses nothing for it, so the pre_backup job freezes that member alone. It refuses a Galera member that is not synced. A session of its own holds FLUSH TABLES WITH READ LOCK, which in Galera desyncs the member and stops it applying, so the others carry on without it; when the writer is the member read, the group’s service selector moves to another synced member. The job then waits until InnoDB has written every dirty page and its log stands still, because the converter boots a copy of the files. The post_backup job ends the session, and a session whose end never comes lets go after 15 minutes. A restore runs at one member: the pre_restore job refuses it at two or more. A dump replays the account tables, which are Aria tables Galera does not replicate, and every other member would be left with no accounts. Scale to one, restore, scale back.

What consumers must meet:

  • InnoDB, with a primary key on every table. Galera replicates InnoDB rows only. MariaDB does not support DELETE on a Galera table without a primary key, and rows of such a table can differ in order between members.
  • One writer. Connect through the socket’s host name. Writes sent to two members at once still work, but conflicting transactions fail at commit with a deadlock error the application must retry.
  • No explicit locks across members. LOCK TABLES and GET_LOCK() are not supported in Galera.

Built (SHC-1796). The app is the group redis: member n is the service redis-<n> with the volume data-<n>, one by default and seven at most. Replication and Sentinel are runtime commands, so no scale change restarts a member it keeps.

One member is a plain server. masterauth, the address it announces and a replica priority equal to its member number are already set, so a second member needs no restart of the first.

Two or more add three sentinel services, redis-sentinel-1 to redis-sentinel-3. They are the third vote at two members and they stay at every higher count. They are a second group, sentinel, of exactly three members, not three replicas of one service. So each has an address of its own and an id that survives a restart, and placement prefers a node of its own for each, as it does for any group’s members. Without a stable id every restarted sentinel is a new one to the others, and the dead one still counts in the majority a failover needs. Leave the sentinel group’s scale alone: fewer than three cannot outvote a lost node. A master that stops answering for 5 seconds is replaced by the lowest-numbered replica in good standing. The sentinels’ config holds the password and lives on a tmpfs.

Addresses, never names. The app’s scripts look every member’s name up themselves, under a timeout, and give Redis and Sentinel addresses only. Both look a name up inside their one thread. Where the name of a member that is gone takes seconds to fail, a replica stalls on it and a sentinel drops into its TILT mode and fails nothing over: a lab with names took 86 seconds to replace a killed master.

Starting. Before its server starts, a member asks the sentinels and the other members who the master is, and starts as that member’s replica. A master that lost a failover comes back as a replica and never as a second master. When nobody answers, only the member that was the master when it last ran starts as one. Each member records its role beside its data, in /data/.shc-role, and a fresh member 1 counts as a master. Every other member waits for one. So a cold start of every member brings back the master that held the newest data, and a new member never serves an empty dataset.

One name for consumers. Each member’s role live label reads INFO replication: primary on the master, replica on a replica. redis: role=primary:1 and redis-ro: role=replica read it, and redis.socket.yaml is named after the group. So the socket’s hostname is the service selector named for the group: a consumer reaches the master at every member count and re-points for nothing. After a failover the service selector follows within two of the label’s 5-second passes and a few seconds more. In between it is refused, never split.

Scale-down. The pre_update job handover runs SENTINEL FAILOVER when the master is a member the update retires. A replica’s priority is its member number, so the master moves to the lowest kept member that replicates. The job checks that and refuses the update otherwise. The post_update job settle resets each sentinel that still lists a retired member, one at a time. Back at one member the sentinels are gone and member 1 is the master.

Backup and restore. The data-1 converter in backup.yaml says from: group=redis, so a backup reads one member, as the backup rule asks: the lowest ready replica, else the master, and member 1 at one member. The pre_backup job snapshot runs in that member: it refuses a replica whose link to the master is down, runs BGREWRITEAOF and BGSAVE in a child process so a master keeps serving, and turns automatic AOF rewrites off until the post_backup job resume, so the snapshot holds an RDB that is only replaced whole and an AOF that only grows. The converter checks the RDB and copies the member’s files, and the restore job place_data writes them into member 1; SHC empties the other members, which replicate from it again. When that leaves no master, for instance a backup read from a replica and restored onto one member, the post_restore job promote makes member 1 the master. shc job run promote -s <stack> does the same by hand.

What it does not do:

  • The socket does not publish the sentinels. A consumer uses the one host name. A Sentinel-aware client inside the deployment can ask redis-sentinel-1:26379 for the master named redis and gets its address.
  • Replication is asynchronous. A write the master acknowledged and had not yet sent to a replica is lost in a failover.
  • It picks no side between two masters. If a master died, a replica took over, and the dead one then comes back in the same cold start as every sentinel, two members answer as masters and nothing remembers which is newer. Members and sentinels wait and log both names, and the group’s service selector is refused while both claim. Make the stale one a replica with redis-cli -h <stale> REPLICAOF <the master's address> 6379.
  • Three sentinels need three nodes to be worth three votes. On fewer, two of them share a node. If that node is also the master’s, losing it loses the majority that allows a failover.

clickhouse and openreplay-ch stay one server. HA by scale is not offered for them (SHC-1798), and x-shc-scale.max: 1 refuses a second member.

The reason is where ClickHouse keeps replication. It belongs to each table, not to the server: only the Replicated MergeTree engines replicate, through Keeper, and the app that creates a table picks its engine.

  • Scaling the server cannot make an app’s tables replicate. A table created as MergeTree holds its rows on the member that took the insert. The Replicated database engine copies the table’s definition to every member and nothing else, so each member would answer with different rows.
  • The open-source server does not swap the engine. Nothing in it turns an app’s ENGINE = MergeTree into a replicated engine. ClickHouse Cloud does, with an engine the open-source server does not have.
  • Rows written before a scale-up would need a conversion per table. A MergeTree table becomes replicated through a flag file in its data directory and a server restart, table by table, and again for every table the app creates later.
  • One member would carry Keeper. Replicated tables need it, and a single member runs no HA machinery. Two members would also need a third Keeper vote.
  • Sharding is out of scope, as for every engine here.

A ClickHouse deployment has what every one-replica stateful app has: its volume, its backups, and a move or a failover of the one replica. An app that wants replicated ClickHouse writes its schema for it and ships its own group.

Clay, 2026-10-07: “we only want one backup, we don’t want to back up every replica or every location”. And there is no new key for it: how a group is backed up follows from backup.yaml itself.

So one rule for every engine (SHC-1826): one backup per group per run, read from one member, never one per member and never one per location. The app says which member through the converter that captures the dataset: from: is a selector, in the language --where reads, and the converter reads one member of the group it picks, chosen when the backup runs.

# backup.yaml: a replicated database
converters:
pgdata-1:
from: group=postgres
image: "{{ vars.images.postgres }}"
command: ...
volumes:
- pgdata-1:/var/lib/postgresql/data
- "@output:/output"
jobs:
restore:
- name: restore_from_dump
image: "{{ vars.images.postgres }}"
command: ...
volumes:
- "@pgdata-1:/backup"
- pgdata-1:/var/lib/postgresql/data
include: [pgdata-1, pgdata-2, pgdata-3, pgdata-4, pgdata-5]
# quiesce.job.yaml, run as the app's pre_backup hook
name: quiesce
context: service
service: postgres-{{ backup.member }}
command: ...

Of the members from: matches, the backup reads the lowest-numbered one that is ready and is not the writer (the member the group-named service selector with :1 picks, here role=primary), and the writer when no standby is ready. from: group=postgres & role!=primary never reads the writer: with no standby ready the run is refused (X409895BKPGRP) and says why each member was passed over. At one member the backup reads that member, as for any one-server app.

The converter is written for member 1’s volume, and SHC points it at the member it reads: it runs over one consistent snapshot of that member, taken after the member’s own quiesce, and writes its artifact under member 1’s key, so the restore job is written for member 1 too. The app lists every member’s volume under include, and the group’s volumes the converter covers are still captured once, through it, and never raw per member. The hooks of the backup (pre_backup, post_backup, pre_pause, post_pause) render with backup.group and backup.member, so the quiesce and its release run in that member alone. They also render backup.atomic, true when SHC captures every volume it reads from that member as one atomic snapshot, decided before the quiesce fires. A quiesce can then skip what only a file-by-file copy needs: postgres runs a CHECKPOINT and keeps taking writes instead of turning its writer read-only, and its converter recovers the snapshot’s WAL. If that snapshot then fails, the backup is refused (X500380BKPATM) rather than copying a member that kept writing. The first job of post_backup is prepared before pre_backup fires, so a member that does quiesce is held only for the two commands. Write the quiesce so it refuses on a member it cannot make consistent, such as a standby that is behind: it prints ::shc-hook:refuse:: <why> and exits non-zero before it locks anything. SHC then runs that member’s post_backup, passes it over and reads the next ready standby, then the writer. With every member refused the run is refused (X409895BKPGRP), naming each member’s reason. A pre_backup that fails without the marker fails the run, as it does for any app. A group backup takes no pause of the deployment’s containers, since that freeze would cover the writer.

An app’s from: uses only labels that hold on every estate: group, member and service, which SHC derives, and the labels the app writes itself, on a service’s labels: or as a live label, such as postgres’s role. A named place (@edge), a node or location term or a label the app does not write differs from one estate to the next, so install refuses it (X400259BKPFRM). A backup whose from: picks a replica outside a group, or members of two groups, is refused with the same code. Reading the backup from a particular place, such as the far member, would be the operator’s setting on the deployment, not the app’s; it is not built.

A group whose members hold distinct data, each member its own shard, writes no converter for it: backup.yaml lists every member’s volumes under include, so each member’s volumes are captured in the same run and each restores into the member of its number.

A restore of a replicated group works at any member count. The archive records the member it read. The restore job writes the data into the target’s lowest-numbered member, here or on that member’s node, every other member’s directories are emptied so they clone from it, and a target whose members cannot be read is refused before anything stops.

shc backup create takes a group backup through the same engine as a scheduled one, wherever the member runs. A one-off --file backup and the daemonless backup read only the disk they run on, every member’s volume on it, so they refuse a converter that reads one member (X400259BKPFRM). MongoDB (SHC-1792, which retired mongodb-replicaset and librechat-db) is the first engine on the rule, then PostgreSQL, MySQL, MariaDB and Redis.

AppScale 1Scale above 1 today
ce modules/postgresone plain serverPatroni with etcd from two members, up to five, see PostgreSQL (SHC-1793)
mysqlone memberup to seven, see MySQL (SHC-1794)
redisone plain serverreplicas and three sentinels, automatic failover, the master handed over on scale-down, consumers on the group’s service selector (SHC-1796)
mongodba one-member replica setup to seven, an arbiter at two, the primary handed over on scale-down, see MongoDB (SHC-1792)
mariadbone plain server, wsrep offa Galera cluster up to seven, member 1 weighing 2 at two, see MariaDB (SHC-1795)
clickhouse, openreplay-chone serverrefused, x-shc-scale.max: 1 (SHC-1798)

Out of scope: caches and datastores an app bundles in its own compose (25 apps bundle valkey), Frappe’s bundled MariaDB, and the external-* apps, which point at a server SHC does not run.