Groups
Layer 6. Services. Services
New line, in development. This page describes SHC as it is on the
devbranch, not the released product. See The new line.
A service scales by interchangeable replicas. A group scales by numbered members, and each member is its own set of services with its own names and volumes. Use a group when the copies are not interchangeable: a database with one primary and two secondaries, a quorum, anything where copy 2 holds different data from copy 3.
| Word | Meaning |
|---|---|
| group | The thing you scale, named in the template (rs). --scale names it. |
| member | One stamped copy of the group’s services, known by its number (rs member 2). The number never changes while the member exists. |
| members block | {% members %} ... {% endmembers %} inside services:, the only way to write a group. There is no x-shc-groups and no x-shc-member; both are refused as unknown directives (X400330XSHUNK). |
The full syntax and every refusal code are in the design, Groups and the members block. This page walks through one example. mongodb ships the block: one member by default, --scale mongodb=N up to seven, an arbiter at two, and its socket fronted by the service selector named for the group. mongodb-replicaset and librechat-db are retired into it (SHC-1792). Checked against ce db676cf4aa0b, pinned by shc 3c88a263038b (core/jinja/members.go, core/pod/members.go, modules/app/groups.go).
A three-member replica set
Section titled “A three-member replica set”# compose.yaml of an app called replicaset (shape only)services:{% members "rs", default=3, max=7 as m %} mongo-{{ m.n }}: image: percona/percona-server-mongodb:7.0 command: ["mongod", "--replSet", "rs0", "--bind_ip_all"] volumes: - data-{{ m.n }}:/data/db expose: - "27017" environment: MEMBER_LOCATION: "{{ m.location }}" healthcheck: test: ["CMD", "mongosh", "--quiet", "--eval", "db.adminCommand('ping').ok"] interval: 10s deploy: resources: limits: cpus: "1" memory: 1G{% endmembers %}With no --scale, a first install renders three members, because default=3:
| Member | Service | Volume |
|---|---|---|
| 1 | mongo-1 | data-1 |
| 2 | mongo-2 | data-2 |
| 3 | mongo-3 | data-3 |
SHC renders the block’s body once per member, in number order, with m.n the member’s number and m.location the location it was given ("" when the scale names none). It declares data-1, data-2 and data-3 at the top level itself (modules/app/install_runner.go AutoDeclareVolumes), so write no top-level volumes: for them. Each member’s data lives in its own named volume. A durable volume on local disk holds that member on its node; numbering alone does not make data portable.
Auth is left out above to keep the group visible. A real replica set also needs a keyfile and a step that runs rs.initiate and adds each member: that is the app’s work, not the group’s.
What a member is called and how it is reached
Section titled “What a member is called and how it is reached”A member’s services are ordinary services named as the body wrote them: mongo-2 is member 2’s database.
- Inside the deployment, any service reaches it by that name:
mongo-2:27017. - The deployment’s containers, and an integration consumer’s, also reach it by its scoped name,
<environment>.<service>.<stack>:prod.mongo-2.mongo-rsfor a deployment calledmongo-rsinprod. On theshcfabric the node resolver answers that name with the service’s VIP to every asker; off the fabric it’s a network alias SHC puts on each of the service’s network entries (install_runner.goserviceNameAliases). - Its full name in the internal zone is
mongo-2.mongo-rs.prod.<tenant>.shc.internal(modules/resolverServiceName), answered by the node resolver, which the render sets as a service’sdnswhen the node has one. Other tenants are refused, and a service that declares no port is private to its deployment. - On a cluster on the
shcfabric each service, a member’s included, also gets its own virtual IP (modules/vipActive). shc logs mongo-2 -s mongo-rsandshc restart -s mongo-rs -S mongo-2act on one member’s service.
A client lists the members it knows, the usual MongoDB way: mongodb://mongo-1:27017,mongo-2:27017,mongo-3:27017/?replicaSet=rs0.
Inside SHC, member 2’s services make up one pod named rs-2 (core/pod/members.go MemberPod). That name is what placement and the replica records use; you never write it.
See Replicas and pods for how SHC renders a pod and which nodes run its replica projects.
Scaling
Section titled “Scaling”Scale the group, never its services:
shc install ./replicaset -s mongo-rs --scale rs=5shc update -s mongo-rs --scale rs=3- Numbers are stable. They are a stored list per deployment (
core/pod/members.goResolveMembers). Going from 3 to 5 adds members 4 and 5 and leaves 1 to 3 exactly as they were, with the same rendered services. - Going down keeps the lowest numbers: 5 to 3 retires members 5 and 4. Their volumes are kept, and scaling back up brings them back by number.
- Retiring a member stops its services; what that means to the data is the app’s. An engine that keeps a membership list needs a
pre_updatehook that removes the retiring members first, or its set keeps a primary only while the running members are a majority of the largest count it ever ran: 5 to 3 is safe, 5 to 2 leaves no primary while every healthcheck passes.mongodb’s handover job does that removal (SHC-1792). maxis the ceiling:--scale rs=9withmax=7is refused (X400093GRPMAX).- Only
--scale rs=...changes the group. An update that names no value for it keeps every member it has, and raisingdefaultin a new app version does not grow a deployment that already runs the group. The exception: a block inside an{% if %}that renders false keeps no member list, so when it renders again the group starts over from its--scalevalue, else itsdefault(modules/app/groups.goAttributeGroups). --scale mongo-2=...is refused (X400095GRPSVC): scale the group.- A member’s service may not set
deploy.replicasorscaleabove 1 (X400099GRPREP). One copy per member for now.
Locations
Section titled “Locations”shc update -s mongo-rs --scale rs=2@home,1@farThis holds two members to home and one to far. Members already in a location stay there; new members take the lowest free numbers. A plain count (--scale rs=5) clears every member’s location; to keep them, scale with counts per location. A location that is a level (incus holding incus/home and incus/far) gives each member a leaf under it, spread over its children, and keeps it there: a member holds data and never floats inside a level. A group takes no <n>/<kind> shape (X400094GRPSHP: a count or counts per place, not that value). The member sees its location as m.location, which the example passes to the container.
Where members run
Section titled “Where members run”Placement records a node for every member and prefers a node no other member of the group holds; with too few nodes, members share rather than wait, and SHC reports the sharing as not redundant (core/placement/engine.go SharedGroupMembers). A member with a location is placed only there.
An app whose members must never share a node says so on the block:
{% members "rs", default=3, max=7, separate="required" as m %}separate is "preferred" (the default) or "required". A required group never puts two members on one node, on any placement: install, update, failover, a move or a rebalance (core/placement/spread.go serviceSpreads, a hard node,max=1 over the group). Only the app declares it; no deployment flag turns it on or off.
- An install, update or
--scalethat asks for more members than the cluster has eligible nodes is refused, with both numbers:X409184FNCSEP, “the cluster has fewer eligible nodes (1) than group rs has members (3)”. A single-node estate can’t run a required group of more than one member. - With enough nodes but no room on the ones a member could take, the install waits like any other (
X409179PLCPND). An update never finishes such a member on the deployment’s own node beside a sibling. - A member whose node is lost waits for a node holding no other member. Its failover hold says
needs a node holding no other member of rs (separate="required"). - An update that makes a group required while two of its members share a node is refused (
X409185FNCSHR). The error names the members, the node andshc cluster rebalance --apply, which moves them apart. Nothing moves on its own because an app version says so. shc doctorreports a required group’s members sharing a node as an error; a preferred group’s sharing stays a warning.
The render stores the declaration beside the member lists (group_separate in the stored spec’s effective state), and every reader takes both through pod.ReadStoredGroups. A spec stored before SHC knew separate, as every v0.1.21 spec is, reads as preferred.
On a fresh install, admission binds each member to a node. The render finishes that member’s replica project for the bound node’s id, location and runtime, including its fabric pins and deployment networks (modules/app/replica_placement.go, replica_projects.go). A dispatched install carries those bindings too (daemon/services_install_placement.go). The runner on the bound node starts the project and refuses one finished for another node (modules/pods/runner.go). Cross-node member placement is built (SHC-1426).
This requires a Linux node on the shc fabric with service VIPs, or a Mac node on the apple runtime (SHC-1557). A node that declines replica projects runs the deployment’s one compose document instead. shc doctor check pods.replica-projects reports the decision (modules/pods/doctor.go).
An update keeps members where they run. Admission places nothing on an ordinary update, so the render finishes each member for the node its live assignment row binds it to, its highest live incarnation, instead of the node that renders (modules/app/replica_placement.go heldReplicas, SHC-1628). A replica with no row and no binding is finished for the render node. DNS stays the render node’s resolver. Global services separately expand into projects for held bindings and eligible ready nodes with room (expandGlobalReplicas).
Install and update render on the cluster leader (SHC-1618): an update and an install from a catalog or git source are dispatched there (daemon/install_leader.go renderLeaderNode, modules/app/update_dispatch.go updateOnHolder), and an install from a path or an archive renders on its placed node, which holds the bytes (daemon/services_install_placement.go). Every render also stores the deployment’s compose before any of it is finished for a node, and the leader finishes that stored render for another node with nothing rendering there (modules/app/stored_render.go FinishStoredRender, SHC-1616). That path carries a failed-over replica to its new node (daemon/scheduler_deliver.go), a move to its target (FinishStoredMove, SHC-1618) and a node joining a global deployment (finishGlobalRerender, SHC-1146).
Witnesses
Section titled “Witnesses”A vote outside the members block declares which group it witnesses:
services:{% members "db", default=2 as m %} pg-{{ m.n }}: image: postgres:17{% endmembers %} postgres-witness: image: etcd x-shc-scale: witness: dbwitness takes one group name declared by this app. A service inside a members block cannot declare it. Lint refuses a missing group or a member declaring a witness, naming the service and group. In shc.yaml it is scale: {witness: db} on the service.
Placement prefers a node holding none of that group’s members when one has room. With only two nodes, or no room on the third, the witness still places and SHC reports the group as not redundant, naming the witness and member sharing a node. The witness does not change how data members spread. Existing bindings stay put until a move or reschedule; the stored spec preserves the declaration for that placement too.
This is an app property. There is no operator flag. Database apps should declare it on postgres-witness, the MongoDB arbiter and the Redis sentinel services, outside their members blocks.
Seeing the members
Section titled “Seeing the members”shc status -s <deployment> prints the deployment’s groups under a Groups title, one line per member: group, member, location, ready, the service selector named for the group on the member it points at, and services (modules/health/commands/status_groups.go). They come from shc.status.get’s groups, which the health service stands from the stored spec (app.GroupLayouts) and the replicas of the same answer. A member is ready when every replica it runs as reads ready. When the deployment declares a service selector under the group’s name (x-shc-service-selectors), the section names it on the member it points at: the only member of a one-member group while that member is ready, otherwise the member whose first replica the service selector picks and reads ready, and none while no member or more than one does, which the section says under its table. -o json carries the same under groups; shc status with no deployment named lists no members. shc ls prints no member table, and its -o json keeps each deployment’s groups as the wire carries them.
Reading the member list
Section titled “Reading the member list”Every render carries groups.<name>, the group’s members in number order, each with n and location (ce core/jinja/members.go GroupsKey, SHC-1790). It is readable inside the block, outside it, and in the hook jobs, socket jobs and other jobs rendered for the same deployment. A pre_update job sees the list the update is moving to, so on a scale-down it can tell which members are about to retire while they still run. A render with no stored list, such as a lint or a catalogue sweep, sees members 1 to default.
services:{% members "db", default=1, max=5 as m %} pg-{{ m.n }}: image: postgres:17{% endmembers %}{% if groups.db|length == 2 %} witness: image: etcd x-shc-scale: {witness: db}{% endif %}Here the witness exists only while the group has exactly two members. Keep a member’s own services free of groups: a member whose rendered bytes change with the count is restarted on every scale change, because the node takes a changed pod down before it brings it up (ce modules/pods/runner.go). If a member’s server really has to change when the group gains HA, let it change only at the edge between one member and two. Database apps, HA by scale uses this for every catalogue database.
Service selectors
Section titled “Service selectors”A group’s own name, and any other name an app wants, is a service selector: a name plus a selector over the labels of the deployment’s replicas (ce modules/vip, SHC-2459). Declare the service selectors at the top level of the compose file and a live label on each member for them to read:
x-shc-service-selectors: postgres: role=primary:1 postgres-ro: role=replicaservices:{% members "postgres", default=1, max=7 as m %} postgres-{{ m.n }}: image: postgres:17 expose: - "5432" x-shc-live-labels: role: test: ["CMD-SHELL", "psql -U postgres -Atc \"select case when pg_is_in_recovery() then 'replica' else 'primary' end\""] interval: 5s timeout: 3s{% endmembers %}x-shc-live-labels maps a label key to a command. SHC runs it inside the container every interval, and the first line it prints is the label’s value on that member’s replica: here role=primary or role=replica. A run that exits 0 and prints nothing a label may hold makes the label absent. A run that times out, cannot start or exits nonzero could not ask, and the label keeps its value. interval is 30s unless stated and never under 5s, and timeout 5s unless stated and below the interval. SHC never knows what primary or replica means. The app measures it, and the app’s own service selectors act on it.
x-shc-service-selectors maps a name to a selector in the node selector grammar, with &, |, != and parentheses. It reads service facts only: service, ordinal, group, member, the app’s compose labels and the live labels. location and node are node selector terms and are refused here (X400454ALSDCL). A trailing :1 is a count: exactly one replica may answer (x-shc directives has the whole grammar). Each name answers as one virtual IP on the shc fabric:
postgresfrom any service of the deployment;postgres.<stack>.<environment>.<tenant>.shc.internalfrom the tenant’s containers;- the socket’s hostname from a consumer, when the app’s socket says
service: postgresor is named after it.
| The service selector | Where it goes |
|---|---|
role=replica | Every ready replica whose labels match, spread like a service’s replicas. |
role=primary:1 | The one ready replica whose labels match. |
| no replica matches | Nowhere. The connection is refused, as for any VIP with no ready replica, and shc status -s names the service selector and says no replica matches role=primary. |
:1, and two replicas match | Nowhere, for as long as both do. It never reaches two, and shc status -s names the replicas that both match. |
Each member keeps its own name and its own virtual IP beside them. The same shape gives a writer and its read replicas, an etcd or Kafka controller’s leader, or a Redis Sentinel master: the app prints what the member is, and the service selector picks by it. A service selector is not tied to a group. web-canary: service=web & track=canary is one too, where track is a compose label.
What to know before you rely on it:
- A service selector may have the group’s name, and that is how a socket named after the group follows its primary. It may not have the name of a service or of a member’s pod (
postgres-2), and the render refuses one that does (X400454ALSDCL). - Name each member’s main service
<group>-{{ m.n }}. A member answers a service selector through the service its pod is named for, and a member’s pod is<group>-<n>. - A service selector reads every replica of the deployment. If a service outside the group prints the same label, hold the service selector to the group:
group=postgres & role=primary:1. - Print the label on one plain member too. At one member postgres prints
primarywhenever it accepts connections, sopostgresanswers from the first install. A service selector with no match is refused, so a label that only prints once the HA pieces run leaves the service selector refused until they do. At one memberpostgres-rohas no replica and is refused. - A label holds down a change. A new answer is reported once two passes in a row give it, so a value that flaps between passes moves nothing and one failed or slow run takes no label away. A run’s first answer is reported at once. After the engine has moved its primary, expect the service selector to follow within two intervals plus a few seconds. Between the old primary’s last
primaryand the new one’s first, a service selector with:1sees two or none and is refused. - A node the cluster no longer sees ready counts for nothing. Its replica is not a backend and its last answer does not hold a service selector with
:1closed, so the member the engine promoted takes over. - A scale-down that drops the primary closes a service selector with
:1for the length of the update. The update’s member list is in force beforepre_updateruns. If a member that list dropped still saysprimary, the service selector is refused from then until the update retires that member, after your hand-over has moved the primary to a kept one. Hand the primary to the lowest member before you scale down if the gap matters. - SHC does not cut connections that are already open. The engine does that when it demotes a member.
- The group has no container. A
context: servicejob names a member’s service (service: postgres-1) and dials a service selector from there.
role is the app’s own label. SHC derives no role, so a backup’s from: and any selector read the value the app printed. A backup that reads one member of a group prefers a member the group-named service selector with :1 does not pick.
Rules worth knowing before you write one
Section titled “Rules worth knowing before you write one”- The body of one pass must read on its own as a mapping of whole services, and the file must read without the block. Write each member’s services whole inside the block. A pass may merge an anchor written outside the block before it (
<<: *common); an anchor written inside the block is for that member only (X400097GRPEMT). - A member’s service may share a namespace or
volumes_fromonly with services of its own member (X400098GRPLNK). - The group’s name may not be a service’s name, and a member pod name (
rs-2) may not be a service outside the group (X400096GRPDUP). - The block may sit inside an
{% if %}, never inside another members block, a{% for %}or a macro, and a group is written once per template (X400512JNJMBR).
Designed, not built
Section titled “Designed, not built”- Replicas inside a member, the read-only copies of one member behind its own virtual IP: the groups design, section 9.
- Dropping one member by number, and an
fqdn()helper: not ruled.
A service stays its replicas plus one VIP. Every other name is a service selector the app declares (above).