Skip to content
SHC Docs

Groups and the members block (design)

Status: open design (SHC-1247). Members render and scale, and each runs as its own replica project (SHC-1259) on the node it is bound to: the render finishes each member for its admission- or row-bound node and that node’s runner starts it (SHC-1426, ce modules/app/replica_placement.go). A member’s sidecar VIP is built (section 4, SHC-1298), and shc status and shc doctor print the not-redundant line for a group whose members share a node (section 5, SHC-1276). Clay, 2026-10-02: a service scales by interchangeable replicas behind one virtual IP; a group scales by stamping distinct numbered members, each an ordinary service with its own name, VIP and volumes. “I think we should only support the loop approach, it’s more explicit and easier to work with.” No top-level x-shc-groups, no per-service x-shc-member, no short-form key. Aliases (db-rw, blue/green, hot/cold), members keyed by name, --drop of one member and an fqdn() helper are not ruled and are out of scope. This builds on the replica project contract (2026-10-02-replica-projects.md) and the VIP design (2026-10-02-service-vip.md) and edits neither.

services:
{% members "db", default=3, max=9 as m %}
pg-{{ m.n }}:
image: postgres:17
volumes: ["data-{{ m.n }}:/var/lib/postgresql/data"]
environment:
PGLOCATION: "{{ m.location }}"
exporter-{{ m.n }}:
image: prometheuscommunity/postgres-exporter
network_mode: service:pg-{{ m.n }}
{% endmembers %}
web:
image: example/web

With no --scale this renders three members: pg-1, exporter-1, pg-2, exporter-2, pg-3, exporter-3, and SHC declares the volumes data-1, data-2, data-3. web is an ordinary service. Every stamped service is an ordinary service with its own VIP (SHC-1238).

Words: the group is the thing you scale (db), a member is one stamped copy (db member 2), identified by its number.

{% members "<group>" [, default=<n>] [, max=<n>] [, separate="preferred"|"required"] as <name> %} ... {% endmembers %}
  • <group> is a string literal matching ^[a-z0-9][a-z0-9_-]*$. It has to be a literal so SHC can find every group before it renders anything.
  • default (an integer literal, 0 or more, default 1) is the member count a group starts with when no --scale names it. max (an integer literal, at least 1) is the ceiling; no max means none. default above max is a parse error.
  • separate (a string literal, "preferred" by default) is how strictly the members keep apart; §5 has what "required" does. Any other value is a parse error. Only the app states it: no deployment flag sets it.
  • <name> binds the member object inside the body: m.n is the member’s stable number, m.location the location SHC assigned it before rendering, or "" when the scale names no locations. Nothing else is on it.
  • The body is rendered once per member, in member-number order, with the rest of the context unchanged. Everything a pass emits belongs to that member.
  • A group appears once per template. A members block inside another members block, a {% for %}, a {% macro %}, a {% call %} or a {% filter %} is a parse error, because each would stamp a group more than once per render or away from where it is written.

These are parse errors (X400512JNJMBR), so shc validate, the catalog parse gate and an install all refuse them the same way, with the line.

The tag is a gonja control structure registered in ce core/jinja next to the loading refusals. The member lists reach it through a reserved context key (shc:members) that no template identifier can spell. With no list for a group (a lint render, a catalog sweep) it stamps members 1..default with no location, so every existing static check renders group templates unchanged. {% members %} emits template text and {{ }} outputs only, so it joins the tags Emissions can attribute and the scalar-shape rule (SHC-1042) still judges every value inside it, m.n included.

Which services belong to which member. The render whose bytes ship records what each pass wrote: RenderCollect’s own render for a tenant’s package, and the RenderComposeYAML render that replaces it for a catalog package. The instrumented passes RenderCollect runs afterwards to find unguarded misses never record, and an output Emissions brackets is recorded without its marks. A second render is never used, because gonja renders are not always reproducible.

Each pass records the byte range of the document it wrote, read from the render’s own output buffer, so nothing searches the document for a pass’s text: the same text written elsewhere is not the pass. Before anything reads the passes, each one’s range has to hold its text, in order. A pass whose output a tag captured instead of writing (a {% set %} block around the members block) has no range and is refused.

The document is taken apart, never searched. The rendered document is cut along the passes’ byte ranges into parts, and each part has to stand on its own:

  • each pass’s bytes are exactly one YAML document, a mapping whose keys are whole services, read as the value of a key of their own so the pass keeps its syntax, block or flow. Those keys are the member’s services. A pass that stamps none, writes a key whose value is not a service, is not a mapping on its own, or carries a document marker is refused;
  • the document with every pass removed is exactly one YAML document;
  • the render, decoded, has to equal the outside document with each pass’s services merged into its services as decoded values, never as text (the same text decodes to the same value, .nan included). The same has to hold with each group’s passes removed and with each pass removed;
  • a pass may not stamp a service the rest of the document already has.

Anchors are document-scoped, so this closes the class rather than one case at a time. A value a pass continues from another service, a key or value it writes into another service (another member’s included), a key under another section, a document marker, and template text after the block that lands in the last member’s service each fail one of the standalone parses or the merge. Comments inside the block are free. Member services may be named freely (pg-{{ m.n }}, replica{{ m.n }}-db): ownership is what a pass’s own document holds, not a name.

Anchors go one way. The anchors the outside document defines before a pass are handed into that pass’s parse, read only, so the compose idiom of x-common: &common and <<: *common inside pg-{{ m.n }} works. An anchor a pass defines may be aliased only inside that same pass: an alias outside it, or in another member’s pass, is refused, which is what stops two members defining the anchor an outside service aliases from hiding each other (scaling the group to nothing would change the outside service). An anchor written after the block is not handed in.

Every refusal here is X400097GRPEMT and names the member and the line of the rendered document it starts on, or for the outside document the line the parse stopped at. So adding or retiring a member never rewrites anything else, which is what lets members 1 to 3 keep their bytes and digests when 4 and 5 arrive.

A block inside an {% if %} that is false renders no pass. The group then has no members in that render and the manifest keeps no list for it; turning the branch back on starts the group like a first install, from its --scale value if the update names one, else its default. A group whose scale is 0 is different when its block ran: the render records which blocks it reached, and the manifest keeps an empty list for one that ran with no member. A block in a false branch keeps no list whatever its scale.

Each deployment keeps, per group, the list of its members: [{n, location, services}], the services being what the last render stamped for the member. It lives in the deployment’s manifest (manifest.json, the persisted state) under groups, beside scale_overrides. No new table.

Every path that renders a deployment reads it:

  • install and update read the manifest on the node that renders;
  • the stored spec carries the manifest whole (deployment_spec.spec.effective), and a rebuild from it (dead-source move) hands the lists to the install and to its hosts replay (SpecReplay.Groups);
  • a backup records the lists (groups in the backup manifest, read from the deployment’s manifest), and a recovery hands them to its install, because the export/import .bak never restores the manifest before that render.

A render that finds no list for a group is a deployment that never had one (a first install, or a package that just gained the block): it starts from the group’s --scale value, else its default.

Only a --scale for the group changes its members. A render that names no value for a group keeps its stored list exactly as it is: every member, every number, every location. The default is where a group starts, not where it returns to. Members hold data, so a group never shrinks because a flag was left out; shrinking is always a statement that names the group. The rule lives in one function (ce core/pod ResolveMembers). It is what keeps the paths that render without the deployment’s scale from retiring members:

  • a rebuild installs from the stored lists first and gets the deployment’s --scale values back in an update after;
  • a backup carries the lists and no scale at all;
  • the clone bring-up renders from the restored manifest and carries neither;
  • --scale is the whole statement of an update, so --scale web=2 names no value for db.

With a default that stood in for the missing value, a list of members 1, 2, 4, 5 under default=3 would lose member 5 and every location, and the value’s return would give the lost place to a new member 3.

This is a place where a group differs from a service, with two effects an operator will meet:

  • A package that changes default does not resize deployments that already run the group; --scale db=<n> does.
  • --scale web=2 resets an ordinary service it leaves out to its count, and does not retire db members. The run says so in words, so the difference is not silent: the install or update prints group db keeps 5 members; name it to change it (--scale db=<n>), its terminal frame carries groups_kept: [{name, members}], and the Terraform plan warns the same way. The line is read from what the render stamped, so a group whose branch turned false in the same run is not reported, and nothing is said when the kept list is what the default would give.

A kept list is still held to max: a package that lowers the ceiling under a deployment’s members is refused (X400093GRPMAX) until someone scales the group. Nothing is cut for them.

The lists a rebuild or a recovery carries stand over a manifest already on the node, so an install that is retried renders from them again and not from what its first attempt left.

When a --scale value names the group (typed on this install or update, or the deployment’s persisted value, which a bare update replays), the list is recomputed from the stored one before the render:

  1. A count (--scale db=5). Keep the stored members with the lowest numbers up to the count and retire the rest. Add new members with the lowest numbers no stored member holds. Every member’s location is "".
  2. Counts per location (--scale db=home:2,far:1). For each location, keep the stored members already there, lowest numbers first, up to its count. Then fill what is still missing in each location from stored members that have no location, lowest first, taking locations in order: the deepest first, then the one asking for the most members, ties by name. Retire every stored member not kept. Add new members for what is still missing, with the lowest numbers that no member, kept or retired in this pass, holds.

A member’s location is always a leaf. Since SHC-1266 a value may name a level (--scale db=incus:3). A member holds data, so it never floats inside a level: a count for a level keeps the stored members already somewhere under it, and gives each new or adopted member a leaf, going down one level at a time to the child holding the fewest of the group’s members, ties by name. incus:3 over incus/far and incus/home is far, home, far; a fourth member goes to home. The leaf is stored with the member, so it stays across later changes and no-scale renders. A deeper location named in the same value claims its members before the level does (incus/home:1,incus:2 puts one member at home and two more under incus, spread). The location paths are the fence store’s, every level of every node’s path, read only when a group’s value names a location, for the admission’s demands and the render alike.

So 3 to 5 adds 4 and 5, and members 1 to 3 render byte for byte as before, keep their spec digests and are not recreated. 5 to 3 retires 5 and 4, and a later 3 to 5 brings back 4 and 5 by number, with their durable volumes. --scale db=home:2,far:1 on a first install therefore numbers home 1 and 2 and far 3, whichever order the value is written in (--scale values are stored in their canonical, sorted spelling, so the typed order is not available to the rule).

A number never moves to another location inside one change, so a member whose location changes is retired and a new number takes the new place. Going from counts per location to a count releases every member’s location: each keeps its number, renders with m.location empty and loses its fence, and stays on the node it is bound to. A later counts-per-location adopts such members by number, and a number retired earlier can come back in another location. In both cases a durable volume holds the member to where its data is (the existing reach rule), so it waits with that reason rather than silently starting empty. Remembering retired members’ locations to avoid that is a follow-up, not built here.

A retired member’s services leave the rendered document, so its replica project goes with them and every one of its services stops together. Its durable volumes are kept, as for any service that leaves.

All services of one member are one pod named <group>-<n> (db-2), whether or not they share a namespace. This is a design decision beyond the ruling’s words, taken so that placement, status and retirement act on a member as a whole (AC5); it changes the replica projects contract, see section 11. A member’s service that shares a namespace or volumes_from with a service outside its member is refused (X400098GRPLNK), since that would weld members together.

A member is a set of services with a replica count of its own. In this ticket that count is always 1: the member pod has one replica (ordinal 0). A member service that authors deploy.replicas or scale other than 1, zero included, is refused as not supported yet (X400099GRPREP). Section 9 sketches how member replicas are added later.

A member’s service that joins another’s network namespace (exporter-1 in pg-1’s) has no fabric pin of its own, like any sidecar today. Its VIP (SHC-1238) has to target the namespace owner’s pin with the sidecar’s own readiness. The VIP design says so (its SHC-1298 paragraph) and ce builds it (modules/vip/section.go, modules/app/readiness.go MemberReady).

replica.json gains group and member; every entry carries the labels shc.group and shc.member beside shc.pod. Both labels are part of what an entry’s spec digest covers, and neither changes while the member keeps its number, so a member’s digest moves only when its own services do. A member pod is not stateful in the x-shc-scale sense: its volumes already carry the member number because the template wrote them that way, and the ordinal suffix would only add -r0.

Volumes. A named volume referenced inside the block (data-{{ m.n }}) is declared by SHC per member, the same way it declares any referenced named volume today (AutoDeclareVolumes), now for the long source: form too. No top-level loop. Whether it is durable follows backup.yaml exactly as for any volume (SHC-1231): with a backup.yaml and no include.mounts allowlist it is kept. Naming a member volume by its base name in backup.yaml is a follow-up.

A member pod’s replica demand is the member’s location as a fence: a member assigned home gets fence=location:home, built by the same scale shape that counts per location use for a service’s ordinals, and held there exactly (the demand’s Exact, nodes whose own location is that leaf), so it is placed only in home and never moves out on its own. Matching a short location name to a hierarchical fence (location:incus/home) is SHC-1260’s: the scale value is made canonical where it is parsed, so m.location and the fence carry the full path and groups add no resolution of their own. A member with no location can land anywhere the deployment’s rules allow.

Siblings prefer separate nodes. Every member demand carries its group, and the engine adds an implicit soft spread=node,skew=1 counted over the group’s members rather than per pod, so with enough nodes no two members share one, and with too few they co-locate instead of waiting. No operator rule is written. The placement decision names a group whose members share a node as not redundant, by member, and shc status and shc doctor print that line for a group whose members share a node (SHC-1276, ce modules/scheduler/shared.go). A demand that names no group, as failover’s and a move’s do, takes it from the placement snapshot, which reads it from the stored spec, so the preference holds on every placement, not only at install and update.

separate="required" makes it hard (Clay, 2026-10-01). The engine gives a required group’s members a hard node,max=1 counted over the group in place of the soft spread, so no placement ever puts two of them on one node. A member that has no node of its own waits: admission refuses an install, update or scale that asks for more members than the cluster has eligible nodes, with both numbers (X409184FNCSEP), and a failed-over member stays pending with a hold that says it needs a node holding no other member. An update that makes a group required while two of its members share a node is refused with shc cluster rebalance --apply in the error (X409185FNCSHR); nothing moves members because an app version says so. shc doctor reports a required group’s sharing as an error. The render stores the declaration beside the member lists (group_separate in the effective state), and the placement snapshot and the scheduler’s group read both take a spec’s groups through one parse (ce pod.ReadStoredGroups). A spec without it, every one v0.1.21 stored, reads as preferred.

Clay’s ruling also has members move apart when a node frees or joins. The engine keeps a bound replica on its node while the node stays eligible, and spread only scores a replica that has to be placed. So the move is a separate step, made two ways: shc cluster rebalance moves co-located members onto separate nodes first (SHC-1218), and the leader’s separation pass makes the same moves on its own when a node joins, frees room or comes back after a failover. Scheduler design §3.19 has its gates, pacing and announcement. Both read each member’s group from the stored spec (ce scheduler.GroupsOf), because the assignment rows carry none.

Before the render the admission gate reads a group template’s members: one demand per member pod <group>-<n> with its location, decided as a first install decides them. Like the rest of the pre-render read, it keeps every group whatever branch surrounds it and reserves nothing for a member, whose services are not known until the render; a group scale it cannot take leaves the template to the render, which refuses it.

Spreading members across nodes falls under the clustering license key like the rest of clustering.

The flag is the existing repeatable --scale on install and update, one name per flag, commas for the per-location split. A group takes two of the three shapes:

valuemeans
--scale db=5five members, anywhere
--scale db=home:2,far:1members in those locations, by count
--scale db=incus:3three members under the level incus, each assigned a leaf
--scale db=per:locationrefused (X400094GRPSHP): a group takes a count or counts per location

The value persists in scale_overrides under the group’s name. A total above max is refused naming the group and its ceiling (X400093GRPMAX). A group and a service may not share a name (X400096GRPDUP); nor may a member pod’s name <group>-<n> be a service outside the group.

--scale on a service a group stamps is refused with the group to scale instead (X400095GRPSVC): “--scale pg names a service group db stamps; scale the group: --scale db=”. Ownership is the render’s record, so the stamped name (pg-2) is always caught. A base name (pg, a stamped name less its member’s -<n>, _<n> or <n>) is caught only when it is not a service of its own: an ordinary pg beside a group’s pg-1 scales as an ordinary service. A name that matches neither is left alone, as an unknown --scale name always is.

codewhen
X400512JNJMBRthe tag is malformed: name, an unknown argument, default over max, a separate other than "preferred" or "required", nesting, inside a loop, macro, call or filter, a group twice
X409184FNCSEPa separate="required" group asks for more members than the cluster has eligible nodes
X409185FNCSHRan update runs a separate="required" group while two of its members share a node
X400093GRPMAXa group’s scale, or the list it keeps, is above its max
X400094GRPSHPa group’s scale is per:<kind>
X400095GRPSVC--scale names a service a group stamps
X400096GRPDUPa group shares a name with a service, or a member pod name is a service outside the group
X400097GRPEMTa pass is not one YAML mapping of whole services on its own, the document without the passes does not parse on its own, the render is not those parts merged, an anchor a pass defines is aliased outside it, or a member stamped none
X400098GRPLNKa member’s service shares a namespace or volumes with a service outside its member
X400099GRPREPa member’s service authors a replica count other than 1, which is not supported yet

x-shc-groups and x-shc-member stay unregistered, so a template that writes either is refused as an unknown directive (X400330XSHUNK) like any other.

shc ls lists, under each deployment that has groups, one line per member: the group, the member number, its location, and its readiness (ready when every service of the member is running and, where it has a healthcheck, healthy; a service whose health could not be read is not ready). -o json carries the same as groups: [{name, members: [{n, location, services, ready}]}]. shc logs and shc restart take a member’s service name (pg-2), which is an ordinary service.

Clay’s own model has members that carry replicas of their own: the read-only members of a database, each with copies behind its own virtual IP. Nothing here builds that, and nothing here closes it off. The shape it would take:

  • A count per member, from the template (deploy.replicas on the member’s services, or a replicas= argument on the tag) and later from --scale. The member pod <group>-<n> then has ordinals 0..k-1 like any pod, each held to the member’s location.
  • Each stamped service keeps its one VIP (SHC-1238), now over that member’s copies, so pg-2 stays one name however many copies member 2 runs.
  • Durable volumes decide what a copy is. A volume shared by the copies needs storage that can share; otherwise each copy needs its own (data-2-r1), and which copy holds the member’s data is the member’s own business (the app’s manager, not SHC). That choice is the open part, and why it waits.
  • The group spread still counts members, and the copies of one member get the ordinary per-pod spread on top.

The tag, attribution, the list rules, member pods, records, the spread and the refusals are pure and tested in ce through the shc tests/_ce corpus. AC2’s “containers are not restarted” is proved there as byte-identical entries and unchanged digests for members 1 to 3, and AC5’s retirement as the member’s replica project leaving the deployment; that no container of members 1 to 3 restarts and that every container of a retired member stops are runtime facts, so a live roll on the Incus rig is owed and named on the ticket. AC3 and AC4 are proved against the placement engine with a fleet of two locations and several nodes; a live multi-node proof is owed for the same reason. A node runs replica projects from its assignments since SHC-1426, so the rig can run both.

  • Replica projects section 1 and 5: a member’s services are one pod named <group>-<n> whatever their namespaces; replica.json has group and member; entries carry shc.group and shc.member.
  • Replica projects section 9: when the stateful replacement is ruled, pod.Scaled.Stateful takes its value from it. Groups are that replacement for distinct members; x-shc-scale: {stateful: true} remains as merged until someone removes it.
  • Scheduler design: the implicit group spread joins the implicit spreads in 3.6.
  • VIP design: the backend of a service that joins another’s network namespace (section 4). Paid: the VIP design’s SHC-1298 paragraph says it.
  • SHC-1218: its rebalance moves a group’s members apart as it does a service’s replicas (section 5).
  • Status and doctor: the not-redundant line for a group’s members sharing a node (section 5). Paid by SHC-1276.