Skip to content
SHC Docs

Replicas and pods

Layer 6. Services. Services

New line, in development. This page describes SHC as it is on the dev branch, not the released product. See The new line.

Checked against shc 0b6ee3547994, which pins ce f9ef568b3746. The code references below are in ce; the cited tests live in shc.

A pod is a set of services that SHC places as one unit. Each replica of that pod gets one compose project, which pods.Runner starts and stops as a whole. The runner’s replica move operations act on that same project (core/pod/scale.go:165, modules/pods/runner.go:903, modules/pods/move.go:10).

SHC joins services into a pod when they name another service through any of these fields:

  • network_mode: service:<service> shares its network namespace.
  • pid: service:<service> shares its process namespace. podman-compose hands it to podman, which has no service: mode, so such a pod is refused on a podman node (X400797EXEPDM) and placed on a docker one.
  • ipc: service:<service> shares its IPC namespace. SHC starts the service it names with ipc: shareable, which docker needs before a second container may join. podman-compose reads no ipc key, so such a pod is refused on a podman node (X400797EXEPDM) and placed on a docker one.
  • volumes_from: [<service>] mounts its volumes, including the :ro form.

These links are transitive. A service with none forms a pod of one. SHC names the pod after the first service in name order that doesn’t join another service’s network namespace (core/pod/pod.go:38, :81). The grouping tests (tests/_ce/core/pod/group_test.go) cover each link and the naming rule.

Two services mounting the same named volume don’t become one pod through that mount alone. SHC records a separate placement constraint for pods sharing a node-local volume or a data bind under the deployment’s mounts directory. Their replicas must share a node, but keep separate projects (modules/app/replica_pods.go:200, core/placement/durability.go:50). See the shared-volume tests (tests/_ce/core/placement/durability_test.go).

Numbered groups have another grouping rule: all services stamped for one group member form its pod (core/pod/members.go, GroupMembers).

Every replica project has one entry per app service in its pod, named <service>-r<ordinal>. Ordinals start at 0. For example, replica 1 of a pod containing web and side has web-r1 and side-r1. An Incus replica also has SHC’s health watcher when its pod runs a service, or a one-shot that declares a healthcheck or that another service awaits with condition: service_healthy (core/pod/healthd.go, NeedsHealthd). A replica without one has no watcher at all: incus-compose makes none of its own. A healthcheck that still runs there, one from the image, is run by the node agent itself, which writes the result where the watcher would. The same goes for any other project that reaches Incus with no watcher, such as the node’s collector (core/executor/incus/health_probe.go).

The project name is <deployment-project>-<pod>-r<ordinal>. The deployment prefix is <tenant>_<environment>_<stack>, with hyphens, dots and spaces replaced by underscores. The pod part folds uppercase to lowercase and other unsupported characters to underscores. A later incarnation adds -i<number> to the project name, while service entries keep their ordinal names (modules/app/install_runner.go:4801, core/pod/record.go:153, core/pod/incarnation.go:12).

The render writes replicas/<pod>-r<ordinal>/compose.yaml and replica.json under the deployment directory. It rewrites references between pod services to their entry names and removes scale, deploy.replicas and deploy.mode from the entries. SHC creates the copies; compose doesn’t scale them again (modules/app/replica_projects.go:654, core/pod/project.go:302). The project tests (tests/_ce/core/pod/project_test.go) pin these transformations.

The daemon decides once per process whether this node runs replica projects. A Linux node with a usable runtime that attaches to the shc fabric, a valid service-VIP configuration and a running VIP applier enables them, and so does a Mac node on the apple runtime with a running pf VIP applier (SHC-1557). When enabled, deployments run through the replica runner, and SHC refuses the deployment’s whole compose project. A node that decides against replica projects runs each deployment as its one compose project (subsystems/impl/replica_projects.go:142, modules/pods/local.go, RefuseWhole).

The apple runtime takes no kernel fabric attachment; its replicas are pinned by the address apple assigns (subsystems/impl/replica_projects.go). A node that decides against replica projects is not refused: it runs each deployment as its one compose project, logs X200769PODOFF with the reason, and shc doctor warns. Missing node identity, unreadable or missing posture, and a VIP applier that hasn’t started leave the decision pending. Installs wait and whole-project verbs refuse until the daemon decides. Once it decides, it keeps that choice for its lifetime. See the boot decision tests (tests/_ce/subsystems/impl/replica_projects_test.go).

Run shc doctor and read pods.replica-projects. It reports enabled, off with a reason, pending with a reason, or that the answering process runs no replica projects (modules/pods/doctor.go:23). An off diagnosis can name X409249VIPNSF for a different fabric, X409243VIPUNR for an unreadable address configuration, or X409250VIPOVR for overlap with the VIP range (modules/vip/require.go:16).

shc status -s <stack> includes replica, node and state rows when available. shc ls shows observed deployment nodes. Neither is the boot-switch diagnosis; use the doctor check for that (modules/health/commands/main.go:731, modules/health/commands/status_replicas.go:3, modules/app/commands/stack_lifecycle.go:473).

--scale on any service of a pod scales the whole pod. If several services state overrides, their shapes must agree. Without an override, SHC reads authored deploy.replicas or scale; conflicting authored counts refuse, and no count means one replica. The smallest x-shc-scale.max in the pod caps it (core/pod/scale.go:44).

Flag valueMeaning
--scale web=2Two replicas of web’s pod. A plain count can be 0.
--scale web=2@east,1@westTwo replicas constrained to east and one to west. Each location count must be at least 1.
--scale web=<n>/<kind>n replicas in every fence of the kind: each leaf location for location, each node for node, each value for a label such as pool. 1/location is one per location. n is 1 or more. A fence with fewer nodes than n puts more than one replica on a node, and shc explain names the fence.

After @, a counted place names a location path. An unambiguous suffix can select a full path; a parent path permits placement beneath it. Parenthesised selectors aren’t accepted here. See CLI symbols and core/placement/scale.go:44, :229.

For ordinary counts, the render uses ordinals 0 through n-1. Scaling 2 to 3 adds r2; scaling back to 2 keeps r0 and r1. Locations sort by name before SHC assigns ordinal ranges. Changing the location counts or the fence list can therefore change an ordinal’s location constraint. This isn’t the stored member identity of a group (core/placement/scale.go:159, :256, core/pod/project.go:258).

deploy.mode: global takes a separate path. SHC expands it from existing assignments and eligible ready nodes with room for the service’s reservation. The global placement planner keeps held ordinal-to-node assignments and gives joining nodes ordinals after the highest held number. A full node gets no new replica and produces X409679SCHGLB. The render derives the global service’s count from these bindings (modules/app/global_replicas.go:45, core/placement/global.go:21). The global render tests (tests/_ce/modules/app/global_replicas_test.go) compare those ordinals with the planner.

For a new deployment, admission places the app’s pods and returns a binding per replica. replicaPlacer uses those bindings to finish each document for its node and location. That node’s runner starts it after receiving the projection. The installing node starts only its own replicas and doesn’t wait for remote replicas (modules/app/service.go:694, modules/app/replica_placement.go:15, modules/app/install_runner.go:2265). The delivery test (tests/_ce/modules/app/replica_delivery_test.go) checks starts on three separate fake nodes.

A dispatched install preserves those bindings in the internal RPC envelope as placed_replicas. The receiving node restores them before running the normal install path (daemon/services_install_placement.go:131, :204). The dispatch test (tests/_ce/daemon/install_replica_spread_test.go) checks this across serialisation.

An ordinary update places nothing through admission, so the render keeps the spread from the assignment rows instead: heldReplicas adds, for each replica without a binding, the node its live row binds it to, the row of its highest live incarnation. replicaPlacer then finishes that replica’s document for the node that runs it, not for the update’s render node (modules/app/replica_placement.go heldReplicas, modules/app/install_runner.go:2170, :2259; SHC-1628). A replica with no live row is finished for the render node. Global services still receive bindings from expandGlobalReplicas.

A rollout, a move, a drain and a scale-down each retire the incarnations they replace. A retired incarnation’s assignment row stays, marked retired, and its fabric pins stay live until its node has taken the replica down. The leader deletes a retired replica row once it retired 24 hours ago (RetiredRetention in modules/scheduler/compact_retired.go), up to 1,000 rows a tick. It keeps a row an unreleased pin still names, each replica’s highest incarnation, so the next start’s number stays above every earlier one, and every stack row. The table then holds the live rows, the last day’s retired rows and one retired row for each replica that has no live row.

The internal service name is <service>.<stack>.<environment>.<tenant>.shc.internal. SHC writes the deployment zone to replica entries’ dns_search, and the node’s DNS answers a name with no dot from a replica in that replica’s own deployment zone, so a short service name resolves within the deployment on every runtime. That holds in a musl image such as alpine too, which never tries the search list at docker’s ndots:0 (modules/resolver/table.go inOwnZone, core/pod/project.go builder.attach, SHC-1766). A short name no service of the deployment has goes to the node’s upstream DNS, as it would without SHC.

For a container allowed to see a service, the node resolver returns that service’s VIP while the node routes the VIP revision. Host queries receive the ready replica addresses instead. A VIP answer can exist even when no replica is ready (modules/resolver/table.go:434).

A replica asking for its own service by the bare name or the full internal name gets its own address, as it would under compose, whether or not a VIP routes and whether or not it is ready yet. The same holds for a service sharing its network namespace. So a compose file that binds the name it advertises works unchanged on every runtime: huly’s redpanda runs --rpc-addr redpanda:33145 and listens on an address its container holds. A VIP is an address no container holds, so binding it would fail with Cannot assign requested address. A group’s or an alias’s name follows one of its members and is never a member’s own: a standby asking for its group is answered the group’s VIP (modules/resolver/table.go own, SHC-2607).

The scoped name, <environment>.<service>.<stack> ({{ hosts.<service> }}), answers the VIP to every asker on every node, the service’s own replica included. On the fabric no service keeps it as a network alias, so docker never answers it with a node-local bridge address first (install_runner.go serviceNameAliases). It’s the one name whose address a member and its peers agree on, so a member that announces or compares addresses asks it: redis’s bin/lib.sh resolves every member and sentinel by its scoped name, from the ENVIRONMENT and STACK its compose passes. See the own-name tests (tests/_ce/modules/resolver/own_name_self_test.go) and tests/_ce/core/jinja/redis_scoped_addr_test.go.

web-r0 and web-r1 are compose entry names, not separate per-replica records in SHC’s internal DNS. The resolver groups replica pins under the base service name, subject to readiness (modules/resolver/source.go:325). See the replica DNS tests (tests/_ce/modules/resolver/replica_source_test.go).

Suppose an app called demo has this compose file, with tenant acme, environment prod and stack demo selected:

services:
web:
image: nginx:alpine
expose: ["80"]
x-shc-scale:
max: 2
side:
image: busybox:1.37
command: ["sleep", "3600"]
network_mode: service:web
Terminal window
shc install demo --scale web=2

This is one pod named web, with two replica projects:

OrdinalProjectEntries
0acme_prod_demo-web-r0web-r0, side-r0
1acme_prod_demo-web-r1web-r1, side-r1

These are the relevant fields of replica 0’s generated compose document. The full document also carries platform labels and network configuration:

name: acme_prod_demo-web-r0
services:
web-r0:
image: nginx:alpine
expose: ["80"]
side-r0:
image: busybox:1.37
command: ["sleep", "3600"]
network_mode: service:web-r0
depends_on:
web-r0:
condition: service_started

Replica 1 makes the same references with r1. Each sidecar joins its own web replica’s namespace. SHC adds the start dependency for the namespace joiner (core/pod/project.go:302; namespace dependency test (tests/_ce/core/pod/project_test.go)). Admission chooses each replica’s node as described above.

CodeCause
X400073SCLVALA malformed scale value, including a per-kind count below 1, a / mixed into counts per place, or a parenthesised node selector. <n>/<kind> and per:<kind> are retired and answer X400225SCLOLD naming <n>/<kind>.
X400225SCLOLDA retired spelling such as east:2 or per:location; the error gives the current spelling.
X400223SCLLOC, X400224SCLAMBA location names no known path, or its suffix matches several paths.
X400071SCLMALA malformed x-shc-scale block, such as a quoted max, an unknown key or the retired stateful key; distinct copies of a service are a {% members %} group.
X400902SCLMAXAn individual service’s declared ceiling is exceeded.
X400083PODSCLPod members disagree on scale, exceed the pod ceiling, reference a missing namespace/volume owner, or all join another member’s network namespace.
X400186FNCSHRLocation constraints leave no common node for pods sharing node-local data.
X400085PODPRJThe document cannot split into replica projects. Examples: container_name with several replicas, extends, top-level include, a cross-pod link alias, a missing required dependency, or colliding folded pod names. On the fabric, an app service named like another service’s replica also refuses.
X400085PODPRJThe document can’t carry SHC’s identity labels safely, including label_file, multiple YAML documents, interpolated label keys or compose merge tags.
X400240PODNETA service joins a network namespace whose owner has no fabric address.
X400241PODPUBAn engine port publish targets a namespace joiner, or names the owner for a port declared only by a member.
X400242PODADRAn --expose route, a port publish or a route’s gRPC path names a service whose network_mode is none, container:<x> or anything but host and service:<x>: it has no address of its own and no VIP.
X400087PODRTMThe chosen runtime refuses a rendered service.

The refusal paths are modules/app/scale_overrides.go, modules/app/replica_ceiling.go, modules/app/replica_pods.go, modules/app/replica_projects.go, modules/app/fabric_render.go:538 and modules/app/spec_record.go:298; their pod codes are in modules/app/errors_pod.go. The shared-volume code is modules/fence/errors.go, X400186FNCSHR. The pod render tests (tests/_ce/modules/app/replica_pods_test.go) and project tests (tests/_ce/core/pod/project_test.go) cover split refusals. When replica projects are off, a split failure logs X200086PODPRJ and the deployment’s one compose project remains the execution path (modules/app/install_runner.go:2234).