Skip to content
SHC Docs

Replica projects (contract)

Status: open design (SHC-1144). Projects are written and recorded, the store keeps each replica’s document as a vault entry, and a node of a cluster on the shc fabric with service VIPs runs them in place of the deployment document (SHC-1259): a Linux node, or a Mac node on the apple runtime (SHC-1557). Each replica runs on the node it is bound to (SHC-1426: ce replicaPlacer, heldReplicas). Section 7 is behind the code: the store fills entry, a document that does not split is refused with X400085PODPRJ once replica projects run, and compose’s own loader reads an include and the rest (SHC-1351, ce core/composenorm/loader.go). Section 8’s shc.pod reading is built (ce nodeagent.replicaOf). The leader side (render, store, admission) and the node side (materialiser, node agent) build against this page. The types and the one implementation of every rule here are ce core/pod. It follows Clay’s rulings of 2026-10-01 and 2026-10-02: one compose project per replica and a replica is a pod, render once on the leader, --scale in three shapes, secrets on tmpfs, shcd answers DNS so SHC writes no extra_hosts, no backwards compatibility. It replaces the scheduler design’s “one compose project per node a deployment touches” (section 3.6, 3.24). It does not edit that proposal; section 9 lists what it owes it.

A pod is the set of an app’s services that must share a node because they share a namespace or a filesystem. The render groups two services into one pod when either names the other in network_mode: service:x, pid: service:x, ipc: service:x or volumes_from. Grouping is transitive. A service linked to nothing is a pod of one.

  • Name. A pod is named for its first member, in name order, that keeps its own network namespace (no network_mode: service:). Assignment rows, --scale output, status and the project directory use that name as the replica’s service.
  • Scale. --scale <member>=<shape> scales the whole pod, whichever member it names. Two members of one pod that state different shapes are refused at render, naming both. With no --scale the pod takes the count its members authored (deploy.replicas or scale), refused when they disagree, and one replica when none states any. The pod’s ceiling is the smallest x-shc-scale.max among its members, checked for every shape, per:<kind> included. The pod is stateful when any member declares x-shc-scale: {stateful: true}.
  • Replica. A replica is one pod at one ordinal: (deployment, pod, ordinal). Ordinals count from 0 per pod. It is placed, started, rolled, restarted and moved as one unit. Its reservation is the sum of its members’.
thingvalue
compose project name<tenant>_<environment>_<stack>-<pod>-r<ordinal>
entry name of a member<member>-r<ordinal>
directory on the nodenode/current/projections/<tenant>/<environment>/<stack>/<pod>-r<ordinal>/
files in itcompose.yaml, replica.json, nothing else

The deployment part of the project name is DockerProjectName (every -, . and space folded to _), so it holds no - and the first - splits the deployment from the replica. The pod part is lowercased and anything outside [a-z0-9_-] becomes _, in the project name and the directory name alike. Two pods of one deployment that fold to the same name do not split.

Example, --scale app=2 where vpn joins app’s network namespace, plus a db pod of one:

projections/corp/prod/wiki/
app-r0/ compose.yaml (entries app-r0, vpn-r0) replica.json
app-r1/ compose.yaml (entries app-r1, vpn-r1) replica.json
db-r0/ compose.yaml (entry db-r0) replica.json

A node holds only the directories of replicas bound to it.

The stored compose document, byte for byte. It is canonical JSON (sorted keys, no insignificant whitespace), which every compose runtime reads as YAML. The materialiser copies it and changes nothing.

What the document always is:

  • Final for one node. The leader finished it for the node in replica.json. The pin and the dns server are that node’s. A move gets a new document from the leader with a new digest.
  • Self-contained. No extends and no include: a document that has either does not split. A platform package that needed compose’s loader (§7) arrives with both already merged. No relative path anywhere: a bind, an env_file, a build context, a secret’s or a config’s file that is not absolute is written out under the deployment’s directory, and a build with no context gets that directory. Config bytes ride inline as configs.<name>.content. The directory it is run from does not matter. Those absolute paths are on the node that rendered; on another node the files arrive by the staging path, which is not built here.
  • One container per entry. No deploy.replicas, no scale, no deploy.mode. Compose never scales. restart is whatever the render’s restart default wrote; SHC-1112 makes that "no" on every entry, with the node agent owning restarts. A container_name on a pod with more than one replica does not split.
  • References inside the pod are rewritten. network_mode: service:app becomes service:app-r0; pid, ipc, volumes_from, links and depends_on between members likewise. depends_on between members stays in the document and compose orders it.
  • No dependency leaves the project. A depends_on on a service of another pod is removed from the document and written to replica.json as after (section 5). One that is required: false on a service the app does not have is dropped. A links entry to another pod under the service’s own name becomes an after with started; under an alias the document does not split, since nothing outside the project answers an alias.
  • Networks are external. Every top-level network is external: true with an explicit name. Compose joins, never creates or removes.
  • Named volumes are external. Every top-level volume is external: true with an explicit name. A stateful pod’s volumes carry the ordinal (<tenant>_<environment>_<stack>_<volume>-r<ordinal>); any other pod’s are shared by its replicas (<tenant>_<environment>_<stack>_<volume>). Host binds are absolute.
  • Names. SHC adds no alias for the bare service name and no extra_hosts. Authored aliases and authored extra_hosts stay. Every entry with its own network namespace carries dns, the node’s resolver address first and any server the app wrote after it, and dns_search with the deployment’s zone (<stack>.<environment>.<tenant>.<cluster zone>), so db resolves through shcd to the ready replicas of db wherever they run. The entry name (db-r0) is the only name the runtime’s own DNS answers.
  • Ports. expose and ports in the app only declare. The document carries a ports entry only for a deployment’s --port mapping, long form with mode: host.
  • Secrets are references. A top-level secret keeps its x-shc-vault reference and an absolute file under the node’s RAM-backed secrets directory. No value is in the document.
  • Labels on every entry, set by the platform and overwriting the app’s:
labelvalue
shc.tenant, shc.environment, shc.stackthe deployment
shc.servicethe member’s app service name (vpn), not the entry name
shc.podthe pod name (app), the assignment row’s service
shc.ordinalthe ordinal
shc.group, shc.memberthe group and the member number, on a group member’s replicas only
shc.node, shc.locationthe node and location the document was finished for (replica.json’s), empty where the render did not know them
shc.spec_digestthe replica’s digest (section 4)

A selector over these labels is how jobs, backups and exec pick a replica (the lowest-numbered one that matches by default), so every entry carries all of them.

A deployment’s containers on a node are found by the three deployment labels, never by a project-name prefix.

One digest per replica, v<n>:sha256:<hex>, from ce core/specdigest, the one implementation both sides call. It reads a canonical form of the compose document that specdigest writes itself (keys in byte order, no whitespace, numbers as the document spells them), with the shc.spec_digest label removed from every entry (and a labels map left empty by that dropped), so the document digests the same before and after it is stamped. <n> is the version of that form. A version never changes once written, and a stored digest is only ever recomputed at its own version.

The leader stores the document only as specdigest.Seal writes it, stamp included, and the node checks it with specdigest.Verify, which refuses the same content spelled any other way. A render that changed nothing in a replica hands it back the document and digest it already has (specdigest.Reseal), so an shcd that writes a newer form recreates nothing. The digest covers what runs, so a change to replica.json alone recreates nothing.

If secret values end up delivered beside the document as a sealed bundle (proposed, not ruled), the replica’s digest folds the bundle’s digest in, so a rotation rolls the replica. Nothing here builds that.

The same value is in three places: the stored replica record, replica.json and the label on each container.

{
"version": 1,
"deployment": {"tenant": "corp", "environment": "prod", "stack": "wiki"},
"pod": "app",
"ordinal": 0,
"project": "corp_prod_wiki-app-r0",
"members": {"app": "app-r0", "vpn": "vpn-r0"},
"ready": ["app"],
"stateful": false,
"reserve": {"cpu_millicores": 350, "memory_bytes": 335544320},
"node": "9f2c",
"location": "eu",
"digest": "sha256:...",
"networks": [
{"name": "corp_prod_wiki_default", "kind": "deployment"},
{"name": "shc_fabric", "kind": "fabric"}
],
"volumes": [
{"name": "corp_prod_wiki_cache", "durable": false}
],
"after": [
{"pod": "db", "condition": "healthy", "required": true},
{"pod": "migrate", "condition": "completed", "required": true},
{"pod": "search", "condition": "started", "required": false}
]
}
  • members maps each member service to its entry name, in name order.
  • ready is the members whose state is the replica’s readiness: those that declare ports, or every member when none does. The replica is ready when each is running and, where it has a healthcheck, healthy.
  • reserve is what the replica’s assignment row reserves on its node: the sum of its members’ deploy.resources.reservations in the rendered document, which the render derives from a service’s limit when the app stated none. Always written.
  • networks and volumes are every external resource the document names, so the node creates them without parsing compose. A network or volume the app authored carries spec, its authored definition without the name (driver, options, labels), for whoever creates it. Network kind is deployment (the node agent creates it from spec, a plain bridge when there is none), fabric (the tenant’s fabric bridge, fabric.EnsureBridge) or system (a network the app declared external: the shared platform networks). A volume the app declared external is not listed. A volume with durable: false is scratch (not in the app’s backup.yaml, SHC-1231) and is created empty when missing. A missing durable: true volume is created only on the replica’s first bind (generation 1); otherwise the start is refused with reason volume_missing and the data move is the relocation path’s job (SHC-1230).
  • after is the ordering SHC enforces across projects. It is the pod’s cross-pod depends_on, merged over its members, sorted by pod.
fieldmeaning
poda pod of the same deployment
ordinaloptional. Absent: any replica of that pod satisfies it. Present: that replica only (a stateful pod’s ordinal n waits on its own n-1, healthy)
conditionstarted: the replica reached starting or later. healthy: it is ready. completed: it ran to exit 0 and reports completed
requiredalways written. False: do not hold the start for a pod that has no replica
restartcompose’s depends_on.restart: restart this replica after the dependency’s replica is recreated

The compose spellings map service_started to started, service_healthy to healthy, service_completed_successfully to completed.

  1. A directory exists for exactly the replicas bound to this node in this revision. A replica that left has no directory in the next one; the agent brings its project down from the engine by project name, not from a file.
  2. compose.yaml hashes to replica.json’s digest and to the stored replica’s. A mismatch never runs.
  3. Applying a replica is up -d --wait on its own project and file: executor.Project{Ref: {Name: project}, Dir: <directory>}. No service targeting, no --no-deps, no --scale, no environment beyond the runtime’s own.
  4. Running up again on an unchanged document changes nothing, from any revision directory.
  5. Everything outside the project is listed in replica.json: networks, volumes, what to wait for.
  6. Pod names and entry names are unique in the deployment, project names on the node.

What it does, in order, before up: the four checks of the scheduler design section 2.2 (revision whole, digest, still this node’s at the current generation and epoch with leases held, after satisfied), then create missing networks and volumes from the lists, then stage the document’s secrets into RAM.

It reads after states from its own replica of the assignment store (scheduler.Store.ForDeployment, a local read inside the lease TTL), for dependencies on this node and on others alike. One path.

What it never does: render, fill a template, add or drop a key, pick an address, reorder entries, let compose create a network or a volume, run a project whose node is not its own.

Whole-deployment verbs (stop, start, restart, down) run over every replica project of the deployment on the node, dependents before dependencies for a stop and the reverse for a start, using after.

deployment_spec.spec.replicas[] (ce modules/deploymentspec) holds one replica per replica project: service is the pod, ordinal, node and location the placement it was finished for, digest the project’s digest as the render computed it, record the replica.json fields, and entry the document. Store.ReplicaDigest(ref, pod, ordinal) is what the node agent’s digest check reads. The materialiser writes compose.yaml from entry and replica.json from record. assignments.json rows carry (deployment, service = pod, ordinal, generation, spec digest, state). A deployment whose document did not split has one stored replica per service at ordinal 0, with its entry’s digest and no record.

Assignment rows. An install or update binds one row per replica project once it has recorded its spec (ce scheduler.Replicas.Sync, SHC-1259): service is the pod, ordinal from 0, incarnation 0, bound to the record’s node, reserving the record’s reserve, stateful as the record says. A replica whose node, reservation or statefulness changed is bound again at its next generation, one a move is taking is left to the move, and the rows of replicas the render no longer has are deleted in the same batch (placement event retire). The bind is fit-guarded like any other. A deployment that has a stored spec and no replica row (an install before any engine held an epoch, or one whose deployment row was written after it) gets its rows from the leader’s scheduler tick, read from that spec.

While the stack row exists. Until the stack-row readers convert, these rows sit beside the deployment’s stack row (service "", ordinal 0). A deployment that has its stack row is one project on that row’s node, and that row alone says which node holds it. The replica rows record what the project holds and what it reserves. Two rules, both in ce modules/scheduler, and both deleted in the slice that removes the stack row:

  1. Nothing acts on a replica row. Every reader that decides where a deployment runs reads the stack row alone while there is one (scheduler.Holds): the node agent’s assigned set and binding, the holders a verb is sent to, the list of deployments a node hosts. The store’s per-node read (what failover moves) leaves out a replica row whose stack row is on another node, and the placement engine’s snapshot reads every replica on the stack row’s node.
  2. A stray lasts one leader tick. Several writers keep the rows (the install’s bind, the stack row’s follow of the deployment’s bytes, a failover’s bind, the leader’s own pass), each reads before it writes, and their guards are per row, so a replica row can land on the node the stack row has just left. The leader’s pass (Replicas.Converge) binds every such row to the stack row’s node, keeping what it reserves, or reserving nothing when that node has no room. A stray a move is taking, and every row while a move is taking the stack row, are left to the move.

What can be off for that one tick: the fit guard counts a stray’s reservation on the node it names, and shc cluster status and the not-redundant check print that node.

When the stack row goes, each row stands alone under its own generation, which is what the store’s guards protect. The follow pass and the stack-row-only reads go in the same change.

Until the document can be replicated. Today’s render interpolates resolved secrets into the compose, so the store keeps each replica’s digest and record and leaves entry empty. The render writes every replica’s project on the node it runs on, under the deployment’s own directory, in the same two-file shape:

<state>/deployments/<tenant>/<stack>/<environment>/replicas/<pod>-r<ordinal>/
compose.yaml replica.json (0600, like the deployment's compose.yaml)

The deployment’s own compose.yaml is still written beside it and is what runs until the node agent applies replica projects. Until then a document that does not split (extends, a link alias across pods, a container_name on a scaled pod) installs as before, gets no replicas/ directory, and logs X200086PODPRJ with the reason. When replica projects are what runs, that becomes the refusal X400085PODPRJ.

What compose reads past the text. Compose reads more from a document than its text when it has a top-level include, an extends from another file, a label_file, a !reset or !override tag, or a further YAML document. A service those bring in would carry none of the deployment’s labels, and a mount they bring in would run unjudged. So before anything is judged, stamped or split, the render checks that the document is plain (ce core/composenorm, SHC-1296). A platform package that is not plain is read through compose’s own loader, below. A tenant’s own package or an adapter’s package that is not plain is refused by name with X400126CMPNRM.

The check fails closed. A document is plain only when all of this holds (composenorm.NotPlain): it is one YAML document and a mapping; no node carries an explicit tag; every key is a scalar the document spells, never an alias; << appears only as the plain merge key; there is no top-level include; services is a mapping of mappings; no service has a label_file or an extends with a file. A shape the check does not know is not plain, and text that is not YAML is refused. Merge keys and anchors are plain YAML, which the stamp reads as YAML does. An extends inside the file stays for compose to merge when the deployment comes up: both services are in the text and stamped, and the extending service’s own labels win.

A plain document is handed on byte for byte. A rendered document is the platform’s dialect of compose: a mount carries a mode, an owner or noinit, a secret names a vault reference and no file. What an install stores for an app must not change because this step exists. Every shipped app is plain, and a test holds each one to that and to coming out byte for byte.

A platform package that is not plain is read through compose’s own loader (ce core/composenorm/loader.go NormalisePackage, SHC-1351), and the plain document it returns is what every later step judges, stamps and splits, so an include, an extends from another file, a label_file, a !reset or !override tag and a further document install. The loader gets no environment and interpolates nothing: no .env, no env_file value and no daemon variable is baked in, and ${VAR} stays literal. It opens every file itself, and only a regular file inside the package directory of at most 8 MiB: a fifo, a device, or a link that resolves outside the package is refused X400126CMPNRM. A tenant’s own package and an adapter’s package are never loaded (ce modules/app/update_funnel.go): the render refuses them as above, since loading them would read the files they name.

Nothing is interpolated at render: ${VAR} stays literal for compose’s own pass when the deployment comes up, so no value from the daemon’s environment lands in a stored spec. A label list entry whose key holds a $ is refused, because compose would fill it in then and it could name a platform label.

The one edit a plain document can get: a long-form mount goes back to the short string that says the same thing, because the volume policy and the mounts rewrite judge short strings. A tmpfs mount stays long. A long-form mount with no short form the mount policy can read (create_host_path: false, recursive, a subpath, an image mount, a source or target holding ${VAR:-default}) is refused: nothing runs unjudged. A mount string the mount policy’s parser cannot read is refused for every package with the parser’s own code, where a trusted package’s used to be left for compose to report: /etc:${TARGET:-/host-etc} does not parse here and is a bind of the host’s /etc once compose fills it in.

The label check (X400085PODPRJ, SHC-1259 step 1) stays behind it, with its own refusal of the same shapes: every service must read exactly the shc.tenant, shc.stack, shc.environment and shc.service SHC computes and an empty shc.node. The labels are how every status and verb finds a deployment’s containers.

8. Requirements on the node lane, not built here

Section titled “8. Requirements on the node lane, not built here”
  • Staged secrets need a replica key. The render stages a deployment’s compose secrets per deployment under the node’s RAM-backed secrets directory, and DropComposeSecrets and the external-provider cache (SHC-1159) work per deployment too. With one project per replica, the staged files, the cache entries and the drop must be keyed by replica, or one replica leaving a node removes files its siblings still mount.
  • Autogen secrets are minted once, on the leader. A ?autogen reference resolved separately on two nodes would mint two values for one secret.
  • The agent names a replica by its pod. A replica project’s containers carry shc.pod (the pod, the stored replica’s service) and shc.service (the member). The node agent’s spec check today reads shc.service as the replica’s service (nodeagent.replicaOf), so a member that is not the pod’s namesake (vpn in pod app) would look up a replica the store does not hold and be refused. When the agent runs replica projects it reads shc.pod first. Nothing breaks before that: the deployment’s own compose.yaml, which is what runs today, carries no shc.spec_digest, so the check skips its containers.
  • An assignment row is refreshed only by a bind. The placement engine’s Place (ce core/placement) keeps a bound replica on its node whenever that node survives the eligibility filters, without rechecking capacity and without re-binding it. The render’s own bind closes this for installs and updates: a replica whose reserve or stateful changed is bound again in place, and a grown reservation is checked against the node. What is left: while the deployment’s own compose is what runs, a refused bind on an update is logged (X200087PODROW) and the rows keep what the last bind wrote until the next render. When replica projects are what runs the bind comes before the up and its refusal fails the update.
  • Scheduler design 3.6 and 3.24: a projection is per replica, at projections/<tenant>/<environment>/<stack>/<pod>-r<ordinal>/, not one compose.yaml per deployment per node.
  • x-shc-scale: {stateful: true} is read as it is today, because it is merged. Clay rejected an authored stateful flag on 2026-10-02 (every service has one virtual IP; non-interchangeable topologies are a separate concept on top of replicas, not ruled yet). Nothing here adds a new authored use, and when the replacement is ruled, pod.Scaled.Stateful takes its value from it instead of the key.
  • Assignment states gain completed for a replica whose members all ran to exit 0 (ce modules/scheduler).
  • Every document is node-specific: the pin comes from the node’s grant, dns is the node’s mesh address, and every entry carries shc.node and shc.location. A move gets a new document and a new digest.
  • The dns line names only the node’s resolver. The host’s own nameservers are no longer appended: the leader cannot read another node’s resolv.conf, and shcd holds port 53 through restarts.