Replica projects (contract)
Status: open design (SHC-1144). Projects are written and recorded, the store keeps each replica’s document as a vault entry, and a node of a cluster on the
shcfabric with service VIPs runs them in place of the deployment document (SHC-1259): a Linux node, or a Mac node on the apple runtime (SHC-1557). Each replica runs on the node it is bound to (SHC-1426: cereplicaPlacer,heldReplicas). Section 7 is behind the code: the store fillsentry, a document that does not split is refused withX400085PODPRJonce replica projects run, and compose’s own loader reads anincludeand the rest (SHC-1351, cecore/composenorm/loader.go). Section 8’sshc.podreading is built (cenodeagent.replicaOf). The leader side (render, store, admission) and the node side (materialiser, node agent) build against this page. The types and the one implementation of every rule here are cecore/pod. It follows Clay’s rulings of 2026-10-01 and 2026-10-02: one compose project per replica and a replica is a pod, render once on the leader,--scalein three shapes, secrets on tmpfs, shcd answers DNS so SHC writes noextra_hosts, no backwards compatibility. It replaces the scheduler design’s “one compose project per node a deployment touches” (section 3.6, 3.24). It does not edit that proposal; section 9 lists what it owes it.
1. Pods
Section titled “1. Pods”A pod is the set of an app’s services that must share a node because
they share a namespace or a filesystem. The render groups two services into
one pod when either names the other in network_mode: service:x,
pid: service:x, ipc: service:x or volumes_from. Grouping is transitive.
A service linked to nothing is a pod of one.
- Name. A pod is named for its first member, in name order, that keeps
its own network namespace (no
network_mode: service:). Assignment rows,--scaleoutput, status and the project directory use that name as the replica’sservice. - Scale.
--scale <member>=<shape>scales the whole pod, whichever member it names. Two members of one pod that state different shapes are refused at render, naming both. With no--scalethe pod takes the count its members authored (deploy.replicasorscale), refused when they disagree, and one replica when none states any. The pod’s ceiling is the smallestx-shc-scale.maxamong its members, checked for every shape,per:<kind>included. The pod is stateful when any member declaresx-shc-scale: {stateful: true}. - Replica. A replica is one pod at one ordinal:
(deployment, pod, ordinal). Ordinals count from 0 per pod. It is placed, started, rolled, restarted and moved as one unit. Its reservation is the sum of its members’.
2. One project per replica
Section titled “2. One project per replica”| thing | value |
|---|---|
| compose project name | <tenant>_<environment>_<stack>-<pod>-r<ordinal> |
| entry name of a member | <member>-r<ordinal> |
| directory on the node | node/current/projections/<tenant>/<environment>/<stack>/<pod>-r<ordinal>/ |
| files in it | compose.yaml, replica.json, nothing else |
The deployment part of the project name is DockerProjectName (every -,
. and space folded to _), so it holds no - and the first - splits the
deployment from the replica. The pod part is lowercased and anything outside
[a-z0-9_-] becomes _, in the project name and the directory name alike.
Two pods of one deployment that fold to the same name do not split.
Example, --scale app=2 where vpn joins app’s network namespace, plus a
db pod of one:
projections/corp/prod/wiki/ app-r0/ compose.yaml (entries app-r0, vpn-r0) replica.json app-r1/ compose.yaml (entries app-r1, vpn-r1) replica.json db-r0/ compose.yaml (entry db-r0) replica.jsonA node holds only the directories of replicas bound to it.
3. compose.yaml
Section titled “3. compose.yaml”The stored compose document, byte for byte. It is canonical JSON (sorted keys, no insignificant whitespace), which every compose runtime reads as YAML. The materialiser copies it and changes nothing.
What the document always is:
- Final for one node. The leader finished it for the node in
replica.json. The pin and thednsserver are that node’s. A move gets a new document from the leader with a new digest. - Self-contained. No
extendsand noinclude: a document that has either does not split. A platform package that needed compose’s loader (§7) arrives with both already merged. No relative path anywhere: a bind, anenv_file, a build context, a secret’s or a config’s file that is not absolute is written out under the deployment’s directory, and a build with no context gets that directory. Config bytes ride inline asconfigs.<name>.content. The directory it is run from does not matter. Those absolute paths are on the node that rendered; on another node the files arrive by the staging path, which is not built here. - One container per entry. No
deploy.replicas, noscale, nodeploy.mode. Compose never scales.restartis whatever the render’s restart default wrote; SHC-1112 makes that"no"on every entry, with the node agent owning restarts. Acontainer_nameon a pod with more than one replica does not split. - References inside the pod are rewritten.
network_mode: service:appbecomesservice:app-r0;pid,ipc,volumes_from,linksanddepends_onbetween members likewise.depends_onbetween members stays in the document and compose orders it. - No dependency leaves the project. A
depends_onon a service of another pod is removed from the document and written toreplica.jsonasafter(section 5). One that isrequired: falseon a service the app does not have is dropped. Alinksentry to another pod under the service’s own name becomes anafterwithstarted; under an alias the document does not split, since nothing outside the project answers an alias. - Networks are external. Every top-level network is
external: truewith an explicitname. Compose joins, never creates or removes. - Named volumes are external. Every top-level volume is
external: truewith an explicitname. A stateful pod’s volumes carry the ordinal (<tenant>_<environment>_<stack>_<volume>-r<ordinal>); any other pod’s are shared by its replicas (<tenant>_<environment>_<stack>_<volume>). Host binds are absolute. - Names. SHC adds no alias for the bare service name and no
extra_hosts. Authored aliases and authoredextra_hostsstay. Every entry with its own network namespace carriesdns, the node’s resolver address first and any server the app wrote after it, anddns_searchwith the deployment’s zone (<stack>.<environment>.<tenant>.<cluster zone>), sodbresolves through shcd to the ready replicas ofdbwherever they run. The entry name (db-r0) is the only name the runtime’s own DNS answers. - Ports.
exposeandportsin the app only declare. The document carries aportsentry only for a deployment’s--portmapping, long form withmode: host. - Secrets are references. A top-level secret keeps its
x-shc-vaultreference and an absolutefileunder the node’s RAM-backed secrets directory. No value is in the document. - Labels on every entry, set by the platform and overwriting the app’s:
| label | value |
|---|---|
shc.tenant, shc.environment, shc.stack | the deployment |
shc.service | the member’s app service name (vpn), not the entry name |
shc.pod | the pod name (app), the assignment row’s service |
shc.ordinal | the ordinal |
shc.group, shc.member | the group and the member number, on a group member’s replicas only |
shc.node, shc.location | the node and location the document was finished for (replica.json’s), empty where the render did not know them |
shc.spec_digest | the replica’s digest (section 4) |
A selector over these labels is how jobs, backups and exec pick a replica (the lowest-numbered one that matches by default), so every entry carries all of them.
A deployment’s containers on a node are found by the three deployment labels, never by a project-name prefix.
4. Digest
Section titled “4. Digest”One digest per replica, v<n>:sha256:<hex>, from ce core/specdigest, the
one implementation both sides call. It reads a canonical form of the compose
document that specdigest writes itself (keys in byte order, no whitespace,
numbers as the document spells them), with the shc.spec_digest label
removed from every entry (and a labels map left empty by that dropped), so
the document digests the same before and after it is stamped. <n> is the
version of that form. A version never changes once written, and a stored
digest is only ever recomputed at its own version.
The leader stores the document only as specdigest.Seal writes it, stamp
included, and the node checks it with specdigest.Verify, which refuses the
same content spelled any other way. A render that changed nothing in a
replica hands it back the document and digest it already has
(specdigest.Reseal), so an shcd that writes a newer form recreates nothing.
The digest covers what runs, so a change to replica.json alone recreates
nothing.
If secret values end up delivered beside the document as a sealed bundle (proposed, not ruled), the replica’s digest folds the bundle’s digest in, so a rotation rolls the replica. Nothing here builds that.
The same value is in three places: the stored replica record, replica.json
and the label on each container.
5. replica.json
Section titled “5. replica.json”{ "version": 1, "deployment": {"tenant": "corp", "environment": "prod", "stack": "wiki"}, "pod": "app", "ordinal": 0, "project": "corp_prod_wiki-app-r0", "members": {"app": "app-r0", "vpn": "vpn-r0"}, "ready": ["app"], "stateful": false, "reserve": {"cpu_millicores": 350, "memory_bytes": 335544320}, "node": "9f2c", "location": "eu", "digest": "sha256:...", "networks": [ {"name": "corp_prod_wiki_default", "kind": "deployment"}, {"name": "shc_fabric", "kind": "fabric"} ], "volumes": [ {"name": "corp_prod_wiki_cache", "durable": false} ], "after": [ {"pod": "db", "condition": "healthy", "required": true}, {"pod": "migrate", "condition": "completed", "required": true}, {"pod": "search", "condition": "started", "required": false} ]}membersmaps each member service to its entry name, in name order.readyis the members whose state is the replica’s readiness: those that declare ports, or every member when none does. The replica is ready when each is running and, where it has a healthcheck, healthy.reserveis what the replica’s assignment row reserves on its node: the sum of its members’deploy.resources.reservationsin the rendered document, which the render derives from a service’s limit when the app stated none. Always written.networksandvolumesare every external resource the document names, so the node creates them without parsing compose. A network or volume the app authored carriesspec, its authored definition without the name (driver, options, labels), for whoever creates it. Networkkindisdeployment(the node agent creates it fromspec, a plain bridge when there is none),fabric(the tenant’s fabric bridge,fabric.EnsureBridge) orsystem(a network the app declared external: the shared platform networks). A volume the app declared external is not listed. A volume withdurable: falseis scratch (not in the app’sbackup.yaml, SHC-1231) and is created empty when missing. A missingdurable: truevolume is created only on the replica’s first bind (generation 1); otherwise the start is refused with reasonvolume_missingand the data move is the relocation path’s job (SHC-1230).afteris the ordering SHC enforces across projects. It is the pod’s cross-poddepends_on, merged over its members, sorted by pod.
| field | meaning |
|---|---|
pod | a pod of the same deployment |
ordinal | optional. Absent: any replica of that pod satisfies it. Present: that replica only (a stateful pod’s ordinal n waits on its own n-1, healthy) |
condition | started: the replica reached starting or later. healthy: it is ready. completed: it ran to exit 0 and reports completed |
required | always written. False: do not hold the start for a pod that has no replica |
restart | compose’s depends_on.restart: restart this replica after the dependency’s replica is recreated |
The compose spellings map service_started to started, service_healthy
to healthy, service_completed_successfully to completed.
6. What the node agent may assume
Section titled “6. What the node agent may assume”- A directory exists for exactly the replicas bound to this node in this revision. A replica that left has no directory in the next one; the agent brings its project down from the engine by project name, not from a file.
compose.yamlhashes toreplica.json’s digest and to the stored replica’s. A mismatch never runs.- Applying a replica is
up -d --waiton its own project and file:executor.Project{Ref: {Name: project}, Dir: <directory>}. No service targeting, no--no-deps, no--scale, no environment beyond the runtime’s own. - Running
upagain on an unchanged document changes nothing, from any revision directory. - Everything outside the project is listed in
replica.json: networks, volumes, what to wait for. - Pod names and entry names are unique in the deployment, project names on the node.
What it does, in order, before up: the four checks of the scheduler design
section 2.2 (revision whole, digest, still this node’s at the current
generation and epoch with leases held, after satisfied), then create
missing networks and volumes from the lists, then stage the document’s
secrets into RAM.
It reads after states from its own replica of the assignment store
(scheduler.Store.ForDeployment, a local read inside the lease TTL), for
dependencies on this node and on others alike. One path.
What it never does: render, fill a template, add or drop a key, pick an
address, reorder entries, let compose create a network or a volume, run a
project whose node is not its own.
Whole-deployment verbs (stop, start, restart, down) run over every replica
project of the deployment on the node, dependents before dependencies for a
stop and the reverse for a start, using after.
7. The stored record
Section titled “7. The stored record”deployment_spec.spec.replicas[] (ce modules/deploymentspec) holds one
replica per replica project: service is the pod, ordinal, node and
location the placement it was finished for, digest the project’s digest
as the render computed it, record the replica.json fields, and entry
the document. Store.ReplicaDigest(ref, pod, ordinal) is what the node
agent’s digest check reads. The materialiser writes compose.yaml from
entry and replica.json from record. assignments.json rows carry
(deployment, service = pod, ordinal, generation, spec digest, state). A
deployment whose document did not split has one stored replica per service
at ordinal 0, with its entry’s digest and no record.
Assignment rows. An install or update binds one row per replica
project once it has recorded its spec (ce scheduler.Replicas.Sync,
SHC-1259): service is the pod, ordinal from 0, incarnation 0, bound to the
record’s node, reserving the record’s reserve, stateful as the record
says. A replica whose node, reservation or statefulness changed is bound
again at its next generation, one a move is taking is left to the move,
and the rows of replicas the render no longer has are deleted in the same
batch (placement event retire). The bind is fit-guarded like any other. A
deployment that has a stored spec and no replica row (an install before
any engine held an epoch, or one whose deployment row was written after
it) gets its rows from the leader’s scheduler tick, read from that spec.
While the stack row exists. Until the stack-row readers convert, these
rows sit beside the deployment’s stack row (service "", ordinal 0). A
deployment that has its stack row is one project on that row’s node, and
that row alone says which node holds it. The replica rows record what the
project holds and what it reserves. Two rules, both in ce
modules/scheduler, and both deleted in the slice that removes the stack
row:
- Nothing acts on a replica row. Every reader that decides where a
deployment runs reads the stack row alone while there is one
(
scheduler.Holds): the node agent’s assigned set and binding, the holders a verb is sent to, the list of deployments a node hosts. The store’s per-node read (what failover moves) leaves out a replica row whose stack row is on another node, and the placement engine’s snapshot reads every replica on the stack row’s node. - A stray lasts one leader tick. Several writers keep the rows (the
install’s bind, the stack row’s follow of the deployment’s bytes, a
failover’s bind, the leader’s own pass), each reads before it writes,
and their guards are per row, so a replica row can land on the node the
stack row has just left. The leader’s pass (
Replicas.Converge) binds every such row to the stack row’s node, keeping what it reserves, or reserving nothing when that node has no room. A stray a move is taking, and every row while a move is taking the stack row, are left to the move.
What can be off for that one tick: the fit guard counts a stray’s
reservation on the node it names, and shc cluster status and the
not-redundant check print that node.
When the stack row goes, each row stands alone under its own generation, which is what the store’s guards protect. The follow pass and the stack-row-only reads go in the same change.
Until the document can be replicated. Today’s render interpolates
resolved secrets into the compose, so the store keeps each replica’s digest
and record and leaves entry empty. The render writes every replica’s project
on the node it runs on, under the deployment’s own directory, in the same
two-file shape:
<state>/deployments/<tenant>/<stack>/<environment>/replicas/<pod>-r<ordinal>/ compose.yaml replica.json (0600, like the deployment's compose.yaml)The deployment’s own compose.yaml is still written beside it and is what
runs until the node agent applies replica projects. Until then a document
that does not split (extends, a link alias across pods, a
container_name on a scaled pod) installs as before, gets no replicas/
directory, and logs X200086PODPRJ with the reason. When replica projects
are what runs, that becomes the refusal X400085PODPRJ.
What compose reads past the text. Compose reads more from a document
than its text when it has a top-level include, an extends from another
file, a label_file, a !reset or !override tag, or a further YAML
document. A service those bring in would carry none of the deployment’s
labels, and a mount they bring in would run unjudged. So before anything is
judged, stamped or split, the render checks that the document is plain (ce
core/composenorm, SHC-1296). A platform package that is not plain is read
through compose’s own loader, below. A tenant’s own package or an adapter’s
package that is not plain is refused by name with X400126CMPNRM.
The check fails closed. A document is plain only when all of this holds
(composenorm.NotPlain): it is one YAML document and a mapping; no node
carries an explicit tag; every key is a scalar the document spells, never an
alias; << appears only as the plain merge key; there is no top-level
include; services is a mapping of mappings; no service has a label_file
or an extends with a file. A shape the check does not know is not plain,
and text that is not YAML is refused. Merge keys and anchors are plain YAML,
which the stamp reads as YAML does. An extends inside the file stays for
compose to merge when the deployment comes up: both services are in the text
and stamped, and the extending service’s own labels win.
A plain document is handed on byte for byte. A rendered document is the
platform’s dialect of compose: a mount carries a mode, an owner or noinit,
a secret names a vault reference and no file. What an install stores for an
app must not change because this step exists. Every shipped app is plain,
and a test holds each one to that and to coming out byte for byte.
A platform package that is not plain is read through compose’s own loader
(ce core/composenorm/loader.go NormalisePackage, SHC-1351), and the plain
document it returns is what every later step judges, stamps and splits, so an
include, an extends from another file, a label_file, a !reset or
!override tag and a further document install. The loader gets no
environment and interpolates nothing: no .env, no env_file value and no
daemon variable is baked in, and ${VAR} stays literal. It opens every file
itself, and only a regular file inside the package directory of at most
8 MiB: a fifo, a device, or a link that resolves outside the package is
refused X400126CMPNRM. A tenant’s own package and an adapter’s package are
never loaded (ce modules/app/update_funnel.go): the render refuses them as
above, since loading them would read the files they name.
Nothing is interpolated at render: ${VAR} stays literal for compose’s own
pass when the deployment comes up, so no value from the daemon’s environment
lands in a stored spec. A label list entry whose key holds a $ is refused,
because compose would fill it in then and it could name a platform label.
The one edit a plain document can get: a long-form mount goes back to the
short string that says the same thing, because the volume policy and the
mounts rewrite judge short strings. A tmpfs mount stays long. A long-form
mount with no short form the mount policy can read (create_host_path: false, recursive, a subpath, an image mount, a source or target holding
${VAR:-default}) is refused: nothing runs unjudged. A mount string the
mount policy’s parser cannot read is refused for every package with the
parser’s own code, where a trusted package’s used to be left for compose to
report: /etc:${TARGET:-/host-etc} does not parse here and is a bind of the
host’s /etc once compose fills it in.
The label check (X400085PODPRJ, SHC-1259 step 1) stays behind it, with its
own refusal of the same shapes: every service must read exactly the
shc.tenant, shc.stack, shc.environment and shc.service SHC computes
and an empty shc.node. The labels are how every status and verb finds a
deployment’s containers.
8. Requirements on the node lane, not built here
Section titled “8. Requirements on the node lane, not built here”- Staged secrets need a replica key. The render stages a deployment’s
compose secrets per deployment under the node’s RAM-backed secrets
directory, and
DropComposeSecretsand the external-provider cache (SHC-1159) work per deployment too. With one project per replica, the staged files, the cache entries and the drop must be keyed by replica, or one replica leaving a node removes files its siblings still mount. - Autogen secrets are minted once, on the leader. A
?autogenreference resolved separately on two nodes would mint two values for one secret. - The agent names a replica by its pod. A replica project’s containers
carry
shc.pod(the pod, the stored replica’sservice) andshc.service(the member). The node agent’s spec check today readsshc.serviceas the replica’s service (nodeagent.replicaOf), so a member that is not the pod’s namesake (vpnin podapp) would look up a replica the store does not hold and be refused. When the agent runs replica projects it readsshc.podfirst. Nothing breaks before that: the deployment’s owncompose.yaml, which is what runs today, carries noshc.spec_digest, so the check skips its containers. - An assignment row is refreshed only by a bind. The placement
engine’s
Place(cecore/placement) keeps a bound replica on its node whenever that node survives the eligibility filters, without rechecking capacity and without re-binding it. The render’s own bind closes this for installs and updates: a replica whosereserveorstatefulchanged is bound again in place, and a grown reservation is checked against the node. What is left: while the deployment’s own compose is what runs, a refused bind on an update is logged (X200087PODROW) and the rows keep what the last bind wrote until the next render. When replica projects are what runs the bind comes before the up and its refusal fails the update.
9. Deltas owed to other pages and lanes
Section titled “9. Deltas owed to other pages and lanes”- Scheduler design 3.6 and 3.24: a projection is per replica, at
projections/<tenant>/<environment>/<stack>/<pod>-r<ordinal>/, not onecompose.yamlper deployment per node. x-shc-scale: {stateful: true}is read as it is today, because it is merged. Clay rejected an authored stateful flag on 2026-10-02 (every service has one virtual IP; non-interchangeable topologies are a separate concept on top of replicas, not ruled yet). Nothing here adds a new authored use, and when the replacement is ruled,pod.Scaled.Statefultakes its value from it instead of the key.- Assignment states gain
completedfor a replica whose members all ran to exit 0 (cemodules/scheduler). - Every document is node-specific: the pin comes from the node’s grant,
dnsis the node’s mesh address, and every entry carriesshc.nodeandshc.location. A move gets a new document and a new digest. - The
dnsline names only the node’s resolver. The host’s own nameservers are no longer appended: the leader cannot read another node’s resolv.conf, and shcd holds port 53 through restarts.