SHC status states
Layer 6. Services. Services
SHC derives deployment status and health from runtime container observations. The same derivation feeds service progress and deployment state. The node agent applies restart policy under the scheduler’s assignments and leases.
Overview
Section titled “Overview”SHC tracks status and health at three layers:
Deployment └── Services └── ContainersEach layer has:
- Status: Lifecycle state (running, stopped, degraded, etc.)
- Health: Application health (healthy, unhealthy, etc.)
Status and health are independent; a container can be running but unhealthy.
Data Model Hierarchy
Section titled “Data Model Hierarchy”| Layer | Fields | Derived From |
|---|---|---|
| Deployment | status, health | Services |
| Service | status, health, desired_replicas, restart_policy | Containers |
| Container | shc_status, health, status, exit_code | Docker/Podman |
Key principle: Lower layers feed into higher layers. Container states derive service states, which derive deployment states.
Key Reference
Section titled “Key Reference”| Var | Meaning |
|---|---|
ds | Docker container status (raw) |
dh | Docker container health (raw) |
ec | Exit code |
rp | Restart policy |
dr | Desired replicas |
cs | SHC container status (derived) |
ch | SHC container health (derived) |
ss | SHC service status |
sh | SHC service health |
rt | Runtime selected for the node |
All Status and Health Values
Section titled “All Status and Health Values”| Layer | Field | Values |
|---|---|---|
| Docker Container | status (raw) | created, running, paused, restarting, exited, dead |
| Docker Container | health (raw) | none, "" (no check), starting, healthy, unhealthy, unknown |
| SHC Container | status | starting, running, paused, complete, failed, degraded, unknown |
| SHC Container | health (derived) | healthy, unhealthy, starting, none, unknown |
| SHC Service | status | stopped, starting, running, restarting, paused, complete, failed, degraded |
| SHC Service | health | unknown, unhealthy, starting, healthy, none |
| SHC Deployment | status (operational) | queued, created, installing, starting, stopping, updating, removing |
| SHC Deployment | status (derived) | running, paused, degraded, complete, failed, stopped; read only, never stored: moving (see Moving) |
| SHC Deployment | health | unknown, unhealthy, starting, healthy, none |
Restart Policy Normalization
Section titled “Restart Policy Normalization”For status derivation, NormalizeRestartPolicy in modules/app/models.go
normalizes the authored policy:
| Input | Status policy | Meaning |
|---|---|---|
no | no | An exit is expected |
on-failure | on-failure | Retry a failed job |
always | always | Keep the service running |
unless-stopped | always | Derive daemon status |
The render writes restart: "no" for the runtime and saves the authored
policy in shc.restart.* labels. The node agent reads those labels.
Compose deploy.restart_policy.condition is translated before this step;
see compose keys.
Layer 1: Container
Section titled “Layer 1: Container”Docker Container Status (Raw)
Section titled “Docker Container Status (Raw)”Docker provides these container states. SHC stores them exactly as Docker reports:
| State | Description |
|---|---|
created | Container exists, never started |
running | Container is running |
paused | Container processes suspended |
restarting | Container is restarting |
exited | Container stopped |
dead | Container failed to stop properly |
Docker Container Health (Raw)
Section titled “Docker Container Health (Raw)”Docker provides health check status. SHC stores the raw value it reads, as Docker reports it:
| Health | Description |
|---|---|
"" | Empty; the container declares no HEALTHCHECK (this is what the engine actually returns; not the literal word none) |
none | Synonym some surfaces emit for “no health check defined” |
starting | Health check defined but not yet passed |
healthy | Health check passing |
unhealthy | Health check failing |
unknown | Health could not be read |
Note: The engine reports "" (empty string) for a container with no HEALTHCHECK, and the recorders write Docker’s own word for that state, none (app.NoHealthcheck), into health for a running container. A row the recorders never stamped reads "" (NULL). Every reader treats "" and none identically: running with no check is ready. unknown is distinct from both: it means “couldn’t determine,” not “no check.”
SHC Container Health (Derived)
Section titled “SHC Container Health (Derived)”The raw docker health string is not a passthrough. SHC derives a canonical container-health word from both the raw health and the raw docker status, because health is only meaningful while a container is live. The derivation is DeriveContainerHealth(rawHealth, dockerStatus) and produces one of five canonical values:
| Health | Meaning |
|---|---|
healthy | Live container whose check is passing or that declares no check at all (a running container with nothing left to check is as healthy as it can report) |
unhealthy | Live container whose health check is failing |
starting | Live container whose health check has not yet passed |
none | Container is not live (created / exited / complete / failed / dead / paused) → health is not applicable. This is a marker, not a displayed word; the UI shows such a container’s STATUS instead. A running container is never none. |
unknown | Health could not be determined. An unrecognised engine value also yields unknown. |
Key invariants:
- A running container is never
none; a running no-check container ishealthy(“live, nothing left to wait on”). The model emits the final word once and every renderer trusts it; there is no per-renderer “promotenone→healthy” logic. noneis the not-applicable marker for non-running containers; a completed job never reports a ghosthealthy; its health derives tononeregardless of any stale raw value, and the display shows its STATUS instead.nonenever displays as a health word.dh=unknownyieldsunknownbefore the container status is consulted.
SHC Container Health Truth Table
Section titled “SHC Container Health Truth Table”| Docker Status | Raw Health | SHC Container Health |
|---|---|---|
| any | unknown | unknown (undeterminable wins) |
running | healthy | healthy |
running | unhealthy | unhealthy |
running | starting | starting |
running | "" / none | healthy (live + no check = healthy) |
running | (unrecognized) | unknown |
created / exited / complete / failed / dead / paused | any (except unknown) | none (not-applicable; display shows STATUS) |
SHC Container Health Derivation
Section titled “SHC Container Health Derivation”flowchart LR D0{"dh (raw)"} D0-->|unknown|UNKNOWN[unknown]
D0-->|not unknown|D1{ds} D1-->|not running|NONE[none]
D1-->|running|D2{dh} D2-->|unhealthy|UNHEALTHY[unhealthy] D2-->|starting|STARTING[starting] D2-->|healthy|HEALTHY[healthy] D2-->|"empty or none"|HEALTHY D2-->|unrecognized|UNKNOWNThe order is load-bearing: dh=unknown is checked first (before the status), then a non-running status forces none, and only a running container consults the remaining raw health words. A running container therefore lands on healthy / unhealthy / starting / unknown but never none; a non-running one lands on none unless its health was already unreadable (unknown).
SHC Container Status (Derived)
Section titled “SHC Container Status (Derived)”SHC interprets Docker status based on exit code and restart policy to produce a semantic status:
| Status | Meaning |
|---|---|
starting | Container initializing or restarting |
running | Container running and healthy |
paused | Container intentionally paused |
complete | Job finished successfully (exit 0, job mode) |
failed | Job failed (exit != 0, no retry) |
degraded | Not running when expected |
Note: The status derivation consumes the raw docker health (dh), not the derived container health (ch) above; its truth table stays documented in raw terms. Both the raw "" (no check) and none map identically to running here, and any unrecognized health word also folds to running (so a new docker health value never silently degrades every container).
SHC Container Status Derivation
Section titled “SHC Container Status Derivation”flowchart LR D1{ds}
D1-->|created|DC{"Active converge?"} DC-->|yes|STARTING[starting] DC-->|no|DEGRADED[degraded] D1-->|restarting|STARTING D1-->|dead|DEGRADED[degraded] D1-->|paused|PAUSED[paused]
D1-->|running|D2{dh} D2-->|starting|STARTING D2-->|healthy|RUNNING[running] D2-->|"empty or none"|RUNNING D2-->|unhealthy|DEGRADED D2-->|"unrecognized"|RUNNING
D1-->|exited|D3{ec}
D3-->|0|D4{rp} D4-->|no|COMPLETE[complete] D4-->|on-failure|COMPLETE D4-->|always|DEGRADED
D3-->|!=0|D5{rp} D5-->|no|FAILED[failed] D5-->|on-failure|DEGRADED D5-->|always|DEGRADED
D3-->|NULL|D6{rp} D6-->|no|UNKNOWN[unknown] D6-->|on-failure|UNKNOWN D6-->|always|DEGRADEDContainer Status Derivation Table
Section titled “Container Status Derivation Table”| Docker Status | Exit Code | Health | Restart Policy | SHC Status |
|---|---|---|---|---|
created during active converge | - | any | any | starting |
created outside active converge | - | any | any | degraded |
restarting | - | any | any | starting |
running | - | starting | any | starting |
running | - | healthy | any | running |
running | - | "" / none | any | running |
running | - | unhealthy | any | degraded |
running | - | (unrecognized) | any | running |
paused | - | any | any | paused |
exited | 0 | any | no | complete |
exited | 0 | any | on-failure | complete |
exited | 0 | any | always | degraded |
exited | !=0 | any | no | failed |
exited | !=0 | any | on-failure | degraded |
exited | !=0 | any | always | degraded |
exited | NULL | any | no | unknown |
exited | NULL | any | on-failure | unknown |
exited | NULL | any | always | degraded |
exited | any, or NULL | any | default (no restart: key) | degraded |
dead | any | any | any | degraded |
A NULL exit code is never read as 0: under no and on-failure only the code tells complete from failed, so the row reads unknown, which the service tier counts in no bucket (degraded) and the health rollup never counts as ready.
Every surface reads a container from the same two facts, its exit code and the restart policy its package authored. The Deployment listing, the Deployment read and the stack status read fold the cluster’s replicated container rows; this node’s engine probe stands in only for a deployment the rows cannot answer (no fresh row, or an operation in flight), and it reads the exit code and the shc.restart.policy and shc.restart.defaulted labels off each container. A run-once that exited 0 therefore reads complete on every route, and one that exited non-zero reads failed, its deployment degraded with the reason traefik-init failed, so it cannot be running.
Layer 2: Service
Section titled “Layer 2: Service”SHC Service Status (Derived)
Section titled “SHC Service Status (Derived)”Service status is derived from container statuses, desired replica count, and restart policy:
| Status | Meaning |
|---|---|
stopped | Intentionally scaled to zero (desired_replicas = 0) |
paused | Every container is paused (an intentional full-service suspension, e.g. a backup quiesce). A partial pause is treated as alive and reads running |
starting | Containers starting up (or no containers observed yet) |
running | Enough replicas alive: count(running + paused) >= desired_replicas |
restarting | A container is mid-restart |
complete | All jobs finished successfully (job mode) |
failed | All jobs failed (job mode) |
degraded | Partial availability or unexpected state |
Paused counts as alive for the running rule: running + paused >= desired_replicas reads running, so a short-lived rolling pause (some containers paused, the rest up) does not flap the status. Only when all containers are paused does the service surface as paused. paused is a derived status.
SHC Service Status Derivation
Section titled “SHC Service Status Derivation”flowchart LR D1{dr} D1-->|= 0|STOPPED[stopped]
D1-->|> 0|DP{"all c : cs=paused (total>0)"} DP-->|yes|PAUSED[paused]
DP-->|no|D2{"n(running + paused) >= dr"} D2-->|yes|RUNNING[running]
D2-->|no|DR{"any c : cs=restarting"} DR-->|yes|RESTARTING[restarting]
DR-->|no|D3{"any c : cs=starting or n(c)=0"} D3-->|yes|STARTING[starting]
D3-->|no|D4{rp} D4-->|no / on-failure|D5{"all c : cs=complete"} D5-->|yes|COMPLETE[complete] D5-->|no|D6{"all c : cs=failed"} D6-->|yes|FAILED[failed] D6-->|no|DEGRADED[degraded]
D4-->|always|DEGRADEDService Status Derivation Logic
Section titled “Service Status Derivation Logic”- If
desired_replicas = 0->stopped - If
total > 0andall(cs = paused)->paused(whole-service suspension; checked before the running rule so an all-paused service does not readrunning) - If
count(cs = running) + count(cs = paused) >= desired_replicas->running(paused counts as alive; a partial pause lands here) - If
any(cs = restarting)->restarting - If
any(cs = starting)ortotal = 0->starting - If job mode (
rp = nooron-failure):- If
all(cs = complete)->complete - If
all(cs = failed)->failed
- If
- Else ->
degraded
Note: Job completion/failure (step 6) requires at least one container and the job restart policy. If a service has desired_replicas > 0 but no containers exist yet, step 5 (total = 0) makes the status starting. A daemon-mode (rp = always) service that reaches step 6 falls through to degraded.
SHC Service Health (Derived)
Section titled “SHC Service Health (Derived)”Service health is derived from the per-container (derived) health using “worst wins” priority:
| Health | Meaning |
|---|---|
unknown | No containers, or all containers were undeterminable (ch = unknown) |
unhealthy | Any container unhealthy |
starting | Any container health starting |
healthy | Any container healthy |
none | All containers non-live / not-applicable (ch = none) |
After checking unhealthy, starting and healthy, any unknown container
health yields unknown. The result is none only when no other signal
exists.
SHC Service Health Derivation
Section titled “SHC Service Health Derivation”flowchart LR D1{"n(c)"} D1-->|= 0|UNKNOWN[unknown]
D1-->|> 0|D2{"any c : ch=unhealthy"} D2-->|yes|UNHEALTHY[unhealthy]
D2-->|no|D3{"any c : ch=starting"} D3-->|yes|STARTING[starting]
D3-->|no|D4{"any c : ch=healthy"} D4-->|yes|HEALTHY[healthy]
D4-->|no|D5{"any c : ch=unknown"} D5-->|yes|UNKNOWN
D5-->|no|NONE[none]Layer 3: Deployment
Section titled “Layer 3: Deployment”SHC Deployment Status
Section titled “SHC Deployment Status”Deployment status has two categories:
Operational states: Set by user commands, block derived updates:
| Status | Trigger |
|---|---|
created | shc install |
starting | shc start or compose up |
stopping | shc stop |
updating | shc update |
removing | shc uninstall |
Derived states: Computed from service statuses when not in operational state (the derivation never overwrites an operational state; callers MUST check IsOperational() first):
| Status | Meaning |
|---|---|
running | All services running (or a partial pause / all-running-or-complete mix) |
paused | Every service is paused (whole-deployment suspension, e.g. a backup quiesce). A partial pause reads running; a paused mix that also has a bad state reads degraded |
degraded | Any service degraded |
complete | All services complete (job deployment) |
failed | All services failed |
stopped | All services stopped |
paused at the deployment tier surfaces only when all services are paused. The check runs first, so an all-paused deployment does not read running. A partial pause (some services running or done, the rest paused, no bad state) is treated as alive and reads running. A pause mixed with any degraded or failed service reads degraded. paused is derived at both tiers.
Moving: a planned move
Section titled “Moving: a planned move”A replica move takes a replica down on purpose. A replica that carries a volume is stopped while its data is copied to the node it goes to; one that carries none is brought up there beside its copy, and the copy then goes. shc update --where and --prefer, a node drain and shc move of a replica all move replicas this way. For that while the deployment is short a ready replica by plan, and that is not a fault.
So shc ls, shc status and their -o json read such a deployment moving, never degraded:
| Word | When |
|---|---|
moving | A planned move holds a replica down, and every container or replica that is not ready is that replica, on the node it leaves or the node it goes to |
degraded | Anything else is down: a replica lost outside a plan (its container killed, its node gone), or a service no move takes |
The row’s reason names each replica being moved, the node it leaves, the node it goes to, since when, and the plan’s downtime when the plan made one:
counter-r0 moving from far10 to home32 since 2026-10-09T18:31:04Z, about 32s downapi-r0 moving from far10 to home32 since 2026-10-09T18:30:51Z, its copy serving until it is upcounter-r0 moved from far10 to home32, starting thereThe replica’s own row in the shc status -s replica table reads moving too, and the services that are not being moved read as they are. -o json carries each move under moving. A move that has landed is read for two minutes more, while the replica comes up on its new node and its old copy goes; the deployment reads healthy again once it is up. A move that fails puts the replica back and stops reading moving at once.
A whole move takes the deployment down as a whole: shc move of a deployment that runs on one node, a drain’s whole moves, and a move with a rename or into another tenant or environment stop it, bring it up on the target and tear the source down. For that while its row reads moving too, from whatever word the move leaves it in (stopped, unknown, starting, degraded, unhealthy), and every replica it took reads moving. A move under another name or scope reads moving on the source row and on the target row; the target reads healthy once it is up. A running or failed row keeps its word, and a deployment lost outside a plan keeps its fault word.
the deployment moving from far10 to home32 since 2026-10-09T20:30:00Z, about 2m4s downthe deployment moved from far10 to home32, starting thereThe move records the replica it takes, or the deployment it moves as a whole, in the replicated store before it takes it down, so every node answers the same. The record counts only while the move holds the deployment’s relocation lease: a move whose process died stops reading moving within the lease’s time to live. moving is read at the answer and never stored, so the stored status and everything that gates on it are unchanged.
SHC Deployment Status: Pure Derivation
Section titled “SHC Deployment Status: Pure Derivation”When the deployment is not in an operational state, its status is derived from the per-service statuses (first matching rule wins):
flowchart LR S0{"n(s)"} S0-->|= 0|STOPPED0[stopped]
S0-->|> 0|DP{"all s : ss=paused"} DP-->|yes|PAUSED[paused]
DP-->|no|D2{"all s : ss=running"} D2-->|yes|RUNNING[running]
D2-->|no|D3{"any s : ss=starting"} D3-->|yes|STARTING0[starting]
D3-->|no|D4{"all s : ss=complete"} D4-->|yes|COMPLETE[complete]
D4-->|no|D5{"all s : ss=failed"} D5-->|yes|FAILED[failed]
D5-->|no|D6{"all s : ss=stopped"} D6-->|yes|STOPPED[stopped]
D6-->|no|D7{"running+complete = total and running>0"} D7-->|yes|RUNNING
D7-->|no|D8{"running+paused+complete = total and (running>0 or paused>0)"} D8-->|yes|RUNNING
D8-->|no|D9{"any s : ss=degraded, or any running/failed/paused"} D9-->|yes|DEGRADED[degraded]
D9-->|no|D10{"any s : ss=complete"} D10-->|yes|COMPLETE
D10-->|no|STOPPEDThe two paused-aware rungs (the all-paused check at the top, and the running+paused+complete partial-pause rung) are what keep a rolling backup pause reading running rather than degraded, while a pause mixed with a bad state still drops to degraded.
SHC Deployment Health (Derived)
Section titled “SHC Deployment Health (Derived)”Deployment health uses the service tier’s priority order. unknown follows
healthy; an all-unknown deployment reports unknown:
| Health | Meaning |
|---|---|
unknown | No services, or all service health was undeterminable (sh = unknown) |
unhealthy | Any service unhealthy |
starting | Any service health starting |
healthy | Any service healthy |
none | All services non-live / not-applicable |
flowchart LR D1{"n(s)"} D1-->|= 0|UNKNOWN[unknown]
D1-->|> 0|D2{"any s : sh=unhealthy"} D2-->|yes|UNHEALTHY[unhealthy]
D2-->|no|D3{"any s : sh=starting"} D3-->|yes|STARTING[starting]
D3-->|no|D4{"any s : sh=healthy"} D4-->|yes|HEALTHY[healthy]
D4-->|no|D5{"any s : sh=unknown"} D5-->|yes|UNKNOWN
D5-->|no|NONE[none]The Single Derivation Pipeline (DeriveDeployment)
Section titled “The Single Derivation Pipeline (DeriveDeployment)”DeriveDeployment runs the same derivation for every set of runtime
observations. Identical inputs produce identical results.
The entry point is DeriveDeployment(containers []RawContainer, desiredReplicas map[string]int) -> (DeploymentStatus, StackHealth). It groups the raw observations by service (preserving first-seen order) and runs the chain:
per container -> DeriveContainerStatus + DeriveContainerHealthper service -> DeriveServiceStatus + DeriveServiceHealthdeployment -> DeriveStackStatus + DeriveStackHealthUniversal input shape: every surface builds a slice of RawContainer from its own probe and hands it to the pipeline, so the transform lives in exactly one place:
type RawContainer struct { Service string // compose service this container belongs to (grouping key) RawStatus string // raw docker status: running/exited/created/restarting/paused/dead RawHealth string // raw docker health: healthy/unhealthy/starting/"", or "unknown" when undeterminable ExitCode *int // meaningful for exited/dead; nil derives unknown (degraded under always), never complete RestartPolicy string // service restart policy; normalized on the way in InFlightConverge bool}Surfaces that use the derivation:
| Surface | Probe it builds RawContainer from |
|---|---|
Live recorder (propagate.go) | DB container rows |
Install spinner (compose_progress.go) | Runtime project observations |
Disk-fallback reader (router.go) | label-filtered docker ps |
Status derivation consumes raw health; health aggregation consumes
DeriveContainerHealth. An unreadable observation stays unknown.
Two variants expose the per-service breakdown and the explicit-service-set case:
DeriveDeploymentServices: returns the per-service results (used by the install spinner to fill per-service status + health rows) and the rolled-up deployment status/health.DeriveDeploymentForServices: runs the chain over an explicit, ordered service set so a service with zero observed containers still counts towardtotal = len(services)(the recorder needs this, orrunning == totalwould spuriously fire when a service has lost all its containers). A service absent fromdesiredReplicasdefaults to 1 desired, so a single observed container readsrunningrather than being judged against zero (which would forcestopped).
Reconciliation Actions
Section titled “Reconciliation Actions”The node agent owns crash restarts. The runtime’s own restart policy is
no; the authored policy lives in labels. At boot the agent waits for the
control plane to confirm its lease and assignments before starting
containers. A restart requires the replica to remain assigned here, and a
single writer also requires a live node lease.
modules/nodeagent/policy.go defines the restart modes and policy labels.
The independent watchdog
stops single writers after lease expiry even when shcd cannot act.
A degraded status alone does not authorise a restart.
OTEL Metrics
Section titled “OTEL Metrics”SHC exports status metrics to OpenTelemetry:
Status Metrics (with status label)
Section titled “Status Metrics (with status label)”| Metric | Labels | Description |
|---|---|---|
shc_container_status | deployment, service, container, status | Container SHC status (gauge=1) |
shc_service_status | deployment, service, status | Service status (gauge=1) |
shc_deployment_status | deployment, status | Deployment status (gauge=1) |
The gauge value is always 1 when exported, allowing aggregation queries like:
sum by (status) (shc_deployment_status{deployment="myapp"})Health Metrics
Section titled “Health Metrics”| Metric | Labels | Description |
|---|---|---|
shc_container_health_status | tenant, environment, stack, app, node_id, container_id, container_name, service | Container health status (1=healthy, 0=unhealthy, -1=none) |
shc_deployment_health | tenant, stack, environment | Deployment health (1=healthy, 0=unhealthy) |
Event Flow
Section titled “Event Flow”flowchart LR subgraph DOCKER DE[Docker Events] DH[Docker Health] end
subgraph SHC_INGEST EW[Event Watcher] HP[Health Poller] end
subgraph SHC_STATE CM[ContainerModel] SM[ServiceModel] DM[DeploymentModel] end
DE-->EW DH-->HP EW-->CM HP-->CM CM-->SM SM-->DMEvent Sources
Section titled “Event Sources”| Source | Events | Frequency |
|---|---|---|
| Docker Events | create, start, stop, die, kill, pause, unpause, destroy, oom, health_status | Real-time |
| Health Poller | Container stats, health status | Every 60 seconds |
Status Propagation
Section titled “Status Propagation”When a container event occurs:
- Container layer: Update
ContainerModelwith Docker status/health - Service layer: Recompute
ServiceModel.statusandServiceModel.healthfrom containers - Deployment layer: Recompute
DeploymentModel.statusandDeploymentModel.healthfrom services
Important: Status propagation is triggered when either:
- Container
statuschanges (e.g., running → exited) - Container
healthchanges (e.g., healthy → unhealthy)
This ensures that health check failures properly propagate to deployment status (causing running → degraded).
Input/Output Summary
Section titled “Input/Output Summary”| Derivation | Inputs | Output |
|---|---|---|
| SHC Container Status | ds, dh (raw), ec, rp | cs |
| SHC Container Health | dh (raw), ds | ch |
| SHC Service Status | cs[], dr, rp | ss (incl. paused) |
| SHC Service Health | ch[] | sh (incl. unknown rung) |
| SHC Deployment Status | ss[], user_action, rt | deployment status (incl. paused) |
| SHC Deployment Health | sh[] | deployment health (incl. unknown rung) |
Whole pipeline (DeriveDeployment) | []RawContainer, dr map | (DeploymentStatus, StackHealth) |
Implementation Files
Section titled “Implementation Files”| Component | File | Description |
|---|---|---|
| Container Status | modules/app/container_status.go | DeriveContainerStatus(ds, ec, dh, rp) |
| Container Health (derived) | modules/app/container_status.go | DeriveContainerHealth(rawHealth, dockerStatus) |
| Service Status | modules/app/service_status.go | DeriveServiceStatus(containers, dr, rp) |
| Service Health | modules/app/service_status.go | DeriveServiceHealth(containers) |
| Deployment status (derived) | modules/app/stack_status.go | DeriveStackStatus(services) |
| Deployment health (derived) | modules/app/stack_status.go | DeriveStackHealth(services) |
| Deployment Pipeline (canonical entry) | modules/app/deployment_status.go | DeriveDeployment / DeriveDeploymentServices / DeriveDeploymentForServices, RawContainer |
Deployment status constants (incl. paused) | modules/app/types.go | DeploymentStatus* consts, IsOperational() |
| Operational States | modules/app/types.go | DeploymentStatus.IsOperational() |
| Status propagation (recorder) | modules/app/propagate.go | builds RawContainer from DB rows, calls the pipeline |
| Restart policy | modules/nodeagent/policy.go | Authored policy labels read by the node agent |
| Status Metrics | modules/metrics/definitions.go | shc_container_status, shc_service_status, shc_deployment_status |
| Restart Policy Normalization | modules/app/models.go | NormalizeRestartPolicy() |
Derivation Logic (Pseudocode)
Section titled “Derivation Logic (Pseudocode)”Container Status Derivation
Section titled “Container Status Derivation”func deriveContainerStatus(dockerStatus, health string, exitCode int, restartPolicy string, converging bool) string { switch dockerStatus { case "created": if converging { return "starting" } return "degraded" case "restarting": return "starting" case "dead": return "degraded" case "paused": return "paused" case "running": switch health { case "starting": return "starting" case "unhealthy": return "degraded" default: return "running" // healthy, "", none, or any unrecognized word } case "exited": isRestartNo := restartPolicy == "no" || restartPolicy == "on-failure" if exitCode == 0 { if isRestartNo { return "complete" } return "degraded" // daemon exited unexpectedly } if restartPolicy == "no" { return "failed" } return "degraded" // will be restarted } return "degraded" // catch-all}Container Health Derivation
Section titled “Container Health Derivation”// Derives the CANONICAL ch from the RAW docker health and the RAW status.// Health is meaningful only while a container is live, so it is a function// of BOTH inputs, not the raw health alone.func deriveContainerHealth(rawHealth, dockerStatus string) string { if rawHealth == "unknown" { return "unknown" } if dockerStatus != "running" { return "none" // not live -> no active check -> not-applicable } switch rawHealth { case "unhealthy": return "unhealthy" case "starting": return "starting" case "healthy", "", "none": return "healthy" // live + (check passing OR no check) = healthy default: return "unknown" // a live container with a word we don't recognize }}Service Status Derivation
Section titled “Service Status Derivation”func deriveServiceStatus(containers []Container, desiredReplicas int, restartPolicy string) string { if desiredReplicas == 0 { return "stopped" }
var running, paused, restarting, starting, complete, failed int for _, c := range containers { switch c.Status { case "running": running++ case "paused": paused++ case "starting": starting++ case "complete": complete++ case "failed": failed++ } // "restarting" is inferred from the raw docker status by the // recorder; the container tier maps it to "starting" today, so // it is rarely a distinct count here. } total := len(containers)
// Whole-service pause (e.g. a backup quiesce), checked before the // running rule so an all-paused service does not read "running". if total > 0 && paused == total { return "paused" }
// Paused counts as "alive enough"; a partial pause lands here as // "running" rather than flapping the status. if running+paused >= desiredReplicas { return "running" } if restarting > 0 { return "restarting" } if starting > 0 || total == 0 { return "starting" } if restartPolicy == "no" || restartPolicy == "on-failure" { if complete == total { return "complete" } if failed == total { return "failed" } } return "degraded" // catch-all (incl. daemon mode)}Service Health Derivation
Section titled “Service Health Derivation”func deriveServiceHealth(containers []Container) string { if len(containers) == 0 { return "unknown" }
// Container health here is the DERIVED ch, not the raw docker health. hasUnhealthy, hasStarting, hasHealthy, hasUnknown := false, false, false, false for _, c := range containers { switch c.Health { case "unhealthy": hasUnhealthy = true case "starting": hasStarting = true case "healthy": hasHealthy = true case "unknown": hasUnknown = true } // "none" (non-live containers) contributes nothing. }
if hasUnhealthy { return "unhealthy" } if hasStarting { return "starting" } if hasHealthy { return "healthy" } if hasUnknown { return "unknown" } return "none" // all containers non-live / not-applicable}Deployment Status Derivation
Section titled “Deployment Status Derivation”Operational states (queued, created, installing, starting, stopping, updating, removing) block derived updates; callers check status.IsOperational() and skip the derivation entirely. The pure derivation over the per-service statuses is count-based:
func deriveDeploymentStatus(services []Service) string { if len(services) == 0 { return "stopped" }
var running, starting, complete, failed, stopped, degraded, paused int total := len(services) for _, s := range services { switch s.Status { case "running": running++ case "starting": starting++ case "complete": complete++ case "failed": failed++ case "stopped": stopped++ case "degraded": degraded++ case "paused": paused++ } }
switch { case paused == total: return "paused" // whole deployment paused (e.g. backup quiesce) case running == total: return "running" case starting > 0: return "starting" case complete == total: return "complete" case failed == total: return "failed" case stopped == total: return "stopped" case running+complete == total && running > 0: return "running" case running+paused+complete == total && (running > 0 || paused > 0): return "running" // partial pause w/ no bad state -> alive case degraded > 0: return "degraded" case running > 0 || failed > 0 || paused > 0: return "degraded" // paused mixed with a bad state -> degraded case complete > 0: return "complete" } return "stopped"}Key Design Decisions
Section titled “Key Design Decisions”1. Status vs Health Separation
Section titled “1. Status vs Health Separation”Status and health are independent concepts:
- Status: Is the container/service/deployment running?
- Health: Is the application inside working correctly?
A service can be running but unhealthy (app responding but health check failing).
2. Operational vs Derived States
Section titled “2. Operational vs Derived States”Deployment status has two categories:
- Operational: Set by user commands, blocks derived updates during operations
- Derived: Computed from service states when no operation is in progress
This prevents status flickering during deployments.
3. Restart Policy Determines Intent
Section titled “3. Restart Policy Determines Intent”The restart policy tells SHC what the user expects:
no/on-failure: Job mode; exit is expectedalways: Daemon mode; should always be running
The same Docker status (exited) produces different SHC statuses based on intent.
4. Runtime-Aware Reconciliation
Section titled “4. Runtime-Aware Reconciliation”SHC selects the node runtime for execution. The node agent uses assignments and lease state to decide whether a container may start or restart.
5. Idempotent Reconciliation
Section titled “5. Idempotent Reconciliation”compose up -d is idempotent; running it multiple times is safe. Combined with exponential backoff (60s base doubling to a 1800s cap, applied only after a failed attempt), this prevents rapid repeated actions.
Podman Considerations
Section titled “Podman Considerations”Podman container states are nearly identical to Docker:
| State | Docker | Podman | Difference |
|---|---|---|---|
created | Yes | Yes | Same |
running | Yes | Yes | Same |
paused | Yes | Yes | Same |
exited | Yes | Yes | Same |
restarting | Yes | Yes | Same |
dead | Yes | No | Podman doesn’t have this |
configured | No | Yes | Podman-only (config exists, not initialized) |
removing | Yes | Yes | Same |
SHC treats Podman containers the same as Docker containers. The configured state maps to starting in SHC, and if dead is not present, SHC won’t see it (containers go directly to exited).
Kubernetes Reference
Section titled “Kubernetes Reference”For context, here’s how Kubernetes handles similar concepts:
Pod Phases (equivalent to container)
Section titled “Pod Phases (equivalent to container)”| Phase | Description |
|---|---|
Pending | Scheduled but not yet running |
Running | At least one container running |
Succeeded | All containers exited with code 0 |
Failed | At least one container exited with non-zero |
Unknown | Cannot determine state (node communication issue) |
Deployment Status (equivalent to deployment)
Section titled “Deployment Status (equivalent to deployment)”status: replicas: 3 # Desired readyReplicas: 2 # Actually ready availableReplicas: 2 # Passed readiness probe unavailableReplicas: 1 conditions: - type: Available status: "True" # At least minAvailable replicas ready - type: Progressing status: "True" # Rollout in progress or completeKey Kubernetes Behaviors
Section titled “Key Kubernetes Behaviors”- Declarative: You specify desired state, K8s reconciles continuously
- Self-healing: Deleted pods are immediately recreated
- Pause not supported: No concept of pausing a pod (you scale to 0 instead)
- Rich status: Conditions, replica counts, rollout status all available
- Probe-based health: Liveness and readiness probes determine health
SHC vs Kubernetes Mapping
Section titled “SHC vs Kubernetes Mapping”| Kubernetes | SHC Equivalent |
|---|---|
Pod Running | Container running |
Pod Succeeded | Container complete |
Pod Failed | Container failed |
Deployment Available=True | Deployment running |
Deployment Progressing=True | Deployment updating or starting |
| Deployment replicas mismatch | Deployment degraded |
| Readiness probe passing | Container health healthy |
| Liveness probe failing | Container killed, status becomes degraded |