Plan: real per-container status during shc install (install-time live service tracking)

Status: IMPLEMENTED — internal/modules/app/compose_progress.go + swarm_progress.go (the swarm path went beyond this plan’s v1 scope), wired into ExecuteCompose for both runtimes; full docker-free unit coverage per the TDD steps below (derive table, both ps shapes, happy path, poller-error isolation, up-failure frame, ctx-cancel).

The shc install spinner (internal/cli/livedisplay) is synthetic. ExecuteCompose parses the compose YAML for service names (serviceNamesFromCompose, install_runner.go:57) and hand-sets every row’s state through one emit() closure: resolving → blanket pulling (install_runner.go:415-419) → run the blocking docker compose up -d --wait (compose_driver.go:23-57, captured into a buffer by runDocker, :92-101) → blanket flip every service to healthy (:491-495). All service rows therefore flip pulling → healthy together the instant up --wait returns. No docker compose ps, no docker inspect, no goroutine runs during the up.

The documented design (docs/architecture/status-states.md, docs/architecture/events/live-service-tracking.md) wants the live display to render the real container→service rollup, refreshed as container facts change — the same semantic states the rest of SHC uses. The Python predecessor achieved this: up --wait blocked, but a concurrent DeploymentTracker._update_loop polled live rows (kept fresh by docker compose ps) and rendered real per-service status/health while the up ran (deployment_tracker.py:124-292, compose.py:309-388,720-738).

Go already has the real observe→rollup→reconcile logic, but only in the out-of-band health crons (2s tick / 1m poll / 1m reconcile) — it is not wired into the install display.

During the up --wait window of ExecuteCompose, run a concurrent poller that shells docker compose ps --format json, derives each service’s real state+health, and feeds the existing Observer.Step with accurate per-service rows. The spinner then shows real creating → starting → healthy (and unhealthy/failed) progression instead of a single blanket flip.

1. The poller is purely additive observability. Any poller error (ps failure, JSON parse error, docker missing) is swallowed and skips that tick — it must NEVER change the install’s success/failure. dockerComposeUp’s return value remains the sole source of truth for the result.
2. No contract change. operations.ServiceInfo / DeploymentPayload / StepObserver shapes are unchanged (shared with the daemon SSE encoder and the CLI decoder — see the warning at observer.go:8-11). The poller only builds richer Services slices.
3. One emitter at a time. During the up window the poller is the only caller of emit/Observer.Step; the main goroutine resumes emitting only after the poller has fully stopped (WaitGroup join), so payloads never interleave with stale slices.
4. No goroutine leak. The poller stops on up-return, on error, and on ctx-cancel, in all paths.
5. CLI self-contained. The poller shells docker compose ps (like compose_driver.go and the reconcile actions), it does NOT pull in the docker SDK.
6. Both paths benefit unchanged. Same ExecuteCompose runs daemon-side with StoreObserver; the extra Step calls become richer SSE step events for free. No daemon-side change required.

• In: the default compose runtime path (dockerComposeUp). TTY and non-TTY render (livedisplay already handles both; poller adds change-detection so non-TTY isn’t spammed).
• Out (v1, documented as follow-up):
  • Swarm path (dockerStackDeploy) — docker stack ps has a different shape; keep the current blanket flip for swarm.
  • SHC_COMPOSE_NO_WAIT=1 — the up returns immediately so the poller stops early; v1 keeps current behavior there (note in code + this doc). A bounded post-up poll window is a separate change because it alters the synchronous-readiness contract.

New file internal/modules/app/compose_progress.go (package app):

// deriveServiceState maps a `docker compose ps` State+Health(+ExitCode) to the
// spinner state string livedisplay's pickIcon understands. Mirrors the Python
// shc_status ladder (service_model.py:205-254) and reuses the health vocabulary
// of metrics/collectors.parseHealth.
func deriveServiceState(state, health string, exitCode int) (spinnerState string)
//   running + healthy|""    -> "healthy"   (✔)
//   running + starting      -> "starting"  (spinner)
//   running + unhealthy     -> "unhealthy" (✘)
//   created                 -> "creating"  (spinner)
//   restarting              -> "restarting"(spinner)
//   exited/dead, exit==0    -> "complete"  (✔)  // one-shot init success
//   exited/dead, exit!=0    -> "failed"    (✘)
//   (absent from ps output) -> keep prior "pulling"/"creating" (spinner)

// composePSEntry is the subset of `docker compose ps --format json` we read.
type composePSEntry struct {
    Service  string `json:"Service"`
    State    string `json:"State"`
    Health   string `json:"Health"`
    ExitCode int    `json:"ExitCode"`
}

// parseComposePS handles BOTH compose output shapes: a JSON array (newer) and
// NDJSON, one object per line (older). Returns []composePSEntry.
func parseComposePS(raw []byte) ([]composePSEntry, error)

// runComposePS is the injectable command seam (real impl shells
// `docker compose -p <project> -f <path> ps -a --format json`); tests swap it.
type psFunc func(ctx context.Context, project, composePath string) ([]byte, error)

// composeUpWithProgress runs the blocking up while a goroutine polls ps every
// ~tickInterval and emits real per-service payloads via emit. Returns the up's
// error verbatim. Guarantees the poller has stopped before returning.
func composeUpWithProgress(
    ctx context.Context,
    project, composePath string,
    services []operations.ServiceInfo, // names + DesiredReplicas, pre-rendered
    emit func(status string, services []operations.ServiceInfo, completed, total int),
    ps psFunc, // nil -> real docker
    up func(ctx context.Context) error, // closure over dockerComposeUp(ctx, project, composePath)
) error

Behavior of composeUpWithProgress:

1. pollCtx, cancel := context.WithCancel(ctx); defer cancel().
2. Start one goroutine (tracked by a sync.WaitGroup):
   • ticker at tickInterval (default 500ms — matches live-service-tracking.md:173-177; overridable via SHC_INSTALL_POLL_MS for tests/tuning).
   • each tick: ps(...); on error → continue (best-effort). Build a fresh services slice = base names with state from deriveServiceState, CurrentReplicas = count of running containers for that service, DesiredReplicas carried from the base. completed = count of healthy/complete. Status string = "deploying".
   • change-detection: keep the last emitted slice; skip emit if identical (prevents TTY redraw churn and non-TTY log spam).
3. Run up(pollCtx) (blocking) in the calling goroutine.
4. cancel(); wg.Wait() — poller fully stopped.
5. Do ONE final ps(...) and emit("running", finalServices, completed, total) so the terminal frame reflects real state (replaces the blanket healthy flip). On up error, still do the final ps so failed services show ✘ where they actually failed, then return the up’s error.

Wiring in install_runner.go (replaces lines 473-495):

if total > 0 {
    switch opts.Runtime {
    case string(RuntimeSwarm):
        // unchanged: blanket flip for swarm (out of v1 scope)
        if err := dockerStackDeploy(ctx, project, composePath); err != nil { /* failed flip + emit */ }
        for i := range observedServices { observedServices[i].State = "healthy"; observedServices[i].CurrentReplicas = 1 }
        emit("running", total, total)
    default:
        upErr := composeUpWithProgress(ctx, project, composePath, observedServices,
            func(status string, svcs []operations.ServiceInfo, c, t int) {
                observedServices = svcs               // keep state.json/down path consistent
                opts.Observer.Step(operations.DeploymentPayload{
                    Deployment: deployment, Services: svcs, Progress: [2]int{c, t}, Status: status,
                })
            }, nil, func(c context.Context) error { return dockerComposeUp(c, project, composePath) })
        if upErr != nil { /* final ps already emitted failed states */ return upErr }
    }
} else {
    // headless: unchanged
    for i := range observedServices { observedServices[i].State = "healthy"; observedServices[i].CurrentReplicas = 1 }
    emit("running", total, total)
}

Note: the emit closure stays for the pre-up phases (resolving/pulling) and the failure paths above the up (install_runner.go:446-449); only the up block delegates to the poller. The poller calls opts.Observer.Step directly (guard opts.Observer != nil).

1. deriveServiceState truth table — compose_progress_test.go, table-driven over the mapping above (running/healthy, running/starting, running/unhealthy, created, restarting, exited-0, exited-1, dead-137). Pure function, no docker.
2. parseComposePS both shapes — feed a JSON array fixture and an NDJSON fixture (captured from docker compose ps --format json on two compose versions); assert equal parsed slices; assert a malformed line returns an error (caller will swallow it).
3. composeUpWithProgress happy path — inject ps returning a scripted sequence (created → starting → healthy) and an up that blocks on a channel for N ticks then returns nil. Assert: emitted payloads show the progression; final emit is "running" with all healthy; completed==total; change-detection collapsed identical ticks; no goroutine left (use goleak or a WaitGroup assertion).
4. composeUpWithProgress poller-error isolation — ps always errors; up returns nil. Assert: function returns nil (success), at least the final emit fired, no panic.
5. composeUpWithProgress up-failure — ps shows one service failed; up returns a wrapped error. Assert: function returns that exact error; final emit shows the failed service as "failed" (✘), not blanket healthy.
6. composeUpWithProgress ctx-cancel — cancel ctx mid-run; assert the poller goroutine exits and the function returns promptly (up closure observes cancellation).
7. Wiring smoke (no docker) — call ExecuteCompose shape with a stub by injecting the ps/up seam through a test-only hook, OR keep the seam internal and assert via an existing app package test that the blanket-flip code path for total==0 and swarm is unchanged. (Real docker behavior stays covered by make test/apps.)

• make build green; make lint green (coded errors only on prod paths — reuse X500059DCKCMP for ps shellouts or add a dedicated factory; no plain errors.New).
• New unit tests (steps 1–6) pass; existing internal/modules/app + internal/core/operations
  • internal/cli/livedisplay tests still pass.
• Manual TTY check: shc install <multi-service-app> --host … shows services converging individually (e.g. db healthy before app starting) rather than all flipping at once.
• make test/apps for one multi-service app (e.g. apps/plane or apps/nextcloud) still green — the poller did not alter install outcome or timing materially.

Gaps 1–3 and 5 from the Python/Go comparison: concurrent per-container tracking during install, layered rollup feeding the spinner, service-level health distinction mid-deploy, and (via the final-ps frame) a truthful terminal state instead of a trusted-exit-code flip. Out of scope here: gap 4 (three-tier SSE event hierarchy) and gap 6 (wiring the 2s tick cache into the deployment poll) — separate changes.