Health module

Source: modules/health/.

modules/health/
├── command.go
├── commands/
├── crons/
│   ├── tick.go          # 15s leader-only cron
│   ├── poll.go          # 1min leader-only cron
│   ├── reconcile.go     # 1min leader-only cron
│   └── maintenance.go   # DB-tracked cron jobs
├── events.go
├── poll_deployments.go
├── poll_nodes.go
├── reconcile.go
├── router.go            # /health, /ready endpoints
├── state.go             # module-level HealthState singleton
├── tasks.go
└── types.go             # HealthState, NodeHealth, DeploymentHealth

Three concerns, all rooted in the health module:

1. Runtime health endpoints — liveness, readiness, and a detailed health report for load balancers, shc status, and monitoring.
2. Cron tiers — the structured periodic work that polls the cluster, reconciles degraded deployments, and purges old rows.
3. The HealthState singleton — the module-level snapshot of cluster state that crons write to and the HUD / endpoints read from.

The module owns no subsystem lifecycle; HealthState is a plain module-level object decoupled from subsystem start/stop.

All periodic work runs through three leader-only cron tiers plus a set of DB-tracked maintenance cron jobs. Every cron uses skip-if-running — if the previous invocation has not completed, the next tick is silently skipped.

All intervals are config-driven via function references resolved at scheduler start — see cron in the communication-patterns doc.

Fast, lightweight data gathering. Runs sequentially within one tick:

1. node_metrics — broadcast gather, each node returns system metrics; leader writes to HealthState.
2. container_metrics — broadcast gather, each node returns docker stats.
3. global_diagnostics — local leader query for task/worker/cron counts.
4. quota_metrics — local leader collection of quota gauges.
5. vault_check — broadcast fire-and-forget; if any node reports locked and the leader holds the password in memory, re-broadcast unlock (see Vault unlock flow).

Moderate-weight state sync:

1. deployment_sync — broadcast docker compose ps to every node; leader writes container/service state to DB and to HealthState.
2. retention_config_check — leader-local audit of retention policies.

Self-healing for degraded deployments plus infrastructure enforcement. Broadcast fire-and-forget; each node acts on its own local state.

Per-node flow:

1. Read in-memory HealthState for this node’s deployments.
2. If any deployment is degraded, do a live docker compose ps re-check before acting — never reconcile on stale in-memory state alone.
3. If still degraded:
   • Skip if already in the resolving set for this stack.
   • Skip if cooldown has not elapsed since last attempt.
   • Apply exponential backoff on repeated failures.
   • Add to resolving set → run docker compose up -d → remove on completion.
4. All cooldown / backoff state is in-memory only and resets on process restart.

Infrastructure enforcement (same cron): Traefik port-binding check and restart, OpenObserve health check, docker network creation.

Run via task cron, so each invocation creates a TaskModel row and is audit-tracked.

The Cleanup job also does task purging:

• Completed tasks older than maintenance.completed_task_retention (default 24 h) are bulk-deleted.
• Failed tasks older than maintenance.failed_task_retention (default 7 d) are bulk-deleted.

This is a single bulk DELETE; the retention defaults keep failed rows longer for debugging.

Lives at modules/health/state.go as a module-level instance of the HealthState struct:

type HealthState struct {
    Nodes       map[string]*NodeHealth
    Deployments map[string]*DeploymentHealth
    LastPoll    time.Time
}

Writers: the Tick and Poll crons (node_metrics, container_metrics, deployment_sync). Readers: the HUD collector, shc status, and the detailed shc.status.get method.

Alongside HealthState, the same module holds:

• _cron_heartbeats: dict[str, datetime] — last successful completion per cron (read by the HUD for heartbeat display).
• Reconcile backoff state — cooldown dict, resolving set — also in-memory only.

Unlock — user POSTs to shc.vault.unlock with the password:

1. Request is routed to the leader (existing behavior).
2. Leader calls unlock_vault(password) locally — password held in memory only.
3. Leader broadcasts POST /internal/vault/unlock to all other nodes via mTLS internal API.
4. Each node unlocks locally.

Ongoing enforcement runs from the Tick cron’s vault_check step: leader broadcasts a vault-status check; any node reporting locked triggers a re-broadcast of the unlock if the leader still holds the password. On leader restart, the leader first tries the configured env var / password manager; if neither yields a password, nodes stay locked until a human unlocks again.

Security constraints:

• There is no endpoint to read the vault password — only write (unlock).
• The password is only transmitted over the mTLS internal API (port 4003).

Broadcasts run concurrently through goroutines and errgroup in RpcTransport.FanOut; 50 nodes = 5 calls/s, comfortably within mTLS connection capacity.

• CLI reference — shc status
• Communication patterns — RPC and task systems
• hud module — consumes HealthState via the internal API.
• vault module