Job Registration System

Jobs in SHC come from multiple sources but have no unified registry:

• Standalone jobs (.job.yaml files) — visible via JobLoader.load_all(), show in shc job ls
• Socket/plug jobs (inline in *.socket.yaml, *.plug.yaml, *.vsocket.yaml) — invisible until they run
• Backup jobs (inline in backup.yaml) — invisible until they run
• Hook jobs (referenced by name in app.yaml hooks) — just references to .job.yaml names

When a socket discovery job or backup pre_backup job runs, it creates a job_run record in the DB, but there’s no job record linking it to a known definition. The HUD, CLI, and any monitoring system can’t enumerate all possible jobs for a stack without scanning multiple YAML files at query time.

Add a job table that acts as a registry/index of all known job definitions for a stack. This table is:

• Not backed up — it can be rebuilt from definition files
• Rebuilt on install/upgrade/restore — always reflects current definitions
• Single source of truth for shc job ls, the HUD jobs tree, and any future job scheduling UI

Unique constraint: (tenant, stack, environment, name)

This means the same job name can’t appear twice in a stack, regardless of source. If a socket job and a .job.yaml have the same name, that’s a conflict caught at registration time.

job_run.job_name + job_run.stack + job_run.tenant + job_run.environment is a logical FK to job. No physical FK constraint — job_run records survive if a job definition is removed (history preserved).

source: standalone
source_name: <filename without .job.yaml>
phase: null

Already parsed by JobLoader.load_all(). Each file produces one JobDefinition.

source: socket
source_name: <socket name, e.g. "postgres">
phase: discovery | integration | post_deployment | unintegration

All phases are nested under a jobs: key in the YAML:

• jobs.discovery → phase discovery
• jobs.integration → phase integration
• jobs.post_deployment → phase post_deployment
• jobs.unintegration → phase unintegration

Each job spec is a dict that can be parsed as JobDefinition. The name field may be missing — derive from context (e.g. <socket>-discovery-<index>).

source: plug
source_name: <plug name>
phase: discovery

Field: jobs.discovery

source: vsocket
source_name: <vsocket name>
phase: discovery | integration | unintegration

Fields: jobs.discovery, jobs.integration, jobs.unintegration

source: backup
source_name: backup
phase: pre_backup | post_backup | pre_restore | post_restore | restore

All phases are nested under a jobs: key:

• jobs.pre_backup → phase pre_backup
• jobs.post_backup → phase post_backup
• jobs.pre_restore → phase pre_restore
• jobs.post_restore → phase post_restore
• jobs.restore → phase restore (these are RestoreJob objects, slightly different schema)

After the stack is deployed and definition files are on disk:

1. Scan all sources (.job.yaml, *.socket.yaml, *.plug.yaml, *.vsocket.yaml, backup.yaml)
2. Parse each job spec into a JobDefinition (or equivalent for backup RestoreJobs)
3. INSERT all into the job table

This happens in StackService.install_stack() after _compose_command_async succeeds, alongside _sync_schedules.

1. Scan all sources from the NEW definition files
2. Compare against existing job table rows for this stack
3. Added jobs → INSERT
4. Removed jobs → DELETE
5. Changed jobs → UPDATE (definition, context, image, etc.)

This is a full reconciliation, not an append. If a .job.yaml is renamed, the old job row is deleted and a new one is inserted.

DELETE all job rows for this stack. job_run history is preserved (no FK cascade).

When connecting a plug to a socket, the socket’s integration/discovery jobs become relevant to the consumer stack. These should be registered on the provider stack (where the socket lives), since that’s where they execute.

Socket/plug jobs are registered when the provider stack is installed. Integration doesn’t create new job registrations — it just runs the already-registered jobs.

After restoring stack files, re-run the full registration scan (same as install). The job table is not in the backup, so it’s always rebuilt.

// modules/job/service.go (RegisterJobs)

// RegisterJobs scans all definition sources and registers jobs in the DB.
// Returns the number of jobs registered.
// Deletes stale jobs that no longer exist in definitions.
func RegisterJobs(
    ctx context.Context,
    db *sql.DB,
    tenant string,
    stack string,
    environment string, // default: "default"
) (int, error)

// UnregisterJobs removes all registered jobs for a stack.
// Returns count removed.
func UnregisterJobs(
    ctx context.Context,
    db *sql.DB,
    tenant string,
    stack string,
    environment string, // default: "default"
) (int, error)

Stays as-is for .job.yaml scanning. The registration function uses it internally. At execution time, jobs can be loaded from the job table instead of the filesystem.

Changes from JobLoader.load_all() to SELECT * FROM job WHERE stack=... AND tenant=.... This now returns ALL jobs, not just standalone ones.

Automatically shows all jobs (standalone + socket + backup + etc.) because it queries the job table. Add a --source filter flag to narrow by source.

Changes from JobLoader.load_all() + DB run lookup to a single query joining job with latest job_run.

For standalone jobs, works as before. For socket/backup jobs, the execution still uses the inline spec (stored in job.definition JSON column). The executor loads the definition from the job table instead of scanning YAML files.

No changes — queries job_run table as before.

No changes — queries job_run table as before.

NAME              SOURCE      PHASE           CONTEXT    SERVICE
backup_daily      standalone  -               service    postgres
migrate           standalone  -               container  -
check-pg-ready    socket      discovery       service    postgres
create-database   socket      integration     container  -
quiesce           backup      pre_backup      service    postgres
resume            backup      post_backup     service    postgres
import-dump       backup      post_restore    container  -

New flags:

• --source <standalone|socket|plug|vsocket|backup> — filter by source
• --phase <discovery|integration|...> — filter by phase

The jobs tree uses the job table. Each job shows source as a dim tag:

└ ▾ jobs  7
    ├ ▾ backup_daily       service   ✓
    │   ├ ✓ #42  2m ago   exit:0
    │   └ ✕ #40  2h ago   exit:1
    ├ ○ check-pg-ready     service   socket:discovery
    ├ ✓ create-database    container socket:integration
    ├ ○ quiesce            service   backup:pre_backup
    └ ✓ resume             service   backup:post_backup

Single Atlas migration (migrations/<timestamp>_job_table.sql) to create the job table. No data migration needed — the table starts empty and gets populated on next install/upgrade.

For existing installations: add a one-time register_jobs() call during server startup if the job table is empty but stacks exist. This backfills the registry for stacks that were installed before this feature.

• Job scheduling UI — cron scheduling stays in app.yaml and schedules table
• Job definition editing via API — definitions are file-based, not mutable via API
• Cross-stack job references — jobs belong to one stack