Origin ingest gate + materialized transpile (2026-07-17)
Status: DESIGN. Part of the platform-compat program — faithful capture of
docs/handoff/2026-07-16-hel1-reprovision-and-platform-compat.md (§C3, §D,
§E), no new design. Siblings:
2026-07-17-manifest-adapters.md (the
per-origin transpilers this gate feeds),
2026-07-17-app-metadata-schema.md (the
canonical fields the transpile targets),
2026-07-17-notes-templating.md (the
post-deploy notes the adapt stage synthesizes),
2026-07-17-origin-repos-and-migrations.md
(the repo surface that runs this pipeline at sync).
Terminology (origin, adapter, adapted app) is the collision-checked
set defined in 2026-07-17-manifest-adapters.md.
Problem
Section titled “Problem”SHC wants to run apps packaged for other self-hosting origins (Umbrel, CasaOS, Runtipi, Coolify, Cloudron, …) — gated, opt-in per app; native SHC apps untouched. Two failure modes have to be designed out up front:
- Runtime shimming. Adapting a foreign manifest in-memory at deploy time produces packages that are unpinnable, undiffable, and unreviewable — and forces the enrichment core to know about origins forever. The transpile must instead be materialized at ingest: the output is a real, on-disk SHC package.
- Silent partial coverage. Not every foreign app can be adapted
(host networking, device passthrough, cross-app dependency shell).
A catalog where ~40% of apps silently break is worse than one that
honestly rejects them. Every reject must be recorded with its
reason and stay visible (
shc repo apps --rejected).
There is also a safety class the gate exists to catch: origin packages
assume a private LAN box. Umbrel’s app_proxy is a transparent
session-auth gateway and many Umbrel apps carry NO auth of their own
(the ecosystem tracks PROXY_AUTH_ADD). Stripping the proxy and
publishing on public DNS + LE would mint unauthenticated admin panels
with a green padlock. The adapt stage therefore flags forward-auth
default-on for every adapted app (see the adapt stage below).
Design
Section titled “Design”The gate: hard reject taxonomy
Section titled “The gate: hard reject taxonomy”Per app, at ingest, before any transpile output is produced:
| Reject | Why |
|---|---|
network_mode: host | breaks tenancy/network isolation; no overlay placement |
device passthrough (/dev) | host hardware coupling (Home Assistant/USB class) |
privileged | unreviewable third-party compose with host root |
torOnly | SHC exposes via public DNS + LE, not Tor hidden services |
dependencies / exports.sh (v1) | arbitrary vendor shell sourced into env resolution + cross-stack volume mounts (${APP_BITCOIN_DATA_DIR}) — the exact backup-boundary violation class the boundary campaign purged |
| >1 HTTP UI | a single host_spec cannot represent it; unique-exposer rule |
| unparseable manifest | nothing trustworthy to transpile |
Non-HTTP (L4) ports are either covered by an explicit L4 publish policy or are an ingest REJECT (unique-exposer + tenant port allocation) — the policy choice is open (see below).
Rejects are first-class output: each rejected app stays listed in the origin repo with its reject reason. Honest coverage, never a silent 40%.
Materialized transpile
Section titled “Materialized transpile”Per accepted app, the adapter transpile writes a complete SHC package
to disk: pinned, diffable, reviewable. Package identity = origin app
version + origin catalog git sha. Never adapt in-memory at deploy —
installing an adapted app is a plain shc install; the enrichment core,
tenancy, ingress, and backup machinery cannot tell it’s adapted.
The ingestion pipeline, stage by stage
Section titled “The ingestion pipeline, stage by stage”What shc repo add --origin umbrel <git-url> + every subsequent sync
does (the repo-registration surface itself is
2026-07-17-origin-repos-and-migrations.md):
- Pin — clone and record the upstream git sha. Nothing floats; every sync is a snapshot.
- Enumerate — discover app packages using the origin’s layout
(umbrel: dir-per-app with
umbrel-app.yml; casaos: compose withx-casaos; runtipi:config.json+ compose; cloudron: manifest + Dockerfile). - Gate — per app, the reject taxonomy above. Every reject recorded
WITH ITS REASON (
shc repo apps --rejected). - Adapt — per accepted app, the transpile:
- manifest → canonical
app.yaml(merit fields +compat:bag — 2026-07-17-app-metadata-schema.md); - strip the origin proxy and synthesize
host_specfrom the proxy service’sAPP_HOST/APP_PORTenv, NOT the manifestport(the manifest port is where the proxy listens; using it routes to nothing for much of the catalog); - bind-mounts → named volumes + seed-file init job (volume
seeding is a missing core primitive: origin packages ship seed
files copied into
${APP_DATA_DIR}pre-start, but named volumes populate from the IMAGE, not the package); - secret contract (
APP_SEED/SERVICE_PASSWORD_*→@secretmints;APP_SEEDis mint-once, IMMUTABLE, non-rotatable — apps persist seed-derived creds at first-init, rotation = lockout the platform can’t see, restore must replay the SAME seed); - auto-
backup.yamlfrom the volume set; notestemplate from default-creds /postInstallMessage(2026-07-17-notes-templating.md);- healthcheck synthesis where the origin declares one;
- forward-auth default-on flag, per-app opt-out requiring
ingest-gate evidence of self-auth (the Huly
?token=mangling class means the opt-out mechanism must exist day one).
- manifest → canonical
- Materialize — write the complete SHC package to disk (above).
- Index — the accepted set is served as a normal SHC repo.
- Re-sync — new upstream sha → re-run 2–6 and DIFF the normalized output; changed packages surface as app updates. This is the rot-prevention CI: without pinned snapshots + re-transpile diffing + conformance fixtures per origin, the adapters rot in a quarter (see the steady-state section of 2026-07-17-manifest-adapters.md).
Decided
Section titled “Decided”- Transpile is materialized at ingest — on-disk, pinned, diffable packages; no runtime adaptation layer (“adapter”, not “shim”).
- Rejects are recorded with reasons and stay visible
(
shc repo apps --rejected). - Forward-auth gate DEFAULT-ON for every adapted app; opt-out is
per-app and requires evidence of self-auth. (Reverses the earlier
draft where stripping
app_proxyhad no replacement gate — that default was dangerous as drafted.) dependencies/exports.share cut from v1 and rejected at ingest (not executed, not emulated).APP_SEEDis immutable: no standard vault-rotation UX on it.host_specsynthesis reads the proxy’s env, never the manifest port.- Compat volumes pin to LOCAL (never network-backed) volume providers.
- L4 ports: explicit publish policy vs plain ingest reject — unresolved (P2P apps: 8333/9735 are common on Umbrel).
exports.shapps: reject forever, or curated reimplementation of high-value dependency-carrying apps later?- Quota defaults for quota-less manifests (noisy-neighbor guard): the requirement is decided, the default values are not.
- Image pinning / provenance policy for unreviewed third-party compose: required by the steady-state analysis, not yet specified.
- Overlapping subpath mounts need a stated shared-parent-volume rule; UID/ownership first-boot failures are expected and need a documented remediation shape.