Skip to content
SHC Docs

Frappe image architecture: baked apps, dynamic apps, and both at once

Status: open design. Baked apps already ship. Installing an app into the image without a rebuild is still the proposal.

The owner’s objective, stated verbatim: the SHC Frappe image must work for vApps (“where you can install an app dynamically instead of them having to be baked into the images”) and support images with apps baked in, and support both, so a builder can bake some apps (Keycloak, because the OIDC plug needs it) while others are still added afterwards.

The open technical question was whether an app can be built once into a tarball and installed without a rebuild, or whether the target must clone source and build every asset and pip-install everything, or some mix.

Both halves are answerable, and they are independent of each other. That is the central finding: build cost and restart cost are separate problems with separate solutions, and conflating them is what made this look like one hard problem.

Evidence: /var/tmp/frappe-arch/0{1..5}-*.md. Every number below is MEASURED on a throwaway bench built from the pinned image, not inferred.


ObjectiveVerdictMechanism
Baked appsAlready shippedbakeApps@boolean in app.yaml, install-time only, guarded twice
Dynamic app install without rebuildYESPrebuilt artifact: no clone, no npm/esbuild, no compiler, no network
Dynamic install without downtimeYES, with one build-time lineA directory pre-seeded onto sys.path at build time
Both modes at onceYES, they are orthogonalThe hot-load directory does not care what was baked

1b. ★ THE DEPLOYMENT MODEL: RULED BY THE OWNER, 2026-08-05

Section titled “1b. ★ THE DEPLOYMENT MODEL: RULED BY THE OWNER, 2026-08-05”

Everything below this section is measurement. This section is the contract those measurements have to serve, and it was ruled after them.

The image is an INPUT, not something SHC manufactures.

SHC must work with whatever image it is handed, carrying whatever packages that image happens to have, and must know what that image can and cannot do. It does not impose a blessed image and it does not build one on the fly.

★ A build sidecar was proposed and REJECTED. (Run the fat image as a one-shot builder, produce the artifact, install it into the slim runtime.) It is technically sound (same base ⇒ ABI-compatible by construction) and it is still the wrong shape, because it manufactures a capability the operator did not ask for and hides which image is actually in play. Do not reintroduce it.

Where an app comes from: baked into the image · added dynamically as a vApp · both at once. The third is the case that must work; it reads as hard only while the axes are conflated.

What the image can do: carries build tooling (compilers, headers, node/yarn) · does not. A lighter image is a legitimate, first-class choice.

external app arrives asimage has build toolingresult
prebuilt artifactirrelevantinstalls (ABI must match, §5)
source (git ref)yesinstalls, compiles on the target, today’s path
source (git ref)noREFUSES
tardetect what is in it, then as aboveas above

Baked apps work in any image, because they are already present.

The owner’s rule, verbatim: “it work with any image that comes with whatever packages they come with as well as work with external packages. however if the external packages are not pre-compiled you have to use an image that comes with the build stuff.”

What “prebuilt” means, and the asymmetry inside it

Section titled “What “prebuilt” means, and the asymmetry inside it”

The two halves of an artifact are not alike, and the difference decides what may be pinned loosely and what may not.

The node/frontend half is compiled in, fully. It is frontend output; there is no reason to run a bundler on a production host. And it is platform-independent; the built assets are just files, and bundle hashes were measured deterministic across independent builds. This half travels anywhere. It is also where the cost is: bench get-app --skip-assets does not skip yarn install (38 s of 43.7 s), and it leaves 680 MB of node_modules + 93 MB of .git in every production bench.

The python half ships as wheels, and a python install step REMAINS, but it degrades from a build to a link:

Terminal window
pip install --no-index --find-links=<bundled-wheels> … # deps: offline, no compiler
pip install -e apps/<app> --no-deps # the app: writes a .pth

★ The asymmetry: the python half is ABI-locked where the node half is not. That is the whole reason §5’s pin is the base image digest and not a python version.

Neither half makes bench install-app cheaper: 24.8 s from an artifact vs 25.9 s from a clone. Database work is the floor.

  • A git ref is source by definition. No probe needed.
  • A tar must be SELF-DESCRIBING: a manifest inside declaring {app, version, commit, base_image_digest, abi_tag, has_assets}. “Is this prebuilt” is really “does this carry built assets and wheels for this runtime”, which cannot be answered by sniffing for public/dist and a wheels directory. It is also the same file the ABI check needs, so it costs nothing extra.
  • Image capability must be PROBED, not declared (a compiler on PATH, python headers, node/yarn for the asset build) and recorded on the deployment rather than re-probed per install. Honesty note: “has build stuff” is not strictly binary, since a source install can succeed without a compiler when every dependency ships a manylinux wheel. Require the toolchain anyway and offer an explicit override; the alternative is attempt-and-discover, whose cost is documented in §4.

★ Where the refusal happens: the load-bearing part

Section titled “★ Where the refusal happens: the load-bearing part”

At RESOLVE time, before anything mutates.

This is the entire lesson of the incident in §4: a failed vApp install left a production site 500ing on every route because the app reached apps.txt before anyone knew it could not work. When the source type plus the image probe are enough to know “this image cannot build this app”, SHC must refuse having changed nothing, which is strictly better than the transactional rollback that now exists to clean up after the attempt.

  • SHC does not need to ship a frappe image at all for the vApp story. The baked-frappe_keycloak image is a convenience for the OIDC plug, not a structural requirement.
  • The factory is not a dependency; see §7.7.

An app builds once in a builder and installs offline. Measured, --network none:

AppClone + build (current)Artifact (offline)
frappe_keycloak3.4 s, 896 KB on disk1.63 s, 356 KB (tarball 837 KB)
hrms v15.36.046 s, 793 MB on disk1.08 s, 19 MB (tarball 9.1 MB)

bench install-app hrms from the artifact: 24.8 s vs a 25.9 s clone-path control, parity, because that step is database work either way. /app returned 200 with all five prebuilt hashed bundles and no bundler on the machine. Bundle hashes are deterministic across independent builds.

Approach A (full pre-build: source + built assets + wheels) wins. Building assets at install time is the worst option: it keeps node, the network, and the upstream frappe-ui/vite failure mode on every target instead of once in a builder.

Two facts that make the builder pattern correct independent of everything else:

  • bench get-app --skip-assets does not skip yarn install. 38 of its 43.7 s were yarn, and it leaves 680 MB of node_modules plus 93 MB of .git in every production bench.
  • frappe_keycloak is a misleadingly easy case. Its public/ holds exactly one file: .gitkeep. No package.json. bench build emits an empty file table in 83 ms. Nothing about assets can be learned from it; every asset finding above is hrms-derived.

3. The restart question, settled, and the two findings that look contradictory

Section titled “3. The restart question, settled, and the two findings that look contradictory”

Two investigations reached opposite-sounding conclusions. They are both correct and they compose, because they tested different mechanisms.

A new .pth requires a restart. Both investigations agree, independently. uv pip install -e writes a plain-path .pth, and site.py processes .pth files once, at interpreter startup. Measured: after a successful offline install + install-app, every route returned 500 ModuleNotFoundError while a fresh interpreter imported the app fine. SIGHUP does not help; it re-forks workers from the same master, whose sys.path was fixed at boot (pids 12,13 → 235,236, still raising). SIGUSR2 does work but is unusable: gunicorn is PID 1, so retiring the old master kills the container.

(Correction for the record: the Dockerfile says pip install -e leaves an .egg-link. Modern pip/uv write a .pth. The conclusion is unchanged; both are startup-only, but the file it names does not exist.)

A directory already on sys.path at boot can receive new packages hot. This was never tested by the tarball investigation. One build-time line:

Terminal window
mkdir -p /home/frappe/frappe-bench/hotapps
printf '/home/frappe/frappe-bench/hotapps\n' > env/lib/python3.11/site-packages/zz-shc-hotapps.pth

Then at runtime, symlinking an app’s package into hotapps/ makes it importable to every already-running worker, with no restart and no signal. Proven end to end with the real payments app on a never-restarted fleet: get-app → symlink → install-app, 7.1 s total, zero downtime, 10 DocTypes created, /login 200.

So: the artifact removes the build; the pre-seeded directory removes the restart. Neither alone is sufficient; together they are.

Directory mtime has a 4 ms quantum (CONFIG_HZ=250), and CPython’s FileFinder rescans a path directory only when mtime changes. Same-instant hot-adds went stale 23 times in 25. importlib.invalidate_caches() or any gap > 4 ms → 15/15 clean. In a real flow install-app supplies 1–3 s of gap, but it is a race, not a guarantee: the airtight form calls invalidate_caches() per worker, or creates the symlink first and lets install-app provide the gap.


4. What the outage actually was: BOTH of the Dockerfile’s premises were wrong

Section titled “4. What the outage actually was: BOTH of the Dockerfile’s premises were wrong”

(Both have since been corrected in place. The conclusions survive as deliberate policy; only the stated mechanisms were false.)

Premise two, for completeness: VOLUME does not stop the write. The Dockerfile argued it cannot register the app because sites/ is a declared volume and Docker discards build-time writes to it. Re-measured on the same base: true only on the deprecated builder (DOCKER_BUILDKIT=0, where the append vanishes at zero exit), and false under buildx/BuildKit, which is exactly what the images job runs, where it persists into the layer, the image, and a fresh named volume. So the restraint is a policy this project holds and a test enforces, not a wall Docker puts up. apps/ is not a volume under either builder.

frappe/docker/Dockerfile states: “Frappe imports every app in apps.txt on every request, so the site 500’d on EVERY route.” Measured false. Frappe consults three lists with three filters:

ConsumerListSource
frappe.init() outside a request, CLI, bench, schedulerget_all_apps(with_internal_apps=True)sites/apps.txt
module map + _load_app_hooks()get_installed_apps(_ensure_on_bench=True)DB ∩ apps.txt
website template resolutionget_installed_apps() rawDB only

An app in apps.txt that is not importable leaves HTTP entirely green (/api/method/ping 200, /login 200, /app 301) while bricking the whole bench CLI, including bench clear-cache, so you cannot clear your way out.

The outage came from the database list via the website renderer, not apps.txt. This inverts which step is dangerous:

bench install-app is the risky step. The apps.txt line is comparatively benign.

It also explains the recovery: deleting the apps.txt line restores the CLI instantly, but /login keeps 500ing, because the renderer walks the raw DB list: only restoring the code or uninstall-app fixes it.

Therefore the safe order is: code first, registration second. Never the reverse.

The Dockerfile’s conclusion (bake frappe_keycloak) remains right. Its stated reason does not survive measurement and should be corrected in place.


5. ★ The constraint that outranks all of this: ABI, and the real pin

Section titled “5. ★ The constraint that outranks all of this: ABI, and the real pin”

An artifact cannot cross the 3.11 → 3.14 hop. The bench venv holds 70 native .so files, 62 ABI-locked to cpython-311. On real python:3.14-slim, cp311-cp311 wheels are refused at the filename. abi3 is a narrower escape than it looks: cryptography’s abi3 wheel installs on 3.14 and is still unusable because its dependency cffi is cp311-only.

frappe_keycloak looks immune and is not. All 15 of its wheels are pure-python, but frappe_keycloak → python-keycloak → jwcrypto → cryptography → cffi hits the wall. It is clean only because it delegates its native dependency to the base image.

The real pin for an artifact is the BASE IMAGE DIGEST, not the Python version. An abi_tag guard is necessary but insufficient.

Compounding it: SHC pins v15.118.2, which predates frappe_docker’s April-2026 asset redesign. SHC is on the old asset contract and inherits the new one only on the v16 move, so the interpreter migration and an asset-layout migration arrive together.


6. The delivery hazard that is live right now

Section titled “6. The delivery hazard that is live right now”

The image is built and pushed, and nothing pins it. docker/docker-bake.hcl builds …/shc/frappe on every main push (five tags published), while apps/frappe/vars.yaml still names upstream frappe/erpnext:v15.118.2. The baked frappe_keycloak the OIDC plug depends on exists on no estate.

And switching the pin may deliver nothing. compose.yaml seeds the bench volume gated on [ "$BAKE_APPS" != "yes" ] && [ ! -f /bench-init/.seeded ]. In the default mode apps/ is a volume seeded once, never refreshed, so an image swap is a silent no-op for the app tree. Only bakeApps=yes puts apps/ in the image layer, and bakeApps is install-time only.

Independent corroboration: frappe_docker rejected a shared apps volume for exactly this reason (issue #790; a named volume is seeded from the image only on first creation, freezing app versions). Two teams reached this from opposite directions.

Also: bench restart is a silent no-op in containers. restart_supervisor_processes returns immediately when supervisorctl is off PATH, with no warning and no error. Nothing in-container can pick up a new app; only an orchestrator-level restart, or the hot-load path in §3.


  1. Do we adopt the hot-load directory? It buys zero-downtime dynamic installs for one build-time line. Cost: a sys.path entry that is writable at runtime, and a 4 ms race needing invalidate_caches(). Restart is cheap (687 ms to first 200), so this buys availability, not speed. Recommendation: yes; availability during app install is exactly the vApp promise.

  2. Artifact format: ★ PARTLY RULED 2026-08-05, see §1b. The artifact is Approach A and carries a self-describing manifest {app, version, commit, base_image_digest, abi_tag, has_assets}; resolution refuses on base-image mismatch rather than installing something that imports in the venv and 500s on request. Still open: where it is built and published. Note this is now a smaller question than it was; under §1b the image is an input, so SHC is not obliged to build artifacts for apps it does not ship. An operator pointing at a git ref with a build-capable image needs no artifact at all.

  3. Retention. ANSWERED 2026-08-05: safe today, with a named landmine. The G19 rot is in the generic package registry. The image lives in the container registry, whose container_expiration_policy.enabled = false; empirically nothing has been pruned (forward-auth retains 534 commit tags, keycloak 270, traefik 269), and the three pins ce/builtin already carries still resolve. The landmine: the disabled policy reads keep_n: 10, older_than: 90d, name_regex: .* with no name_regex_keep; enabling it as written would delete this pin and the three ce ones together. Do not enable it without an exclusion.

  4. How does a plug declare a host-side prerequisite? The OIDC plug needs frappe_keycloak present. Today that is solved by baking, and there is no way to state it; the consumer’s only phase is post_deployment, which runs after the consumer is live, i.e. after the failure window. This is G1’s cousin.

  5. Does an image record what it carries? The vsocket’s refusal is mode-shaped (bakeApps=yes → refuse everything) rather than set-shaped, because the fact does not exist in readable form. “Bake some, add others” needs it to.

  6. Existing deployments: STILL OPEN, and now measured. The pin landed (apps/frappe/vars.yaml → …/shc/frappe:76624a36@sha256:114137d9…, verified by pulling and probing the image: apps/ carries frappe, erpnext and frappe_keycloak, and ./env/bin/python -c "import frappe_keycloak" resolves). The delivery matrix is now confirmed rather than suspected:

    casedoes frappe_keycloak arrive?
    new install, bakeApps=true✔ apps/ is the image every boot
    new install, bakeApps=false✔ volume created empty, seeded from this image
    existing deployment, bakeApps=false, i.e. both live benches✘ silent no-op

    The seed is keyed on [ ! -f /bench-init/.seeded ], never on image identity. The pin switch must not be read as a delivered fix for bit/ffx. One thing improved though: the bump does deliver everything outside the bench mount, including the recorded manifest, so the drift job now reports stale and names the once-only seed as the cause, making the no-op visible instead of silent.

    Do we accept new-installs-only, add an image-keyed re-seed, or require a bakeApps=true reinstall? This is a consented estate decision, not an engineering one.

  7. Does SHC’s frappe image come from the factory? ANSWERED 2026-08-05: no, and under §1b it does not need to. Measured rather than assumed:

    • SHC’s image is FROM frappe/erpnext:v15.118.2@sha256:2372e19…, straight upstream. The factory (~/frappe/factory, a different repo tree entirely) is not in SHC’s image lineage today.
    • The factory does already implement the fat/slim split, and implements it well: base = python:${PYTHON_VERSION}-slim-${DEBIAN} + runtime deps → builder = base + build deps (build-essential, gcc, libffi/libmariadb/libpq/ libsasl2/libldap2/libtiff5-dev …) → frappe = FROM builder AS b; FROM base; COPY --from=b /home/frappe/frappe-bench. It compiles in the fat image and ships on the slim one.
    • ★ It also carries ARG PYTHON_VERSION=3.11.6, a parameterised interpreter, which is the only lever anyone has for the v16 wall (§5: frappe v16 requires ≥3.14; upstream ships 3.11.6).

    Under §1b this is a convenience, not a dependency. The factory is simply one source of images where builder has the toolchain and base does not. Adopting it as SHC’s base remains a live option with a real trade-off: it buys control of the interpreter (and therefore a v16 path that does not wait on upstream), and it costs divergence from the upstream image the community survey identified as the supported path. Not required by anything in this document.


8. Defects found along the way (tracked separately, not fixed here)

Section titled “8. Defects found along the way (tracked separately, not fixed here)”
  • ★ sites/apps.json, where bench get-app records commit_hash/branch/version, is never maintained by SHC’s vsocket. Combined with the builder’s rm -rf apps/$_NAME/.git, that is the whole of G17 (UNVERSIONED).
  • The builder soft-fails clones (exit 0): an image can ship green missing a declared app.
  • bench get-app is not atomic (leaves a .pth + apps.txt entry when the build step fails) and scrambles apps.txt order.
  • Neither the baked apps.txt nor bench’s rewrite ends in a newline, so echo x >> silently corrupts the last entry (measured: a literal probeappghostapp).
  • ls -1 apps > sites/apps.txt writes alphabetical order (erpnext before frappe). Harmless today, get_all_apps() forces frappe first, but it clobbers deliberate ordering.
  • If an app is absent from apps.txt before bench build, esbuild dies as path.resolve(undefined) with exit 143, which reads like SIGTERM/OOM and is not.
  • The bench-init seeding loop is not idempotent.

The prebuilt path was proven for frappe_keycloak (trivial, no assets) and hrms (real, asset-heavy). It was not proven for an app with native extensions of its own; that case still needs the base-image pin of §5 to be enforced, not just recorded. And one install-app hrms failed once with a DocType-sync error that did not reproduce across two subsequent clean runs with the same artifact; the clone-path control also succeeded. Recorded as unattributed rather than explained away.