Bundled services vs. sockets: a catalog-wide audit

STATUS: ANALYSIS ONLY. No recipe was changed by this audit. The one app that materially violated the rule (huly) was already being converted in parallel and is fixed at the time of writing. Everything else in the catalog either complies or has a documented upstream constraint. There are no data-loss bugs in the bundled-service surface.

The owner’s formulation:

if it doesn’t need backed up it can go in the compose, else it should be an integration (with the exception of course of volumes that need persistent)

Refined to the sharper operational test:

is the bundled thing a stateful service SHC already operates as its own app? Bundling such a service means a second copy to patch, a second backup path, and a second thing that can silently diverge.

Both readings agree on the interesting cases, and where they disagree (a bundled cache that is a real SHC app but holds nothing worth keeping) the catalog has already settled on an explicit house rule, quoted below.

Programmatic, not by eye. compose.yaml is Jinja-templated, so the scan neutralises {{ ... }} into inspectable placeholders and strips {% ... %} control tags while keeping every branch body — an over-approximating union view, which is the correct bias when the question is “could this app ever run its own database”. Service names are separately re-scanned line-wise to recover which {% if %} guards each one, so a conditional fallback bundle is distinguishable from an unconditional one.

Per app the scan collects: consumed sockets (*.plug.yaml), provided sockets (*.socket.yaml), every compose service image (resolved through vars.yaml images.*), each service’s mounts, and backup.yaml’s exclude.mounts, converters and jobs.restore. Images are matched against a table of stateful-service families and mapped to the socket that would replace them.

Coverage: 78 app directories, all parsed clean (two needed YAML merge-key support; zero parse failures remain).

Anyone re-running an image-name scan will hit these:

1. *-pgclient / *-mongo-client sidecars (23 services). They run the postgres/mongo image but the command is sleep infinity and the environment points at integrations.<x>.socket.*. These are the correct pattern — a client container that gives the dump/restore jobs a shell against the integrated database. They are not bundles.
2. One-shot migrators/initialisers (4 services), e.g. hyperswitch-migration, windmill-db-init.

The scan classifies each image-matched service as server / sidecar / oneshot and only server counts. Of 70 image-matched services, 43 are real servers.

Symmetrically, plain substring matching under-reports: vendor-prefixed database images are easy to miss. Ones that had to be added by hand after sweeping every unclassified image in the catalog: registry.community.greenbone.net/community/pg-gvm (a real PostgreSQL), percona/percona-distribution-postgresql, wazuh/wazuh-indexer (OpenSearch).

43 distinct socket names are published across the catalog. The ones that matter here — a service an app could integrate with instead of bundling:

Socket	Provided by	Consumers
postgres	postgres, external-postgres	31
smtp	mail, mailcow, external-smtp	38
sso	keycloak, external-oidc	32
s3	minio, s3	10
mysql	mysql, mariadb, external-mysql	7
mongodb	mongodb, external-mongodb	1
redis	redis, external-redis	0
clickhouse	clickhouse	1
imap	mailcow, external-imap	0

Plus app-specific sockets (matrix, wazuh, ethereum, pmm, keygen, immich, …) and six vsockets (database, bucket, cache, plugin, site, realm).

redis has a socket and zero consumers. That is the single most striking number in this audit, and section “The valkey pattern” explains why it is defensible.

• 78 apps; 51 consume at least one socket; 129 plug files; 64 have a backup.yaml.
• 43 real bundled stateful servers. 7 of those are the provider apps themselves (postgres, mysql, mariadb, mongodb, redis, minio, clickhouse) legitimately being the thing they provide.
• 36 bundled servers across 27 non-provider apps. 26 duplicate an available socket; 10 have no socket equivalent.
• Of the 26: 22 are redis/valkey, 2 postgres, 1 mariadb, 1 postfix.
• Data-loss bugs found: 0.

Every real bundled stateful server in a non-provider app. “Backup” is the status of the service’s state volumes (socket/config/cert/log mounts filtered out) against the app’s current backup.yaml.

App	Service	Kind	Socket it duplicates	State volumes	Backup	Class
affine	valkey	valkey	redis	valkey-data	excluded	b
blockscout	blockscout-redis	redis	redis	redis-data	excluded	b
ciso-assistant	qdrant	vector db	—	qdrant-data	captured	b
defectdojo	valkey	valkey	redis	valkey-data	excluded	b
fleet	valkey	valkey	redis	valkey-data	excluded	b
frappe	valkey	valkey	redis	valkey-data	excluded	b
gitlab	valkey	valkey	redis	valkey-data	excluded	b
graphiti	neo4j	neo4j	—	neo4j-data, neo4j-import, neo4j-plugins	captured	b
greenbone	pg-gvm	postgres	postgres	pg-gvm-data, greenbone-shc-backup	captured	b (documented)
greenbone	redis-server	redis	redis	none (socket only)	n/a	b
highlight	kafka	kafka	—	kafka-data	excluded	b
highlight	zookeeper	zookeeper	—	zookeeper-data	excluded	b
highlight	redis	redis	redis	redis-data	excluded	b
huly	elastic	elasticsearch	—	elastic	captured	b (the model)
huly	redpanda	kafka	—	redpanda	captured	b (the model)
hyperswitch	valkey	valkey	redis	valkey-data	excluded	b
immich	valkey	valkey	redis	valkey-data	excluded	b
keygen	valkey	valkey	redis	valkey-data	excluded	b
kutt	valkey	valkey	redis	none	n/a	b
librechat	valkey	valkey	redis	valkey-data	excluded	b
mailcow	mysql-mailcow	mariadb	mysql	mysql-data	captured	b (appliance)
mailcow	redis-mailcow	redis	redis	redis-data	captured	b (appliance)
mailcow	postfix-mailcow	smtp relay	smtp	postfix-data, crypt-data, rspamd-data	captured	b (is the provider)
mailcow	memcached-mailcow	memcached	—	none	n/a	b
nextcloud	valkey	valkey	redis	valkey-data	excluded	b
paperless	valkey	valkey	redis	none	n/a	b
peertube	valkey	valkey	redis	valkey-data	excluded	b
penpot	valkey	valkey	redis	valkey-data	excluded	b
plane	plane-rabbitmq	rabbitmq	—	plane-rabbitmq-data	excluded	b
plane	valkey	valkey	redis	valkey-data	excluded	b
postiz	temporal	temporal server	—	none	n/a	b
postiz	valkey	valkey	redis	valkey-data	excluded	b
pulp	pulp-redis	valkey	redis	pulp-redis-data	excluded	b
supabase	postgres	postgres	postgres	pgdata	captured	b (is the product)
test-multi	redis	redis	redis	none	n/a	fixture
wazuh	wazuh-indexer	OpenSearch	—	wazuh-indexer-data	captured	b

Class (a) — should convert — is empty.

huly was the one true case: unconditional CockroachDB (all platform data) plus MinIO (all blobs), with only smtp and sso plugs. At the time of this audit the sibling conversion has landed apps/huly/postgres.plug.yaml and apps/huly/s3.plug.yaml, and the scan no longer reports cockroach or minio under huly. Nothing else in the catalog matched the pattern.

Four distinct justifications, all verified against the recipe rather than assumed:

1. Derived / transient — the huly model. elastic (a fulltext index the fulltext service rebuilds from the primary store) and redpanda (a transient event queue) have no other possible consumer and regenerate themselves. Same shape: highlight’s kafka + zookeeper, plane’s plane-rabbitmq, postiz’s temporal, mailcow’s memcached, and every valkey.

2. Upstream cannot use an external instance. This is the make-or-break question and three apps fail it for concrete, checkable reasons:

• greenbone / pg-gvm. gvmd requires the pg-gvm PostgreSQL extension — a shared library plus custom SQL types — shipped only in Greenbone’s own image; a stock PostgreSQL cannot load it. Worse for conversion, start-gvmd runs psql -U gvmd -d gvmd with no host flag, i.e. over the local unix socket, and does not honour libpq PG* env vars for TCP. This is already documented in apps/greenbone/app.yaml. Not convertible without patching upstream’s start scripts.
• supabase / postgres. The image is supabase/postgres, carrying Supabase’s custom shared_preload_libraries and extension set; the restore job in apps/supabase/backup.yaml initialises the cluster with that image specifically for this reason. The database is the product.
• mailcow / mysql-mailcow + redis-mailcow. mailcow is a fixed appliance whose components resolve each other by bare compose service name; upstream does not support an external MySQL. Its postfix-mailcow is not a duplicate of the smtp socket — mailcow provides that socket.

3. No socket exists to convert to. graphiti’s neo4j, wazuh’s OpenSearch indexer, ciso-assistant’s qdrant, and the kafka/zookeeper/ rabbitmq/temporal set. These hold real state (graphiti’s neo4j and wazuh’s indexer are primary stores, and both are correctly captured), but there is nothing to integrate with. They become interesting only if SHC ever ships a neo4j, elasticsearch or amqp socket.

4. Provider apps being their own service — the 7 excluded from the table.

Every bundle resolved to (b) with a concrete reason. The nearest thing to an open question is whether graphiti/ciso-assistant/wazuh justify new sockets; that is a roadmap question, not a rule violation.

22 apps bundle their own redis/valkey while a redis socket sits unused. Under the literal rule this is fine — a cache does not need backing up — and the catalog has committed to that in writing. From apps/blockscout/backup.yaml:

House rule: redis/valkey/kafka/rabbitmq volumes are ephemeral.

The scan confirms the rule is applied consistently: of the 22, 17 exclude the redis volume from backup, 4 mount no volume at all, and exactly one captures it — mailcow, which is the deliberate exception (mailcow’s redis holds real state and has a matching mailcow-redis converter and a paired mailcow-restore-redis job, unlike every other app’s cache).

Two small inconsistencies, both cosmetic, neither a bug:

• Most bundled valkeys run with redis/valkey default RDB persistence on and then exclude the volume. Only keygen and paperless pass --save '' --appendonly no. The others write periodic snapshots nobody will ever read. Harmless; tidying it would mean either dropping the volume (as kutt/paperless/test-multi already do) or disabling persistence explicitly.
• highlight’s redis runs redis-server --save 60 1, i.e. persistence deliberately enabled, with redis-data excluded from backup. The backup.yaml prose explains the exclusion (restoring stale queue state breaks kafka/zookeeper cluster-id pairing) so the intent is clear, but the --save flag reads as contradicting it.

Whether 22 bundled caches is the right answer is a cost-of-ownership call, not a data-safety one: it is 22 images to bump and 22 things that can drift, against the operational simplicity of not making every app depend on a shared redis stack. Worth an explicit decision either way, since the redis socket currently has no consumers at all and is effectively dead surface.

The dangerous shape is a converter that claims a mount — which removes it from the raw volume pass — without a paired jobs.restore entry to replay it, producing a restore that reports success and comes up empty. (The trap is documented at length in apps/huly/backup.yaml.)

A mount is claimed when a converter’s name matches a compose volume name. Checking that collision rather than merely “has converters” across all 15 apps that declare any:

Result: zero unpaired claiming converters. Every claiming converter has restore jobs — ciso-assistant (2/2), crowdsec, kubo (3/3), mail, mariadb, matrix, mission-control, mongodb, mysql, openobserve, postgres, supabase, traefik (2/2).

Two near-misses that a naive check flags and that are both fine:

• pmm declares a converter pmm-postgres with no restore job. Safe: the converter’s name does not match any compose volume (pmm-data), so it claims nothing, and it mounts the snapshot root (.:/snapshot:ro) read-only. apps/pmm/backup.yaml states this explicitly — the SQL dump is a “consistency-normalized SUPPLEMENT, not a replacement”, and the raw /srv bytes remain the primary capture. A paired restore job is called out there as a follow-up needing live PMM validation.
• mailcow has 7 converters, none matching a volume name (they are mailcow-* prefixed against volumes like vmail-data), and 7 restore jobs. Correct by construction.

Bundled stateful services whose state volume is excluded from backup are all in the ephemeral class above; none is a primary store. Nothing to flag.

Ordered by (data at risk x cleanness of conversion). The list is short because the catalog is in good shape.

1. Nothing to convert. Once huly lands there is no app whose primary state sits in a bundled copy of a service SHC operates. No action.

2. Decide the redis question explicitly (no data at risk; medium ownership cost). Either bless the house rule — 22 bundled caches, and consider retiring or repositioning the redis socket, which has zero consumers — or pick a few high-service-count apps to prove the conversion. If converting, the specific plug is the same everywhere; for e.g. gitlab it would be apps/gitlab/redis.plug.yaml against the redis socket schema in apps/redis/redis.socket.yaml, with the compose valkey service and its valkey-data exclusion deleted and the redis URL env repointed at integrations.redis.socket.*. Recommend bless the rule: the caches hold nothing, and a shared redis turns a per-app restart into a fleet-wide blast radius.

3. Tidy the persistence/exclusion mismatch (cosmetic, trivially safe). Give the 14 bundled valkeys that mount valkey-data the same --save '' --appendonly no that keygen and paperless already use, or drop the volume. Purely hygiene — it removes pointless disk writes and makes the “this is ephemeral” intent legible in the compose rather than only in backup.yaml. Each touched app’s suite must pass.

4. Consider a pmm restore job (low; already documented as a follow-up in apps/pmm/backup.yaml, needs live PMM validation). Not a bug today — the raw volume restores — but the dump is currently write-only.

5. Roadmap-only: new sockets. If neo4j, elasticsearch/OpenSearch or an AMQP socket ever ship, revisit graphiti, wazuh and plane. Not worth creating a socket for a single consumer today.

Requested as a note, not a recommendation.

• valkey/redis — 22 apps. Covered above; the dominant duplication in the catalog by a wide margin.
• busybox / alpine init containers — ~12 apps (fleet-db-conf, mailcow-config-init, hockeypuck-config, traefik-init, netbird-management-config, mission-control-init, backend-init, mysql-socket-init, …). Stateless one-shots; duplication here is fine and arguably preferable to a shared abstraction.
• otel/opentelemetry-collector-contrib — 2 apps (otel-collector standalone at 0.96.0 and highlight-collector at 0.128.0, a 32-minor version skew). Stateless, so no backup concern, but this is exactly the “second copy to patch” cost the rule warns about, in miniature.
• SMTP relays. Only mailcow bundles one, and it provides the smtp socket, so there is no duplication to speak of — 38 apps consume smtp, the healthiest integration number in the catalog.

SUPERSEDED — this section was wrong about uptime-kuma and was acted on. It is kept, corrected, because the correction is the interesting part: the original claim came from reputation, not from the pinned image. The follow-up pass verified every candidate against the shipped image and converted the two that hold up. See the table below.

Out of scope for the rule as written — SQLite is a library in a persistent volume, not “a stateful service SHC operates as its own app”, and the rule explicitly excepts “volumes that need persistent”. But an integrated database gets SHC’s backup, quiesce and consistency-group machinery, while an embedded SQLite file is just bytes in a volume, so the question is worth settling per app.

Every app whose primary datastore is embedded, checked against the image pinned in its own vars.yaml:

App	Primary store	Upstream Postgres at the pinned version	Verdict
ciso-assistant	Django SQLite, /code/db	Yes — settings.py switches to django.db.backends.postgresql on POSTGRES_NAME; psycopg[binary] 3.3.4 is in the image venv	CONVERTED, optional (backend: sqlite default)
openobserve	/data/db/metadata.sqlite	Yes — ZO_META_STORE=postgres + ZO_META_POSTGRES_DSN; the only mode constraint the binary enforces is the reverse (“cluster mode is not supported for ZO_META_STORE=sqlite”)	CONVERTED, optional (metaStore: sqlite default), metadata only
uptime-kuma	/app/data/kuma.db	NO. Database.connect in louislam/uptime-kuma:2 (v2.4.0) accepts sqlite, mariadb, embedded-mariadb and nothing else. The pg dependency in package.json powers the postgres MONITOR TYPE (server/monitor-types/postgres.js), i.e. probing someone else’s database as a health check — it is not a backend	Leave
crowdsec	/var/lib/crowdsec/data/crowdsec.db	Upstream yes (ent + pgx are in the v1.7.6 binary), image no — docker_start.sh owns /etc/crowdsec/config.yaml and rewrites it in place with conf_set, and exposes no DB_* env. Wiring it means SHC forking upstream’s whole config file	Leave
mission-control	better-sqlite3, /app/.data	No — better-sqlite3 is the only driver in the 2.0.1 image; no ORM, no pg	Leave
jellyfin	library.db / jellyfin.db	No — SQLite-only at 10.10.7	Leave
mail (mailpit)	/data/mailpit.db	No — mailpit’s --database takes a file or an rqlite DSN	Leave
openhands	plain files under /.openhands	N/A — the 0.39 OSS image has no sqlalchemy, no psycopg, no SQLite at all; state is a FileStore (local/s3/gcs). Postgres exists only in the closed SaaS build	Leave (an s3 plug is the interesting one here, not postgres)
kubo	IPFS flatfs/badger datastore	No — not a relational store	Leave
graphiti	bundled Neo4j	No — graph store, no socket exists	Leave
greenbone	bundled pg-gvm	Already documented above: needs the pg-gvm extension and talks over a local unix socket	Leave
wazuh	OpenSearch indexer + local files	No — no socket exists	Leave
supabase, mailcow	their own postgres / mariadb	The database is the appliance	Leave

gitlab-runner was on the original list by mistake — it holds no primary state at all.

Both conversions are optional with a working local fallback, which is the catalog convention for an app that genuinely has one (gitea and vaultwarden are the precedents). A plain shc install <app> still comes up with zero dependencies; attaching -i postgres=<stack> selects postgres on its own. That also settles the existing-operator question without a migration: an install already on SQLite keeps its default and is untouched.

The scan is three small Python passes (parse -> role-classify -> volume/backup cross-check). It was run from a scratch directory and intentionally not committed; the method is described in full above so it can be rebuilt, and the classifier’s two blind spots (client sidecars over-reporting, vendor-prefixed images under-reporting) are the parts worth keeping.