Skip to content
SHC Docs

Hookshot's first deployment: what a live estate settled

Status: open design. A second appservice registration, hand-run job context, and the transform render arms are still owed.

Companion to 2026-08-09-matrix-notification-bus.md (the design) and 2026-08-10-hookshot-phase0-readiness.md (the ruling map). Those two documents end with the same sentence in different words: the package has never been deployed. It has now. This document records what the deployment measured, what it changed in the package, and what each producer’s edge proved on the far side, bitspur corp/prod, 2026-08-14/15.

The ruling this lane executes is the notification-bus row of docs/decisions/2026-08-14-capability-and-backlog-rulings.md: “Hookshot for everything: one delivery path: wazuh, gitlab and GlitchTip all route through the hookshot bridge”, which overrode §9.1’s phase-0 recommendation (native Matrix for gitlab, a custom- script straight to the client-server API for wazuh). Every producer below therefore rides the bridge, including the two the proposal argued should not.


1. ★ The finding that reshaped the package: url was already taken

Section titled “1. ★ The finding that reshaped the package: url was already taken”

matrixAppservice is ONE registration shared by every consumer, and the registration’s url (the address Synapse pushes transactions to) is a single string authored on the provider. Measured on bitspur before anything was installed:

$ docker exec corp_prod_matrix-synapse-1 cat /data/appservices/shc.yaml
{ "id": "shc", "url": "http://matrix:8090", "sender_localpart": "_shc_appservice", … }
$ docker inspect … | grep 10.0.44.126
/corp_prod_huly-matrix-1 aliases=[… matrix prod.matrix.huly]

http://matrix:8090 is huly’s two-way Matrix bridge, deployed and working (43h uptime, its log shows inbound-event handling). The readiness doc’s §2.4 called two consumers on one registration a standing risk; on a live estate it is not a risk but an allocation; the pointer is spent, and hookshot cannot have it without taking it from a running product surface.

So hookshot is deployed SEND-ONLY, and that is sufficient, because of what the bridge actually needs an inbound transaction for. Measured in the pinned image (ghcr.io/matrix-org/matrix-hookshot:7.4.4):

where a static connection is builttriggerneeds a transaction?
BotUsersManager.getJoinedRooms() → Bridge.js:459 → createConnectionsForRoomId(roomId, false)startup, over the rooms the bot is already inno, a client-server GET /joined_rooms
Bridge.js:793 onRoomJoin → createConnectionsForRoomId(roomId, true)a pushed membership eventyes

Posting is likewise a client-server call with the appservice token. The only thing the transaction half buys a generic-webhook bus is the auto-accept of an invite, and an appservice can simply join itself.

The invite + join moved into the boot script, before exec, and hookshot-init.job.yaml was deleted rather than left beside it (M12: the replaced surface goes in the same program). The ordering is the whole point:

  • a post_install hook job necessarily runs after the bridge has finished enumerating its rooms, so an invite issued there converges one restart too late, measured exactly that way on the first install, where the job’s invite landed at 23:37:39 and the bridge kept reporting Found 0 connections;
  • the boot script invites as the alerts bot (matrix socket), joins as the appservice with its own token, and reads back joined_members before starting the daemon.

First boot after the change:

hookshot boot: rendered config with 3 static connection(s)
hookshot boot: hook glitchtip -> http://prod.hookshot.hookshot:9000/webhook/844abdf5... (rest masked …)
hookshot boot: invited @_shc_appservice:matrix.corp.bitspur.com to !IqVzZGjkiYuEdXYyiA:…
hookshot boot: read-back confirms @_shc_appservice:matrix.corp.bitspur.com JOINED !IqVzZGjkiYuEdXYyiA:…
INFO [Bridge] Bridge is now ready. Found 3 connections

and the acceptance ladder’s step 4 (§9.2) answered on the first try:

POST /webhook/844abdf5... -> 202 {"ok":true}
… @_shc_appservice:matrix.corp.bitspur.com m.room.message "shc lane probe: hookshot generic webhook, glitchtip hook"

★ This also closes readiness §4 unknown 7: a URL derived by the recipe has never been POSTed to a bridge whose config rendered the same recipe. It has now, and the glitchtip edge derived the identical id in Python (844abdf5-a19f-4123-b425-d990f221b9e9 on both sides).


2. The other measurements the first deployment produced

Section titled “2. The other measurements the first deployment produced”
#measuredconsequence
D1--var hooks=[{"name":…}] cannot set a list-typed var: --var values arrive as strings, and jinja iterated the JSON text character by character, rendering hooks named '' and crash-looping the containervars.hooks is now a STRING spec, name=room;name=room, which is the wire format compose always flattened to anyway
D2a MAS-fronted Synapse refuses an appservice registration without inhibit_login: 400 IO.ELEMENT.MSC4190.M_APPSERVICE_LOGIN_UNSUPPORTEDthe boot script’s register call sets inhibit_login: true; the bridge wants no device and no access token, it has the as_token
D3the appservice bot joining the alerts room makes that room appservice-interesting, so huly’s bridge now also receives its transactionsobserved: no errors, no behaviour change in corp_prod_huly-matrix-1 over the following half hour. Recorded because it is the shared registration showing through
D4shc job run <hook job> renders the job with neither the deployment’s vars nor its integration context; the same job that had MATRIX_TOKEN, ROOM_ID and HOOKS_SPEC during install came back empty-handed when run by handa platform defect worth its own lane: a hand-run job that silently sees defaults is a debugging trap. It is not a blocker here; the package no longer has a hook job
D5wazuh-control start inherits the job’s stdout through the daemons it forks, so a job that lets it write to the pipe never sees EOF and burns its timeoutthe wazuh edge redirects it to a file; start (not restart) leaves the running SIEM alone and brings up only integratord

3. Per-producer verdicts, each with its far-side proof

Section titled “3. Per-producer verdicts, each with its far-side proof”

All three ride the bridge, per the ruling. Each edge is a webhook.plug.yaml with a post_deployment job that converges the producer’s OWN notification surface and an unintegration job that removes it.

apps/flagship/glitchtip/webhook.plug.yaml. The job writes an AlertRecipient of type webhook through the ORM (manage.py shell), because the package seeds no API token and the REST surface has no standalone recipient endpoint (readiness M3). One dedicated ProjectAlert per project, named shc-matrix, so the edge never bolts a row onto an alert a human made.

SSRF posture: ruled in favour of the in-fabric URL. GlitchTip refuses a private-IP recipient twice (pydantic validator at creation, check_url_safe at send) unless GLITCHTIP_ALLOW_PRIVATE_IPS is set; default False. The alternative was a public vars.urlBase on the hookshot stack, i.e. a bearer secret on the internet. The package gained allowPrivateWebhookIps (default "false"), the job refuses loudly when the URL is private and the flag is off, and the estate is bound with it true.

Far side, from a real ingested error event through GlitchTip’s own alert path:

2026-08-15T00:03:12Z @_shc_appservice:… m.room.message "GlitchTip Alert"

(The issue detail rides attachments, which hookshot carries in uk.half-shot.hookshot.webhook_data rather than the visible body, the measured text-only render, with JS transformation functions off by policy.)

apps/flagship/wazuh/webhook.plug.yaml, and it is the one the proposal warned is “larger than it looks”. The job writes /var/ossec/integrations/custom-shc-webhook + .py (750 root:wazuh, the shipped convention) into the integrations volume and inserts a marker-fenced <integration> block into ossec.conf; there is no env var and no drop-in directory, and wazuh-integratord does not exist as a process until such a block does. Volume-written rather than image-baked because this package does not build the vendor image; re-asserted every deploy so an image upgrade that reseeds the volume converges instead of silently dropping the route.

vars.webhookAlertLevel (default 10) is the only volume control the edge has; the bridge renders every POST it receives.

Far side, from nine synthetic failed-SSH lines (rule 5712, level 10):

2026-08-15T00:15:22Z @_shc_appservice:… m.room.message
"[wazuh] level 10 sshd: brute force trying to get access to the system. Non existent user.
on 489b7a0c87f2 (rule 5712) - /var/ossec/logs/active-responses.log"

★ One diagnostic worth keeping: before the edge was bound, the same script failed with POST failed: <urlopen error [Errno -2] Name or service not known>; the manager could not resolve prod.hookshot.hookshot until the plug put it on the bridge’s network. The bind is what makes the name exist, which is the argument for proving edges by binding them rather than by rehearsing their commands.

3.3 GitLab: edge built, bind REFUSED by an orchestrator boundary

Section titled “3.3 GitLab: edge built, bind REFUSED by an orchestrator boundary”

apps/flagship/gitlab/webhook.plug.yaml drives GitLab’s instance-level Slack-notifications integration through gitlab-rails runner, which emits the Slack incoming-webhook dialect (text + attachments), the dialect the socket advertises and hookshot renders natively. Instance level, not per project, because a bus that carries only the projects somebody remembered to wire is not a bus; the job propagates the instance row so projects inherit it. Events: issues, merge requests, tag pushes and pipelines (webhookBrokenPipelinesOnly, default true). Push, note and wiki events stay off; they are the firehose.

★ The file states plainly that this edge overrides §6.3’s recommendation (GitLab 17.3+ has a native Matrix integration) on the owner’s ruling: one delivery path for the whole estate beats three that fail differently.

★★ AND THE BIND DOES NOT LAND ON THIS ESTATE, for a reason that has nothing to do with either package. Measured, shc update gitlab -i webhook=hookshot:

gitlab: degraded
shc: [X500059DCKCMP] docker stack failed, exit status 1, output …
network "corp_prod_hookshot_default" is declared as external,
but it is not in the right scope: "local" instead of "swarm"

The update rolled back cleanly (gitlab running, its other three edges intact). gitlab is a SWARM service (node-ca97810c782b, deployed with docker stack deploy); hookshot is a compose deployment on node-d5879f311b04, so the network an integration attaches a consumer to is a LOCAL bridge. A swarm service cannot attach a local-scope network, and no amount of plug authoring changes that; this is a placement fact, and the first time the estate has been asked to wire a swarm consumer to a compose provider.

The fork, stated rather than taken, because each arm costs something an edge may not spend on its own:

  1. Place hookshot on the swarm. Its network becomes an attachable overlay, which BOTH orchestrators can join; compose containers can attach an attachable overlay, which is why this is the only arm that serves all three producers at once. Costs a shc move of a stack whose other two edges are proven and would be re-rendered.
  2. Publish the listener on the node and advertise that as urlBase. Reachable from every node over the private network, at the price of a bearer URL that is no longer confined to the fabric, and urlBase is provider-global, so it changes the posture for glitchtip and wazuh too.
  3. Public ingress urlBase: the §6.1 posture this estate’s SSRF ruling already declined; it puts the bearer secret on the internet.

Until one is chosen, GitLab’s own notifications stay unrouted; nothing was left half-configured on the GitLab side (the failed update never reached the job, and the instance-level integration row does not exist).

3.4 Same day, second pass: the boundary resolved WITHOUT the move

Section titled “3.4 Same day, second pass: the boundary resolved WITHOUT the move”

Arm 1’s intent (hookshot’s network becomes an attachable overlay both orchestrators can join) turned out not to need shc move at all, because the literal move has a landmine the fork missed: a swarm-runtime stack’s default network is an overlay but not attachable, so moving hookshot onto the swarm would have handed gitlab its network while cutting off the two standalone-compose producers. The engine already owns the correct mechanism: ensureSwarmAttachableNetworks materialises any AUTHORED non-default network as an --attachable overlay when swarm is active (it deliberately never touches <project>_default), and this estate has proven that arm live for months (gitlab, a swarm service, reaches its compose-deployed postgres over corp_prod_gitlab_db_public).

So the hookshot service now authors public beside default (apps/standalone/hookshot/compose.yaml). One shc update hookshot later, corp_prod_hookshot_public existed as swarm overlay attachable=true, the socket-hostname alias (prod.hookshot.hookshot) landed on it, the bridge came back with its 3 connections, and shc update gitlab -i webhook=hookshot sailed through the very docker stack deploy that had refused; the gitlab task carries the overlay as its fifth network. glitchtip and wazuh were re-rendered onto the overlay and both re-proven end to end (wazuh 2026-08-15T01:41:30Z, glitchtip 2026-08-15T01:57:15Z, same room). The SSRF posture is untouched: urlBase stays in-fabric.

The bind then measured two more GitLab-18.x realities the plug had not met, both surfaced by one save! when the job finally ran:

Validation failed: Organization can't be blank, Exactly one of project_id,
group_id, organization_id must be present, Webhook is blocked: Requests to
the local network are not allowed
  • instance-level integrations are organization-owned in 18.x; the job now assigns Organizations::Organization::DEFAULT_ORGANIZATION_ID when the row carries none;
  • GitLab has its own SSRF gate, the exact counterpart of glitchtip’s GLITCHTIP_ALLOW_PRIVATE_IPS: a local-network webhook is refused at save unless allow_local_requests_from_web_hooks_and_services is on. The package gained vars.webhookAllowLocalNetwork (default "false"): true lets the job enable the setting loudly, false makes it refuse loudly, the same consent shape the glitchtip edge ruled, and the estate binds with it true.

Two operational traps this pass measured, worth their own records:

  • D6: a bare shc update on a version-unpinned deployment re-renders it from the CATALOGUE (X200971UPDHED warns; X200093APPORP names the dropped files). The three producer stacks ran --from <local path> packages, and one bare update silently reverted glitchtip’s app tree; the GLITCHTIP_ALLOW_PRIVATE_IPS env vanished and every alert died at the send gate (“webhook delivery blocked: URL resolves to a private/internal IP”) while the recipient row still looked configured. Until a release carries these packages, every update of the three producers must restate --from.
  • D7: a consumer whose only render change is a provider network’s external NAME is not recreated by docker compose up: the service’s own config hash keys on the network KEY (__shc_int_2), not the external name under it, so glitchtip kept running on the dead local bridge while its rendered compose said overlay. One --force-recreate converged it; wazuh, whose render changed in other ways too, recreated on its own.

3.5 Same day, third pass: GitLab producer proven; the silent-skip landmine

Section titled “3.5 Same day, third pass: GitLab producer proven; the silent-skip landmine”

The §3.4 bind left the instance Slack row active (org=1, allow_local=true, in-fabric URL) and (despite Admin::PropagateIntegrationService being nil on 18.11) all six projects inherited it (inherit_from_id set, same masked URL). 18.x save! of an instance integration enqueues propagation on its own, so the job’s if defined?(Admin::…) skip was a no-op that happened to converge. That is not a contract: a GitLab that does not auto-enqueue would report success with zero inherited rows, which is the silent all-alerts-dropped edge the second pass named. The job now calls Integrations::PropagateService.new(integration).execute (the living surface, defined? == "constant" on this estate) and raises if neither constant exists.

Far-side producer proof, 2026-08-15T03:03:13Z, from swarm02 (gitlab’s node) through the overlay: Integrations::Slack#execute of an issue payload returned true; hookshot logged onGenericHook for the gitlab hook id and MatrixSender Sent event; the room gained:

2026-08-15T03:03:13Z @_shc_appservice:… m.room.message
"Received webhook data: ```json { \"payload\": \"{\\\"fallback\\\":\\\"[<https://git.corp.bitspur.com/root/clone-proof|root/clone-proof>] Issue …"

Reachability was independently confirmed from the gitlab container: prod.hookshot.hookshot → 10.0.93.2, POST /webhook/shc-health-probe → 404 {"ok":false,"error":"Webhook not found"} (listener up, unknown id refused). Sidekiq was running.

★ Render gap, measured: GitLab’s Slack integration POSTs the incoming-webhook dialect as application/x-www-form-urlencoded with a payload=<json> field, not a JSON body with top-level text. Hookshot’s generic handler therefore takes the “Received webhook data” dump rather than the text extract. Delivery is the ruling’s bar and it cleared; making the visible body a one-line issue title is a transform/Content-Type follow-up, not a missing producer.

3.6 Same day, fifth pass: the render gap closes producer-side

Section titled “3.6 Same day, fifth pass: the render gap closes producer-side”

The follow-up §3.5 called for was assumed to live on the bridge. It did not. GitLab’s chat notifiers all build the same Integrations::ChatMessage::* object and differ only in how they encode it, and Integrations::Pumble#notify (18.11 omnibus, app/models/concerns/integrations/base/pumble.rb) posts

header = { 'Content-Type' => 'application/json' }
Clients::HTTP.post(webhook, headers: header, body: Gitlab::Json.dump({ text: message.summary }))

Exactly the one shape transformHookData renders. Both rows were live against hook 9ddb6e23-… briefly, which measured the difference with no other variable: issue #5 at 2026-08-15T07:51:56Z produced two room messages 15 ms apart, …889Z the sentence (Pumble) and …904Z the fenced dump (Slack).

The edge moved to Pumble, and the converge job destroys any instance-level Slack row whose webhook is on this bus, scoped by URL prefix so an operator’s real Slack row is never touched, and destroyed rather than deactivated because two active notifiers on one hook id double-deliver every event (M12: the replaced surface goes in the same program). Run verbatim from the plug against the live instance at 07:58:40Z: removed 1 superseded instance-level Slack row(s) pointing at this bus, seven Slack rows (instance + six inherited) gone, seven Pumble rows in their place. Re-running it is a no-op.

Fresh proof at 07:59:32Z: a real issue (root/clone-proof #6) produced exactly one onGenericHook (08:00:12:355), one MatrixSender Sent event (08:00:12:428), and one room message:

2026-08-15T08:00:12.394Z @_shc_appservice:… m.room.message
"[<https://git.corp.bitspur.com/root/clone-proof|Administrator / clone-proof>]
Issue <https://git.corp.bitspur.com/root/clone-proof/-/work_items/6|#6
hs-r5b-075931 far-side proof> opened by Administrator (root)"

gitlab: AMBER → GREEN, with no transformation function. The allowJsTransformationFunctions ruling is still owed, now for glitchtip alone; all eight GlitchTip 6.2.5 RecipientTypes were re-read at the running pin and none carries issue detail in a top-level string text or html.

3.7 The edge ran from the package, not from a transcript, and what that exposed

Section titled “3.7 The edge ran from the package, not from a transcript, and what that exposed”

§3.6’s converge was run verbatim out of the plug file against the live instance. That proves the Ruby and not the packaging, and the difference was measurable: the gitlab deployment’s stored app tree (/var/lib/shc/deployments/corp/gitlab/prod/app/) was still a revision that predated §3.5, so it wrote Integrations::Slack; every future replay of post_deployment, for any reason, would have put the form-encoded row back beside the Pumble one and started double-delivering. A converged estate was one unrelated shc update away from the defect it had just fixed.

So the landed package was pushed to the estate’s local app dir and the real update was run: shc update gitlab -t corp -e prod --from /home/admin/gitlab-app -i webhook=hookshot, 2026-08-15T19:24:50Z, job configure-matrix-webhook on service gitlab at 19:25:46Z, stack back to running, exit 0. Stored plug b5875286 → b5ecd5ec (the landed bytes), app_digest 3fe052f2 → 0f2f408b. The row state is what proves which job ran: afterwards the instance row is still the single Integrations::Pumble on the bus URL with six inherited children and zero Slack rows; the tree it replaced would have created one.

Fresh far-side proof, all three producers fired after that update, each landing exactly one onGenericHook and one MatrixSender Sent event:

19:28:22.985Z [wazuh] level 10 sshd: brute force trying to get access to the system.
Non existent user. on 70f94003ca68 (rule 5712) - …/active-responses.log
19:29:03.797Z GlitchTip Alert
19:29:14.996Z [<…|Administrator / clone-proof>] Issue <…/work_items/9|#9 r7-192819
far-side proof> opened by Administrator (root)

wazuh GREEN, gitlab GREEN (one message, no dump twin), glitchtip AMBER: delivers, renders the constant banner, and that is the one edge the transformation-function ruling is still owed for.

★ AND THE PUBLISHED CATALOGUE IS THE LANDMINE D6 DESCRIBED, ARMED. Measured 19:20Z: the estate’s shc repo (https://selfhostedcloud.org/apps/index.yaml, synced 18:02:18Z) carries the Slack-writing gitlab edge (Integrations::Slack.where(instance: true).first_or_initialize at line 138 of its webhook.plug.yaml), and shc update gitlab --check reports the deployment STALE against it. Until a release publishes the fix, a bare shc update gitlab on any estate re-renders the form-encoded edge, which renders every alert as a dump and, beside a live Pumble row, double-delivers. Restate --from on every update of this stack until a release carries the package.


  • apps/standalone/hookshot/tests/tests.bats: 26 tests, including four new ones: no hook job may reintroduce the too-late invite; the boot script invites, joins with the appservice token, and reads back; the convergence is never fatal and never blocks the boot; derived URLs are printed masked. Two guards that had been silently passing over a nonexistent path since the apps/ re-tiering ('..', 'matrix'; the provider is apps/flagship/matrix, not a sibling) now resolve the provider by name and fail if it moves again.
  • tests/unit/catalog/webhook_derivation_parity_test.go: new. The derivation is now written in three languages (JavaScript on the bridge, Python on the glitchtip and wazuh edges, Ruby on the gitlab edge), and a single divergent nibble produces a URL that 404s forever while every stack reports itself configured. The test pins the canonical VALUE (recomputed in Go from the recipe’s definition) and the canonical TEXT of each language’s spelling, and additionally requires every webhook.plug.yaml to declare a [a-z0-9-]+ hook, to carry an unintegration job, and to print its URL masked.
  • (second pass) the hookshot bats guard that pinned “default network only” now demands default AND public; dropping public reopens exactly the §3.3 scope refusal for every swarm-runtime producer while the compose producers keep working, which is the asymmetry that earns it a name. And apps/flagship/gitlab/tests/tests.bats gained a guard pinning the organization_id assignment, the consent-var wiring, the SSRF-gate name, the loud refusal, and the living Integrations::PropagateService call ending in a raise (not a defined? skip); mutation-proofed red-by-name.

5. State at hand-off, and what is still owed

Section titled “5. State at hand-off, and what is still owed”

On the estate (bitspur corp/prod), after the §3.5 third pass: hookshot running with three static connections (glitchtip, wazuh, gitlab) into the alerts room !IqVzZGjkiYuEdXYyiA:matrix.corp.bitspur.com, its public network an attachable swarm overlay every producer attaches; all three producers bound and proven end to end over that overlay: glitchtip 00:03:12Z and 01:57:15Z, wazuh 00:15:22Z and 01:41:30Z, gitlab 03:03:13Z. The §3.3 boundary is closed by the authored provider network, not by a move.

The one drift this lane keeps deliberately: the producer stacks were brought up from --from <local path> packages so the estate could run the edges before the catalogue carried them. D6 above is what happens when anyone forgets: a bare shc update reverts the tree to the catalogue and the edge dies in whichever way its package machinery was holding open. Once a release publishes these packages, each stack should be updated back onto the catalogue app; until then, restate --from on every update.

★ Corrected 19:20Z (§3.7): “the stacks run --from packages” is not a property the estate records, and only hookshot has it. Read off the manifests: hookshot carries app_name=/home/admin/hookshot-app, while gitlab, glitchtip and wazuh all carry catalogue app names; their local-path origin left a stale snapshot, not a pin. --check says so: gitlab and glitchtip are STALE against the published catalogue and wazuh is CURRENT with it. So the mitigation is a habit, not a mechanism, and the catalogue it would fall back to still carries the form-encoded gitlab edge (§3.7).

Still owed:

  1. A second appservice registration on the matrix package (§11.4, now concrete rather than hypothetical). Until it exists, whichever consumer binds first owns url and every other consumer is send-only. That is survivable for a webhook bus and fatal for anything that needs inbound events.
  2. shc job run context (D4): a hand-run job that silently renders defaults.
  3. The html and fenced-JSON render arms of transformHookData, and enableHttpGet: false → 405, remain source-read only (readiness §4.4/4.5). GitLab’s form-encoded payload= Slack body is now a measured sibling of that gap (§3.5): the event lands, the visible body is the generic dump.
  4. Nothing here touched promanager; the same two-stack bring-up applies there and its matrix stack’s appservice.url should be read before assuming it is free.