SHC Telemetry Fabric (Design)

Status: slices 1 and 3 are implemented (shc/collector/ — the ocb distro with the shcscope processor, plus the shc/core/collectorcfg config renderer; deployment wiring is the recorded seam). The rest is design. Written 2026-07-31, and revised the same day against measured evidence: a container-log parsing benchmark over 28,657 real lines from both live estates, and a secret-surface census over the same corpus. Two sections that were speculative are now empirical — Log parsing and The scrub — and the parsing question has an owner ruling on it, as does routing precedence.

This document proposes a telemetry fabric owned by the SHC daemon: a per-node collector that ingests container logs and app-emitted traces and metrics, stamps every signal with the deployment scope SHC already knows, redacts secrets at a single chokepoint, applies operator filter policy, and routes the result to a destination chosen by scoped config.

It is one design covering four things that are separate today and should not be:

• Container logs are not collected at all. Application stdout lands in the node’s docker json-file logs and stays there.
• SHC’s own telemetry rides a single global exporter with no per-scope destination.
• Per-scope routing exists as scaffolding with zero consumers.
• The support bundle (below) is a scraper today by necessity, because the observability store does not hold the data it needs.

For today’s scoped-config ladder see configuration/config-files.md; for the shape this document imitates — a design proposal built in independently provable slices — see cluster/network-fabric.md.

Three claims, each verified against the tree rather than assumed.

Path convention, as everywhere else in these docs: a bare path is in the pinned ce module; an shc/… prefix is this repo.

1. SHC pushes its own telemetry, and only its own. core/otel defines one collector shared by the daemon, the cron runner and the client; the SDK half that builds the exporters is shc/core/otelboot/otelboot.go. Export is OTLP-HTTP and push-only — core/otel/otel.go says so in its own header (“No /metrics endpoint. No prometheus client. Push-only.”), and neither tree carries a Prometheus client dependency or a /metrics route. otel.provider selects one otel.providers.<name> blob; unset engages a managed default (the system OpenObserve behind SHC_HOST). The daemon’s boot path (shc/daemon/wiring/otel_wire.go, resolving through ce’s daemon/otel_boot.go) selects one provider among the configured options — it does not fan out to several.

2. Container logs are not shipped. There is no filelog receiver, no Fluent Bit, no Vector, no docker-json pipe. A container’s stdout reaches the host’s json-file log driver and stops there — node-local, unqueryable from the control plane, invisible to a support bundle, and lost when the node is reprovisioned. Jobs are containers, so job output has the same fate with a shorter lifetime.

3. Per-scope TAGGING is live; per-scope ROUTING is dead scaffolding. This distinction is the one most often collapsed, so it is worth stating precisely:

So “per-tenant telemetry” today means an instrumentation-scope label on one destination, not a per-tenant destination. Setting otel.provider at tenant scope does not send that tenant’s data anywhere different. The multi-endpoint machinery is half-present and unreachable: the seam is injectable, nothing injects.

The collector runs on every node. This is forced, not chosen: container logs are physically local to the node running the container, and no amount of control-plane cleverness moves bytes that were never written anywhere else.

The split that keeps this coherent:

• The leader owns POLICY. Scoped config — which destination serves which scope, which filters apply where — is resolved on the leader and distributed, exactly as every other scoped setting is.
• Each node EXECUTES collection. It reads its own container logs, receives its own apps’ pushes, enriches, scrubs, filters, and pushes onward.

A node that is offline during a policy change converges when it returns, in the same reconcile-based shape as the rest of the daemon.

The expensive part of any container-log pipeline is answering “whose log is this?”. SHC already answered it at deploy time:

• Every container carries shc.tenant, shc.stack, shc.environment and a per-service shc.service label (modules/app/inject_labels.go, InjectSHCLabels). That is the whole scope tuple plus the service name, already on the container, written by the same renderer that deployed it.
• The Compose project name is <tenant>_<environment>_<stack> (core/dockerscope/dockerscope.go, ProjectName) — the same scope again, in the container’s own metadata, as a fallback and a cross-check.

The collector reads the labels off the container it is tailing and stamps the scope. No cluster-wide lookup table, no cache to go stale, no control-plane round trip per log line. This is the same property that makes k8sattributesprocessor cheap in Kubernetes, and SHC has the equivalent already written.

One shape, every signal, every node:

receive → RECOMBINE (multi-line) → ANSI-STRIP → enrich (scope)
        → SCRUB (security) → FILTER (operator policy) → route (scoped provider) → push

There is no parse stage. That is a ruling, not an omission — see Log parsing for the benchmark behind it. The two stages that would have sat in front of a parser survive the ruling because neither is a parser feature.

This matches the canonical OTel processor order (memory_limiter → enrichment → redaction → filter → sampling → batch → export), and the ordering carries real weight:

• Recombination is its own stage and it comes first. A per-stream recombine keyed on an anchor pattern. On the measured corpus 6.8% of physical lines are continuations and no candidate joins them — passthrough included, so a 40-frame stack trace becomes 40 orphan records either way. Recombination is orthogonal to parsing and has to be built regardless.
• The ANSI strip stays, at roughly 20 bytes of work per line. Without it, raw lines reach the destination full of escape sequences: unreadable in the UI, and hostile to any parsing done at ingest time. Measured: every huly and affine winston line is wrapped in ESC[33m…ESC[39m, and hulykvs interleaves ANSI inside the key=value separator.
• Stripping comes before the scrub. Measured, not assumed: hulykvs prints a Postgres DSN whose key and value are separated by ANSI escape sequences, so a raw-byte rule scanning the unstripped line cannot fire on it.
• Scrub comes before filter, and before anything leaves the node. A secret that reaches a destination is a secret that leaked; redaction has to happen at the earliest point where the whole record is assembled. Putting it after filtering would make the security guarantee depend on operator-tunable policy.
• Enrichment comes before both, because filter rules address scope attributes and the scrub wants to know what it is looking at.
• Filtering comes before routing, so data dropped at source is never processed, never stored and never sent — the cheapest possible outcome, and the one that matters when the destination bills by ingested volume.

The per-node collector is also the single chokepoint for secret redaction. One scrub there makes the master OpenObserve clean by construction, which makes the support bundle’s store query clean by construction, which makes a tenant’s own destination clean by construction. Three guarantees from one implementation site.

Turning an arbitrary line of container stdout into a record with a level, a timestamp and fields has a simple answer (“ship the line, let the destination parse it at query time”) and an elaborate one SHC already designed and shipped once, in the Python era.

The ruling is the simple answer: passthrough. Owner ruling, 2026-07-31 — “I’m ok with passthrough. People can filter on stacks, environments, tenants and/or regex if they really care about something.”

The rest of this section is the evidence that ruling was made against, and it stays in the document deliberately. A benchmark on real data ran first and found a clear technical optimum; the ruling went the other way, on grounds the benchmark itself supports. Keeping the measurements records exactly what is being traded away, and means revisiting the decision later does not mean re-running the work.

The corpus is not in git and must not be. It is mode 700 on the development box and it contains live credentials from the estates it was harvested from — see The scrub, which is written from exactly those findings.

Where it is. The full Python tree survives only on backup/pre-purge-main (4182 commits back to the initial commit); the working reference copy at /var/tmp/shc_pyref is a 42-file, five-module subset that does not include it. The files were last live at c7abf2114^, the Go-port deletion commit: shc/logparse.py (700 lines, born 2025-12-16) and a whole shc/modules/logs/ module — parser.py (with DrainLogParser), tailer.py, export.py, service.py, router.py, tasks.py, types.py, plus unit tests.

The shipped design: a structured cascade first, template mining as a fallback enricher on the residue.

1. _try_json → _try_logfmt → _try_syslog → _try_weblog. First match wins and returns immediately.
2. Otherwise _try_generic: scrape a level and a timestamp with regexes (_detect_log_level, _extract_timestamp) and decode a syslog priority if one is present.
3. Then, if Drain3 was available, mine a template from the line, attach template and template_id, and flip degraded to false.

The ordering is the insight. Where a structured parser applies it is objectively correct — valid JSON parsed as JSON is not a guess — so the cheap deterministic attempts run first and the template miner only ever sees what none of them could claim.

Two corrections to the recollection, verified rather than inferred (the whole 4182-commit history was pickaxed):

• pygrok was never wired. It was capability-detected, reserved in the ParserType enum beside DRAIN3, and reported in features() — and referenced nowhere else except its own test. A stub.
• The parser duel was never implemented. The remembered design — run both parsers, measure which did better per line, lock the winner in per stream — produced zero commits for confidence, best_parser, winner or duel, and the first version of logparse.py (691 lines) is about the same size as the last (700). It was an intention that never reached code.

The duel — run both parsers, measure which did better per line, lock the winner in per stream — was rejected on its own merits before the benchmark ran. It doubles CPU on the hottest path in the product; “which did better” has no objective metric, because a template miner manufactures plausible structure out of noise and scores well on it; and where the cascade applies it is already right, so the measurement is wasted work. The corpus supplied a fourth reason: two sources put two grammars on one stream with no delimiter — nginx interleaves access and error lines, keygen-worker interleaves Ruby Logger and Sidekiq output — so a per-stream lock would have been locked to the loser on real data.

A faithful Go port of the Python cascade costs 20,233 ns/line, and 84% of that is two regex helpers — not the cascade, not the ordering, not the fallback:

The cause is a property of the engine, not of the patterns’ authorship: these patterns have no literal prefix, so RE2 runs its NFA from every byte offset of every line. Python’s backtracking re has literal prefilters that made the identical patterns cheap. The Python original was fine. A literal transcription into Go would have been a disaster — and it would have looked like a faithful port the whole time.

Same architecture minus the regex engine is 30x cheaper and better on every quality axis. That is the whole finding: this is a Go-runtime result, not an architecture result.

For the record, because the ruling is a decision not to build this: the winning candidate was C9-tuned at 666 ns/line, 4 B/line, 0 allocs/line, 0 false structure, 74.6% timestamps / 79.5% levels / 61.7% structured fields. It was the Python cascade’s architecture with the regex engine removed — a single-pass byte router, ANSI-stripped before routing, hand-written scanners throughout with no regexp on the hot path, per-stream memoization used as a hint that falls through rather than a lock, and a generic fallback that still scrapes level and timestamp.

It is cheaper and better than every regex candidate on every axis. It is not being built.

Twelve candidates, sorted by cost. Quality denominators exclude the 56 blank lines.

There is no cost/quality trade above 666 ns. C9 is cheaper and better than every regex candidate on every axis. The only genuine trade is along C1 → C1L → C9, and it spans 663 ns.

The C3 row also carries the only false structure in the whole benchmark: 8 traefik lines where zerolog’s interleaved ANSI (ESC[36macmeCA=ESC[0m) manufactured spurious k=v pairs that the Python logfmt matcher claimed. The byte router rejects them by requiring the = to be preceded by an identifier run starting at a word boundary. After the prefilter: zero false structure across 28,601 lines, for every candidate. The prefilter’s real value is correctness, not the 20% of CPU it saves.

The rate columns below are measured from timestamps in the corpus itself, not invented:

Against that, the fraction of one core on one node:

Read the table this way. At every rate SHC has actually been observed to produce, all twelve candidates are free — the entire CPU argument between passthrough and the most expensive scanner design is seven hundredths of one percent of a single core. CPU only becomes decisive in the flood column, and there the split is binary rather than graded: the scanner candidates keep up and the RE2 candidates do not.

This is the first half of why the ruling is sound. The decision was never really about CPU, in either direction. Nobody is buying meaningful headroom by choosing passthrough, and nobody would have been paying a meaningful price by choosing C9. Whatever settles this question, it is not the core fraction.

Independent oracle census over the corpus:

Of 4,772 JSON lines, the naive line[0] == '{' fast path sees 269 — 5.6%. Every Node service (huly ×7, affine ×2) wraps winston JSON in ESC[33m…ESC[39m. A ~20-byte SGR strip recovers all of them and is worth +15.7 percentage points of structured-field yield on its own. It is also cheaper than not stripping: on the huly frontend stream the ANSI-blind candidate costs 1,406 ns/line because it fails to route to JSON and then pays a full generic scan of a 400-byte line; the stripping candidate costs 453 ns.

The 35% plain tail is roughly twenty mutually incompatible positional grammars — six non-RFC3339 timestamp formats, one source that spells the level with a single glyph, one with no timestamp at all.

This is the second half of why the ruling is sound, and it is the decisive half. The benchmark’s own strongest finding is that parser quality is not bounded by the parsing architecture at all — it is bounded by a table of per-format patterns that somebody has to keep writing. Three gaps, all found on real data, all free in CPU terms and none of them free in attention:

• Slash dates. 2026/07/31 15:10:00 (Go stdlib log, nginx error log) and 07/31/2026, 3:03:21 PM (NestJS). 4,472 lines — 15.6% of the corpus — carry a timestamp that no candidate can read. Two whole sources score 0%.
• Timezone-less and comma-fraction ISO. 2026-07-31 14:04:03,881 (Quarkus), 2026-07-27T19:14:10,937Z (Elasticsearch). The Python pattern required a timezone, which is exactly why keycloak scores 0.2% under the port and 93.1% under C9. This single entry is most of C9’s timestamp advantage.
• Level vocabulary. NestJS VERBOSE and LOG (1,545 occurrences), zerolog INF/WRN/ERR.

Adding these three entries moves the corpus from 74.6% → ~90% timestamps and 79.5% → ~88% levels at zero CPU cost. The remaining quality lives in the table, not in the code.

Which is the point. Those three gaps were found by pointing a benchmark at 23 images from two estates that SHC controls. The catalogue is open-ended and the formats drift: every new app, every upstream logging-library change, every customer running software nobody here has seen produces another entry someone has to notice, diagnose and add. That is a permanent maintenance treadmill attached to a component whose measured value is a percentage-point yield on fields.

Passthrough deletes the treadmill and the entire parser as engineering surface — not 666 ns of CPU, which was never the issue, but an indefinite obligation to chase other people’s log formats.

Ship the line. The collector recombines multi-line records, strips ANSI, stamps scope, scrubs, filters and forwards; it does not attempt to understand the line’s grammar. Queries filter on scope, on literal substrings, and on regular expressions at the destination if someone really cares about something.

Scope filtering is completely unaffected, and this is the load-bearing detail. (tenant, stack, environment, service) comes from container labels — modules/app/inject_labels.go, written at deploy time by the same renderer that deployed the container — not from parsing the line. The fabric’s primary filter axis never depended on the parser and does not lose anything here.

Stated plainly, because the benchmark measured both of these and softening them would waste the measurement:

1. There is no single otel.severity: warn knob. Severity filtering becomes per-app literal or regex rules, and the corpus proves that genuinely is per-app: valkey signals its level with one glyph, zerolog uses INF/WRN/ERR, NestJS uses VERBOSE and LOG, and a scanned level only reached 79.5% of lines with a curated vocabulary. A uniform severity floor across a whole scope is not available; what is available is a rule per noisy app. See Filtering, where the T1 severity knob is scoped down accordingly.

2. SHC pays to parse its own logs at the destination. 40.3% of the corpus is logfmt from SHC’s own daemon — already structured, thrown away at the node and reconstructed at query time. Practically this means an ingest-time transform at OpenObserve for SHC’s own streams. The cost does not disappear; it moves from N nodes to one destination instance, which is a smaller total and a different bill, and it is one transform against a format SHC itself controls rather than a table against formats it does not.

Two stages people will be tempted to remove along with the parser. Both stay, and both are pipeline stages in their own right:

• The ANSI strip. ~20 bytes of work per line. Without it, raw lines land at the destination full of escape sequences — unreadable in the UI and hostile to the ingest-time parsing that passthrough now depends on. Measured: every huly and affine winston line is wrapped in ESC[33m…ESC[39m, and hulykvs interleaves ANSI inside the key=value separator, so even a destination-side logfmt transform needs the strip to have happened first.
• Multi-line recombination. Orthogonal to parsing. No candidate joins continuations, passthrough included: 6.8% of physical lines are continuations and a 40-frame stack trace becomes 40 orphan records under every one of the twelve. It has to be built regardless of this ruling.

Recorded rather than deleted, so revisiting the ruling does not mean re-running the benchmark. All of these were live questions for a parser SHC is no longer building:

• This is a parser benchmark, not a collector benchmark. No file tailing, no docker json-file decoding, no OTLP marshalling, no compression, no network, no batching. In a real per-node collector those very likely dominate 666 ns/line. The report answers “which parser”, not “what does the collector cost”. The flood-column conclusions still hold, because a slow parser cannot be rescued by cheap I/O. Under the ruling this caveat becomes the whole remaining cost question: with no parser in the pipeline, the collector’s footprint is exactly the part the benchmark did not measure, and open decision 4 is where it gets answered.
• The corpus is SHC’s own two estates. That is the point for SHC’s own telemetry, and it is not representative of arbitrary customer applications. The 40%-logfmt share exists because shcd and keygen dominate the sample; a customer estate running different software would shift the mix. Note which way that cuts: it makes the pattern-table treadmill worse, not better, and so it supports the ruling rather than undermining it.
• The Python-port candidate is biased in its own favour — one sanity check was computed arithmetically instead of materializing a string as the Python does. A literal transcription would be slower still.
• Single goroutine, pooled records. Nothing contends; cold-start allocation is not measured.

The redaction stage used to be specified from reasoning about what secrets look like. The same corpus that settled the parser question replaced that reasoning with a census, and the census changes the design.

18 real credential occurrences across 5 of the 36 files, from SHC’s own shipped applications running on SHC’s own estates:

1. The worst leak matches none of the obvious regexes. The Keygen keys are ALL_CAPS_NAME=<hex> on an unstructured line from a migration container. No bearer prefix, no PEM header, no recognizable envelope — the entire shape of the rule set that would have been written from memory misses the single most damaging finding in the corpus.
2. Four of the five findings are identifiable by FIELD NAME, not by value shape. Secret, token, the DSN’s password position, the ?token= query parameter. Provenance and field-name redaction beats value-shape regex — this was suspected before and is now measured.
3. A pure value-shape rule is unusable alone. “32 or more hex characters” ran at roughly 76% false positives over the corpus — build hashes, certificate DNS labels, container digests. Do not ship a hex-length rule on its own; it trains people to ignore redactions.
4. The scrub must run after the ANSI strip. hulykvs separates the DSN’s key from its value with escape sequences, so a raw-byte field rule cannot fire on the unstripped line. This is why the pipeline puts stripping ahead of the scrub.
5. A dedicated query-string rule is required. ?token= alone is 11 of the 18 occurrences. Redacting query-parameter values by parameter name is the single highest-yield rule available.
6. A collector that samples only RUNNING containers misses the worst finding. It was in an exited one-shot migration container. Both the fabric and the support bundle must cover exited and one-shot containers and jobs, or the rule set is aimed at the wrong population.

Under passthrough the scrub has no parsed fields to work from, and consequence 2 says field names are where the signal is. So the scrub carries its own minimal key-name scan over the ANSI-stripped line — name=, "name":, ?name= — sufficient to find the four field-name findings and nothing more. That is a redaction concern, deliberately not a parse stage wearing a different hat, and it is scoped by a curated key list rather than by a general grammar.

The exact rule set stays blocked on the secret-surface audit. What the corpus supplies is its shape and its priorities, which is what was missing.

Separately actionable, and it is the empirical case for scrub-at-collector being non-negotiable: SHC-shipped applications leak secrets to stdout today. Keygen prints live crypto keys; huly puts JWTs in access-log URLs. This is close to harmless while logs stay host-local in the json-file driver, which is where they are now. The fabric changes that — it would ship those bytes into OpenObserve, and a support bundle would mail them to whoever asked for it.

Two consequences: the scrub stage is a prerequisite for the fabric rather than a follow-up to it, and the applications themselves want a log-hygiene audit as a separate piece of work.

The recurring fork — “vendored otel-collector-contrib (has every receiver, but it is a large external binary against SHC’s pure-Go single-binary ethos)” versus “purpose-built Go collector (on-brand, scrub inline)” — is a false choice.

ocb, the OpenTelemetry Collector Builder, takes a manifest naming only the components you want and compiles a custom distro: still pure Go, still one binary, with no unused receivers linked in. SHC gets the standard components and the small footprint at the same time.

The recommended manifest:

Exactly one component is ours to write, and it is the direct analog of k8sattributesprocessor. The passthrough ruling shrinks this manifest rather than complicating it: there is no parser component, custom or otherwise, and the receiver’s own operators cover everything that remains.

Why hand-rolling is rejected. A hand-written collector must reimplement OTTL — an entire expression language — to express filter and transform rules, which is a months-long project whose output would still be non-standard. Users arriving from any other OTel deployment expect OTTL semantics. The two receivers above also do double duty: dockerstatsreceiver and hostmetricsreceiver feed the support bundle’s state view as continuous time-series, so we get “what the node looked like an hour ago” for free rather than only “what it looks like now”.

Deployment follows the OTel canon: an agent tier on every node (receive/enrich/redact/cheap-filter) and an optional gateway tier later, needed only for tail-based sampling, which requires a whole-trace view no single agent has.

Filtering happens at three layers, cheapest first: SDK (severity floors, Views, head sampling — free, the signal is never generated), agent collector (where roughly 90% of policy belongs, SHC’s included), gateway (tail sampling only).

Owner ruling, final. Observability is a premium capability, so everything goes through SHC’s fabric. This is the open-core image-coupling doctrine applied to observability.

There is no otel=OPENOBSERVE integration and no app-facing otel socket. Both were considered and deliberately killed: a direct pipe from an app to the observability store is a path that bypasses SHC entirely — no scope stamp, no scrub, no filter, no scoped routing, and no reason to keep paying for the fabric. Removing the possibility is the point.

Done. What the ruling removed, so this is read against facts rather than a blank slate: the socket was named openobserve, not otel (builtin/openobserve/openobserve.socket.yaml), and it published the ingest ports — port (HTTP API and OTLP-HTTP ingest, default 5080) and grpcPort (OTLP/gRPC, default 5081) — alongside the organization and basic-auth credentials. The cross-tenant magic-token table (modules/app/integration_magic.go) carried an OPENOBSERVE entry beside KEYCLOAK, resolving to the master tenant’s openobserve stack with a per-tenant org override. Both are gone; KEYCLOAK is now the sole magic token. No app had ever bound the socket — no openobserve.plug.yaml was defined in any builtin or catalog app — so nothing needed migrating.

Implementation nuance, so this is not misread: OpenObserve still receives OTLP. Those ingest ports stay as internal plumbing that SHC’s own collector pushes to. The store listens, the collector pushes, and no app is ever wired to the store directly.

The one blessed path is -i otel=OTEL:

1. The app pushes to SHC’s collector.
2. SHC stamps shc.tenant, shc.stack, shc.environment.
3. The signal routes through the scoped otel.provider system.

Third-party destinations — ELK, a hosted vendor, a customer’s own store — connect as a provider, configured by the operator at whatever scope they choose. They are never wired per-app.

OTEL is a token, not a stack. It is deliberately not a magic provider: a magic token resolves to a real deployment in the master tenant, and the collector is a per-node system workload — so the edge is headless (no provider network join, no socket jobs, no tenant crossing) and the work happens at render time (ce modules/app/inject_otel.go, running right after InjectSHCLabels). Every service gains the OTel spec’s standard env — which is what makes the integration passthrough-compatible: every SDK honours it with zero app code:

The scope-derivation decision. The obvious passthrough answer — inject the scope tuple itself into OTEL_RESOURCE_ATTRIBUTES — was rejected: it stamps app-side, puts the tenant name in the app’s env, and fails the slice-6 proof (“without the app knowing what a tenant is”). What is injected instead is one opaque join key: shc.instance, a deterministic hash of (tenant, environment, stack, service), written BOTH as the sole injected resource attribute AND as a container label beside the shc.* scope labels. The collector’s shcscope processor indexes containers by that label (same config.v2.json census the log path reads), matches the OTLP resource’s attribute verbatim, stamps shc.tenant / shc.stack / shc.environment / shc.service from the labels, and removes the join key. Scope stamping stays collector-side; the app forwards opaque baggage. An unresolvable key is left in place as the operator’s debug handle.

The tuple is collector-OWNED, not collector-preferred: the stamp overwrites whatever the emitter sent under those names, and a resource the collector cannot attribute (unknown key, or no key at all) has the tuple stripped. The resource is app-controlled — any process can set OTEL_RESOURCE_ATTRIBUTES and any container on the node can reach the receiver — so a put-if-absent stamp would let an app forge its own scope, with two consequences both measured in review: the forged value rides to the destination through the scrub’s stamped-attribute skip, and it steers the record into ANOTHER tenant’s slice-5 route branch. Unattributable data still reaches the default destination; it just cannot claim a tenant.

Existing app-authored OTEL_* env always wins (the injection is put-if-absent), and network_mode services are skipped whole (the traefik-reload P0 rule — compose rejects extra_hosts against container:/service: modes).

Collector-side, core/collectorcfg renders the OTLP receiver (grpc + http) into two pipelines — traces and metrics/otlp, both through the same memory_limiter → shcscope → shcscrub → batch chain as the log path (the scrub’s traces/metrics halves apply the R7 attr_kv rule to every app-supplied attribute: span, event, link, scope, resource minus the collector-stamped tuple, metric datapoint — and the full line rules to every app-supplied free-text field beside them: span name, event names, scope name/version, status message, metric name/description/unit. App attributes are a secret surface exactly like log bodies, and so is a client span NAMED after the URL it fetched) — and deliberately no third: with no logs pipeline wired to the receiver, an SDK pushing to /v1/logs is refused at the receiver. The render also feeds integrations.<plug>.socket.{endpoint,protocol} into the consumer’s jinja namespace for apps whose OTel surface is file-configured rather than env-read. The edge persists as an ordinary plug-keyed sidecar row (-i otel! removes, bare update and recover replay), and the deployment seam owes the node-local publish of 4318 (see collector/README.md).

otel=OTEL carries TRACES and METRICS only. Never logs. Logs are the container-log collector’s exclusive domain: stdout → collector → scoped routing. An app cannot push its own stdout anywhere, so accepting logs on the integration would create a second, partial, unscoped log path beside the complete one. Not an app concern, not an integration signal, ever.

The generic plug and socket system still exists — apps wire to each other, so a determined user could build their own logs socket or route telemetry into an app they socketed themselves. That path is unsupported, unrecommended, and receives zero investment. The product value is otel=OTEL plus SHC’s providers, so that is the only thing built.

The side effect is the point: this collapses the build to one story — container logs via the collector, app traces and metrics via otel=OTEL, both enriched, both scrubbed, both scope-routed. No dual-target design, no cheap interim direct pipe that becomes permanent. The coupling is deliberate.

The model has no explicit scope argument. Setting otel.provider at a scope means “data from this scope goes there” — the same shape as dns.target and the vault providers, resolved through the one SHC scope ladder (most specific wins, always ending at Global):

Deployment > Stack > Environment > Tenant > Global

Stack outranks Environment; Deployment is Stack and Environment together; the walk always ends at Global. That order is spelled out authoritatively on Service.Get in modules/config/service.go. The resolver pattern to copy is DefaultScopedConfigValue in modules/app/dns_target.go — the seam behind the DNS target, which threads tenant/stack/environment into the config context and lets the underlying Get perform the full walk.

The master OpenObserve remains the node-wide default tier and therefore receives every tenant’s data unless a more specific row says otherwise.

An earlier estimate called this a small gap. That was wrong, and the corrected breakdown is the honest input to any scheduling decision:

Total ≈ 8–17 engineering days across two repos as originally scoped; the fan-out ruling deletes C, the largest line item. The cross-repo half is what makes this a program rather than a ticket. Do not half-build it — A alone is dead code, and B without D has nowhere to send the second stream.

The four-tier walk today fans out: a signal matching several tiers is recorded against all of them. This document recommended most-specific-wins — a tenant’s data goes to their destination only, matching how dns.target and the vault providers resolve.

Owner ruling, 2026-07-31: FAN-OUT. The recommendation was rejected; the existing four-tier fan-out walk is the ruled behavior. A signal matching several tiers goes to all of them — a tenant-scope destination receives that tenant’s data in addition to the master default, never instead of it.

Two consequences:

• Step C — the expensive emit-path rewrite — is DEAD. The hot path already fans out; there is nothing to re-encode. What remains of the A–F program is plumbing (B, D, E, F), not the 3–5-day hot-path slice, and slice 5 becomes prove the existing behavior, not build new behavior.
• The support bundle keeps one place to look. Under fan-out the master store always holds every scope’s data, so the bundle’s “query the destination serving that scope” edge case never bites — the most-specific-wins caveat in the support bundle is moot unless this ruling is ever reversed.

Filtering is expressible in two places and they are not two spellings of one thing:

Half of this already exists in the data model: CollectorConfig carries a per-destination Filter with Tenants/Stacks/Environments allowlists (AND’d, empty meaning all) plus a per-signal SignalConfig for logs, traces and metrics. What does not exist is content-level filtering — no severity floors, no trace sampling, no attribute rules. The fabric wants both stages.

A per-provider filter can only NARROW, never WIDEN. If the top-level filter dropped a record at source, the record does not exist, so otel.severity: warn globally plus providers.debug-sink.severity: debug yields nothing extra on the debug sink. This is a natural consequence of the pipeline order and a natural mistake for an operator to make; the implementation should document it and ideally validate-and-warn on the combination.

They compose, and confusing them is the main documentation hazard:

1. Config scope = whose telemetry. otel.severity: warn at tenant acme affects acme’s signals only.
2. Top-level versus per-provider = which pipeline stage. Drop-once versus per-branch view.
3. The rule itself = what is matched.

Filter policy is scoped config like everything else — the scope of the row is what it filters. Cranking one chatty app down is otel.filter.severity: warn at that stack’s scope. One more entry in the existing grammar, not a new mechanism.

Filtering before the provider is also a money decision, not only a noise decision: a paid observability tier bills on ingestion, so at-source filtering is the lever, and trace sampling is the biggest one. For third-party destinations it is also data minimization.

Container logs have no parsed severity, by ruling, so severity is not a uniform axis across all signals and the documentation must not pretend otherwise:

The corpus is why this is per-app rather than a shared vocabulary: valkey signals its level with one glyph, zerolog uses INF/WRN/ERR, NestJS uses VERBOSE and LOG, and even a curated vocabulary only reached 79.5% of lines. There is no honest way to compile otel.severity: warn into a container- log rule that means the same thing for every app.

So T1’s severity knob applies to SDK-borne signals, and quieting a chatty container is a T2-shaped rule scoped at that stack — still one entry in the existing scoped-config grammar, still no new mechanism, but a literal match, not a severity floor. Note the cost direction: at-source dropping for container logs now needs a rule per noisy app instead of one estate-wide knob.

Never expose raw collector YAML. It leaks the implementation, it lets a config error break the pipeline, and — decisively — it lets an operator delete the scrub stage. SHC owns the collector config and compiles intent into YAML, the same generator pattern already used for Traefik’s dynamic config.

• Filters DROP; the scrub REDACTS. A filter must never be the security mechanism. An operator-tunable “drop the secrets” rule is a footgun — it fails open the moment someone edits it.
• The compiled pipeline ALWAYS contains the scrub stage, whatever the user config says. Non-removable by construction, in the same way the airgapped telemetry gate is non-overridable.

With scoped routing built, there would be two ways to express one outcome:

• otel.provider = acme-oo at tenant scope — config-scope-is-the-rule.
• providers.acme-oo.scope.tenants = [acme] — the existing per-destination ScopeFilter allowlist.

Two mechanisms with one visible outcome is how operators end up setting both, inconsistently.

Recommendation: scope-routing is canonical for where data goes. The provider ScopeFilter is demoted to a defensive assertion — “this destination must NEVER receive anything but acme, regardless of what the routing config says”. That is a genuine compliance guarantee and a useful hard wall, but it stops being the normal steering tool and the documentation should say so in those words.

shc support bundle is local, inspectable, and never auto-uploaded — the customer generates it, reads it, and chooses to send it. That is load-bearing: scrubbing freeform text is never 100%, so a human review step is part of the design, not a courtesy.

This is the post-fabric bundle. It is included here rather than in its own document because it is the strongest argument for building the fabric: today the observability store holds no container logs, so “just query the store” is not yet possible.

shc support bundle --scope <tenant/stack/env> --since <range> produces:

1. A scoped observability dump — the bulk, and the default. Logs, metrics and traces over (date range × scope). Because the fabric routes everything — the daemon’s own signals, container logs, app traces and metrics — there is no separate journald or docker-log collection; it is already there. Clean by construction (scrubbed at the collector), and exported in an ingestable form so support can replay it into a throwaway OpenObserve — the same interface the customer is looking at. This is the time-series half: what happened.
2. A no-ref config render. Every in-scope deployment re-templated in a bundle mode that skips the vault-ref resolution pass entirely (and skips autogen resolution). The templating runs and produces the full faithful compose — all structure, all configuration — with every secret left as its literal ref+vault://… pointer. The plaintext is never produced, so there is nothing to mask and nothing to leak. A vault ref is a POINTER, not a secret, and seeing which key a variable points at is diagnostic. Never collect the on-disk resolved compose or docker inspect environments — those hold plaintext; always re-render fresh in no-ref mode.
3. A point-in-time state snapshot — what IS, now: deployment rows; every container’s state, health, restart count, uptime, CPU and memory, and running image ref plus digest (drift against the pin is diagnostic gold); per-node CPU cores, memory, disk and load; per stack, tenant and estate aggregates including capacity against the licence; the health rollup; versions; licence status; cluster and raft state; recent coded errors.

Part of the state snapshot, and cheap enough to include in --quick. Metadata only — never keys. The public half of a certificate is not a secret, and its metadata answers the questions that otherwise cost an hour of back-and-forth. For every certificate:

• Subject and the full issuer chain with fingerprints — which instantly distinguishes LE-staging from LE-production from internal from self-signed, a question that has cost real time on live estates.
• not-before / not-after and days remaining.
• SANs, serial, fingerprint.
• Where it lives — host, node, purpose.

Covering: edge/LE certificates per ingress host; the internal cluster CA and per-node mTLS leaves (core/paths CertsDir(), /var/lib/shc/certs on a root Linux install); the Nebula CA and node certificates (a separate tree — shc/modules/nebula, /var/lib/shc/nebula/{ca.crt,host.crt}); SSO signing certificates; and the ACME account and configuration state, so which caServer is in use is visible at a glance. Fingerprints compared across nodes give split-brain-trust detection for free. This is a strong candidate for a future shc certs status operator command in its own right.

The licence file itself (shc/modules/license/store.go, /var/lib/shc/license/license.lic, plus the overage file beside it) is metadata for this purpose too — its status belongs in the snapshot, its bytes do not.

Never read, under any flag: private keys of any kind — edge, mTLS, the cluster CA, and Nebula’s ca.key and host.key — the vault store and its password, and SSH keys. Mesh IPs, node IDs and raft addresses are mask-optional (--redact-topology), not hard-excluded — they are usually needed to debug a cluster.

Raw application container logs stay behind an explicit opt-in with a loud warning even after the fabric ships, wherever the bundle would reach past SHC’s own signal: they are the customer’s arbitrary output and may hold their end users’ data.

• The fabric must ship first. Without container-log collection the store does not hold what the bundle promises.
• Scoped routing changes where to look. If a tenant is routed to their own destination under most-specific-wins, the master store will not have their data — the bundle must query the destination serving that scope, resolved from the scoped-provider config, not blindly the master store.
• Retention bounds the dump. It reaches exactly as far back as the destination retained, and the bundle should say so rather than silently returning less.
• Exited and one-shot containers are in scope, not an edge case. The corpus census found the estates’ single worst credential leak in an exited migration container — a bundle that enumerates running containers would have missed it entirely, and so would a collector that only tails what is up. Jobs are containers. See The scrub.

Run a read-only secret-surface audit before building the scrubber. What the logger actually emits, real deployment-directory contents, exact docker inspect exposure, every @secret and autogen provenance path, and every certificate, key, Nebula and vault-password location on a live node → an authoritative include/mask/exclude spec with tests → then build. Building a redactor from memory is how a token leaks.

The corpus is a down payment on that audit, not a substitute for it. It supplies measured shape and priorities — six consequences with a finding behind each — over one population (36 container streams on two estates). The audit still owes the rest: what the daemon’s own logger emits, deployment-directory contents, docker inspect exposure, and every key location on a live node.

Distinct from everything above, sharing only the word. Recorded here so the two are not conflated:

• One config key, on by default, opt-out: telemetry.enabled: true plus SHC_TELEMETRY=off, bundling product analytics (version, feature and catalog usage, aggregate error-code counts) with error reporting. No payloads, no config values, no domains.
• Genuinely anonymous, and that property is load-bearing. A random install ID with no path back to the licence identity, sent on a separate channel from the licence reverify. SHC already knows the estate identity from the licence domain, so if telemetry rode that channel or keyed to it, “anonymous” would be false — and the licence channel’s promise (it sends the billing meter and nothing else) has to stay literally true.
• A loud one-time first-run notice, shc telemetry status showing exactly what is sent, and a trivial documented opt-out, named in the tier docs.
• Airgapped licence means telemetry is ALWAYS OFF and never even attempted. Gated on licence type before any outbound activity — no connection, no DNS lookup, because airgapped egress monitoring flags an attempt as an incident. Non-overridable: licence type beats telemetry.enabled: true. Same gate the licence reverify already takes under license.offline.

Setting the default before the first sale is the honest moment; flipping it on after release would not be.

Each slice is independently provable, in the shape network-fabric.md uses.

1. Slice 1 — container logs land, single destination. The ocb distro with the filelog receiver (container operator, recombine, ANSI strip — no grammar parsing), the custom scope-enrichment processor, and an OTLP exporter to the existing managed destination. No scoped routing, no operator filtering. Proof: a container’s stdout is queryable by (tenant, stack, environment) from the control plane; a 40-frame stack trace arrives as one record rather than 40; a coloured winston line arrives free of escape sequences; and a job’s output survives the job, including a container that has already exited.
2. Slice 2 — the scrub. Gated on the secret-surface audit, shaped by the corpus census. The redaction processor, always present in the compiled pipeline, with tests asserting it cannot be configured away. Proof: replaying the corpus’s own five offending files through the pipeline redacts all 18 occurrences — including the ALL_CAPS=<hex> migration-container case that matches no conventional secret regex — and the false-positive rate on the other 31 files is measured and reported, not assumed.
3. Slice 3 — node and container metrics. hostmetricsreceiver and dockerstatsreceiver. Proof: the support bundle’s state view can be drawn as a time-series, not only as a snapshot.
4. Slice 4 — operator filtering, T1 then T2. Intent compiled to collector YAML, scoped like every other config row. Proof: a chatty stack quieted at stack scope with nothing else affected; a raw OTTL rule accepted; the narrow-only rule warned about.
5. Slice 5 — scoped routing. The A–F program minus step C — precedence is ruled (fan-out, 2026-07-31), so the emit-path rewrite is not built and the existing four-tier walk is proven rather than replaced. Proof: a tenant-scope provider row sends that tenant’s data to a second destination — and, under the fan-out ruling, still to the master default.
6. Slice 6 — otel=OTEL. The app-facing integration for traces and metrics, riding the same enrich/scrub/filter/route pipeline. Proof: an app’s spans arrive scope-stamped without the app knowing what a tenant is. BUILT (2026-07-31): the token + render-time injection in ce (modules/app/integration_otel.go, inject_otel.go), the OTLP receiver and traces/metrics/otlp pipelines in core/collectorcfg, the shc.instance join-key stamping in collector/processor/shcscopeprocessor — see the mechanism. The proof runs hermetically in tests/collector (TestOTLPAppTelemetryEndToEnd): the posted payload names no scope word, arrives stamped, and /v1/logs is refused. Owed: the deployment seam landed (2026-07-31) without the OTLP flip — modules/collector/desired.go never sets Params.OTLP, so the app-side env is inert-but-harmless until someone flips it AND publishes port 4318 on the docker bridge (collector/README.md, “Still owed by the seam”); the live proof (an app’s span queryable by scope in the managed store from a real -i otel=OTEL install) is deferred with it.
7. Slice 7 — the support bundle. Scoped store query, no-ref render, state snapshot, certificate inventory. Proof: a bundle from a live estate replays into a throwaway OpenObserve and contains no plaintext secret.

Slices 1–3 are useful on their own and unblock the support bundle’s hardest dependency. Slices 1 and 3 are BUILT AND DEPLOYED, slice 6 is BUILT (2026-07-31): the collector distro and its scope-enrichment processor live at shc/collector/, the config renderer at shc/core/collectorcfg, the image bakes as collector in shc/docker/docker-bake.hcl, and the proofs run as shc/tests/collector (make -C collector test/e2e) — synthetic fixtures only, the corpus stays out of git.

The per-node deployment wiring landed the same day. The collector is a daemon-managed node-local workload, not a fifth system app: system stacks are manager-pinned singletons and the codebase has no “run this container on every node” primitive, whereas the collector is per-node by physics. It ships as the collector_node subsystem (shc/subsystems, ModeAny, not leader-only — the per-node, not leader-only position, enforced by a test) driving the desired-state compiler and converger in shc/modules/collector. Every seam it consumes was already exported, so the slice needed zero ce changes. The rendered config is written 0600 (it carries the export credential) outside the backup-captured deployments/ tree; union trust arrives as the /etc/shc/tls.ca platform mount, without which an LE-staging estate drops 100% of telemetry; and the SHC_HOST physical-dial answer is baked into extra_hosts once per render, since a container cannot consult the daemon’s per-connection rewriter. See shc/collector/README.md.

The airgapped hard-off applies to the fabric too, and it gates the export, not the collector: under license.offline the pipeline terminates in nop — it still collects and still stamps scope, and emits nothing at all, so an airgapped estate’s egress monitoring never sees an attempt. Non-overridable: the gate is consulted before any provider config, so an explicit otel.provider cannot re-enable egress.

Slice 5 is still the cross-repo program, but its precedence question is answered — fan-out, so the expensive emit-path rewrite is off its critical path.

• Cluster network fabric (design) — the sibling proposal this document is shaped after
• Configuration files and the scope ladder
• Configuration keys — otel
• OpenTelemetry Collector Builder (ocb) — manifest-driven custom distros
• OTTL (OpenTelemetry Transformation Language) — the expression language behind filterprocessor and transformprocessor
• k8sattributesprocessor — the prior art the SHC scope-enrichment processor is modeled on
• Drain (log template mining) — Grafana Loki ships a Go implementation used for pattern detection
• SHC’s own Python-era parser — shc/logparse.py and shc/modules/logs/, last live at c7abf2114^ on the backup/pre-purge-main branch

Not in this repository, and not in any repository. They live on the development box that produced them:

• /var/tmp/logbench — the harness. A standalone Go module: 12 candidate parsers, the independent quality oracle, the microbenchmarks behind the RE2 figures. Nothing was added to this repo or to ce.
• /var/tmp/logbench/REPORT.md — the full report. Master table, per-file cost and quality, eight numbered findings, per-node projections, caveats, and the commands to reproduce every number quoted above.
• /var/tmp/logcorpus — the corpus. 36 files, 28,657 lines, 23 container images, both live estates. Mode 700, contains live credentials, and must never be committed — it is the evidence behind The scrub precisely because it is unsanitized.