SHC Engineering Standards

This document defines the quality bar for the Self-Hosted Cloud (SHC) project. The long-term ambition is to be to self-hosted infrastructure what Linux is to operating systems: a foundation others can build on for decades. That ambition sets the bar.

These are the rules. They apply to the maintainer first, then to every contributor.

The Go rewrite addressed the issues identified in the original Python codebase audit. The standards below now apply to the Go implementation.

The CLI, HTTP API, config schema, and on-disk layout are contracts. Once a contract ships, it cannot be broken without a deprecation period and a compelling reason.

• Every public surface gets a version.
• Every breaking change requires a deprecation cycle of at least one minor release with warnings.
• A compatibility test suite runs in CI and pins behavior, not just signatures.
• “We didn’t mean for anyone to use it that way” is not an excuse. If they’re using it, it’s the contract.

• Broad recover() is forbidden except at well-defined top-level boundaries (CLI entry, HTTP handler, background task runner, RPC transports, callback dispatch loops). At every such site the recover must:
  1. Use a // Broad recover: … comment stating why the breadth is correct.
  2. Either re-panic, surface a typed error to the caller, or log at WARN or above with a typed error code (see 2.16.1).
• Handle the specific errors you intend to handle. Everything else propagates.
• Never return nil as a stand-in for failure without a typed sentinel and a documented contract.
• Background jobs must record structured failure state. “Unknown failed unknown” is a bug.
• defer func() { recover() }() without logging is specifically forbidden. If you don’t know what operation was being attempted, you cannot tell the operator reading the log either.

• If it is not imported, it is deleted.
• If it is legacy, it lives behind a deprecation shim with a removal date, not in a parallel directory.
• shc-old/ must be removed before the next release.
• No commented-out code. Git remembers.

• Functions: target under 50 lines, hard ceiling at 100 except in generated code or parsers.
• Files: target under 1000 lines, hard ceiling at 2000.
• Classes: one responsibility. If you need “and” to describe what a class does, split it.
• Nesting: four levels is a smell, five is a bug.

• Integration tests exercise real Docker, real filesystems, real networks.
• Mocks are allowed only at true external boundaries (paid APIs, nondeterministic third parties).
• Every bug fix ships with a test that would have caught it.
• “The test passes” is not evidence the feature works — evidence is running the feature.

For unit tests that exercise SQL — service methods, CRUD, graph traversal, DB query predicates — use an in-memory SQLite instance with the relevant schema. Each test file typically defines a newTestDB(t *testing.T) *sql.DB helper that creates the database, applies the necessary schema DDL, and registers cleanup via t.Cleanup.

func TestSkipIfPendingTaskExists(t *testing.T) {
    db := newTestDB(t)
    _, err := db.Exec(`INSERT INTO task (name, trigger, status) VALUES (?, ?, ?)`,
        "my_cron_fn", "cron", "pending")
    require.NoError(t, err)
    // ...run production query...
    require.NotNil(t, existing)
}

When to use real DB vs. mocks:

• Use real DB when the test’s correctness depends on the SQL doing the right thing: a WHERE clause filter, an ORDER BY, a UNIQUE constraint, a DELETE/UPDATE affecting specific rows, tenant/env isolation, cycles in a graph, foreign-key cascades.
• Keep mocking at external-system boundaries: Docker/subprocess, restic binary, HTTP webhooks, RPC proxies, external KV providers.
• Don’t test mocks. Testing that a mock returns what you told it to is not a test of production code.

Seed helpers: tests that need related rows (e.g., DeploymentModel needs NodeModel FK) should factor seeding into a private helper inside the test file.

Performance: an in-memory SQLite with the SHC schema costs ~5 ms per test. A test file with 20-30 tests typically runs in under a second.

• Get the types right first. If the types are wrong, the code is wrong.
• Prefer plain data (Go structs, typed constants) over interfaces with methods that just shuffle state.
• One source of truth per piece of data. Duplicated state is a bug waiting to happen.

• Abstract only when you have two concrete call sites that demand it.
• Interfaces with one implementation are deleted.
• Plugin systems used once are inlined.
• “We might want to swap this out later” is not a reason to build a proxy today.

• golangci-lint run must pass with no per-file ignores. If a rule is wrong for the project, disable it globally in .golangci.yml and explain why.
• gofmt -d must produce zero diffs.
• Formatting is automated. Style arguments are not allowed.

• Names explain what. Comments explain why.
• If a comment restates the code, delete it.
• Non-obvious constraints, invariants, workarounds, and references to specific bugs or RFCs are the only reasons to comment.

• No build artifacts, debug dumps, or scratch notes in version control.
• fix.md, report.txt, shc.db move out or get gitignored.
• Source tree reflects the project, not its history.

• “No, rewrite it” is a valid and frequent answer.
• Praise and harshness are both calibrated to the code, not the contributor.
• The code outlives every contributor. Protect it.

SHC runs workloads, holds secrets, and exposes network surface. Security is not a feature — it’s a precondition.

• Threat model is documented. Every subsystem with a network or filesystem boundary declares what it trusts and what it doesn’t. If the threat model isn’t written down, the code hasn’t been designed.
• Secrets never touch disk unencrypted. The vault is the only place secrets live. No .env files in repos, no plaintext in logs, no secrets in error messages.
• Rotation is routine, not exceptional. Every credential has a rotation path. If a credential cannot be rotated without downtime, that’s a bug.
• Principle of least privilege. Containers run non-root by default. Volumes get the tightest permissions that work. Capabilities are dropped unless explicitly needed.
• Dependencies are audited. go.sum is committed. Known-vulnerable versions fail CI (govulncheck).
• Security fixes jump the queue. A CVE in a dependency or a reported vulnerability is a drop-everything event.

• Semantic versioning, strictly. MAJOR breaks contracts, MINOR adds, PATCH fixes. No “we’ll bump major when it feels right.”
• v0.x is the prototype phase. During v0, contracts can still change but require a migration note. After v1.0, breaking changes require a full deprecation cycle.
• LTS branches live for two years minimum. When you cut an LTS, you commit to patching it. If you can’t commit, don’t cut it.
• Release notes are written by the maintainer, not generated. Every release explains what changed, what broke, and what to do about it.
• “Stable” means stable. A release tagged stable has been exercised in production, not just tested in CI.

• Every public CLI command, HTTP endpoint, config key, and extension point is documented before it merges.
• Documentation lives in the repo, next to the code. External wikis drift and die.
• Examples in documentation are tested. Broken examples are worse than no examples.
• A feature without documentation is a feature that doesn’t exist — and cannot be merged.

Infrastructure software that gets slow gets replaced. Performance is a feature.

• Startup: shc cold-start under 500 ms for list/status commands. Measured in CI.
• Memory: core daemon under 200 MB resident at idle.
• Backup throughput: snapshot and restore track disk speed within 20%. Regressions fail the build.
• API latency: p50 under 50 ms, p99 under 500 ms for read endpoints.
• Every subsystem declares its budget. If a subsystem has no budget, it has no contract and can’t be depended on.

Numbers above are starting targets — refine them as the project matures, but once set, regressions block merges.

Every subsystem emits the same shape of signal. Operators should not need to learn a new telemetry dialect per module.

• Structured logs only. JSON or key-value. No printf debugging in main branch.
• Every log at WARN or above carries a typed error code (the existing X200924SYSINS-style codes). Free-text-only warnings are rejected.
• Metrics for every boundary: request counts, latency histograms, error rates. Exposed in a standard format (Prometheus-compatible).
• Traces for every cross-subsystem call. Context propagates through background jobs, not just HTTP requests.
• Log levels mean what they say. ERROR means a human needs to look. WARN means something unusual happened. Nothing routine is WARN or above.

SHC logs through the typed core/log (corelog) wrapper — raw slog.Warn / slog.Error are lint-banned in production paths. The standard shapes are:

logger.Debug("free-text message", "key", value)   // developer diagnostics (slog-style OK below WARN)
logger.Info("free-text message", "key", value)    // routine operator signal
corelog.Warn(ctx, X200003OPFAIL(op, output, err)) // something unusual happened — typed coded error
corelog.Error(ctx, X500907STKFAL(op, err), err)   // a human needs to look — coded error + cause

Guidelines:

• WARN and above take a *coded.CodedError, never a free-text string — the human message lives in the factory’s registered template, and a string argument is a compile error
• ERROR means a human needs to investigate
• WARN means something unusual happened but was handled
• Generic “suppressed exception” messages tell the reader nothing — name the operation that failed
• Every WARN/ERROR therefore carries an X-code from the module’s errors.go by construction

A platform not in the matrix is not supported. A platform in the matrix is tested in CI.

• Host OS: Debian stable, Ubuntu LTS. Others are community best-effort.
• Container runtime: Docker Engine current stable and one previous. Swarm mode required in production.
• Go: the version pinned in go.mod. One version at a time.
• Architecture: amd64 first-class, arm64 supported, others opportunistic.
• Adding a platform is a commitment. Dropping a platform requires a deprecation cycle.

Who decides what. Explicit, not implicit.

• Benevolent dictator model. The maintainer has final say on what merges, what ships, and what gets rejected. This is the model until the project outgrows it.
• Disputes resolve in public. Technical disagreements are hashed out on the issue tracker or mailing list, not in private. The decision and its reasoning are recorded.
• Succession plan. If the maintainer is unavailable for more than 30 days, a named deputy has merge authority for security and critical fixes only. Feature work waits.
• Bus factor is tracked. Every subsystem has at least one other person who has read it and could patch it in an emergency. If only one person understands a subsystem, that’s a risk logged as a bug.
• The model can change. When the project outgrows benevolent dictatorship, the replacement model is proposed, debated in public, and documented here.

Mechanics of getting code in.

• One logical change per patch. Commits are atomic. “Fix X and also refactor Y” is two commits.
• Commit messages follow the kernel style. Subject line imperative, under 72 chars. Body explains why, not what. Reference the issue, bug, or RFC that motivates the change.
• Developer Certificate of Origin. Every commit is signed off (Signed-off-by:). No CLA required, but provenance must be clear.
• Reviews are technical, not social. Reviewers comment on the code. Contributors respond to the comments. Nobody merges their own PR without an approving review, including the maintainer.
• CI is green before review starts. Don’t waste reviewer time on broken patches.

Standards 2.1 and 2.13 say we don’t break contracts. This section says how we prove it.

• Every tagged release gets a compatibility test suite snapshot. The suite exercises the public CLI, HTTP API, and on-disk formats from that version.
• The next release’s CI runs every prior supported version’s compatibility suite. If v1.2 fails the v1.0 compat suite, v1.2 does not ship.
• Migration paths are tested, not inferred. Upgrading from vN to vN+1 is a CI job, not a hope.
• Data format changes are backward-readable. A newer version reads older data. If that’s impossible, a migration tool ships in the release before the format change lands.

SHC moves data. Losing user data is the worst thing this project can do. Given the backup history, this deserves its own wall.

• Destructive operations require explicit confirmation. Restore, wipe, volume delete, stack destroy — all prompt for confirmation by default. Non-interactive mode requires an explicit flag (--yes-really).
• Dry-run is a first-class mode. Every destructive command supports --dry-run and prints exactly what it would do.
• Audit log for destructive ops. Who ran what, when, with what arguments. Persisted outside the data the operation might destroy.
• Backups are verified. A backup that cannot be restored is not a backup. The backup subsystem periodically runs restore-to-scratch to prove the archive is readable.
• Irreversible ops have a cooling-off period where possible. Deleted volumes move to a trash namespace for 24 hours before actual removal.
• Tests for data-loss paths are mandatory. Every code path that can delete or overwrite user data ships with a test that proves the guard holds.

Bugs ship. What happens next is what separates infrastructure software from hobby projects.

• Severity classification. SEV1: data loss or security breach in the wild. SEV2: outage or unrecoverable failure. SEV3: degraded behavior with a workaround. SEV4: cosmetic.
• SEV1 drops everything. The maintainer and all active contributors stop feature work until the fix is shipped and users are notified.
• Postmortem within one week of every SEV1 or SEV2. Public, blameless, and linked from the release notes. Template: what happened, timeline, root cause, contributing factors, what prevented detection, what changes land as a result.
• CVE process. Security bugs get a CVE if they qualify. Disclosure timeline is coordinated with reporters and follows a standard 90-day window unless actively exploited.
• User notification. Breaking incidents are announced on a known channel (release notes, mailing list, security advisory feed). Silence is not an option.
• Every postmortem produces at least one test or rule change. If a bug could happen twice, the postmortem didn’t finish.

Projects die from accretion, not from malice. Things get added and never removed. This section defines when something leaves.

• Subsystems: a subsystem with no active maintainer for two releases, no users (measurable by telemetry or explicit removal of the feature flag), or a critical unfixed bug for more than 90 days is a candidate for deprecation.
• Platforms: a platform whose CI fails repeatedly without a fix, or whose user base drops below a meaningful threshold, moves to best-effort and then to unsupported on a published schedule.
• Features: a feature that can be replaced by a simpler one is deprecated, not left parallel. Two ways to do the same thing is one too many.
• Dependencies: a dependency that is unmaintained upstream for more than a year gets replaced or vendored with explicit ownership.
• Deprecation cycle is public and dated. “Deprecated in v1.4, removed in v1.6.” Not “deprecated eventually.”
• Saying no is the maintainer’s most important skill. Features that don’t fit the project’s scope are rejected early, before they accumulate investment.

Order of operations. Do these before adding new features.

1. Clean up legacy code. Remove any deprecated or orphaned code paths. One commit. Today.
2. Error handling audit. Replace every broad recover() with specific error handling or let it propagate. Add top-level handlers only where needed.
3. Define the public API. Pin the CLI surface, HTTP routes, and config schema. Write the compatibility test suite. Tag a v0 that commits to them.
4. Split the god modules. app/service.go and backup/service.go along existing seams (swarm vs compose, snapshot vs restore).
5. Replace the top 10 most-mocked tests with real integration tests. Real Docker, real volumes, real network.
6. Turn on golangci-lint strict mode and clean all exemptions in CI. Fail the build on violations.
7. Write CONTRIBUTING.md. Point it here. Enforce it on yourself first.
8. Document the threat model and rotation paths for every subsystem that holds credentials or exposes network surface.
9. Write the observability contract and retrofit existing subsystems to emit structured logs, metrics, and traces consistently.
10. Set and publish performance budgets for each subsystem. Wire regression checks into CI.
11. Publish the governance model and succession plan. Name the deputy. Log the bus-factor-one subsystems as bugs.
12. Adopt the contribution workflow. Enable DCO sign-off enforcement. Document the commit message rules. Require reviews even for maintainer PRs.
13. Build the backwards-compatibility test harness. Snapshot the first tagged release’s public surface. Run it against every subsequent release in CI.
14. Harden every destructive code path. Add --dry-run, confirmation prompts, and audit logging. Write the data-loss-path tests.
15. Write the incident response playbook. Define severity levels. Create the postmortem template. Set up the security advisory channel.
16. Draft the deprecation policy. Pick the first subsystem, platform, or feature to deprecate as a test run of the process.

Do these and SHC is at the starting line of Linux-grade discipline. Not there — at the starting line.

The maintainer holds this bar for themselves before holding anyone else to it. Every PR the maintainer writes gets reviewed against this document. Every PR the maintainer merges gets reviewed against this document. No exceptions for convenience, deadlines, or mood.

If the maintainer cannot hold the bar, the bar is wrong and the document gets updated — not quietly ignored.

A codebase grows spaghetti when the same concept gets implemented many ways because no contributor (human or agent) holds the whole model — architectural drift. SHC fights it with three things: one canonical pattern per concern, written down; CI checks that fail the build on regression; and a visible backlog of the deviations not yet normalized.

Before adding a service, a wiring path, an error type, an HTTP client, a scope, or a test fixture: find the existing pattern and reuse it. Do not invent a parallel one. If you genuinely need a new pattern, you change the canonical doc + the fitness check in the same change and say why. “I didn’t know it existed” is the drift this section exists to prevent.

Each is recorded in docs/decisions/ with context, the decision, and consequences:

• ADR-0001 — Pure DI at a composition root (daemon/composition.go), no container. The four transform classes (W1a/W1b/W1c/W2) are named there; injected resolvers are per-call closures, never wire-time captures. Modules/subsystems must never import daemon.
• ADR-0002 — Stack state: the DB = identity/status authority, manifest.json = offline SPEC, docker = observed STATUS, the in-memory map = read-through cache. One authority per role.
• ADR-0003 — One data-ladder core/scope.Scope; auth.AuthScope is the separate RBAC marker.
• ADR-0004 — Hermetic default make test/unit (pure Go, no cgo); live-engine rqlited tests skip unless a rqlited binary is resolvable (SHC_RQLITED_BIN / make vendor/rqlite).
• ADR-0005 — the enforcement mechanism itself.

The cross-module wiring pattern is documented in docs/architecture/patterns/cross-module.md; the package DAG in import-dag.md. (Docs that predate the DI/scope/ent refactors and are known-stale are catalogued in the drift register below — trust the register over an unmarked older doc.)

scripts/arch-fitness.sh (wired into make lint) fails the build on a regression of a converged pattern. ratchet checks may only decrease (a migration lowers the ceiling, never raises it); hard checks must be zero:

To add a pattern: write its rule into arch-fitness.sh, set the baseline, record an ADR, and reference it here.

docs/architecture/DRIFT-REGISTER.md is the prioritized backlog of every code deviation and every stale-doc claim found in the architecture audit, plus the pending fitness rules (greppable checks ready to be promoted into the script as each class is normalized). It is the normalization worklist for the from-scratch convergence.