#113 quorum-safe removal v2 — reachability-aware guard + bounded RPCs + recovery hint

LANDED then reshaped; #113 CLOSED. The reachability-aware guard shipped as 207e61370 (EnsureVoterRemovable + the reachability gate, live in internal/modules/cluster). The removal/recovery path was then folded into wedge-safe automatic voter demotion and the clean-slate shc cluster repair recovery sweep (939c59c97, 119dba304); #113 is closed (a0ca64af3). Supersedes the original v1 fix design (db9cf6517, since removed — see git log).

v1 (db9cf6517) added EnsureVoterRemovable counting configured voters (Role==RoleVoter from LeaderInfo.Members()). Live proxmox proved that’s the wrong axis: a force-rejoined node-3 that crashed (its /internal/health :4003 was connection refused) is still a configured voter, so the guard counted it, allowed the removal, and removing node-2 left {node-1, node-3-down} → 1 functional voter of 2 → permanent quorum wedge.

The demote is the wedge point, not the remove. removeFromRaft does DemoteNode(node-2) then RemoveNode(node-2). The demote (cli.Assign → StandBy) commits under the current {n1,n2,n3} majority ({n1,n2}=2-of-3, node-2 acks). After it, the live voter set is {n1, n3-down}; cli.Remove then needs 2-of-2 with node-3 down → blocks until ctx deadline → X500935INTUNH, and the cluster is already wedged. So a guard must refuse before the demote — bounding the RPC cannot un-commit it. The guard MUST live in EnsureVoterRemovable (the only force-path call whose error propagates; the downstream DemoteNode error is swallowed at dqlite_adapters.go:308).

go-dqlite ceiling (confirmed): no follower match-index/last-contact/progress at any layer (request enum closed at RequestWeight=19; Cluster/Describe expose only {ID,Address,Role}/{FailureDomain,Weight}). Upstream’s own roles loop infers liveness via a 2s client.New+Describe reachability probe (app.go:716-758). So a reachability proxy is the strongest available signal — the same ceiling go-dqlite itself accepts. A true caught-up gate would need an upstream go-dqlite feature; OUT OF SCOPE.

NOT “never wedge”. The achievable guarantee: removal never causes a silent permanent wedge — worst case is a clean typed refusal (the deterministic #113 down-voter case) or a fast recoverable failure (the residual reachable-but-stale TOCTOU race), with an operator force-leader recovery path. Mirrors etcd --strict-reconfig-check ∩ MongoDB majority-reachable reconfig (gap vs them: they gate on match-index; we gate on reachability — go-dqlite’s limit).

internal/modules/cluster/service.go EnsureVoterRemovable (the guard called non-bypassably at dqlite_adapters.go:206 before removeFromRaft):

• Keep the existing configured-voter check (X409201CLSRMQ) as the FIRST, cheap gate — covers early-boot / peer-client-unwired where reachability must fail OPEN (no probe transport → fall back to the structural gate only).
• Then, for each surviving Role==RoleVoter member, probe liveness with a short bounded timeout (~2–3s, mirroring upstream’s 2s) over the wired mTLS mesh transport (cluster.CurrentMeshPeerClient() — already used by sibling requestRemoteLeave, dqlite_adapters.go:255) against the registry row’s /internal/health. Match member→row by DqliteID == ParseUint(m.ID) first, fall back to HostOnly(m.Address) == HostOnly(row.InternalAddress).
• Count only reachable survivors. Refuse with a NEW typed code X409202CLSRMR (“refusing to remove voter, {reachable}/{required} reachable voters would remain”) when reachableAfter < floor(votersBefore/2)+1.
• A non-voter / absent target → no-op (unchanged). LeaderInfo/peer-client nil → fail-open to the structural gate (unchanged no-op posture).

A new probe seam (interface) is injected so the unit test can fake it — do NOT hard-call the mesh client from the build-tag-free service.go in a way that breaks the stub lane. Define a small voterReachabilityProbe func(ctx, addr) bool field on ServiceOptions (nil → treat all configured voters as reachable = v1 behaviour, fail-open); the daemon wires it to the mesh-client /internal/health probe.

Thread context.WithTimeout(ctx, ~10s) through removeFromRaft → DemoteNode/RemoveNode and the a.Leader(ctx) dial (internal/modules/cluster/init.go:347,372,342,367; internal/subsystems/impl/dqlite_adapters.go:306-314). protocol.Call honors a deadline when set (today the rm handler passes raw r.Context()). Defense-in-depth only: converts a guard-miss/TOCTOU hang into a fast clean coded error.

When a removal times out post-demote, surface a typed error that NAMES the recovery (“quorum lost after removal; if peers are confirmed down, run shc cluster force-leader on this node”) instead of the bare X500935INTUNH. Do NOT auto-execute the destructive single-node rewrite (split-brain risk without a single-recoverer lease).

Apply the same reachable-voter count to changeNodeRole’s demote guard (service.go quorum-guard block, X409200CLSQUR) so the standalone shc cluster demote path can’t re-open the wedge. NOTE: on the force-REMOVE path the DemoteNode error is swallowed, so P1 is the load-bearing gate; this only closes the direct-demote path.

Add a quorum_removal_test.go case modelling the REAL #113 scenario: a 3-MEMBER config (voters 111, 222=target, 333) where 333 is UNREACHABLE per the injected probe → expect X409202CLSRMR refusal. (Today’s “#113 wedge” case models node-3 as ABSENT — a 2-member config — so it never exercises down-but-configured.) Also assert: all-reachable 3→2 still succeeds; probe-nil fails open to v1 behaviour.

tests/proxmox/test_multinode.bats: gate the node-2 removal on node-3 being BOTH 3-voter AND cluster ping status==ok (new wait_for_node_pingable helper using the shipped shc.cluster.ping endpoint), fail loudly otherwise. On the node-3 rejoin, assert the LEADER’s cluster ping shows node-3 ok. Do NOT add Restart= to the shc-e2e systemd-run units (teardown uses pkill -9 at 7 sites — a restart policy would race the #113 :4002 rebind).

• The reachability probe MUST gate EnsureVoterRemovable (the propagating call), NOT the swallowed DemoteNode.
• Probe counts REACHABLE Role==RoleVoter survivors from the LIVE membership; never the registry cluster_state.
• FAIL OPEN when the probe transport is unwired (early boot / CLI / stub) — never block a removal because we couldn’t probe; the configured-count gate still applies.
• A legitimate all-healthy 3→2 removal MUST still succeed.
• A non-voter / absent target MUST NOT be blocked.
• Do NOT change WithVoters(3), the int64 dqlite_id, or the reconciler.
• Do NOT auto-execute force-leader.
• Bounded RPCs are defense-in-depth — they MUST NOT change the refusal semantics.