feat(node-removal): online node removal as inverse of cluster expansion

[schmidt-scaled]

Summary

Implements an online storage-node removal function that works as the inverse of cluster expansion (add_node). It is background, task-driven, idempotent and resumable — sn remove validates preconditions then queues an FN_NODE_REMOVAL task; the new tasks_runner_node_removal service drives the flow:

shutdown → in_removal → rewire LVS replicas → remove/fail/migrate devices → removed

This replaces the old remove_storage_node semantics (which only handled an already-offline node and did no LVS rewiring).

Behavior

Preconditions (enforced before anything is queued):

• target node is online;
• every other non-removed node is online;
• FTT headroom allows losing the node (_check_ftt_allows_node_removal);
• the node hosts no LVols and no snapshots (operator migrates those separately, at a higher level);
• every secondary/tertiary replica the node hosts for other primaries has a host-disjoint relocation target (catches e.g. 2-node clusters).

LVS rewiring:

• Case A — the node's own primary LVS: tear down its (empty) secondary/tertiary replicas on the peers that host them, clear cross-references.
• Case B — replicas this node hosts for other primaries: re-host each on a fresh, anti-affinity-valid node (get_secondary_nodes / get_secondary_nodes_2 + recreate_lvstore_on_non_leader) so those primaries keep their fault tolerance. Bookkeeping back-ref on the removed node is cleared only after a successful rebuild, so retries resume cleanly.

Devices: each data device is driven online → removed → failed (queuing failure-migration on the surviving online nodes), then the flow waits for failed_and_migrated before flipping the node to removed.

Changes

• models/base_model.py: new STATUS_IN_REMOVAL (code 13)
• models/job_schedule.py: new FN_NODE_REMOVAL
• controllers/tasks_controller.py: add_node_removal_task + dedup branch + get_active_node_removal_task; skip IN_REMOVAL nodes when fanning out device-migration tasks (their SPDK is dead)
• storage_node_ops.py: rewrite remove_storage_node (validate + queue) + node_removal_orchestrate + Case A/B + device-decommission/finalize helpers
• services/tasks_runner_node_removal.py: new runner service (lease-aware, suspend-and-revisit for the long migration wait)
• scripts/docker-compose-swarm.yml: register TasksNodeRemovalRunner
• simplyblock_cli/cli.py: updated sn remove help; --force-remove now only cancels active tasks
• tests/test_node_removal.py: 21 unit tests

Testing

• New: tests/test_node_removal.py — 21 tests (preconditions, relocation feasibility/picking, Case A/B bookkeeping incl. idempotency/resume, device completion gate, status mapping). ✅
• Full non-migration suite: 1137 passed, 213 skipped, no regressions.

Notes / review focus

• Case B (recreating a replica on a fresh node while the cluster is online) is the net-new, highest-risk piece — it reuses recreate_lvstore_on_non_leader with the primary as the online leader. Worth a careful look from someone close to the LVS restart/recreate machinery.
• Unit tests mock all data-plane RPCs; this has not yet been validated on a live cluster.

🤖 Generated with Claude Code

[mxsrc]

The new functions that are being introduced use return-value based error reporting, since they are new I feel they should use exceptions for the error handling. Other than that I think the code looks good, I just have a few comments on two locations. It requires changes in the helm-chart to add the container there as well.

[mxsrc]

I think this should be moved to the task module, including its helpers. The tasks should be as self-contained as possible, this code should not be executed from anywhere else.