feat(spam-stream): per-signer concurrent sends#590
Open
jelias2 wants to merge 4 commits into
Open
Conversation
drive_stream sent serially (send_one().await per spec), capping throughput at ~one RPC round-trip per tx (~20/s). Replace it with a per-signer worker pool: spawn one worker per pool signer, route each spec to the worker that owns its signer (round-robin by idx, matching make_strict_call's signer selection), and let workers send concurrently. Each signer's sends stay serial within its worker, so nonce assignment and reclaim-on-rejection (reuse the nonce, no gap) remain correct with no shared nonce state or locks. Concurrency == pool size; a pool of 1 reproduces the original serial behavior. Workers build/sign against a shared Arc<TestScenario> with a worker-local nonce (bypassing prepare_tx_request's &mut), and the gas price is shared via an atomic refreshed once per interval. Validated relaying interop executing messages: at a sustained rate above the serial ceiling, cross-chain inclusion latency dropped from p50 80s (queue-bound) to p50 4s with a 32-signer pool.
…igner concurrency Extract the two correctness invariants of the per-signer worker model into pure helpers and unit-test them (no RPC needed): - worker_index(idx, n): round-robin routing; assert idx and idx+n map to the same worker (so the same signer) and that it matches make_strict_call's idx % signers.len() pick — the property that keeps each signer serial. - next_nonce(nonce, submitted): advance on accept, reuse on rejection (no gap). Also assert pool signers are distinct (per-signer workers depend on it). Update docs/stream-mode.md: data-flow now shows the per-signer worker pool (concurrency == --pool-size, pool of 1 = serial) and the reuse table reflects workers building directly with a worker-local nonce instead of prepare_tx_request.
…urrency knob The per-signer concurrent-sends change resolves the last open question (concurrency bounded by pool size). Drop the resolved questions, keep the one real follow-up (Spammer-trait reuse) under Follow-ups, and note in the CLI table that --pool-size sets send concurrency (no new flag).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
spam-stream'sdrive_streamcurrently sends serially —send_one(...).awaitper spec — so throughput is capped at roughly one RPC round-trip per tx (~20/s). When the stream supplies specs faster than that (e.g. relaying interop executing messages), a send queue builds and end-to-end latency balloons.This makes the send path concurrent while keeping nonce handling correct.
Approach: per-signer worker pool
Spawn one worker per pool signer. Each incoming spec is routed (round-robin by
idx, matchingmake_strict_call'sidx % signers.len()selection) to the worker that owns its signer, and workers send concurrently.Why per-signer rather than a semaphore over all sends: a bounced/rejected send must reuse its nonce (otherwise a gap stalls every later tx from that signer). That reclaim is only correct if a given signer's sends are serialized. Pinning each signer to one worker gives exactly that:
Arc<TestScenario>using a worker-local nonce (bypassingprepare_tx_request's&mut), with the gas price shared via an atomic refreshed once per interval.Compatibility
--pool-size 1reproduces the original serial behavior exactly.StreamEventschema are unchanged (the Go-side/reactive consumer correlates byidx, so out-of-order emits are fine).spam_stream.rs.Validation
Relaying interop executing messages at a sustained rate above the serial ceiling, with a 32-signer pool: cross-chain inclusion latency dropped from p50 80s (queue-bound) to p50 4s, with send→receipt p50 ~1.5s and 0 failures. At
--pool-size 1behavior is unchanged.