release(0.13.5): perf — cancellable flush sleep + CI hygiene by maltsev-dev · Pull Request #60 · nullrunio/nullrun-sdk-python

maltsev-dev · 2026-07-08T12:19:26Z

release(0.13.5)

This is the 0.13.5 perf release — pairs the LangChain elif-chain
fix from 0.13.4 with a CI/runtime perf pass that the audit revealed.

What's in this PR

Three commits on top of origin/master (note: the 0.13.4
elif-chain fix already landed on master as PR #59, so the matching
c17a582 version-bump commit is intentionally not cherry-
picked — its changelog text was already on master and would have
been a no-op):

6ec7671 — perf(ci): Transport._flush_loop swapped
time.sleep for threading.Event.wait so
runtime.shutdown() returns in ms, not the full
flush_interval (5s default). Pin:
tests/test_transport.py::test_stop_interrupts_flush_sleep
uses 30s flush_interval and asserts stop() < 5s.
84fc9d5 — docs: remove docs/drift.md and
docs/sdk-v3-migration-gaps.md (drift content already in
CHANGELOG.md and the per-release __version__.py headers).
99bc197 — chore(release): bump 0.13.4 → 0.13.5
(pyproject.toml + src/nullrun/__version__.py) with a full
changelog block describing the perf fix + CI hygiene.

CI hygiene (in the perf commit)

setup-python action: cache: pip +
cache-dependency-path: pyproject.toml — saves ~60-90s of
cold install per matrix leg.
strategy.fail-fast: true on the test matrix.
pip install "pytest-xdist>=3.6" + pytest -n auto so the
suite uses all runner cores. xdist also added to
[project.optional-dependencies.dev] for local parity.
pyproject.toml: dropped the global -q from addopts so
CI logs surface the full PASSED line per test;
--tb=short keeps tracebacks compact. -n auto stays in
the workflow (not addopts) so a developer running
pytest tests/test_x.py locally gets a single process.

Wire format

No change. The default FlushConfig is unchanged (5s
flush_interval, 50 batch_size); production flush cadence is
identical. The fix only shortens the worst-case shutdown latency.

Compatibility

Backends on 1.0.0 keep working unchanged. No SDK_MIN_VERSION
bump. Recommended upgrade path: 0.13.4 → 0.13.5.

Validation

test_stop_interrupts_flush_sleep passes in 0.26s (was 30s).
test_legacy_key_warning — 0.19s (was 5.18s).
test_remote_states_race ×4 — no 5.0s teardowns.
test_state_compare_case_insensitive ×10 — all <0.2s.
ruff check clean on changed files.
mypy clean on transport.py.

Branch / commit policy

New branch release/0.13.5 cut from origin/master (not
from a prior release/*, per repo convention).
No Co-Authored-By trailers.
Real user.name/user.email from local git config.

The Transport flush loop used `time.sleep(self.config.flush_interval)` — uncancellable, so any test or process that called `runtime.shutdown()` while the thread was mid-sleep blocked on `thread.join()` for the full default 5s flush_interval. With 1222 tests in the suite and many paths calling shutdown() (or its fixture teardowns), this multiplied into ~10-15 minutes of pure teardown wall-clock per Python in the matrix. Replace the bare sleep with `Event.wait`, which returns the instant `stop()` sets the event. `stop()` now sets the event before `join()`, and `start()` clears it so a restart-after-stop is clean. Pin contract in tests/test_transport.py:: test_stop_interrupts_flush_sleep …uses a 30s flush_interval; pre-fix this took 30s, post-fix <5s. CI hygiene in the same commit so the suite can actually use the freed time: - ci.yml / publish*.yml: enable pip cache (`cache: pip` + `cache-dependency-path: pyproject.toml`) — saves ~60-90s of cold install per matrix leg. - ci.yml: `fail-fast: true` on the matrix — don't burn two more runner legs once one Python leg is red. - ci.yml / coverage / publish*.yml: install `pytest-xdist>=3.6` and pass `-n auto` to pytest. `pytest-xdist` is also added to `[project.optional-dependencies.dev]` so a local `pip install -e .[dev]` brings it in. - pyproject.toml: drop `-q` from `addopts` so CI logs show the full PASSED line per test (`--tb=short` keeps tracebacks compact). `-n auto` stays in the workflow, not the addopts, so a developer running `pytest tests/test_x.py` gets a single process. No public API change. The runtime default FlushConfig is unchanged (5s interval, 50 batch size); production flush cadence is identical. The fix only shortens the worst-case shutdown latency.

Pairs with the preceding release/0.13.5 commits: * perf(ci): cancel flush-thread sleep (transport.py:816) * remove redundant docs (drift.md, sdk-v3-migration-gaps.md) Wire format unchanged; pure version bump + changelog entry covering both the perf fix and the CI hygiene so the SDK_MIN_VERSION floor is up to date. No on-wire breaking change; backends on 1.0.0 keep working unchanged. Recommended upgrade path: 0.13.4 -> 0.13.5.

…ace respx PR #60 landed the cancellable-sleep fix in Transport._flush_loop and expected CI wall-clock to drop to 3-5 minutes. The first green run on PR #60 (PR #60 run #1) actually took 9m 47s — the test step dominated by a retry storm: Request failed (attempt 5/11), retrying in 8.46s: ConnectError Request failed (attempt 6/11), retrying in 9.16s: ConnectError ... Circuit breaker OPEN. Batch of 10 events will be re-queued. Root cause: `tests/conftest.py:reset_runtime` teardown nulled the runtime reference WITHOUT calling `runtime.shutdown()`. The transport flush thread therefore kept running across tests, the buffer drained through httpx with no respx context active, and the xdist workers spent the next 9 minutes retry-sending the buffer against the real (unreachable in CI) backend. `_retry_with_backoff (max_retries=10, max_delay=10s)` is 65s of pure sleep per failed batch, and with 4 xdist workers and many buffered batches this multiplied into 9m 47s — i.e. a CI-noise fix that hid a deeper lifecycle bug. Pre-fix CI was already paying this cost (5s shutdown-sleep × 200+ tests ≈ 17 min of teardown per Python leg); the retry storm was always there but masked by the dominant 5s cost. PR #60's 5s fix exposed it. Fix: add `flush: bool = True` to both `Transport.stop()` and `NullRunRuntime.shutdown()`. When False, the transport thread is cancelled WITHOUT a final `_do_flush()` / `_persist_to_wal()`. `tests/conftest.py:reset_runtime` teardown now calls `inst.shutdown(flush=False)` before nilling the reference. This makes the conftest teardown a true no-op for the buffer — the test that wrote the events is responsible for asserting on what it cared about. The production default (`flush=True`) is preserved, so the `nullrun.shutdown()` audit contract ("drain in-flight events") is unchanged. Pins: * `tests/test_transport.py::test_stop_flush_false_skips_final_flush ` — buffers an event, calls `stop(flush=False)` with no respx active, asserts the call returns in <1s AND the buffer is left untouched. Pre-fix this would have hung for 65s+ on the first retry. * `tests/test_init_contract.py::TestShutdownFlushKwarg:: test_runtime_shutdown_flush_false_skips_final_flush` — same contract at the `NullRunRuntime` level: `shutdown(flush=False )` propagates the `flush=False` flag to `Transport.stop()`. Public API additions: * `Transport.stop(timeout=10.0, flush: bool = True)` — `flush =False` is the new flag. * `NullRunRuntime.shutdown(flush: bool = True)` — propagates. * `nullrun.shutdown(timeout=2.0, flush: bool = True)` — passes `flush` through to the runtime. No on-wire or production behaviour change. CI step is expected to drop from ~9m 47s (PR #60 run #1) to ~30-60s on the next run.

maltsev-dev · 2026-07-08T12:42:31Z

Follow-up commit `a037b3a` — conftest teardown fix

Pushed a 4th commit on top of the original 3 to address the 9m 47s
retry storm that surfaced on the first green run of PR #60 (run
#1, the log you saw in chat).

What was wrong

The first commit (6ec7671) made Transport._flush_loop use
Event.wait instead of time.sleep, so shutdown() returns
fast. But the tests/conftest.py:reset_runtime teardown was
never calling shutdown() — it just nulled the runtime
reference. The transport flush thread therefore kept running across
tests with a non-empty buffer, the next _do_flush raced the
respx context exit, and CI spent 9m 47s doing:

Request failed (attempt 5/11), retrying in 8.46s: ConnectError
Request failed (attempt 6/11), retrying in 9.16s: ConnectError
...
Circuit breaker OPEN. Batch of 10 events will be re-queued.

_retry_with_backoff(max_retries=10, max_delay=10s) is ~65s of
sleep per failed batch; with 4 xdist workers and many buffered
batches it dominated the wall clock.

The fix

Transport.stop(timeout, flush: bool = True) — new flush
flag; when False, skip the final _do_flush() /
_persist_to_wal().
NullRunRuntime.shutdown(flush: bool = True) — propagates.
nullrun.shutdown(timeout, flush: bool = True) — propagates.
tests/conftest.py:reset_runtime teardown now calls
inst.shutdown(flush=False) before nilling the reference.

Production default (flush=True) is unchanged — the public
nullrun.shutdown() audit contract ("drain in-flight events")
still holds. The flush=False path is for the test conftest
only, where the test that wrote the events is responsible for
asserting on what it cared about.

Pins

tests/test_transport.py::test_stop_flush_false_skips_final_flush
tests/test_init_contract.py::TestShutdownFlushKwarg:: test_runtime_shutdown_flush_false_skips_final_flush

Both assert stop(flush=False) / shutdown(flush=False) return
in <1s even with a non-empty buffer and no respx active.

Expected CI delta

From ~9m 47s (PR #60 run #1) down to ~30-60s on the next green run.
The fail-fast matrix + pip cache + -n auto xdist were already
in place from 6ec7671; this commit kills the last noisy
bottleneck.

codecov · 2026-07-08T12:56:10Z

Codecov Report

❌ Patch coverage is 0% with 15 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/nullrun/transport.py	0.00%	10 Missing ⚠️
src/nullrun/__init__.py	0.00%	2 Missing ⚠️
src/nullrun/runtime.py	0.00%	2 Missing ⚠️
src/nullrun/__version__.py	0.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

#61) * perf(ci): cancel flush-thread sleep so shutdown() returns in ms, not 5s The Transport flush loop used `time.sleep(self.config.flush_interval)` — uncancellable, so any test or process that called `runtime.shutdown()` while the thread was mid-sleep blocked on `thread.join()` for the full default 5s flush_interval. With 1222 tests in the suite and many paths calling shutdown() (or its fixture teardowns), this multiplied into ~10-15 minutes of pure teardown wall-clock per Python in the matrix. Replace the bare sleep with `Event.wait`, which returns the instant `stop()` sets the event. `stop()` now sets the event before `join()`, and `start()` clears it so a restart-after-stop is clean. Pin contract in tests/test_transport.py:: test_stop_interrupts_flush_sleep …uses a 30s flush_interval; pre-fix this took 30s, post-fix <5s. CI hygiene in the same commit so the suite can actually use the freed time: - ci.yml / publish*.yml: enable pip cache (`cache: pip` + `cache-dependency-path: pyproject.toml`) — saves ~60-90s of cold install per matrix leg. - ci.yml: `fail-fast: true` on the matrix — don't burn two more runner legs once one Python leg is red. - ci.yml / coverage / publish*.yml: install `pytest-xdist>=3.6` and pass `-n auto` to pytest. `pytest-xdist` is also added to `[project.optional-dependencies.dev]` so a local `pip install -e .[dev]` brings it in. - pyproject.toml: drop `-q` from `addopts` so CI logs show the full PASSED line per test (`--tb=short` keeps tracebacks compact). `-n auto` stays in the workflow, not the addopts, so a developer running `pytest tests/test_x.py` gets a single process. No public API change. The runtime default FlushConfig is unchanged (5s interval, 50 batch size); production flush cadence is identical. The fix only shortens the worst-case shutdown latency. * remove redundant docs * chore(release): bump version 0.13.4 -> 0.13.5 Pairs with the preceding release/0.13.5 commits: * perf(ci): cancel flush-thread sleep (transport.py:816) * remove redundant docs (drift.md, sdk-v3-migration-gaps.md) Wire format unchanged; pure version bump + changelog entry covering both the perf fix and the CI hygiene so the SDK_MIN_VERSION floor is up to date. No on-wire breaking change; backends on 1.0.0 keep working unchanged. Recommended upgrade path: 0.13.4 -> 0.13.5. * fix(tests): stop transport flush thread between tests so it doesn't race respx PR #60 landed the cancellable-sleep fix in Transport._flush_loop and expected CI wall-clock to drop to 3-5 minutes. The first green run on PR #60 (PR #60 run #1) actually took 9m 47s — the test step dominated by a retry storm: Request failed (attempt 5/11), retrying in 8.46s: ConnectError Request failed (attempt 6/11), retrying in 9.16s: ConnectError ... Circuit breaker OPEN. Batch of 10 events will be re-queued. Root cause: `tests/conftest.py:reset_runtime` teardown nulled the runtime reference WITHOUT calling `runtime.shutdown()`. The transport flush thread therefore kept running across tests, the buffer drained through httpx with no respx context active, and the xdist workers spent the next 9 minutes retry-sending the buffer against the real (unreachable in CI) backend. `_retry_with_backoff (max_retries=10, max_delay=10s)` is 65s of pure sleep per failed batch, and with 4 xdist workers and many buffered batches this multiplied into 9m 47s — i.e. a CI-noise fix that hid a deeper lifecycle bug. Pre-fix CI was already paying this cost (5s shutdown-sleep × 200+ tests ≈ 17 min of teardown per Python leg); the retry storm was always there but masked by the dominant 5s cost. PR #60's 5s fix exposed it. Fix: add `flush: bool = True` to both `Transport.stop()` and `NullRunRuntime.shutdown()`. When False, the transport thread is cancelled WITHOUT a final `_do_flush()` / `_persist_to_wal()`. `tests/conftest.py:reset_runtime` teardown now calls `inst.shutdown(flush=False)` before nilling the reference. This makes the conftest teardown a true no-op for the buffer — the test that wrote the events is responsible for asserting on what it cared about. The production default (`flush=True`) is preserved, so the `nullrun.shutdown()` audit contract ("drain in-flight events") is unchanged. Pins: * `tests/test_transport.py::test_stop_flush_false_skips_final_flush ` — buffers an event, calls `stop(flush=False)` with no respx active, asserts the call returns in <1s AND the buffer is left untouched. Pre-fix this would have hung for 65s+ on the first retry. * `tests/test_init_contract.py::TestShutdownFlushKwarg:: test_runtime_shutdown_flush_false_skips_final_flush` — same contract at the `NullRunRuntime` level: `shutdown(flush=False )` propagates the `flush=False` flag to `Transport.stop()`. Public API additions: * `Transport.stop(timeout=10.0, flush: bool = True)` — `flush =False` is the new flag. * `NullRunRuntime.shutdown(flush: bool = True)` — propagates. * `nullrun.shutdown(timeout=2.0, flush: bool = True)` — passes `flush` through to the runtime. No on-wire or production behaviour change. CI step is expected to drop from ~9m 47s (PR #60 run #1) to ~30-60s on the next run. * fix(langgraph): attach LLM spans to parent chain via callback run_id Sprint 2026-07-12 (multi-agent span attachment). Previously on_llm_end called runtime.track() with no trace context, so the runtime's _enrich_event generated a FRESH trace_id for every LLM call. The downstream effect on multi-agent / reflection flows was 4/5 empty rows in the workflow detail 'Recent executions' panel: https://nullrun.io/control-center/workflows/<id> ┌────────────────────────────────────────────────┐ │ 1cf7f505-… trace: 1cf7 cost: /usr/bin/bash.00 │ ← orchestration span only │ c4be95fe-… trace: c4be cost: /usr/bin/bash.00 │ ← orchestration span only │ 9295df0f-… trace: 9295 cost: /usr/bin/bash.00 │ ← orchestration span only │ 019f5060-… trace: 019f cost: $0.00013 ✓ │ ← cost_events orphan, by luck └────────────────────────────────────────────────┘ The cost_summary LEFT JOIN in db/mod.rs::get_execution_records_* keyed on cs.join_kind='trace_id' AND cs.join_id=u.execution_id and the orchestration spans' trace_ids never matched any cost_events row because every LLM call wrote under a brand-new trace_id. Fix: - on_llm_start now opens a child span from the active chain (looked up by parent_run_id) or the contextvar-set parent, mirrors the existing on_chain_* pattern. Stores the SpanContext under the LangChain run_id key. - on_llm_end looks up that span, threads trace_id / span_id / parent_span_id / depth / parent_trace_id (alias for trace_id since SpanContext invariants make them identical) into the cost event dict BEFORE runtime.track(). _enrich_event's 'if X not in enriched: generate fresh' checks skip already-set values, so the parent chain's trace_id survives onto the wire. - finally: emits span_end via _end_run so the dashboard sees both span_start and span_end for the LLM span, even if the cost-event path raised. Backward compatibility: - LangChain builds that omit run_id fall through to legacy behaviour (fresh trace_id per event). Tested by test_on_llm_without_run_id_is_silent_no_op. - Pre-existing cost_events rows (older SDKs without span attachment) keep their own fresh trace_ids; the new unified SELECT arm on the backend will JOIN via parent_trace_id (NULL for legacy rows) and via trace_id for new rows, so the dashboard migrates incrementally. Wire contract: - Old backends that strip parent_trace_id at the wire boundary are unaffected (the field is unknown but harmless). - New backends write it to cost_events.parent_trace_id once the migration that adds the column ships (matching change in breaker-core/master). Tests (test_langgraph_callback.py): - test_on_llm_start_then_end_attaches_parent_chain_trace_id: - chain span root depth=0 (parent_run_id chain-1) - LLM span child depth>=1, span_kind=llm, parent_span_id matches chain span_id - cost event trace_id == chain trace_id (the contract) - parent_trace_id on cost event == chain trace_id (alias) - span_start + span_end both fire around the cost event - test_on_llm_without_run_id_is_silent_no_op: legacy LangChain path doesn't crash, no spans opened, cost event fallback - test_on_llm_end_emits_span_end_even_if_track_raises: finally block guarantees cleanup on backend errors 42/42 langgraph tests pass after the change (was 39 before). * chore(release): 0.13.6 — multi-agent span attachment (parent_trace_id) Bump __version__ to 0.13.6 and add changelog entry covering the new on_llm_start / on_llm_end parent-span attach behavior (commit efff530 on this branch). No public API change. Wire format: backward-compatible. The new parent_trace_id field is serde(default) absent on older SDKs and ignored by older backends. Operators upgrading from 0.13.5 must upgrade both sides together (SDK to 0.13.6 + backend with migration 217); the SDK alone still works on 1.0.0 backends. Recommended upgrade path: 0.13.5 -> 0.13.6. SDK_MIN_VERSION_FOR_V3 unchanged (0.12.0).

maltsev-dev added 4 commits July 8, 2026 16:18

remove redundant docs

84fc9d5

maltsev-dev merged commit f79312c into master Jul 8, 2026
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

release(0.13.5): perf — cancellable flush sleep + CI hygiene#60

release(0.13.5): perf — cancellable flush sleep + CI hygiene#60
maltsev-dev merged 4 commits into
masterfrom
release/0.13.5

maltsev-dev commented Jul 8, 2026

Uh oh!

maltsev-dev commented Jul 8, 2026

Uh oh!

Uh oh!

codecov Bot commented Jul 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

maltsev-dev commented Jul 8, 2026

release(0.13.5)

What's in this PR

CI hygiene (in the perf commit)

Wire format

Compatibility

Validation

Branch / commit policy

Uh oh!

maltsev-dev commented Jul 8, 2026

Follow-up commit a037b3a — conftest teardown fix

What was wrong

The fix

Pins

Expected CI delta

Uh oh!

Uh oh!

codecov Bot commented Jul 8, 2026

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Follow-up commit `a037b3a` — conftest teardown fix