Node silently stalls at UTC-midnight log rotation: 0-byte new log file, network I/O drops to zero, RSS grows 300 MB -> 6 GB, service still Running

[jordglob]

Summary

An ant-node 0.14.2 Windows service went invisible-dead at exactly 00:00 UTC: the daily log rotated, the new file was created but stayed 0 bytes for 8.5 hours, network throughput dropped to ~0, CPU sat at ~1 %, and RSS climbed monotonically from ~300 MB to 6 GB until we killed it manually. The service manager saw a healthy "Running" service the whole time — unlike the panics we've filed separately (saorsa-labs/x0x#190, WithAutonomi/saorsa-transport#108), this failure mode has no self-healing path and silently takes a node off the network.

We don't have a root-cause-certain diagnosis; we're filing the observations plus a mechanism analysis and are set up to capture a thread dump if it recurs.

Environment & timeline

Host	Windows 10 Pro N for Workstations 19045
Binary	ant-node 0.14.2, WinSW auto-start service, SYSTEM, --enable-logging --log-dir H:\...\logs
Instance start	2026-07-06 20:40:59 UTC (this was the recovery-restart after the JoinHandle abort we filed separately: WithAutonomi/saorsa-transport#108)
Last normal log line	2026-07-06 23:59:23 UTC (routine replication/audit activity, nothing anomalous)
Rotation	new ant-node.2026-07-07.log created 00:00:36 UTC — stays 0 bytes
Observed 08:30–08:40 UTC	UDP sockets still bound; CPU delta 0.22 s per 20 s (~1 %); per-process I/O ≈ 0 over 2 min; RSS 6 036 MB; nothing in service stderr
Manual kill + restart 08:38 UTC	fresh instance immediately healthy (85 MB RSS, 13 peers in 30 s, log growing)

What the 0-byte file proves

The logging setup (from src/bin/ant-node/main.rs) is tracing_appender::rolling Rotation::DAILY wrapped in the default tracing_appender::non_blocking — bounded 128k-line channel, lossy = true. Rotation is lazy/write-triggered: the worker thread creates the new file inside refresh_writer on the first post-midnight write. So:

• at least one log event arrived after midnight (the file exists),
• the appender worker created the file and then never completed a single write in 8.5 h — it stalled between file creation and the first write (or every write failed silently),
• because the channel is lossy, application threads never block on logging — they just silently drop lines. The logging stack can therefore explain the silent part, but not the node-wide stall or the memory: the bounded channel caps out at tens of MB (tracing #2415).

Something else stalled the node at the same moment — the log file is the visible symptom, not the disease.

Mechanism hypothesis (medium confidence)

Best fit for all four data points (midnight timing, 0-byte file, ~0 network with a live process, monotonic RSS growth):

1. A filesystem/volume stall around midnight (classic time for scheduled AV scans/backups/indexing; the log dir and record storage share a volume here) blocks the appender worker on its first write and the storage layer's spawn_blocking I/O.
2. Async tasks awaiting storage pend; packet processing stops (I/O ≈ 0, CPU ≈ 1 % from timers).
3. Timer-driven producers keep pushing into the many unbounded channels in the stack (heaviest concentration in saorsa-transport's nat_traversal_api.rs event/hole-punch/peer-addr channels; also src/node.rs:126, replication/mod.rs:343) whose consumers are stalled → unbounded queue growth → 300 MB → 6 GB. mimalloc's page retention amplifies the reported RSS.

Two code-level observations that make this worth hardening regardless of the trigger:

• main.rs:91-97's own comment acknowledges that blocking work inside poll() "freezes the entire runtime" with few workers — the same freeze reachable via blocked spawn_blocking+storage on 4 workers.
• The unbounded channels turn any prolonged consumer stall into unbounded memory growth. Bounding them (or adding lag metrics) would convert this failure mode from "6 GB zombie" into observable back-pressure.

Related upstream fragility notes, for context: tracing #3131 (rolling-appender concurrency around rotation), tracing #1931 (NonBlocking flush starvation), tracing #3027 (prune_old_logs runs on the worker thread at rotation, does directory scan + deletes — ant-node sets max_log_files, so this path is active at midnight).

What would settle it

We run this node 24/7 and it reproduced once in one midnight so far. If it recurs we'll capture:

• a full thread dump (ProcDump) — expectation under the hypothesis: appender worker and spawn_blocking threads parked in NtWriteFile/NtCreateFile,
• handle count + per-volume disk queue at the time,
• whether the other node on the same volume family stalls simultaneously.

Suggestions for making the node robust to whatever the trigger is: watchdog on own-log liveness (the node can notice "I haven't logged in N minutes" and exit so SCM recovery restarts it), bounded channels with drop metrics, and moving prune_old_logs/rotation I/O off the hot path.

[jordglob]

Correction from the reporter — please read before spending time on this.

Continued investigation invalidated the two central observations:

1. The "0-byte log" was stale NTFS directory metadata for an open file — reading the file content shows continuous, normal-rate logging all night (~2,000 lines/h, no gap at any hour between 00:00 and 08:38 UTC). Rotation worked fine.
2. The 6 GB RSS was WorkingSet dominated by LMDB's memory-mapped DB (private bytes were ~300 MB) — page-cache accounting, not a leak.

What remains is only ~1 % CPU (which we now believe is this node's normal idle profile) and one ambiguous 2-minute I/O sample — no longer credible evidence of a hang. Closing as reporter-error / not reproducible.

Two things may still be of value to the project:

• The NTFS stale-metadata pitfall: anyone monitoring node liveness via log-file size or mtime on Windows will get false positives — the directory entry for an open file is not updated until the handle is flushed/closed. Liveness checks must read the file's tail content.
• The hardening notes about unbounded channels stand on their own merits, independent of this incident.

Apologies for the noise. The three companion issues (#167, WithAutonomi/saorsa-transport#108, saorsa-labs/x0x#190) are unaffected — their evidence is service stderr + WER crash records, not file metadata.