fix: resolve shrunk DataMap in data_download#142
Open
mickvandijke wants to merge 3 commits into
Open
Conversation
Large uploads (more than 3 chunks, roughly >12 MiB) produce a shrunk (child) DataMap whose infos() reference wrapper chunks rather than the root content chunks. data_download handed those straight to self_encryption::decrypt, which unshrinks the root map internally but then fails fetching the root content chunks that were never requested, erroring with "Missing chunk: <hex>". Such blobs were unreadable through data_download even though the data was fully intact on the network; the data_upload/data_download primitive pair was asymmetric. Add resolve_root_data_map: when the map is a child, fetch the wrapper chunks and unshrink via self_encryption::get_root_data_map before downloading the root map's content chunks, mirroring how file_download already handles shrinking. Non-child maps are returned unchanged with no network access, so the common flat-map path is untouched. Add e2e regression test test_data_shrunk_datamap_round_trip using a 4 * MAX_CHUNK_SIZE payload that deterministically forces shrinking; confirmed it fails with the exact "Missing chunk" error without the fix. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The prior shrunk-DataMap fix introduced two issues on the public data_download API: - Its doc comment linked to the private resolve_root_data_map via an intra-doc link. cargo doc (run in CI with RUSTDOCFLAGS="-D warnings") rejects a public item linking to a private one, breaking the docs job. Reworded the doc to describe the behavior inline instead. - Resolving a shrunk map bridges self-encryption's synchronous fetcher onto the async network via block_in_place, which panics on a current-thread runtime. data_download thus panicked for large (shrunk) payloads while succeeding for small (flat) ones on the same runtime. A fully-async path isn't possible (self-encryption's per-level decrypt_full_set is pub(crate)), so keep the block_in_place bridge but guard it: ensure_shrunk_resolution_runtime maps a current-thread runtime to a descriptive Error::Config instead of panicking, and the requirement is now documented on the public method (mirroring file_download). Add unit tests covering both guard branches. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mickvandijke
force-pushed
the
fix/data-download-shrunk-datamap
branch
from
99f454c to
f7bacdf
Compare
… DataMap resolve Refines the shrunk-DataMap resolver added earlier in this PR: - Fetch wrapper chunks via `chunk_get_observed` instead of plain `chunk_get`, so the wrapper-chunk fetches feed the adaptive fetch limiter consistently with the content-chunk fetches in the same `data_download` call. Same close-group peer count; only the observation feed is added. - The self-encryption fetcher may only yield `self_encryption::Error`, which previously flattened every resolution failure to `Error::Encryption`. Capture the underlying `ant-core` error out-of-band so a missing wrapper chunk surfaces as `Error::InvalidData` (matching the content-chunk path) and a network failure keeps its `Timeout`/`Network` classification. No change to the successful path or the public API. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Large uploads (more than 3 chunks, roughly >12 MiB) produce a shrunk (child)
DataMapwhoseinfos()reference wrapper chunks rather than the root content chunks.data_downloadhanded those straight toself_encryption::decrypt, which unshrinks the root map internally but then fails fetching the root content chunks that were never requested, erroring withMissing chunk: <hex>.The result: any blob large enough to shrink was unreadable through
data_downloadeven though the data was fully intact on the network — thedata_upload/data_downloadprimitive pair was asymmetric. Small payloads (≤3 chunks, flat map) round-trip fine, which is why this hid in tests.Fix
resolve_root_data_map: when the map is a child, fetch the wrapper chunks and unshrink viaself_encryption::get_root_data_mapbefore downloading the root map's content chunks — mirroring howfile_downloadalready handles shrinking.Test plan
test_data_shrunk_datamap_round_tripuploads a4 * MAX_CHUNK_SIZEpayload that deterministically forces shrinking (assertsis_child()), then verifies a full byte-for-byte round-trip.Missing chunk: <hex>error, and passes with it.cargo clippy --all-targets --all-features -- -D warnings✅ ·cargo fmt --check✅ · fulle2e_datasuite (7 tests) ✅.🤖 Generated with Claude Code