From b33e389d83e4c84b58a220e8e4203644c4236e57 Mon Sep 17 00:00:00 2001 From: Richard Taylor Date: Wed, 13 May 2026 16:36:36 +0100 Subject: [PATCH] DownloadManager: document redact_url kwarg for auth-bearing URLs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Companion to MicroPythonOS#136 (adds the kwarg to DownloadManager.download_url). Adds a new "Redacting Sensitive URLs" section explaining when callers should opt in, what gets redacted (URL path/query, response-headers, exception messages — host kept for failure triage), and why the default is False (preserves useful debug output for public-URL callers). Updates the parameters table with the new kwarg. Co-Authored-By: Claude Opus 4.7 (1M context) --- docs/frameworks/download-manager.md | 31 ++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-) diff --git a/docs/frameworks/download-manager.md b/docs/frameworks/download-manager.md index ab9b0ab..cb3010f 100644 --- a/docs/frameworks/download-manager.md +++ b/docs/frameworks/download-manager.md @@ -85,7 +85,7 @@ This means you can use the same API in both async and sync code without any wrap ```python def download_url(url, outfile=None, total_size=None, progress_callback=None, chunk_callback=None, - headers=None, speed_callback=None) + headers=None, speed_callback=None, redact_url=False) ``` **Note:** This method works in both async and sync contexts. When called from an async function, it returns a coroutine. When called from a sync function, it runs synchronously. @@ -101,6 +101,7 @@ def download_url(url, outfile=None, total_size=None, | `chunk_callback` | async function | Callback for streaming chunks (optional) | | `headers` | dict | Custom HTTP headers (optional) | | `speed_callback` | async function | Callback for download speed (optional) | +| `redact_url` | bool | Opt-in: hide the URL in log output when it carries an auth secret in its path or query string. Default `False` keeps existing debug output for public URLs. See [Redacting Sensitive URLs](#redacting-sensitive-urls) below. | **Returns:** - **Memory mode**: `bytes` on success @@ -144,6 +145,34 @@ else: await DownloadManager.download_url(url, outfile=outfile) ``` +## Redacting Sensitive URLs + +`DownloadManager.download_url` prints the request URL three times per download (start, finished, exception path) plus the full response-headers dict — useful debug output for typical callers fetching public URLs (app icons, OS updates, weather data), but it silently leaks any auth secret embedded in the URL. + +Common cases where a URL carries a secret: + +- **API-key-in-URL** auth: `https://api.example.com/v1/data?api_key=ABC123` +- **OAuth-token-in-URL**: `https://service.example.com/resource?access_token=eyJhbGci…` +- **Wallet xpubs** in indexer URLs: `https://btc1.trezor.io/api/v2/xpub/zpub6q…` +- **LNBits readkey** in URL path on some endpoints + +A leaked secret-bearing URL ends up in serial / REPL output, screenshots of debug logs, and anywhere those logs are shared. Pass `redact_url=True` to scrub it: + +```python +await DownloadManager.download_url( + "https://btc1.trezor.io/api/v2/xpub/zpub6q…", + redact_url=True, +) +``` + +When `redact_url=True`: + +- The URL is logged as `scheme://host[:port]/...REDACTED...` (path + query stripped). The host is intentionally kept so failure triage (DNS, connectivity, wrong endpoint) is still possible. +- The response-headers dump is suppressed entirely (``). Response headers often contain `set-cookie`, `cf-ray`, and other tokens that correlate to the secret-bearing request. +- Exception messages have any embedded URL substring scrubbed (aiohttp's `ClientConnectorError` typically embeds the URL). + +Default `False` is deliberate — most callers fetch public URLs and want the full log line for diagnostics. Opt-in is per call. + ## Common Patterns ### Download with Timeout