Problem
scripts/seed-fuel-prices.mjs fetchNewZealand() can no longer retrieve the MBIE weekly fuel CSV. Around 2026-05-20, mbie.govt.nz moved its entire domain (apex, the weekly-fuel-price-monitoring data page, and the weekly-table.csv asset) behind an Incapsula (Imperva) JS bot-challenge. The URL now returns:
- HTTP
200, content-type: text/html, ~212 bytes
- body is an
_Incapsula_Resource challenge stub, not CSV
Verified blocked from all of: a residential IP, a datacenter IP, and the Decodo residential proxy. data.govt.nz and figure.nz mirrors are also Incapsula-walled. A plain fetch/curl cannot pass a JS challenge from any IP.
Interim mitigation (shipped in PR #4007)
'New Zealand' added to TOLERATED_FAILURES so the dead source no longer rejects the whole validated fuel-prices publish (same precedent as Brazil). fetchNewZealand() still runs every cycle and NZ is carried automatically the moment the source returns CSV again — so if this is fixed, no code change to the validator is needed.
To actually restore NZ (pick one)
- Headless / challenge-solver fetch — integrate FlareSolverr / browserless / a Zyte/ScrapingBee unblocker for
fetchNewZealand() (reusable for any future JS-walled source). New infra + dependency + cost.
- Alternative source — find a non-walled host that publishes the same MBIE weekly board prices (e.g. an official API, a data.govt.nz CKAN resource that serves the raw CSV without the wall, or another aggregator).
- Accept NZ as permanently best-effort like Brazil and close this (data stays absent).
Acceptance
fetchNewZealand() returns NZ gasoline + diesel in NZD/L again, and
- once NZ is reliably present, optionally remove it from
TOLERATED_FAILURES to re-tighten the gate.
Context: WM 2026-06-01 DEGRADED triage. See scripts/seed-fuel-prices.mjs TOLERATED_FAILURES comment + fetchNewZealand() (~line 499).
Problem
scripts/seed-fuel-prices.mjsfetchNewZealand()can no longer retrieve the MBIE weekly fuel CSV. Around 2026-05-20,mbie.govt.nzmoved its entire domain (apex, the weekly-fuel-price-monitoring data page, and theweekly-table.csvasset) behind an Incapsula (Imperva) JS bot-challenge. The URL now returns:200,content-type: text/html, ~212 bytes_Incapsula_Resourcechallenge stub, not CSVVerified blocked from all of: a residential IP, a datacenter IP, and the Decodo residential proxy.
data.govt.nzandfigure.nzmirrors are also Incapsula-walled. A plainfetch/curlcannot pass a JS challenge from any IP.Interim mitigation (shipped in PR #4007)
'New Zealand'added toTOLERATED_FAILURESso the dead source no longer rejects the whole validated fuel-prices publish (same precedent as Brazil).fetchNewZealand()still runs every cycle and NZ is carried automatically the moment the source returns CSV again — so if this is fixed, no code change to the validator is needed.To actually restore NZ (pick one)
fetchNewZealand()(reusable for any future JS-walled source). New infra + dependency + cost.Acceptance
fetchNewZealand()returns NZ gasoline + diesel in NZD/L again, andTOLERATED_FAILURESto re-tighten the gate.Context: WM 2026-06-01 DEGRADED triage. See
scripts/seed-fuel-prices.mjsTOLERATED_FAILUREScomment +fetchNewZealand()(~line 499).