Skip to content

fuel-prices: restore New Zealand (MBIE) — source moved behind Incapsula JS bot-wall #4010

@koala73

Description

@koala73

Problem

scripts/seed-fuel-prices.mjs fetchNewZealand() can no longer retrieve the MBIE weekly fuel CSV. Around 2026-05-20, mbie.govt.nz moved its entire domain (apex, the weekly-fuel-price-monitoring data page, and the weekly-table.csv asset) behind an Incapsula (Imperva) JS bot-challenge. The URL now returns:

  • HTTP 200, content-type: text/html, ~212 bytes
  • body is an _Incapsula_Resource challenge stub, not CSV

Verified blocked from all of: a residential IP, a datacenter IP, and the Decodo residential proxy. data.govt.nz and figure.nz mirrors are also Incapsula-walled. A plain fetch/curl cannot pass a JS challenge from any IP.

Interim mitigation (shipped in PR #4007)

'New Zealand' added to TOLERATED_FAILURES so the dead source no longer rejects the whole validated fuel-prices publish (same precedent as Brazil). fetchNewZealand() still runs every cycle and NZ is carried automatically the moment the source returns CSV again — so if this is fixed, no code change to the validator is needed.

To actually restore NZ (pick one)

  1. Headless / challenge-solver fetch — integrate FlareSolverr / browserless / a Zyte/ScrapingBee unblocker for fetchNewZealand() (reusable for any future JS-walled source). New infra + dependency + cost.
  2. Alternative source — find a non-walled host that publishes the same MBIE weekly board prices (e.g. an official API, a data.govt.nz CKAN resource that serves the raw CSV without the wall, or another aggregator).
  3. Accept NZ as permanently best-effort like Brazil and close this (data stays absent).

Acceptance

  • fetchNewZealand() returns NZ gasoline + diesel in NZD/L again, and
  • once NZ is reliably present, optionally remove it from TOLERATED_FAILURES to re-tighten the gate.

Context: WM 2026-06-01 DEGRADED triage. See scripts/seed-fuel-prices.mjs TOLERATED_FAILURES comment + fetchNewZealand() (~line 499).

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Medium priority, schedule when capacity allows

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions