refactor(pipeline): Fetch all inputs up front, in parallel

[brawer]

Closes #830.

Summary

The pipeline now starts with a single fetch_inputs step that downloads every external input concurrently, before any processing. Every later step works only on local files. This is a refactor: no output changes.

• New src/pipeline/inputs.rs. fetch_inputs runs the AllThePlaces and planet fetches with try_join, each logged as its own sub-step (fetch_inputs.atp, fetch_inputs.osm) with timings. If one fails, the others are cancelled immediately and the run aborts. A cancelled fetch still logs its end snapshot plus a "cancelled" warning, through a new run_step_async that shares its logging with run_step. The runtime is shut down with shutdown_background(), so a cancelled fetch's blocking hash work isn't waited on.
• import_atp / import_osm now take the fetched Input (local path plus metadata) instead of an HTTP client, and no longer download anything. import_atp is no longer async. The import_osm.fetch sub-step is gone; that work is now fetch_inputs.osm.
• fetch_planet is async: no nested runtime, and blocking work runs in spawn_blocking. It now hashes while streaming, which saves a full ~90 GB read pass. A resumed .part download first hashes the prefix already on disk.
• Release date is now computed right after fetch_inputs from the fetched metadata (provenance::anchor_timestamp). read_anchor is removed.

Cache rule (module doc of pipeline::inputs)

Per input, nothing is downloaded if the metadata sidecar exists and either the raw file or the artifact derived from it (alltheplaces.parquet / osm-features.index) exists. In the second case, Input::path doesn't exist, and the downstream step reuses its artifact. A planet .pbf without a sidecar (a regional extract for scripts/test-on-hetzner) is still accepted, and its metadata is computed on the spot.

One behavioural tightening: import_atp now rebuilds alltheplaces.parquet when alltheplaces.zip exists and is newer than it. This mirrors import_osm's existing mtime check. It guards against an old parquet being reused after the zip was downloaded again, which the new cache rule can cause when the sidecar is missing.

Tests

• inputs:
  • a complete workdir downloads nothing;
  • derived artifacts stand in for deleted raw inputs;
  • a failing ATP endpoint (mockito 503), with a planet server that accepts connections but never answers, makes fetch_inputs fail well within the test's 30 s limit and leaves no downloaded files. With no request timeout on the planet download, only cancellation can end that fetch.
• osm::fetch:
  • a regional .pbf without a sidecar is accepted (now async);
  • the index stands in for a deleted PBF;
  • a sidecar alone still triggers a download;
  • a resumed download (Range/206) hashes the whole file, checked against utils::hash_file.
• atp::fetch: the parquet stands in for a deleted zip, but only together with its sidecar.

Run locally:

• cargo fmt --check, cargo clippy --all-targets -- -D warnings and cargo test are clean, including test_pipeline and test_reproducible_artifacts.
• Output unchanged: main and this branch, run on the fixtures in the same workdir path, give byte-identical conflated.parquet, all .pmtiles, datapackage.json and the BOM.
  • Side note, not caused by this PR: in two different workdir paths, the .pmtiles files (and therefore datapackage.json) differ by one byte on main too. The tiles seem to depend on the workdir path. test_reproducible_artifacts doesn't catch this.
• Not run: the full planet pipeline (needs ~200 GB of disk).

Docs

• docs/TECHNICAL_DESIGN.md: an "Inputs" box and the fetch_inputs step in the diagram, and the step list updated.
• docs/LOGGING.md and docs/PRODUCTION.md: step names updated. A note says the --mem-limit table's import_osm timings still include the download.
• scripts/test-on-hetzner/remote/fetch_test_extract.sh: a comment updated.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WtzCoNtkdZsv5JqAkTcU6j

---

Generated by Claude Code