Skip to content

geo: a missing jurisdiction name reads as empty, not a crash - #40

Open
danohu wants to merge 1 commit into
mainfrom
vector-missing-names
Open

danohu wants to merge 1 commit into
mainfrom
vector-missing-names

Conversation

@danohu

@danohu danohu commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Why

Ingesting WORLD_BANK_ADMIN_2 into an ingest root that doesn't already have it crashes:

AttributeError: 'float' object has no attribute 'encode'

geo.convert_vector_to_flatgeobuf builds each jurisdiction's name with .astype(str).map(maybe_repair_mojibake). Under pandas 3 (main locks 3.0.3), astype(str) keeps a missing value missing instead of turning it into "nan". A null name therefore reaches maybe_repair_mojibake as a float. Some World Bank admin-2 districts have no name, so the admin-2 ingest always fails.

An ingest root that already has vector/world-bank/admin-2/v0/whole-world/world.fgb (from before the pandas 3 upgrade) reuses that file and never hits this. That probably explains why it hasn't surfaced. We hit it ingesting into an empty root. FDaP's ingest reads admin 2, so a fresh root can't run FDaP without it.

Where names are missing

175 of the 39,370 admin-2 districts have no name of their own. Each is the only district in its province (ids ending 001), i.e. a province the World Bank doesn't subdivide at admin 2:

Country Districts
AZE 74
ROU 42
MDA 37
LSO 10
CYP 6
ESP (Melilla and the North African territories) 5
GIB 1

The missing part is always the district's own name, never the country or the province. Nothing reads admin-2 names today: the per-jurisdiction rollups work at the provincial level, and FDaP uses admin 2 only for its geometry and ids.

Change

  • jdluc/geo.py: fill missing names with "" before astype(str). A district with no name of its own reads as e.g. Honduras | Olancho | .
  • jdluc/__tests__/geo_test.py: a test that a missing name reads as empty. It fails on main without the fix.

The file format is unchanged, so the dataset version stays v0. Existing admin-2 files are unaffected.

To reproduce

Unit level (seconds): check out this branch's test file onto main and run it.

git checkout vector-missing-names -- jdluc/__tests__/geo_test.py
uv run pytest jdluc/__tests__/geo_test.py -k missing_name

On main's geo.py this fails with the AttributeError above.

The real ingest, on an empty local ingest root. I haven't run this exact sequence locally, so treat it as a sketch. We saw the crash ingesting into an empty remote root, with geo.py identical to main.

mkdir -p /tmp/luc-repro/ingest /tmp/luc-repro/scratch

In .env, point both roots there:

INGEST_ROOT = '/tmp/luc-repro/ingest'
SCRATCH_ROOT = '/tmp/luc-repro/scratch'
uv run python -m jdluc.ingest WORLD_BANK_ADMIN_2 --backfill

--backfill avoids resolving a country to tiles, which would need admin 0 first. On main this should download the admin-2 GeoPackage (large), then crash in convert_vector_to_flatgeobuf. On this branch it should write vector/world-bank/admin-2/v0/whole-world/world.fgb.

Open questions

  • Only the name columns are filled. The id columns go through the same kind of astype(str), but I haven't seen a missing id.
  • A missing part leaves a trailing | in the name. Dropping empty parts instead would be easy if you prefer.
  • Alternatively, a district with no name could take its province's name (Romania | Bihor | Bihor).

Checked

  • The new test fails without the fix and passes with it; the rest of geo_test.py passes, as does pre-commit.
  • With this fix, our own pipeline ingested admin 2 into a root that lacked it, writing world.fgb in about 35 seconds after the download.

pandas keeps a missing value missing through astype(str), so a null name reached
maybe_repair_mojibake as a float. Some World Bank admin-2 districts have no name,
so ingesting admin 2 failed.
@danohu
danohu marked this pull request as ready for review November 17, 2026 14:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant