Repository navigation
Conversation
pandas keeps a missing value missing through astype(str), so a null name reached maybe_repair_mojibake as a float. Some World Bank admin-2 districts have no name, so ingesting admin 2 failed.
danohu
marked this pull request as ready for review
November 17, 2026 14:47
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Ingesting
WORLD_BANK_ADMIN_2into an ingest root that doesn't already have it crashes:geo.convert_vector_to_flatgeobufbuilds each jurisdiction's name with.astype(str).map(maybe_repair_mojibake). Under pandas 3 (main locks 3.0.3),astype(str)keeps a missing value missing instead of turning it into"nan". A null name therefore reachesmaybe_repair_mojibakeas a float. Some World Bank admin-2 districts have no name, so the admin-2 ingest always fails.An ingest root that already has
vector/world-bank/admin-2/v0/whole-world/world.fgb(from before the pandas 3 upgrade) reuses that file and never hits this. That probably explains why it hasn't surfaced. We hit it ingesting into an empty root. FDaP's ingest reads admin 2, so a fresh root can't run FDaP without it.Where names are missing
175 of the 39,370 admin-2 districts have no name of their own. Each is the only district in its province (ids ending
001), i.e. a province the World Bank doesn't subdivide at admin 2:The missing part is always the district's own name, never the country or the province. Nothing reads admin-2 names today: the per-jurisdiction rollups work at the provincial level, and FDaP uses admin 2 only for its geometry and ids.
Change
jdluc/geo.py: fill missing names with""beforeastype(str). A district with no name of its own reads as e.g.Honduras | Olancho |.jdluc/__tests__/geo_test.py: a test that a missing name reads as empty. It fails on main without the fix.The file format is unchanged, so the dataset version stays
v0. Existing admin-2 files are unaffected.To reproduce
Unit level (seconds): check out this branch's test file onto main and run it.
On main's
geo.pythis fails with theAttributeErrorabove.The real ingest, on an empty local ingest root. I haven't run this exact sequence locally, so treat it as a sketch. We saw the crash ingesting into an empty remote root, with
geo.pyidentical to main.In
.env, point both roots there:--backfillavoids resolving a country to tiles, which would need admin 0 first. On main this should download the admin-2 GeoPackage (large), then crash inconvert_vector_to_flatgeobuf. On this branch it should writevector/world-bank/admin-2/v0/whole-world/world.fgb.Open questions
astype(str), but I haven't seen a missing id.|in the name. Dropping empty parts instead would be easy if you prefer.Romania | Bihor | Bihor).Checked
geo_test.pypasses, as does pre-commit.world.fgbin about 35 seconds after the download.