IPF drops 2.35 M people: LAUs on the NUTS3 fallback get zero broad-age shares, so A_i collapses to 0

[MaartenHilferink]

EU 2011, 100 m, GeoDms 20.19.1.m on main @ 735cad7.

/Analysis is handed 445 356 960 persons and writes 443 008 640. 2 348 320 people are in the input map and in no output map. It is not rounding and it is not spread thinly: it is fully accounted for by grid cells whose balancing factor A_i ends at exactly 0.

measured over the EU run	value
sum(Checks/P_r), the ARDECO input	445 356 960
sum(Result/X_asr/Total), the output	443 008 640
difference	2 348 320
sum(P_i) over the cells where Result/A_i == 0	2 348 314
LAUs with Checks/Qr_0_and_Pr_gt0	930
of those, LAUs where Share_LT15_LAU + Share_15_64_LAU + Share_GE65_LAU == 0	928

The 6-person gap between rows 3 and 4 is float32 rounding over 97 428 sums; per country the two agree to within a person (FR -2 318 976 vs -2 318 974, DE -26 632 vs -26 635, EL -1 980 vs -1 980, LT -667 vs -667, everything else nets to zero).

Mechanism

Checks/Share_<broad>_LAU is the per-LAU age structure that E_ib falls back on for 1 km cells where ESTAT has no age data:

attribute<Share> Share_LT15_LAU (r) := (sum(<raw census pop, LT15>, LAU_rel) / Q_asr/Total)[Share];

The numerator is the raw census from Population/AGEGROUP5_SEX_perLAU; the denominator is Q_asr/Total, which already carries the NUTS3 fallback of #12. For a LAU with no census the numerator is 0 while the denominator is P_r > 0, so all three broad shares come out 0. That is the case for 928 of the 930 LAUs on the fallback.

E_ib then hands 0 to every cell in those LAUs where ESTAT_Age_0_and_with_ARDECO_pop holds (158 728 cells EU-wide), and:

A_i_raw = P_i / sum_as(E_is * E_ib * B_asr)   ->  division by zero
A_i     = MakeDefined(A_i_raw, 0f)            ->  0
X_asi   = E_is * E_ib * A_i * B_asr           ->  0

so the cell's population disappears. The NUTS3 fallback repairs the target Q_asr but not the shares E_ib uses, and the two disagree in exactly the LAUs where the fallback was needed.

Where it lands

47 LAUs lose more than 100 persons, and 43 of them have POP_2021 = 0 in the LAU gpkg:

GISCO_ID	lost	input P_r	POP_2021	LAU_NAME
FR_75056	2 318 836	2 318 836	2 187 526	Paris (#19)
DE_03153504	2 111	3 757	0	Harz (Landkreis Goslar), gemfr. Gebiet
DE_09188451	1 800	3 532	0	Starnberger See
EL_99010000	1 798	1 798	1 811	Ψευδοκοινοτητα Αγιο Ορος (Αυτοδιοίκητο)
DE_09184452	1 532	2 535	0	Forstenrieder Park
DE_09184454	1 442	2 231	0	Grünwalder Forst
DE_03159501	1 205	3 610	0	Harz (Landkreis Göttingen), gemfr. Geb.
DE_01053105	1 072	2 066	0	Sachsenwald (Forstgutsbez.),gemfr.Geb.

The German entries are almost all gemeindefreie Gebiete - forests and lakes such as Starnberger See, Chiemsee, Ammersee and Perlacher Forst - that genuinely have no inhabitants. There the census is right and it is the ARDECO 100 m disaggregation that puts some 27 000 people in them. Dropping those is defensible; doing it silently is not, and the same code path is what loses Paris.

Suggestion

• Derive Share_<broad>_LAU from Q_asr rather than from the raw census, so a LAU on the NUTS3 fallback gets the fallback's age structure instead of zeros.
• Add A_i == 0 && P_i > 0 next to Checks/Qr_0_and_Pr_gt0, so a run can state how many people it dropped.
• Decide separately what should happen to ARDECO population in LAUs the census says are empty: reallocate over the neighbours, or accept the loss explicitly.

Reproducing

Measured with a Diag container appended to a scratch copy of the config, which writes per_country.csv, lau_lost.csv and unmatched_census_ids.csv. The two items that carry the finding:

attribute<Person> P_in_A0_cells := sum(/Analysis/Result/A_i == 0f ? /Analysis/P_i : 0f[Person], /Analysis/i/r_rel);
attribute<Share>  share_sum_LAU := /Analysis/Checks/Share_LT15_LAU + /Analysis/Checks/Share_15_64_LAU + /Analysis/Checks/Share_GE65_LAU;

Happy to add the whole container to the repo as cfg/main/Diagnostics.dms if that is wanted.

[MaartenHilferink]

The 2 348 320 double-counts #19: this issue's own scope is 29 484

Raising this because at first sight #19 and this issue look like the same bug - they end at the same symptom (A_i == 0) and both sit downstream of the #12 fallback. They are not, and the headline figure here is what makes them look identical: 98.7 % of it is Paris.

Splitting the per-country figures already measured above:

	persons	belongs to
FR_75056	2 318 836	#19
DE, gemeindefreie Gebiete	26 632	here
EL, EL_99010000 (Mount Athos)	1 980	here / #18
LT	667	here
FR, remainder (Marseille and rounding)	~140	here
total excluding Paris	29 484

The country rows account for 29 419 of that 29 484; the remaining 65 is the float32 rounding over 97 428 sums already noted above, spread over countries that net to roughly zero.

Why they are two issues and not one

The two sit on opposite sides of whether the #12 fallback worked.

#19 - the fallback fails. FR101's pool is {FR_75056} and that LAU is zero-filled, so the denominator in Q_asr_shares_based_on_NUTS3 is 0, the share is 0f/0f = null, and Q_asr is null. The target itself is missing. Paris is the only LAU in the continental study area in that position.

Here - the fallback succeeds, but the shares do not follow it. For the other 928 LAUs Q_asr comes out perfectly well-defined (P_r x a valid NUTS3 pyramid). What breaks is Checks/Share_<broad>_LAU: numerator the raw census (0), denominator the repaired Q_asr/Total (P_r > 0), so 0/P_r = 0. That mismatch between a repaired target and unrepaired shares exists whether or not any pool is empty, and it is what this issue is about.

The 930 / 928 split above already separates them. Paris should be one of the two exceptions: with Q_asr/Total null its share sum is 0/null = null rather than 0, so it fails the == 0 test. That is read off the config rather than measured - share_sum_LAU for FR_75056 would confirm it in one look. Its cells still reach A_i = 0, but by the other route: B_asr collapses to 0 at the end of iteration 1 (MakeDefined(1f * null / X, 0f)), and iteration 2 then divides P_i by 0.

Neither fix covers the other's case. Derive Share_<broad>_LAU from Q_asr as suggested above and Paris gets null/null - still null, still lost. Give the fallback a pool that cannot be empty as proposed in #19 and the 928 still have shares of 0, because their Q_asr was never the problem. Both changes are needed, and they touch different lines.

Suggested scope for this issue

• The bug: Checks/Share_<broad>_LAU (Reference_data.dms:30-34) derived from the raw census over a repaired denominator - 928 LAUs, 29 484 persons.
• The check: A_i == 0 && P_i > 0, as already proposed. This one is worth keeping global rather than per-issue: it catches both failure modes and anything else that reaches the same MakeDefined(A_i_raw, 0f).
• The policy question, which is neither this bug nor FR_75056 (Paris) drops out of the reference maps: zero-filled 2011 census and a NUTS3 pool of one #19: roughly 27 000 ARDECO persons that the 100 m disaggregation places in German gemeindefreie Gebiete - Starnberger See, Chiemsee, Ammersee, Perlacher Forst - which the census correctly reports as uninhabited. Dropping them may well be the right answer; it should be an explicit decision with a number attached rather than a side effect of a division by zero.