diff --git a/docs/further_research.md b/docs/further_research.md index 9de5455..78bcf6a 100644 --- a/docs/further_research.md +++ b/docs/further_research.md @@ -154,7 +154,7 @@ Three Paraná windows on the neighbouring tiles read 99–100% wetland class, an **Issue**: The 2000 snapshot is a different MapSPAM release (v3.0.7) from the ones that follow (v3.2), and its unmodeled crop vocabulary is disjoint from 2005's: a catch-all `OTHE` holds 560 Mha where the nine crops that replace it hold 161 Mha, correlating at r = 0.17 per pixel. The 2000 release simply accounts for ~400 Mha that 2005 does not, and no mapping between the two would be honest. -The consequence is that the nine 2005 crops read as expansion from zero across the 2000→2005 span, whose clipped unmodeled term is +77 Mha. Restricting that span to crops both snapshots report — the treatment applied to the 2005-onward boundaries, where the vocabularies do correspond — destroys 96% of the term rather than stabilizing it, so it is deliberately not applied there. +The consequence is that the nine 2005 crops read as expansion from zero across the 2000→2005 span, whose clipped unmodeled term is +77 Mha. Restricting that span to crops both snapshots report destroys 96% of the term rather than stabilizing it, so the unmodeled crops are compared as one total across it, as they are across every span. **Potential impact:** Small and bounded. The affected span carries 6.3% of the linear discount weight, the least of the three, and the error is one-directional: a denominator inflated by area the earlier release did not name depresses modeled crops' shares and cannot over-attribute. diff --git a/docs/methodology.md b/docs/methodology.md index f2163be..eaf3f5f 100644 --- a/docs/methodology.md +++ b/docs/methodology.md @@ -274,6 +274,8 @@ Each quantity is split by its own distribution — production by the pooled prod Where no reference year places the group in a pixel there is no basis for a split at all. Two of the six 2000 groups have a catch-all constituent (`GROUP_TO_RESIDUAL_NAME`: "other oil crops" and "other pulses"), which absorbs the whole unattributable remainder rather than handing a slice to every named sibling. The other four have none, so the total is split evenly across constituents as a last resort. That last resort is not rare: measured across the MapSPAM snapshots it divides 18.1 Mt of 2000 production and 4.84 Mha of area — about 7.7% of the four affected groups — among crops that may not grow in the pixel at all. Its effect on a finished factor is nonetheless bounded well under a percent, because the 2000 snapshot is the only thing the decomposition touches and it carries 6.25% of the emissions numerator and 3.13% of the production denominator. See "MapSPAM group decomposition falls back to an even split" in `further_research.md`. +The crops this leg does not model are compared between snapshots as one total. MapSPAM 2020 is the first snapshot to name citrus, onion, rubber and tomato, which 2005 and 2010 count inside tropical fruit, vegetables and rest of crops (`NEWLY_NAMED_CROP_TO_EARLIER_GROUP_NAME`), so rubber taking up land that 2010 counted as rest of crops is not expansion. + **Tropical woody-perennial factors other than oil palm are lower bounds.** GACED30 admits woody crops in principle, but measured against independent references it recovers only a small fraction of them — 1.1% and 0.3% of mapped oil palm on two Riau windows — so a woody perennial with no layer of its own is one of the destinations bucket 4 above is blind to, and the conversion emissions reaching coconut and rubber are far below what external references report. Those crops' factors should be read as lower bounds rather than estimates until a per-crop destination layer supplies what the general cropland mask cannot. Oil palm has one, and coffee has FDaP's between 24°S and 24°N; `further_research.md` covers the mechanism for the rest, and separately a smaller allocation gap specific to oil palm. The asymmetry worth naming is that MapSPAM's expansion denominator *does* see these crops, so they dilute every other crop's share in the cells they grow in while collecting almost none of the emissions themselves. **Windowed production denominator (a departure from WRI).** The emissions numerator is time-resolved: it links conversion emissions to each crop's *historical* expansion, weighted by recency. Dividing that by a single *current-year* production snapshot — as in WRI's methodology — would place the numerator and denominator on inconsistent time bases. A crop that expanded early in the window and then contracted would carry real conversion emissions against a shrunken (or zero) present-day production, yielding an inflated or undefined emissions factor. Instead, production is reduced over the same 20-year lookback window using the same linear temporal ramp applied to emissions: each conversion year's discount weight is split evenly between the two MapSPAM snapshots of the span holding it, a year after 2020 gives its whole weight to the 2020 snapshot, and production is the snapshots' mean under those weights (`get_snapshot_year_to_weight` in `statistical.py`). This ties emission allocation and production to the same years with the same recency weighting. When production is flat across the window the result reduces exactly to the current-production value; it diverges only when production actually changed. This consistent linkage is a material improvement over attributing historical-expansion emissions to present-day production alone. diff --git a/jdluc/__tests__/datasets/ifpri_mapspam_test.py b/jdluc/__tests__/datasets/ifpri_mapspam_test.py index 4a3cddf..f7547eb 100644 --- a/jdluc/__tests__/datasets/ifpri_mapspam_test.py +++ b/jdluc/__tests__/datasets/ifpri_mapspam_test.py @@ -1,5 +1,3 @@ -import itertools - import pytest from jdluc.datasets import ifpri_mapspam @@ -59,27 +57,6 @@ def test_is_reported_as_group_is_exactly_the_constituents_in_2000() -> None: } -def test_comparable_spans_agree_with_the_vocabularies() -> None: - # Not a restatement of the constant: it checks the constant against how much of the - # unrecoverable vocabulary the two snapshots actually share. Listing a pair as comparable - # when their names barely overlap is what would make name-matching measure the taxonomy - # instead of the land. - for before, after in itertools.combinations(sorted(ifpri_mapspam.YEARS), 2): - one, two = ( - ifpri_mapspam.YEAR_TO_UNRECOVERABLE_CROP_NAMES[before], - ifpri_mapspam.YEAR_TO_UNRECOVERABLE_CROP_NAMES[after], - ) - shared = len(one & two) / len(one | two) - if (before, after) in ifpri_mapspam.SPANS_WITH_COMPARABLE_CROP_NAMES: - assert shared > 0.5, ( - f"{before}->{after} called comparable but shares {shared:.0%}" - ) - else: - assert shared < 0.2, ( - f"{before}->{after} called incomparable but shares {shared:.0%}" - ) - - @pytest.mark.parametrize("quantity", sorted(ifpri_mapspam.Quantity)) @pytest.mark.parametrize("year", sorted(ifpri_mapspam.YEARS)) def test_get_band_name_is_positionally_consistent( diff --git a/jdluc/__tests__/statistical_test.py b/jdluc/__tests__/statistical_test.py index ffd8f8f..f2ea98f 100644 --- a/jdluc/__tests__/statistical_test.py +++ b/jdluc/__tests__/statistical_test.py @@ -46,13 +46,6 @@ UNATTRIBUTED_0, UNATTRIBUTED_1 = sorted( ifpri_mapspam.YEAR_TO_UNRECOVERABLE_CROP_NAMES[2005] )[:2] -# A crop no `Crop` claims that MapSPAM only starts reporting in 2020. Compared in 2020's own -# naming, since two attributed crops are renamed there (ACOF -> COFF, SMIL -> MILL) and would -# otherwise look 2020-only themselves. -UNATTRIBUTED_2020_ONLY = sorted( - ifpri_mapspam.YEAR_TO_UNRECOVERABLE_CROP_NAMES[2020] - - ifpri_mapspam.YEAR_TO_UNRECOVERABLE_CROP_NAMES[2010] -)[0] # `get_hectares_per_pixel` reads its spacing off these coords, and uniform values keep @@ -82,7 +75,8 @@ def get_dset( ) -> xarray.Dataset: # A band for every crop of every (quantity, year) -- the real dset always carries all four # snapshots. Unset crops, and unset years, are 0. Pasture rides along as a cell fraction, and - # each grazer as a density in heads/ha, weighed into livestock units as `workflow` weighs them. + # each grazer as a density in heads/ha, weighed into livestock units as `workflow` weighs them, + # and each cell's hectares are measured on the whole grid, as `workflow` measures them. dset = xarray.Dataset( { ifpri_mapspam.get_band_name( @@ -113,6 +107,11 @@ def get_dset( get_band_name_to_livestock_units( dset=dset, species_to_units_per_head=units_per_head or UNITS_PER_HEAD ) + | { + "hectares-per-cell:ha": emit.get_hectares_per_pixel( + darray=dset["pasture:fraction:2020"] + ) + } ) @@ -139,9 +138,7 @@ def get_density( } -HECTARES_PER_CELL = get_value( - emit.get_hectares_per_pixel(darray=get_dset(values={})["pasture:fraction:2020"]) -) +HECTARES_PER_CELL = get_value(darray=get_dset(values={})["hectares-per-cell:ha"]) @pytest.mark.parametrize( @@ -211,17 +208,21 @@ def get_density( id="crops-it-cannot-attribute-stay-lumped-and-net-against-each-other-leaving-maize-the-whole-cell", ), pytest.param( - # Counting UNATTRIBUTED_2020_ONLY's whole area as expansion from zero would halve - # MAIZE's share, although nothing was necessarily planted. - {2010: {}, 2020: {"MAIZ": 100.0, UNATTRIBUTED_2020_ONLY: 100.0}}, + # MapSPAM 2010 counted rubber in rest of crops + {2010: {"REST": 100.0}, 2020: {"MAIZ": 100.0, "RUBB": 100.0}}, 2010, 2020, {Crop.MAIZE: 1.0}, - id="a-crop-the-earlier-snapshot-never-reported-is-newly-named-not-newly-grown-so-it-is-dropped", + id="a-crop-2020-first-names-is-not-expansion-where-it-only-left-its-earlier-group", + ), + pytest.param( + {2010: {"REST": 100.0}, 2020: {"MAIZ": 100.0, "RUBB": 200.0}}, + 2010, + 2020, + {Crop.MAIZE: 0.5}, + id="a-crop-2020-first-names-that-outgrew-its-earlier-group-dilutes", ), pytest.param( - # The converse of the case above -- same shape, one crop different -- and the reason - # it cannot simply drop every unattributed crop. {2010: {}, 2020: {"MAIZ": 100.0, UNATTRIBUTED_0: 100.0}}, 2010, 2020, @@ -230,9 +231,7 @@ def get_density( ), pytest.param( # 2000 is a different MapSPAM release and shares almost no unattributed names with - # 2005, so the "reported in both" test above would reject nearly the whole lump here - # rather than stabilise it -- hence the span's absence from - # `SPANS_WITH_COMPARABLE_CROP_NAMES`. + # 2005, so the lump is compared as a whole rather than name by name {2000: {}, 2005: {"MAIZ": 100.0, UNATTRIBUTED_0: 100.0}}, 2000, 2005, @@ -984,6 +983,39 @@ def test_get_commodity_name_to_totals( ) +def test_a_clip_one_cell_wide_keeps_the_hectares_of_its_cells() -> None: + # One column of the 2x2 grid holds half of every total the whole grid holds + dset = get_dset( + pasture_fraction=dict.fromkeys(ifpri_mapspam.YEARS, 0.5), + values={ + (AREA, 2020): {Crop.WHEAT.value: 100.0}, + (PRODUCTION, 2020): {Crop.WHEAT.value: 300.0}, + }, + ) + whole, column = ( + get_name_to_totals( + commodity_to_share={Crop.WHEAT: 0.5, Livestock.PASTURE: 0.25}, + cropland_occupation=2.0, + dset=clipped, + forest_emissions=10.0, + occupation_shares={Crop.WHEAT: 0.5}, + ) + for clipped in (dset, dset.isel(x=[0])) + ) + assert { + (name, key): value + for name, totals in column.items() + for key, value in totals.items() + } == pytest.approx( + { + (name, key): value / 2 + for name, totals in whole.items() + for key, value in totals.items() + }, + nan_ok=True, + ) + + @pytest.mark.parametrize( ("heads_per_ha", "expected"), ( diff --git a/jdluc/datasets/ifpri_mapspam.py b/jdluc/datasets/ifpri_mapspam.py index 92346ba..365bfd7 100644 --- a/jdluc/datasets/ifpri_mapspam.py +++ b/jdluc/datasets/ifpri_mapspam.py @@ -411,22 +411,22 @@ def get_reported_crop_name(canonical_crop_name: str, year: int) -> str: } == YEAR_TO_UNRECOVERABLE_CROP_NAMES -# Snapshot pairs whose crop names can be matched directly. Every pair involving the decompose year -# is excluded: it is a different MapSPAM release (v3.0.7 against v3.2) and its unrecoverable -# vocabulary is nearly disjoint from 2005's -- a catch-all OTHE leaves and nine crops appear in its -# place, holding 560 Mha against their 161 and correlating at r = 0.17 per pixel. A name on only -# one side of that boundary says the release did not use it, not that the crop was absent. -SPANS_WITH_COMPARABLE_CROP_NAMES: set[tuple[int, int]] = { - (2005, 2010), - (2005, 2020), - (2010, 2020), +# The unrecoverable crops 2020 is the first snapshot to name, each with the group 2005 and 2010 +# counted its area in, so the unrecoverable lump holds the same crops in 2005, 2010 and 2020 +NEWLY_NAMED_CROP_TO_EARLIER_GROUP_NAME: dict[str, str] = { + "CITR": "TROF", + "ONIO": "VEGE", + "RUBB": "REST", + "TOMA": "VEGE", } -assert { - (before, after) - for before in YEARS - for after in YEARS - if before < after and DECOMPOSITION_YEAR not in (before, after) -} == SPANS_WITH_COMPARABLE_CROP_NAMES +assert YEAR_TO_UNRECOVERABLE_CROP_NAMES[2005] == YEAR_TO_UNRECOVERABLE_CROP_NAMES[2010] +assert YEAR_TO_UNRECOVERABLE_CROP_NAMES[2020] == ( + YEAR_TO_UNRECOVERABLE_CROP_NAMES[2010] | set(NEWLY_NAMED_CROP_TO_EARLIER_GROUP_NAME) +) +assert ( + set(NEWLY_NAMED_CROP_TO_EARLIER_GROUP_NAME.values()) + <= (YEAR_TO_UNRECOVERABLE_CROP_NAMES[2010]) +) class Quantity(enum.StrEnum): diff --git a/jdluc/statistical.py b/jdluc/statistical.py index 3386343..e97d4b3 100644 --- a/jdluc/statistical.py +++ b/jdluc/statistical.py @@ -182,10 +182,9 @@ def get_commodity_hectares( commodity: Commodity, dset: xarray.Dataset, year: int ) -> xarray.DataArray: if isinstance(commodity, gpw_livestock.Livestock): - pasture_fraction = dset[f"pasture:fraction:{year:d}"] return ( - pasture_fraction - * emit.get_hectares_per_pixel(darray=pasture_fraction) + dset[f"pasture:fraction:{year:d}"] + * dset["hectares-per-cell:ha"] * gpw_livestock.get_livestock_to_grazing_share(dset=dset, year=year)[ commodity ] @@ -217,15 +216,11 @@ def get_cropland_expansion( for crop in sorted(Crop) ) - # Drop any crops which are being newly tracked so they aren't interpreted as an expansion from zero - before_names = ifpri_mapspam.YEAR_TO_UNRECOVERABLE_CROP_NAMES[before] - after_names = ifpri_mapspam.YEAR_TO_UNRECOVERABLE_CROP_NAMES[after] - if (before, after) in ifpri_mapspam.SPANS_WITH_COMPARABLE_CROP_NAMES: - before_names = after_names = before_names & after_names - - def get_unattributed(names: set[str], year: int) -> xarray.DataArray: + def get_unattributed(year: int) -> xarray.DataArray: # NB: no canonical lookup is required -- these names are already `year`'s own, and - # nothing divides by them, so the raw band is the whole answer + # nothing divides by them, so the raw band is the whole answer. They include the crops + # 2020 names first, whose area 2005 and 2010 held in a group + # (`ifpri_mapspam.NEWLY_NAMED_CROP_TO_EARLIER_GROUP_NAME`) return sum( ( ifpri_mapspam.get_reported_quantity( @@ -234,14 +229,13 @@ def get_unattributed(names: set[str], year: int) -> xarray.DataArray: reported_crop_name=name, year=year, ) - for name in sorted(names) + for name in sorted(ifpri_mapspam.YEAR_TO_UNRECOVERABLE_CROP_NAMES[year]) ), start=xarray.DataArray(numpy.float32(0)), ) unattributed_expansion = ( - get_unattributed(names=after_names, year=after) - - get_unattributed(names=before_names, year=before) + get_unattributed(year=after) - get_unattributed(year=before) ).clip(min=0) return attributed_expansion + unattributed_expansion @@ -252,13 +246,9 @@ def get_commodity_to_share( """Each commodity's share of everything a cell emitted in a span, whatever each pixel became.""" # Pastureland expands alongside the crops, measured net and clipped exactly as a crop's is: # GPW is annual, and its gross gain would charge pasture for churn MapSPAM cannot see - pasture_fraction_after = dset[f"pasture:fraction:{after:d}"] - pasture_fraction_before = dset[f"pasture:fraction:{before:d}"] pasture_expansion = ( - pasture_fraction_after - * emit.get_hectares_per_pixel(darray=pasture_fraction_after) - - pasture_fraction_before - * emit.get_hectares_per_pixel(darray=pasture_fraction_before) + (dset[f"pasture:fraction:{after:d}"] - dset[f"pasture:fraction:{before:d}"]) + * dset["hectares-per-cell:ha"] ).clip(min=0) total_expansion = ( get_cropland_expansion(after=after, before=before, dset=dset) @@ -383,11 +373,10 @@ def get_commodity_name_to_totals( MapSPAM crops no row models, the forest and grassland charge where neither cropland nor pasture expanded, and the cropland peat pulse where no crop expanded. """ - cropland_occupation_per_hectare = dset[ - f"{emit.CROPLAND:s}-{emit.PEATLAND_OCCUPATION:s}:tco2e-per-ha" - ] - hectares = emit.get_hectares_per_pixel(darray=cropland_occupation_per_hectare) - cropland_occupation = cropland_occupation_per_hectare * hectares + hectares = dset["hectares-per-cell:ha"] + cropland_occupation = ( + dset[f"{emit.CROPLAND:s}-{emit.PEATLAND_OCCUPATION:s}:tco2e-per-ha"] * hectares + ) pastureland_occupation = ( dset[f"{emit.PASTURELAND:s}-{emit.PEATLAND_OCCUPATION:s}:tco2e-per-ha"] * hectares @@ -647,6 +636,10 @@ def workflow( }, ) ) + # NB: on the whole tile, since a clip one cell wide has no spacing to measure a cell by + merged["hectares-per-cell:ha"] = emit.get_hectares_per_pixel( + darray=merged["dropped-emissions:tco2e-per-ha"] + ) # NB: usually we avoid dumping to local mmemory, but since this xarray is small there is no # OOM risk. Doing so speeds up the many downstream reads for each jurisdiction merged = merged.load()