Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/further_research.md
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,7 @@ Three Paraná windows on the neighbouring tiles read 99–100% wetland class, an

**Issue**: The 2000 snapshot is a different MapSPAM release (v3.0.7) from the ones that follow (v3.2), and its unmodeled crop vocabulary is disjoint from 2005's: a catch-all `OTHE` holds 560 Mha where the nine crops that replace it hold 161 Mha, correlating at r = 0.17 per pixel. The 2000 release simply accounts for ~400 Mha that 2005 does not, and no mapping between the two would be honest.

The consequence is that the nine 2005 crops read as expansion from zero across the 2000→2005 span, whose clipped unmodeled term is +77 Mha. Restricting that span to crops both snapshots report — the treatment applied to the 2005-onward boundaries, where the vocabularies do correspond — destroys 96% of the term rather than stabilizing it, so it is deliberately not applied there.
The consequence is that the nine 2005 crops read as expansion from zero across the 2000→2005 span, whose clipped unmodeled term is +77 Mha. Restricting that span to crops both snapshots report destroys 96% of the term rather than stabilizing it, so the unmodeled crops are compared as one total across it, as they are across every span.

**Potential impact:** Small and bounded. The affected span carries 6.3% of the linear discount weight, the least of the three, and the error is one-directional: a denominator inflated by area the earlier release did not name depresses modeled crops' shares and cannot over-attribute.

Expand Down
2 changes: 2 additions & 0 deletions docs/methodology.md
Original file line number Diff line number Diff line change
Expand Up @@ -274,6 +274,8 @@ Each quantity is split by its own distribution — production by the pooled prod

Where no reference year places the group in a pixel there is no basis for a split at all. Two of the six 2000 groups have a catch-all constituent (`GROUP_TO_RESIDUAL_NAME`: "other oil crops" and "other pulses"), which absorbs the whole unattributable remainder rather than handing a slice to every named sibling. The other four have none, so the total is split evenly across constituents as a last resort. That last resort is not rare: measured across the MapSPAM snapshots it divides 18.1 Mt of 2000 production and 4.84 Mha of area — about 7.7% of the four affected groups — among crops that may not grow in the pixel at all. Its effect on a finished factor is nonetheless bounded well under a percent, because the 2000 snapshot is the only thing the decomposition touches and it carries 6.25% of the emissions numerator and 3.13% of the production denominator. See "MapSPAM group decomposition falls back to an even split" in `further_research.md`.

The crops this leg does not model are compared between snapshots as one total. MapSPAM 2020 is the first snapshot to name citrus, onion, rubber and tomato, which 2005 and 2010 count inside tropical fruit, vegetables and rest of crops (`NEWLY_NAMED_CROP_TO_EARLIER_GROUP_NAME`), so rubber taking up land that 2010 counted as rest of crops is not expansion.

**Tropical woody-perennial factors other than oil palm are lower bounds.** GACED30 admits woody crops in principle, but measured against independent references it recovers only a small fraction of them — 1.1% and 0.3% of mapped oil palm on two Riau windows — so a woody perennial with no layer of its own is one of the destinations bucket 4 above is blind to, and the conversion emissions reaching coconut and rubber are far below what external references report. Those crops' factors should be read as lower bounds rather than estimates until a per-crop destination layer supplies what the general cropland mask cannot. Oil palm has one, and coffee has FDaP's between 24°S and 24°N; `further_research.md` covers the mechanism for the rest, and separately a smaller allocation gap specific to oil palm. The asymmetry worth naming is that MapSPAM's expansion denominator *does* see these crops, so they dilute every other crop's share in the cells they grow in while collecting almost none of the emissions themselves.

**Windowed production denominator (a departure from WRI).** The emissions numerator is time-resolved: it links conversion emissions to each crop's *historical* expansion, weighted by recency. Dividing that by a single *current-year* production snapshot — as in WRI's methodology — would place the numerator and denominator on inconsistent time bases. A crop that expanded early in the window and then contracted would carry real conversion emissions against a shrunken (or zero) present-day production, yielding an inflated or undefined emissions factor. Instead, production is reduced over the same 20-year lookback window using the same linear temporal ramp applied to emissions: each conversion year's discount weight is split evenly between the two MapSPAM snapshots of the span holding it, a year after 2020 gives its whole weight to the 2020 snapshot, and production is the snapshots' mean under those weights (`get_snapshot_year_to_weight` in `statistical.py`). This ties emission allocation and production to the same years with the same recency weighting. When production is flat across the window the result reduces exactly to the current-production value; it diverges only when production actually changed. This consistent linkage is a material improvement over attributing historical-expansion emissions to present-day production alone.
Expand Down
23 changes: 0 additions & 23 deletions jdluc/__tests__/datasets/ifpri_mapspam_test.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
import itertools

import pytest

from jdluc.datasets import ifpri_mapspam
Expand Down Expand Up @@ -59,27 +57,6 @@ def test_is_reported_as_group_is_exactly_the_constituents_in_2000() -> None:
}


def test_comparable_spans_agree_with_the_vocabularies() -> None:
# Not a restatement of the constant: it checks the constant against how much of the
# unrecoverable vocabulary the two snapshots actually share. Listing a pair as comparable
# when their names barely overlap is what would make name-matching measure the taxonomy
# instead of the land.
for before, after in itertools.combinations(sorted(ifpri_mapspam.YEARS), 2):
one, two = (
ifpri_mapspam.YEAR_TO_UNRECOVERABLE_CROP_NAMES[before],
ifpri_mapspam.YEAR_TO_UNRECOVERABLE_CROP_NAMES[after],
)
shared = len(one & two) / len(one | two)
if (before, after) in ifpri_mapspam.SPANS_WITH_COMPARABLE_CROP_NAMES:
assert shared > 0.5, (
f"{before}->{after} called comparable but shares {shared:.0%}"
)
else:
assert shared < 0.2, (
f"{before}->{after} called incomparable but shares {shared:.0%}"
)


@pytest.mark.parametrize("quantity", sorted(ifpri_mapspam.Quantity))
@pytest.mark.parametrize("year", sorted(ifpri_mapspam.YEARS))
def test_get_band_name_is_positionally_consistent(
Expand Down
72 changes: 52 additions & 20 deletions jdluc/__tests__/statistical_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -46,13 +46,6 @@
UNATTRIBUTED_0, UNATTRIBUTED_1 = sorted(
ifpri_mapspam.YEAR_TO_UNRECOVERABLE_CROP_NAMES[2005]
)[:2]
# A crop no `Crop` claims that MapSPAM only starts reporting in 2020. Compared in 2020's own
# naming, since two attributed crops are renamed there (ACOF -> COFF, SMIL -> MILL) and would
# otherwise look 2020-only themselves.
UNATTRIBUTED_2020_ONLY = sorted(
ifpri_mapspam.YEAR_TO_UNRECOVERABLE_CROP_NAMES[2020]
- ifpri_mapspam.YEAR_TO_UNRECOVERABLE_CROP_NAMES[2010]
)[0]


# `get_hectares_per_pixel` reads its spacing off these coords, and uniform values keep
Expand Down Expand Up @@ -82,7 +75,8 @@ def get_dset(
) -> xarray.Dataset:
# A band for every crop of every (quantity, year) -- the real dset always carries all four
# snapshots. Unset crops, and unset years, are 0. Pasture rides along as a cell fraction, and
# each grazer as a density in heads/ha, weighed into livestock units as `workflow` weighs them.
# each grazer as a density in heads/ha, weighed into livestock units as `workflow` weighs them,
# and each cell's hectares are measured on the whole grid, as `workflow` measures them.
dset = xarray.Dataset(
{
ifpri_mapspam.get_band_name(
Expand Down Expand Up @@ -113,6 +107,11 @@ def get_dset(
get_band_name_to_livestock_units(
dset=dset, species_to_units_per_head=units_per_head or UNITS_PER_HEAD
)
| {
"hectares-per-cell:ha": emit.get_hectares_per_pixel(
darray=dset["pasture:fraction:2020"]
)
}
)


Expand All @@ -139,9 +138,7 @@ def get_density(
}


HECTARES_PER_CELL = get_value(
emit.get_hectares_per_pixel(darray=get_dset(values={})["pasture:fraction:2020"])
)
HECTARES_PER_CELL = get_value(darray=get_dset(values={})["hectares-per-cell:ha"])


@pytest.mark.parametrize(
Expand Down Expand Up @@ -211,17 +208,21 @@ def get_density(
id="crops-it-cannot-attribute-stay-lumped-and-net-against-each-other-leaving-maize-the-whole-cell",
),
pytest.param(
# Counting UNATTRIBUTED_2020_ONLY's whole area as expansion from zero would halve
# MAIZE's share, although nothing was necessarily planted.
{2010: {}, 2020: {"MAIZ": 100.0, UNATTRIBUTED_2020_ONLY: 100.0}},
# MapSPAM 2010 counted rubber in rest of crops
{2010: {"REST": 100.0}, 2020: {"MAIZ": 100.0, "RUBB": 100.0}},
2010,
2020,
{Crop.MAIZE: 1.0},
id="a-crop-the-earlier-snapshot-never-reported-is-newly-named-not-newly-grown-so-it-is-dropped",
id="a-crop-2020-first-names-is-not-expansion-where-it-only-left-its-earlier-group",
),
pytest.param(
{2010: {"REST": 100.0}, 2020: {"MAIZ": 100.0, "RUBB": 200.0}},
2010,
2020,
{Crop.MAIZE: 0.5},
id="a-crop-2020-first-names-that-outgrew-its-earlier-group-dilutes",
),
pytest.param(
# The converse of the case above -- same shape, one crop different -- and the reason
# it cannot simply drop every unattributed crop.
{2010: {}, 2020: {"MAIZ": 100.0, UNATTRIBUTED_0: 100.0}},
2010,
2020,
Expand All @@ -230,9 +231,7 @@ def get_density(
),
pytest.param(
# 2000 is a different MapSPAM release and shares almost no unattributed names with
# 2005, so the "reported in both" test above would reject nearly the whole lump here
# rather than stabilise it -- hence the span's absence from
# `SPANS_WITH_COMPARABLE_CROP_NAMES`.
# 2005, so the lump is compared as a whole rather than name by name
{2000: {}, 2005: {"MAIZ": 100.0, UNATTRIBUTED_0: 100.0}},
2000,
2005,
Expand Down Expand Up @@ -984,6 +983,39 @@ def test_get_commodity_name_to_totals(
)


def test_a_clip_one_cell_wide_keeps_the_hectares_of_its_cells() -> None:
# One column of the 2x2 grid holds half of every total the whole grid holds
dset = get_dset(
pasture_fraction=dict.fromkeys(ifpri_mapspam.YEARS, 0.5),
values={
(AREA, 2020): {Crop.WHEAT.value: 100.0},
(PRODUCTION, 2020): {Crop.WHEAT.value: 300.0},
},
)
whole, column = (
get_name_to_totals(
commodity_to_share={Crop.WHEAT: 0.5, Livestock.PASTURE: 0.25},
cropland_occupation=2.0,
dset=clipped,
forest_emissions=10.0,
occupation_shares={Crop.WHEAT: 0.5},
)
for clipped in (dset, dset.isel(x=[0]))
)
assert {
(name, key): value
for name, totals in column.items()
for key, value in totals.items()
} == pytest.approx(
{
(name, key): value / 2
for name, totals in whole.items()
for key, value in totals.items()
},
nan_ok=True,
)


@pytest.mark.parametrize(
("heads_per_ha", "expected"),
(
Expand Down
30 changes: 15 additions & 15 deletions jdluc/datasets/ifpri_mapspam.py
Original file line number Diff line number Diff line change
Expand Up @@ -411,22 +411,22 @@ def get_reported_crop_name(canonical_crop_name: str, year: int) -> str:
} == YEAR_TO_UNRECOVERABLE_CROP_NAMES


# Snapshot pairs whose crop names can be matched directly. Every pair involving the decompose year
# is excluded: it is a different MapSPAM release (v3.0.7 against v3.2) and its unrecoverable
# vocabulary is nearly disjoint from 2005's -- a catch-all OTHE leaves and nine crops appear in its
# place, holding 560 Mha against their 161 and correlating at r = 0.17 per pixel. A name on only
# one side of that boundary says the release did not use it, not that the crop was absent.
SPANS_WITH_COMPARABLE_CROP_NAMES: set[tuple[int, int]] = {
(2005, 2010),
(2005, 2020),
(2010, 2020),
# The unrecoverable crops 2020 is the first snapshot to name, each with the group 2005 and 2010
# counted its area in, so the unrecoverable lump holds the same crops in 2005, 2010 and 2020
NEWLY_NAMED_CROP_TO_EARLIER_GROUP_NAME: dict[str, str] = {
"CITR": "TROF",
"ONIO": "VEGE",
"RUBB": "REST",
"TOMA": "VEGE",
}
assert {
(before, after)
for before in YEARS
for after in YEARS
if before < after and DECOMPOSITION_YEAR not in (before, after)
} == SPANS_WITH_COMPARABLE_CROP_NAMES
assert YEAR_TO_UNRECOVERABLE_CROP_NAMES[2005] == YEAR_TO_UNRECOVERABLE_CROP_NAMES[2010]
assert YEAR_TO_UNRECOVERABLE_CROP_NAMES[2020] == (
YEAR_TO_UNRECOVERABLE_CROP_NAMES[2010] | set(NEWLY_NAMED_CROP_TO_EARLIER_GROUP_NAME)
)
assert (
set(NEWLY_NAMED_CROP_TO_EARLIER_GROUP_NAME.values())
<= (YEAR_TO_UNRECOVERABLE_CROP_NAMES[2010])
)


class Quantity(enum.StrEnum):
Expand Down
43 changes: 18 additions & 25 deletions jdluc/statistical.py
Original file line number Diff line number Diff line change
Expand Up @@ -182,10 +182,9 @@ def get_commodity_hectares(
commodity: Commodity, dset: xarray.Dataset, year: int
) -> xarray.DataArray:
if isinstance(commodity, gpw_livestock.Livestock):
pasture_fraction = dset[f"pasture:fraction:{year:d}"]
return (
pasture_fraction
* emit.get_hectares_per_pixel(darray=pasture_fraction)
dset[f"pasture:fraction:{year:d}"]
* dset["hectares-per-cell:ha"]
* gpw_livestock.get_livestock_to_grazing_share(dset=dset, year=year)[
commodity
]
Expand Down Expand Up @@ -217,15 +216,11 @@ def get_cropland_expansion(
for crop in sorted(Crop)
)

# Drop any crops which are being newly tracked so they aren't interpreted as an expansion from zero
before_names = ifpri_mapspam.YEAR_TO_UNRECOVERABLE_CROP_NAMES[before]
after_names = ifpri_mapspam.YEAR_TO_UNRECOVERABLE_CROP_NAMES[after]
if (before, after) in ifpri_mapspam.SPANS_WITH_COMPARABLE_CROP_NAMES:
before_names = after_names = before_names & after_names

def get_unattributed(names: set[str], year: int) -> xarray.DataArray:
def get_unattributed(year: int) -> xarray.DataArray:
# NB: no canonical lookup is required -- these names are already `year`'s own, and
# nothing divides by them, so the raw band is the whole answer
# nothing divides by them, so the raw band is the whole answer. They include the crops
# 2020 names first, whose area 2005 and 2010 held in a group
# (`ifpri_mapspam.NEWLY_NAMED_CROP_TO_EARLIER_GROUP_NAME`)
return sum(
(
ifpri_mapspam.get_reported_quantity(
Expand All @@ -234,14 +229,13 @@ def get_unattributed(names: set[str], year: int) -> xarray.DataArray:
reported_crop_name=name,
year=year,
)
for name in sorted(names)
for name in sorted(ifpri_mapspam.YEAR_TO_UNRECOVERABLE_CROP_NAMES[year])
),
start=xarray.DataArray(numpy.float32(0)),
)

unattributed_expansion = (
get_unattributed(names=after_names, year=after)
- get_unattributed(names=before_names, year=before)
get_unattributed(year=after) - get_unattributed(year=before)
).clip(min=0)
return attributed_expansion + unattributed_expansion

Expand All @@ -252,13 +246,9 @@ def get_commodity_to_share(
"""Each commodity's share of everything a cell emitted in a span, whatever each pixel became."""
# Pastureland expands alongside the crops, measured net and clipped exactly as a crop's is:
# GPW is annual, and its gross gain would charge pasture for churn MapSPAM cannot see
pasture_fraction_after = dset[f"pasture:fraction:{after:d}"]
pasture_fraction_before = dset[f"pasture:fraction:{before:d}"]
pasture_expansion = (
pasture_fraction_after
* emit.get_hectares_per_pixel(darray=pasture_fraction_after)
- pasture_fraction_before
* emit.get_hectares_per_pixel(darray=pasture_fraction_before)
(dset[f"pasture:fraction:{after:d}"] - dset[f"pasture:fraction:{before:d}"])
* dset["hectares-per-cell:ha"]
).clip(min=0)
total_expansion = (
get_cropland_expansion(after=after, before=before, dset=dset)
Expand Down Expand Up @@ -383,11 +373,10 @@ def get_commodity_name_to_totals(
MapSPAM crops no row models, the forest and grassland charge where neither cropland nor
pasture expanded, and the cropland peat pulse where no crop expanded.
"""
cropland_occupation_per_hectare = dset[
f"{emit.CROPLAND:s}-{emit.PEATLAND_OCCUPATION:s}:tco2e-per-ha"
]
hectares = emit.get_hectares_per_pixel(darray=cropland_occupation_per_hectare)
cropland_occupation = cropland_occupation_per_hectare * hectares
hectares = dset["hectares-per-cell:ha"]
cropland_occupation = (
dset[f"{emit.CROPLAND:s}-{emit.PEATLAND_OCCUPATION:s}:tco2e-per-ha"] * hectares
)
pastureland_occupation = (
dset[f"{emit.PASTURELAND:s}-{emit.PEATLAND_OCCUPATION:s}:tco2e-per-ha"]
* hectares
Expand Down Expand Up @@ -647,6 +636,10 @@ def workflow(
},
)
)
# NB: on the whole tile, since a clip one cell wide has no spacing to measure a cell by
merged["hectares-per-cell:ha"] = emit.get_hectares_per_pixel(
darray=merged["dropped-emissions:tco2e-per-ha"]
)
# NB: usually we avoid dumping to local mmemory, but since this xarray is small there is no
# OOM risk. Doing so speeds up the many downstream reads for each jurisdiction
merged = merged.load()
Expand Down