diff --git a/README.md b/README.md index e9818cf..fa540aa 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ As a proof of concept, the jurisdictional-direct leg covers eleven crops grown i ![Forest cleared for cropland in a few concentrated blocks of Matopiba, Brazil, when it happened, and the emissions it released beneath today's soybean](docs/figures/soy-brazil-matopiba-methodology.png) -*Matopiba, Brazil. **Top left:** the five-year span each converted pixel was charged to, darkest most recent — clearance falls in a few blocks rather than advancing as a front, and most of it between 2006 and 2015. **Top right:** which of the five conversions fired — forest → cropland in those blocks, rangeland → cropland as the thin scatter across the western half, and a band of pasture → cropland down the southern frontier where soybean is now densest. **Bottom:** MapSPAM soybean area in 2000 and 2020, settling on the same blocks the top row shows converting. **Right:** the 20-year linearly-discounted per-hectare emissions those conversions released, before any crop is named — bright on the forest blocks and not on the rangeland scatter, because forest carries far more carbon per hectare. The statistical leg then divides these emissions among the crops and pastureland that expanded locally, in proportion to each one's share of that expansion.* +*Matopiba, Brazil. **Top left:** the year each converted pixel's source class ended, 2001 to 2020, darker more recent — clearance falls in a few blocks rather than advancing as a front. **Top right:** which of the five conversions fired — forest → cropland in those blocks, a band of pasture → cropland down the southern frontier where soybean is now densest, and forest → pasture and rangeland conversions scattered across the rest. **Bottom:** MapSPAM soybean area in 2000 and 2020, settling on the same blocks the top row shows converting. **Right:** the per-hectare emissions those conversions released, vegetation plus soil, before they are discounted by conversion year and before any crop is named — bright on the forest blocks and dim on the scatter, because forest carries far more carbon per hectare. The statistical leg then divides the emissions on land that became cropland among the crops that expanded locally, in proportion to each one's share of that expansion, and the emissions on land that became pasture among the herds grazing it.* ## Why are we publishing this? diff --git a/docs/figures/soy-brazil-matopiba-methodology.png b/docs/figures/soy-brazil-matopiba-methodology.png index 8f9e979..0d4df83 100644 Binary files a/docs/figures/soy-brazil-matopiba-methodology.png and b/docs/figures/soy-brazil-matopiba-methodology.png differ diff --git a/docs/further_research.md b/docs/further_research.md index 78bcf6a..0f4d8d0 100644 --- a/docs/further_research.md +++ b/docs/further_research.md @@ -162,7 +162,7 @@ The consequence is that the nine 2005 crops read as expansion from zero across t ## Emissions attributed to no crop are reported rather than spread over the modeled crops -**Issue**: Shares add up to at most one, and whatever they leave over goes to no commodity. It is published for each jurisdiction as the `UNATTRIBUTED` row, so the commodity rows plus that row add up to the emissions total, but no emissions factor includes it. The row is an allocation remainder, so its amount does not by itself identify the crop responsible. It holds the part of each cell taken by the fourteen MapSPAM crops this leg does not model, and also conversion in cells without positive measured expansion: where neither cropland nor pasture grew, every share is zero and the whole forest, grassland and cropland peat conversion charge lands in this row. That conversion may be gross change hidden by net area change, or disagreement between the input maps. Both kinds of share leave something over, and the peatland-occupation shares leave more. The figures below predate the GACED30 and Global Pasture Watch destinations and need a re-measure on the current pipeline: +**Issue**: Shares add up to at most one, and whatever they leave over goes to no commodity. It is published for each jurisdiction as the `UNATTRIBUTED` row, so the commodity rows plus that row add up to the emissions total, but no emissions factor includes it. The row is an allocation remainder, so its amount does not by itself identify the crop responsible. It holds the part of each cell taken by the fourteen MapSPAM crops this leg does not model, and also conversion to cropland in cells without positive measured crop expansion: where no crop grew, every crop's share is zero and everything converted to cropland lands in this row. That conversion may be gross change hidden by net area change, or disagreement between the input maps. Both kinds of share leave something over, and the peatland-occupation shares leave more. The figures below predate the GACED30 and Global Pasture Watch destinations and need a re-measure on the current pipeline: | window | conversion shares | peatland-occupation shares | | --------- | ----------------: | -------------------------: | @@ -185,7 +185,7 @@ WRI reaches the same fork and answers it without an external layer, which makes ## A crop's emissions are not bounded by the area it holds -**Issue**: the statistical leg gives each crop a cell's conversion emissions in proportion to the crop's share of the cell's expansion. The share is relative and is applied to an absolute amount of emissions, so a small absolute expansion can take a large share: a crop that expanded by 1 ha, in a cell where no other crop or pasture expanded, takes all of that cell's emissions however small its area or production there, and nothing in the factor, `emissions_mt / production_kg`, ties the two back together. The same symptom has a second cause: where the cropland map misses a crop's land, the crop's production stays in the denominator while the conversion behind it does not. That is the question of how much of a crop's land the cropland map finds, in "Woody perennial crops resolve to no destination". +**Issue**: the statistical leg gives each crop a cell's conversion emissions in proportion to the crop's share of the cell's expansion. The share is relative and is applied to an absolute amount of emissions, so a small absolute expansion can take a large share: a crop that expanded by 1 ha, in a cell where no other crop expanded, takes all of what that cell converted to cropland however small its area or production there, and nothing in the factor, `emissions_mt / production_kg`, ties the two back together. The same symptom has a second cause: where the cropland map misses a crop's land, the crop's production stays in the denominator while the conversion behind it does not. That is the question of how much of a crop's land the cropland map finds, in "Woody perennial crops resolve to no destination". **Measured** on an earlier table, counting crops only (`PASTURE` and `DROPPED` left out, since including them shrinks every share about 4.6×): rows with a factor above 10 kg CO₂e/kg, which marks where to look rather than an error, carry 3.2% of the crops' emissions nationally and 9.3% provincially. 90% of those emissions are in rows over 10,000 ha, so this is not an effect of rows with tiny areas, and the `OTHER_*` catch-all crops carry 30% of it. The table had no livestock commodities and did not divide the peat drainage pulse by destination, so these figures need measuring again on the next release. diff --git a/docs/methodology.md b/docs/methodology.md index eaf3f5f..70847ef 100644 --- a/docs/methodology.md +++ b/docs/methodology.md @@ -69,7 +69,7 @@ A worked example of that flow on real data — soy in Matopiba, Brazil: ![Forest cleared for cropland in a few concentrated blocks of Matopiba, Brazil, when it happened, and the emissions it released beneath today's soybean](figures/soy-brazil-matopiba-methodology.png) -*Matopiba, Brazil. **Top left:** the five-year span each converted pixel was charged to, darkest most recent — clearance falls in a few blocks rather than advancing as a front, and most of it between 2006 and 2015. **Top right:** which of the five conversions fired — forest → cropland in those blocks, rangeland → cropland as the thin scatter across the western half, and a band of pasture → cropland down the southern frontier where soybean is now densest. **Bottom:** MapSPAM soybean area in 2000 and 2020, settling on the same blocks the top row shows converting. **Right:** the 20-year linearly-discounted per-hectare emissions those conversions released, before any crop is named — bright on the forest blocks and not on the rangeland scatter, because forest carries far more carbon per hectare. The statistical leg then divides these emissions among the crops and pastureland that expanded locally, in proportion to each one's share of that expansion.* +*Matopiba, Brazil. **Top left:** the year each converted pixel's source class ended, 2001 to 2020, darker more recent — clearance falls in a few blocks rather than advancing as a front. **Top right:** which of the five conversions fired — forest → cropland in those blocks, a band of pasture → cropland down the southern frontier where soybean is now densest, and forest → pasture and rangeland conversions scattered across the rest. **Bottom:** MapSPAM soybean area in 2000 and 2020, settling on the same blocks the top row shows converting. **Right:** the per-hectare emissions those conversions released, vegetation plus soil, before they are discounted by conversion year and before any crop is named — bright on the forest blocks and dim on the scatter, because forest carries far more carbon per hectare. The statistical leg then divides the emissions on land that became cropland among the crops that expanded locally, in proportion to each one's share of that expansion, and the emissions on land that became pasture among the herds grazing it.* ______________________________________________________________________ @@ -245,26 +245,24 @@ Because direct attribution relies only on **current** production, production is ### Statistical (`statistical.py`) -When crop production is only available at low spatial resolution, spatial intersection is ambiguous and a statistical model is needed instead. The assumption is that emissions are driven by agricultural expansion: within a coarse spatial cell, emissions are attributed to each commodity in proportion to its share of total expansion (retractions are ignored). Whereas direct attribution relies only on current production, the statistical leg is estimated from multiple spans — the emissions factor is reduced over a traceability region by summing attributed emissions across spans and dividing by the sum of linearly discounted production across those same spans, again aggregating to district-, provincial-, or national-level jurisdictions. +When crop production is only available at low spatial resolution, spatial intersection is ambiguous and a statistical model is needed instead. The assumption is that emissions are driven by agricultural expansion: within a coarse spatial cell, emissions on land that became cropland are attributed to each crop in proportion to its share of crop expansion (retractions are ignored), and emissions on land that became pasture to the herds grazing it. Whereas direct attribution relies only on current production, the statistical leg is estimated from multiple spans — the emissions factor is reduced over a traceability region by summing attributed emissions across spans and dividing by the sum of linearly discounted production across those same spans, again aggregating to district-, provincial-, or national-level jurisdictions. Concretely: -1. Discount each pixel's emissions by its conversion year and downsample them to the MapSPAM ~10 km grid — forest and grassland conversion emissions and the peat drainage pulse split by destination (cropland or pastureland), each in one band per MapSPAM span holding the conversion year, the cropland and pastureland peatland-occupation bands, and pasture extent at each MapSPAM snapshot. -2. For each MapSPAM span, compute each commodity's **expansion** and its **share** of total expansion in that cell. A crop's expansion is the positive change in its MapSPAM physical area; pastureland's is the positive change in the hectares Global Pasture Watch calls cultivated grassland, measured the same way. MapSPAM has no 2015 snapshot and none after 2020, so conversions from 2011 onward all take the 2010→2020 expansion. The total is built by clipping each commodity's own movement, exactly as its numerator is, so the shares sum to **at most** one. -3. Attribute each span's discounted forest and grassland conversion emissions to commodities by that span's expansion share, and its peat drainage pulse by destination (below). +1. Discount each pixel's emissions by its conversion year and downsample them to the MapSPAM ~10 km grid — forest and grassland conversion emissions and the peat drainage pulse, each split by destination (cropland or pastureland) into one band per MapSPAM span holding the conversion year, the cropland and pastureland peatland-occupation bands, and pasture extent at each MapSPAM snapshot. +2. For each MapSPAM span, compute each crop's **expansion** and its **share** of crop expansion in that cell. A crop's expansion is the positive change in its MapSPAM physical area. MapSPAM has no 2015 snapshot and none after 2020, so conversions from 2011 onward all take the 2010→2020 expansion. The total is built by clipping each crop's own movement, exactly as its numerator is, so the shares sum to **at most** one. +3. Attribute each span's discounted conversion emissions by destination: what became cropland to crops by that span's expansion share, and what became pasture to livestock by grazing share (below). 4. Compute each commodity's production denominator as a discount-weighted average over the lookback window (described below), rather than a single current-year snapshot. -**Pastureland is an expansion term, divided among its grazers.** Grazing land is inside the boundary the LSRS draws, and forest cleared for pasture emits whether or not a crop ever follows, so pasture takes its share of the cell alongside the crops. This matches the external references, which divide by cropland plus cultivated pasture rather than by cropland alone. The consequence for every crop is that its share is diluted by whatever pasture expanded beside it — in pasture-led frontiers that is most of the cell. +**Conversion emissions are divided by destination before they are shared.** Grazing land is inside the boundary the LSRS draws, and forest cleared for pasture emits whether or not a crop ever follows, so pasture carries its own conversions. Each span's forest, grassland and peat conversion emissions are split on the 30 m grid by what each converted pixel became, cropland or cultivated grassland, and each half goes only to the commodities of its own land class. The cropland half goes to crops by each crop's share of crop expansion, so pasture expanding beside them takes none of it; where no crop expanded in the span, it goes to `UNATTRIBUTED`. The pastureland half goes whole to the livestock commodities, whatever the crops did, so none of it is discarded. The external references divide a cell's emissions by the expansion of cropland and cultivated pasture together; this leg divides each destination's emissions separately, as the proxy leg does. -Pasture's share is then divided among livestock commodities by grazing pressure at the end of each span. Each species' livestock units per hectare are its Global Pasture Watch headcount density times its country's livestock units per head, as FAOSTAT Livestock Patterns counts them: FAO's coefficient for the country's region (FAO, 2011), fixed over time, and recovered as the median over the years of Livestock Patterns' stocks in livestock units over FAOSTAT Production's stocks in heads. Cattle take 0.50 to 1.00 by region, horses 0.40 to 0.80, buffalo 0.50 to 0.70, and sheep and goats 0.10 almost everywhere; a species FAOSTAT gives none, such as South America's buffalo, counts none. `BEEF_CATTLE` takes cattle's fraction of the cell's units. `PASTURE` keeps the rest: the units of species no commodity names, and all of pasture's share where no grazer is mapped. Within a cell, every livestock unit is taken to graze an equal slice of the pasture. Dividing the pool rather than measuring each grazer's own expansion means a shift between grazers on unchanged pasture is not read as new pasture, and no crop's share depends on the herd. +The pastureland half is divided among livestock commodities by grazing pressure at the end of each span. Each species' livestock units per hectare are its Global Pasture Watch headcount density times its country's livestock units per head, as FAOSTAT Livestock Patterns counts them: FAO's coefficient for the country's region (FAO, 2011), fixed over time, and recovered as the median over the years of Livestock Patterns' stocks in livestock units over FAOSTAT Production's stocks in heads. Cattle take 0.50 to 1.00 by region, horses 0.40 to 0.80, buffalo 0.50 to 0.70, and sheep and goats 0.10 almost everywhere; a species FAOSTAT gives none, such as South America's buffalo, counts none. `BEEF_CATTLE` takes cattle's fraction of the cell's units. `PASTURE` keeps the rest: the units of species no commodity names, and all of pasture's share where no grazer is mapped. Within a cell, every livestock unit is taken to graze an equal slice of the pasture. Dividing the pool rather than measuring each grazer's own expansion means a shift between grazers on unchanged pasture is not read as new pasture, and no crop's share depends on the herd. `BEEF_CATTLE`'s production is the cattle GPW maps in cells holding any cultivated grassland, times FAOSTAT's national carcass weight per standing head — meat production (carcass weight, bone in) over stocks — at each snapshot, reduced over the window like a crop's. A cell's herd counts whole rather than weighted by the pasture fraction, since it grazes the cell's pasture rather than standing in its forest or cropland, and herds in cells without mapped pasture — feedlots, rangeland — earn no production. Production therefore sits where the herd grazes, as a crop's sits where it grows, and follows the same standing herd the grazing split does. The carcass carries all of beef's emissions — offal, fat and hides take none — and meat is not split from milk, so `BEEF_CATTLE` holds dairy herds' pasture and culled dairy animals' meat along with beef herds'. The simplifications still open — GPW's herd against FAOSTAT's, one national rate per head, and herds off mapped pasture — are measured in "Livestock attribution rests on national rates" in `further_research.md`. `PASTURE` carries area but no production, so no emissions factor is published for it. -**The peat drainage pulse is divided by destination before it is shared.** The pulse is split on the 30 m grid by what each converted peat pixel became, cropland or cultivated grassland, and each half goes only to the commodities of its own land class. The cropland half goes to crops by each crop's share of *crop* expansion alone, so pasture expanding beside them takes none of it; where no crop expanded in the span, it goes to `UNATTRIBUTED`. The pastureland half goes whole to the livestock commodities by grazing share at the end of the span, whatever the crops did, so none of it is discarded: `PASTURE` takes what no named herd grazes. Forest and grassland conversion emissions stay pooled across destinations and are divided by the combined crop-and-pasture share above. - Peatland occupation is allocated differently from conversion, because it is a land-management flux on land that is drained *now* and has no relationship to expansion. It is split by each commodity's share of occupied area instead — a crop's share of MapSPAM cropland at 2020, and a livestock commodity's grazing share of the pasture at 2020 — and the two occupation bands keep the crop and pasture halves apart: the split is made on the 30 m grid, where cropland and cultivated grassland are separable, so nothing has to assume how peat, pasture and cropland sit relative to one another inside a 10 km cell. -**Emissions that no modeled commodity takes are reported, not handed to other commodities.** This leg divides a cell's emissions by each commodity's share of the cell, and those shares add up to less than one wherever the cell holds crops this leg does not model — citrus, cocoa, rubber, vegetables and ten others. The part left over goes to the `UNATTRIBUTED` row. That happens with conversion emissions, which are shared by expansion, and with peat occupation, which is shared by area. It also happens where a cell has no expansion to share by: where neither cropland nor pasture grew, the cell's whole forest and grassland conversion charge goes to `UNATTRIBUTED`, and so does the cropland peat pulse wherever no crop grew. That conversion may be gross change hidden by net area change, or disagreement between the input maps, so the row's amount does not by itself identify the crop responsible. Pasture leaves nothing over, because `PASTURE` takes whatever no named herd grazes. Handing the leftover to the crops we do model would charge them for land that other crops took, so it is kept apart, and each crop's factor is unaffected by it. Its largest single source is described in "MapSPAM crops that first appear in a later snapshot read as expansion from zero" in `further_research.md`. +**Emissions that no modeled commodity takes are reported, not handed to other commodities.** This leg divides a cell's emissions by each commodity's share of the cell, and those shares add up to less than one wherever the cell holds crops this leg does not model — citrus, cocoa, rubber, vegetables and ten others. The part left over goes to the `UNATTRIBUTED` row. That happens with conversion emissions, which are shared by expansion, and with peat occupation, which is shared by area. It also happens where a cell has no crop expansion to share by: wherever no crop grew in a span, everything converted to cropland goes to `UNATTRIBUTED`. That conversion may be gross change hidden by net area change, or disagreement between the input maps, so the row's amount does not by itself identify the crop responsible. Pasture leaves nothing over, because `PASTURE` takes whatever no named herd grazes. Handing the leftover to the crops we do model would charge them for land that other crops took, so it is kept apart, and each crop's factor is unaffected by it. Its largest single source is described in "MapSPAM crops that first appear in a later snapshot read as expansion from zero" in `further_research.md`. This leg is global; MapSPAM crops are modeled with a broader `Crop` enum (maize, soybean, wheat, rice, oil palm, coffee, and more; see `statistical.py`), and livestock with the `Livestock` enum (`BEEF_CATTLE` and the residual `PASTURE`). diff --git a/jdluc/__tests__/statistical_test.py b/jdluc/__tests__/statistical_test.py index f2ea98f..e8c24b5 100644 --- a/jdluc/__tests__/statistical_test.py +++ b/jdluc/__tests__/statistical_test.py @@ -20,7 +20,6 @@ Commodity, Crop, get_commodity_name_to_totals, - get_commodity_to_peatland_conversion_share, get_commodity_to_share, get_crop_to_area_share, get_mapspam_span, @@ -248,16 +247,14 @@ def test_get_commodity_to_share( ) -> None: dset = get_dset_for_areas(areas=areas) # Every case asks for all of `Crop`, so each one also checks the invariant that the shares - # never over-attribute the cell + # never over-attribute what was converted to cropland shares = get_commodity_to_share( after=after, before=before, crops=tuple(Crop), dset=dset ) - assert sum(get_value(share) for share in shares.values()) <= 1.0 + assert sum(get_value(darray=shares[crop]) for crop in Crop) <= 1.0 assert {crop: get_value(shares[crop]) for crop in expected} == pytest.approx( expected ) - # No pasture moves in any of these cases, so it takes nothing off the crops - assert all(get_value(shares[livestock]) == 0.0 for livestock in Livestock) def test_get_commodity_to_share_does_not_renormalise_onto_the_crops_asked_for() -> None: @@ -575,19 +572,23 @@ def test_get_mapspam_span(year: int, span: tuple[int, int]) -> None: # The default for `get_name_to_totals`'s shares, read-only so every call can share it NO_SHARES: collections.abc.Mapping[Commodity, float] = types.MappingProxyType({}) +NO_EMISSIONS: collections.abc.Mapping[emit.EmissionComponent, float] = ( + types.MappingProxyType({}) +) def get_name_to_totals( dset: xarray.Dataset, commodity_to_share: collections.abc.Mapping[Commodity, float] = NO_SHARES, + cropland_emissions: collections.abc.Mapping[ + emit.EmissionComponent, float + ] = NO_EMISSIONS, cropland_occupation: float = 0.0, - cropland_pulse: float = 0.0, - forest_emissions: float = 0.0, - grassland_emissions: float = 0.0, occupation_shares: collections.abc.Mapping[Commodity, float] = NO_SHARES, + pasture_emissions: collections.abc.Mapping[ + emit.EmissionComponent, float + ] = NO_EMISSIONS, pasture_occupation: float = 0.0, - pasture_pulse: float = 0.0, - peatland_conversion_shares: collections.abc.Mapping[Commodity, float] = NO_SHARES, ) -> dict[str, dict[str, float]]: """`get_commodity_name_to_totals` over WHEAT and the livestock rows on `get_dset`'s 2x2 grid. @@ -619,14 +620,19 @@ def get_shares( + pasture_occupation, } | { - f"{name:s}-emissions:tco2e-per-ha:{before:d}-{after:d}": grid - + (emissions if (before, after) == (2010, 2020) else 0.0) - for name, emissions in { - "forest": forest_emissions, - "grassland": grassland_emissions, - f"{emit.CROPLAND:s}-peatland-conversion": cropland_pulse, - f"{emit.PASTURELAND:s}-peatland-conversion": pasture_pulse, - }.items() + f"{land_class:s}-{component!s}-emissions:tco2e-per-ha:{before:d}-{after:d}": grid + + ( + (dict.fromkeys(emit.EmissionComponent, 0.0) | dict(emissions))[ + component + ] + if (before, after) == (2010, 2020) + else 0.0 + ) + for land_class, emissions in ( + (emit.CROPLAND, cropland_emissions), + (emit.PASTURELAND, pasture_emissions), + ) + for component in emit.EmissionComponent for before, after in MAPSPAM_SPANS } ), @@ -634,9 +640,6 @@ def get_shares( snapshot_year_to_weight=get_snapshot_year_to_weight( assessment_year=emit.DEFAULT_ASSESSMENT_YEAR ), - span_to_commodity_to_peatland_conversion_share=dict.fromkeys( - MAPSPAM_SPANS, get_shares(shares=peatland_conversion_shares) - ), span_to_commodity_to_share=dict.fromkeys( MAPSPAM_SPANS, get_shares(shares=commodity_to_share) ), @@ -700,77 +703,6 @@ def test_a_constant_yield_survives_the_window() -> None: SHEEP = gpw_livestock.Species.SHEEP -@pytest.mark.parametrize( - ("areas", "pasture_fraction", "density", "expected"), - ( - pytest.param( - {2005: {}, 2010: {"MAIZ": 100.0}}, - {2010: 100.0 / HECTARES_PER_CELL}, - {}, - {Crop.MAIZE: 0.5, Livestock.BEEF_CATTLE: 0.0, Livestock.PASTURE: 0.5}, - id="pasture-expanding-by-the-crops-own-hectares-takes-half-the-cell-from-it", - ), - pytest.param( - {2005: {}, 2010: {}}, - {2010: 0.25}, - {}, - {Crop.MAIZE: 0.0, Livestock.BEEF_CATTLE: 0.0, Livestock.PASTURE: 1.0}, - id="the-frontier-cell-where-only-pasture-expands-charges-pasture-not-nobody", - ), - pytest.param( - {2005: {}, 2010: {"MAIZ": 100.0}}, - {2005: 0.5, 2010: 0.0}, - {}, - {Crop.MAIZE: 1.0, Livestock.BEEF_CATTLE: 0.0, Livestock.PASTURE: 0.0}, - id="contracting-pasture-clips-to-zero-rather-than-shrinking-the-denominator", - ), - pytest.param( - {2005: {}, 2010: {"MAIZ": 100.0}}, - {2010: 100.0 / HECTARES_PER_CELL}, - get_density(heads_per_ha={CATTLE: 0.01}), - {Crop.MAIZE: 0.5, Livestock.BEEF_CATTLE: 0.5, Livestock.PASTURE: 0.0}, - id="cattle-alone-take-the-pasture-share-and-the-crop-keeps-its-own", - ), - pytest.param( - {2005: {}, 2010: {"MAIZ": 100.0}}, - {2010: 100.0 / HECTARES_PER_CELL}, - get_density(heads_per_ha={CATTLE: 0.1, SHEEP: 0.7}), - {Crop.MAIZE: 0.5, Livestock.BEEF_CATTLE: 0.25, Livestock.PASTURE: 0.25}, - id="grazers-divide-the-pasture-share-by-livestock-units-and-the-crop-keeps-its-own", - ), - pytest.param( - {2005: {}, 2010: {"MAIZ": 100.0}}, - {2005: 0.5, 2010: 0.5}, - {(SHEEP, 2005): 0.7, (CATTLE, 2010): 0.1}, - {Crop.MAIZE: 1.0, Livestock.BEEF_CATTLE: 0.0, Livestock.PASTURE: 0.0}, - id="cattle-replacing-sheep-on-unchanged-pasture-is-not-expansion", - ), - pytest.param( - {2005: {}, 2010: {}}, - {2010: 0.25}, - {(SHEEP, 2005): 0.7, (CATTLE, 2010): 0.1}, - {Crop.MAIZE: 0.0, Livestock.BEEF_CATTLE: 1.0, Livestock.PASTURE: 0.0}, - id="new-pasture-goes-to-whoever-grazes-it-at-the-end-of-the-span", - ), - ), -) -def test_get_commodity_to_share_charges_pasture_and_its_grazers( - areas: dict[int, dict[str, float]], - pasture_fraction: dict[int, float], - density: dict[tuple[gpw_livestock.Species, int], float], - expected: dict[Commodity, float], -) -> None: - dset = get_dset_for_areas( - areas=areas, density=density, pasture_fraction=pasture_fraction - ) - shares = get_commodity_to_share( - after=2010, before=2005, crops=(Crop.MAIZE,), dset=dset - ) - assert { - commodity: get_value(shares[commodity]) for commodity in expected - } == pytest.approx(expected) - - @pytest.mark.parametrize( ("areas", "pasture_fraction", "density", "expected"), ( @@ -779,39 +711,39 @@ def test_get_commodity_to_share_charges_pasture_and_its_grazers( {2020: 100.0 / HECTARES_PER_CELL}, {}, {Crop.MAIZE: 1.0, Livestock.BEEF_CATTLE: 0.0, Livestock.PASTURE: 1.0}, - id="pasture-expanding-beside-the-crop-takes-none-of-the-cropland-pulse", + id="pasture-expanding-beside-the-crop-takes-none-of-what-was-converted-to-cropland", ), pytest.param( {2010: {}, 2020: {}}, {2020: 0.25}, {}, {Crop.MAIZE: 0.0, Livestock.BEEF_CATTLE: 0.0, Livestock.PASTURE: 1.0}, - id="where-no-crop-expanded-the-cropland-pulse-goes-to-nobody", + id="where-no-crop-expanded-what-was-converted-to-cropland-goes-to-nobody", ), pytest.param( {2010: {}, 2020: {"MAIZ": 100.0, UNATTRIBUTED_0: 100.0}}, {}, {}, {Crop.MAIZE: 0.5}, - id="crops-it-cannot-attribute-keep-their-share-of-the-cropland-pulse", + id="crops-it-cannot-attribute-keep-their-share-of-what-was-converted-to-cropland", ), pytest.param( {2010: {}, 2020: {"MAIZ": 100.0}}, {2010: 0.5, 2020: 0.5}, {(CATTLE, 2020): 0.1, (SHEEP, 2020): 0.7}, {Crop.MAIZE: 1.0, Livestock.BEEF_CATTLE: 0.5, Livestock.PASTURE: 0.5}, - id="the-pasture-pulse-goes-to-the-grazers-where-only-crops-expanded", + id="what-was-converted-to-pasture-goes-to-the-grazers-where-only-crops-expanded", ), pytest.param( {2010: {}, 2020: {}}, {2010: 0.5, 2020: 0.5}, {(SHEEP, 2010): 0.7, (CATTLE, 2020): 0.1}, {Crop.MAIZE: 0.0, Livestock.BEEF_CATTLE: 1.0, Livestock.PASTURE: 0.0}, - id="the-pasture-pulse-goes-to-whoever-grazes-at-the-end-of-the-span", + id="what-was-converted-to-pasture-goes-to-whoever-grazes-at-the-end-of-the-span", ), ), ) -def test_get_commodity_to_peatland_conversion_share( +def test_get_commodity_to_share_divides_each_land_class_among_its_own_commodities( areas: dict[int, dict[str, float]], pasture_fraction: dict[int, float], density: dict[tuple[gpw_livestock.Species, int], float], @@ -820,7 +752,7 @@ def test_get_commodity_to_peatland_conversion_share( dset = get_dset_for_areas( areas=areas, density=density, pasture_fraction=pasture_fraction ) - shares = get_commodity_to_peatland_conversion_share( + shares = get_commodity_to_share( after=2020, before=2010, crops=(Crop.MAIZE,), dset=dset ) assert { @@ -836,6 +768,9 @@ def test_get_commodity_to_peatland_conversion_share( BEEF = Livestock.BEEF_CATTLE.name PASTURE = Livestock.PASTURE.name UNATTRIBUTED = emit.NonCommodity.UNATTRIBUTED.name +FOREST = emit.EmissionComponent.FOREST +GRASSLAND = emit.EmissionComponent.GRASSLAND +PEAT = emit.EmissionComponent.PEATLAND_CONVERSION # Expected values are per hectare of the cell. Each emissions band is already discounted and the @@ -865,78 +800,63 @@ def test_get_commodity_to_peatland_conversion_share( 0.0, { "commodity_to_share": { - Crop.WHEAT: 0.5, - Livestock.BEEF_CATTLE: 0.3, - Livestock.PASTURE: 0.2, - }, - "cropland_pulse": 2.0, - "pasture_pulse": 5.0, - "peatland_conversion_shares": { Crop.WHEAT: 0.25, Livestock.BEEF_CATTLE: 0.6, Livestock.PASTURE: 0.4, }, + "cropland_emissions": {FOREST: 2.0, PEAT: 4.0}, + "pasture_emissions": {GRASSLAND: 5.0}, }, { - (WHEAT, "peatland_conversion_emissions_mt"): 0.5, - (BEEF, "peatland_conversion_emissions_mt"): 3.0, - (PASTURE, "peatland_conversion_emissions_mt"): 2.0, - }, - id="crops-divide-the-cropland-pulse-and-livestock-the-pasture-pulse-by-their-own-shares", - ), - pytest.param( - 0.0, - { - "commodity_to_share": {Livestock.PASTURE: 0.25}, - "forest_emissions": 10.0, + (WHEAT, "forest_emissions_mt"): 0.5, + (WHEAT, "grassland_emissions_mt"): 0.0, + (WHEAT, "peatland_conversion_emissions_mt"): 1.0, + (BEEF, "forest_emissions_mt"): 0.0, + (BEEF, "grassland_emissions_mt"): 3.0, + (PASTURE, "grassland_emissions_mt"): 2.0, }, - {(PASTURE, "emissions_mt"): 2.5}, - id="the-pasture-row-takes-its-pooled-share-of-every-conversion-not-only-pasture-ones", + id="crops-divide-what-was-converted-to-cropland-and-livestock-what-was-converted-to-pasture", ), pytest.param( 0.0, { "commodity_to_share": {Crop.WHEAT: 0.5}, + "cropland_emissions": {FOREST: 1.0, GRASSLAND: 10.0, PEAT: 100.0}, "cropland_occupation": 1000.0, - "cropland_pulse": 100.0, - "forest_emissions": 1.0, - "grassland_emissions": 10.0, "occupation_shares": {Crop.WHEAT: 0.1}, - "peatland_conversion_shares": {Crop.WHEAT: 0.25}, }, { (WHEAT, "forest_emissions_mt"): 0.5, (WHEAT, "grassland_emissions_mt"): 5.0, - (WHEAT, "peatland_conversion_emissions_mt"): 25.0, + (WHEAT, "peatland_conversion_emissions_mt"): 50.0, (WHEAT, "peatland_occupation_emissions_mt"): 100.0, - (WHEAT, "emissions_mt"): 130.5, + (WHEAT, "emissions_mt"): 155.5, }, - id="emissions-mt-is-the-components-plus-occupation-each-by-its-own-share", + id="emissions-mt-is-the-conversion-components-plus-occupation-each-by-its-own-share", ), pytest.param( 0.0, { - "commodity_to_share": {Crop.WHEAT: 0.5}, + "commodity_to_share": {Crop.WHEAT: 0.5, Livestock.PASTURE: 1.0}, + "cropland_emissions": {FOREST: 1.0, GRASSLAND: 10.0, PEAT: 100.0}, "cropland_occupation": 1000.0, - "cropland_pulse": 100.0, - "forest_emissions": 1.0, - "grassland_emissions": 10.0, "occupation_shares": {Crop.WHEAT: 0.1}, - "peatland_conversion_shares": {Crop.WHEAT: 0.25}, }, # Charged: forest 1, grassland 10, peat conversion 100 and occupation 1000, so 1111 in - # all; wheat's row and the unattributed row split each between them + # all, every unit of it on cropland; wheat's row and the unattributed row split each + # between them, and the pasture row takes none of it { (WHEAT, "forest_emissions_mt"): 0.5, (WHEAT, "grassland_emissions_mt"): 5.0, - (WHEAT, "peatland_conversion_emissions_mt"): 25.0, + (WHEAT, "peatland_conversion_emissions_mt"): 50.0, (WHEAT, "peatland_occupation_emissions_mt"): 100.0, - (WHEAT, "emissions_mt"): 130.5, + (WHEAT, "emissions_mt"): 155.5, + (PASTURE, "emissions_mt"): 0.0, (UNATTRIBUTED, "forest_emissions_mt"): 0.5, (UNATTRIBUTED, "grassland_emissions_mt"): 5.0, - (UNATTRIBUTED, "peatland_conversion_emissions_mt"): 75.0, + (UNATTRIBUTED, "peatland_conversion_emissions_mt"): 50.0, (UNATTRIBUTED, "peatland_occupation_emissions_mt"): 900.0, - (UNATTRIBUTED, "emissions_mt"): 980.5, + (UNATTRIBUTED, "emissions_mt"): 955.5, (UNATTRIBUTED, "production_mt"): math.nan, }, id="the-unattributed-row-is-what-the-shares-leave-of-the-charge-so-the-rows-sum-to-it", @@ -995,10 +915,11 @@ def test_a_clip_one_cell_wide_keeps_the_hectares_of_its_cells() -> None: whole, column = ( get_name_to_totals( commodity_to_share={Crop.WHEAT: 0.5, Livestock.PASTURE: 0.25}, + cropland_emissions={FOREST: 10.0}, cropland_occupation=2.0, dset=clipped, - forest_emissions=10.0, occupation_shares={Crop.WHEAT: 0.5}, + pasture_emissions={GRASSLAND: 10.0}, ) for clipped in (dset, dset.isel(x=[0])) ) diff --git a/jdluc/geo.py b/jdluc/geo.py index d429d39..aa1a024 100644 --- a/jdluc/geo.py +++ b/jdluc/geo.py @@ -323,6 +323,7 @@ def downscale_dset( row_weights=rows.weights, valid=valid, ) + logger.info(f"Downscaled rows {row_stop:d} of {fine_height:d}") coords = affine_to_coords(affine=affine, height=height, width=width) name_to_darray: dict[collections.abc.Hashable, xarray.DataArray] = {} diff --git a/jdluc/statistical.py b/jdluc/statistical.py index e97d4b3..cbc11cc 100644 --- a/jdluc/statistical.py +++ b/jdluc/statistical.py @@ -71,15 +71,8 @@ class Crop(enum.StrEnum): f"{emit.PASTURELAND:s}-{emit.PEATLAND_OCCUPATION:s}:tco2e-per-ha", ] -# Components divided by the pooled expansion share. The peat pulse is not one of them: it is stored -# as two bands, peat drained for cropland and peat drained for pasture, each with its own share. -POOLED_COMPONENTS = (emit.EmissionComponent.FOREST, emit.EmissionComponent.GRASSLAND) -assert set(POOLED_COMPONENTS) == set(emit.EmissionComponent) - { - emit.EmissionComponent.PEATLAND_CONVERSION -} - -@storage.cache_to_zarr(version=0) +@storage.cache_to_zarr(version=1) def get_downscaled_luc_emissions(assessment_year: int, tile_id: str) -> xarray.Dataset: logger.info("Computing emissions on the GLAD grid") luc_and_emissions = geo.exact_merge( @@ -120,25 +113,19 @@ def get_downscaled_luc_emissions(assessment_year: int, tile_id: str) -> xarray.D if get_mapspam_span(year=year) == (before, after) ] ) - for component in POOLED_COMPONENTS: - name = f"{component!s}-emissions:tco2e-per-ha:{before:d}-{after:d}" - luc_and_emissions[name] = component_to_emissions[component].where( - in_span, other=0 - ) - derived_variable_names.append(name) - # NB: `emit` puts soil emissions only on converted pixels, and every conversion ends in - # cropland or pasture, so the two bands add up to the whole pulse - for land_class, conversions in ( - (emit.CROPLAND, emit.TO_CROPLAND), - (emit.PASTURELAND, emit.TO_PASTURE), - ): - name = f"{land_class:s}-{emit.EmissionComponent.PEATLAND_CONVERSION!s}-emissions:tco2e-per-ha:{before:d}-{after:d}" - luc_and_emissions[name] = component_to_emissions[ - emit.EmissionComponent.PEATLAND_CONVERSION - ].where( - in_span & luc_and_emissions["conversion"].isin(conversions), other=0 - ) - derived_variable_names.append(name) + # NB: `emit` charges only converted pixels, and every conversion ends in cropland or + # pasture, so each component's two bands add up to the whole component + for component in emit.EmissionComponent: + for land_class, conversions in ( + (emit.CROPLAND, emit.TO_CROPLAND), + (emit.PASTURELAND, emit.TO_PASTURE), + ): + name = f"{land_class:s}-{component!s}-emissions:tco2e-per-ha:{before:d}-{after:d}" + luc_and_emissions[name] = component_to_emissions[component].where( + in_span & luc_and_emissions["conversion"].isin(conversions), + other=0, + ) + derived_variable_names.append(name) logger.info("Discounting the source carbon no destination claimed") name = "dropped-emissions:tco2e-per-ha" @@ -243,46 +230,12 @@ def get_unattributed(year: int) -> xarray.DataArray: def get_commodity_to_share( after: int, before: int, crops: tuple[Crop, ...], dset: xarray.Dataset ) -> dict[Commodity, xarray.DataArray]: - """Each commodity's share of everything a cell emitted in a span, whatever each pixel became.""" - # Pastureland expands alongside the crops, measured net and clipped exactly as a crop's is: - # GPW is annual, and its gross gain would charge pasture for churn MapSPAM cannot see - pasture_expansion = ( - (dset[f"pasture:fraction:{after:d}"] - dset[f"pasture:fraction:{before:d}"]) - * dset["hectares-per-cell:ha"] - ).clip(min=0) - total_expansion = ( - get_cropland_expansion(after=after, before=before, dset=dset) - + pasture_expansion - ) - total_expansion = total_expansion.where(total_expansion > 0) - # Share is zero when there is no expansion at all - pasture_share = (pasture_expansion / total_expansion).fillna(0) - return { - crop: ( - get_crop_expansion(after=after, before=before, crop=crop, dset=dset) - / total_expansion - ).fillna(0) - for crop in crops - } | { - # NB: pasture's share is split by who grazes at the end of the span. Measuring each - # livestock commodity's own expansion instead would count one grazer replacing another as - # new pasture. - livestock: pasture_share * grazing_share - for livestock, grazing_share in gpw_livestock.get_livestock_to_grazing_share( - dset=dset, year=after - ).items() - } - - -def get_commodity_to_peatland_conversion_share( - after: int, before: int, crops: tuple[Crop, ...], dset: xarray.Dataset -) -> dict[Commodity, xarray.DataArray]: - """Each commodity's share of the pulse on peat drained for its own land class. + """Each commodity's share of what a cell emitted in a span on land converted to its own class. - A crop takes its share of crop expansion alone, so pasture expanding beside it takes none of - the pulse on peat drained for cropland, and where no crop expanded that pulse goes to nobody. - The grazers divide the pulse on peat drained for pasture by grazing share at the end of the - span, which always sums to one. + A crop takes its share of crop expansion, so pasture expanding beside it takes none of what + was converted to cropland, and where no crop expanded that goes to nobody. The grazers divide + what was converted to pasture by grazing share at the end of the span, which always sums to + one. """ cropland_expansion = get_cropland_expansion(after=after, before=before, dset=dset) cropland_expansion = cropland_expansion.where(cropland_expansion > 0) @@ -355,9 +308,6 @@ def get_commodity_name_to_totals( dset: xarray.Dataset, occupation_shares: dict[Commodity, xarray.DataArray], snapshot_year_to_weight: dict[int, float], - span_to_commodity_to_peatland_conversion_share: dict[ - tuple[int, int], dict[Commodity, xarray.DataArray] - ], span_to_commodity_to_share: dict[ tuple[int, int], dict[Commodity, xarray.DataArray] ], @@ -365,13 +315,12 @@ def get_commodity_name_to_totals( """Each commodity's emissions and area, a crop's production and a herd's heads, summed over a jurisdiction's cells. - Both share mappings are keyed by MapSPAM span, and each divides that span's discounted - emissions bands. + The conversion shares are keyed by MapSPAM span, and each divides that span's discounted + emissions bands for its commodity's own land class. Beside the commodities sit `DROPPED` and `UNATTRIBUTED` (see `emit.NonCommodity`). Here `UNATTRIBUTED` is what the shares leave of the charge: conversion and occupation by the - MapSPAM crops no row models, the forest and grassland charge where neither cropland nor - pasture expanded, and the cropland peat pulse where no crop expanded. + MapSPAM crops no row models, and everything converted to cropland where no crop expanded. """ hectares = dset["hectares-per-cell:ha"] cropland_occupation = ( @@ -394,29 +343,17 @@ def get_commodity_name_to_totals( ) component_to_emissions = { component: sum( - ( - dset[f"{component!s}-emissions:tco2e-per-ha:{before:d}-{after:d}"] - * hectares - * span_to_commodity_to_share[(before, after)][commodity] - for before, after in MAPSPAM_SPANS - ), - start=xarray.zeros_like(hectares), - ) - for component in POOLED_COMPONENTS - } | { - emit.EmissionComponent.PEATLAND_CONVERSION: sum( ( dset[ - f"{land_class:s}-{emit.EmissionComponent.PEATLAND_CONVERSION!s}-emissions:tco2e-per-ha:{before:d}-{after:d}" + f"{land_class:s}-{component!s}-emissions:tco2e-per-ha:{before:d}-{after:d}" ] * hectares - * span_to_commodity_to_peatland_conversion_share[(before, after)][ - commodity - ] + * span_to_commodity_to_share[(before, after)][commodity] for before, after in MAPSPAM_SPANS ), start=xarray.zeros_like(hectares), ) + for component in emit.EmissionComponent } totals |= { component.column: component_to_emissions[component] @@ -484,19 +421,9 @@ def get_commodity_name_to_totals( # The whole charge each component carries, as a commodity holding every share would take it component_to_charged = { component: sum( - ( - dset[f"{component!s}-emissions:tco2e-per-ha:{before:d}-{after:d}"] - * hectares - for before, after in MAPSPAM_SPANS - ), - start=xarray.zeros_like(hectares), - ) - for component in POOLED_COMPONENTS - } | { - emit.EmissionComponent.PEATLAND_CONVERSION: sum( ( dset[ - f"{land_class:s}-{emit.EmissionComponent.PEATLAND_CONVERSION!s}-emissions:tco2e-per-ha:{before:d}-{after:d}" + f"{land_class:s}-{component!s}-emissions:tco2e-per-ha:{before:d}-{after:d}" ] * hectares for land_class in (emit.CROPLAND, emit.PASTURELAND) @@ -504,6 +431,7 @@ def get_commodity_name_to_totals( ), start=xarray.zeros_like(hectares), ) + for component in emit.EmissionComponent } unattributed = commodity_to_totals[emit.NonCommodity.UNATTRIBUTED] unattributed |= { @@ -587,7 +515,7 @@ def get_crop_to_area_share( } | dict.fromkeys(HERD_HEADS_COLUMNS, float) -@storage.cache_to_parquet(version=0) +@storage.cache_to_parquet(version=1) def workflow( assessment_year: int, commodity_names: tuple[str, ...], @@ -667,15 +595,6 @@ def it() -> collections.abc.Iterator[dict[str, float | str]]: ) for (before, after) in MAPSPAM_SPANS } - span_to_commodity_to_peatland_conversion_share = { - (before, after): get_commodity_to_peatland_conversion_share( - after=after, - before=before, - crops=crops, - dset=clipped, - ) - for (before, after) in span_to_commodity_to_share - } logger.info( f"Populating emissions for {jurisdiction.id=!s} over " @@ -698,7 +617,6 @@ def it() -> collections.abc.Iterator[dict[str, float | str]]: dset=clipped, year=max(ifpri_mapspam.YEARS) ), snapshot_year_to_weight=snapshot_year_to_weight, - span_to_commodity_to_peatland_conversion_share=span_to_commodity_to_peatland_conversion_share, span_to_commodity_to_share=span_to_commodity_to_share, ) for commodity_name, totals in commodity_name_to_totals.items(): diff --git a/jdluc/trace.py b/jdluc/trace.py index 6cce7e4..aba325d 100644 --- a/jdluc/trace.py +++ b/jdluc/trace.py @@ -335,7 +335,7 @@ def safe_div(numer: float, denom: float) -> float: } # type: ignore -@storage.cache_to_parquet(ignored_args=["concurrency"], version=1) +@storage.cache_to_parquet(ignored_args=["concurrency"], version=2) def workflow( assessment_year: int, commodity_names: tuple[str, ...],