Four pinned remote data sources have moved or died

[EnvDroneSense]

Describe the bug

Four hardcoded remote sources no longer resolve to what the code expects.

#	Where	What happens now
1	climada/util/dwd_icon_loader.py:299	The pinned icon_grid_0028_R02B07_N02.nc.bz2 is still served but is the wrong grid for the GRIB the same function downloads. StormEurope.from_icon_grib dies with Invalid Hazard.intensity column size: 75948 != 164984
2	climada/entity/exposures/litpop/gpw_population.py:164	sedac.ciesin.columbia.edu accepts no connections. Used only in a FileNotFoundError, but that message is the only instruction a user gets for an input CLIMADA cannot download itself
3	climada/conf/climada.conf:33 (noaa_url)	404 on directory, tarball and parent. The resulting "Try a different year" is misleading; the year is fine, the host is gone
4	climada/conf/climada.conf:36 and :39	Contain a literal * that nothing glob-expands, so they can never resolve

To Reproduce

1. StormEurope.from_icon_grib() for a current ICON-EU-EPS run.
2. LitPop.from_countries("CHE") without the GPW raster, then follow the URL in the error.
3. nightlight.load_nightlight_noaa(2013).
4. read the config; nightlight.py:372 concatenates the string directly, nothing expands the *.

Code example:

# 1. from the artifacts themselves
#   icon_grid_0028_R02B07_N02.nc.bz2  ->  75,948 cells   (pinned)
#   icon_grid_0037_R03B07_N02.nc.bz2  ->  164,984 cells
# and from a live icon-eu-eps vmax_10m GRIB, read with eccodes:
#   numberOfGridUsed    -> 37
#   numberOfDataPoints  -> 164984

# 2., 3. the dead hosts
import urllib.request
for url in ["https://sedac.ciesin.columbia.edu/",                     # connection fails
            "https://ngdc.noaa.gov/eog/data/web_data/v4composites/"]:  # 404
    try:
        urllib.request.urlopen(url, timeout=30)
    except Exception as e:
        print(url, type(e).__name__, e)

Expected behavior

from_icon_grib uses the grid the GRIB declares. A user following an error message reaches the data. No config entry is unreachable by construction.

Climada Version: develop @ dea46fac (6.1.1-dev)

System Information (please complete the following information):

• Operating system and version: Windows 11 Home 10.0.26200
• Python version: 3.12.14 | packaged by conda-forge | (main, Sep 2 2026, 23:18:20) [MSC v.1944 64 bit (AMD64)]

Additional context

Probed 2026-09-07. Each grid was downloaded, decompressed and its cell dimension read; one live GRIB per model was read with eccodes.

1. DWD added 0037 without removing 0028, so the pinned URL still returns 200 and any liveness check calls it healthy. Both numbers in the error come straight from the artifacts: 75,948 cells in the pinned grid, 164,984 in grid 0037 and in the GRIB's numberOfDataPoints. Reading numberOfGridUsed from the GRIB and selecting the grid from it would avoid re-pinning next time.

The other two ICON pins are fine: icon-d2-eps (line 303) reports grid 47 / 542,040 points, matching its pinned file exactly, and the icon-eu branch at line 301 is unreachable because _create_icon_grib_name raises for any model but the two -eps ones. Only line 299 is wrong.

2. sedac.ciesin.columbia.edu resolves to 129.236.39.66 but nothing answers on 443; confirmed from two independent networks, with earthdata.nasa.gov and github.com returning 200 as controls. Content moved to data.earthdata.nasa.gov; CMR lists CIESIN_SEDAC_GPWv4_POPCOUNT_R11, the pinned version. The translated path:

https://data.earthdata.nasa.gov/nasa-earth/human-dimensions/sedac-root/downloads/data/gpw-v4/
    gpw-v4-population-count-rev11/gpw-v4-population-count-rev11_2015_30_sec_tif.zip

returns 401 -> Earthdata OAuth unauthenticated, and 206 with a login. I downloaded the full 405,108,819 bytes and unzipped it. A Earthdata login is still needed, as before only host and path change.

3. EOG moved to the Colorado School of Mines, and eogdata.mines.edu/wwwdata/dmsp/v4composites_rearrange/F182013.v4.tar returned 200 on 2026-08-27, but on 2026-09-07 it returns 302 to a Keycloak login. Repointing noaa_url would trade a 404 for an HTML login page. This likely needs the GPW treatment (documented manual download) or another mirror. The misleading error message is worth fixing regardless.

4. The tiles are healthy: record 144896 serves 8/8 tiles for 2012, 144897 8/8 for 2016, both years in blackmarble_years. Only the * entries are dead, and download_nl_files skips them silently. Removing them costs nothing.

Priority. 1 decides whether a documented feature runs at all. 2 and 4 are small and self-contained. 3 needs a decision before any patch.

Two climada_petals sources fail the same way (ETH gumbel-fit at rf_glofas/setup.py:55, and the WRI Aqueduct base URL). Both replacements are verified.

[emanuel-schmid]

👀 many thanks for pointing this out, @EnvDroneSense 🙌