Full lockfile regen is broken: dependency names are not PEP 503 canonicalized

[schloerke]

Summary

make update_packages_lock (full regeneration of shinylive_lock.json from shinylive_requirements.json) currently produces a broken build. Any PyPI package whose parent declares its dependency with an underscore instead of a hyphen gets written into the lockfile under a non-canonical key, which pyodide then cannot resolve at runtime.

This blocks refreshing the "version": "latest" PyPI pins, several of which are well behind (e.g. shinyswatch sat at 0.8.0 while PyPI was at 0.11.0 — see #230).

Reproduction

make update_packages_lock
make retrieve_packages update_pyodide_lock_json
make serve

Then load any app that imports shinywidgets (the bundled Plotly, Map, and altair examples all qualify). The app never starts; the browser console shows:

pyodide.ffi.JsException: Error: No known package with name 'jupyterlab_widgets'
  at _load_packages_from_dir

Root cause

_recurse_dependencies_lockfile() keys each discovered dependency by the literal name string taken from the parent package's metadata:

https://github.com/posit-dev/shinylive/blob/main/scripts/pyodide_packages.py#L332

pkgs[dep_name] = _find_package_info_lockfile_one(
    {
        "name": dep_name,
        "source": "pypi",
        ...

ipywidgets changed how it spells that dependency between releases:

ipywidgets	declares dependency as
8.1.3 (currently pinned)	jupyterlab-widgets (hyphen — PEP 503 canonical)
8.1.8 (latest)	jupyterlab_widgets (underscore)

So a regen writes the lockfile key as jupyterlab_widgets, and update_pyodide_lock_json injects that key verbatim into pyodide-lock.json. Pyodide normalizes _ → - when resolving a package name, finds no jupyterlab-widgets entry, and throws.

Note this is not a version-incompatibility problem — jupyterlab_widgets 3.0.16 itself is fine. The wheel is downloaded and present on disk; it is purely the lookup key that is wrong. The current lockfile works only by luck, because the pinned ipywidgets happens to use the hyphen spelling.

Suggested fix

Canonicalize dependency names (PEP 503: lowercase, runs of -_. collapsed to -) at the point where they become lockfile keys, in _recurse_dependencies_lockfile() and/or _filter_requires(). packaging.utils.canonicalize_name already does exactly this, and packaging is available.

Care is needed on two fronts:

• _to_basic_package_info() / the orig_pyodide_lock() comparison already lowercases keys but does not normalize separators, so the "do we already have this package?" check has the same blind spot and can double-add a package under two spellings.
• The wheel filename legitimately uses underscores (jupyterlab_widgets-3.0.16-py3-none-any.whl); only the lookup key should be canonicalized, not filename/url.

Related: dependency version constraints are ignored

While verifying the above, a second contributing factor showed up. _recurse_dependencies_lockfile() resolves every transitive dependency to "version": "latest" regardless of what the parent actually supports:

https://github.com/posit-dev/shinylive/blob/main/scripts/pyodide_packages.py#L336

        # TODO: Use version from dependencies
        "version": "latest",

The specs are captured into the lockfile but never used for resolution. This makes a full regen much higher-variance than it needs to be — a single regen moved 28 packages at once, including a starlette 0.38.1 → 1.3.1 major bump.

For what it's worth, that starlette jump was verified to be fine in practice (shiny 1.6.3 rendered and round-tripped reactive updates over the websocket on starlette 1.3.1), and some of the regen's churn is genuinely desirable cleanup — it correctly drops python-multipart (which is marked platform_system != 'Emscripten' and should never have been in the lock), tenacity (dropped by plotly 6), and pypng (dropped by qrcode 8.2). So the regen is worth unblocking rather than avoiding.

Workaround until fixed

Edit the single package's entry in shinylive_lock.json by hand instead of regenerating. Using the repo's own helper keeps the formatting identical:

import json, importlib.util
spec = importlib.util.spec_from_file_location("pp", "scripts/pyodide_packages.py")
pp = importlib.util.module_from_spec(spec); spec.loader.exec_module(pp)

lock = json.load(open("shinylive_lock.json"))
lock["shinyswatch"] = pp._get_pypi_package_info("shinyswatch", "latest")

with open("shinylive_lock.json", "w") as f:
    json.dump(pp._mark_no_indent(lock, pp._is_lockfile_dependency), f, indent=2, cls=pp.NoIndentEncoder)

followed by make retrieve_packages update_pyodide_lock_json. Worth pruning stale wheels from build/shinylive/pyodide/ afterwards, since retrieve_packages only adds.