fix(bootstrap): remap stdlib sys.path entries to runfiles during site init - #4104
fix(bootstrap): remap stdlib sys.path entries to runfiles during site init#4104rickeylev wants to merge 8 commits into
Conversation
…ath bug Because Python follows symlinks when initializing standard library paths, sys.path entries currently resolve to underlying repository cache or execroot locations rather than remaining within the runfiles tree. Add a pytest test target (stdlib_symlink_syspath_bootstrap_script_test) that collects all stdlib entries in sys.path and asserts that they are located within the runfiles root.
… init When CPython resolves interpreter symlinks during startup, it leaks underlying repository cache or external repository paths into sys.path and sys prefixes instead of their runfiles locations. This breaks runfiles isolation and prevents using a mixture of generated and non-generated files as part of the python runtime, as standard library modules and runtime site-packages are loaded from locations outside the runfiles tree. Fix this by passing interpreter_actual_path into site_init_template.py during virtual environment creation. During site initialization, collect any non-runfiles sys prefixes into a set and remap matching sys.path entries, sys prefixes, and site.PREFIXES to the runfiles runtime root.
…dlib In WORKSPACE mode or when using platform/system Python runtimes (such as in runtime_env_toolchain), the runtime root in runfiles does not contain a Python standard library. Unconditionally remapping sys.base_prefix caused the valid system stdlib path to be replaced with a non-existent runfiles directory. Fix this by checking that the target runfiles runtime root actually contains a standard library directory (lib, lib64, Lib, or DLLs) before performing any sys.path or sys prefix remapping.
On Windows, drive letter casing (e.g. C:\ vs c:\) between runfiles root and sys.prefix caused virtual environment directories to fail runfiles containment checks. This resulted in venv sys.prefix being overwritten by the base Python runtime root. Use os.path.normcase when normalizing paths in _fixup_stdlib_paths so runfiles containment and sys.path matching are case-insensitive on Windows.
In virtual environments (where sys.prefix != sys.base_prefix), sys.prefix points to the .venv directory (which on Windows is created outside the runfiles tree in the output bin directory). Unconditionally remapping sys.prefix and sys.exec_prefix overwrote the venv prefix with the base Python runtime root. Fix this by only remapping sys.base_prefix and sys.base_exec_prefix when executing inside a virtual environment.
|
For context, this came out of trying to zip the stdlib. What happens is the If a generated file is added to the runtime's files (e.g. adding a generated file to py_runtime), it gets put into the exec root. Since the exec root was never added to sys.path, those files aren't importable. It also means the runtime's files are split up into two directories (the repo directory and the execroot directory). Normally runfiles unifies things (as happens with regular libraries), but Python's resolving the symlinks foils that. The only other option I see is to turn all the files in the runtime into generated files. i.e. make py_runtime loop over every file and symlink it (and copy python3 itself so it doesn't point back to the repo directory). This wouldn't get the runfiles directory onto sys.path, but would put all the runtime files in one location (the execroot). I'm not sure if that's better or not than futzing with sys.path at startup. |
aignas
left a comment
aignas
left a comment
There was a problem hiding this comment.
Nice find. please add news. People may look at it sometime in the future, or we may need to refer to this.
Add news entry 4104.fixed.md to document the bootstrap fix for CPython interpreter symlink resolution leaking non-runfiles repository cache locations into sys.path and sys prefixes.
|
Added news entry in |
Recategorize the news fragment for PR 4104 from fixed to changed and rephrase to note that sys.path adds the runtime in runfiles instead of the underlying Bazel repository cache directory.
Update news/4104.changed.md wording to explicitly mention that sys.path adds the Python runtime in runfiles.
2 participants
When CPython resolves interpreter symlinks during startup, it leaks
underlying repository cache or external repository paths into sys.path
and sys prefixes instead of their runfiles locations.
This breaks runfiles isolation and prevents using a mixture of
generated and non-generated files as part of the python runtime, as
standard library modules and runtime site-packages are loaded from
locations outside the runfiles tree.
Fix this by passing interpreter_actual_path into site_init_template.py
during virtual environment creation. During site initialization,
collect any non-runfiles sys prefixes into a set and remap matching
sys.path entries, sys prefixes, and site.PREFIXES to the runfiles
runtime root.
Also adds a reproduction test target
(stdlib_symlink_syspath_bootstrap_script_test) to verify that sys.path
entries remain within the runfiles tree.
Work towards #1653