René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"add2virtualenv logic breaks builds of packages with modern build backend","articleBody":"The logic in add2virtualenv of adding paths to the front of sys.path breaks installing packages in the virtualenv that use a PEP517 build. In preparing the build environment pip tries to determine the necessary sys.path using a \"sitecustomize.py\" written in pip's `_internal/build_env.py`, which gets confused with paths not being added to the end of sys.path. The result is that the lib-dynload path of the Python environment gets removed from sys.path and pip install fails with a traceback like this\r\n\r\n```python\r\n# pip install .\r\nProcessing /Devel/foo\r\n  Installing build dependencies ... done\r\n  Getting requirements to build wheel ... done\r\nERROR: Exception:\r\nTraceback (most recent call last):\r\n  File \"/root/.virtualenvs/foo/lib/python3.11/site-packages/pip/_internal/cli/base_command.py\", line 180, in exc_logging_wrapper\r\n    status = run_func(*args)\r\n             ^^^^^^^^^^^^^^^\r\n  File \"/root/.virtualenvs/foo/lib/python3.11/site-packages/pip/_internal/cli/req_command.py\", line 245, in wrapper\r\n    return func(self, options, args)\r\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/root/.virtualenvs/foo/lib/python3.11/site-packages/pip/_internal/commands/install.py\", line 377, in run\r\n    requirement_set = resolver.resolve(\r\n                      ^^^^^^^^^^^^^^^^^\r\n  File \"/root/.virtualenvs/foo/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/resolver.py\", line 76, in resolve\r\n    collected = self.factory.collect_root_requirements(root_reqs)\r\n                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/root/.virtualenvs/foo/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/factory.py\", line 534, in collect_root_requirements\r\n    reqs = list(\r\n           ^^^^^\r\n  File \"/root/.virtualenvs/foo/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/factory.py\", line 490, in _make_requirements_from_install_req\r\n    cand = self._make_base_candidate_from_link(\r\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/root/.virtualenvs/foo/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/factory.py\", line 228, in _make_base_candidate_from_link\r\n    self._link_candidate_cache[link] = LinkCandidate(\r\n                                       ^^^^^^^^^^^^^^\r\n  File \"/root/.virtualenvs/foo/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/candidates.py\", line 290, in __init__\r\n    super().__init__(\r\n  File \"/root/.virtualenvs/foo/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/candidates.py\", line 156, in __init__\r\n    self.dist = self._prepare()\r\n                ^^^^^^^^^^^^^^^\r\n  File \"/root/.virtualenvs/foo/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/candidates.py\", line 222, in _prepare\r\n    dist = self._prepare_distribution()\r\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/root/.virtualenvs/foo/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/candidates.py\", line 301, in _prepare_distribution\r\n    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)\r\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/root/.virtualenvs/foo/lib/python3.11/site-packages/pip/_internal/operations/prepare.py\", line 525, in prepare_linked_requirement\r\n    return self._prepare_linked_requirement(req, parallel_builds)\r\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/root/.virtualenvs/foo/lib/python3.11/site-packages/pip/_internal/operations/prepare.py\", line 640, in _prepare_linked_requirement\r\n    dist = _get_prepared_distribution(\r\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/root/.virtualenvs/foo/lib/python3.11/site-packages/pip/_internal/operations/prepare.py\", line 71, in _get_prepared_distribution\r\n    abstract_dist.prepare_distribution_metadata(\r\n  File \"/root/.virtualenvs/foo/lib/python3.11/site-packages/pip/_internal/distributions/sdist.py\", line 54, in prepare_distribution_metadata\r\n    self._install_build_reqs(finder)\r\n  File \"/root/.virtualenvs/foo/lib/python3.11/site-packages/pip/_internal/distributions/sdist.py\", line 124, in _install_build_reqs\r\n    build_reqs = self._get_build_requires_wheel()\r\n                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/root/.virtualenvs/foo/lib/python3.11/site-packages/pip/_internal/distributions/sdist.py\", line 101, in _get_build_requires_wheel\r\n    return backend.get_requires_for_build_wheel()\r\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/root/.virtualenvs/foo/lib/python3.11/site-packages/pip/_internal/utils/misc.py\", line 745, in get_requires_for_build_wheel\r\n    return super().get_requires_for_build_wheel(config_settings=cs)\r\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/root/.virtualenvs/foo/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_impl.py\", line 166, in get_requires_for_build_wheel\r\n    return self._call_hook('get_requires_for_build_wheel', {\r\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/root/.virtualenvs/foo/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_impl.py\", line 321, in _call_hook\r\n    raise BackendUnavailable(data.get('traceback', ''))\r\npip._vendor.pyproject_hooks._impl.BackendUnavailable: Traceback (most recent call last):\r\n  File \"/root/.virtualenvs/foo/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py\", line 77, in _build_backend\r\n    obj = import_module(mod_path)\r\n          ^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/usr/lib64/python3.11/importlib/__init__.py\", line 126, in import_module\r\n    return _bootstrap._gcd_import(name[level:], package, level)\r\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"\u003cfrozen importlib._bootstrap\u003e\", line 1204, in _gcd_import\r\n  File \"\u003cfrozen importlib._bootstrap\u003e\", line 1176, in _find_and_load\r\n  File \"\u003cfrozen importlib._bootstrap\u003e\", line 1126, in _find_and_load_unlocked\r\n  File \"\u003cfrozen importlib._bootstrap\u003e\", line 241, in _call_with_frames_removed\r\n  File \"\u003cfrozen importlib._bootstrap\u003e\", line 1204, in _gcd_import\r\n  File \"\u003cfrozen importlib._bootstrap\u003e\", line 1176, in _find_and_load\r\n  File \"\u003cfrozen importlib._bootstrap\u003e\", line 1147, in _find_and_load_unlocked\r\n  File \"\u003cfrozen importlib._bootstrap\u003e\", line 690, in _load_unlocked\r\n  File \"\u003cfrozen importlib._bootstrap_external\u003e\", line 940, in exec_module\r\n  File \"\u003cfrozen importlib._bootstrap\u003e\", line 241, in _call_with_frames_removed\r\n  File \"/tmp/pip-build-env-pitma8s7/overlay/lib/python3.11/site-packages/setuptools/__init__.py\", line 8, in \u003cmodule\u003e\r\n    import _distutils_hack.override  # noqa: F401\r\n    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/root/.virtualenvs/foo/lib/python3.11/site-packages/_distutils_hack/override.py\", line 1, in \u003cmodule\u003e\r\n    __import__('_distutils_hack').do_override()\r\n  File \"/root/.virtualenvs/foo/lib/python3.11/site-packages/_distutils_hack/__init__.py\", line 70, in do_override\r\n    ensure_local_distutils()\r\n  File \"/root/.virtualenvs/foo/lib/python3.11/site-packages/_distutils_hack/__init__.py\", line 53, in ensure_local_distutils\r\n    importlib.import_module('distutils')\r\n  File \"/usr/lib64/python3.11/importlib/__init__.py\", line 126, in import_module\r\n    return _bootstrap._gcd_import(name[level:], package, level)\r\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/root/.virtualenvs/foo/lib/python3.11/site-packages/_distutils_hack/__init__.py\", line 90, in find_spec\r\n    return method()\r\n           ^^^^^^^^\r\n  File \"/root/.virtualenvs/foo/lib/python3.11/site-packages/_distutils_hack/__init__.py\", line 97, in spec_for_distutils\r\n    import importlib.abc\r\n  File \"/usr/lib64/python3.11/importlib/abc.py\", line 19, in \u003cmodule\u003e\r\n    from .resources.abc import ResourceReader, Traversable, TraversableResources\r\n  File \"/usr/lib64/python3.11/importlib/resources/__init__.py\", line 3, in \u003cmodule\u003e\r\n    from ._common import (\r\n  File \"/usr/lib64/python3.11/importlib/resources/_common.py\", line 3, in \u003cmodule\u003e\r\n    import tempfile\r\n  File \"/usr/lib64/python3.11/tempfile.py\", line 45, in \u003cmodule\u003e\r\n    from random import Random as _Random\r\n  File \"/usr/lib64/python3.11/random.py\", line 49, in \u003cmodule\u003e\r\n    from math import log as _log, exp as _exp, pi as _pi, e as _e, ceil as _ceil\r\nModuleNotFoundError: No module named 'math'\r\n```\r\n\r\nThis can be reproduced with a fresh virtualenv that contains a couple - in this setup it was three, but in other environments two were enough - paths added via add2virtualenv\r\n\r\n```bash\r\nmkvirtualenv foo\r\nmkdir dummy1 dummy2 dummy3\r\nadd2virtualenv dummy1 dummy2 dummy3\r\n```\r\n\r\nand then installing a basically empty package with a minimal pyproject.toml\r\n\r\n```bash\r\necho \"[project]\r\nname = 'bar'\r\nversion = '1.0.0'\r\n\r\n[build-system]\r\nrequires = ['setuptools']\r\nbuild-backend = 'setuptools.build_meta'\" \u003e pyproject.toml\r\npip install .\r\n```","author":{"url":"https://github.com/dfn-certling","@type":"Person","name":"dfn-certling"},"datePublished":"2024-04-17T16:02:50.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":1},"url":"https://github.com/98/virtualenvwrapper/issues/98"}