René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"sys._base_executable resolves symlinks too eagerly, resulting in undesirable venv from venv behaviour","articleBody":"# Bug report\n\n### Bug description:\n\nIn #106045, the use case of putting a symlink in `/usr/local/bin` to a Python binary from another prefix was highlighted. For homebrew, this use case is used with the idea that the symlink is the public interface/location, and the prefix where it is actually installed is an implementation detail (and can change over time) (more details in https://github.com/astral-sh/uv/issues/1640).\n\n#106045 seems to have now been resolved by #127974 ([I wrote a test for that](https://github.com/python/cpython/pull/127974#issuecomment-2579501759)). The originating issue proposed a solution which would fix the problem with the broken virtual environment in that case, but would actually break the aforementioned homebrew example by eagerly resolving the symlinked executable and using the resolved path to determine the `home` value in `pyvenv.cfg` - thereby exposing the \"internal\" prefix into `pyvenv.cfg`.\n\nThe problem I am reporting here is that\n * because `sys._base_executable` eagerly resolves symlinks in [`getpath.py`](https://github.com/python/cpython/blob/b2adf556747d080f04b53ba4063b627c2dbe41d1/Modules/getpath.py#L388)\n * and `venv`, `virtualenv` and `uv` all use `sys._base_executable` to determine the `home` location\n  * then when you take a virtual environment _from another virtual environment_ in the aforementioned setup, the second virtual environment's `home` location will be resolved, whereas the first will not be. Thereby bleeding the implementation detail once more.\n\nIn `uv`, the behaviour was solidified in https://github.com/astral-sh/uv/pull/8433 to avoid exposing the internal prefix in the venv. Note that `virtualenv` is (accidentally?) now exposing the internal prefix even on the first virtual environment https://github.com/pypa/virtualenv/issues/2770.\n\n\nTo reproduce this:\n\n```\n$ mkdir -p /tmp/public/bin/\n$ ln -s /usr/local/bin/python3.14 /tmp/public/bin/\n\n$ /tmp/public/bin/python3.14 -m venv /tmp/venv1\n$ /tmp/venv1/bin/python -m venv /tmp/venv2\n\n$ cat /tmp/venv1/pyvenv.cfg \nhome = /tmp/public/bin\ninclude-system-site-packages = false\nversion = 3.14.0\nexecutable = /usr/local/bin/python3.14\ncommand = /tmp/public/bin/python3.14 -m venv /tmp/venv1\n\n$ cat /tmp/venv2/pyvenv.cfg \nhome = /usr/local/bin\ninclude-system-site-packages = false\nversion = 3.14.0\nexecutable = /usr/local/bin/python3.14\ncommand = /tmp/venv1/bin/python -m venv /tmp/venv2\n```\n\nAnd observe that the second venv's home is not `/tmp/public/bin` but the \"internal detail\" one.\n\nA quick test for getpath, and a complete test for venv are included below (both failing):\n\n```diff\ndiff --git a/Lib/test/test_getpath.py b/Lib/test/test_getpath.py\nindex f86df9d0d03..1c2e2a0b3fc 100644\n--- a/Lib/test/test_getpath.py\n+++ b/Lib/test/test_getpath.py\n@@ -864,6 +864,55 @@ def test_PYTHONHOME_in_venv(self):\n         actual = getpath(ns, expected)\n         self.assertEqual(expected, actual)\n \n+    def test_venv_w_symlinked_base_executable(self):\n+        \"\"\"\n+        If we symlink the base executable, and point to it via home in pyvenv.cfg,\n+        we should have it as sys.executable (and sys.prefix should be the resolved location)\n+        \"\"\"\n+        ns = MockPosixNamespace(\n+            argv0=\"/venv/bin/python3\",\n+            PREFIX=\"/some/_internal/prefix\",\n+        )\n+        # Setup venv\n+        ns.add_known_xfile(\"/venv/bin/python3\")\n+        ns.add_known_xfile(\"/usr/local/bin/python3\")\n+        ns.add_known_xfile(\"/some/_internal/prefix/bin/python3\")\n+\n+        ns.add_known_file(\"/venv/pyvenv.cfg\", [\n+            # The published based executable location is /usr/local/bin - we don't want to\n+            # expose /some/internal/directory (this location can change under our feet)\n+            r\"home = /usr/local/bin\"\n+        ])\n+        ns.add_known_link(\"/venv/bin/python3\", \"/usr/local/bin/python3\")\n+        ns.add_known_link(\"/usr/local/bin/python3\", \"/some/_internal/prefix/bin/python3\")\n+\n+        ns.add_known_file(\"/some/_internal/prefix/lib/python9.8/os.py\")\n+        ns.add_known_dir(\"/some/_internal/prefix/lib/python9.8/lib-dynload\")\n+\n+        # Put a file completely outside of /usr/local to validate that the issue\n+        # in https://github.com/python/cpython/issues/106045 is resolved.\n+        ns.add_known_dir(\"/usr/lib/python9.8/lib-dynload\")\n+\n+        expected = dict(\n+            executable=\"/venv/bin/python3\",\n+            prefix=\"/venv\",\n+            exec_prefix=\"/venv\",\n+            base_prefix=\"/some/_internal/prefix\",\n+            base_exec_prefix=\"/some/_internal/prefix\",\n+            # It is important to maintain the link to the original executable, as this\n+            # is used when creating a new virtual environment (which should also have home\n+            # set to /usr/local/bin to avoid bleeding the internal path to the venv)\n+            base_executable=\"/usr/bin/python3\",\n+            module_search_paths_set=1,\n+            module_search_paths=[\n+                \"/some/_internal/prefix/lib/python98.zip\",\n+                \"/some/_internal/prefix/lib/python9.8\",\n+                \"/some/_internal/prefix/lib/python9.8/lib-dynload\",\n+            ],\n+        )\n+        actual = getpath(ns, expected)\n+        self.assertEqual(expected, actual)\n+\n # ******************************************************************************\n \n DEFAULT_NAMESPACE = dict(\ndiff --git a/Lib/test/test_venv.py b/Lib/test/test_venv.py\nindex 0b09010c69d..bd1b19a9c15 100644\n--- a/Lib/test/test_venv.py\n+++ b/Lib/test/test_venv.py\n@@ -756,6 +756,53 @@ def test_zippath_from_non_installed_posix(self):\n         out, err = check_output(cmd)\n         self.assertTrue(zip_landmark.encode() in out)\n \n+    @unittest.skipIf(os.name == 'nt', 'not relevant on Windows')\n+    @requireVenvCreate\n+    def test_venv_from_venv_with_symlink(self):\n+        \"\"\"\n+        Test that we can create a venv from a venv using a base Python that is\n+        a symlink, without exposing the location of the symlink in pyvenv.cfg.\n+        \"\"\"\n+        rmtree(self.env_dir)\n+        public_prefix = os.path.realpath(tempfile.mkdtemp())\n+        self.addCleanup(rmtree, public_prefix)\n+        public_bin_dir = os.path.join(public_prefix, 'bin')\n+        os.mkdir(public_bin_dir)\n+        public_exe = os.path.join(public_bin_dir, self.exe)\n+        os.symlink(sys.executable, public_exe)\n+        cmd = [public_exe,\n+               \"-m\",\n+               \"venv\",\n+               \"--without-pip\",\n+               \"--without-scm-ignore-files\",\n+               self.env_dir]\n+\n+        subprocess.check_call(cmd)\n+\n+        # Verify that we don't expose the internal prefix to the first venv config:\n+        contents = (pathlib.Path(self.env_dir) / 'pyvenv.cfg').read_text()\n+        self.assertIn(f'home = {public_bin_dir}\\n', contents)\n+\n+        # Now use the venv to make another, and assert that the internal env is\n+        # also not exposed there.\n+        second_venv = os.path.realpath(tempfile.mkdtemp())\n+        self.addCleanup(rmtree, second_venv)\n+\n+        cmd = [os.path.join(self.env_dir, 'bin', 'python3'),\n+               \"-m\",\n+               \"venv\",\n+               \"--without-pip\",\n+               \"--without-scm-ignore-files\",\n+               second_venv]\n+\n+        subprocess.check_call(cmd)\n+\n+        contents = (pathlib.Path(second_venv) / 'pyvenv.cfg').read_text()\n+        self.assertIn(f'home = {public_bin_dir}\\n', contents)\n+\n     @requireVenvCreate\n     def test_activate_shell_script_has_no_dos_newlines(self):\n         \"\"\"\n```\n\ncc @FFY00 following on from our conversation in https://github.com/python/cpython/pull/127974#issuecomment-2579408023 (I would love to get a commit for the tests, if not the fix :stuck_out_tongue_winking_eye:).\n\n### CPython versions tested on:\n\nCPython main branch\n\n### Operating systems tested on:\n\nLinux\n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-128913\n* gh-150999\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/pelson","@type":"Person","name":"pelson"},"datePublished":"2025-01-09T10:57:48.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":4},"url":"https://github.com/128670/cpython/issues/128670"}