René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"gh-101607 causes regression in third-party psutil package on macOS","articleBody":"As part of the cPython release process for macOS installers, besides the standard cPython test suite, we run a small set of additional smoke tests. One of these involves an install from source of [the psutil package](https://github.com/giampaolo/psutil) which includes the building of a C extension module. In the run up to the 3.12.0a6 release, we found that our long-unchanged simple test was now failing, a regression from 3.12.0a5 and 3.11.x. The commit causing the failure was bisected to feec49c40736fc05626a183a8d14c4ebbea5ae28 from GH-101607, a part of issue #101578. From a cursory glance at the traceback and the psutil code, it appears that psutil uses a [decorator to translate certain OSError exceptions](https://github.com/giampaolo/psutil/blob/master/psutil/_psosx.py#L339) from its C module into other Python exceptions that its calling code is prepared to handle or ignore. For some reason, that exception translation is now failing. FWIW, this is the first failure of this psutil smoke test in recent memory including previous 3.12.0 alphas. Also, FWIW, this same smoke test does not fail on a reasonably recent Debian Linux system but, of course, that exercises a different platform-dependent code path in psutil.\r\n\r\nBecause of its dependencies, it is a bit complicated to build and reproduce without changes. Here is one way to do it; it is likely not optimal. This assumes a fairly recent macOS system with Homebrew installed (as described in the devguide) and assumes git checkouts of the cpython and psutil Github repos.\r\n\r\n```\r\nbrew install pkg-config openssl@1.1 xz\r\n\r\nmkdir /tmp/test_psutil\r\ncd /tmp/test_psutil\r\ngit clone https://github.com/giampaolo/psutil.git\r\ngit clone https://github.com/python/cpython.git # or from an existing repo\r\n\r\ncd cpython\r\ngit checkout feec49c40736fc05626a183a8d14c4ebbea5ae28^ # last good commit\r\ngit clean -fdxq\r\n./configure -q  --prefix=/tmp/none --with-pydebug \\\r\n              --with-openssl=\"$(brew --prefix openssl@1.1)\"\r\nmake -s -j2\r\ncd ..\r\ngit -C ./psutil clean -fdxq\r\nrm -rf venv\r\n./cpython/python.exe -m venv venv\r\nvenv/bin/python -m pip install --no-binary psutil ./psutil/\r\nvenv/bin/python -c 'import psutil, pprint; pprint.pprint(list(psutil.process_iter()))'\r\n```\r\nThe expected result should be something like:\r\n\r\n\r\n```\r\n[psutil.Process(pid=0, name='kernel_task', status='running', started='03:25:52'),\r\n psutil.Process(pid=1, name='launchd', status='running', started='03:25:52'),\r\n psutil.Process(pid=75, name='logd', status='running', started='03:25:54'),\r\n psutil.Process(pid=76, name='UserEventAgent', status='running', started='03:25:54'),\r\n ...\r\n\r\n```\r\n\r\nWith the failing commit applied (up to and including the current head of the main branch):\r\n\r\n`git checkout feec49c40736fc05626a183a8d14c4ebbea5ae28 # failing commit\r\n`\r\n\r\nand repeating the above build and test, the result is now something like:\r\n\r\n \r\n```\r\nTraceback (most recent call last):\r\n  File \"\u003cstring\u003e\", line 1, in \u003cmodule\u003e\r\n  File \"/private/tmp/cpython/Lib/pprint.py\", line 55, in pprint\r\n    printer.pprint(object)\r\n  File \"/private/tmp/cpython/Lib/pprint.py\", line 153, in pprint\r\n    self._format(object, self._stream, 0, 0, {}, 0)\r\n  File \"/private/tmp/cpython/Lib/pprint.py\", line 175, in _format\r\n    rep = self._repr(object, context, level)\r\n          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/private/tmp/cpython/Lib/pprint.py\", line 455, in _repr\r\n    repr, readable, recursive = self.format(object, context.copy(),\r\n                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/private/tmp/cpython/Lib/pprint.py\", line 468, in format\r\n    return self._safe_repr(object, context, maxlevels, level)\r\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/private/tmp/cpython/Lib/pprint.py\", line 619, in _safe_repr\r\n    orepr, oreadable, orecur = self.format(\r\n                               ^^^^^^^^^^^^\r\n  File \"/private/tmp/cpython/Lib/pprint.py\", line 468, in format\r\n    return self._safe_repr(object, context, maxlevels, level)\r\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/private/tmp/cpython/Lib/pprint.py\", line 629, in _safe_repr\r\n    rep = repr(object)\r\n          ^^^^^^^^^^^^\r\n  File \"/private/tmp/venv/lib/python3.12/site-packages/psutil/__init__.py\", line 389, in __str__\r\n    info[\"name\"] = self.name()\r\n                   ^^^^^^^^^^^\r\n  File \"/private/tmp/venv/lib/python3.12/site-packages/psutil/__init__.py\", line 628, in name\r\n    cmdline = self.cmdline()\r\n              ^^^^^^^^^^^^^^\r\n  File \"/private/tmp/venv/lib/python3.12/site-packages/psutil/__init__.py\", line 681, in cmdline\r\n    return self._proc.cmdline()\r\n           ^^^^^^^^^^^^^^^^^^^^\r\n  File \"/private/tmp/venv/lib/python3.12/site-packages/psutil/_psosx.py\", line 346, in wrapper\r\n    return fun(self, *args, **kwargs)\r\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n  File \"/private/tmp/venv/lib/python3.12/site-packages/psutil/_psosx.py\", line 404, in cmdline\r\n    return cext.proc_cmdline(self.pid)\r\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\nOSError: [Errno 1] Operation not permitted (originated from sysctl(KERN_PROCARGS2))\r\n\r\n```\r\n\r\nNote this seems to come from trying to inspect a privileged process and the point of the exception wrapping is to effectively ignore such errors.\r\n\r\nFlagging as a potential release-blocker for @Yhg1s\r\n\n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-102502\n* gh-102602\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/ned-deily","@type":"Person","name":"ned-deily"},"datePublished":"2023-03-07T09:56:33.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":9},"url":"https://github.com/102493/cpython/issues/102493"}