René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Building and linking C extensions in a post-distutils world","articleBody":"With the removal of distutils from 3.12 alphas, I've recently taken a new look at the use of distutils within https://mesonbuild.com in the hopes that we were finally unblocked and could migrate to sysconfig.\r\n\r\n- install schemes turn out to finally be viable since 3.10.3, when the `deb_system` scheme patch on Debian operating systems for distutils was patched into sysconfig. This hard blocker is gone, thank G-d.\r\n- linking to libpython itself may or may not be an issue, originally I thought it was, but now I think it may not be.\r\n\r\n### How can a C-extension supporting PEP 517 build backend know the best way to compile and link?\r\n\r\n## python \u003c3.12\r\n\r\nMeson has some code: https://github.com/mesonbuild/meson/blob/master/mesonbuild/modules/python.py#L338-L407\r\n\r\nHere's the interesting bit:\r\n```python\r\ndef links_against_libpython():\r\n    from distutils.core import Distribution, Extension\r\n    cmd = Distribution().get_command_obj('build_ext')\r\n    cmd.ensure_finalized()\r\n    return bool(cmd.get_libraries(Extension('dummy', [])))\r\n```\r\n\r\n## python \u003e=3.8\r\n\r\nIn bpo-36721, bpo-21536 etc the above code changes from returning True to False, on many Unix platforms.\r\n\r\nNew additional methods suitable for getting C extension arguments are available. Well, mostly suitable.\r\n\r\n- `sysconfig.get_config_var('LIBPYTHON')`\r\n\r\nthis is configured into:\r\n- `pkg-config --cflags --libs python3`\r\n- `python-config --embed`\r\n\r\nThere's a couple problems with this:\r\n- on Cygwin and Android, LIBPYTHON and thus pkg-config is hardcoded to link to libpython, but distutils only does so when `Py_ENABLE_SHARED`\r\n- none of it works on Windows, which since it is built with PCBuild, doesn't have Makefile config vars (and also / consequently? distributes neither of the latter two)\r\n- pkg-config may not always be installed, so we want a fallback for that\r\n- python-config has tons of CPython build-time junk so we cannot use it\r\n\r\n...\r\n\r\nIt feels uncomfortably like there's still way too much undocumented magic here.\r\n\r\n@vstinner, I assume the Cygwin/Android inconsistency is probably a configure.ac bug that should behave the same way distutils does/did? I can provide a patch to fix it but would like confirmation of which side to fix.\r\n\r\n@FFY00, I think in the long term, sysconfig should expose a function that returns cflags / libs required to build an extension (and works on Windows without `_generate_posix_vars`, and doesn't include lots of `-O3 -pipe -fstack-protector-strong -fdiagnostics-color=always` and more -- i.e. works like pkg-config, not like python-config). Even without the question of \"whether to link to libpython\", there's a 140-line class for figuring out what the name of the library is, which directory to find it in, and what the include directory is too. Of course, this is specific to the case where pkg-config is not available, and most of it is for the Windows case. \n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-100356\n* gh-100967\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/eli-schwartz","@type":"Person","name":"eli-schwartz"},"datePublished":"2022-12-02T04:30:55.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":6},"url":"https://github.com/99942/cpython/issues/99942"}