Title: New sysconfig API 🤞 (meta-issue) · Issue #103480 · python/cpython · GitHub
Open Graph Title: New sysconfig API 🤞 (meta-issue) · Issue #103480 · python/cpython
X Title: New sysconfig API 🤞 (meta-issue) · Issue #103480 · python/cpython
Description: Feature or enhancement Provide new syconfig API that is aligned to how things have evolved. Fixes GH-99942 Fixes GH-99560 Pitch The sysconfig module was added in Python 3.2, over 10 years ago, and though there were some minor changes and...
Open Graph Description: Feature or enhancement Provide new syconfig API that is aligned to how things have evolved. Fixes GH-99942 Fixes GH-99560 Pitch The sysconfig module was added in Python 3.2, over 10 years ago, and ...
X Description: Feature or enhancement Provide new syconfig API that is aligned to how things have evolved. Fixes GH-99942 Fixes GH-99560 Pitch The sysconfig module was added in Python 3.2, over 10 years ago, and ...
Opengraph URL: https://github.com/python/cpython/issues/103480
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"New sysconfig API 🤞 (meta-issue)","articleBody":"# Feature or enhancement\r\n\r\nProvide new `syconfig` API that is aligned to how things have evolved.\r\n\r\nFixes GH-99942\r\nFixes GH-99560\r\n\u003c!--\r\nFixes GH-77620? (XXX: python-config.py should be part of the stdlib)\r\n-->\r\n\r\n# Pitch\r\n\r\nThe `sysconfig` module was added in Python 3.2, over 10 years ago, and though there were some minor changes and additions, its API stayed mostly the same. Since then things have evolved, with new platform support (eg. WASM) and features being added, and we are now at a point where the current design is not great and has been the source of several issues, especially since the `distutils` deprecation (see GH-99942 for eg.).\r\n\r\nChanging the existing API to better represent modern Python installations would cause a lot of breakage on existing code. Considering that, on top of several of its shortcomings, we think the best step at the moment would be to introduce a new API and mark part of the existing one as either deprecated, or pending deprecation, depending how development goes and on feedback from the impacted parties.\r\n\r\nHere are the main points we'd like to improve on:\r\n\r\n1) Users currently rely on [`sysconfig.get_config_vars`](https://docs.python.org/3/library/sysconfig.html) for **a lot** of things\r\n\r\nOn Posix, `sysconfig.get_config_vars` exports all the variables present in the installation's `Makefile`, and on Windows, it exports a handful of selected variables.\r\n\r\nhttps://github.com/python/cpython/blob/52f96d3ea39fea4c16e26b7b10bd2db09726bd7c/Lib/sysconfig.py#L540-L556\r\n\r\nThere are several issues with this, the main being that the exported variables are not documented, so there are no guarantees about what is information is gonna be available, or the variable names.\r\n\r\nWhile this is a good escape-hatch mechanism, it is a very poor option for the intended interface for key information. We should be exporting that information in a proper API.\r\n\r\n2) The installation paths model is outdated\r\n\r\nThere are a couple of issues and things to improve here here:\r\n- A scheme should be able to not provide non-critial paths (`scripts` for eg.)\r\n- We should introduce the concept of writable and read-only paths (users should not install to any path in the installed base, like `include`, for eg.)\r\n- We should introduce the concept of active location schemes (eg. `posix_user` should not be available on virtual environments)\r\n - This would extend to vendor schemes outside the default environment, if we ever get around to standardize that\r\n\r\n**Discussion on this should go to GH-103482**\r\n\r\n3) Cross-compilation support is very poor\r\n\r\nCross-compilation at the moment is done by patching several aspects of Python, with the `sysconfig` module being the main thing. We provide several undocumented features to support this, like the allowing the project base path and sysconfig data module name to be overwritten via the `_PYTHON_PROJECT_BASE` and `_PYTHON_SYSCONFIGDATA_NAME` environment variables, respectively.\r\n\r\nWe believe one of the reasons that plays into users choosing to support cross-compilation by monkey patching Python, instead of directly adding cross-compilation support to the 3rd party building code, is because with the current `sysconfig` design, a lot of custom code would have to be added to properly support cross builds.\r\n\r\nOur proposal to solve this is to make the new API standalone, meaning it would provide all the information users may need for binary interfacing (eg. building extension modules) and similar applications. On top of this, we'd shift the design from using module functions to data holder objects, meaning all the relevant details of the Python installation would be represented by just a couple objects. Users that adopt this new API should then be able to support the cross build by simply changing the data objects for ones that represent the target installation.\r\n\r\nThis has the drawback that we'd be re-exporting information already present in other places (eg. `sys.version`, `sys.implementation`), but we believe it is worth.\r\n\r\n###### Future work\r\n\r\n*In a future step* we would also like to get Python to provide a static file detailing all the information required by the new `sysconfig` API. With this, we could then allow the new API to use an external installation/interpreter config file as the data source, making cross builds even simpler to support.\r\nSee the discussion at https://discuss.python.org/t/what-information-is-useful-to-know-statically-about-an-interpreter/25563 for examples of users that would benefit from this feature.\r\n\r\n## Migration plan\r\n\r\nTo help migration, and to make these improvements available on older Python versions, we plan to provide a backport package on PyPI, similarly with we already do with [`importlib_metadata`](https://github.com/python/importlib_metadata) (for [`importlib.metadata`](https://docs.python.org/3/library/importlib.metadata.html)), [`importlib_resources`](https://github.com/python/importlib_resources) (for [`importlib.resources`](https://docs.python.org/3/library/importlib.resources.html)), and [`typing_extensions`](https://github.com/python/typing_extensions) (for [`typing`](https://docs.python.org/3/library/typing.html)).\r\n\r\n## Sub-issues\r\n\r\nThe proposal is quite big, and may require extensive discussions with different parties, so I am splitting it into a couple separate issues.\r\n\r\n- Install paths (GH-103481)\r\n- Build information\r\n - Compilation (GH-103482)\r\n - Compatibility (GH-103483)\r\n\r\n# Previous discussion\r\n\r\nThis has been mentioned in a couple places, here are some of the most relevant:\r\n\r\nhttps://discuss.python.org/t/building-extensions-modules-in-a-post-distutils-world/23938\r\nhttps://discuss.python.org/t/sysconfig-should-provide-more-information-to-help-packaging/22950\r\nhttps://discuss.python.org/t/pep-582-python-local-packages-directory/963/391\r\nhttps://discuss.python.org/t/linux-distro-patches-to-sysconfig-are-changing-pip-install-prefix-outside-virtual-environments/18240/28\r\nhttps://github.com/python/cpython/issues/102522#issuecomment-1497727808\r\nhttps://github.com/PyO3/maturin/blob/5d5b96a9974eac26b8cdf601cd2faf64f4999de9/src/python_interpreter/sysconfig-freebsd.json\n\n\u003c!-- gh-linked-prs -->\n### Linked PRs\n* gh-110785\n* gh-110792\n\u003c!-- /gh-linked-prs -->\n","author":{"url":"https://github.com/FFY00","@type":"Person","name":"FFY00"},"datePublished":"2023-04-12T18:48:07Z","url":"https://github.com/python/cpython/issues/103480"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:7afbf9c9-7129-7d8a-91a5-5632a1050b5f |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | A064:19AA62:3E32E48:4EE0109:6B020F4D |
| html-safe-nonce | 044e86010278afc4abf09352fcc2f5697ec6ef760bc048f47185f57fa88129d1 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJBMDY0OjE5QUE2MjozRTMyRTQ4OjRFRTAxMDk6NkIwMjBGNEQiLCJ2aXNpdG9yX2lkIjoiMjYxNzQ0NDE5ODI5ODgxNjMzMyIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | 2e2ff599c32ea854a9c3f6a15d1241e3e211e862b61d69dbac1f7a81e3e889a3 |
| hovercard-subject-tag | issue:1665071363 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/python/cpython/103480/issue_layout |
| twitter:image | https://opengraph.githubassets.com/c1d3e479809a97687c33d309bf7dfe675dbe4473ad27445fd44b92f81b0afe7c/python/cpython/issues/103480 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/c1d3e479809a97687c33d309bf7dfe675dbe4473ad27445fd44b92f81b0afe7c/python/cpython/issues/103480 |
| og:image:alt | Feature or enhancement Provide new syconfig API that is aligned to how things have evolved. Fixes GH-99942 Fixes GH-99560 Pitch The sysconfig module was added in Python 3.2, over 10 years ago, and ... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | FFY00 |
| hostname | github.com |
| expected-hostname | github.com |
| None | 87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456 |
| turbo-cache-control | no-preview |
| go-import | github.com/python/cpython git https://github.com/python/cpython.git |
| octolytics-dimension-user_id | 1525981 |
| octolytics-dimension-user_login | python |
| octolytics-dimension-repository_id | 81598961 |
| octolytics-dimension-repository_nwo | python/cpython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 81598961 |
| octolytics-dimension-repository_network_root_nwo | python/cpython |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 423dbb54c5e7bfecc838eb387c1435eb3d0e760d |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width