Title: Add pyo3 module bridging to an embedded CPython by youknowone · Pull Request #6301 · RustPython/RustPython · GitHub
Open Graph Title: Add pyo3 module bridging to an embedded CPython by youknowone · Pull Request #6301 · RustPython/RustPython
X Title: Add pyo3 module bridging to an embedded CPython by youknowone · Pull Request #6301 · RustPython/RustPython
Description: Adds an optional pyo3 module (crate rustpython-module_pyo3, cargo feature pyo3) that loads CPython into the process through PyO3 and lets RustPython code use CPython modules, including C extensions RustPython does not implement. import pyo3 np = pyo3.import_module("numpy") a = np.array([1, 2, 3]) print(a * 2, a.sum()) @pyo3.wraps def max_prec(): import _decimal return _decimal.MAX_PREC cargo build --release --features pyo3 ./target/release/rustpython crates/module_pyo3/tests/test_module_pyo3.py Module API pyo3.import_module(name) imports a module in the embedded CPython. It does not touch this interpreter's sys.modules or import machinery. pyo3.wraps compiles the decorated function's source in CPython and runs calls there. pyo3.load_embedded(data) is the reduce target used when pickling embedded objects. rustpython_module_pyo3::borrow_module(name, vm) (Rust API) registers a CPython module in sys.modules under its own name, e.g. to use the embedded _ctypes. Conversion rules None, bool, int, float, complex, str, bytes and tuples of them are copied in both directions. CPython objects arrive as instances of a RustPython wrapper of their class. Attribute access, slots and identity are forwarded. A wrapper's metaclass is the wrapper of the CPython metaclass. An immutable metaclass whose own metaclass is type (e.g. ctypes' PyCStructType) gets type. RustPython list, dict, set, frozenset and bytearray are copied into CPython as the exact built-in, because C code checks them with PyList_Check and similar. CPython's changes to such a copy do not reach the RustPython original. A subclass of those containers is pickled, so e.g. OrderedDict, Counter and defaultdict arrive as the CPython class of the same name. When pickling fails, the subclass is copied as the base built-in. It is never proxied. Subclasses of int, float, complex, str and bytes are copied as the base value. Other buffers are shared through a memoryview. Callables, data descriptors and objects with a number protocol become rustpython.object proxies. The proxy class carries exactly the slots the wrapped class defines, and one proxy class is cached per slot set. Non-slot dunders the class defines (e.g. __array__, __array_ufunc__) are published, and a None definition opts out. Anything else is pickled into CPython when its class overrides __eq__, because only then can a copy not be told apart from the original. It is proxied otherwise. An object compared by identity, such as io.BytesIO, is always proxied, so CPython reads and writes the original. Exceptions cross with their type mapped. The original RustPython exception is restored when it comes back. copy/deepcopy of embedded objects delegate to the embedded copy module. pickle stores the embedded pickle bytes behind pyo3.load_embedded. Embedded warnings.showwarning is delivered to RustPython's warnings while a RustPython thread runs. The unraisable hook is routed the same way. The GIL is released while RustPython code runs and is attached only around CPython calls. Changes in crates/vm PyType::new_heap selects the most derived solid-layout base instead of bases[0]. Python-level BASETYPE checks remain in best_base; low-level native type construction retains its existing contract. New PyType::assign_version_tag, which assigns a version tag under the type lock and returns it, or 0 when none can be assigned. Testing crates/module_pyo3/tests/test_module_pyo3.py covers the bridge using only the stdlib through pyo3. A new pyo3 CI job builds with --features pyo3 against CPython 3.14 and runs it on Linux, macOS and Windows. Locally (Windows, CPython 3.14.2) I also ran test_ctypes with the embedded _ctypes borrowed in place of RustPython's. 4 tests fail: 2 depend on refcounts or on id() being a C address, and the 2 bitfield message tests also fail on CPython 3.14.2 itself. A numpy 2.5.3 probe of 47 cases passes 44. Plain CPython gives the same result for 2 of the misses: the memoryview format 'l' on Windows and det rounding. The third, frombuffer(bytearray) not sharing memory, follows from the copy rule above. A probe of 79 cases ran across pandas, scipy, pillow, lxml, orjson, msgpack, regex, cryptography, pydantic and pyyaml, and 76 pass. The 3 that fail all pass a RustPython dataclass or YAML representer hook, which arrives as a proxy that the library's exact type check rejects. The module name pyo3 is provisional. 🤖 Generated with Claude Code Rebase validation Rebased onto f1f38c1594af492dac1cc934aa9431a51a0f9a83. Removed the redundant retained-copy Drop implementation and Option, adapted metaclass dispatch to the constructor/instance vectorcall split, and updated exception notes to the typed add_note_str API. Native constructor identity checks remain unchanged. On Linux with Rust 1.99.0, against the published tree: Workspace tests with threading,pyo3, excluding wasm, venvlauncher and C-API: 1,500 passed, 0 failed, 20 ignored, including doctests. Separate C-API tests: 149 passed, 0 failed, 2 ignored. Strict Clippy (--all-targets -- -D warnings) passed for the workspace and separately for C-API. Pre-commit hooks and formatting passed. Debug runtime probes passed for retained ctypes byte copies, exception notes, and Python BASETYPE restrictions. The unmodified bridge script with the available CPython 3.12 stops at line 316 when CPython rejects type.__new__(_ctypes.PyCStructType). The same allocation operation fails directly in CPython 3.12; RustPython's constructor safety check is not the source. Full bridge validation against CPython 3.14 is still pending the existing Linux/macOS/Windows pyo3 CI matrix. This remains a draft. The rebase, cleanup, integration fixes and automated validation were assisted by OpenAI Codex (model version identifier unavailable).
Open Graph Description: Adds an optional pyo3 module (crate rustpython-module_pyo3, cargo feature pyo3) that loads CPython into the process through PyO3 and lets RustPython code use CPython modules, including C extensions...
X Description: Adds an optional pyo3 module (crate rustpython-module_pyo3, cargo feature pyo3) that loads CPython into the process through PyO3 and lets RustPython code use CPython modules, including C extensions...
Opengraph URL: https://github.com/RustPython/RustPython/pull/6301
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:39743d02-e726-5698-3666-21e87617e65a |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | E94C:EAD6D:25524:2DD8D:6B01F9B3 |
| html-safe-nonce | 1f7a1bca823bebc57c8abd96f62cc501e43e7593915f5d6072db5d4d579fab7b |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJFOTRDOkVBRDZEOjI1NTI0OjJERDhEOjZCMDFGOUIzIiwidmlzaXRvcl9pZCI6IjYxODA5NzI3MTgzMzk1MjA5NDciLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | ecbbbf9e495e96fc529ca5dd18d4b1236635ba1b8276cf341dcc8fdebc521235 |
| hovercard-subject-tag | pull_request:3052896161 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/RustPython/RustPython/pull/6301/files |
| twitter:image | https://avatars.githubusercontent.com/u/69878?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/69878?s=400&v=4 |
| og:image:alt | Adds an optional pyo3 module (crate rustpython-module_pyo3, cargo feature pyo3) that loads CPython into the process through PyO3 and lets RustPython code use CPython modules, including C extensions... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | 87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/RustPython/RustPython git https://github.com/RustPython/RustPython.git |
| octolytics-dimension-user_id | 39710557 |
| octolytics-dimension-user_login | RustPython |
| octolytics-dimension-repository_id | 135201145 |
| octolytics-dimension-repository_nwo | RustPython/RustPython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 135201145 |
| octolytics-dimension-repository_network_root_nwo | RustPython/RustPython |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 423dbb54c5e7bfecc838eb387c1435eb3d0e760d |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width