Title: Uniform capability-scoped file I/O on wasm32 (wasi/non-web/web) and native via host_io · Issue #9853 · RustPython/RustPython · GitHub
Open Graph Title: Uniform capability-scoped file I/O on wasm32 (wasi/non-web/web) and native via host_io · Issue #9853 · RustPython/RustPython
X Title: Uniform capability-scoped file I/O on wasm32 (wasi/non-web/web) and native via host_io · Issue #9853 · RustPython/RustPython
Description: Goal The same file I/O behavior on every target, behind one feature: wasm32-wasip1 wasm32-unknown-unknown without JS (embedded guest) wasm32-unknown-unknown in the browser native targets rustpython-vm/host_io becomes capability-scoped fi...
Open Graph Description: Goal The same file I/O behavior on every target, behind one feature: wasm32-wasip1 wasm32-unknown-unknown without JS (embedded guest) wasm32-unknown-unknown in the browser native targets rustpython...
X Description: Goal The same file I/O behavior on every target, behind one feature: wasm32-wasip1 wasm32-unknown-unknown without JS (embedded guest) wasm32-unknown-unknown in the browser native targets rustpython...
Opengraph URL: https://github.com/RustPython/RustPython/issues/9853
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Uniform capability-scoped file I/O on wasm32 (wasi/non-web/web) and native via host_io","articleBody":"## Goal\n\nThe same file I/O behavior on every target, behind one feature:\n\n- `wasm32-wasip1`\n- `wasm32-unknown-unknown` without JS (embedded guest)\n- `wasm32-unknown-unknown` in the browser\n- native targets\n\n`rustpython-vm/host_io` becomes **capability-scoped file I/O on every platform**. It works through preopened directories and WASI rights, so the same Python code behaves the same way everywhere. `host_env` keeps the existing ambient behavior: libc plus the whole host OS, CPython semantics.\n\nThe contract is the **WASI preview1 subset** introduced in #9833 (`crates/host_env/src/wasi_p1.rs`). Platforms differ only in who implements that contract.\n\n| Target | Guest code | Contract provided by |\n|---|---|---|\n| wasm32-wasip1 | wasi-libc (unchanged, see \"Out of scope\") | WASI runtime (wasmtime, wasmer) |\n| wasm32-unknown, non-web | `host_env::wasi_p1` imports (#9833) | embedder, through the shared host crate (step 1) |\n| wasm32-unknown, web | `host_env::wasi_p1` imports | JS shim (step 4) |\n| native, `host_io` without `host_env` | same semantics, in-process calls | shared host crate, called directly (step 2) |\n| native, `host_env` | existing libc `crt_fd`/`io` | host OS (ambient, unchanged) |\n\n## Decisions already made\n\n1. Native `host_io` (without `host_env`) is a sandbox. It opens only what the embedder preopened, and FileIO on a raw integer fd only reaches descriptors from the capability table: fds 0, 1 and 2 are not the process stdio unless the embedder lends them. `host_env` stays ambient.\n2. Renumber `errno_wasm` to WASI errno values so that `e.errno` is numerically identical on wasm32-wasip1 and wasm32-unknown (`ENOENT` = 44, `EACCES` = 2, ...). This changes errno numbers in the browser build; that is accepted.\n3. Stay on WASI **preview1**. CPython's tier-2 target is `wasm32-wasip1`, and wasmer, wasmtime, `node:wasi` and the browser shims all support it. Preview2 can be evaluated separately later.\n4. Out of scope for now: making `wasm32-wasip1` use `wasi_p1` directly instead of wasi-libc (it would need its own cwd/chdir). Keep wasi-libc there.\n\n## Invariants for every step\n\n- One implementation of access rules: preopens, rights, `..`/absolute/symlink escape, UTF-8 paths → `INVAL`, guest memory → `FAULT`. The example runner, native `host_io` and test hosts all use it; no second copy.\n- No process-global side tables for interpreter state. The native preopen/descriptor table belongs to the interpreter (`PyGlobalState` or an object it owns), not a `static`. The wasm guest's preopen cache (`OnceLock` in `wasi_p1.rs`) mirrors the host's process environment and can stay.\n- Every new WASI function lands in the shared host crate, the guest (`wasi_p1.rs`) and the contract test (`example_projects/wasm32_without_js/test_wasi_fs.py`) in the same PR, so all targets gain it together.\n- Contract tests assert exception classes and symbolic `errno.E*` names, never numeric literals.\n- Hot paths get no new runtime checks for the ambient `host_env` build.\n\n## Step 1: extract the shared WASI host crate\n\n- Move `example_projects/wasm32_without_js/wasm-runtime/src/wasi.rs` (preopen table, rights, cap-std path resolution, errno mapping, iovec/memory validation) into a workspace crate, proposed `crates/wasi_host` (`rustpython-wasi-host`). Its dependencies are `cap-std` (new workspace dependency) and `std`. It must not depend on `rustpython-host_env` or `rustpython-vm`.\n- Split it into two layers:\n - **core**: descriptor table and operations on Rust types (`&[u8]` buffers, `Result\u003c_, Errno>`);\n - **guest-memory adapter**: pointer/length validation and the `wasi_snapshot_preview1` signatures. The example runner wires this layer into wasmer; the native backend in step 2 uses only the core.\n- `wasm-runtime` becomes a thin wasmer binding over the crate. Its unit tests move with the code.\n- Acceptance: the `wasm-runtime` unit tests and `test_wasi_fs.py` (runner and wasmer modes) pass unchanged; `cargo test -p rustpython-wasi-host` runs in the main Rust CI job.\n\n## Step 2: native `host_io` backend (capability-scoped)\n\n- `rustpython-host_env` with `host_io` on native gets a capability backend over the step-1 core. FileIO uses it when `rustpython-vm/host_env` is **off**; with `host_env` on, FileIO keeps the ambient `crt_fd`/`io` (the `os` module needs ambient fds anyway).\n - Select this once, e.g. a single `fileio_backend` alias module chosen by cfg, instead of scattering cfgs through `crates/vm/src/stdlib/_io.rs`.\n - Note: the `host_env` crate cannot see the vm feature, so either forward a dedicated feature from vm or choose the backend in vm. Keep it to one switch for embedders.\n- Embedder API on `InterpreterBuilder` (`crates/vm/src/vm/interpreter.rs`):\n - `preopen_dir(host_path, guest_path, DirPerms)`, with read-only and read-write permissions;\n - optionally `lend_stdio(...)` to expose fds 0–2.\n - The table is stored in interpreter state, never in a static.\n- Path semantics are identical to the wasm guest: the current directory is `/`, the longest preopen prefix wins, and paths must be UTF-8.\n- Rewrite `crates/vm/tests/host_io.rs`. Today it uses ambient pipes and temp paths, so it becomes preopen-based: read/write/seek/truncate, read-only denial, escape denial, the `OSError.filename` identity, non-blocking read returning `None` on a lent pipe, and no posix/ctypes modules.\n- Add a third mode to `test_wasi_fs.py`: a small native embedding binary built with `host_io` only, accepting the same `--dir`/`--ro-dir` flags. It can be an example or test binary; do not add flags to the main `rustpython` CLI.\n- Acceptance: `test_wasi_fs.py` passes in runner, wasmer and native modes with the same snippets; the native `host_io` CI step runs the rewritten test.\n\n## Step 3: errno renumbering\n\n- `crates/host_env/src/errno_wasm.rs`: use WASI errno numbers (the table is in `wasi_p1.rs` and the WASI preview1 witx). The guest's WASI→guest errno translation in `wasi_p1.rs` then becomes the identity mapping; remove it.\n- Check every user of `errno_wasm` constants (`crates/vm/src/exceptions.rs` `errno_to_exc_type`, `crates/vm/src/stdlib/errno.rs` `ERROR_CODES`, the browser crate) and the strerror texts.\n- Acceptance: on wasm32-unknown and wasm32-wasip1, `errno.errorcode` agrees for every name both define; `test_wasi_fs.py` gains an assertion that `e.errno` is the same number in both wasm modes for ENOENT, EACCES/ENOTCAPABLE and EINVAL.\n\n## Step 4: web (browser crate)\n\n- `crates/wasm` enables `rustpython-vm/host_io`. The wasm-bindgen output then imports the `wasi_snapshot_preview1` subset, and the JS side must supply it.\n- Provide a JS shim in `wasm/` with the same semantics, rather than a hand-written partial one:\n - an in-memory directory tree by default;\n - OPFS-backed preopens available only inside a Worker (WASI calls are synchronous, and OPFS sync access handles exist only in Workers).\n - Either adapt an existing preview1 browser shim (for example `@bjorn3/browser_wasi_shim`) with a rights layer matching the shared host crate, or write a small one. Document the choice.\n- The JS API exposes `preopen(guestPath, files | handle, {readonly})`.\n- Run the contract snippets in CI under node (`node:wasi` or the shim) and/or in the existing headless browser test (`wasm/demo`, selenium + geckodriver).\n- Acceptance: the same `test_wasi_fs.py` snippets pass in a fourth \"web\" mode (read-only preopen, read/write/truncate, escapes, errno names).\n\n## Step 5: grow the contract\n\nAdd, in this order, each across all targets at once (host crate, guest, contract test):\n\n1. `fd_filestat_get`, `path_filestat_get` → `os.stat`/`fstat` subset\n2. `fd_readdir` → `os.listdir`/`scandir`\n3. `path_create_directory`, `path_remove_directory`, `path_unlink_file`\n4. `path_rename`\n5. `fd_fdstat_set_flags` (O_NONBLOCK / O_APPEND changes)\n\nOn top of that comes a WASI-backed `posix` subset for `host_io` builds plus importlib loading from preopens. The end goal is `-m test` on non-WASI wasm targets. That needs a separate design note when it starts; do not grow `host_io` into ambient `host_env`.\n\n## Risks and notes\n\n- **Runtime divergences seen in #9833 (wasmer 7.5):**\n - writes to a volume mounted on the guest root do not reach the host, so tests mount at `/work`;\n - wasmer allows writes on a read-only borrowed fd;\n - it reports escapes with a different `OSError` subclass.\n - Keep these as per-mode assertions in the contract test and report them upstream if they are bugs.\n- **Non-UTF-8 paths:** WASI preview1 paths are UTF-8, so non-UTF-8 byte names are rejected under `host_io` everywhere, including native. `host_env` builds keep raw bytes.\n- **Behavior change for native `host_io`-only embedders:** after step 2 they get sandbox semantics. That is intentional; call it out in the release notes and in `architecture/architecture.md` (\"Embedded file I/O\").\n\n## Suggested PR split\n\n1. Step 1, mechanical extraction, no behavior change\n2. Step 3, independent of 1 and 2, can go first\n3. Step 2\n4. Step 4\n5. Step 5, one PR per WASI function group\n\nEach PR: the contract test passes in every mode that exists at that point; clippy/fmt; and for Rust changes, the checks in AGENTS.md.\n\nContext: #9833 (WASI preview1 subset for the wasm32-unknown guest, example runner, contract test), #9843 (FileIO decoupled from `os`/`msvcrt`).\n\n— *commented by Claude*\n","author":{"url":"https://github.com/youknowone","@type":"Person","name":"youknowone"},"datePublished":"2026-11-15T11:39:07Z","url":"https://github.com/RustPython/RustPython/issues/9853"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:8919fb98-6c41-d215-09fa-da001067ba4d |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | B582:36A3E:2D34C78:3C54A18:6B01D7D5 |
| html-safe-nonce | aa478afba8a1d19be49488d20a96d651a964531bd48345ec33f8aabd5fd0dc75 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJCNTgyOjM2QTNFOjJEMzRDNzg6M0M1NEExODo2QjAxRDdENSIsInZpc2l0b3JfaWQiOiIyMjMyNTQzNTUyOTQ3MjE0MjkzIiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | b9b7075f349a3c70e5709b70dfffd8a41527ce0db844ec804521a8be259e9192 |
| hovercard-subject-tag | issue:6403480290 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/RustPython/RustPython/9853/issue_layout |
| twitter:image | https://opengraph.githubassets.com/1f2912e4dd0dd82787f1628ec6939b4d9355909d79300ea3c86344b0c1de6c09/RustPython/RustPython/issues/9853 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/1f2912e4dd0dd82787f1628ec6939b4d9355909d79300ea3c86344b0c1de6c09/RustPython/RustPython/issues/9853 |
| og:image:alt | Goal The same file I/O behavior on every target, behind one feature: wasm32-wasip1 wasm32-unknown-unknown without JS (embedded guest) wasm32-unknown-unknown in the browser native targets rustpython... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | youknowone |
| hostname | github.com |
| expected-hostname | github.com |
| None | cc5dfdd32f62ebd3223cab3af3aa2b268d3605218237326a48a99d1efd89495c |
| turbo-cache-control | no-preview |
| go-import | github.com/RustPython/RustPython git https://github.com/RustPython/RustPython.git |
| octolytics-dimension-user_id | 39710557 |
| octolytics-dimension-user_login | RustPython |
| octolytics-dimension-repository_id | 135201145 |
| octolytics-dimension-repository_nwo | RustPython/RustPython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 135201145 |
| octolytics-dimension-repository_network_root_nwo | RustPython/RustPython |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 8c77c9e8439031ad54c75a250694842892f693e1 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width