René's URL Explorer Experiment


Title: Uniform capability-scoped file I/O on wasm32 (wasi/non-web/web) and native via host_io · Issue #9853 · RustPython/RustPython · GitHub

Open Graph Title: Uniform capability-scoped file I/O on wasm32 (wasi/non-web/web) and native via host_io · Issue #9853 · RustPython/RustPython

X Title: Uniform capability-scoped file I/O on wasm32 (wasi/non-web/web) and native via host_io · Issue #9853 · RustPython/RustPython

Description: Goal The same file I/O behavior on every target, behind one feature: wasm32-wasip1 wasm32-unknown-unknown without JS (embedded guest) wasm32-unknown-unknown in the browser native targets rustpython-vm/host_io becomes capability-scoped fi...

Open Graph Description: Goal The same file I/O behavior on every target, behind one feature: wasm32-wasip1 wasm32-unknown-unknown without JS (embedded guest) wasm32-unknown-unknown in the browser native targets rustpython...

X Description: Goal The same file I/O behavior on every target, behind one feature: wasm32-wasip1 wasm32-unknown-unknown without JS (embedded guest) wasm32-unknown-unknown in the browser native targets rustpython...

Opengraph URL: https://github.com/RustPython/RustPython/issues/9853

X: @github

direct link

Domain: github.com


Hey, it has json ld scripts:
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Uniform capability-scoped file I/O on wasm32 (wasi/non-web/web) and native via host_io","articleBody":"## Goal\n\nThe same file I/O behavior on every target, behind one feature:\n\n- `wasm32-wasip1`\n- `wasm32-unknown-unknown` without JS (embedded guest)\n- `wasm32-unknown-unknown` in the browser\n- native targets\n\n`rustpython-vm/host_io` becomes **capability-scoped file I/O on every platform**. It works through preopened directories and WASI rights, so the same Python code behaves the same way everywhere. `host_env` keeps the existing ambient behavior: libc plus the whole host OS, CPython semantics.\n\nThe contract is the **WASI preview1 subset** introduced in #9833 (`crates/host_env/src/wasi_p1.rs`). Platforms differ only in who implements that contract.\n\n| Target | Guest code | Contract provided by |\n|---|---|---|\n| wasm32-wasip1 | wasi-libc (unchanged, see \"Out of scope\") | WASI runtime (wasmtime, wasmer) |\n| wasm32-unknown, non-web | `host_env::wasi_p1` imports (#9833) | embedder, through the shared host crate (step 1) |\n| wasm32-unknown, web | `host_env::wasi_p1` imports | JS shim (step 4) |\n| native, `host_io` without `host_env` | same semantics, in-process calls | shared host crate, called directly (step 2) |\n| native, `host_env` | existing libc `crt_fd`/`io` | host OS (ambient, unchanged) |\n\n## Decisions already made\n\n1. Native `host_io` (without `host_env`) is a sandbox. It opens only what the embedder preopened, and FileIO on a raw integer fd only reaches descriptors from the capability table: fds 0, 1 and 2 are not the process stdio unless the embedder lends them. `host_env` stays ambient.\n2. Renumber `errno_wasm` to WASI errno values so that `e.errno` is numerically identical on wasm32-wasip1 and wasm32-unknown (`ENOENT` = 44, `EACCES` = 2, ...). This changes errno numbers in the browser build; that is accepted.\n3. Stay on WASI **preview1**. CPython's tier-2 target is `wasm32-wasip1`, and wasmer, wasmtime, `node:wasi` and the browser shims all support it. Preview2 can be evaluated separately later.\n4. Out of scope for now: making `wasm32-wasip1` use `wasi_p1` directly instead of wasi-libc (it would need its own cwd/chdir). Keep wasi-libc there.\n\n## Invariants for every step\n\n- One implementation of access rules: preopens, rights, `..`/absolute/symlink escape, UTF-8 paths → `INVAL`, guest memory → `FAULT`. The example runner, native `host_io` and test hosts all use it; no second copy.\n- No process-global side tables for interpreter state. The native preopen/descriptor table belongs to the interpreter (`PyGlobalState` or an object it owns), not a `static`. The wasm guest's preopen cache (`OnceLock` in `wasi_p1.rs`) mirrors the host's process environment and can stay.\n- Every new WASI function lands in the shared host crate, the guest (`wasi_p1.rs`) and the contract test (`example_projects/wasm32_without_js/test_wasi_fs.py`) in the same PR, so all targets gain it together.\n- Contract tests assert exception classes and symbolic `errno.E*` names, never numeric literals.\n- Hot paths get no new runtime checks for the ambient `host_env` build.\n\n## Step 1: extract the shared WASI host crate\n\n- Move `example_projects/wasm32_without_js/wasm-runtime/src/wasi.rs` (preopen table, rights, cap-std path resolution, errno mapping, iovec/memory validation) into a workspace crate, proposed `crates/wasi_host` (`rustpython-wasi-host`). Its dependencies are `cap-std` (new workspace dependency) and `std`. It must not depend on `rustpython-host_env` or `rustpython-vm`.\n- Split it into two layers:\n  - **core**: descriptor table and operations on Rust types (`&[u8]` buffers, `Result\u003c_, Errno>`);\n  - **guest-memory adapter**: pointer/length validation and the `wasi_snapshot_preview1` signatures. The example runner wires this layer into wasmer; the native backend in step 2 uses only the core.\n- `wasm-runtime` becomes a thin wasmer binding over the crate. Its unit tests move with the code.\n- Acceptance: the `wasm-runtime` unit tests and `test_wasi_fs.py` (runner and wasmer modes) pass unchanged; `cargo test -p rustpython-wasi-host` runs in the main Rust CI job.\n\n## Step 2: native `host_io` backend (capability-scoped)\n\n- `rustpython-host_env` with `host_io` on native gets a capability backend over the step-1 core. FileIO uses it when `rustpython-vm/host_env` is **off**; with `host_env` on, FileIO keeps the ambient `crt_fd`/`io` (the `os` module needs ambient fds anyway).\n  - Select this once, e.g. a single `fileio_backend` alias module chosen by cfg, instead of scattering cfgs through `crates/vm/src/stdlib/_io.rs`.\n  - Note: the `host_env` crate cannot see the vm feature, so either forward a dedicated feature from vm or choose the backend in vm. Keep it to one switch for embedders.\n- Embedder API on `InterpreterBuilder` (`crates/vm/src/vm/interpreter.rs`):\n  - `preopen_dir(host_path, guest_path, DirPerms)`, with read-only and read-write permissions;\n  - optionally `lend_stdio(...)` to expose fds 0–2.\n  - The table is stored in interpreter state, never in a static.\n- Path semantics are identical to the wasm guest: the current directory is `/`, the longest preopen prefix wins, and paths must be UTF-8.\n- Rewrite `crates/vm/tests/host_io.rs`. Today it uses ambient pipes and temp paths, so it becomes preopen-based: read/write/seek/truncate, read-only denial, escape denial, the `OSError.filename` identity, non-blocking read returning `None` on a lent pipe, and no posix/ctypes modules.\n- Add a third mode to `test_wasi_fs.py`: a small native embedding binary built with `host_io` only, accepting the same `--dir`/`--ro-dir` flags. It can be an example or test binary; do not add flags to the main `rustpython` CLI.\n- Acceptance: `test_wasi_fs.py` passes in runner, wasmer and native modes with the same snippets; the native `host_io` CI step runs the rewritten test.\n\n## Step 3: errno renumbering\n\n- `crates/host_env/src/errno_wasm.rs`: use WASI errno numbers (the table is in `wasi_p1.rs` and the WASI preview1 witx). The guest's WASI→guest errno translation in `wasi_p1.rs` then becomes the identity mapping; remove it.\n- Check every user of `errno_wasm` constants (`crates/vm/src/exceptions.rs` `errno_to_exc_type`, `crates/vm/src/stdlib/errno.rs` `ERROR_CODES`, the browser crate) and the strerror texts.\n- Acceptance: on wasm32-unknown and wasm32-wasip1, `errno.errorcode` agrees for every name both define; `test_wasi_fs.py` gains an assertion that `e.errno` is the same number in both wasm modes for ENOENT, EACCES/ENOTCAPABLE and EINVAL.\n\n## Step 4: web (browser crate)\n\n- `crates/wasm` enables `rustpython-vm/host_io`. The wasm-bindgen output then imports the `wasi_snapshot_preview1` subset, and the JS side must supply it.\n- Provide a JS shim in `wasm/` with the same semantics, rather than a hand-written partial one:\n  - an in-memory directory tree by default;\n  - OPFS-backed preopens available only inside a Worker (WASI calls are synchronous, and OPFS sync access handles exist only in Workers).\n  - Either adapt an existing preview1 browser shim (for example `@bjorn3/browser_wasi_shim`) with a rights layer matching the shared host crate, or write a small one. Document the choice.\n- The JS API exposes `preopen(guestPath, files | handle, {readonly})`.\n- Run the contract snippets in CI under node (`node:wasi` or the shim) and/or in the existing headless browser test (`wasm/demo`, selenium + geckodriver).\n- Acceptance: the same `test_wasi_fs.py` snippets pass in a fourth \"web\" mode (read-only preopen, read/write/truncate, escapes, errno names).\n\n## Step 5: grow the contract\n\nAdd, in this order, each across all targets at once (host crate, guest, contract test):\n\n1. `fd_filestat_get`, `path_filestat_get` → `os.stat`/`fstat` subset\n2. `fd_readdir` → `os.listdir`/`scandir`\n3. `path_create_directory`, `path_remove_directory`, `path_unlink_file`\n4. `path_rename`\n5. `fd_fdstat_set_flags` (O_NONBLOCK / O_APPEND changes)\n\nOn top of that comes a WASI-backed `posix` subset for `host_io` builds plus importlib loading from preopens. The end goal is `-m test` on non-WASI wasm targets. That needs a separate design note when it starts; do not grow `host_io` into ambient `host_env`.\n\n## Risks and notes\n\n- **Runtime divergences seen in #9833 (wasmer 7.5):**\n  - writes to a volume mounted on the guest root do not reach the host, so tests mount at `/work`;\n  - wasmer allows writes on a read-only borrowed fd;\n  - it reports escapes with a different `OSError` subclass.\n  - Keep these as per-mode assertions in the contract test and report them upstream if they are bugs.\n- **Non-UTF-8 paths:** WASI preview1 paths are UTF-8, so non-UTF-8 byte names are rejected under `host_io` everywhere, including native. `host_env` builds keep raw bytes.\n- **Behavior change for native `host_io`-only embedders:** after step 2 they get sandbox semantics. That is intentional; call it out in the release notes and in `architecture/architecture.md` (\"Embedded file I/O\").\n\n## Suggested PR split\n\n1. Step 1, mechanical extraction, no behavior change\n2. Step 3, independent of 1 and 2, can go first\n3. Step 2\n4. Step 4\n5. Step 5, one PR per WASI function group\n\nEach PR: the contract test passes in every mode that exists at that point; clippy/fmt; and for Rust changes, the checks in AGENTS.md.\n\nContext: #9833 (WASI preview1 subset for the wasm32-unknown guest, example runner, contract test), #9843 (FileIO decoupled from `os`/`msvcrt`).\n\n— *commented by Claude*\n","author":{"url":"https://github.com/youknowone","@type":"Person","name":"youknowone"},"datePublished":"2026-11-15T11:39:07Z","url":"https://github.com/RustPython/RustPython/issues/9853"}

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format)
route-controllervoltron_issues_fragments
route-actionissue_layout
fetch-noncev2:8919fb98-6c41-d215-09fa-da001067ba4d
current-catalog-service-hash81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114
request-idB582:36A3E:2D34C78:3C54A18:6B01D7D5
html-safe-nonceaa478afba8a1d19be49488d20a96d651a964531bd48345ec33f8aabd5fd0dc75
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJCNTgyOjM2QTNFOjJEMzRDNzg6M0M1NEExODo2QjAxRDdENSIsInZpc2l0b3JfaWQiOiIyMjMyNTQzNTUyOTQ3MjE0MjkzIiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0=
visitor-hmacb9b7075f349a3c70e5709b70dfffd8a41527ce0db844ec804521a8be259e9192
hovercard-subject-tagissue:6403480290
github-keyboard-shortcutsrepository,issues,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///voltron/issues_fragments/issue_layout
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/RustPython/RustPython/9853/issue_layout
twitter:imagehttps://opengraph.githubassets.com/1f2912e4dd0dd82787f1628ec6939b4d9355909d79300ea3c86344b0c1de6c09/RustPython/RustPython/issues/9853
twitter:cardsummary_large_image
og:imagehttps://opengraph.githubassets.com/1f2912e4dd0dd82787f1628ec6939b4d9355909d79300ea3c86344b0c1de6c09/RustPython/RustPython/issues/9853
og:image:altGoal The same file I/O behavior on every target, behind one feature: wasm32-wasip1 wasm32-unknown-unknown without JS (embedded guest) wasm32-unknown-unknown in the browser native targets rustpython...
og:image:width1200
og:image:height600
og:site_nameGitHub
og:typeobject
og:author:usernameyouknowone
hostnamegithub.com
expected-hostnamegithub.com
Nonecc5dfdd32f62ebd3223cab3af3aa2b268d3605218237326a48a99d1efd89495c
turbo-cache-controlno-preview
go-importgithub.com/RustPython/RustPython git https://github.com/RustPython/RustPython.git
octolytics-dimension-user_id39710557
octolytics-dimension-user_loginRustPython
octolytics-dimension-repository_id135201145
octolytics-dimension-repository_nwoRustPython/RustPython
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id135201145
octolytics-dimension-repository_network_root_nwoRustPython/RustPython
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release8c77c9e8439031ad54c75a250694842892f693e1
ui-targetfull
theme-color#1e2327
color-schemelight dark
react-app-nameissues

Links:

Skip to contenthttps://github.com/RustPython/RustPython/issues/9853#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2FRustPython%2FRustPython%2Fissues%2F9853
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2FRustPython%2FRustPython%2Fissues%2F9853
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fvoltron%2Fissues_fragments%2Fissue_layout&source=header-repo&source_repo=RustPython%2FRustPython
Reloadhttps://github.com/RustPython/RustPython/issues/9853
Reloadhttps://github.com/RustPython/RustPython/issues/9853
Reloadhttps://github.com/RustPython/RustPython/issues/9853
RustPythonhttps://github.com/RustPython
RustPythonhttps://github.com/RustPython/RustPython
Codehttps://github.com/RustPython/RustPython
Issues195 (195)https://github.com/RustPython/RustPython/issues
Pull requests132 (132)https://github.com/RustPython/RustPython/pulls
Discussionshttps://github.com/RustPython/RustPython/discussions
Actionshttps://github.com/RustPython/RustPython/actions
Projectshttps://github.com/RustPython/RustPython/projects
Wikihttps://github.com/RustPython/RustPython/wiki
Security and qualityhttps://github.com/RustPython/RustPython/security
Insightshttps://github.com/RustPython/RustPython/insights
https://github.com/RustPython/RustPython
New issuehttps://github.com/login?return_to=https://github.com/RustPython/RustPython/issues/9853
New issuehttps://github.com/login?return_to=https://github.com/RustPython/RustPython/issues/9853
#9869https://github.com/RustPython/RustPython/pull/9869
Uniform capability-scoped file I/O on wasm32 (wasi/non-web/web) and native via host_iohttps://github.com/RustPython/RustPython/issues/9853#top
#9869https://github.com/RustPython/RustPython/pull/9869
https://github.com/youknowone
youknowonehttps://github.com/youknowone
on Nov 15, 2026https://github.com/RustPython/RustPython/issues/9853#issue-6403480290
#9833https://github.com/RustPython/RustPython/pull/9833
#9833https://github.com/RustPython/RustPython/pull/9833
Add WASI preview1 file I/O to the wasm32-unknown guest via host_io #9833https://github.com/RustPython/RustPython/pull/9833
#9833https://github.com/RustPython/RustPython/pull/9833
#9843https://github.com/RustPython/RustPython/pull/9843
youknowonehttps://github.com/youknowone
on Nov 16, 2026https://github.com/RustPython/RustPython/issues/9853#event-36242976340
Align wasm32-unknown errno values with WASI preview1 (#9868)https://github.com/RustPython/RustPython/commit/519dec078f0301ce2c49e77920caa95a41b4ea72
#9868https://github.com/RustPython/RustPython/pull/9868
519dec0https://github.com/RustPython/RustPython/commit/519dec078f0301ce2c49e77920caa95a41b4ea72
youknowonehttps://github.com/youknowone
completedhttps://github.com/RustPython/RustPython/issues?q=is%3Aissue%20state%3Aclosed%20archived%3Afalse%20reason%3Acompleted
#9869https://github.com/RustPython/RustPython/pull/9869
on Nov 16, 2026https://github.com/RustPython/RustPython/issues/9853#event-36281781400
Sign up for freehttps://github.com/signup?return_to=https://github.com/RustPython/RustPython/issues/9853
Sign in to commenthttps://github.com/login?return_to=https://github.com/RustPython/RustPython/issues/9853
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.