Title: Add tracing profiler and Python 3.15 offline profiling tools by youknowone · Pull Request #9271 · RustPython/RustPython · GitHub
Open Graph Title: Add tracing profiler and Python 3.15 offline profiling tools by youknowone · Pull Request #9271 · RustPython/RustPython
X Title: Add tracing profiler and Python 3.15 offline profiling tools by youknowone · Pull Request #9271 · RustPython/RustPython
Description: Summary Split the tracing profiler and canonical profiling tools from #8954 after prerequisite #9215 landed. Target remains CPython 3.15.0, pinned to d220409a13b77de7a6c30abe16ea90db01528edc. Add native _lsprof with operation exclusion, short payload borrows, owned callback roots, interpreter-affinity checks and cyclic-GC traversal of owned code references/caches. Import canonical cProfile, profiling.tracing, profiling.sampling offline tools, corresponding tests/assets and profile/pstats updates through update_lib. Include canonical asyncio.tools and its paired tests for the FrameInfo location schema. No manually edited Lib snippets or new test skips. Add native LocationInfo/FrameInfo/ThreadInfo/InterpreterInfo records, six thread-status constants, and a real uncompressed BinaryWriter/BinaryReader implementation. The codec supports FULL, REPEAT, SUFFIX and POP_PUSH records, tables, footer/statistics, repeat batching and opposite-endian reading. File access uses host_env; callback invocation occurs outside native state locks. Preserve accurate builtin labels through raw-receiver normalization and constructor-time static-method metadata. Match CPython's classmethod convention for float.getformat. Scope: 79 changed files, including 51 canonical/updater-generated Lib outputs. License and copyright terms are preserved. Canonical package closure, without namespace emulation The previous frozen namespace-container extension is completely reverted in a separate logical commit. Its synthetic-package field, serialization marker and priority changes are absent. The resource support below is a separate loader capability for canonical regular packages; no namespace emulation or FrozenModule/FrozenLib wire-format change is introduced. The canonical profiling root and selected sampling production/test/assets closure are restored together, including the binary reader/writer Python wrappers and tests. Normal packages therefore work through existing filesystem and frozen import semantics, including canonical wildcard exports. Tests were not removed in isolation to avoid failures. Explicit capability boundaries This is offline file support, not live process sampling. RemoteUnwinder remains the existing NotImplemented implementation. PROCESS_VM_READV_SUPPORTED remains absent exactly as on main; no false capability flag or process reader/writer/attachment implementation is added. Importing the canonical offline production modules works, but constructing a live SampleProfiler still reaches the unsupported unwinder. zstd_available() is false; zstd files/requests receive an unsupported-compression error rather than being mislabeled or silently decoded as raw data. Existing canonical capability gates still skip test_collectors, test_binary_format and test_external_inspection as whole modules. These are skips, not passing suites. No capability injection or skip edits were used for validation. Canonical pstats sampled-file sort_stats("samples") still reproduces an upstream KeyError on pinned CPython; this PR does not claim to fix it. Reader safety deliberately differs from pinned CPython in repeated/simultaneous replay: each replay owns independent decoding state, concurrent replay on the same open reader is rejected, and close/reinitialization cannot invalidate the owned snapshot during callbacks. File parsing checks lengths/arithmetic and allocates from actual validated payload, not advertised counts. These checks are not claims of malformed-input, OOM or race-stress coverage. Fresh offline-correction validation Strict all-target Clippy for rustpython-stdlib, plus final threading and non-threading stdlib compile checks. Fresh source-bound default and freeze-stdlib Linux CLI builds, separately sealed. Both report version 3.15.0; source manifests and binary hashes were recorded. Embedded git-description text is not used as the source receipt. A bounded valid owned-file matrix with pinned CPython: both writers produced byte-identical 362-byte files, containing ten observations and all four encodings. CPython→RustPython, RustPython→CPython and both self-reads matched records, callback ordering, progress, metadata and statistics exactly. Canonical root/tracing/sampling/cProfile and binary-module imports passed from the exact filesystem Lib tree. The exact canonical all helper passed for profiling and profiling.tracing. Fresh frozen CLI import-only proof with sys.path empty: root, tracing, sampling, cProfile and both binary modules all had frozen origins. All 20 specifically reviewed canonical asyncio.tools mock cases passed individually on candidate and pinned CPython. No external inspection was invoked. All eight previously reviewed canonical CProfileTest cases passed individually again: calling conventions, context manager, enable/disable, run, sorted run, runctx, second profiler and generator throw. Existing expected-failure decorators/assertions were preserved. Normal hooks and commit hooks passed; post-hook source was checked against the validated source manifest. These are scoped local gates, not a complete profiler/binary/native suite. Full workspace/CAPI compilation and runtime suites were not repeated for the offline correction. No live-process, fork/ptrace, malformed-file, large-allocation, deep-recursion, reentrant close/timer/deallocation or crash probes were run. One Cargo job, no incremental compilation and time/memory/disk guards were used. Correctness gates are not performance measurements. Earlier proof and inherited limitations Earlier revision 94f8b2b passed workspace/CAPI no-run compilation and strict all-target Clippy, 16 selected VM and 11 selected CAPI cases, and paired builtin-label/module-mutation/float-binding oracles. That broad proof belongs to that historical source, not the new offline codec. A bounded owned-thread oracle recorded three worker calls on both pinned CPython and this implementation. CPython also uses interpreter-wide monitoring and does not filter to the enabling thread. Disable callback/event ordering likewise follows pinned CPython. Neither observation establishes race safety. The lock acquisition/handoff concerns inherited from #9215 are not fixed by this consumer or by the prerequisite's merge, including #9215 (comment) . Ordinary passing tests do not resolve those races. Earlier CI failures for the incomplete root all and the review finding against the frozen namespace workaround are addressed by the coherent canonical package restoration, not suppressed assertions or fabricated sampling capability. Public CI remains separate from local proof. Skipped pyperf or CodSpeed execution is not a performance pass. AI assistance Prepared, adapted and validated with dot (GPT-6.1). Native codec/API, callback ownership and compatibility changes were independently source-reviewed. Canonical library files were generated with the repository updater, not rewritten manually. Windows conversion lint follow-up Windows Clippy on 7e3b98e flagged the c_long-to-i32 location conversion as same-type. e8f0bcf adds a documented allowance on that conversion only, retaining LP64 truncation and Windows identity semantics. Linux strict affected all-target Clippy and normal commit hooks passed again. Runtime/codec behavior is unchanged; hosted Windows Clippy passed on e8f0bcf (job 119328656630). Packaging and resource follow-up eab90b8 keeps the profiling asset notice solely at crates/pylib/LICENSE-profiling, where the vendored Lib assets are packaged, and removes the redundant root copy. Desktop resource metadata references that crate-local path. The notice bytes and copyright/license terms are unchanged; cargo package --list includes the notice alongside the assets, all desktop resource inputs exist, and normal hooks passed. This is a packaging-only correction, not a new native/runtime proof. The frozen report-resource finding is addressed by the native resource-reader correction below. This is resource availability/protocol proof, not an executed HTML exporter, browser or wasm-runtime test. The sampled-pstats dump marker issue also reproduces identically on pinned CPython: a 74-byte sampled input loads as SampledStats, dump_stats writes 58 bytes without the marker, and the result reloads as Stats. No manual Lib fix is applied or claimed. Native frozen resource correction 1200603 adds a native FrozenImporter ResourceReader/Traversable capability without canonical importlib edits or a frozen bytecode format change. Exactly 16 canonical profiling assets (663,390 bytes) are build-time SHA-256 verified and embedded as immutable snapshots. Other Lib data is not blanket-embedded; there is no filesystem fallback or spoofed resource path. The existing crate-local license notice is retained unchanged. Code and assets are registered together. New readers require matching live package/anchor registration tokens. Code-only replacement invalidates bindings even for identical bytecode; existing readers may retain their immutable snapshots. Exact names are required, so unbound aliases cannot borrow another registration's assets. Replacing a nested package invalidates its own/descendant anchors but does not revoke the unchanged parent owner's independently registered asset paths. Logical resource paths reject absolute paths, parent traversal, drive/UNC/backslash and NUL forms. Only r/rb opens are allowed, with read-only BufferedReader/TextIOWrapper streams. Standard importlib.resources.as_file handles explicitly requested owned temporary extraction. Public Rust API impact: PyGlobalState.frozen changes from HashMap to FrozenModuleRegistry. Controlled get/keys/len/is_empty/contains_key/insert/extend/remove/clear/clone remain available; raw mutable-map access, direct HashMap assignment and other map-specific APIs require migration. add_frozen_modules remains valid code-only registration; embedders wanting assets use add_frozen_modules_with_resources with FROZEN_STDLIB_RESOURCES. Main, wasm and shipped embedding examples use the paired API. No FrozenModule/FrozenLib serialization fields or markers change. Final source-bound local gates passed: All 11 native structural registry/index tests, with zero ignored; 163 unrelated tests filtered out, not claimed passed. Non-threading VM check and strict all-target Clippy for VM, pylib and the main crate (including its default threading feature). Fresh frozen CLI build, with matching pre/post-build and post-commit manifests for all 3,682 recorded source files; binary separately sealed. With sys.path empty, a bounded probe verified frozen module origins, all sixteen asset hashes, finite tree/path/mode operations, module-adjacent lookup, read-only stream errors and standard as_file cleanup of one 24,651-byte template. Normal hooks/commit hooks, exact one-edge Cargo.lock update to the existing sha2 0.11.1 dependency, and package listing including the notice plus all sixteen assets. The resource correction is 14 source/build files plus Cargo.lock, with no Lib changes. These are scoped Linux local gates, not a full workspace/CAPI suite, wasm runtime, HTML exporter/browser execution, malformed-input/stress coverage or performance measurement. Public CI for this exact new head is tracked separately; skipped benchmark execution is not a performance pass.
Open Graph Description: Summary Split the tracing profiler and canonical profiling tools from #8954 after prerequisite #9215 landed. Target remains CPython 3.15.0, pinned to d220409a13b77de7a6c30abe16ea90db01528edc. Add ...
X Description: Summary Split the tracing profiler and canonical profiling tools from #8954 after prerequisite #9215 landed. Target remains CPython 3.15.0, pinned to d220409a13b77de7a6c30abe16ea90db01528edc. Add ...
Opengraph URL: https://github.com/RustPython/RustPython/pull/9271
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:14128e25-97c9-9087-ba16-c7cb1b099b8b |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | B912:692CD:11902:163F2:6B01E690 |
| html-safe-nonce | def1e18b0e759e7cd718fbcecd9bd50853aa0abe22b8d2d3a5cd6ea9e117e177 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJCOTEyOjY5MkNEOjExOTAyOjE2M0YyOjZCMDFFNjkwIiwidmlzaXRvcl9pZCI6IjYxODk0NzY5ODEyMTg4NjI3MzYiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | 505d37a6f06a42a81aea94aa046bff48cb0b09b8a3ed65c65e66e64b604608c1 |
| hovercard-subject-tag | pull_request:4950986112 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/RustPython/RustPython/pull/9271/files |
| twitter:image | https://avatars.githubusercontent.com/u/69878?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/69878?s=400&v=4 |
| og:image:alt | Summary Split the tracing profiler and canonical profiling tools from #8954 after prerequisite #9215 landed. Target remains CPython 3.15.0, pinned to d220409a13b77de7a6c30abe16ea90db01528edc. Add ... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | cc5dfdd32f62ebd3223cab3af3aa2b268d3605218237326a48a99d1efd89495c |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/RustPython/RustPython git https://github.com/RustPython/RustPython.git |
| octolytics-dimension-user_id | 39710557 |
| octolytics-dimension-user_login | RustPython |
| octolytics-dimension-repository_id | 135201145 |
| octolytics-dimension-repository_nwo | RustPython/RustPython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 135201145 |
| octolytics-dimension-repository_network_root_nwo | RustPython/RustPython |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 8c77c9e8439031ad54c75a250694842892f693e1 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width