Title: Race condition in tracemalloc causes segfaults · Issue #128679 · python/cpython · GitHub
Open Graph Title: Race condition in tracemalloc causes segfaults · Issue #128679 · python/cpython
X Title: Race condition in tracemalloc causes segfaults · Issue #128679 · python/cpython
Description: Crash report What happened? This is a bit of a tricky situation, but it is real and impacting my ability to use tracemalloc. As background, I've added code to Polars to make it record all of its allocations in tracemalloc, and this is en...
Open Graph Description: Crash report What happened? This is a bit of a tricky situation, but it is real and impacting my ability to use tracemalloc. As background, I've added code to Polars to make it record all of its al...
X Description: Crash report What happened? This is a bit of a tricky situation, but it is real and impacting my ability to use tracemalloc. As background, I've added code to Polars to make it record all of it...
Opengraph URL: https://github.com/python/cpython/issues/128679
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Race condition in tracemalloc causes segfaults","articleBody":"# Crash report\r\n\r\n### What happened?\r\n\r\nThis is a bit of a tricky situation, but it is real and impacting my ability to use tracemalloc. As background, I've added code to Polars to make it record all of its allocations in tracemalloc, and this is enabled in debug builds. This then allows writing unit tests that check memory usage, which is very useful in ensuring high memory usage is fixed, and making sure it doesn't get high again.\r\n\r\nUnfortunately, I'm hitting a situation where tracemalloc causes segfaults in multi-threaded situations. I believe that this is a race condition between `PyTraceMalloc_Track()` in a new non-Python thread that does not hold the GIL, and `tracemalloc.stop()` being called in another thread. My hypothesis in detail:\r\n\r\n* Thread 1 does `tracemalloc.start()`.\r\n* Thread 2 does not hold the GIL.\r\n* Thread 2 calls PyTraceMalloc_Track().\r\n* Thread 2 checks if tracemalloc is enabled, the answer is yes.\r\n* Thread 2 starts acquiring the GIL (see PyTraceMalloc_Track implementation). As part of acquiring the GIL memory needs to be allocated.\r\n* Thread 2 therefore once again checks if tracemalloc is enabled (I haven't actually read source code enough to verify if this part is true, but the coredump suggests it probably is). The answer is yes.\r\n* At this point, Thread 1 calls `tracemalloc.stop()`.\r\n* Thread 2 continues as if tracemalloc enabled, but it is not!\r\n* BOOM\r\n\r\nIf this hypothesis is correct, the solution would for GIL acquisition to bypass tracemalloc altogether if it allocates; it's not like it allocates a lot of memory, so not tracking it is fine. This may be difficult in practice, so another approach would involve having an additional lock so there's no race condition around checking if tracemalloc is enabled.\r\n\r\nHere is a stack trace from a coredump from the reproducer (see below) that led me to the above hypothesis:\r\n\r\n```\r\n#0 0x00000000005f0482 in traceback_new () at ../Python/tracemalloc.c:371\r\n#1 0x00000000005f0083 in tracemalloc_add_trace (domain=0, ptr=132539872185888, size=336) at ../Python/tracemalloc.c:471\r\n#2 0x00000000006930db in tracemalloc_alloc (elsize=336, nelem=\u003coptimized out>, ctx=0x9ec4f8 \u003c_PyRuntime+10136>, use_calloc=\u003coptimized out>) at ../Python/tracemalloc.c:541\r\n#3 tracemalloc_raw_alloc (use_calloc=\u003coptimized out>, ctx=0x9ec4f8 \u003c_PyRuntime+10136>, nelem=\u003coptimized out>, elsize=336) at ../Python/tracemalloc.c:715\r\n#4 0x0000000000639a52 in alloc_threadstate (interp=0x9ff6b0 \u003c_PyRuntime+88400>) at ../Python/pystate.c:1452\r\n#5 new_threadstate (interp=0x9ff6b0 \u003c_PyRuntime+88400>, whence=4) at ../Python/pystate.c:1563\r\n#6 0x0000000000480ef0 in PyGILState_Ensure () at ../Python/pystate.c:2766\r\n#7 0x00000000005203fc in PyTraceMalloc_Track (domain=123, ptr=10, size=1) at ../Python/tracemalloc.c:1318\r\n#8 0x0000788b7088774c in tracemalloc_repro::in_thread () at src/lib.rs:23\r\n```\r\n\r\nTo run the reproducer you will need to `pip install rustimport` and have Rust installed. (I tried with Cython, had a hard time, gave up.)\r\n\r\nHere's the Python file:\r\n\r\n```python\r\nimport gc\r\nimport time\r\nimport tracemalloc\r\n\r\nimport rustimport.import_hook\r\n\r\n# rustimport will automatically compile tracemalloc_repro.rs into an extension:\r\nimport tracemalloc_repro\r\n\r\ntracemalloc.start()\r\n# This launches ~50 threads that run in parallel to this one:\r\ntracemalloc_repro.run()\r\n# This happens in parallel to the new threads running:\r\ntracemalloc.stop()\r\ngc.collect()\r\n\r\ntime.sleep(10)\r\n```\r\n\r\nAnd here is the Rust file, you should call it `tracemalloc_repro.rs`:\r\n\r\n```rust\r\n// rustimport\r\n//: [package]\r\n//: name = \"tracemalloc_repro\"\r\n//: version = \"0.1.0\"\r\n//: edition = \"2021\"\r\n//:\r\n//: [lib]\r\n//: name = \"tracemalloc_repro\"\r\n//: crate-type = [\"cdylib\"]\r\n//:\r\n//: [dependencies]\r\n//: pyo3 = {version = \"0.23\", features = [\"extension-module\"]}\r\n//: libc = \"0.2\"\r\n\r\nuse pyo3::prelude::*;\r\nuse libc::{c_int, c_uint, size_t, uintptr_t};\r\n\r\nextern \"C\" {\r\n fn PyTraceMalloc_Track(domain: c_uint, ptr: uintptr_t, size: size_t) -> c_int;\r\n}\r\n\r\nfn in_thread() {\r\n let result = unsafe { PyTraceMalloc_Track(123, 10, 1) };\r\n println!(\"Result of tracking: {result}\");\r\n}\r\n\r\n#[pyfunction]\r\nfn run(py: Python) {\r\n py.allow_threads(|| {\r\n // With GIL released, run function in a thread.\r\n for _ in 0..50 {\r\n let _ = std::thread::spawn(in_thread);\r\n }\r\n });\r\n}\r\n\r\n#[pymodule]\r\nfn tracemalloc_repro(_py: Python, m: &Bound\u003c'_, PyModule>) -> PyResult\u003c()> {\r\n m.add_function(wrap_pyfunction!(run, m)?)?;\r\n Ok(())\r\n}\r\n```\r\n\r\nYou can reproduce by calling `repro.py`. Because this is a race condition, you may need to run it a few times; I had more consistent crashes with Python 3.12, but it does crash on Python 3.13. You may need to tweak the number 50 above to make it happen.\r\n\r\n### CPython versions tested on:\r\n\r\n3.12, 3.13\r\n\r\n### Operating systems tested on:\r\n\r\nLinux\r\n\r\n### Output from running 'python -VV' on the command line:\r\n\r\nPython 3.13.1 (main, Dec 4 2024, 08:54:15) [GCC 13.2.0]\n\n\u003c!-- gh-linked-prs -->\n### Linked PRs\n* gh-128695\n* gh-128710\n* gh-128888\n* gh-128893\n* gh-128897\n* gh-128988\n* gh-129022\n* gh-129126\n* gh-129258\n\u003c!-- /gh-linked-prs -->\n","author":{"url":"https://github.com/itamarst","@type":"Person","name":"itamarst"},"datePublished":"2025-01-09T15:20:30Z","url":"https://github.com/python/cpython/issues/128679"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:1fd1059e-dd43-c3a2-3651-735746d89496 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | CB90:19F84C:3C94DA9:4CF68CA:6B01F3E1 |
| html-safe-nonce | 5ac397a4afa300ce1fa5ce450a434476e7a0286501a4551ab7f623db3e82df3b |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJDQjkwOjE5Rjg0QzozQzk0REE5OjRDRjY4Q0E6NkIwMUYzRTEiLCJ2aXNpdG9yX2lkIjoiMzQ1NDk4MjU3MzU3MzQwMzYxNyIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | d0fe50481efd00a16a223ba7d94ca7dd1515bb43313bf1ff1a12d9e997f126dc |
| hovercard-subject-tag | issue:2778112954 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/python/cpython/128679/issue_layout |
| twitter:image | https://opengraph.githubassets.com/f2969e2d3f3951304cecb0cf184399271f754088e2ccc186ceb947244269015a/python/cpython/issues/128679 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/f2969e2d3f3951304cecb0cf184399271f754088e2ccc186ceb947244269015a/python/cpython/issues/128679 |
| og:image:alt | Crash report What happened? This is a bit of a tricky situation, but it is real and impacting my ability to use tracemalloc. As background, I've added code to Polars to make it record all of its al... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | itamarst |
| hostname | github.com |
| expected-hostname | github.com |
| None | 87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456 |
| turbo-cache-control | no-preview |
| go-import | github.com/python/cpython git https://github.com/python/cpython.git |
| octolytics-dimension-user_id | 1525981 |
| octolytics-dimension-user_login | python |
| octolytics-dimension-repository_id | 81598961 |
| octolytics-dimension-repository_nwo | python/cpython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 81598961 |
| octolytics-dimension-repository_network_root_nwo | python/cpython |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 423dbb54c5e7bfecc838eb387c1435eb3d0e760d |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width