Title: gh-151424: Fix impossible stack traces in `RemoteUnwinder(..., cache_frames=True)` by copying chunks on cache miss by maurycy · Pull Request #151426 · python/cpython · GitHub
Open Graph Title: gh-151424: Fix impossible stack traces in `RemoteUnwinder(..., cache_frames=True)` by copying chunks on cache miss by maurycy · Pull Request #151426 · python/cpython
X Title: gh-151424: Fix impossible stack traces in `RemoteUnwinder(..., cache_frames=True)` by copying chunks on cache miss by maurycy · Pull Request #151426 · python/cpython
Description: The issue is caused by memory walks with cache_frames=True when there's a cache miss.
If I understand the whole machniery correctly.
Basically, we don't copy chunks in cache mode:
cpython/Modules/_remote_debugging/threads.c
Lines 506 to 512
in
f4f1020
// In cache mode, copying stack chunks is more expensive than direct memory reads
if (!unwinder->cache_frames) {
if (copy_stack_chunks(unwinder, *current_tstate, &chunks) < 0) {
set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to copy stack chunks");
goto error;
}
}
So there are never any chunks in the cache mode, not a problem with full cache hits:
cpython/Modules/_remote_debugging/frames.c
Lines 328 to 351
in
f4f1020
if (ctx->chunks && ctx->chunks->count > 0) {
if (parse_frame_from_chunks(unwinder, &frame, frame_addr, &next_frame_addr, &stackpointer, ctx->chunks) == 0) {
goto parsed_frame;
}
PyErr_Clear();
}
{
uintptr_t address_of_code_object = 0;
int parse_result;
if (ctx->prefetch.frame && ctx->prefetch.frame_addr == frame_addr) {
parse_result = parse_frame_buffer(
unwinder, &frame, ctx->prefetch.frame,
&address_of_code_object, &next_frame_addr);
}
else {
parse_result = parse_frame_object(
unwinder, &frame, frame_addr,
&address_of_code_object, &next_frame_addr);
}
if (parse_result < 0) {
set_exception_cause(unwinder, PyExc_RuntimeError, "Failed to parse frame object in chain");
return -1;
}
}
But when there's no cache hit:
cpython/Modules/_remote_debugging/frames.c
Lines 578 to 585
in
f4f1020
int full_hit = try_full_cache_hit(unwinder, ctx, thread_id);
if (full_hit != 0) {
return full_hit < 0 ? -1 : 0;
}
Py_ssize_t frames_before = PyList_GET_SIZE(ctx->frame_info);
if (process_frame_chain(unwinder, ctx) < 0) {
it falls back to many memory reads:
cpython/Modules/_remote_debugging/frames.c
Line 315
in
f4f1020
while ((void*)frame_addr != NULL) {
cpython/Modules/_remote_debugging/frames.c
Lines 236 to 242
in
f4f1020
char frame[SIZEOF_INTERP_FRAME];
Py_ssize_t bytes_read = _Py_RemoteDebug_ReadRemoteMemory(
&unwinder->handle,
address,
SIZEOF_INTERP_FRAME,
frame
);
which isn't very atomic... the target mutates between reads and we're lost.
That's definitely the cause. Is there a cleaner fix ?
One side-note - let's not tighten the validation yet?
Generally, it's a statistical profiler so we should have some statistical tests for correctness, too.
Proof
Reusing
def leaf_a(): return sum(range(50))
def leaf_b(): return sum(range(50))
def hot_a(): return leaf_a()
def hot_b(): return leaf_b()
while True:
hot_a(); hot_b();
from #151424 we get
2026-06-13T03:14:37.706845000+0200 maurycy@gimel /Users/maurycy/src/github.com/maurycy/cpython (gh-151424 141f623*?) % sudo ./python.exe -m profiling.sampling run --collapsed -o /tmp/stacks.txt -d 10 repro.py
Captured 10,000 samples in 10.00 seconds
Sample rate: 999.99 samples/sec
Error rate: 14.91
Collapsed stack output written to /tmp/stacks.txt
2026-06-13T03:14:51.568490000+0200 maurycy@gimel /Users/maurycy/src/github.com/maurycy/cpython (gh-151424 141f623*?) % cat /tmp/stacks.txt
tid:13199240;
Open Graph Description: The issue is caused by memory walks with cache_frames=True when there's a cache miss. If I understand the whole machniery correctly. Basically, we don't copy chunks in cache mode: ...
X Description: The issue is caused by memory walks with cache_frames=True when there's a cache miss. If I understand the whole machniery correctly. Basically, we don't copy chunks in cache mode: ...
Opengraph URL: https://github.com/python/cpython/pull/151426
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:c0de6545-620e-e696-1738-516004b4a6c7 |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | D1C4:AFBD9:22F887E:2DDD8AC:6B024BAD |
| html-safe-nonce | e2ba4c2f70be8f9080c4a47337b1d58ed1dd22f7434726e3c09e755c1c03c465 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJEMUM0OkFGQkQ5OjIyRjg4N0U6MkRERDhBQzo2QjAyNEJBRCIsInZpc2l0b3JfaWQiOiI2MDg1Mjg0MDMzODUwMzk1NTY1IiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | 84b60d8b93684672445637714301ec01d6a7a0a0b3a3435e6a91830ce64c4cc9 |
| hovercard-subject-tag | pull_request:3858518128 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/python/cpython/pull/151426/files |
| twitter:image | https://avatars.githubusercontent.com/u/5383?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/5383?s=400&v=4 |
| og:image:alt | The issue is caused by memory walks with cache_frames=True when there's a cache miss. If I understand the whole machniery correctly. Basically, we don't copy chunks in cache mode: ... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | 87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/python/cpython git https://github.com/python/cpython.git |
| octolytics-dimension-user_id | 1525981 |
| octolytics-dimension-user_login | python |
| octolytics-dimension-repository_id | 81598961 |
| octolytics-dimension-repository_nwo | python/cpython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 81598961 |
| octolytics-dimension-repository_network_root_nwo | python/cpython |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 423dbb54c5e7bfecc838eb387c1435eb3d0e760d |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width