Title: profiling.sampling: exception mode misses except blocks running in generators and coroutines · Issue #158539 · python/cpython · GitHub
Open Graph Title: profiling.sampling: exception mode misses except blocks running in generators and coroutines · Issue #158539 · python/cpython
X Title: profiling.sampling: exception mode misses except blocks running in generators and coroutines · Issue #158539 · python/cpython
Description: Bug report Background I have been writing a set of guided workshops on Tachyon as a way of learning it (https://github.com/GrahamDumpleton/tachyon-workshops), using an AI agent to generate the workshop content. While working on a worksho...
Open Graph Description: Bug report Background I have been writing a set of guided workshops on Tachyon as a way of learning it (https://github.com/GrahamDumpleton/tachyon-workshops), using an AI agent to generate the work...
X Description: Bug report Background I have been writing a set of guided workshops on Tachyon as a way of learning it (https://github.com/GrahamDumpleton/tachyon-workshops), using an AI agent to generate the work...
Opengraph URL: https://github.com/python/cpython/issues/158539
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"profiling.sampling: exception mode misses except blocks running in generators and coroutines","articleBody":"### Bug report\n\n## Background\n\nI have been writing a set of guided workshops on Tachyon as a way of learning it (https://github.com/GrahamDumpleton/tachyon-workshops), using an AI agent to generate the workshop content. While working on a workshop about exception mode, the agent found that a function which catches exceptions had far fewer samples in `--mode exception` when called from a generator expression than when called from a plain loop, with nothing else changed. It reduced that to the script below and believes there is a disparity between what the documentation says exception mode records and what it actually records.\n\nThat seems reasonable to me, but I do not have the knowledge of the internals to say whether this is a bug, a known shortcoming, or by design, so I am reporting what was found and how to reproduce it. The analysis of the cause at the end is the agent's, and I cannot evaluate it myself.\n\n## What the documentation says\n\nThe exception mode section of the `profiling.sampling` docs says samples are recorded when an exception is being propagated, \"or when code is executing inside an `except` block where exception information is still present in the thread state\". There is no mention of generators or coroutines being treated differently.\n\n## What happens\n\nAn `except` block that runs inside a generator or a coroutine, or in an ordinary function called from one, is never sampled in exception mode. The same `except` block in a plain function is sampled as expected.\n\nThe script below has five cases. Each raises `ValueError`, catches it, and spends one second in a busy loop inside the `except` block. They differ only in what kind of frame the handler, or its caller, is.\n\n```python\nimport asyncio\nimport time\n\n\ndef busy(seconds):\n end = time.perf_counter() + seconds\n while time.perf_counter() \u003c end:\n pass\n\n\ndef handler_in_function():\n try:\n raise ValueError(\"handled in a function\")\n except ValueError:\n busy(1.0)\n\n\ndef handler_in_generator():\n try:\n raise ValueError(\"handled in a generator\")\n except ValueError:\n busy(1.0)\n yield\n\n\ndef handler_in_genexpr_callee():\n try:\n raise ValueError(\"handled in a function called from a generator expression\")\n except ValueError:\n busy(1.0)\n\n\nasync def handler_in_coroutine():\n try:\n raise ValueError(\"handled in a coroutine\")\n except ValueError:\n busy(1.0)\n\n\nasync def calls_handler_from_coroutine():\n handler_in_coroutine_callee()\n\n\ndef handler_in_coroutine_callee():\n try:\n raise ValueError(\"handled in a function called from a coroutine\")\n except ValueError:\n busy(1.0)\n\n\nhandler_in_function()\nfor _ in handler_in_generator():\n pass\nlist(handler_in_genexpr_callee() for _ in range(1))\nasyncio.run(handler_in_coroutine())\nasyncio.run(calls_handler_from_coroutine())\n```\n\nRun it in wall-clock mode and then in exception mode:\n\n```\npython -m profiling.sampling run --mode wall --sort cumtime -l 40 --no-summary repro.py\npython -m profiling.sampling run --mode exception --sort cumtime -l 40 --no-summary repro.py\n```\n\nWall-clock mode sees all five handlers for about a second each (rows for `repro.py` only):\n\n```\nCaptured 5080 samples in 5.08 seconds\n nsamples sample% tottime (s) cumul% cumtime (s) filename:lineno(function)\n 4969/4969 98.0 4.969 98.0 4.969 repro.py:7(busy)\n 0/1000 0.0 0.000 19.7 1.000 repro.py:15(handler_in_function)\n 0/1000 0.0 0.000 19.7 1.000 repro.py:51(\u003cmodule>)\n 0/1000 0.0 0.000 19.7 1.000 repro.py:22(handler_in_generator)\n 0/1000 0.0 0.000 19.7 1.000 repro.py:52(\u003cmodule>)\n 0/1000 0.0 0.000 19.7 1.000 repro.py:30(handler_in_genexpr_callee)\n 0/1000 0.0 0.000 19.7 1.000 repro.py:54(\u003cgenexpr>)\n 0/1000 0.0 0.000 19.7 1.000 repro.py:54(\u003cmodule>)\n 0/1000 0.0 0.000 19.7 1.000 repro.py:37(handler_in_coroutine)\n 0/1000 0.0 0.000 19.7 1.000 repro.py:55(\u003cmodule>)\n 0/1000 0.0 0.000 19.7 1.000 repro.py:48(handler_in_coroutine_callee)\n 0/1000 0.0 0.000 19.7 1.000 repro.py:41(calls_handler_from_coroutine)\n 0/1000 0.0 0.000 19.7 1.000 repro.py:56(\u003cmodule>)\n 0/66 0.0 0.000 1.3 0.066 repro.py:1(\u003cmodule>)\n 31/31 0.6 0.031 0.6 0.031 repro.py:8(busy)\n```\n\nException mode sees only the first:\n\n```\nCaptured 5075 samples in 5.08 seconds\n nsamples sample% tottime (ms) cumul% cumtime (s) filename:lineno(function)\n 0/1000 0.0 0.000 100.0 1.000 repro.py:15(handler_in_function)\n 0/1000 0.0 0.000 100.0 1.000 repro.py:51(\u003cmodule>)\n 993/993 99.3 993.000 99.3 0.993 repro.py:7(busy)\n 7/7 0.7 7.000 0.7 0.007 repro.py:8(busy)\n```\n\n| Case | Wall-clock samples | Exception mode samples |\n| --- | --- | --- |\n| `handler_in_function` | 1000 | 1000 |\n| `handler_in_generator` | 1000 | 0 |\n| `handler_in_genexpr_callee` | 1000 | 0 |\n| `handler_in_coroutine` | 1000 | 0 |\n| `handler_in_coroutine_callee` | 1000 | 0 |\n\nThe exception mode run reports an error rate of 0.00 and captured over 5000 samples, so the samples were taken; they were then discarded by the mode's filter. From inside the program, `sys.exception()` returns the `ValueError` in every one of the five handlers.\n\nThis was reproduced on two builds of 3.15.0rc2, the `python:3.15-rc-slim` Docker image (GCC) and the python-build-standalone build installed by uv (Clang), both on Linux aarch64 in a container, as a non-root user, using `run` so no ptrace permission is involved.\n\nIf the behaviour is intended, the practical effect seems worth noting in the docs: in an asyncio program every `except` block runs in a coroutine or in something called from one, so exception mode reports almost nothing for the whole program, which reads as \"exceptions are not your problem\" rather than as a limitation.\n\n## AI analysis of the cause (not verified by me)\n\nWhat follows is the agent's reading of the source. I am including it in case it saves someone time, but I cannot judge whether it is right.\n\nIn `Modules/_remote_debugging/threads.c`, the check that sets `THREAD_STATUS_HAS_EXCEPTION` reads `current_exception` and then `exc_state.exc_value`, the exception stack entry embedded in `PyThreadState`, with a comment that \"nested exception handlers where exc_info points elsewhere are rare\":\n\n```c\n // Check exc_state.exc_value (exception being handled in except block)\n // exc_state is embedded in PyThreadState, so we read it directly from\n // the thread state buffer. This catches most cases; nested exception\n // handlers where exc_info points elsewhere are rare.\n if (!has_exception) {\n uintptr_t exc_value = GET_MEMBER(uintptr_t, ts,\n unwinder->debug_offsets.thread_state.exc_state +\n unwinder->debug_offsets.err_stackitem.exc_value);\n if (exc_value != 0) {\n has_exception = 1;\n }\n }\n```\n\nThe agent's explanation is that the interpreter records a handled exception in `tstate->exc_info->exc_value`, and that while a generator or coroutine is running, `exc_info` points at the generator's own `_PyErr_StackItem` rather than the thread's `exc_state`. A handler running in the generator, or in anything it calls, writes to the generator's entry, so `exc_state` stays empty and the profiler sees no exception. That would make the \"rare\" case every `except` block in every `async def`. It also matches the case of a plain function called from a generator expression being missed even though the handler is not itself in a generator.\n\nIt also noted that PR #142561, which added the mode, describes the check as using `exc_info`, and that a review comment on the docs (https://github.com/python/cpython/pull/142561#discussion_r2611748072) pointed out the code reads `exc_state`; the field name was then removed from the docs. The tests in `test_external_inspection.py` (`TestExceptionDetectionScenarios`) describe the condition as \"`exc_info->exc_value` is set\" but only put the handler in a plain function, where the two coincide.\n\nThe suggested fix was to add `exc_info` to the debug offsets, use the embedded `exc_state` value when `exc_info` points at it (as now, at no extra cost), and otherwise read the `_PyErr_StackItem` it points at and walk `previous_item`, as `_PyErr_GetTopmostException` does. Whether that is the right approach, or whether changing the debug offsets is acceptable at this stage, is beyond what I can assess.\n\n\n### CPython versions tested on:\n\n3.15\n\n### Operating systems tested on:\n\nLinux\n\n\u003c!-- gh-linked-prs -->\n### Linked PRs\n* gh-158581\n* gh-158852\n* gh-158906\n\u003c!-- /gh-linked-prs -->\n","author":{"url":"https://github.com/GrahamDumpleton","@type":"Person","name":"GrahamDumpleton"},"datePublished":"2026-09-30T23:54:40Z","url":"https://github.com/python/cpython/issues/158539"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:3b8ad652-9e08-35b4-6c43-e4945e88f221 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | CFE6:269AD9:38FB787:4C0A12E:6B021B79 |
| html-safe-nonce | 1ab13686e6bc6fca63bacacc8518f68fb5883d22719ee90fda02cef61cf06fb1 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJDRkU2OjI2OUFEOTozOEZCNzg3OjRDMEExMkU6NkIwMjFCNzkiLCJ2aXNpdG9yX2lkIjoiODA3MDU2MzA3OTM2NjI1MzQzMyIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | eb3bb0a64ae0e68a46ce7e18dbc3b8b2f65f60f4e88289070fe4781b6a85175c |
| hovercard-subject-tag | issue:5654595395 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/python/cpython/158539/issue_layout |
| twitter:image | https://opengraph.githubassets.com/3e0020471975232005ae45bf66a113199b48c8149d8b97634c6e970ce50d546b/python/cpython/issues/158539 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/3e0020471975232005ae45bf66a113199b48c8149d8b97634c6e970ce50d546b/python/cpython/issues/158539 |
| og:image:alt | Bug report Background I have been writing a set of guided workshops on Tachyon as a way of learning it (https://github.com/GrahamDumpleton/tachyon-workshops), using an AI agent to generate the work... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | GrahamDumpleton |
| hostname | github.com |
| expected-hostname | github.com |
| None | 87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456 |
| turbo-cache-control | no-preview |
| go-import | github.com/python/cpython git https://github.com/python/cpython.git |
| octolytics-dimension-user_id | 1525981 |
| octolytics-dimension-user_login | python |
| octolytics-dimension-repository_id | 81598961 |
| octolytics-dimension-repository_nwo | python/cpython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 81598961 |
| octolytics-dimension-repository_network_root_nwo | python/cpython |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 423dbb54c5e7bfecc838eb387c1435eb3d0e760d |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width