Title: Segfault in faulthandler signal handler with threads · Issue #116008 · python/cpython · GitHub
Open Graph Title: Segfault in faulthandler signal handler with threads · Issue #116008 · python/cpython
X Title: Segfault in faulthandler signal handler with threads · Issue #116008 · python/cpython
Description: Crash report What happened? We use faulthandler.register to dump stacks on SIGUSR1. faulthandler.register(signal.SIGUSR1, chain=True) Sending the signal to a running multi-threaded process, it segfaulted. It successfully dumped several t...
Open Graph Description: Crash report What happened? We use faulthandler.register to dump stacks on SIGUSR1. faulthandler.register(signal.SIGUSR1, chain=True) Sending the signal to a running multi-threaded process, it segf...
X Description: Crash report What happened? We use faulthandler.register to dump stacks on SIGUSR1. faulthandler.register(signal.SIGUSR1, chain=True) Sending the signal to a running multi-threaded process, it segf...
Opengraph URL: https://github.com/python/cpython/issues/116008
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Segfault in faulthandler signal handler with threads","articleBody":"# Crash report\r\n\r\n### What happened?\r\n\r\nWe use faulthandler.register to dump stacks on SIGUSR1.\r\n```python\r\nfaulthandler.register(signal.SIGUSR1, chain=True)\r\n```\r\nSending the signal to a running multi-threaded process, it segfaulted. It successfully dumped several thread stacks, but segfaulted part way through:\r\n\r\n```\r\nThread 0x00007f4fbd4a3640 (most recent call first):\r\n File \"/usr/tideway/lib/python3.11/threading.py\", line 320 in wait\r\n File \"/usr/tideway/lib/python3.11/threading.py\", line 622 in wait\r\n File \"/usr/tideway/python/reasoning/consolidation/sender.py\", line 1151 in _threadedSendingOfData\r\n File \"/usr/tideway/lib/python3.11/threading.py\", line 975 in run\r\n File \"/usr/tideway/lib/python3.11/threading.py\", line 1038 in _bootstrap_inner\r\n File \"/usr/tideway/lib/python3.11/threading.py\", line 995 in _bootstrap\r\n\r\nThread 0x00007f4fafca0640 (most recent call first):\r\n File \"/usr/tideway/python/stubs/search_idl.py\", line 720 in search\r\n File \"/usr/tideway/python/reasoning/ecaengine/tpl/tpl_search.py\", line 87 in _doSearch\r\n File \"/usr/tideway/python/reasoning/ecaengine/tpl/tpl_search.py\", line 103 in search\r\n File \"/usr/tideway/var/code/tpl_py/SupportDetail/SoftwareInstance/__init__.py\", line 233 in execute\r\n File \"/usr/tideway/python/reasoning/ecaengine/engine/rule.py\", line 141 in doExecute\r\n File \"/usr/tideway/python/reasoning/ecaengine/engine/__init__.py\", line 715 in processEvent\r\n File \"/usr/tideway/python/reasoning/ecaengine/transactional_server.py\", line 258 in processEvent\r\n File \"/usr/tideway/pylib/00_hack/common/threadpool.py\", line 93 in run\r\n File \"/usr/tideway/lib/python3.11/threading.py\", line 1038 in _bootstrap_inner\r\n File \"/usr/tideway/lib/python3.11/threading.py\", line 995 in _bootstrap\r\n\r\nThread 0x00007f4fb04a1640 (most recent call first):\r\n File \"/usr/tideway/lib/python3.11/threading.py\", line 338 in notify_all\r\n File \"/usr/tideway/lib/python3.11/concurrent/futures/_base.py\", line 550 in set_result\r\n File Tue Feb 27 14:17:40 2024 : Watchdog : Service failed after 1674.6 seconds (signal 11; exit 0). Restarting (1 restart so far).\r\n```\r\nThe last line shows that it wrote \"File\", then segfaulted. (That was caught by a parent watchdog process that restarted the process. The timestamp and rest of the message is from the watchdog process.) \r\n\r\nLooking at the core file:\r\n\r\n```\r\nCore was generated by `python3 -m reasoning.main --daemon start'.\r\nProgram terminated with signal SIGSEGV, Segmentation fault.\r\n#0 0x00007f4fcd393e70 in dump_frame (frame=0x7f4f9533bac0, fd=2) at Python/traceback.c:1190\r\n1190\t && PyUnicode_Check(code->co_filename))\r\n\r\n(gdb) bt\r\n#0 0x00007f4fcd393e70 in dump_frame (frame=0x7f4f9533bac0, fd=2) at Python/traceback.c:1190\r\n#1 dump_traceback (fd=fd@entry=2, tstate=tstate@entry=0xbd23860, write_header=write_header@entry=0) at Python/traceback.c:1242\r\n#2 0x00007f4fcd3940b7 in _Py_DumpTracebackThreads (fd=fd@entry=2, interp=\u003coptimized out>, interp@entry=0x0, current_tstate=0x7f4fcd75d2d8 \u003c_PyRuntime+166328>) at Python/traceback.c:1344\r\n#3 0x00007f4fcd39a380 in faulthandler_dump_traceback (fd=2, all_threads=1, interp=\u003coptimized out>) at ./Modules/faulthandler.c:252\r\n#4 0x00007f4fcd39a539 in faulthandler_user (signum=10) at ./Modules/faulthandler.c:862\r\n#5 \u003csignal handler called>\r\n#6 0x00007f4fcce9c3aa in __futex_abstimed_wait_common () from /lib64/libc.so.6\r\n#7 0x00007f4fccea6bae in __new_sem_wait_slow64 () from /lib64/libc.so.6\r\n#8 0x00007f4fcd466270 in PyThread_acquire_lock_timed (lock=lock@entry=0x50d6510, microseconds=microseconds@entry=5000000, intr_flag=intr_flag@entry=1) at Python/thread_pthread.h:483\r\n#9 0x00007f4fcd4f5c72 in acquire_timed (lock=0x50d6510, timeout=5000000000) at ./Modules/_threadmodule.c:98\r\n#10 0x00007f4fcd4f5abe in lock_PyThread_acquire_lock (self=0x7f4fb1dabdc0, args=args@entry=0x7f4fa0b4fdc0, kwds=kwds@entry=0x0) at ./Modules/_threadmodule.c:179\r\n#11 0x00007f4fcd3ea0dc in method_vectorcall_VARARGS_KEYWORDS (func=\u003coptimized out>, args=0x7f4fcd9394b0, nargsf=\u003coptimized out>, kwnames=\u003coptimized out>) at Objects/descrobject.c:364\r\n#12 0x00007f4fcd3e59a8 in _PyObject_VectorcallTstate (kwnames=\u003coptimized out>, nargsf=\u003coptimized out>, args=\u003coptimized out>, callable=0x7f4fcd18af20, tstate=0x7f4fcd75d2d8 \u003c_PyRuntime+166328>)\r\n at ./Include/internal/pycore_call.h:92\r\n#13 PyObject_Vectorcall (callable=0x7f4fcd18af20, args=\u003coptimized out>, nargsf=\u003coptimized out>, kwnames=\u003coptimized out>) at Objects/call.c:299\r\n#14 0x00007f4fcd4438ca in _PyEval_EvalFrameDefault (tstate=\u003coptimized out>, frame=\u003coptimized out>, throwflag=\u003coptimized out>) at Python/ceval.c:4774\r\n#15 0x00007f4fcd442557 in _PyEval_EvalFrame (throwflag=0, frame=0x7f4fcd9391b8, tstate=0x7f4fcd75d2d8 \u003c_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73\r\n#16 _PyEval_Vector (tstate=0x7f4fcd75d2d8 \u003c_PyRuntime+166328>, func=\u003coptimized out>, locals=0x7f4fcd1f30c0, args=0x0, argcount=\u003coptimized out>, kwnames=\u003coptimized out>) at Python/ceval.c:6439\r\n#17 0x00007f4fcd4bf2f4 in PyEval_EvalCode (co=co@entry=0x7f4fccd230a0, globals=globals@entry=0x7f4fcd1f30c0, locals=locals@entry=0x7f4fcd1f30c0) at Python/ceval.c:1154\r\n#18 0x00007f4fcd4be3b4 in builtin_exec_impl (module=\u003coptimized out>, closure=\u003coptimized out>, locals=0x7f4fcd1f30c0, globals=0x7f4fcd1f30c0, source=0x7f4fccd230a0) at Python/bltinmodule.c:1077\r\n#19 builtin_exec (module=\u003coptimized out>, args=\u003coptimized out>, nargs=\u003coptimized out>, kwnames=\u003coptimized out>) at Python/clinic/bltinmodule.c.h:465\r\n#20 0x00007f4fcd40c4c6 in cfunction_vectorcall_FASTCALL_KEYWORDS (func=0x7f4fcd188f90, args=0x7f4fcd939180, nargsf=\u003coptimized out>, kwnames=0x0) at ./Include/cpython/methodobject.h:52\r\n#21 0x00007f4fcd3e59a8 in _PyObject_VectorcallTstate (kwnames=\u003coptimized out>, nargsf=\u003coptimized out>, args=\u003coptimized out>, callable=0x7f4fcd188f90, tstate=0x7f4fcd75d2d8 \u003c_PyRuntime+166328>)\r\n at ./Include/internal/pycore_call.h:92\r\n#22 PyObject_Vectorcall (callable=0x7f4fcd188f90, args=\u003coptimized out>, nargsf=\u003coptimized out>, kwnames=\u003coptimized out>) at Objects/call.c:299\r\n#23 0x00007f4fcd4438ca in _PyEval_EvalFrameDefault (tstate=\u003coptimized out>, frame=\u003coptimized out>, throwflag=\u003coptimized out>) at Python/ceval.c:4774\r\n#24 0x00007f4fcd442557 in _PyEval_EvalFrame (throwflag=0, frame=0x7f4fcd939020, tstate=0x7f4fcd75d2d8 \u003c_PyRuntime+166328>) at ./Include/internal/pycore_ceval.h:73\r\n#25 _PyEval_Vector (tstate=0x7f4fcd75d2d8 \u003c_PyRuntime+166328>, func=\u003coptimized out>, locals=0x0, args=0x7f4fccda6c18, argcount=\u003coptimized out>, kwnames=\u003coptimized out>) at Python/ceval.c:6439\r\n#26 0x00007f4fcd4dc089 in pymain_run_module (modname=\u003coptimized out>, set_argv0=\u003coptimized out>) at Modules/main.c:300\r\n#27 0x00007f4fcd4dbc8d in pymain_run_python (exitcode=0x7ffda6d1e874) at Modules/main.c:595\r\n#28 Py_RunMain () at Modules/main.c:680\r\n#29 0x00007f4fcd4dba39 in Py_BytesMain (argc=\u003coptimized out>, argv=\u003coptimized out>) at Modules/main.c:734\r\n#30 0x00007f4fcce3feb0 in __libc_start_call_main () from /lib64/libc.so.6\r\n#31 0x00007f4fcce3ff60 in __libc_start_main_impl () from /lib64/libc.so.6\r\n#32 0x0000000000401075 in _start ()\r\n(gdb) p code\r\n$1 = (PyCodeObject *) 0x7f4fcd63ebe0 \u003cempty_keys_struct>\r\n(gdb) p *code\r\n$2 = {ob_base = {ob_base = {ob_refcnt = 118616, ob_type = 0x100010000}, ob_size = 0}, co_consts = 0x0, co_names = 0xffffffffffffffff, co_exceptiontable = 0x0, co_flags = 0, co_warmup = 0, _co_linearray_entry_size = 0, \r\n co_argcount = 0, co_posonlyargcount = 0, co_kwonlyargcount = 11, co_stacksize = 0, co_firstlineno = -849062080, co_nlocalsplus = 32591, co_nlocals = 0, co_nplaincellvars = 0, co_ncellvars = -850270115, \r\n co_nfreevars = 32591, co_localsplusnames = 0x18, co_localspluskinds = 0x0, co_filename = 0x7f4fcd408c90 \u003cdictview_dealloc>, co_name = 0x0, co_qualname = 0x0, co_linetable = 0x0, co_weakreflist = 0x0, \r\n _co_code = 0x7f4fcd499e60 \u003cdictview_repr>, _co_linearray = 0x0, _co_firsttraceable = -849084352, co_extra = 0x0, co_code_adaptive = \" \"}\r\n(gdb) p *code->co_filename\r\n$3 = {ob_refcnt = -177530513340544939, ob_type = 0x8418b48f8578b48}\r\n(gdb) p *code->co_filename->ob_type\r\nCannot access memory at address 0x8418b48f8578b48\r\n```\r\nco_filename is clearly invalid here.\r\n\r\nMost of the time dumping the stack works fine, but doing it repeatedly I can reproduce this when the process is busy. A second time, it failed in a slightly different place, accessing co_name:\r\n\r\n```\r\n(gdb) bt\r\n#0 0x00007f4ab7993f05 in dump_frame (frame=0x7f4ab3ab43a8, fd=2) at Python/traceback.c:1210\r\n#1 dump_traceback (fd=fd@entry=2, tstate=tstate@entry=0xae9d1f0, write_header=write_header@entry=0) at Python/traceback.c:1242\r\n#2 0x00007f4ab79940b7 in _Py_DumpTracebackThreads (fd=fd@entry=2, interp=\u003coptimized out>, interp@entry=0x0, current_tstate=0x7f4ab7d5d2d8 \u003c_PyRuntime+166328>) at Python/traceback.c:1344\r\n#3 0x00007f4ab799a380 in faulthandler_dump_traceback (fd=2, all_threads=1, interp=\u003coptimized out>) at ./Modules/faulthandler.c:252\r\n#4 0x00007f4ab799a539 in faulthandler_user (signum=10) at ./Modules/faulthandler.c:862\r\n#5 \u003csignal handler called>\r\n#6 0x00007f4ab749c3aa in __futex_abstimed_wait_common () from /lib64/libc.so.6\r\n...\r\n\r\n\r\n(gdb) p *code\r\n$2 = {ob_base = {ob_base = {ob_refcnt = 336273, ob_type = 0x7f4ab7c42940 \u003c_PyNone_Type>}, ob_size = 0}, co_consts = 0x0, co_names = 0xa2, co_exceptiontable = 0x7f4ab7c45740 \u003cPyType_Type>, co_flags = 0, co_warmup = 0, \r\n _co_linearray_entry_size = 0, co_argcount = -1213067338, co_posonlyargcount = 32586, co_kwonlyargcount = 16, co_stacksize = 0, co_firstlineno = 0, co_nlocalsplus = 0, co_nlocals = -1215091588, co_nplaincellvars = 32586, \r\n co_ncellvars = 0, co_nfreevars = 0, co_localsplusnames = 0x0, co_localspluskinds = 0x0, co_filename = 0x0, co_name = 0x7f4ab7a0eb60 \u003cnone_repr>, co_qualname = 0x7f4ab7c42c40 \u003cnone_as_number>, co_linetable = 0x0, \r\n co_weakreflist = 0x0, _co_code = 0x7f4ab7a61a20 \u003c_Py_HashPointer>, _co_linearray = 0x0, _co_firsttraceable = -1214131376, co_extra = 0x7f4ab7a0dfe0 \u003cPyObject_GenericGetAttr>, co_code_adaptive = \"P\"}\r\n\r\n(gdb) p code->co_name\r\n$4 = (PyObject *) 0x7f4ab7a0eb60 \u003cnone_repr>\r\n(gdb) p *code->co_name\r\n$5 = {ob_refcnt = -1657306674253427384, ob_type = 0x401f0fffee6194}\r\n```\r\n\r\n\r\n### CPython versions tested on:\r\n\r\n3.11\r\n\r\n### Operating systems tested on:\r\n\r\nLinux\r\n\r\n### Output from running 'python -VV' on the command line:\r\n\r\nPython 3.11.4 (main, Jun 19 2023, 17:32:00) [GCC 11.3.1 20221121 (Red Hat 11.3.1-4.3.0.1)]\n\n\u003c!-- gh-linked-prs -->\n### Linked PRs\n* gh-141267\n* gh-141988\n* gh-142013\n* gh-142017\n\u003c!-- /gh-linked-prs -->\n","author":{"url":"https://github.com/dgrisby","@type":"Person","name":"dgrisby"},"datePublished":"2024-02-27T21:49:33Z","url":"https://github.com/python/cpython/issues/116008"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:234839c9-92d5-bd41-b3ab-c9899bb87faa |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | 8F04:313081:1CC2FDD:25DF617:6B022046 |
| html-safe-nonce | d23496265921d0c609dc96b010462a7168e82ef5995f8c83954f8ce4a8eb582a |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI4RjA0OjMxMzA4MToxQ0MyRkREOjI1REY2MTc6NkIwMjIwNDYiLCJ2aXNpdG9yX2lkIjoiODIwMTU3MDk1Mzk0NTQ5ODMwIiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | 5a21d371504b79550056a0b781c6c52852f11fe1dc5a3b6047d10cace6e8e686 |
| hovercard-subject-tag | issue:2157665754 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/python/cpython/116008/issue_layout |
| twitter:image | https://opengraph.githubassets.com/742325881167011d8731d66aac4529e5502be66bdcb70c7f7cc9c617ecd55dd5/python/cpython/issues/116008 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/742325881167011d8731d66aac4529e5502be66bdcb70c7f7cc9c617ecd55dd5/python/cpython/issues/116008 |
| og:image:alt | Crash report What happened? We use faulthandler.register to dump stacks on SIGUSR1. faulthandler.register(signal.SIGUSR1, chain=True) Sending the signal to a running multi-threaded process, it segf... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | dgrisby |
| hostname | github.com |
| expected-hostname | github.com |
| None | 87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456 |
| turbo-cache-control | no-preview |
| go-import | github.com/python/cpython git https://github.com/python/cpython.git |
| octolytics-dimension-user_id | 1525981 |
| octolytics-dimension-user_login | python |
| octolytics-dimension-repository_id | 81598961 |
| octolytics-dimension-repository_nwo | python/cpython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 81598961 |
| octolytics-dimension-repository_network_root_nwo | python/cpython |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 423dbb54c5e7bfecc838eb387c1435eb3d0e760d |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width