René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"segmentation fault in PyContextVar_set under py3.9 ARM64","articleBody":"# Bug report\r\n\r\n### Bug description:\r\n\r\nA segmentation fault sometimes occurs in an application using Datadog's ``ddtrace-run`` 2.9.0, CPython 3.9, and sqlalchemy on ARM64. Our (@sanchda and my) analysis suggests that this is the result of a bug in CPython 3.9, which may have been fixed in CPython 3.12. We are unable to replicate this issue directly because it happens sporadically on infrastructure operated by a Datadog customer.\r\n\r\nOne instance of the segfault for which we managed to capture a core dump looks like this:\r\n\r\n```\r\nCurrent thread 0x0000ffff9a4915a0 (most recent call first):,1719935856047\r\n\"File \"\"/usr/local/lib/python3.9/site-packages/ddtrace/_trace/provider.py\"\", line 126 in activate\",1719935856047\r\n\"File \"\"/usr/local/lib/python3.9/site-packages/ddtrace/_trace/tracer.py\"\", line 807 in _start_span\",1719935856047\r\n\"File \"\"/usr/local/lib/python3.9/site-packages/ddtrace/_trace/tracer.py\"\", line 898 in trace\",1719935856047\r\n\"File \"\"/usr/local/lib/python3.9/site-packages/ddtrace/contrib/dbapi/__init__.py\"\", line 311 in _trace_method\",1719935856047\r\n\"File \"\"/usr/local/lib/python3.9/site-packages/ddtrace/contrib/dbapi/__init__.py\"\", line 335 in rollback\",1719935856047\r\n\"File \"\"/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/default.py\"\", line 683 in do_rollback\",1719935856047\r\n\"File \"\"/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py\"\", line 1038 in _reset\",1719935856047\r\n\"File \"\"/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py\"\", line 763 in _finalize_fairy\",1719935856047\r\n\"File \"\"/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py\"\", line 1008 in _checkin\",1719935856047\r\n\"File \"\"/usr/local/lib/python3.9/site-packages/sqlalchemy/pool/base.py\"\", line 1166 in close\",1719935856047\r\n\"File \"\"/usr/local/lib/python3.9/site-packages/sqlalchemy/engine/base.py\"\", line 1251 in close\",1719935856047\r\n\"File \"\"/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py\"\", line 928 in close\",1719935856047\r\n\"File \"\"/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py\"\", line 846 in commit\",1719935856047\r\n\"File \"\"/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/session.py\"\", line 1454 in commit\",1719935856047\r\n# application stack frames....\r\n```\r\n\r\nThe leaf node of this Python stacktrace is [this call](https://github.com/DataDog/dd-trace-py/blob/1e4ecf948e3a12b82ca6307f7bbcf52ceedad074/ddtrace/_trace/provider.py#L126) to `contextvars.ContextVar().set()`, which operates on a contextvar instance internal to Datadog code that's initialized [here](https://github.com/DataDog/dd-trace-py/blob/1e4ecf948e3a12b82ca6307f7bbcf52ceedad074/ddtrace/_trace/provider.py#L17-L19).\r\n\r\nThe core dump also includes this native backtrace:\r\n\r\n```\r\n*** Segmentation fault,1719935856047\r\nBacktrace:,1719935856047\r\n/lib/aarch64-linux-gnu/libc.so.6(+0x80a10)[0xffff99fb0a10],1719935856048\r\n/lib/aarch64-linux-gnu/libc.so.6(gsignal+0x1c)[0xffff99f6a76c],1719935856048\r\nlinux-vdso.so.1(__kernel_rt_sigreturn+0x0)[0xffff9a49d850],1719935856048\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(+0xd2538)[0xffff9a1b2538],1719935856048\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(+0xd2ce0)[0xffff9a1b2ce0],1719935856048\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(+0xd3414)[0xffff9a1b3414],1719935856048\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(+0xcd82c)[0xffff9a1ad82c],1719935856048\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(PyContextVar_Set+0x10c)[0xffff9a1add0c],1719935856048\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(+0x13a4c8)[0xffff9a21a4c8],1719935856048\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(_PyEval_EvalFrameDefault+0x548)[0xffff9a249e28],1719935856048\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(+0x168c24)[0xffff9a248c24],1719935856048\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(_PyFunction_Vectorcall+0xb0)[0xffff9a215bd0],1719935856048\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(_PyEval_EvalFrameDefault+0x548)[0xffff9a249e28],1719935856048\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(+0x168c24)[0xffff9a248c24],1719935856048\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(_PyFunction_Vectorcall+0xb0)[0xffff9a215bd0],1719935856048\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(+0x137df0)[0xffff9a217df0],1719935856048\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(_PyEval_EvalFrameDefault+0x1540)[0xffff9a24ae20],1719935856048\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(+0x168c24)[0xffff9a248c24],1719935856048\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(_PyFunction_Vectorcall+0xb0)[0xffff9a215bd0],1719935856048\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(+0x137df0)[0xffff9a217df0],1719935856048\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(_PyEval_EvalFrameDefault+0x1540)[0xffff9a24ae20],1719935856048\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(+0x168c24)[0xffff9a248c24],1719935856048\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(_PyFunction_Vectorcall+0xb0)[0xffff9a215bd0],1719935856048\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(+0x137eec)[0xffff9a217eec],1719935856048\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(_PyEval_EvalFrameDefault+0x1be8)[0xffff9a24b4c8],1719935856048\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(+0x168c24)[0xffff9a248c24],1719935856048\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(_PyFunction_Vectorcall+0xb0)[0xffff9a215bd0],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(+0x137df0)[0xffff9a217df0],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(_PyEval_EvalFrameDefault+0x3134)[0xffff9a24ca14],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(+0x135da0)[0xffff9a215da0],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(_PyEval_EvalFrameDefault+0x548)[0xffff9a249e28],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(+0x168c24)[0xffff9a248c24],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(_PyFunction_Vectorcall+0xb0)[0xffff9a215bd0],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(+0x137df0)[0xffff9a217df0],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(_PyEval_EvalFrameDefault+0x3134)[0xffff9a24ca14],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(+0x168c24)[0xffff9a248c24],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(_PyFunction_Vectorcall+0xb0)[0xffff9a215bd0],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(_PyEval_EvalFrameDefault+0x1540)[0xffff9a24ae20],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(+0x168c24)[0xffff9a248c24],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(_PyFunction_Vectorcall+0xb0)[0xffff9a215bd0],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(+0x137df0)[0xffff9a217df0],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(_PyEval_EvalFrameDefault+0x3134)[0xffff9a24ca14],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(+0x135da0)[0xffff9a215da0],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(+0x137df0)[0xffff9a217df0],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(_PyEval_EvalFrameDefault+0x3134)[0xffff9a24ca14],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(+0x135da0)[0xffff9a215da0],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(_PyEval_EvalFrameDefault+0x548)[0xffff9a249e28],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(+0x168c24)[0xffff9a248c24],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(_PyFunction_Vectorcall+0xb0)[0xffff9a215bd0],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(_PyEval_EvalFrameDefault+0x548)[0xffff9a249e28],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(+0x168c24)[0xffff9a248c24],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(_PyFunction_Vectorcall+0xb0)[0xffff9a215bd0],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(+0x137df0)[0xffff9a217df0],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(_PyEval_EvalFrameDefault+0x1540)[0xffff9a24ae20],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(+0x135da0)[0xffff9a215da0],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(_PyEval_EvalFrameDefault+0x548)[0xffff9a249e28],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(+0x135da0)[0xffff9a215da0],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(_PyEval_EvalFrameDefault+0x6d4)[0xffff9a249fb4],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(+0x135da0)[0xffff9a215da0],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(_PyEval_EvalFrameDefault+0x3134)[0xffff9a24ca14],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(+0x135da0)[0xffff9a215da0],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(_PyEval_EvalFrameDefault+0x548)[0xffff9a249e28],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(+0x135da0)[0xffff9a215da0],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(_PyEval_EvalFrameDefault+0x548)[0xffff9a249e28],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(+0x168c24)[0xffff9a248c24],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(_PyFunction_Vectorcall+0xb0)[0xffff9a215bd0],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(_PyEval_EvalFrameDefault+0x548)[0xffff9a249e28],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(+0x135da0)[0xffff9a215da0],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(_PyEval_EvalFrameDefault+0x548)[0xffff9a249e28],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(+0x135da0)[0xffff9a215da0],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(_PyEval_EvalFrameDefault+0x548)[0xffff9a249e28],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(+0x168c24)[0xffff9a248c24],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(_PyEval_EvalCodeWithName+0x64)[0xffff9a2aeb74],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(PyEval_EvalCodeEx+0x40)[0xffff9a2aeb00],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(PyEval_EvalCode+0x2c)[0xffff9a2aeaac],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(+0x1e08fc)[0xffff9a2c08fc],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(+0x1e0868)[0xffff9a2c0868],1719935856049\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(+0x1e0798)[0xffff9a2c0798],1719935856050\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(PyRun_SimpleFileExFlags+0x190)[0xffff9a2c03e4],1719935856050\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(Py_RunMain+0x3c4)[0xffff9a2c9624],1719935856050\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(Py_BytesMain+0x38)[0xffff9a2c9078],1719935856050\r\n/lib/aarch64-linux-gnu/libc.so.6(+0x27780)[0xffff99f57780],1719935856050\r\n/lib/aarch64-linux-gnu/libc.so.6(__libc_start_main+0x98)[0xffff99f57858],1719935856050\r\n/usr/local/bin/python3(_start+0x30)[0xaaaaad2308b0],1719935856050\r\n```\r\n\r\nThese lines in particular warrant scrutiny:\r\n\r\n```\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(+0xd2538)[0xffff9a1b2538],1719935856048\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(+0xd2ce0)[0xffff9a1b2ce0],1719935856048\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(+0xd3414)[0xffff9a1b3414],1719935856048\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(+0xcd82c)[0xffff9a1ad82c],1719935856048\r\n/usr/local/bin/../lib/libpython3.9.so.1.0(PyContextVar_Set+0x10c)[0xffff9a1add0c],1719935856048\r\n```\r\n\r\nWe grabbed the `libpython3.9.so.1.0` from Dockerhub's ``python:3.9.19-bookworm, (arm64)`` image, version ``sha256:37823bd8e7266bac0399e3b55e67cfe00297686f09fe5cf73411ffe1d75fd93e``. This .so file is bitwise-identical to the artifact used in the production environment that the core dump came from.\r\n\r\nWe used `objdump` to symbolize the containing functions within `libpython3.9.so.1.0` and found the following:\r\n\r\n```\r\n0xd2538 -\u003e hamt_node_bitmap_clone\r\n0xd2ce0 -\u003e hamt_node_bitmap_assoc\r\n0xd3414 -\u003e _PyHamt_Assoc\r\n0xcd82c -\u003e contextvar_set\r\n0x10c   -\u003e PyContextVar_set\r\n```\r\n\r\nThe leaf of this call stack is defined in Py3.9 [here](https://github.com/python/cpython/blob/06fa244666ec6335a3b9bf2367e31b42b9a89b20/Python/hamt.c#L589).\r\n\r\n`ghidra` gives us the following decompilation of the `hamt_node_bitmap_clone` function:\r\n\r\n```\r\nvoid hamt_node_bitmap_clone(long param_1)\r\n\r\n{\r\n  long lVar1;\r\n  long lVar2;\r\n  long *plVar3;\r\n  long lVar4;\r\n\r\n  lVar1 = hamt_node_bitmap_new(*(undefined8 *)(param_1 + 0x10));\r\n  if (lVar1 != 0) {\r\n    lVar4 = *(long *)(param_1 + 0x10);\r\n    for (lVar2 = 0; lVar2 \u003c lVar4; lVar2 = lVar2 + 1) {\r\n      plVar3 = *(long **)(param_1 + 0x20 + lVar2 * 8);                     // 0xd2530\r\n      if (plVar3 != (long *)0x0) {\r\n        *plVar3 = *plVar3 + 1;\r\n      }\r\n      *(long **)(lVar1 + 0x20 + lVar2 * 8) = plVar3;\r\n    }\r\n    *(undefined4 *)(lVar1 + 0x18) = *(undefined4 *)(param_1 + 0x18);\r\n  }\r\n  return;\r\n}\r\n```\r\n\r\n`objdump` tells us that the specific fault site is `0xd2538`, which ghidra tells us is on the line\r\n\r\n```\r\nplVar3 = *(long **)(param_1 + 0x20 + lVar2 * 8);\r\n```\r\n\r\nThe source code counterpart of this decompiled line is [here](https://github.com/python/cpython/blob/06fa244666ec6335a3b9bf2367e31b42b9a89b20/Python/hamt.c#L602), a call to `Py_XINCREF`.\r\n\r\n`ghidra` gives the failing instructions from this line as\r\n\r\n```\r\n001d2530 a2 78 61 f8     ldr        x2,[x5, x1, LSL #0x3]\r\n001d2534 82 00 00 b4     cbz        x2,LAB_001d2544\r\n001d2538 43 00 40 f9     ldr        x3,[x2]               # this is the site of the fault\r\n001d253c 63 04 00 91     add        x3,x3,#0x1\r\n```\r\n\r\n* `ldr`: load the given value into the register x2.\r\n* `cbz`: conditional jump if zero.  This is looking at x2 as a value, not as a pointer.\r\n    It’s probably part of ``Py_XINCREF`` where it checks non-null. Since the next instruction is executed, x2 is NOT null.\r\n* `ldr`: de-references x2 and puts it in x3. x2 is not NULL (we know that because the computation on the ldr line is an iteration)\r\n    So, x2 is an invalid address\r\n* `add`: this is just the increment operation from the ``Py_XINCREF``\r\n\r\nIn terms of the source code, this suggests that `node-\u003eb_array[i]` is an invalid nonzero pointer [here](https://github.com/python/cpython/blob/06fa244666ec6335a3b9bf2367e31b42b9a89b20/Python/hamt.c#L602).\r\n\r\nSome speculative reasons why this could happen:\r\n\r\n1. The given item was cleaned up and is no longer valid, although sometimes it is valid because it is freed but not unmapped\r\n    This might explain why the problem occurs on arm but not x86, since the mapping behavior at the level of the allocator may just so happen to be different for system reasons\r\n2. The size of the array is incorrect\r\n3. The item is uninitialized\r\n\r\nIn this case, the \"item\" is a HAMT node underlying a `ContextVar` instance being `set()` during a sqlalchemy rollback operation.\r\n\r\nAside from this analysis, another reason we're opening this issue on CPython is that the `Py_XINCREF` line in question was changed since Python 3.9. #99317 replaced it with a call to `Py_XNewRef` [here](https://github.com/python/cpython/pull/99317/files#diff-57efcedae5cd47def53e512cf0bae5554f794563c3c6269b3a9ac3fa24055001R578). While we haven't tested this against Python 3.12 (the first to include the change), the presence of this change suggests that a bug may have been fixed in newer versions that could be worth backporting. Even without a backport, confirmation that this is or was a real issue in CPython itself would be helpful.\r\n\r\n### CPython versions tested on:\r\n\r\n3.9\r\n\r\n### Operating systems tested on:\r\n\r\nLinux","author":{"url":"https://github.com/emmettbutler","@type":"Person","name":"emmettbutler"},"datePublished":"2024-07-31T18:07:58.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":9},"url":"https://github.com/122529/cpython/issues/122529"}