Title: Py_NewInterpreterFromConfig end with hard crash · Issue #123488 · python/cpython · GitHub
Open Graph Title: Py_NewInterpreterFromConfig end with hard crash · Issue #123488 · python/cpython
X Title: Py_NewInterpreterFromConfig end with hard crash · Issue #123488 · python/cpython
Description: Crash report What happened? I started using Py_NewInterpreterFromConfig to add an PER-THREAD interpreter with my library. The library is thread safe and supports many languages C C++ JAVA C# VB.Net Perl Python GO TCL Ruby Php including o...
Open Graph Description: Crash report What happened? I started using Py_NewInterpreterFromConfig to add an PER-THREAD interpreter with my library. The library is thread safe and supports many languages C C++ JAVA C# VB.Net...
X Description: Crash report What happened? I started using Py_NewInterpreterFromConfig to add an PER-THREAD interpreter with my library. The library is thread safe and supports many languages C C++ JAVA C# VB.Net...
Opengraph URL: https://github.com/python/cpython/issues/123488
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Py_NewInterpreterFromConfig end with hard crash","articleBody":"# Crash report\r\n\r\n### What happened?\r\n\r\nI started using **Py_NewInterpreterFromConfig** to add an **PER-THREAD interpreter** with my library. \r\nThe library is thread safe and supports many languages\r\n\r\n- C C++ JAVA C# VB.Net Perl Python GO TCL Ruby Php\r\n\r\nincluding other languages with thread support\r\n\r\n- C C++ JAVA C# TCL GO\r\n\r\nare working fine. Even _python_ works fine without thread support.\r\n\r\nTask: now I started to use thread support in python. \r\nI had to change my code to to Multi-phase initialization, isolate the python data on an per-thread-level and finally get an easy thread example working -> ok\r\n\r\nI test a client server application. the server create a thread in **my** library and this thread is initialized with\r\n```\r\n if (create == MQ_FACTORY_NEW_THREAD) {\r\n//mk_debug_color(MK_YELLOW,\"%s\",\"MQ_FACTORY_NEW_THREAD\");\r\n static PyInterpreterConfig config = {\r\n .use_main_obmalloc = 0,\r\n .allow_fork = 0,\r\n .allow_exec = 0,\r\n .allow_threads = 1,\r\n .allow_daemon_threads = 0,\r\n .check_multi_interp_extensions = 1,\r\n .gil = PyInterpreterConfig_OWN_GIL,\r\n };\r\n\r\n PyThreadState *tstate = NULL;\r\n PyStatus status = Py_NewInterpreterFromConfig(&tstate, &config);\r\n MK_RT_REF.threadData = tstate;\r\n\r\n//printV(\"#2 PyThreadState_Get\u003c%p>\", PyThreadState_Get())\r\n\r\n if (PyStatus_Exception(status)) {\r\n return MkErrorSetC_3_NULL(status.err_msg, status.func, 300+status.exitcode);\r\n }\r\n\r\n // mark new interpreter as \"MQ_STARTUP_IS_THREAD\"\r\n PyObject *mainM = PyImport_AddModule(\"__main__\");\r\n PyObject *strO = PyUnicode_FromString(\"MQ_STARTUP_IS_THREAD\");\r\n PyObject_SetAttrString(mainM,\"__name__\", strO);\r\n Py_CLEAR(strO);\r\n // source initial script\r\n FILE *FH = fopen(MqInitGetArg0()->data[1]->storage.first.C, \"r\");\r\n check_LNG(PyRun_SimpleFileEx(FH,MqInitGetArg0()->data[1]->storage.first.C,true)) {\r\n return MkErrorSetV_1E(\"script '%s' failed\",MqInitGetArg0()->data[1]->storage.first.C);\r\n }\r\n }\r\n```\r\nI figure out that always after **4** successful simple round trip __clinet → server → client__ the server crash with the message below\r\n- The stack-trace comes from my `MqDisasterSignal` handler.\r\n\r\n```\r\nexec[#6] -> 'NHI1_EXT/debug/bin/python3' 'NHI1_HOME/example/py/MyServer.py' '--thread' '--uds' '--file' '/tmp/test.uds'\r\nDebug memory block at address p=0x7f0012d1a730: API '^@'\r\n 18302063728033398269 bytes originally requested\r\n The 7 pad bytes at p-7 are not all FORBIDDENBYTE (0xfd):\r\n at p-7: 0x00 *** OUCH\r\n at p-6: 0x00 *** OUCH\r\n at p-5: 0x00 *** OUCH\r\n at p-4: 0x00 *** OUCH\r\n at p-3: 0x00 *** OUCH\r\n at p-2: 0x00 *** OUCH\r\n at p-1: 0x00 *** OUCH\r\n Because memory is corrupted at the start, the count of bytes requested\r\n may be bogus, and checking the trailing pad bytes may segfault.\r\n The 8 pad bytes at tail=0xfdfe7cfe10cfa52d are X> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: BackTrace {\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ library : filename : lineno ] function\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ ------- : -------- : ------ ] --------\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ theLink : c/sys_mq.c : 705 ] MqDisasterSignal\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ unknown : unknown : 0 ] unknown\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/obmalloc.c : 2408 ] _PyObject_DebugDumpAddress\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/obmalloc.c : 2326 ] _PyMem_DebugCheckAddress\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/obmalloc.c : 2159 ] _PyMem_DebugRawFree\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/obmalloc.c : 685 ] PyMem_RawFree\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/obmalloc.c : 1853 ] _PyObject_Free\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/obmalloc.c : 2163 ] _PyMem_DebugRawFree\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/obmalloc.c : 2296 ] _PyMem_DebugFree\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/obmalloc.c : 830 ] PyObject_Free\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/dictobject.c : 1569 ] dictresize\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/dictobject.c : 1194 ] insertion_resize\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/dictobject.c : 1261 ] insertdict\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/dictobject.c : 1865 ] _PyDict_SetItem_Take2\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/dictobject.c : 1883 ] PyDict_SetItem\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/typeobject.c : 7618 ] add_subclass\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/typeobject.c : 7352 ] type_ready_add_subclasses\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/typeobject.c : 7515 ] type_ready\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/typeobject.c : 7553 ] PyType_Ready\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/typeobject.c : 3795 ] type_new_impl\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/typeobject.c : 3929 ] type_new\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/typeobject.c : 1664 ] type_call\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/call.c : 240 ] _PyObject_MakeTpCall\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/typeobject.c : 3949 ] type_vectorcall\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/call.c : 133 ] _PyObject_FastCallDictTstate\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/call.c : 157 ] PyObject_VectorcallDict\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Python/bltinmodule.c : 208 ] builtin___build_class__\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/methodobject.c : 438 ] cfunction_vectorcall_FASTCALL_KEYWORDS\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Include/internal/pycore_call.h : 92 ] _PyObject_VectorcallTstate\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Objects/call.c : 325 ] PyObject_Vectorcall\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Python/bytecodes.c : 2714 ] _PyEval_EvalFrameDefault\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : nclude/internal/pycore_ceval.h : 89 ] _PyEval_EvalFrame\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Python/ceval.c : 1683 ] _PyEval_Vector\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Python/ceval.c : 578 ] PyEval_EvalCode\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Python/pythonrun.c : 1722 ] run_eval_code_obj\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Python/pythonrun.c : 1743 ] run_mod\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Python/pythonrun.c : 1643 ] pyrun_file\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Python/pythonrun.c : 433 ] _PyRun_SimpleFileObject\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ system : Python/pythonrun.c : 466 ] PyRun_SimpleFileExFlags\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ theLink : py/MqFactoryC_py.c : 195 ] py_mqmsgque_sFactoryCTor\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ theLink : c/MqFactoryS_mq.c : 289 ] MqFactoryInvoke_RT\r\nX> {PRINT :pid(66272):tid(0x7f000f479700):X:dlv(0):ctxId( 0):rc(1):ctx(0x1b87fb8 ):MqDisasterSignal }: [ theLink : c/sys_mq.c : 269 ] MqSysServerThreadMain\r\n```\r\n\r\nThe server code (file sourced into the new interpreter) is quite simple.\r\n\r\n```python\r\n# Add a code block here, if required\r\nimport sys\r\nfrom pymqmsgque import *\r\n\r\n# package-item\r\nclass MyServer(MqContextC):\r\n \r\n # factory startup\r\n def __init__(self, tmpl=None):\r\n self.ConfigSetServerSetup(self.ServerSetup)\r\n super().__init__(tmpl)\r\n \r\n # service to serve all incoming requests for token \"HLWO\"\r\n def MyFirstService(self):\r\n self.SendSTART()\r\n self.SendSTR(self.ReadSTR() + \" World\")\r\n self.SendRETURN()\r\n \r\n # define a service as link between the token \"HLWO\" and the callback \"MyFirstService\"\r\n def ServerSetup(self):\r\n self.ServiceCreate(\"HLWO\",self.MyFirstService)\r\n \r\n# package-main\r\nif __name__ == \"__main__\":\r\n\r\n # create the \"MyServer\" factory… and the object\r\n srv = MqFactoryC.Add(MyServer).New()\r\n\r\n try:\r\n srv.LinkCreate(sys.argv)\r\n srv.ProcessEvent(MqWaitOnEventE.FOREVER)\r\n except Exception as ex:\r\n srv.ErrorCatch(ex)\r\n finally:\r\n srv.Exit()\r\n```\r\n\r\nthe problem is that running **this** example with a server with **valgrind** nothing happen (**no** crash)\r\nthe _same_ example using **fork** or using **spawn** to startup the server instances works fine. \r\nthe problem is **only** with thread support.\r\n\r\nadditional Information: because my library **only** support **one** server-instance per thread, I choose `PyInterpreterConfig_OWN_GIL` to be clear that I **don't** want to have interaction between different python interpreters.\r\n**all** the communication (between processes and threads) is done by my library. my library uses _thread-local-storage_ to isolate the data **per-thread**.\r\n\r\n\r\n### CPython versions tested on:\r\n\r\n3.12\r\n\r\n### Operating systems tested on:\r\n\r\nLinux\r\n\r\n### Output from running 'python -VV' on the command line:\r\n\r\nPython 3.12.4 (main, Aug 29 2024, 18:00:55) [GCC 13.3.0]","author":{"url":"https://github.com/aotto1968","@type":"Person","name":"aotto1968"},"datePublished":"2024-08-30T05:38:09Z","url":"https://github.com/python/cpython/issues/123488"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:7df90179-f183-a542-a408-f0b19efa6403 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | DFA4:8AE25:1751325:1EE151E:6B0209C8 |
| html-safe-nonce | af50b6b2613ba60f81236051f377046c9198babcbeebbe5a63e7e29357e9e06f |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJERkE0OjhBRTI1OjE3NTEzMjU6MUVFMTUxRTo2QjAyMDlDOCIsInZpc2l0b3JfaWQiOiI0NTkzMjU5OTIzMzI4MDc2MjMyIiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | 9a43aeed0725f0aaa666f0d8eca93a19cc2a9276b5e5a96a6c2b269496dc475c |
| hovercard-subject-tag | issue:2496284710 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/python/cpython/123488/issue_layout |
| twitter:image | https://opengraph.githubassets.com/1949d0e723db217d35adffc4d19132209da5583f69a5d2b9c7fc17e993a6d266/python/cpython/issues/123488 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/1949d0e723db217d35adffc4d19132209da5583f69a5d2b9c7fc17e993a6d266/python/cpython/issues/123488 |
| og:image:alt | Crash report What happened? I started using Py_NewInterpreterFromConfig to add an PER-THREAD interpreter with my library. The library is thread safe and supports many languages C C++ JAVA C# VB.Net... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | aotto1968 |
| hostname | github.com |
| expected-hostname | github.com |
| None | 87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456 |
| turbo-cache-control | no-preview |
| go-import | github.com/python/cpython git https://github.com/python/cpython.git |
| octolytics-dimension-user_id | 1525981 |
| octolytics-dimension-user_login | python |
| octolytics-dimension-repository_id | 81598961 |
| octolytics-dimension-repository_nwo | python/cpython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 81598961 |
| octolytics-dimension-repository_network_root_nwo | python/cpython |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 423dbb54c5e7bfecc838eb387c1435eb3d0e760d |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width