René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Creating an ungodly amount of sub interpreters in a short amount of time causes memory debug assertions.","articleBody":"# Bug report\n\n### Bug description:\n\nHello. While working on a small joke program, I found a possible memory corruption issue (it could also be a threading issue?) when using the Python C API in a debug only build to quickly create, execute python code, and then destroy *463 sub interpreters*. Before I post the code sample and the debug output I'm using a somewhat unique build environment for a Windows developer.\n\n - clang++ 18.1.7 for x86_64-pc-windows-msvc\n - Visual Studio Build Tools 2022 17.11.0\n -  CMake 3.30\n - Python 3.12.5\n - ninja 1.12.1\n\nWhen running the code sample I've attached at the bottom of this post, I am unable to get the exact same output each time, though the traceback does fire in the same location (Due to the size of the traceback I've not attached it, as it's about 10 MB of text for each thread). Additionally, I sometimes have to run the executable several times to get the error to occur. Lastly, release builds do *not* exhibit any thread crashes or issues as the debug assertions never fire or execute.\n\nThe error output seems to also halt in some cases, either because of stack failure or some other issue I was unable to determine and seemed to possibly be outside the scope of the issue presented here. I have the entire error output from one execution where I was able to save the output.\n\n```\nDebug memory block at address p=00000267272D6030: API 'p'\n    16718206241729413120 bytes originally requested\n    The 7 pad bytes at p-7 are not all FORBIDDENBYTE (0xfd):\n        at p-7: 0xa0 *** OUCH\n        at p-6: 0x00 *** OUCH\n        at p-5: 0x00Hello, world! From Thread 31\nHello, world! From Thread 87\n *** OUCH\n        at p-4: 0xfd\nHello, world! From Thread 43\nHello, world! From Thread 279\nGenerating thread state 314\n        at p-3: 0xfd\n        at p-2: 0xfd\nHello, world! From Thread 168\nGenerating thread state 315\n        at p-1: 0xfd\n    Because memory is corrupted at the start, the count of bytes requested\n       may be bogus, and checking the trailing pad bytes may segfault.\nGenerating thread state 316\nGenerating thread state 317\n    The 8 pad bytes at tail=E8030267272D6030 are\n```\n\nThe output cut off after this, as the entire program crashed, taking my terminal with it 😅 \n\nYou'll find the MRE code below. I've also added a minimal version of CMakeLists.txt file I used so anyone can recreate the build with the code below (Any warnings, or additional settings I have do not affect whether the error occurs or not). The code appears to breaks inside of `_PyObject_DebugDumpAddress`, based on what debugging I was able to do with WinDbg. \n\n\u003e [!IMPORTANT]\n\u003e `std::jthread` calls `.join()` on destruction, so all threads auto-join once the `std::vector` goes out of scope.\n\u003e \n\u003e Additionally this code exhibits the same behavior regardless of whether it is a `thread_local` or declared within the lambda passed to `std::thread`\n\nmain.cxx\n```cpp\n#include \u003cvector\u003e\n#include \u003cthread\u003e\n#include \u003ccstdlib\u003e\n#include \u003cprint\u003e\n\n#include \u003cPython.h\u003e\n\nnamespace {\n\nstatic thread_local inline PyThreadState* state = nullptr;\nstatic inline constexpr auto MAX_STATES = 463; \nstatic inline constexpr auto config = PyInterpreterConfig {\n  .use_main_obmalloc = 0,\n  .allow_fork = 0,\n  .allow_exec = 0,\n  .allow_threads = 0,\n  .allow_daemon_threads = 0,\n  .check_multi_interp_extensions = 1,\n  .gil = PyInterpreterConfig_OWN_GIL,\n};\n\n} /* nameless namespace */\n\nvoid execute () {\n  std::vector\u003cstd::jthread\u003e tasks { };\n  tasks.reserve(MAX_STATES);\n  for (auto count = 0zu; count \u003c tasks.capacity(); count++) {\n    std::println(\"Generating thread state {}\", count);\n    tasks.emplace_back([count] {\n      if (auto status = Py_NewInterpreterFromConfig(\u0026state, \u0026config); PyStatus_IsError(status)) {\n        std::println(\"Failed to initialize thread state {}\", count);\n        return;\n      }\n      auto text = std::format(R\"(print(\"Hello, world! From Thread {}\"))\", count);\n      auto globals = PyDict_New();\n      auto code = Py_CompileString(text.data(), __FILE__, Py_eval_input);\n      auto result = PyEval_EvalCode(code, globals, globals);\n      Py_DecRef(result);\n      Py_DecRef(code);\n      Py_DecRef(globals);\n      Py_EndInterpreter(state);\n      state = nullptr;\n    });\n  }\n}\n\nint main() {\n  PyConfig config {};\n  PyConfig_InitIsolatedConfig(\u0026config);\n  if (auto status = Py_InitializeFromConfig(\u0026config); PyStatus_IsError(status)) {\n    std::println(\"Failed to initialize with isolated config: {}\", status.err_msg);\n    return EXIT_FAILURE;\n  }\n  PyConfig_Clear(\u0026config);\n  execute();\n  Py_Finalize();\n}\n```\nCMakeLists.txt\n```cmake\ncmake_minimum_required(VERSION 3.30)\nproject(463-interpreters LANGUAGES C CXX)\n\nfind_package(Python 3.12 REQUIRED COMPONENTS Development.Embed)\n\nset(CMAKE_MSVC_RUNTIME_LIBRARY \"MultiThreaded$\u003c$\u003cCONFIG:Debug\u003e:Debug\u003e\")\n\nadd_executable(${PROJECT_NAME})\ntarget_sources(${PROJECT_NAME} PRIVATE main.cxx)\ntarget_compile_features(${PROJECT_NAME} PRIVATE cxx_std_23)\ntarget_precompile_headers(${PROJECT_NAME} PRIVATE \u003cPython.h\u003e)\ntarget_link_libraries(${PROJECT_NAME} PRIVATE Python::Python)\n```\n\nCommand to build + run\n```shell\n$ cmake -Bbuild -S. -G \"Ninja\"\n$ cmake --build build \u0026\u0026 .\\build\\463-interpreters.exe\n```\n\n### CPython versions tested on:\n\n3.12\n\n### Operating systems tested on:\n\nWindows","author":{"url":"https://github.com/bruxisma","@type":"Person","name":"bruxisma"},"datePublished":"2024-08-19T05:11:50.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":88},"url":"https://github.com/123134/cpython/issues/123134"}