René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Under-estimated stack size for recursion limit (macOS + python \u003e= 3.14)","articleBody":"# Bug report\n\n### Bug description:\n\nGH-130398 introduced estimation of stack size (for the purpose of recursion-limit management) via `pthread_getattr_np()` on platforms that support it. This non-portable function is not available on macOS, and so the following fallback codepath is used:\n\nhttps://github.com/python/cpython/blob/4fb338d844a1e992857a17b5bd1269837e847fb2/Python/ceval.c#L485-L487\n\nwith `Py_C_STACK_SIZE` being set to 4 MB on macOS:\n\nhttps://github.com/python/cpython/blob/4fb338d844a1e992857a17b5bd1269837e847fb2/Python/ceval.c#L369-L381\n\nHowever, this is too conservative, as the python's build script explicitly increases the stack to 16 MB:\n\nhttps://github.com/python/cpython/blob/4fb338d844a1e992857a17b5bd1269837e847fb2/configure.ac#L3601-L3628\n\nThis was observed in GH-131543, but was not investigated further, because the primary issue there turned out to be excessive stack consumption due to inlining (GH-137573).\n\nNevertheless, the under-estimated stack size still negatively impacts the recursion limit.\n\nUsing slightly modified reproducer from https://github.com/python/cpython/issues/131543#issuecomment-2907899988 that keeps trying to dynamically increase recursion limit as it keeps recursing:\n\n```c\n// stackpointer.c\n//\n// gcc -shared -fpic -o stackpointer.dylib stackpointer.c\n//\n// import ctypes\n// sp = ctypes.CDLL('./stackpointer.dylib')\n// address = sp.get_machine_stack_pointer()\n\n#include \u003cstdint.h\u003e\n\nuintptr_t get_machine_stack_pointer(void)\n{\n    return (uintptr_t)__builtin_frame_address(0);\n}\n```\n\n```python\n# recursion_limit_test.py\nimport sys\nimport ctypes\nimport ctypes.util\n\n# Obtain stack address and stack size as reported by non-portable pthread functions:\nlibc = ctypes.CDLL(ctypes.util.find_library('c'))\n\nlibc.pthread_self.restype = ctypes.c_void_p\n\nlibc.pthread_get_stackaddr_np.argtypes = [ctypes.c_void_p]\nlibc.pthread_get_stackaddr_np.restype = ctypes.c_void_p\n\nlibc.pthread_get_stacksize_np.argtypes = [ctypes.c_void_p]\nlibc.pthread_get_stacksize_np.restype = ctypes.c_ulonglong\n\nthis_thread = libc.pthread_self()\n\nstack_address = libc.pthread_get_stackaddr_np(this_thread)\nstack_size = libc.pthread_get_stacksize_np(this_thread)\n\nprint(f\"Stack address: {stack_address} = 0x{stack_address:X}\")\nprint(f\"Stack size: {stack_size} = {stack_size / 1024.0} kB = {stack_size / 1024**2} MB\")\n\n# Helper for tracking stack pointer location\nsplib = ctypes.CDLL('./stackpointer.dylib')\nsplib.get_machine_stack_pointer.restype = ctypes.c_void_p\nstack_pointer = splib.get_machine_stack_pointer()\nprint(f\"Stack pointer: 0x{stack_pointer:X}, depth: {(stack_address - stack_pointer)/1024:.2f} kB\")\n\n# Recursion limit test\nlimit = sys.getrecursionlimit()\ncounter = 0\n\nclass A:\n    def __getattribute__(self, name):\n        global counter\n        counter += 1\n        stack_pointer = splib.get_machine_stack_pointer()\n        print(f\"Recursion level: {counter}, stack pointer: 0x{stack_pointer:X}, depth: {(stack_address - stack_pointer)/1024:.2f} kB\")\n\n        # Increase recursion limit, if necessary\n        global limit\n        if counter + 1 \u003e= limit:\n            limit *= 2\n            print(f\"Increasing recursion limit: {limit}\")\n            sys.setrecursionlimit(limit)\n\n        # Recurse\n        return getattr(self, name)\n\n\na = A()\nprint(\"Testing Recursion Limit\")\nprint(f\"Initial limit: {limit}\")\ntry:\n    a.test\nexcept RecursionError:\n    print(f\"Recursion Limit ok (reached level {counter})\")\n``` \n\n----\nRunning with python 3.13:\n\n```\n% python3.13 recursion_limit_test.py\nStack address: 6101024768 = 0x16BA64000\nStack size: 16777216 = 16384.0 kB = 16.0 MB\nStack pointer: 0x16BA622A0, depth: 7.34 kB\nTesting Recursion Limit\nInitial limit: 1000\nRecursion level: 1, stack pointer: 0x16BA61CD0, depth: 8.80 kB\nRecursion level: 2, stack pointer: 0x16BA616A0, depth: 10.34 kB\nRecursion level: 3, stack pointer: 0x16BA610B0, depth: 11.83 kB\n...\nRecursion level: 4997, stack pointer: 0x16B323CD0, depth: 7424.80 kB\nRecursion level: 4998, stack pointer: 0x16B3236E0, depth: 7426.28 kB\nRecursion Limit ok (reached level 4999)\n```\n\nNote that even with the old approach, only about half of the actual 16 MB stack was used before the recursion limit kicked in  (as evident from estimated stack depth).\n\n---\nWith 3.14(.0rc2):\n\n```\n% python3.14 recursion_limit_test.py\nStack address: 6091063296 = 0x16B0E4000\nStack size: 16777216 = 16384.0 kB = 16.0 MB\nStack pointer: 0x16B0E1CC0, depth: 8.81 kB\nTesting Recursion Limit\nInitial limit: 1000\nRecursion level: 1, stack pointer: 0x16B0E1120, depth: 11.72 kB\nRecursion level: 2, stack pointer: 0x16B0E0550, depth: 14.67 kB\n...\nRecursion level: 1320, stack pointer: 0x16AD13470, depth: 3906.89 kB\nRecursion level: 1321, stack pointer: 0x16AD128A0, depth: 3909.84 kB\nRecursion Limit ok (reached level 1322)\n``` \n\nWith 3.14, we don't get far above the original limit before the recursion limit kicks in, under assumption of 4 MB stack.\n\n### CPython versions tested on:\n\n3.14\n\n### Operating systems tested on:\n\nmacOS\n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-139232\n* gh-139290\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/rokm","@type":"Person","name":"rokm"},"datePublished":"2025-09-22T12:15:50.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/139231/cpython/issues/139231"}