René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Segfaults on 3.12 when using PySR and running Julia's GC","articleBody":"# Crash report\r\n\r\n### What happened?\r\n\r\nThis is a segfault I am seeing on Python 3.12, when trying to use the Python and Julia runtimes simultaneously via the PyJulia package.\r\n\r\nIt seems like when there is an object that is referenced by both the Julia and Python runtimes, there can be memory access errors. It seems as though Python is trying to free memory which has already been freed in Julia or vice versa.\r\n\r\nI am raising the issue here since the issue has only started occurring on Python 3.12, but does not occur on 3.11. The Julia version does not seem to affect this behavior. So I am trying to understand what changes were made to the Python GC that might have triggered this, and if perhaps the GC is more aggressive in some way?\r\n\r\nHere is my current MWE based on a package I maintain ([PySR](https://github.com/MilesCranmer/PySR)) that uses Julia as  backend for a Python frontend:\r\n\r\n```python\r\nfrom pysr import PySRRegressor\r\n\r\nmodel = PySRRegressor()\r\n\r\n# This runs fine:\r\nmodel.fit([[1]], [1])\r\n\r\n# This second run results in a segmentation fault:\r\nmodel.fit([[1]], [1])\r\n```\r\n\r\nThis is the smallest MWE I have been able to create thus far.\r\n\r\nI also see the issue in my continuous integration tests on Python 3.12, but never before 3.12: https://github.com/MilesCranmer/PySR/pull/450\r\n\r\nFor example, in one of those segfaults, I see the following backtrace:\r\n\r\n```\r\nPlease submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.\r\nException: EXCEPTION_ACCESS_VIOLATION at 0x7ffb0691a026 -- PyObject_Free at C:\\hostedtoolcache\\windows\\Python\\3.12.1\\x64\\python312.dll (unknown line)\r\nin expression starting at none:0\r\nPyObject_Free at C:\\hostedtoolcache\\windows\\Python\\3.12.1\\x64\\python312.dll (unknown line)\r\npydecref_ at C:\\Users\\runneradmin\\.julia\\packages\\PyCall\\1gn3u\\src\\PyCall.jl:118\r\npydecref at C:\\Users\\runneradmin\\.julia\\packages\\PyCall\\1gn3u\\src\\PyCall.jl:123\r\njfptr_pydecref_1039 at C:\\Users\\runneradmin\\.julia\\compiled\\v1.9\\PyCall\\GkzkC_As42O.dll (unknown line)\r\nrun_finalizer at C:/workdir/src\\gc.c:417\r\njl_gc_run_finalizers_in_list at C:/workdir/src\\gc.c:507\r\nrun_finalizers at C:/workdir/src\\gc.c:553\r\nrun_finalizers at C:/workdir/src\\gc.c:534 [inlined]\r\nijl_gc_collect at C:/workdir/src\\gc.c:3732\r\nmaybe_collect at C:/workdir/src\\gc.c:1083 [inlined]\r\njl_gc_pool_alloc_inner at C:/workdir/src\\gc.c:1450 [inlined]\r\njl_gc_pool_alloc_noinline at C:/workdir/src\\gc.c:1511\r\njl_gc_alloc_ at C:/workdir/src\\julia_internal.h:460 [inlined]\r\n_new_array_ at C:/workdir/src\\array.c:144\r\n_new_array at C:/workdir/src\\array.c:198 [inlined]\r\nijl_alloc_array_1d at C:/workdir/src\\array.c:436\r\nArray at .\\boot.jl:477 [inlined]\r\nArray at .\\boot.jl:486 [inlined]\r\nsimilar at .\\array.jl:374 [inlined]\r\nsimilar at .\\abstractarray.jl:839 [inlined]\r\ndeg2_l0_r0_eval at C:\\Users\\runneradmin\\.julia\\packages\\DynamicExpressions\\KRT17\\src\\EvaluateEquation.jl:257\r\njfptr_deg2_l0_r0_eval_1577 at C:\\Users\\runneradmin\\.julia\\compiled\\v1.9\\DynamicExpressions\\BQC8W_As42O.dll (unknown line)\r\n_eval_tree_array at C:\\Users\\runneradmin\\.julia\\packages\\DynamicExpressions\\KRT17\\src\\EvaluateEquation.jl:117\r\n_eval_tree_array at C:\\Users\\runneradmin\\.julia\\packages\\DynamicExpressions\\KRT17\\src\\EvaluateEquation.jl:131\r\n_eval_tree_array at C:\\Users\\runneradmin\\.julia\\packages\\DynamicExpressions\\KRT17\\src\\EvaluateEquation.jl:125\r\n#eval_tree_array#1 at C:\\Users\\runneradmin\\.julia\\packages\\DynamicExpressions\\KRT17\\src\\EvaluateEquation.jl:65 [inlined]\r\neval_tree_array at C:\\Users\\runneradmin\\.julia\\packages\\DynamicExpressions\\KRT17\\src\\EvaluateEquation.jl:59 [inlined]\r\n#eval_tree_array#1 at C:\\Users\\runneradmin\\.julia\\packages\\SymbolicRegression\\OYvt5\\src\\InterfaceDynamicExpressions.jl:57 [inlined]\r\neval_tree_array at C:\\Users\\runneradmin\\.julia\\packages\\SymbolicRegression\\OYvt5\\src\\InterfaceDynamicExpressions.jl:56 [inlined]\r\n_eval_loss at C:\\Users\\runneradmin\\.julia\\packages\\SymbolicRegression\\OYvt5\\src\\LossFunctions.jl:48\r\n#eval_loss#3 at C:\\Users\\runneradmin\\.julia\\packages\\SymbolicRegression\\OYvt5\\src\\LossFunctions.jl:101\r\neval_loss at C:\\Users\\runneradmin\\.julia\\packages\\SymbolicRegression\\OYvt5\\src\\LossFunctions.jl:93 [inlined]\r\n#score_func#5 at C:\\Users\\runneradmin\\.julia\\packages\\SymbolicRegression\\OYvt5\\src\\LossFunctions.jl:160 [inlined]\r\nscore_func at C:\\Users\\runneradmin\\.julia\\packages\\SymbolicRegression\\OYvt5\\src\\LossFunctions.jl:157 [inlined]\r\n#next_generation#1 at C:\\Users\\runneradmin\\.julia\\packages\\SymbolicRegression\\OYvt5\\src\\Mutate.jl:235\r\nnext_generation at C:\\Users\\runneradmin\\.julia\\packages\\SymbolicRegression\\OYvt5\\src\\Mutate.jl:60 [inlined]\r\nreg_evol_cycle at C:\\Users\\runneradmin\\.julia\\packages\\SymbolicRegression\\OYvt5\\src\\RegularizedEvolution.jl:37\r\n#s_r_cycle#1 at C:\\Users\\runneradmin\\.julia\\packages\\SymbolicRegression\\OYvt5\\src\\SingleIteration.jl:42\r\ns_r_cycle at C:\\Users\\runneradmin\\.julia\\packages\\SymbolicRegression\\OYvt5\\src\\SingleIteration.jl:17 [inlined]\r\n#_dispatch_s_r_cycle#81 at C:\\Users\\runneradmin\\.julia\\packages\\SymbolicRegression\\OYvt5\\src\\SymbolicRegression.jl:1053\r\n_dispatch_s_r_cycle at C:\\Users\\runneradmin\\.julia\\packages\\SymbolicRegression\\OYvt5\\src\\SymbolicRegression.jl:1036\r\n```\r\n\r\nI found this quite odd as it seems as though both the Julia and Python garbage collection are interfering with eachother. Here, it seems as though `PyObject_Free` is trying to free memory that was already freed? Perhaps one of the GCs is trying to free the memory access by the other runtime. Looking at the backtrace, I suppose this could also be an issue with [PyCall.jl](https://github.com/JuliaPy/PyCall.jl) (which calls Python functions from Julia), although it hasn't occurred in any previous Python version, so I'm not sure where the issue is coming from.\r\n\r\nAny help is appreciated. I am happy to provide you with as much debugging information as I can, as this issue is quite urgent to fix in the ecosystem of Python \u003c-\u003e Julia packages.\r\n\r\n\r\n\r\n### CPython versions tested on:\r\n\r\n3.12\r\n\r\n### Operating systems tested on:\r\n\r\nLinux, macOS, Windows\r\n\r\n### Output from running 'python -VV' on the command line:\r\n\r\nLinux test performed on: `Python 3.12.1 (main, Dec 30 2023, 22:23:57) [GCC 8.5.0 20210514 (Red Hat 8.5.0-20)]`","author":{"url":"https://github.com/MilesCranmer","@type":"Person","name":"MilesCranmer"},"datePublished":"2023-12-31T00:11:35.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":7},"url":"https://github.com/113591/cpython/issues/113591"}