René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"GC in free-threaded build has problems with `Py_INCREF()`/`Py_DECREF()` in `tp_traverse` handlers","articleBody":"# Bug report\n\nThis came up in the context of [nanobind](https://github.com/wjakob/nanobind/). Nanobind implements (in a test) a traverse function:\n\n```c++\nint funcwrapper_tp_traverse(PyObject *self, visitproc visit, void *arg) {\n    FuncWrapper *w = nb::inst_ptr\u003cFuncWrapper\u003e(self);\n\n    nb::object f = nb::cast(w-\u003ef, nb::rv_policy::none);\n    Py_VISIT(f.ptr());\n\n    return 0;\n};\n```\n\nThe ``nb::object`` smart pointer is internally reference counted. In other words, the above is roughly equivalent to:\n\n```c\nint funcwrapper_tp_traverse(PyObject *self, visitproc visit, void *arg) {\n    PyObject *f = self-\u003ew-\u003ef;\n    Py_INCREF(f);\n    Py_VISIT(f);\n    Py_DECREF(f);\n    return 0;\n};\n```\n\nThis leads to a leak in the free-threaded GC for subtle reasons: when determining resurrected objects, the free-threaded GC uses `ob_ref_local` to compute the refcount - incoming references, which may be (temporarily) negative. In this case, `Py_INCREF()` adds 1 to the refcount, but by the time `Py_DECREF()` is called, the local refcount is `-1` which makes the object appear immortal.\n\nThere are a number of limitations on the implementations of traverse functions, which are not well documented. For example, it's not safe to allocate, free, track, or untrack Python objects. It's unclear to me whether there are other issues with calling refcounting functions in traverse callbacks.\n\nI think we can make [`handle_resurrected_objects`](https://github.com/python/cpython/blob/297f2e093ec95800ae2184330b8408c875523467/Python/gc_free_threading.c#L974) more robust to this by splitting the first pass over `state-\u003eunreachable` into two passes.\n\nSee also: https://github.com/PyO3/pyo3/issues/3165, which was not related to the free-threaded build.\n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-142232\n* gh-142271\n* gh-142272\n* gh-142422\n* gh-142423\n* gh-142567\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/colesbury","@type":"Person","name":"colesbury"},"datePublished":"2024-08-22T19:28:14.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":11},"url":"https://github.com/123241/cpython/issues/123241"}