René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Reference counting bug with manually allocated heap types","articleBody":"# Bug report\n\nFound by @vfdev-5.\n\nThis is specific to the free threading build and 3.14.\n\nXLA/Jax uses the following code to create a heap type:\n\n```c++\n  // We need to use heap-allocated type objects because we want to add\n  // additional methods dynamically.\n...\n    nb::str name = nb::str(\"PmapFunction\");\n    nb::str qualname = nb::str(\"PmapFunction\");\n    PyHeapTypeObject* heap_type = reinterpret_cast\u003cPyHeapTypeObject*\u003e(\n        PyType_Type.tp_alloc(\u0026PyType_Type, 0));\n    // Caution: we must not call any functions that might invoke the GC until\n    // PyType_Ready() is called. Otherwise the GC might see a half-constructed\n    // type object.\n    CHECK(heap_type) \u003c\u003c \"Unable to create heap type object\";\n    heap_type-\u003eht_name = name.release().ptr();\n    heap_type-\u003eht_qualname = qualname.release().ptr();\n   ...\n```\n\nhttps://github.com/openxla/xla/blob/19a8e8e05fb34c5c4b8c38c9a8225e89f008c8c1/xla/python/pmap_lib.cc#L1027-L1058\n\nIn other words, the heap type is created by by calling `PyType_Type.tp_alloc` and filling in the fields, instead of the more common use of `PyType_FromSpec`. This leaves [`unique_id`](https://github.com/python/cpython/blob/211f41316b7f205d18eb65c1ececd7f7fb30b02d/Include/cpython/object.h#L284-L286) zero initialized. The problem is that `unique_id=0` currently looks like a valid unique id for per-thread reference counting, which leads to reference counting errors and use-after-frees.\n\nI think we should change the per-thread reference counting so that `unique_id=0` is the sentinel value indicating that it's not assigned instead of the current `unique_id=-1` convention.\n\n### Full repro\n\n* https://gist.github.com/vfdev-5/a2f8f0716611afe2e0721c4332bedcd5\n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-128925\n* gh-128951\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/colesbury","@type":"Person","name":"colesbury"},"datePublished":"2025-01-16T19:29:13.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":1},"url":"https://github.com/128923/cpython/issues/128923"}