René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"C API: Investigate how the PyTypeObject members can be removed from the public C API","articleBody":"I propose to investigate an incompatible C API change: make the ``PyTypeObject`` and ``PyHeapTypeObject`` structures opaque, remove their members from the public C API (move them to the internal C API). We have to investigate how it's outside in 3rd party C extensions (ex: Cython, pybind11, etc.), and design a smooth migration plan.\r\n\r\nThe PyTypeObject structure is exposed as part of the public Python C API. For example, ``Py_TYPE(obj)-\u003etp_name`` directly gets a type name (as a UTF-8 encoded byte string, ``char*``).\r\n\r\nThe PyTypeObject members are **NOT** part of the limited C API (PEP 384).\r\n\r\n---\r\n\r\nIn Python 3.9 (2020), I reworked the C API to avoid accessing directly PyTypeObject members at the ABI level: issue #84351. For example, Python 3.8 implements ``PyObject_IS_GC()`` as a macro:\r\n\r\n```c\r\n/* Test if an object has a GC head */\r\n#define PyObject_IS_GC(o) \\\r\n    (PyType_IS_GC(Py_TYPE(o)) \\\r\n     \u0026\u0026 (Py_TYPE(o)-\u003etp_is_gc == NULL || Py_TYPE(o)-\u003etp_is_gc(o)))\r\n```\r\n\r\nwhereas Python 3.9 only provides an opaque function call:\r\n\r\n```c\r\n/* Test if an object implements the garbage collector protocol */\r\nPyAPI_FUNC(int) PyObject_IS_GC(PyObject *obj);\r\n```\r\n\r\nAt the ABI level, the direct access to the ``PyTypeObject.tp_is_gc`` member became an opaque function call.\r\n\r\n---\r\n\r\nChanging PyTypeObject API and ABI caused a lot of troubles in the past. Example:\r\n\r\n* issue GH-81431: Cython broken by ``PyTypeObject.tp_print`` removal -- related to ``PyTypeObject.tp_vectorcall_offset`` addition\r\n* issue GH-76569: ABI issue with the addition of ``PyTypeObject.tp_finalize`` (PEP 442). Is ``Py_TPFLAGS_HAVE_FINALIZE`` needed for ABI compatibility? See also the [python-dev thread](https://mail.python.org/pipermail/python-dev/2017-December/151345.html).\r\n* issue #86913: remove ``Py_TPFLAGS_HAVE_VERSION_TAG`` flag (ABI compatibility)\r\n* issue GH-91271: Documentation inconsistency with the stable ABI (PyTypeObject)\r\n* The [PEP 620](https://peps.python.org/pep-0620/#the-c-api-blocks-cpython-evolutions) uses PyTypeObject incompatible changes in its rationale.\r\n\r\n---\r\n\r\nFor many years, there is a work-in-progress to convert all Python built-in types and types of stdlib extensions from static types to heap types. See for example issue #84258 and [PEP 630](https://peps.python.org/pep-0630/).\r\n\r\nThe API and ABI for heap type was also enhanced over the years. Examples:\r\n\r\n* Python 3.12 adds ``PyType_FromMetaclass()``\r\n* Add the buffer protocol to the limited C API (to PyType_FromSpec()): issue #89622\r\n* issue GH-86201: get tp_name with PyType_GetSlot()\r\n\r\n---\r\n\r\nIn the past, other structure members were removed:\r\n\r\n* PyInterpreterState: Python 3.8\r\n* PyGC_Head: Python 3.9\r\n* PyFrameObject: Python 3.11\r\n\r\nThe work was also prepared for:\r\n\r\n* PyObject: issue #83754, Python 3.10 and Python 3.11, add Py_SET_REFCNT() and Py_SET_TYPE()\r\n* PyTheadState: issue #84128, Python 3.11","author":{"url":"https://github.com/vstinner","@type":"Person","name":"vstinner"},"datePublished":"2023-06-21T14:32:28.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":8},"url":"https://github.com/105970/cpython/issues/105970"}