René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"PEP 670: Convert _PyObject_SIZE() and _PyObject_VAR_SIZE() macros to functions","articleBody":"The _PyObject_SIZE() and _PyObject_VAR_SIZE() macros should be converted to functions: see [PEP 670](https://peps.python.org/pep-0670/) for the rationale.\r\n\r\nMy problem is that I don't know if the return type should be signed (Py_ssize_t) or unsigned (size_t).\r\n\r\nCPython usage of _PyObject_SIZE():\r\n\r\n* Signed: 18. Implementation of ``__sizeof__()`` methods.\r\n* Unsigned: 0\r\n*  Implicit cast to unsigned: 2. Calls ``PyObject_Malloc(_PyObject_SIZE(tp))`` and ``gc_alloc(_PyObject_SIZE(tp), presize)`` where the first argument type is ``size_t``.\r\n\r\nCPython usage of _PyObject_VAR_SIZE():\r\n\r\n* Signed: 5\r\n* Unsigned: 1\r\n* Implicit cast to unsigned: 1. Call ``_PyDebugAllocatorStats(..., _PyObject_VAR_SIZE(\u0026PyTuple_Type, len))`` where the argument type is ``size_t``.\r\n\r\nTo get a **container length**, the C API uses signed type (``Py_ssize_t``): PyList_Size(), PyDict_Size(), Py_SIZE(), etc.\r\n\r\nTo **allocate memory**, the C API prefers unsigned type (``size_t``): PyMem_Malloc(), PyObject_Realloc(), etc.\r\n\r\nPython allocator functions reject size greater than ``PY_SSIZE_T_MAX``:\r\n\r\n```\r\nvoid *\r\nPyMem_RawMalloc(size_t size)\r\n{\r\n    /*\r\n     * Limit ourselves to PY_SSIZE_T_MAX bytes to prevent security holes.\r\n     * Most python internals blindly use a signed Py_ssize_t to track\r\n     * things without checking for overflows or negatives.\r\n     * As size_t is unsigned, checking for size \u003c 0 is not required.\r\n     */\r\n    if (size \u003e (size_t)PY_SSIZE_T_MAX)\r\n        return NULL;\r\n    return _PyMem_Raw.malloc(_PyMem_Raw.ctx, size);\r\n}\r\n```\r\n\r\nSome \"sizeof\" functions freely mix signed and unsigned types. Example:\r\n\r\n```\r\nstatic PyObject *\r\ndeque_sizeof(dequeobject *deque, void *unused)\r\n{\r\n    Py_ssize_t res;\r\n    Py_ssize_t blocks;\r\n\r\n    res = _PyObject_SIZE(Py_TYPE(deque));\r\n    blocks = (size_t)(deque-\u003eleftindex + Py_SIZE(deque) + BLOCKLEN - 1) / BLOCKLEN;\r\n    assert(deque-\u003eleftindex + Py_SIZE(deque) - 1 ==\r\n           (blocks - 1) * BLOCKLEN + deque-\u003erightindex);\r\n    res += blocks * sizeof(block);\r\n    return PyLong_FromSsize_t(res);\r\n}\r\n```\r\n\r\n``blocks`` and ``sizeof(block)`` are unsigned, but ``res`` is signed.\r\n\r\n---\r\n\r\nAnother problem is that _PyObject_VAR_SIZE() has an undefined behavior on integer overflow. Maybe it should return ``SIZE_MAX`` on oveflow, to make sure that Python allocator function fail (return ``NULL``)?\n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-99846\n* gh-99847\n* gh-99848\n* gh-99850\n* gh-99903\n* gh-99922\n* gh-99924\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/vstinner","@type":"Person","name":"vstinner"},"datePublished":"2022-11-28T16:00:17.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":7},"url":"https://github.com/99845/cpython/issues/99845"}