René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Speed regression in the decimal module due to using heap types","articleBody":"This is essentially a rebirth of #114682, which was closed due to @skrah ban (unfortunately, such things usually don't fix issues).\n\nThe problem seems to be valid and I open a new issue per kindly @gpshead permission.  I labeled issue as \"type-feature\", but IMO it's looks rather as a regression, i.e. \"type-bug\".\n\nHere benchmarks results (with PGO) for 3.9-3.15:\n| Benchmark         | 39       | 310                    | 311                    | 312                    | 313                    | 314                    | 315                    |\n|-------------------|:--------:|:----------------------:|:----------------------:|:----------------------:|:----------------------:|:----------------------:|:----------------------:|\n| decimal_factorial | 804 ms   | 843 ms: 1.05x slower   | 763 ms: 1.05x faster   | 794 ms: 1.01x faster   | 1.02 sec: 1.27x slower | 950 ms: 1.18x slower   | 951 ms: 1.18x slower   |\n| decimal_pi        | 1.12 sec | 1.26 sec: 1.13x slower | 1.29 sec: 1.15x slower | 1.31 sec: 1.17x slower | 1.86 sec: 1.67x slower | 1.66 sec: 1.48x slower | 1.66 sec: 1.49x slower |\n| Geometric mean    | (ref)    | 1.09x slower           | 1.05x slower           | 1.07x slower           | 1.45x slower           | 1.32x slower           | 1.33x slower           |\n\nBenchmarks were available in the pyperformance package long time ago but are disabled, see https://github.com/python/pyperformance/pull/453 (I use *first* commit from above PR to enable *both* benchmarks.  Though, the \"decimal_factorial\" benchmark seems to be less affected, I guess because it has more large numbers).\n\nN.B.: the \"decimal_pi\" benchmark do computation with low precision (9 and 19).  Here is what happens with the default precision (28):\n| Benchmark  | 39       | 313                    |\n|------------|:--------:|:----------------------:|\n| decimal_pi | 1.20 sec | 1.99 sec: 1.65x slower |\n\nNot much better.\n\nIt seems that major slowdown come in the 3.13 (around 1.67x slower for \"decimal_pi\" on my system) with https://github.com/python/cpython/pull/106079.  Unfortunately, nobody asked to do performance measurements during PR review :(\n\nSomething like this happens already on toy extension types:\n| Benchmark           | ref    | heap                 |\n|---------------------|:------:|:--------------------:|\n| xyz(123) + xyz(321) | 204 ns | 357 ns: 1.75x slower |\n\n\u003cdetails\u003e\n\u003csummary\u003ecode and benchmark\u003c/summary\u003e\n\n```py\n# bench.py\nimport pyperf\nfrom operator import add\nfrom example import xyz\n\nx, y = map(xyz, [123, 321])\nrunner = pyperf.Runner()\ns = repr(x) + \" + \" + repr(y)\nrunner.bench_func(s, add, x, y)\n```\n\n```c\n/* static type */\n\n#define PY_SSIZE_T_CLEAN\n#include \u003cPython.h\u003e\n\ntypedef struct {\n    PyObject_HEAD\n    long value;\n} XYZ_Object;\n\nPyTypeObject XYZ_Type;\n#define XYZ_CheckExact(u) Py_IS_TYPE((u), \u0026XYZ_Type)\n\nstatic XYZ_Object *\nXYZ_new(long value)\n{\n    XYZ_Object *res = PyObject_New(XYZ_Object, \u0026XYZ_Type);\n\n    if (res) {\n        res-\u003evalue = value;\n    }\n    return res;\n}\n\nstatic PyObject *\nnew(PyTypeObject *type, PyObject *args, PyObject *keywds)\n{\n    Py_ssize_t argc = PyTuple_GET_SIZE(args);\n\n    if (argc == 1) {\n        PyObject *arg = PyTuple_GET_ITEM(args, 0);\n        long value = PyLong_AsLong(arg);\n\n        if (value == -1 \u0026\u0026 PyErr_Occurred()) {\n            return NULL;\n        }\n        return (PyObject *)XYZ_new(value);\n    }\n    PyErr_SetString(PyExc_TypeError, \"value required\");\n    return NULL;\n}\n\nstatic PyObject *\nadd(PyObject *self, PyObject *other)\n{\n    XYZ_Object *x = (XYZ_Object *)self;\n    XYZ_Object *y = (XYZ_Object *)other;\n\n    if (XYZ_CheckExact(x) \u0026\u0026 XYZ_CheckExact(y)) {\n        return (PyObject *)XYZ_new(x-\u003evalue + y-\u003evalue);\n    }\n    Py_RETURN_NOTIMPLEMENTED;\n}\n\nstatic PyObject *\nrepr(PyObject *self)\n{\n    return PyUnicode_FromFormat(\"xyz(%ld)\", ((XYZ_Object *)self)-\u003evalue);\n}\n\nstatic PyNumberMethods xyz_as_number = {\n    .nb_add = add,\n};\n\nPyTypeObject XYZ_Type = {\n    PyVarObject_HEAD_INIT(NULL, 0)\n    .tp_name = \"xyz\",\n    .tp_basicsize = sizeof(XYZ_Object),\n    .tp_new = new,\n    .tp_repr = repr,\n    .tp_as_number = \u0026xyz_as_number,\n    .tp_flags = Py_TPFLAGS_DEFAULT,\n};\n\nstatic int\nexample_exec(PyObject *module)\n{\n    if (PyModule_AddType(module, \u0026XYZ_Type) \u003c 0) {\n        return -1;\n    }\n    return 0;\n}\n\n#ifdef __GNUC__\n#  pragma GCC diagnostic push\n#  pragma GCC diagnostic ignored \"-Wpedantic\"\n#endif\nstatic PyModuleDef_Slot example_slots[] = {\n    {Py_mod_exec, example_exec},\n    {0, NULL}};\n#ifdef __GNUC__\n#  pragma GCC diagnostic pop\n#endif\n\nstatic struct PyModuleDef example_module = {\n    PyModuleDef_HEAD_INIT,\n    .m_name = \"example\",\n    .m_doc = \"Test module.\",\n    .m_size = 0,\n    .m_slots = example_slots,\n};\n\nPyMODINIT_FUNC\nPyInit_example(void)\n{\n    return PyModuleDef_Init(\u0026example_module);\n}\n```\n\n```c\n/* heap type */\n\n#define PY_SSIZE_T_CLEAN\n#include \u003cPython.h\u003e\n\ntypedef struct {\n    PyObject_HEAD\n    long value;\n} XYZ_Object;\n\ntypedef struct {\n    PyTypeObject *XYZ_Type;\n} example_state;\n\n#define XYZ_CheckExact(st, u) Py_IS_TYPE((u), (st)-\u003eXYZ_Type)\n\nstatic XYZ_Object *\nXYZ_new(example_state *Py_UNUSED(state), PyTypeObject *type, long value)\n{\n    XYZ_Object *res = PyObject_GC_New(XYZ_Object, type);\n\n    if (res) {\n        PyObject_GC_Track((PyObject *)res);\n        res-\u003evalue = value;\n    }\n    return res;\n}\n\nstatic struct PyModuleDef example_module;\n\nstatic example_state *\nget_state(PyTypeObject *type)\n{\n    PyObject *module = PyType_GetModuleByDef(type, \u0026example_module);\n\n    return PyModule_GetState(module);\n}\n\nstatic PyObject *\nnew(PyTypeObject *type, PyObject *args, PyObject *keywds)\n{\n    Py_ssize_t argc = PyTuple_GET_SIZE(args);\n    example_state *state = get_state(type);\n\n    if (argc == 1) {\n        PyObject *arg = PyTuple_GET_ITEM(args, 0);\n        long value = PyLong_AsLong(arg);\n\n        if (value == -1 \u0026\u0026 PyErr_Occurred()) {\n            return NULL;\n        }\n        return (PyObject *)XYZ_new(state, type, value);\n    }\n    PyErr_SetString(PyExc_TypeError, \"value required\");\n    return NULL;\n}\n\nstatic int\ntraverse(PyObject *self, visitproc visit, void *arg)\n{\n    Py_VISIT(Py_TYPE(self));\n    return 0;\n}\n\nstatic void\ndealloc(PyObject *self)\n{\n    PyTypeObject *type = Py_TYPE(self);\n\n    PyObject_GC_UnTrack(self);\n    type-\u003etp_free(self);\n    Py_DECREF(type);\n}\n\nstatic inline example_state *\nfind_state_left_or_right(PyObject *left, PyObject *right)\n{\n    PyObject *module = PyType_GetModuleByDef(Py_TYPE(left), \u0026example_module);\n\n    if (module) {\n        return PyModule_GetState(module);\n    }\n    PyErr_Clear();\n    return PyType_GetModuleState(Py_TYPE(right));\n}\n\nstatic PyObject *\nadd(PyObject *self, PyObject *other)\n{\n    example_state *state = find_state_left_or_right(self, other);\n    XYZ_Object *x = (XYZ_Object *)self;\n    XYZ_Object *y = (XYZ_Object *)other;\n\n    if (XYZ_CheckExact(state, x) \u0026\u0026 XYZ_CheckExact(state, y)) {\n        return (PyObject *)XYZ_new(state, state-\u003eXYZ_Type, x-\u003evalue + y-\u003evalue);\n    }\n    Py_RETURN_NOTIMPLEMENTED;\n}\n\nstatic PyObject *\nrepr(PyObject *self)\n{\n    return PyUnicode_FromFormat(\"xyz(%ld)\", ((XYZ_Object *)self)-\u003evalue);\n}\n\n#ifdef __GNUC__\n#  pragma GCC diagnostic push\n#  pragma GCC diagnostic ignored \"-Wpedantic\"\n#endif\nstatic PyType_Slot xyz_slots[] = {\n    {Py_tp_repr, repr},\n    {Py_tp_new, new},\n    {Py_nb_add, add},\n    {Py_tp_traverse, traverse},\n    {Py_tp_dealloc, dealloc},\n};\n#ifdef __GNUC__\n#  pragma GCC diagnostic pop\n#endif\n\nstatic PyType_Spec xyz_spec = {\n    .name = \"xyz\",\n    .basicsize = sizeof(XYZ_Object),\n    .flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC | Py_TPFLAGS_IMMUTABLETYPE,\n    .slots = xyz_slots,\n};\n\nstatic int\nexample_exec(PyObject *module)\n{\n    example_state *state = PyModule_GetState(module);\n\n    state-\u003eXYZ_Type = (PyTypeObject *)PyType_FromModuleAndSpec(module,\n                                                               \u0026xyz_spec,\n                                                               NULL);\n    if (!state-\u003eXYZ_Type || PyModule_AddType(module, state-\u003eXYZ_Type) \u003c 0) {\n        return -1;\n    }\n    return 0;\n}\n\nstatic int\nexample_clear(PyObject *module)\n{\n    example_state *state = PyModule_GetState(module);\n\n    Py_CLEAR(state-\u003eXYZ_Type);\n    return 0;\n}\n\nstatic int\nexample_traverse(PyObject *module, visitproc visit, void *arg)\n{\n    example_state *state = PyModule_GetState(module);\n\n    Py_VISIT(state-\u003eXYZ_Type);\n    return 0;\n}\n\n#ifdef __GNUC__\n#  pragma GCC diagnostic push\n#  pragma GCC diagnostic ignored \"-Wpedantic\"\n#endif\nstatic PyModuleDef_Slot example_slots[] = {\n    {Py_mod_exec, example_exec},\n    {0, NULL}};\n#ifdef __GNUC__\n#  pragma GCC diagnostic pop\n#endif\n\nstatic struct PyModuleDef example_module = {\n    PyModuleDef_HEAD_INIT,\n    .m_name = \"example\",\n    .m_doc = \"Test module.\",\n    .m_size = 0,\n    .m_slots = example_slots,\n    .m_clear = example_clear,\n    .m_traverse = example_traverse,\n};\n\nPyMODINIT_FUNC\nPyInit_example(void)\n{\n    return PyModuleDef_Init(\u0026example_module);\n}\n```\n\n\u003c/details\u003e\n\nSo, what we could do?  Can heap types (at least immutable) be less costly c.f. static types?  Can we convert the decimal module back to static types?\n\nI tried to add a freelist for Decimal's (quick patch is there: https://github.com/skirpichev/cpython/pull/16).  It looks this will mitigate the problem, but not entirely fix regression.\n","author":{"url":"https://github.com/skirpichev","@type":"Person","name":"skirpichev"},"datePublished":"2026-02-10T07:29:14.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":7},"url":"https://github.com/144650/cpython/issues/144650"}