René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Improve the Efficiency of Python3.11.1 __getattr__","articleBody":"# Feature or enhancement\r\n\r\n`__getattr__` in Python3.11.1 is much slower than `@Property` and visiting an object's attribute. It's even slower than Python3.10.4.\r\n`_PyObject_GenericGetAttrWithDict` is the key reason. If Python3 fails finding an attribute in normal ways, it will return NULL and raise an exception. But raising an exception has performance cost. Python3.11.1 add set_attribute_error_context to support `Fine Grained Error Locations in Tracebacks`. It makes things worser.\r\n\r\n# Pitch\r\n\r\nWe can use this test code:\r\n```py\r\nimport time\r\nimport sys\r\n\r\nclass A:\r\n    def foo(self):\r\n        print(\"Call A.foo!\")\r\n\r\n    def __getattr__(self, name):\r\n        return 2\r\n\r\n    @property\r\n    def ppp(self):\r\n        return 3\r\n\r\nclass B(A):\r\n    def foo(self):\r\n        print(\"Call B.foo!\")\r\n\r\n\r\nclass C(B):\r\n\r\n    def __init__(self) -\u003e None:\r\n        self.pps = 1\r\n\r\n    def foo(self):\r\n        print(\"Call C.foo!\")\r\n\r\n\r\ndef main():\r\n    start = time.time()\r\n    for i in range(1, 1000000):\r\n        pass\r\n    end = time.time()\r\n    peer = end - start\r\n    c = C()\r\n    print(f\"Python version of {sys.version}\")\r\n    start = time.time()\r\n    for i in range(1, 1000000):\r\n        s = c.pps\r\n    end = time.time()\r\n    print(f\"Normal getattr spend time: {end - start - peer}\")\r\n    start = time.time()\r\n    for i in range(1, 1000000):\r\n        s = c.ppa\r\n    end = time.time()\r\n    print(f\"Call __getattr__ spend time: {end - start - peer}\")\r\n    start = time.time()\r\n    for i in range(1, 1000000):\r\n        s = c.ppp\r\n    end = time.time()\r\n    print(f\"Call property spend time: {end - start - peer}\")\r\n\r\nif __name__ == \"__main__\":\r\n    main()\r\n```\r\n\r\nThe result shows how slow `__getattr__` is:\r\n```\r\nPython version of 3.11.1 (main, Dec 26 2022, 16:32:50) [GCC 8.3.0]\r\nNormal getattr spend time: 0.03204226493835449\r\nCall __getattr__ spend time: 0.4767305850982666\r\nCall property spend time: 0.06345891952514648\r\n```\r\n\r\nWhen we define `__getattr__`, failed to find an attribute is what we expected. If we can get this result and then call `__getattr__` without exception handling, it will be faster.\r\nI tried to modify Python3.11.1 like this:\r\n1. add a new function in `object.c`:\r\n```c\r\nPyObject *\r\nPyObject_GenericTryGetAttr(PyObject *obj, PyObject *name)\r\n{\r\n    return _PyObject_GenericGetAttrWithDict(obj, name, NULL, 1);\r\n}\r\n```\r\n\r\n2. change `typeobject.c` :\r\n```c\r\nif (getattribute == NULL ||\r\n        (Py_IS_TYPE(getattribute, \u0026PyWrapperDescr_Type) \u0026\u0026\r\n         ((PyWrapperDescrObject *)getattribute)-\u003ed_wrapped ==\r\n         (void *)PyObject_GenericGetAttr))\r\n        // res = PyObject_GenericGetAttr(self, name);\r\n        res = PyObject_GenericTryGetAttr(self, name);\r\n    else {\r\n        Py_INCREF(getattribute);\r\n        res = call_attribute(self, getattribute, name);\r\n        Py_DECREF(getattribute);\r\n    }\r\n    if (res == NULL) {\r\n        if (PyErr_ExceptionMatches(PyExc_AttributeError))\r\n            PyErr_Clear();\r\n        res = call_attribute(self, getattr, name);\r\n    }\r\n    Py_DECREF(getattr);\r\n    return res;\r\n```\r\n\r\nRebuild python, it really become faster: `spend time: 0.13772845268249512.`\r\n\r\n# Previous discussion\r\n\r\n\u003c!--\r\n  New features to Python should first be discussed elsewhere before creating issues on GitHub,\r\n  for example in the \"ideas\" category (https://discuss.python.org/c/ideas/6) of discuss.python.org,\r\n  or the python-ideas mailing list (https://mail.python.org/mailman3/lists/python-ideas.python.org/).\r\n  Use this space to post links to the places where you have already discussed this feature proposal:\r\n--\u003e\r\n[\r\n__getattr__ is much slower in Python3.11](https://discuss.python.org/t/getattr-is-much-slower-in-python3-11/24028)\r\n\r\n\u003c!--\r\nYou can freely edit this text. Remove any lines you believe are unnecessary.\r\n--\u003e\r\n\n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-102248\n* gh-103332\n* gh-103761\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/wangxiang-hz","@type":"Person","name":"wangxiang-hz"},"datePublished":"2023-02-24T12:23:14.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":4},"url":"https://github.com/102213/cpython/issues/102213"}