René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Inline dict/list/set comprehensions in the compiler for better performance","articleBody":"# Feature or enhancement\r\n\r\nIn [Cinder](https://github.com/facebookincubator/cinder) we inline some list/dict/set comprehensions in the compiler for better performance. That is, instead of creating a function object every time and calling it, for some comprehensions we just emit bytecode directly in the outer function to implement the comprehension loop.\r\n\r\n# Pitch\r\n\r\nThis change was a significant CPU efficiency improvement on the Instagram production web tier. Allocating a new Python function object is a significant and not strictly necessary cost.\r\n\r\nThere are some backward-compatibility considerations. In the Cinder implementation, we refuse to inline comprehensions if there is a name collision between a name assigned within the comprehension and a name in the containing scope, and we delete any names defined in the comprehension immediately after the inlined bytecode, to preserve the same visibility of names defined in the comprehension. There can still be some visible changes, e.g. if `locals()` is called within the comprehension it will show names from the containing function too. If a traceback occurs within the comprehension (or `sys._getframe` is called), it will not show an additional last frame for the comprehension itself. In practice we've not observed either of these to be a problem.\r\n\r\nI discussed this idea with @markshannon, and he suggested a couple possible improvements. Instead of refusing to inline comprehensions with name clashes, we could push the prior value of the colliding name(s) onto the stack before running the comprehension, and then pop it back into the name afterward. This would allow inlining ~all comprehensions. He also suggested that we could add `PUSH_FRAME` and `POP_FRAME` opcodes and wrap the inlined bytecode in these. Then we would still elide the cost of function object creation, but the comprehension bytecode (although part of the parent bytecode) would still run inside its own frame, eliminating the above-mentioned incompatibilities.\n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-101310\n* gh-101441\n* gh-105126\n* gh-105142\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/carljm","@type":"Person","name":"carljm"},"datePublished":"2022-10-05T20:36:58.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":14},"url":"https://github.com/97933/cpython/issues/97933"}