René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Reorder some fields to facilitate out-of-process inspection","articleBody":"Some of the relevant fields in the interpreter state and the frame state in 3.12 are **very** challenging to fetch from out of process tools because they are in offsets that depend on compilation or platform variables that are different in different platforms. Not only that but they require the tools to copy **a huge** amount of intermediate structures making the whole thing very verbose.\r\n\r\nAs an example, this is the list of stuff that needs to be copied so out-of-process debuggers can fetch the interpreters, runtime state and thread list:\r\n\r\nhttps://gist.github.com/godlygeek/271951b20bb4c3783c2dd7c80908b116\r\n\r\nWith a simple reordering, that would shrink this to\r\n\r\n https://gist.github.com/godlygeek/341ce879a638c0fece9d0081d63e5ad9\r\n\r\nFor the interpreter state is also quite bad. Here is the things that need to be copied:\r\n\r\nhttps://gist.github.com/godlygeek/2468ff3d0f648a1aca7a8305bad7f825\r\n\r\nNot only that, but this depends on the compile-time value of all of this:\r\n\r\n```\r\n int PYSTACK_SIZEOF_VOID_P = sizeof(void*);\r\n int ALIGNMENT = PYSTACK_SIZEOF_VOID_P \u003e 4 ? 16 : 8;\r\n int SMALL_REQUEST_THRESHOLD = 512;\r\n int NB_SMALL_SIZE_CLASSES   = (SMALL_REQUEST_THRESHOLD / ALIGNMENT);\r\n int OBMALLOC_USED_POOLS_SIZE = (2 * ((NB_SMALL_SIZE_CLASSES + 7) / 8) * 8);\r\n\r\n int USE_LARGE_ARENAS = PYSTACK_SIZEOF_VOID_P \u003e 4;\r\n int ARENA_BITS = USE_LARGE_ARENAS ? 20 : 18;\r\n int ARENA_SIZE = 1 \u003c\u003c ARENA_BITS;\r\n int ARENA_SIZE_MASK = ARENA_SIZE - 1;\r\n\r\n int POINTER_BITS = 8 * PYSTACK_SIZEOF_VOID_P;\r\n int IGNORE_BITS = 0;\r\n int USE_INTERIOR_NODES = PYSTACK_SIZEOF_VOID_P \u003e 4;\r\n int ADDRESS_BITS = (POINTER_BITS - IGNORE_BITS);\r\n int INTERIOR_BITS = USE_INTERIOR_NODES ? ((ADDRESS_BITS - ARENA_BITS + 2) / 3) : 0;\r\n\r\n int MAP_TOP_BITS = INTERIOR_BITS;\r\n int MAP_TOP_LENGTH = (1 \u003c\u003c MAP_TOP_BITS);\r\n int MAP_TOP_MASK = (MAP_TOP_LENGTH - 1);\r\n\r\n int MAP_MID_BITS = INTERIOR_BITS;\r\n int MAP_MID_LENGTH = (1 \u003c\u003c MAP_MID_BITS);\r\n\r\n int MAP_BOT_BITS = (ADDRESS_BITS - ARENA_BITS - 2 * INTERIOR_BITS);\r\n int MAP_BOT_LENGTH = (1 \u003c\u003c MAP_BOT_BITS);\r\n\r\n int WITH_PYMALLOC_RADIX_TREE = 1;\r\n int USE_LARGE_POOLS = USE_LARGE_ARENAS ? WITH_PYMALLOC_RADIX_TREE : 0;\r\n int POOL_BITS = USE_LARGE_POOLS ? 14 : 12;\r\n int POOL_SIZE = (1 \u003c\u003c POOL_BITS);\r\n int MAX_POOLS_IN_ARENA = (ARENA_SIZE / POOL_SIZE);\r\n```\r\n\r\nIf the user changes any of these (like `WITH_PYMALLOC_RADIX_TREE`) when compiling Python, then the tools won't be able to work correctly.\r\n\r\nWe can easily reorder these two structures because they are not in the hot path of anything (unlike frames and code objects).\n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-106143\n* gh-106147\n* gh-106148\n* gh-106155\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/pablogsal","@type":"Person","name":"pablogsal"},"datePublished":"2023-06-27T14:12:03.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":3},"url":"https://github.com/106140/cpython/issues/106140"}