René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"asyncio_websockets tests the implementation of `zlib`, not of Python","articleBody":"I found out that the `asyncio_websockets` [benchmark](https://github.com/python/pyperformance/blob/main/pyperformance/data-files/benchmarks/bm_asyncio_websockets/run_benchmark.py) spends ~87% of runtime in `zlib` (i.e. in the shared library `libz.so`), or whatever compression library is the default on the system-under-test.\n\nIn other words, `asyncio_websockets` tests the implementation of compression/decompression algorithms rather than anything to do with the Python interpreter or the `websockets` Python module. Websockets indeed enables compression by default: https://websockets.readthedocs.io/en/stable/topics/compression.html, excerpt from that official documentation:\n\u003e [connect()](https://websockets.readthedocs.io/en/stable/reference/asyncio/client.html#websockets.asyncio.client.connect) and [serve()](https://websockets.readthedocs.io/en/stable/reference/asyncio/server.html#websockets.asyncio.server.serve) enable compression by default because the reduction in network bandwidth is usually worth the additional memory and CPU cost.\n\n### Problem\n\nI believe this benchmark may not be measuring what's intended in its current form. For example, a replacement of `zlib` with `zlib-ng` or `zlib-rs` (which are newer drop-in replacement of `zlib`) may significantly affect the performance score of this benchmark, even though *nothing changed in Python and/or websockets implementations*. It is hard to root cause such performance modification, without knowing this detail about the `asyncio_websockets` benchmark.\n\nIt is also used in e.g. Phoronix testing, which may lead to unexpected conclusions for readers who aren't aware of this detail. Example: https://www.phoronix.com/review/cachyos-ubuntu-2510-f43/5.\n\n### Solutions\n\nI see the following solutions:\n1. Clearly document this behavior in https://pyperformance.readthedocs.io/benchmarks.html (in fact, there is no mention of websockets benchmark at all).\n2. Remove this benchmark, since the workload is dominated by native zlib rather than Python or websockets logic.\n3. Modify this benchmark to disable compression, as described here: https://websockets.readthedocs.io/en/stable/topics/compression.html#configuring-compression\n\nI'd lean towards option 3 (disabling compression) as it preserves the benchmark's intent while removing the zlib dependency from results. I'm happy to submit a PR if the maintainers agree.\n\n### Reproducing\n\nI ran it with a Amazon Linux 2023 docker container (OS distro similar to Fedora):\n```bash\ndocker run --rm -it amazonlinux:2023 /bin/bash # --\u003e\n\ndnf install -y pip perf dnf-utils\ndnf debuginfo-install zlib python3.9\npython3 -m pip install pyperformance\npython3 -m pip install websockets==11.0.3 pyperf==2.6.3\n\nperf record -g --call-graph dwarf -- \\\n  python3 -u /usr/local/lib/python3.9/site-packages/pyperformance/data-files/benchmarks/bm_asyncio_websockets/run_benchmark.py\n```\n\nAfter running the benchmark, we can examine the resulting `perf.data` file:\n```console\n$ perf report --hierarchy\n...\n-   99.93%        python3\n   -   87.22%        libz.so.1.2.11\n      +   40.11%        [.] inflate_fast\n      +   36.28%        [.] deflate_slow\n      ...\n   +    5.20%        libpython3.9.so.1.0\n   +    1.60%        libc.so.6\n   ...\n```\n\n---\n\nP.S. Thank you for maintaining the pyperformance project!","author":{"url":"https://github.com/dimakuv","@type":"Person","name":"dimakuv"},"datePublished":"2026-03-19T09:17:12.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/460/pyperformance/issues/460"}