René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"`_remote_debugging`: returning the same samples over and over","articleBody":"# Bug report\n\n### Bug description:\n\n## Diagnosis\n\nNote the **consecutive identical** here:\n\n```bash\n2026-05-12T10:27:01.788483000+0200 maurycy@gimel /Users/maurycy/src/github.com/maurycy/cpython (main c6fd7de*?) % sudo ./python.exe -m profiling.sampling run -r 300khz --pstats -o /dev/null -m test\n\n[...]\n\nCaptured 31,970,196 samples in 851.80 seconds\nSample rate: 37,532.68 samples/sec (consecutive identical: 30,295,039/31,457,858)\nError rate: 1.60\nWarning: missed 251961921 samples from the expected total of 283932117 (88.74%)\n```\n\nThat's with:\n\n```diff\ndiff --git i/Lib/profiling/sampling/sample.py w/Lib/profiling/sampling/sample.py\nindex 5bbe2483581..41c6dec4f6d 100644\n--- i/Lib/profiling/sampling/sample.py\n+++ w/Lib/profiling/sampling/sample.py\n@@ -109,6 +109,8 @@ def sample(self, collector, duration_sec=None, *, async_aware=False):\n         last_sample_time = start_time\n         realtime_update_interval = 1.0  # Update every second\n         last_realtime_update = start_time\n+        prev_stack = None\n+        consecutive_identical = 0\n         try:\n             while duration_sec is None or running_time_sec \u003c duration_sec:\n                 # Check if live collector wants to stop\n@@ -125,6 +127,9 @@ def sample(self, collector, duration_sec=None, *, async_aware=False):\n                         stack_frames = self._get_stack_trace(\n                             async_aware=async_aware\n                         )\n+                        if stack_frames == prev_stack:\n+                            consecutive_identical += 1\n+                        prev_stack = stack_frames\n                         collector.collect(stack_frames)\n                     except ProcessLookupError as e:\n                         running_time_sec = current_time - start_time\n@@ -178,7 +183,9 @@ def sample(self, collector, duration_sec=None, *, async_aware=False):\n         if not is_live_mode:\n             s = \"\" if num_samples == 1 else \"s\"\n             print(f\"Captured {num_samples:n} sample{s} in {fmt(running_time_sec, 2)} seconds\")\n-            print(f\"Sample rate: {fmt(sample_rate, 2)} samples/sec\")\n+            comparable_samples = max(1, num_samples - errors - 1)\n+            print(f\"Sample rate: {fmt(sample_rate, 2)} samples/sec \"\n+                  f\"(consecutive identical: {consecutive_identical:n}/{comparable_samples:n})\")\n             print(f\"Error rate: {fmt(error_rate, 2)}\")\n \n             # Print unwinder stats if stats collection is enabled\n```\n\n## Discussion\n\nThis hints why RLE in the binary format is so efficient.\n\nI propose a Python-level improvement in #149719, leveraging the fact that `timestamps_us` in the `collect` is plural:\n\nhttps://github.com/python/cpython/blob/b546cc10f5c659344ce3cf49db6d9c92307ed1fc/Lib/profiling/sampling/collector.py#L147\n\nThis gives 3x improvement throughput for `--pstats` for example:\n\n```\n2026-05-12T11:08:55.568015000+0200 maurycy@gimel /Users/maurycy/src/github.com/maurycy/cpython (main c6fd7de*?) % sudo ./python.exe -m profiling.sampling run -r 300khz --pstats -o /dev/null -m test\n\n[...]\n\n463 tests OK.\n\nTotal duration: 14 min 46 sec\nTotal tests: run=49,788 failures=9 skipped=2,696\nTotal test files: run=494/505 failed=5 env_changed=1 skipped=25 resource_denied=11\nResult: FAILURE\nCaptured 100,236,311 samples in 886.43 seconds\nSample rate: 113,079.14 samples/sec (consecutive identical: 98,417,466/99,721,944)\nError rate: 0.51\nWarning: missed 195239075 samples from the expected total of 295475386 (66.08%)\n```\n\nOn the C-level, there'd be still millions of unnecessary allocations, though.\n\nI did not measure nor explored this yet, but quick shots go along the lines of returning a non-mutable objects from `get_stack_trace()` and friends. Maybe the ideal would be a common `StackTrace`, instead of a list. Another would be to add `prev_stack_trace` (so we don't make the `RemoteUnwinder` stateful) and perhaps return something like same-as-before. \n\nPerhaps we could already think in terms of batches and dedups in the `RemoteUnwinder` layer, instead of `Binary{Writer,Reader}`? cc @LalitMaganti\n\nMy intuition is that it starts to resemble `epoll()`, so my favourite idea is an `async` API where unwinder would wake up the profiler. We'd keep the current API inact, and be able to come up with future-proof types of wake up events to which the consumer could subscribe (eg: new, batch etc.)\n\nI believe that mining binary recordings would also reveal better shaped internal data structs.\n\nI'm not exactly sure about hinting the user so she'd decrease the sampling rate. There still might be new unique data among the sea of duplicates.\n\n### CPython versions tested on:\n\nCPython main branch\n\n### Operating systems tested on:\n\nmacOS\n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-149719\n* gh-149747\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/maurycy","@type":"Person","name":"maurycy"},"datePublished":"2026-05-12T10:52:29.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":3},"url":"https://github.com/149718/cpython/issues/149718"}