Title: Regression tests do not support exclusion and pgo in the same invocation · Issue #135494 · python/cpython · GitHub
Open Graph Title: Regression tests do not support exclusion and pgo in the same invocation · Issue #135494 · python/cpython
X Title: Regression tests do not support exclusion and pgo in the same invocation · Issue #135494 · python/cpython
Description: Bug report Bug description: Background I've been working to build from source on Alpine. Starting in 3.13 pgo optimizations now error and that led to the test_re tests failing and blocking builds. I didn't want to fully disable pgo optim...
Open Graph Description: Bug report Bug description: Background I've been working to build from source on Alpine. Starting in 3.13 pgo optimizations now error and that led to the test_re tests failing and blocking builds. ...
X Description: Bug report Bug description: Background I've been working to build from source on Alpine. Starting in 3.13 pgo optimizations now error and that led to the test_re tests failing and blocking buil...
Opengraph URL: https://github.com/python/cpython/issues/135494
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Regression tests do not support exclusion and pgo in the same invocation","articleBody":"# Bug report\n\n### Bug description:\n\n## Background\n\nI've been working to build from source on Alpine. Starting in 3.13 pgo optimizations now error and that led to the test_re tests failing and blocking builds. I didn't want to fully disable pgo optimizations and miss out on all of the related performance wins.\n\n## Attempted Fix and Problem Identification\n\nI initially attempted to preclude this by leveraging `PROFILE_TASK=\"-m test --pgo -x test_re\"` when calling make. This had the impact of making all tests *except* the pgo tests run. After some spelunking I found that this logic from `Lib/test/libregrtest/main.py` is the culprit.\n\n```python\ndef find_tests(self, tests: TestList | None = None) -> tuple[TestTuple, TestList | None]:\n if tests is None:\n tests = []\n if self.single_test_run:\n self.next_single_filename = os.path.join(self.tmp_dir, 'pynexttest')\n try:\n with open(self.next_single_filename, 'r') as fp:\n next_test = fp.read().strip()\n tests = [next_test]\n except OSError:\n pass\n\n if self.fromfile:\n tests = []\n # regex to match 'test_builtin' in line:\n # '0:00:00 [ 4/400] test_builtin -- test_dict took 1 sec'\n regex = re.compile(r'\\btest_[a-zA-Z0-9_]+\\b')\n with open(os.path.join(os_helper.SAVEDCWD, self.fromfile)) as fp:\n for line in fp:\n line = line.split('#', 1)[0]\n line = line.strip()\n match = regex.search(line)\n if match is not None:\n tests.append(match.group())\n\n strip_py_suffix(tests)\n\n if self.pgo:\n # add default PGO tests if no tests are specified\n setup_pgo_tests(self.cmdline_args, self.pgo_extended)\n\n if self.tsan:\n setup_tsan_tests(self.cmdline_args)\n\n if self.tsan_parallel:\n setup_tsan_parallel_tests(self.cmdline_args)\n\n exclude_tests = set()\n if self.exclude:\n for arg in self.cmdline_args:\n exclude_tests.add(arg)\n self.cmdline_args = []\n\n alltests = findtests(testdir=self.test_dir,\n exclude=exclude_tests)\n\n if not self.fromfile:\n selected = tests or self.cmdline_args\n if selected:\n selected = split_test_packages(selected)\n else:\n selected = alltests\n else:\n selected = tests\n\n if self.single_test_run:\n selected = selected[:1]\n try:\n pos = alltests.index(selected[0])\n self.next_single_test = alltests[pos + 1]\n except IndexError:\n pass\n\n # Remove all the selected tests that precede start if it's set.\n if self.starting_test:\n try:\n del selected[:selected.index(self.starting_test)]\n except ValueError:\n print(f\"Cannot find starting test: {self.starting_test}\")\n sys.exit(1)\n\n random.seed(self.random_seed)\n if self.randomize:\n random.shuffle(selected)\n\n for priority_test in reversed(self.prioritize_tests):\n try:\n selected.remove(priority_test)\n except ValueError:\n print(f\"warning: --prioritize={priority_test} used\"\n f\" but test not actually selected\")\n continue\n else:\n selected.insert(0, priority_test)\n\n return (tuple(selected), tests)\n```\n\nDue to the order of operations what ends up happening is the pgo test suite gets added to `cmdline_args` and then they all get excluded leaving everything else to run.\n\n## Hacky Workaround\n\nMy eventual workaround is... less than great. I ultimately settled on two choices: explicitly maintain my own copy of the PGO test list in my dockerfile or update the list defined in the testing framework. I opted for the later and have added this to my build script.\n\n```bash\nsed -Ei \"s/'test_re',?$//g\" Lib/test/libregrtest/pgo.py\n```\n\nThis felt like the less likely option to break over time as we not only build 3.13 but other versions as well.\n\n## Suggested Improvement Options\n\n**1. Automatic Exclusion for Musl:** \nAutomatically exclude incompatible tests on musl-based systems, which appears consistent with other test behaviors. This minimizes friction for maintainers who otherwise may forgo PGO optimizations.\n\nFrom my view, this is the preferred option. \n\n**2. Improved Test Exclusion Logic:**\nAdjust `--pgo` and `-x` flag logic to allow intuitive exclusion of PGO tests, or otherwise enable this behavior, and combine with clearer documentation for builders.\n\n**3. Explicit Error Messaging:**\nExplicitly error when these flags are used together to reduce confusion and accidental behaviors. This aligns with other checks for mutually exclusive flags. \n\n\n### CPython versions tested on:\n\n3.13\n\n### Operating systems tested on:\n\nOther\n\n\u003c!-- gh-linked-prs -->\n### Linked PRs\n* gh-135713\n* gh-135880\n* gh-135881\n\u003c!-- /gh-linked-prs -->\n","author":{"url":"https://github.com/Kaelten","@type":"Person","name":"Kaelten"},"datePublished":"2025-06-13T22:52:58Z","url":"https://github.com/python/cpython/issues/135494"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:a14ee9fa-9693-afe9-b586-8ad33846b933 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | E324:19AA62:44E7059:5748C3F:6B023325 |
| html-safe-nonce | 0e1f466e2d260ed7bee90102378ce6faa0bfe8de33def1d2a4eec7633ad75b03 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJFMzI0OjE5QUE2Mjo0NEU3MDU5OjU3NDhDM0Y6NkIwMjMzMjUiLCJ2aXNpdG9yX2lkIjoiODA5OTYzMzU2OTgwNDIwMjc4OSIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | 16d886d259481e414c4858f8ddb96f52189c5306be7b0f29c45bed62fc980eae |
| hovercard-subject-tag | issue:3145003114 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/python/cpython/135494/issue_layout |
| twitter:image | https://opengraph.githubassets.com/f1e7f11d4a89b45cca175e6f0f981fede234d09cd1dfeae5a7ccf85c61e435e1/python/cpython/issues/135494 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/f1e7f11d4a89b45cca175e6f0f981fede234d09cd1dfeae5a7ccf85c61e435e1/python/cpython/issues/135494 |
| og:image:alt | Bug report Bug description: Background I've been working to build from source on Alpine. Starting in 3.13 pgo optimizations now error and that led to the test_re tests failing and blocking builds. ... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | Kaelten |
| hostname | github.com |
| expected-hostname | github.com |
| None | 87e774ee7c6b3bee0490394e4d3e57ec9dbc956f74b87ea685dc8996c1676456 |
| turbo-cache-control | no-preview |
| go-import | github.com/python/cpython git https://github.com/python/cpython.git |
| octolytics-dimension-user_id | 1525981 |
| octolytics-dimension-user_login | python |
| octolytics-dimension-repository_id | 81598961 |
| octolytics-dimension-repository_nwo | python/cpython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 81598961 |
| octolytics-dimension-repository_network_root_nwo | python/cpython |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 423dbb54c5e7bfecc838eb387c1435eb3d0e760d |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width