René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Unify ways to run the Python test suite","articleBody":"There are at least 15 known ways to run the Python test suite. IMO it's too much, it's hard to keep all these code paths consistent. IMO ``python -m test`` should be enough to fit all use cases.\r\n\r\nMaybe we need a ``--ci``, ``--buildbot``, or ``--strict`` option which would run tests is \"strict\" mode.\r\n\r\n---\r\n\r\nPortable way to run tests:\r\n\r\n```\r\n./python -m test\r\n./python -m test.regrtest\r\n./python -m test.libregrtest\r\n./python -m test.autotest\r\n\r\n./python Lib/test/regrtest.py\r\n./python Lib/test/autotest.py\r\n\r\n./python -c 'from test import autotest'\r\n```\r\n\r\nThe main drawback is that running ``./python`` doesn't work when Python is built with ``./configure --enable-shared``, and it doesn't work when Python is cross-compiled (WASM/WASI buildbots). Moreover, on macOS, ``./python.exe`` must be run instead. On Windows, it's just ``python``.\r\n\r\nUnix, from Python source code:\r\n\r\n```\r\nmake buildbottest TESTOPTS=\"...\"\r\nmake test\r\nmake testall\r\nmake testuniversal\r\nmake hostrunnertest\r\n\r\n./python Tools/scripts/run_tests.py\r\n```\r\n\r\nWindows, from Python source code:\r\n\r\n```\r\nPCbuild\\rt.bat\r\nTools\\buildbot\\test.bat\r\n```\r\n\r\n---\r\n\r\n* Lib/test/autotest.py just runs ``test.libregrtest.main()``.\r\n* Lib/test/regrtest.py runs ``test.libregrtest.main()`` **and** remove ``Lib/test/`` from ``sys.path`` **and** make ``__file__`` an absolute path.\r\n* ``PCbuild\\rt.bat``:\r\n\r\n  * python: pass ``-u -Wd -E -bb`` options\r\n  * regrtest: pass options passed to ``rt.bat`` (except of options specific to rt.bat)\r\n  * most options are to get the path to the python.exe program in PCbuild/\r\n\r\n* ``Tools\\buildbot\\test.bat`` runs ``PCbuild\\rt.bat``:\r\n\r\n  * ``rt.bat``: pass ``-q -d`` options.\r\n  * regrtest: pass ``-j1 -uall -rwW --slowest --timeout=1200 --fail-env-changed`` options.\r\n  * ARM32 SSH pass ``-unetwork -udecimal -usubprocess -uurlfetch -utzdata -rwW --slowest --timeout=1200 --fail-env-changed`` options.\r\n\r\n* ``make buildbottest`` runs ``Tools/scripts/run_tests.py``:\r\n\r\n  * python: pass ``$(TESTPYTHONOPTS)`` options.\r\n  * regrtest: pass ``-j 1 -u all -W --slowest --fail-env-changed --timeout=$(TESTTIMEOUT) $(TESTOPTS)`` options.\r\n\r\n* ``Tools/scripts/run_tests.py``:\r\n\r\n  * python: pass ``-u -W default -bb -E`` options -- also cross-compilation options, not detailed here\r\n  * regrtest: pass ``-r -w -j0 -u all,-largefile,-audio,-gui`` options, add ``-n`` on Windows.\r\n\r\n---\r\n\r\nGitHub Action workflow uses, [.github/workflows/build.yml](https://github.com/python/cpython/blob/afa7b0d743d9b7172d58d1e4c28da8324b9be2eb/.github/workflows/build.yml):\r\n\r\n* Windows x86: ``.\\PCbuild\\rt.bat -p Win32 -d -q -uall -u-cpu -rwW --slowest --timeout=1200 -j0``\r\n* Windows x64: ``.\\PCbuild\\rt.bat -p x64 -d -q -uall -u-cpu -rwW --slowest --timeout=1200 -j0``\r\n* macOS: ``make buildbottest TESTOPTS=\"-j4 -uall,-cpu\"``\r\n* Ubuntu: ``xvfb-run make buildbottest TESTOPTS=\"-j4 -uall,-cpu\"``\r\n* Address Sanitizer: ``xvfb-run make buildbottest TESTOPTS=\"-j4 -uall,-cpu\"``\r\n\r\nBuildbot use, [master/custom/factories.py](https://github.com/python/buildmaster-config/blob/0fbab0b010c8d24145e304505463bdf562439d85/master/custom/factories.py):\r\n\r\n* UnixBuild and UnixCrossBuild: ``make buildbottest TESTOPTS=\"...\" TESTPYTHONOPTS=\"...\" TESTTIMEOUT=\"...\"`` with code to select TESTOPTS, TESTPYTHONOPTS and TESTTIMEOUT.\r\n* UnixInstalledBuild: ``/path/to/python -m test ...`` with code to select ``-m test`` options.\r\n* BaseWindowsBuild: ``Tools\\buildbot\\test.bat ...`` with code to select ``....`` options\n\n\u003c!-- gh-linked-prs --\u003e\n### Linked PRs\n* gh-109567\n* gh-109570\n* gh-109701\n* gh-109909\n* gh-109926\n* gh-109954\n* gh-109969\n* gh-110062\n* gh-110120\n* gh-110121\n* gh-110122\n\u003c!-- /gh-linked-prs --\u003e\n","author":{"url":"https://github.com/vstinner","@type":"Person","name":"vstinner"},"datePublished":"2023-09-19T10:28:13.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":1},"url":"https://github.com/109566/cpython/issues/109566"}