René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Testing third-party stubs in isolated environments","articleBody":"At the moment, whenever there is change to typeshed, we test all stdlib and third-party packages in the same testing environment, using the `--custom-typeshed-dir` so that all packages can see each other. There are several practical problems with that:\r\n\r\n* We can't test the `requires` fields for correctness; all packages are always available.\r\n* We can't have packages that depend on external packages, see for example #5768, #5618, #5847, and the discussion in #5769 (all related to `cryptography`). (Except if we'd just install all non-types dependencies for all package into our venv.)\r\n* It doesn't scale if we increase the number of third-party packages in typeshed.\r\n* Ideas like per-package configuration is not possible at the moment, see for example #1526.\r\n\r\nMy suggestion: Only run the tests for the third-party stubs that have actually changed, and run each test in a separate environment. This environment only has the requirements from `METADATA.toml` installed. This fixes the problems above and also means:\r\n\r\n* The stubs will not be able to use features from stdlib that are not yet in the latest type checker distributions.\r\n* Changes that depend on changes in another third-party stub need to wait until that change has been released. (Happens every three hours, so it shouldn't be too much of a problem.)\r\n* Running the tests for all stubs will be slower, since each package needs its own venv installation.","author":{"url":"https://github.com/srittau","@type":"Person","name":"srittau"},"datePublished":"2021-08-23T14:33:11.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":9},"url":"https://github.com/5952/typeshed/issues/5952"}