René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Defer environment installation if hook would be skipped","articleBody":"### Rationale\r\nWhen running `pre-commit run` or during `git commit`, all the virtual environments are initialized (fast) and dependencies for them are installed. The latter can be slow depending on many factors (number of dependencies, whether they are cached already, etc.).\r\n\r\npre-commit has logic to decide if certain hooks should be invoked at all and if there are no matching files the hook would be run on, it is \"(no files to check)Skipped\". However, even when this is the case, the hook's environment still would be installed.\r\n\r\n#### Possible implementation\r\n\r\nThis could be improved by leveraging the `Classifier` to check if there would be modified files at all just like in [`_run_single_hook`](https://github.com/pre-commit/pre-commit/blob/6740a1795c99f5af0afcd0b8b77a56f5f0d4e106/pre_commit/commands/run.py#L151), and if not, don't just skip _running_ the hook but also skip _installing its environment_.\r\n\r\n\r\n#### Our use case\r\n\r\n\u003cdetails\u003e\r\n\u003csummary\u003eClick for details....\u003c/summary\u003e\r\n\r\nMany monorepos, where hooks are possibly defined to run on `files` in certain `^subdirectory/.*$`s would greatly benefit from this. In our case, many C++ developers only work in C++-related subdirectories; however, although they would never touch python files, if another, python-related hook's dependencies change, the other hook's python virtualenv is recreated.\r\n\r\nCurrently, even with cached packages, this recreation (just installing packages with pip) takes 2-3 minutes on modern hardware, multiplied by the number of hooks whose `additional_dependencies` changed.  In our repository, one of these virtualenvs takes around 8+ minute to install (from already cached wheels). This is acceptable for the python developers (after all they are changing code which is expected to be checked), however not for the rest of the team.\r\n\r\n\u003c/details\u003e\r\n\r\n### Example\r\n\r\n```sh\r\n$ export XDG_CACHE_HOME=/tmp/.PRECOMMITCACHE/\r\n$ tree\r\n$ .\r\n├── project-with-arrow\r\n│   └── test.py\r\n└── project-with-hdfs\r\n    └── test.py\r\n$ cat .pre-commit-config.yaml\r\nrepos:\r\n  - repo: https://github.com/PyCQA/pylint\r\n    rev: v2.14.5\r\n    hooks:\r\n      - name: pylint-arrow\r\n        alias: pylint-arrow\r\n        id: pylint\r\n        additional_dependencies:\r\n          - arrow\r\n        files: ^project-with-arrow/.*\r\n        args: [\"--disable=W,C,R\"]\r\n      - name: pylint-hdfs\r\n        alias: pylint-hdfs\r\n        id: pylint\r\n        additional_dependencies:\r\n          - hdfs\r\n        files: ^project-with-hdfs/.*\r\n        args: [\"--disable=W,C,R\"]\r\n$ \r\n$ ####################\r\n$ # Current behavior #\r\n$ ####################\r\n$ pip install --force-reinstall pre-commit\r\n...\r\nSuccessfully installed ... pre-commit-2.20.0\r\n$ rm -rf $XDG_CACHE_HOME\r\n$ echo \"# Let's modify project-with-arrow!\" \u003e\u003e project-with-arrow/test.py\r\n$ git commit -a -m \"Modified arrow. During commit, both environment will be installed, but only one will be used\"\r\n[INFO] Initializing environment for https://github.com/PyCQA/pylint.\r\n[INFO] Initializing environment for https://github.com/PyCQA/pylint:arrow.\r\n[INFO] Initializing environment for https://github.com/PyCQA/pylint:hdfs.\r\n[INFO] Installing environment for https://github.com/PyCQA/pylint. # \u003c---------------- Two envs installed\r\n[INFO] Once installed this environment will be reused.\r\n[INFO] This may take a few minutes...\r\n[INFO] Installing environment for https://github.com/PyCQA/pylint. # \u003c---------------- Two envs installed\r\n[INFO] Once installed this environment will be reused.\r\n[INFO] This may take a few minutes...\r\npylint-arrow.............................................................Passed\r\npylint-hdfs..........................................(no files to check)Skipped  # \u003c-- One env used\r\n[master bd2d394] Modified arrow. During commit, both environment will be installed, but only one will be used\r\n 1 file changed, 1 insertion(+)\r\n$\r\n$ #####################\r\n$ # Proposed behavior #\r\n$ #####################\r\n$ git reset --hard HEAD~\r\n$ pip install --force-reinstall git+https://github.com/imc-trading/pre-commit.git@imc\r\n...\r\nSuccessfully installed ... pre-commit-2.20.1\r\n$ rm -rf $XDG_CACHE_HOME\r\n$ echo \"# Let's modify again project-with-arrow!\" \u003e\u003e project-with-arrow/test.py\r\n$ git commit -a -m \"Modified arrow. During commit, only one environment will be installed\"\r\n[INFO] Initializing environment for https://github.com/PyCQA/pylint.\r\n[INFO] Initializing environment for https://github.com/PyCQA/pylint:arrow.\r\n[INFO] Initializing environment for https://github.com/PyCQA/pylint:hdfs.\r\n[INFO] Installing environment for https://github.com/PyCQA/pylint. # \u003c---------------- Only one env installed\r\n[INFO] Once installed this environment will be reused.\r\n[INFO] This may take a few minutes...\r\npylint-arrow.............................................................Passed\r\npylint-hdfs..........................................(no files to check)Skipped\r\n[master ba4bdd5] Modified arrow. During commit, only one environment will be installed\r\n 1 file changed, 1 insertion(+)\r\n```\r\n\r\nA PoC quick\u0026dirty implementation is [available here](https://github.com/pre-commit/pre-commit/compare/main...imc-trading:pre-commit:imc).\r\n\r\n### Workarounds\r\n* When running manually `pre-commit run \u003chook\u003e`, only given hooks' environments are installed\r\n* specifying hooks in `SKIP`\r\n\r\n### Searched:\r\n* All currently open issues\r\n* subdirectory OR subfolder\r\n* defer\r\n* monorepo\r\n\r\n### Related:\r\n*  https://github.com/pre-commit/pre-commit/issues/2395\r\n    Good solution is to manually invoke `pre-commit run hook-id`, however during `git commit` it would still install unused environments\r\n* https://github.com/pre-commit/pre-commit/issues/1110\r\n    Confining pre-commit hooks to separate subdirectories can be done by either `cd`ing in a system hook, or simply running (aliased) hooks with a `files` include pattern of `subdirectory/.*`\r\n* https://github.com/pre-commit/pre-commit/pull/2480#issuecomment-1213047024 \r\n    `cd`ing out-of-the-box by pre-commit has already been discussed, and I also don't think it's a good idea","author":{"url":"https://github.com/kmARC","@type":"Person","name":"kmARC"},"datePublished":"2022-08-12T14:10:40.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":6},"url":"https://github.com/2481/pre-commit/issues/2481"}