René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"GitCommand missing stderr","articleBody":"With GitPython 3.1.14 `GitCommandError`, for failed clones would pass on the captured stderr output into its `stderr` attribute. With 3.1.15 this contains the empty string instead.\r\n\r\nThis was caught by this unit test: https://github.com/tomtom-international/hopic/blob/9a35520388f8109ccff6a89407bc2429ed0d0557/hopic/test/test_checkout.py#L84-L103\r\n\r\nA reduced test that reduces this to _only_ testing GitPython (instead of the full integration with hopic) looks like the code below. That exhibits the exact same problem:\r\n\r\n```python\r\nimport git\r\nimport pytest\r\n\r\ndef test_checkout_in_non_empty_dir(tmp_path):\r\n    orig_repo = tmp_path / \"orig\"\r\n    with git.Repo.init(orig_repo, expand_vars=False) as repo:\r\n        repo.index.commit(message='Initial commit', **_commitargs)\r\n\r\n    non_empty_dir = tmp_path / 'non-empty-clone'\r\n    non_empty_dir.mkdir(parents=True)\r\n    garbage_file = non_empty_dir / 'not-empty'\r\n    garbage_file.write_text('Garbage!')\r\n\r\n    # Verify that the clone fails complaining about the target directory not being empty/non-existent\r\n    with pytest.raises(git.GitCommandError, match=r'(?is).*\\bfatal:\\s+destination\\s+path\\b.*\\bexists\\b.*\\bnot\\b.*\\bempty\\s+directory\\b'):\r\n        git.Repo.clone_from(orig_repo, non_empty_dir)\r\n    assert garbage_file.exists()\r\n```\r\n\r\nWith 3.1.14 this passes. With the `pytest.raises` expectation removed this output is displayed:\r\n```console\r\n___________________________________________________________________ test_checkout_in_non_empty_dir ____________________________________________________________________\r\n\r\ntmp_path = PosixPath('/tmp/pytest-of-vanschig/pytest-334/test_checkout_in_non_empty_dir0')\r\n\r\n    def test_checkout_in_non_empty_dir(tmp_path):\r\n        orig_repo = tmp_path / \"orig\"\r\n        with git.Repo.init(orig_repo, expand_vars=False) as repo:\r\n            repo.index.commit(message='Initial commit', **_commitargs)\r\n\r\n        non_empty_dir = tmp_path / 'non-empty-clone'\r\n        non_empty_dir.mkdir(parents=True)\r\n        garbage_file = non_empty_dir / 'not-empty'\r\n        garbage_file.write_text('Garbage!')\r\n\r\n        # Verify that the clone fails complaining about the target directory not being empty/non-existent\r\n\u003e       git.Repo.clone_from(orig_repo, non_empty_dir)\r\n\r\nhopic/test/test_checkout.py:95:\r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _\r\nv3.8/lib/python3.8/site-packages/git/repo/base.py:1032: in clone_from\r\n    return cls._clone(git, url, to_path, GitCmdObjectDB, progress, multi_options, **kwargs)\r\nv3.8/lib/python3.8/site-packages/git/repo/base.py:973: in _clone\r\n    finalize_process(proc, stderr=stderr)\r\nv3.8/lib/python3.8/site-packages/git/util.py:329: in finalize_process\r\n    proc.wait(**kwargs)\r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _\r\n\r\nself = \u003cgit.cmd.Git.AutoInterrupt object at 0x7f6a5883f9a0\u003e\r\nstderr = b\"fatal: destination path '/tmp/pytest-of-vanschig/pytest-334/test_checkout_in_non_empty_dir0/non-empty-clone' already exists and is not an empty directory.\\n\"\r\n\r\n    def wait(self, stderr=b''):  # TODO: Bad choice to mimic `proc.wait()` but with different args.\r\n        \"\"\"Wait for the process and return its status code.\r\n\r\n        :param stderr: Previously read value of stderr, in case stderr is already closed.\r\n        :warn: may deadlock if output or error pipes are used and not handled separately.\r\n        :raise GitCommandError: if the return status is not 0\"\"\"\r\n        if stderr is None:\r\n            stderr = b''\r\n        stderr = force_bytes(data=stderr, encoding='utf-8')\r\n\r\n        status = self.proc.wait()\r\n\r\n        def read_all_from_possibly_closed_stream(stream):\r\n            try:\r\n                return stderr + force_bytes(stream.read())\r\n            except ValueError:\r\n                return stderr or b''\r\n\r\n        if status != 0:\r\n            errstr = read_all_from_possibly_closed_stream(self.proc.stderr)\r\n            log.debug('AutoInterrupt wait stderr: %r' % (errstr,))\r\n\u003e           raise GitCommandError(self.args, status, errstr)\r\nE           git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)\r\nE             cmdline: git clone -v /tmp/pytest-of-vanschig/pytest-334/test_checkout_in_non_empty_dir0/orig /tmp/pytest-of-vanschig/pytest-334/test_checkout_in_non_empty_dir0/non-empty-clone\r\nE             stderr: 'fatal: destination path '/tmp/pytest-of-vanschig/pytest-334/test_checkout_in_non_empty_dir0/non-empty-clone' already exists and is not an empty directory.\r\nE           '\r\n\r\nv3.8/lib/python3.8/site-packages/git/cmd.py:408: GitCommandError\r\n```\r\n\r\nWith 3.1.15 this output is produced instead (notice the absence of the \"stderr: ...\" line):\r\n```console\r\n___________________________________________________________________ test_checkout_in_non_empty_dir ____________________________________________________________________\r\n\r\ntmp_path = PosixPath('/tmp/pytest-of-vanschig/pytest-336/test_checkout_in_non_empty_dir0')\r\n\r\n    def test_checkout_in_non_empty_dir(tmp_path):\r\n        orig_repo = tmp_path / \"orig\"\r\n        with git.Repo.init(orig_repo, expand_vars=False) as repo:\r\n            repo.index.commit(message='Initial commit', **_commitargs)\r\n\r\n        non_empty_dir = tmp_path / 'non-empty-clone'\r\n        non_empty_dir.mkdir(parents=True)\r\n        garbage_file = non_empty_dir / 'not-empty'\r\n        garbage_file.write_text('Garbage!')\r\n\r\n        # Verify that the clone fails complaining about the target directory not being empty/non-existent\r\n\u003e       git.Repo.clone_from(orig_repo, non_empty_dir)\r\n\r\nhopic/test/test_checkout.py:95:\r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _\r\nv3.8/lib/python3.8/site-packages/git/repo/base.py:1087: in clone_from\r\n    return cls._clone(git, url, to_path, GitCmdObjectDB, progress, multi_options, **kwargs)\r\nv3.8/lib/python3.8/site-packages/git/repo/base.py:1017: in _clone\r\n    handle_process_output(proc, None, progress_checked.new_message_handler(),\r\nv3.8/lib/python3.8/site-packages/git/cmd.py:116: in handle_process_output\r\n    return finalizer(process)\r\nv3.8/lib/python3.8/site-packages/git/util.py:354: in finalize_process\r\n    proc.wait(**kwargs)\r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _\r\n\r\nself = \u003cgit.cmd.Git.AutoInterrupt object at 0x7fcc8fd95820\u003e, stderr = b''\r\n\r\n    def wait(self, stderr=b''):  # TODO: Bad choice to mimic `proc.wait()` but with different args.\r\n        \"\"\"Wait for the process and return its status code.\r\n\r\n        :param stderr: Previously read value of stderr, in case stderr is already closed.\r\n        :warn: may deadlock if output or error pipes are used and not handled separately.\r\n        :raise GitCommandError: if the return status is not 0\"\"\"\r\n        if stderr is None:\r\n            stderr = b''\r\n        stderr = force_bytes(data=stderr, encoding='utf-8')\r\n\r\n        status = self.proc.wait()\r\n\r\n        def read_all_from_possibly_closed_stream(stream):\r\n            try:\r\n                return stderr + force_bytes(stream.read())\r\n            except ValueError:\r\n                return stderr or b''\r\n\r\n        if status != 0:\r\n            errstr = read_all_from_possibly_closed_stream(self.proc.stderr)\r\n            log.debug('AutoInterrupt wait stderr: %r' % (errstr,))\r\n\u003e           raise GitCommandError(remove_password_if_present(self.args), status, errstr)\r\nE           git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)\r\nE             cmdline: git clone -v --progress /tmp/pytest-of-vanschig/pytest-336/test_checkout_in_non_empty_dir0/orig /tmp/pytest-of-vanschig/pytest-336/test_checkout_in_non_empty_dir0/non-empty-clone\r\n\r\nv3.8/lib/python3.8/site-packages/git/cmd.py:409: GitCommandError\r\n```\r\n\r\nWhen catching the exception and dumping it's `stderr` attribute it turns out to be an empty string (`''`).\r\n\r\nThis seems related to #1220. But it's presentation is different as that issue still reports having content, just wrongly encoded.","author":{"url":"https://github.com/muggenhor","@type":"Person","name":"muggenhor"},"datePublished":"2021-04-21T15:00:37.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":4},"url":"https://github.com/1221/GitPython/issues/1221"}