René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Fuzz Tests Are Crashing at Start-up on ClusterFuzz","articleBody":"PR #1901 was successful in fixing the broken build but resurfaced another issue that is preventing the fuzzer from running.\r\n\r\n## The Problem\r\n\r\nIt appears that a Git executable is not available in the ClusterFuzz container environment where fuzz tests are executed, causing an error in the fuzz harnesses when GitPython attempts to initialize.\r\n\r\n This issue has been previously seen and reported on the OSS-Fuzz issue tracker: https://github.com/google/oss-fuzz/issues/10600\r\n\r\n\r\n\u003cdetails\u003e\u003csummary\u003eRelevant Portion of the ClusterFuzz Crash Logs from 2024-04-20\u003c/summary\u003e\r\n\r\n```shell\r\n# \u003cPrior output omitted for brevity\u003e\r\nINFO: Instrumenting git.index.util\r\nINFO: Instrumenting git.remote\r\nINFO: Instrumenting [git.repo.fun](http://git.repo.fun/)\r\nTraceback (most recent call last):\r\n  File \"git/__init__.py\", line 296, in \u003cmodule\u003e\r\n  File \"git/__init__.py\", line 287, in refresh\r\n  File \"git/cmd.py\", line 631, in refresh\r\nImportError: Bad git executable.\r\nThe git executable must be specified in one of the following ways:\r\n    - be included in your $PATH\r\n    - be set via $GIT_PYTHON_GIT_EXECUTABLE\r\n    - explicitly set via git.refresh(\u003cfull-path-to-git-executable\u003e)\r\n\r\nAll git commands will error until this is rectified.\r\n\r\nThis initial message can be silenced or aggravated in the future by setting the\r\n$GIT_PYTHON_REFRESH environment variable. Use one of the following values:\r\n    - quiet|q|silence|s|silent|none|n|0: for no message or exception\r\n    - warn|w|warning|log|l|1: for a warning message (logging level CRITICAL, displayed by default)\r\n    - error|e|exception|raise|r|2: for a raised exception\r\n\r\nExample:\r\n    export GIT_PYTHON_REFRESH=quiet\r\n\r\n\r\nThe above exception was the direct cause of the following exception:\r\n\r\nTraceback (most recent call last):\r\n  File \"[fuzz_config.py](http://fuzz_config.py/)\", line 26, in \u003cmodule\u003e\r\n  File \"PyInstaller/loader/pyimod02_importers.py\", line 419, in exec_module\r\n  File \"git/__init__.py\", line 298, in \u003cmodule\u003e\r\nImportError: Failed to initialize: Bad git executable.\r\nThe git executable must be specified in one of the following ways:\r\n    - be included in your $PATH\r\n    - be set via $GIT_PYTHON_GIT_EXECUTABLE\r\n    - explicitly set via git.refresh(\u003cfull-path-to-git-executable\u003e)\r\n\r\nAll git commands will error until this is rectified.\r\n\r\nThis initial message can be silenced or aggravated in the future by setting the\r\n$GIT_PYTHON_REFRESH environment variable. Use one of the following values:\r\n    - quiet|q|silence|s|silent|none|n|0: for no message or exception\r\n    - warn|w|warning|log|l|1: for a warning message (logging level CRITICAL, displayed by default)\r\n    - error|e|exception|raise|r|2: for a raised exception\r\n\r\nExample:\r\n    export GIT_PYTHON_REFRESH=quiet\r\n\r\n[80625] Failed to execute script 'fuzz_config' due to unhandled exception!\r\ncf::fuzzing_strategies: fork:2,value_profile:1\r\n```\r\n\r\n\u003c/details\u003e \r\n\r\n\r\n## Possible Solution\r\n\r\nOSS-Fuzz uses Pyinstaller to bundle fuzz harnesses and their dependencies in the `compile_python_fuzzers` function called by `build.sh`. Arguments passed to `compile_python_fuzzers` after the fuzz harness are forwarded to Pyinstaller, which accepts an [`--add-binary`](https://pyinstaller.org/en/stable/usage.html#what-to-bundle-where-to-search) flag to add arbitrary binaries to the bundle and [are made available to the bundled program at runtime](https://pyinstaller.org/en/stable/runtime-information.html#placing-data-files-at-expected-locations-inside-the-bundle).\r\n\r\nWe should be able to:\r\n1. ~Download a pre built Git binary from [kernal.org](https://mirrors.edge.kernel.org/pub/software/scm/git/) in the `container-environment-bootstrap.sh` script.~ Never mind, the downloadable archives are source, not builds.\r\n2. Bundle the `git` available in the OSS-Fuzz build container with the fuzz harness in `build.sh`\r\n3. And use GitPython's `git.refresh(\u003cfull-path-to-git-executable\u003e)` method inside a [Pyintaller runtime check](https://pyinstaller.org/en/stable/runtime-information.html#run-time-information) to initialize GitPython with the bundled Git executable when running from the bundled application.\r\n\r\n\r\n## Next Steps\r\n\r\n- [x] ~I'll test out the possible solution described above and open a PR if it works as expected.~ Done in #1906\r\n- ~If it doesn't work, I'll document the outcome in this issue.~\r\n\r\n\r\n","author":{"url":"https://github.com/DaveLak","@type":"Person","name":"DaveLak"},"datePublished":"2024-04-22T16:33:41.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":2},"url":"https://github.com/1905/GitPython/issues/1905"}