René's URL Explorer Experiment

Title: Delete unused CodeQL bundles from the toolcache before downloading by henrymercer · Pull Request #4124 · github/codeql-action · GitHub

Open Graph Title: Delete unused CodeQL bundles from the toolcache before downloading by henrymercer · Pull Request #4124 · github/codeql-action

X Title: Delete unused CodeQL bundles from the toolcache before downloading by henrymercer · Pull Request #4124 · github/codeql-action

Description: When a job downloads a CodeQL bundle, the toolcache is often already holding a different one that the job will not use, typically the bundle baked into the runner image. On GitHub-hosted runners the toolcache sits on the same filesystem we measure for the disk space check, so that unused bundle takes around 2 GB away from the analysis. This adds a feature flag, cleanup_toolcache_bundles, that empties $RUNNER_TOOL_CACHE/CodeQL immediately before we download a bundle. It is off by default. Emptying it before the download rather than after means the space also covers the archive during extraction. We only get as far as downloading when the tools were not resolved from the toolcache, and a lookup there requires both the version directory and its completion marker, so nothing left behind is usable by the job. That stops holding once a step has obtained the tools, because a later step may run a path it was given, as our own codeql.yml does. The job records that it has the tools and skips the cleanup from then on. The cleanup only runs when RUNNER_ENVIRONMENT reports github-hosted, so we never delete from a toolcache that outlives the job, and only when the toolcache is on the same filesystem as the workspace, since otherwise the space we free is not space the analysis can use. Windows runner images are laid out that way, keeping the toolcache on C: while the workspace is on D:. Deletion is confined to the CodeQL directory, refuses to follow a symlink, and logs failures rather than throwing. This also renames isHostedRunner to looksLikeHostedRunner, with no behaviour change. It infers hostedness from the runner name and filesystem layout, so it can disagree with the new isGitHubHostedRunner check. Its callers are deciding whether caching is worthwhile, where the inferred answer is arguably the one they want, so switching them would be a behaviour change and is left for a separate PR. Risk assessment Low risk: Changes are fully under feature flags, or have been fully tested and validated in pre-production environments and are highly observable, or are documentation or test only. Which use cases does this change impact? Workflow types: Advanced setup - Impacts users who have custom CodeQL workflows. Managed - Impacts users with dynamic workflows (Default Setup, Code Quality, ...). Products: Code Scanning - The changes impact analyses when analysis-kinds: code-scanning. Code Quality - The changes impact analyses when analysis-kinds: code-quality. Other first-party - The changes impact other first-party analyses. Third-party analyses - The changes affect the upload-sarif action. The cleanup sits on the shared bundle download path, so it can run for anything that obtains the CodeQL tools. Environments: Dotcom - Impacts CodeQL workflows on github.com and/or GitHub Enterprise Cloud with Data Residency. How did/will you validate this change? End-to-end tests - I am depending on PR checks (i.e. tests in pr-checks). Unit tests - I am depending on unit test coverage (i.e. tests in .test.ts files). Other - Please provide details. Unit tests cover the flag being off, a non-hosted runner, an unreported runner environment, a toolcache on a different filesystem to the workspace, a hosted runner with bundles present, a download that will not be cached in the toolcache, a step having already obtained the tools, deletion failing, a toolcache we cannot inspect, and a symlinked CodeQL directory. The multi-language PR check runs with the flag enabled, so the cleanup happens ahead of a real build and analysis of seven languages. This has not been run on a real hosted runner outside the PR checks, and the disk saving is inferred from bundle size rather than measured, so the rollout should confirm it. If something goes wrong after this change is released, what are the mitigation and rollback strategies? Feature flags - All new or changed code paths can be fully disabled with corresponding feature flags. The rename is not behind the flag, but it has no behaviour change. How will you know if something goes wrong after this change is released? Telemetry - I rely on existing telemetry or have made changes to the telemetry. A codeql-action/toolcache-bundle-cleanup diagnostic reports which versions were deleted and whether the attempt failed. Note that diagnostics are only flushed by the init action, so a cleanup under the standalone setup-codeql action reports nothing. That is pre-existing behaviour for diagnostics raised from setup-codeql.ts, but it does leave that entry point unmeasurable. Are there any special considerations for merging or releasing this change? No special considerations - This change can be merged at any time. Merge / deployment checklist Confirm this change is backwards compatible with existing workflows. Consider adding a changelog entry for this change. Confirm the readme and docs have been updated if necessary.

Open Graph Description: When a job downloads a CodeQL bundle, the toolcache is often already holding a different one that the job will not use, typically the bundle baked into the runner image. On GitHub-hosted runners th...

X Description: When a job downloads a CodeQL bundle, the toolcache is often already holding a different one that the job will not use, typically the bundle baked into the runner image. On GitHub-hosted runners th...

Opengraph URL: https://github.com/github/codeql-action/pull/4124

X: @github

direct link

Domain: github.com

Links:

Viewport: width=device-width