Title: improvement(ci): collect idle node_modules sticky disks at 3 days by waleedlatif1 · Pull Request #7513 · simstudioai/sim · GitHub
Open Graph Title: improvement(ci): collect idle node_modules sticky disks at 3 days by waleedlatif1 · Pull Request #7513 · simstudioai/sim
X Title: improvement(ci): collect idle node_modules sticky disks at 3 days by waleedlatif1 · Pull Request #7513 · simstudioai/sim
Description: What's going on
The Docker layer-cache caps we shipped last week worked — layer caches gave back 431 GB. But total storage barely moved, because node_modules sticky disks grew +350 GB over the same period and ate the win. They're now 44 disks / 601 GB (~$305/mo) and the single largest storage line, with total storage back over the 500 GB quota.
Why they grew — checked, not assumed
Two possible causes, and they need different fixes:
Disks accumulating cruft over their life → the fix would be in test-build.yml, not retention.
More disks alive at once → the fix is retention.
It's the second. Disks created yesterday are already 14.8 GB, and the whole population now clusters tightly at 14.7–14.9 GB rather than the bimodal spread it had before. They're born that size — no accumulation. What changed is churn: bun.lock moved 24 times in 14 days on staging alone, and disks are created at ~6.3/day.
Steady state is new disks/day × GB/disk × retention:
predicted
observed
7d (today)
606 GB
601 GB
3d (this PR)
260 GB
—
The model reproduces reality to within 1%, so the projection is trustworthy.
Why 3 days
Not a judgement call — the data has a cliff:
Retention
Disks kept
GB kept
Reclaimed now
7d
41
558
43 GB
4d
34
458
144 GB
3d
20
277
348 GB
2d
19
263
362 GB
Going 4d→3d reclaims another 204 GB; going 3d→2d reclaims only 14 GB more. So 3 days captures essentially all of the win, and it clears a long weekend — a PR whose last CI run was Friday afternoon still hits a warm disk on Monday.
Expected: ~$173/mo steady-state, ~$176/mo reclaimed on the first run.
Why age-based and never PR-triggered
The key contains no PR identifier. Every open PR whose checkout has the same bun.lock mounts the same disk — there are far more open PRs than distinct lockfile hashes (28 hashes across 44 disks), so sharing is the common case. A pull_request: [closed] hook would destroy a disk many other open PRs are actively using, most aggressively for the most-shared disk.
Safety — every guard was executed, not reasoned about
Test
Result
Normal run at 3d against the live listing
24/44 selected, 348 GB
retention_days: 0
refused
retention_days: "3d" (non-numeric)
refused
Every disk crafted to look idle
refused — that's a listing/clock fault, not idleness
A turbo-cache-build-push key injected into the listing
0 selected
Two independent filters: a server-side --search, then a local regex re-proving the full key shape. The event segment is [a-z_]+ rather than an enumerated push|pull_request — the key interpolates ${{ github.event_name }}, and a workflow_dispatch disk already exists that an enumerated list would have skipped forever.
Other choices worth a look in review:
GitHub-hosted runner, so collection keeps working during a Blacksmith outage or CI_PROVIDER break-glass switch — exactly when disks idle and still bill.
CLI pinned by version + SHA256, verified against the vendor's published .sha256 sidecar, not curl | sh; the job holds an org-wide token. Auto-update disabled so the pin holds.
permissions: {}, no checkout.
Group by key before the staleness test, since delete without --arch removes every arch variant.
Worst case of a wrong delete is one cold bun install — and the bun cache disk is not lockfile-keyed, so it never idles, is never collected, and a refill links from a warm store with no network fetch.
Setup required before this does anything
A GitHub org admin runs blacksmith org-token create --label ci-stickydisk-gc --organization simstudioai (printed once).
Add it as repository secret BLACKSMITH_CLI_TOKEN. The job fails with a clear message if it's missing.
Must land on main — schedule only runs the default-branch copy.
First run via Actions → Sticky Disk GC → Run workflow → dry_run = true; confirm every listed key is a node-modules-
Open Graph Description: What's going on The Docker layer-cache caps we shipped last week worked — layer caches gave back 431 GB. But total storage barely moved, because node_modules sticky disks grew +350 GB over the ...
X Description: What's going on The Docker layer-cache caps we shipped last week worked — layer caches gave back 431 GB. But total storage barely moved, because node_modules sticky disks grew +350 GB over ...
Opengraph URL: https://github.com/simstudioai/sim/pull/7513
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:d3c86346-70f3-7de1-3cf6-8c7a9aeded9a |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | A480:63A32:24818F:337A05:6AFB51DB |
| html-safe-nonce | 3509b38980ad279db466ae4b85a8e3cd1533243e08615c9880ceb3229f71c754 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJBNDgwOjYzQTMyOjI0ODE4RjozMzdBMDU6NkFGQjUxREIiLCJ2aXNpdG9yX2lkIjoiNjI0NTczOTI3ODk3NDUzODIwMyIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | 068ab59dff12f88fe02d96b0724832dab6c74a4cedd8f8d58d4db76434d5884f |
| hovercard-subject-tag | pull_request:4448448787 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/simstudioai/sim/pull/7513/files |
| twitter:image | https://avatars.githubusercontent.com/u/40672544?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/40672544?s=400&v=4 |
| og:image:alt | What's going on The Docker layer-cache caps we shipped last week worked — layer caches gave back 431 GB. But total storage barely moved, because node_modules sticky disks grew +350 GB over the ... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | 7122bb414b542dba14d71a3f0b9f6fd2abce715b9f5912d082800563e6c806d1 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/simstudioai/sim git https://github.com/simstudioai/sim.git |
| octolytics-dimension-user_id | 199344406 |
| octolytics-dimension-user_login | simstudioai |
| octolytics-dimension-repository_id | 912559512 |
| octolytics-dimension-repository_nwo | simstudioai/sim |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 912559512 |
| octolytics-dimension-repository_network_root_nwo | simstudioai/sim |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | cbf66c024bb6d10e6955baa7413c54cf13a201bb |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width