René's URL Explorer Experiment


Title: improvement(cleanup): batchTrigger fan-out, chunked queries, batched S3, faster outlier drain by waleedlatif1 · Pull Request #4688 · simstudioai/sim · GitHub

Open Graph Title: improvement(cleanup): batchTrigger fan-out, chunked queries, batched S3, faster outlier drain by waleedlatif1 · Pull Request #4688 · simstudioai/sim

X Title: improvement(cleanup): batchTrigger fan-out, chunked queries, batched S3, faster outlier drain by waleedlatif1 · Pull Request #4688 · simstudioai/sim

Description: Summary Fan cleanup-tasks/logs/soft-deletes out via tasks.batchTrigger (500 ws/chunk); bump to large-1x with concurrencyLimit: 5 Chunk bulk DELETEs (1000 IDs/stmt) and collectChatFiles JSONB SELECT (500 chats/stmt) to bound worker memory and lock duration Replace per-key position() table scans with one LATERAL unnest scan per 200-key chunk Route storage deletes through StorageService.deleteFiles (S3 DeleteObjects: 1000 keys/HTTP) Raise per-run row cap to 100K so long-tail tenants (one prod workspace has 723K doomed rows) drain in days, not weeks Type of Change Improvement Testing Verified position-query SQL rewrite returns identical results to original against local Postgres with seeded data tsc, biome check, check:api-validation all pass 98 adjacent tests pass (uploads, snapshot service, billing core) Trigger.dev batchTrigger usage validated against official docs (SDK 4.4.3, all options within documented caps) Checklist Code follows project style guidelines Self-reviewed my changes Tests added/updated and passing No new warnings introduced I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

Open Graph Description: Summary Fan cleanup-tasks/logs/soft-deletes out via tasks.batchTrigger (500 ws/chunk); bump to large-1x with concurrencyLimit: 5 Chunk bulk DELETEs (1000 IDs/stmt) and collectChatFiles JSONB SELEC...

X Description: Summary Fan cleanup-tasks/logs/soft-deletes out via tasks.batchTrigger (500 ws/chunk); bump to large-1x with concurrencyLimit: 5 Chunk bulk DELETEs (1000 IDs/stmt) and collectChatFiles JSONB SELEC...

Opengraph URL: https://github.com/simstudioai/sim/pull/4688

X: @github

direct link

Domain: github.com

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf
route-pattern/:user_id/:repository/pull/:id/files(.:format)
route-controllerpull_requests
route-actionfiles
fetch-noncev2:9fbb777a-c70b-0aa2-35c2-c7dca104edca
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-idCD16:1C135C:4AB9:648C:6AFB63F8
html-safe-nonce8802591b015a5d0fb5d5202f47b95e65f313d50b70fe667300b7a9dd16b09fa0
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJDRDE2OjFDMTM1Qzo0QUI5OjY0OEM6NkFGQjYzRjgiLCJ2aXNpdG9yX2lkIjoiNTY3MTMxNjAyOTA5MTU3MDY4MCIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9
visitor-hmac45f1351a3037416c015612ec31ba7af96a00ac1dc1de36da8bd4a7c4bd79011e
hovercard-subject-tagpull_request:3720525421
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///pull_requests/show/files
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/simstudioai/sim/pull/4688/files
twitter:imagehttps://avatars.githubusercontent.com/u/40672544?s=400&v=4
twitter:cardsummary_large_image
og:imagehttps://avatars.githubusercontent.com/u/40672544?s=400&v=4
og:image:altSummary Fan cleanup-tasks/logs/soft-deletes out via tasks.batchTrigger (500 ws/chunk); bump to large-1x with concurrencyLimit: 5 Chunk bulk DELETEs (1000 IDs/stmt) and collectChatFiles JSONB SELEC...
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
Noneea33d13c96d0ff66bd8fa906b193d2213536b78cb06b0c50a4632c58b1418f94
turbo-cache-controlno-preview
diff-viewunified
go-importgithub.com/simstudioai/sim git https://github.com/simstudioai/sim.git
octolytics-dimension-user_id199344406
octolytics-dimension-user_loginsimstudioai
octolytics-dimension-repository_id912559512
octolytics-dimension-repository_nwosimstudioai/sim
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id912559512
octolytics-dimension-repository_network_root_nwosimstudioai/sim
turbo-body-classeslogged-out env-production page-responsive full-width
disable-turbotrue
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release59ef97164c52cf59bd10558ddec9c4f482763988
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/simstudioai/sim/pull/4688/files#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fsimstudioai%2Fsim%2Fpull%2F4688%2Ffiles
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fsimstudioai%2Fsim%2Fpull%2F4688%2Ffiles
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Ffiles&source=header-repo&source_repo=simstudioai%2Fsim
Reloadhttps://github.com/simstudioai/sim/pull/4688/files
Reloadhttps://github.com/simstudioai/sim/pull/4688/files
Reloadhttps://github.com/simstudioai/sim/pull/4688/files
simstudioaihttps://github.com/simstudioai
simhttps://github.com/simstudioai/sim
Codehttps://github.com/simstudioai/sim
Issues129 (129)https://github.com/simstudioai/sim/issues
Pull requests197 (197)https://github.com/simstudioai/sim/pulls
Actionshttps://github.com/simstudioai/sim/actions
Projectshttps://github.com/simstudioai/sim/projects
Security and qualityhttps://github.com/simstudioai/sim/security
Insightshttps://github.com/simstudioai/sim/pulse
Sign up for GitHub https://github.com/signup?return_to=%2Fsimstudioai%2Fsim%2Fissues%2Fnew%2Fchoose
terms of servicehttps://docs.github.com/terms
privacy statementhttps://docs.github.com/privacy
Sign inhttps://github.com/login?return_to=%2Fsimstudioai%2Fsim%2Fissues%2Fnew%2Fchoose
waleedlatif1https://github.com/waleedlatif1
staginghttps://github.com/simstudioai/sim/tree/staging
waleedlatif1/trigger-cleanup-larger-machinehttps://github.com/simstudioai/sim/tree/waleedlatif1/trigger-cleanup-larger-machine
Conversation 9 https://github.com/simstudioai/sim/pull/4688
Commits 2 https://github.com/simstudioai/sim/pull/4688/commits
Checks 13 https://github.com/simstudioai/sim/pull/4688/checks
Files changed https://github.com/simstudioai/sim/pull/4688/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/4688/files
improvement(cleanup): batchTrigger fan-out, chunked queries, batched S3, faster outlier drain https://github.com/simstudioai/sim/pull/4688/files#top
Show all changes 2 commits https://github.com/simstudioai/sim/pull/4688/files
3f3f200 improvement(cleanup): batchTrigger fan-out, chunked queries, batched … waleedlatif1 May 21, 2026 https://github.com/simstudioai/sim/pull/4688/commits/3f3f2005c4519e4f37f9a34600cfa580f15fe836
ddbdacb improvement(cleanup): chunk-index labels, clarify upper-bound failure… waleedlatif1 May 21, 2026 https://github.com/simstudioai/sim/pull/4688/commits/ddbdacbd653b2a28c3a16e1447142a51a5f841d0
Clear filters https://github.com/simstudioai/sim/pull/4688/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/4688/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/4688/files
cleanup-logs.ts https://github.com/simstudioai/sim/pull/4688/files#diff-5aa920679fbd8b67e6e550268dd0c6d9031297895cf596a415e6317d97d83be8
cleanup-soft-deletes.ts https://github.com/simstudioai/sim/pull/4688/files#diff-0d27a0e21a9e9287bfa78ce2693f0694d417ae9bde9b2379b06f1d7bdd4052d6
cleanup-tasks.ts https://github.com/simstudioai/sim/pull/4688/files#diff-729994db54353b6880c11d202dcca660a4f07f694366c9b496ac179f126fa211
cleanup-dispatcher.ts https://github.com/simstudioai/sim/pull/4688/files#diff-9b0785fc05035d15847118630b773f0c10848c5292b12139dc229074e6b3e868
batch-delete.ts https://github.com/simstudioai/sim/pull/4688/files#diff-f07b958aaec9193a4b4c14e384154d004e0aa8d937949fcf60403269307315a5
chat-cleanup.ts https://github.com/simstudioai/sim/pull/4688/files#diff-5f80865555d40f6c376c0d53f72944d721b340b03cf01b8f43a0d41fb28640e3
storage-service.ts https://github.com/simstudioai/sim/pull/4688/files#diff-8768f26f6a6d271a9bf26da5650f60add71370858edf7f9fd68abbfe423030fd
client.ts https://github.com/simstudioai/sim/pull/4688/files#diff-5b5c75fd8b0e84e44544dfe3eb319a86603568755f5568c5b4ce3007b519927e
apps/sim/background/cleanup-logs.tshttps://github.com/simstudioai/sim/pull/4688/files#diff-5aa920679fbd8b67e6e550268dd0c6d9031297895cf596a415e6317d97d83be8
View file https://github.com/simstudioai/sim/blob/ddbdacbd653b2a28c3a16e1447142a51a5f841d0/apps/sim/background/cleanup-logs.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/4688/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/4688/files#diff-5aa920679fbd8b67e6e550268dd0c6d9031297895cf596a415e6317d97d83be8
https://github.com/simstudioai/sim/pull/4688/files#diff-5aa920679fbd8b67e6e550268dd0c6d9031297895cf596a415e6317d97d83be8
https://github.com/simstudioai/sim/pull/4688/files#diff-5aa920679fbd8b67e6e550268dd0c6d9031297895cf596a415e6317d97d83be8
https://github.com/simstudioai/sim/pull/4688/files#diff-5aa920679fbd8b67e6e550268dd0c6d9031297895cf596a415e6317d97d83be8
https://github.com/simstudioai/sim/pull/4688/files#diff-5aa920679fbd8b67e6e550268dd0c6d9031297895cf596a415e6317d97d83be8
https://github.com/simstudioai/sim/pull/4688/files#diff-5aa920679fbd8b67e6e550268dd0c6d9031297895cf596a415e6317d97d83be8
apps/sim/background/cleanup-soft-deletes.tshttps://github.com/simstudioai/sim/pull/4688/files#diff-0d27a0e21a9e9287bfa78ce2693f0694d417ae9bde9b2379b06f1d7bdd4052d6
View file https://github.com/simstudioai/sim/blob/ddbdacbd653b2a28c3a16e1447142a51a5f841d0/apps/sim/background/cleanup-soft-deletes.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/4688/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/4688/files#diff-0d27a0e21a9e9287bfa78ce2693f0694d417ae9bde9b2379b06f1d7bdd4052d6
https://github.com/simstudioai/sim/pull/4688/files#diff-0d27a0e21a9e9287bfa78ce2693f0694d417ae9bde9b2379b06f1d7bdd4052d6
https://github.com/simstudioai/sim/pull/4688/files#diff-0d27a0e21a9e9287bfa78ce2693f0694d417ae9bde9b2379b06f1d7bdd4052d6
https://github.com/simstudioai/sim/pull/4688/files#diff-0d27a0e21a9e9287bfa78ce2693f0694d417ae9bde9b2379b06f1d7bdd4052d6
https://github.com/simstudioai/sim/pull/4688/files#diff-0d27a0e21a9e9287bfa78ce2693f0694d417ae9bde9b2379b06f1d7bdd4052d6
https://github.com/simstudioai/sim/pull/4688/files#diff-0d27a0e21a9e9287bfa78ce2693f0694d417ae9bde9b2379b06f1d7bdd4052d6
https://github.com/simstudioai/sim/pull/4688/files#diff-0d27a0e21a9e9287bfa78ce2693f0694d417ae9bde9b2379b06f1d7bdd4052d6
apps/sim/background/cleanup-tasks.tshttps://github.com/simstudioai/sim/pull/4688/files#diff-729994db54353b6880c11d202dcca660a4f07f694366c9b496ac179f126fa211
View file https://github.com/simstudioai/sim/blob/ddbdacbd653b2a28c3a16e1447142a51a5f841d0/apps/sim/background/cleanup-tasks.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/4688/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/4688/files#diff-729994db54353b6880c11d202dcca660a4f07f694366c9b496ac179f126fa211
https://github.com/simstudioai/sim/pull/4688/files#diff-729994db54353b6880c11d202dcca660a4f07f694366c9b496ac179f126fa211
https://github.com/simstudioai/sim/pull/4688/files#diff-729994db54353b6880c11d202dcca660a4f07f694366c9b496ac179f126fa211
https://github.com/simstudioai/sim/pull/4688/files#diff-729994db54353b6880c11d202dcca660a4f07f694366c9b496ac179f126fa211
https://github.com/simstudioai/sim/pull/4688/files#diff-729994db54353b6880c11d202dcca660a4f07f694366c9b496ac179f126fa211
https://github.com/simstudioai/sim/pull/4688/files#diff-729994db54353b6880c11d202dcca660a4f07f694366c9b496ac179f126fa211
https://github.com/simstudioai/sim/pull/4688/files#diff-729994db54353b6880c11d202dcca660a4f07f694366c9b496ac179f126fa211
https://github.com/simstudioai/sim/pull/4688/files#diff-729994db54353b6880c11d202dcca660a4f07f694366c9b496ac179f126fa211
Please reload this pagehttps://github.com/simstudioai/sim/pull/4688/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/4688/files
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.