René's URL Explorer Experiment


Title: fix(redis): bound the three unbudgeted stream writers by bytes by waleedlatif1 · Pull Request #7568 · simstudioai/sim · GitHub

Open Graph Title: fix(redis): bound the three unbudgeted stream writers by bytes by waleedlatif1 · Pull Request #7568 · simstudioai/sim

X Title: fix(redis): bound the three unbudgeted stream writers by bytes by waleedlatif1 · Pull Request #7568 · simstudioai/sim

Description: Why Three Redis writers were bounded by entry count and nothing else. An entry cap bounds how many entries a key holds and says nothing about how large each one is — a writer emitting large entries reaches gigabytes well inside its cap. That is what filled the shared Redis and evicted the entire keyspace: one stream under a 100,000-entry cap. Redis has no per-tenant memory limit, so the bound has to be applied in the application. This adds one, shaped to what each writer's readers can tolerate. What Copilot stream buffer — a replay buffer read from a cursor, so it must stay contiguous. Bytes are now reserved against per-stream and per-user ceilings inside the same Lua that appends (a budget checked in a separate round-trip is one two concurrent writers both pass). A refusal is returned, never thrown: the writer stops persisting for replay and the live stream is unaffected. Tables event log (table:stream:*) — a live feed whose readers already handle a prune by refetching and resuming from latest. So it drops oldest-first past a byte ceiling rather than refusing, riding the existing pruned path. The running total lives in the meta hash under the same TTL as the bytes it counts. File-doc streams (filedoc:stream:*) — Yjs deltas, replayed in full by a task attaching later. Dropping the oldest loses edits outright, so a native MAXLEN retention bound is not safe here. Compaction is, because it folds deltas into a snapshot before trimming — it now triggers on appended bytes as well as entry count, checked every publish since one entry can cross the ceiling alone. Consolidation — lib/execution/redis-budget.server.ts folded into the shared module. Two files were defining the same key prefix and the same ceilings while writing the same Redis keys; a change to one would have silently disagreed with the other. Key layout and every execution limit are unchanged, and now pinned by test. Verification New tests for each bound; each was verified to fail with its fix reverted. apps/sim lib suites (execution, uploads, copilot, realtime, table, core/redis): 4,991 passed. apps/realtime: 289 passed. bun run lint, bun run check:audits (45 audits), bun run type-check across all workspaces: clean.

Open Graph Description: Why Three Redis writers were bounded by entry count and nothing else. An entry cap bounds how many entries a key holds and says nothing about how large each one is — a writer emitting large entries...

X Description: Why Three Redis writers were bounded by entry count and nothing else. An entry cap bounds how many entries a key holds and says nothing about how large each one is — a writer emitting large entries...

Opengraph URL: https://github.com/simstudioai/sim/pull/7568

X: @github

direct link

Domain: github.com

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf
route-pattern/:user_id/:repository/pull/:id/files(.:format)
route-controllerpull_requests
route-actionfiles
fetch-noncev2:fa2e09a4-2336-2cfd-2875-a6cb86c9ee80
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-id8D0C:3EFFC5:8E986C:C9DE5A:6AFB508E
html-safe-nonce1d034be0a748f12580019e5e68bcf7fd55d130dae7e7d3f3789abb7c5dfe80e0
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI4RDBDOjNFRkZDNTo4RTk4NkM6QzlERTVBOjZBRkI1MDhFIiwidmlzaXRvcl9pZCI6IjY5MTA4ODQ4MzUyODgxMDkxOTgiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ==
visitor-hmac90f22dcb110c7f7f1cf1f99cc3f7184a369a545e2649b641cc69dacd5f749dc6
hovercard-subject-tagpull_request:4467726630
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///pull_requests/show/files
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/simstudioai/sim/pull/7568/files
twitter:imagehttps://avatars.githubusercontent.com/u/40672544?s=400&v=4
twitter:cardsummary_large_image
og:imagehttps://avatars.githubusercontent.com/u/40672544?s=400&v=4
og:image:altWhy Three Redis writers were bounded by entry count and nothing else. An entry cap bounds how many entries a key holds and says nothing about how large each one is — a writer emitting large entries...
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
None7122bb414b542dba14d71a3f0b9f6fd2abce715b9f5912d082800563e6c806d1
turbo-cache-controlno-preview
diff-viewunified
go-importgithub.com/simstudioai/sim git https://github.com/simstudioai/sim.git
octolytics-dimension-user_id199344406
octolytics-dimension-user_loginsimstudioai
octolytics-dimension-repository_id912559512
octolytics-dimension-repository_nwosimstudioai/sim
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id912559512
octolytics-dimension-repository_network_root_nwosimstudioai/sim
turbo-body-classeslogged-out env-production page-responsive full-width
disable-turbotrue
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
releasecbf66c024bb6d10e6955baa7413c54cf13a201bb
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/simstudioai/sim/pull/7568/files#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fsimstudioai%2Fsim%2Fpull%2F7568%2Ffiles
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fsimstudioai%2Fsim%2Fpull%2F7568%2Ffiles
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Ffiles&source=header-repo&source_repo=simstudioai%2Fsim
Reloadhttps://github.com/simstudioai/sim/pull/7568/files
Reloadhttps://github.com/simstudioai/sim/pull/7568/files
Reloadhttps://github.com/simstudioai/sim/pull/7568/files
simstudioaihttps://github.com/simstudioai
simhttps://github.com/simstudioai/sim
Codehttps://github.com/simstudioai/sim
Issues129 (129)https://github.com/simstudioai/sim/issues
Pull requests197 (197)https://github.com/simstudioai/sim/pulls
Actionshttps://github.com/simstudioai/sim/actions
Projectshttps://github.com/simstudioai/sim/projects
Security and qualityhttps://github.com/simstudioai/sim/security
Insightshttps://github.com/simstudioai/sim/pulse
Sign up for GitHub https://github.com/signup?return_to=%2Fsimstudioai%2Fsim%2Fissues%2Fnew%2Fchoose
terms of servicehttps://docs.github.com/terms
privacy statementhttps://docs.github.com/privacy
Sign inhttps://github.com/login?return_to=%2Fsimstudioai%2Fsim%2Fissues%2Fnew%2Fchoose
waleedlatif1https://github.com/waleedlatif1
staginghttps://github.com/simstudioai/sim/tree/staging
fix/redis-payload-budgetshttps://github.com/simstudioai/sim/tree/fix/redis-payload-budgets
Conversation 95 https://github.com/simstudioai/sim/pull/7568
Commits 10 https://github.com/simstudioai/sim/pull/7568/commits
Checks 31 https://github.com/simstudioai/sim/pull/7568/checks
Files changed https://github.com/simstudioai/sim/pull/7568/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/7568/files
fix(redis): bound the three unbudgeted stream writers by bytes https://github.com/simstudioai/sim/pull/7568/files#top
Show all changes 10 commits https://github.com/simstudioai/sim/pull/7568/files
cbe11c8 fix(redis): bound the three unbudgeted stream writers by bytes waleedlatif1 Sep 7, 2026 https://github.com/simstudioai/sim/pull/7568/commits/cbe11c89815c2c55d1595c3ee61c40a4beabfea0
c42db99 fix(realtime): count only deltas toward the compaction byte threshold waleedlatif1 Sep 7, 2026 https://github.com/simstudioai/sim/pull/7568/commits/c42db991de0ccc335e3edc8ef397a93d52fda40b
2d6defb fix(redis): address review findings on the byte bounds waleedlatif1 Sep 7, 2026 https://github.com/simstudioai/sim/pull/7568/commits/2d6defb014f5d3d6c57a0afc6e6f10461b4aa5aa
8e68c31 fix(redis): account for deltas a fold retains, and release cleared co… waleedlatif1 Sep 7, 2026 https://github.com/simstudioai/sim/pull/7568/commits/8e68c315a0a6aabbac13e687f73c0de58d12b8e2
4af6e71 fix(redis): make buffer cleanup atomic and match MINID's inclusive bo… waleedlatif1 Sep 7, 2026 https://github.com/simstudioai/sim/pull/7568/commits/4af6e71b81cfece8b7d322f5688fb0a10b57f18b
a40f1be fix(redis): never credit the shared user counter from a buffer delete waleedlatif1 Sep 7, 2026 https://github.com/simstudioai/sim/pull/7568/commits/a40f1bef59d197ea8a8803eb2c25e23994a1bb2d
6794cab fix(redis): keep counters outliving their data, and stop a failed fol… waleedlatif1 Sep 7, 2026 https://github.com/simstudioai/sim/pull/7568/commits/6794cabc8cb820b091409e6113586c6cfba00d5f
aefcaa1 fix(realtime): adopt byte accounting for a stream taken over waleedlatif1 Sep 7, 2026 https://github.com/simstudioai/sim/pull/7568/commits/aefcaa1af5b62562c93c2234a62e319bd01320fa
f2f63a1 fix(realtime): mark a fold's output explicitly, and account in the ta… waleedlatif1 Sep 7, 2026 https://github.com/simstudioai/sim/pull/7568/commits/f2f63a19c5ff16c522cc25208eb4d90c6b7d7a37
235c87c refactor(redis): drop the clear-buffer script and the unused error class waleedlatif1 Sep 7, 2026 https://github.com/simstudioai/sim/pull/7568/commits/235c87c5546f8889ef2e63583f3ee7dcda2e1d9f
Clear filters https://github.com/simstudioai/sim/pull/7568/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/7568/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/7568/files
file-doc-store.test.ts https://github.com/simstudioai/sim/pull/7568/files#diff-58c1d3c2df2d2addcfb82a354e84729e7678b1363a0bcb22e86028034659f3fc
file-doc-store.ts https://github.com/simstudioai/sim/pull/7568/files#diff-0d8e8ff53357fcf40d72aa618fbdca8f29efe869c7b227b75f304bdece2cde6a
start.ts https://github.com/simstudioai/sim/pull/7568/files#diff-6d1d121ff4ddd703b0fe2a11c19ce949a7d3d0532e5e7c84bb0fecb2e8e48f8e
buffer.test.ts https://github.com/simstudioai/sim/pull/7568/files#diff-e06e6d3d44d833568c2265fceb3d133bf7fe8820950bbe215c1133062e88c3a2
buffer.ts https://github.com/simstudioai/sim/pull/7568/files#diff-33507f0dcbd3ab01d2b49d9a505c75f684e407b0ded131020f907df6d0902db6
writer.test.ts https://github.com/simstudioai/sim/pull/7568/files#diff-fc9d3191b2504db86a4c354b0fbc59cd59158af21eef50b041c34a97aec89787
writer.ts https://github.com/simstudioai/sim/pull/7568/files#diff-cc7f0b72ae545f9596d5616d720115eb14b76df4aab03fe483c6ca36e72a1c42
byte-budget.server.test.ts https://github.com/simstudioai/sim/pull/7568/files#diff-8fcbd1c22e5352801a8668b4e07bace8998466421488893320cfc264cc23fbf0
byte-budget.server.ts https://github.com/simstudioai/sim/pull/7568/files#diff-d27f814dc324ffc076144a10984df0e0d5434afda975f4f477f349730bd60d55
event-buffer.ts https://github.com/simstudioai/sim/pull/7568/files#diff-7d7525d18ecf7583efaa0d408ec6beace469e423fbbf40183f61673a4f0fc802
redis-budget.server.test.ts https://github.com/simstudioai/sim/pull/7568/files#diff-eea3bba0edab094730c69f9d71f043cac47fc6537d3589123814879dc7c78ce5
redis-budget.server.ts https://github.com/simstudioai/sim/pull/7568/files#diff-ef75a1d85215e0147119836b4684db6c1c9df11fee79d06b56b0293ba8ab57ce
event-log.test.ts https://github.com/simstudioai/sim/pull/7568/files#diff-f55d51798ac8a615d9556d40037477d696af37d47e75dd38981bbb7a52dc6937
event-log.ts https://github.com/simstudioai/sim/pull/7568/files#diff-50740783dd2c50901dcbc5409dbdcc3fc1d88b4652912ff18611494002e437af
events.ts https://github.com/simstudioai/sim/pull/7568/files#diff-f4db8c424928e747aaffb7c0352fdb9690e85dd886cf357a8deca0cdaf29e6e3
user-file-base64.server.ts https://github.com/simstudioai/sim/pull/7568/files#diff-df6aa2815c3351b752a7e2ae94c7769597ae4291706c82f4bd4138969a389f02
apps/realtime/src/handlers/file-doc-store.test.tshttps://github.com/simstudioai/sim/pull/7568/files#diff-58c1d3c2df2d2addcfb82a354e84729e7678b1363a0bcb22e86028034659f3fc
View file https://github.com/simstudioai/sim/blob/235c87c5546f8889ef2e63583f3ee7dcda2e1d9f/apps/realtime/src/handlers/file-doc-store.test.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/7568/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/7568/files#diff-58c1d3c2df2d2addcfb82a354e84729e7678b1363a0bcb22e86028034659f3fc
https://github.com/simstudioai/sim/pull/7568/files#diff-58c1d3c2df2d2addcfb82a354e84729e7678b1363a0bcb22e86028034659f3fc
https://github.com/simstudioai/sim/pull/7568/files#diff-58c1d3c2df2d2addcfb82a354e84729e7678b1363a0bcb22e86028034659f3fc
https://github.com/simstudioai/sim/pull/7568/files#diff-58c1d3c2df2d2addcfb82a354e84729e7678b1363a0bcb22e86028034659f3fc
https://github.com/simstudioai/sim/pull/7568/files#diff-58c1d3c2df2d2addcfb82a354e84729e7678b1363a0bcb22e86028034659f3fc
https://github.com/simstudioai/sim/pull/7568/files#diff-58c1d3c2df2d2addcfb82a354e84729e7678b1363a0bcb22e86028034659f3fc
https://github.com/simstudioai/sim/pull/7568/files#diff-58c1d3c2df2d2addcfb82a354e84729e7678b1363a0bcb22e86028034659f3fc
https://github.com/simstudioai/sim/pull/7568/files#diff-58c1d3c2df2d2addcfb82a354e84729e7678b1363a0bcb22e86028034659f3fc
https://github.com/simstudioai/sim/pull/7568/files#diff-58c1d3c2df2d2addcfb82a354e84729e7678b1363a0bcb22e86028034659f3fc
Please reload this pagehttps://github.com/simstudioai/sim/pull/7568/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/7568/files
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.