René's URL Explorer Experiment


Title: fix(knowledge): bound chunking separators so one config can't stall processing by waleedlatif1 · Pull Request #6735 · simstudioai/sim · GitHub

Open Graph Title: fix(knowledge): bound chunking separators so one config can't stall processing by waleedlatif1 · Pull Request #6735 · simstudioai/sim

X Title: fix(knowledge): bound chunking separators so one config can't stall processing by waleedlatif1 · Pull Request #6735 · simstudioai/sim

Description: Summary Cap strategyOptions.separators at 32 entries of at most 100 characters on knowledge-base create/update. It sat next to a pattern field already capped at 500 chars, but had no bound of its own. RecursiveChunker splits the whole document once per separator and restarts the list for every oversized fragment, so a saved config with thousands of non-matching separators burned seconds of synchronous CPU on every later document upload. The processing Promise.race timeout and the chunk-count cap can't interrupt a synchronous split loop. On a 21.3 MB document: 632 ms at 100 separators, 6.1 s at 1000, 36.8 s at 5000 — now ~300 ms at every count. Keep the stored/read shape unbounded so a config written before this change still lists instead of failing response validation. Clamp in the chunker too, so an already-saved oversized list can't reach the split loop. An over-long separator is dropped rather than truncated — a truncated separator matches where the configured one never did, which would silently re-cut the document. A list left empty falls back to the recipe. Walk non-matching separators iteratively instead of recursively, so stack depth no longer tracks the separator count. Validate in the create-base modal so the limit surfaces inline instead of as a 400. Type of Change Bug fix Testing New unit tests for the separator bounds and the contract; each verified to fail against the unfixed code. Diffed the rewritten split against the previous implementation over 4000 randomized configs (custom separators, all three recipes, varying chunk size and overlap) — byte-identical output, so the loop rewrite changes nothing for in-bound configs. bun run check:audits (27 audits), bun run type-check, and 750 tests across lib/chunkers, lib/api/contracts/knowledge, and lib/knowledge pass. Checklist Code follows project style guidelines Self-reviewed my changes Tests added/updated and passing No new warnings introduced I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

Open Graph Description: Summary Cap strategyOptions.separators at 32 entries of at most 100 characters on knowledge-base create/update. It sat next to a pattern field already capped at 500 chars, but had no bound of its ...

X Description: Summary Cap strategyOptions.separators at 32 entries of at most 100 characters on knowledge-base create/update. It sat next to a pattern field already capped at 500 chars, but had no bound of its ...

Opengraph URL: https://github.com/simstudioai/sim/pull/6735

X: @github

direct link

Domain: github.com

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf
route-pattern/:user_id/:repository/pull/:id/files(.:format)
route-controllerpull_requests
route-actionfiles
fetch-noncev2:81820a01-203e-761f-21ac-ff6e21f42c46
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-idCE00:14B945:76511A:A3EFBD:6AFB45CE
html-safe-nonce149705bee6e4610031c21c890f35c261fdcc6dd571b3e0dbfd3bf91384604148
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJDRTAwOjE0Qjk0NTo3NjUxMUE6QTNFRkJEOjZBRkI0NUNFIiwidmlzaXRvcl9pZCI6IjQxNTUwOTgxMzU4NjI2NTg1MTAiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ==
visitor-hmac20bea2686ec134cebe7504c215a09ad82eaefef9a82a78269cfbd926edb9699a
hovercard-subject-tagpull_request:4287358639
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///pull_requests/show/files
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/simstudioai/sim/pull/6735/files
twitter:imagehttps://avatars.githubusercontent.com/u/40672544?s=400&v=4
twitter:cardsummary_large_image
og:imagehttps://avatars.githubusercontent.com/u/40672544?s=400&v=4
og:image:altSummary Cap strategyOptions.separators at 32 entries of at most 100 characters on knowledge-base create/update. It sat next to a pattern field already capped at 500 chars, but had no bound of its ...
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
Noned849b928f455c800985398e7f1b659425adb312a5fa14fba5d9a14e62e9fe5dd
turbo-cache-controlno-preview
diff-viewunified
go-importgithub.com/simstudioai/sim git https://github.com/simstudioai/sim.git
octolytics-dimension-user_id199344406
octolytics-dimension-user_loginsimstudioai
octolytics-dimension-repository_id912559512
octolytics-dimension-repository_nwosimstudioai/sim
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id912559512
octolytics-dimension-repository_network_root_nwosimstudioai/sim
turbo-body-classeslogged-out env-production page-responsive full-width
disable-turbotrue
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
releasea6c2f87cbcb6414452377b4967858ed6a66f1109
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/simstudioai/sim/pull/6735/files#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fsimstudioai%2Fsim%2Fpull%2F6735%2Ffiles
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fsimstudioai%2Fsim%2Fpull%2F6735%2Ffiles
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Ffiles&source=header-repo&source_repo=simstudioai%2Fsim
Reloadhttps://github.com/simstudioai/sim/pull/6735/files
Reloadhttps://github.com/simstudioai/sim/pull/6735/files
Reloadhttps://github.com/simstudioai/sim/pull/6735/files
simstudioaihttps://github.com/simstudioai
simhttps://github.com/simstudioai/sim
Codehttps://github.com/simstudioai/sim
Issues129 (129)https://github.com/simstudioai/sim/issues
Pull requests199 (199)https://github.com/simstudioai/sim/pulls
Actionshttps://github.com/simstudioai/sim/actions
Projectshttps://github.com/simstudioai/sim/projects
Security and qualityhttps://github.com/simstudioai/sim/security
Insightshttps://github.com/simstudioai/sim/pulse
Sign up for GitHub https://github.com/signup?return_to=%2Fsimstudioai%2Fsim%2Fissues%2Fnew%2Fchoose
terms of servicehttps://docs.github.com/terms
privacy statementhttps://docs.github.com/privacy
Sign inhttps://github.com/login?return_to=%2Fsimstudioai%2Fsim%2Fissues%2Fnew%2Fchoose
waleedlatif1https://github.com/waleedlatif1
staginghttps://github.com/simstudioai/sim/tree/staging
fix/chunking-separators-boundhttps://github.com/simstudioai/sim/tree/fix/chunking-separators-bound
Conversation 10 https://github.com/simstudioai/sim/pull/6735
Commits 2 https://github.com/simstudioai/sim/pull/6735/commits
Checks 30 https://github.com/simstudioai/sim/pull/6735/checks
Files changed https://github.com/simstudioai/sim/pull/6735/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/6735/files
fix(knowledge): bound chunking separators so one config can't stall processing https://github.com/simstudioai/sim/pull/6735/files#top
Show all changes 2 commits https://github.com/simstudioai/sim/pull/6735/files
367aff4 fix(knowledge): bound chunking separators so one config can't stall p… waleedlatif1 Aug 15, 2026 https://github.com/simstudioai/sim/pull/6735/commits/367aff4834c96046218d77c9ac530d2a97ab23de
6f94d58 fix(knowledge): gate separator validation on the recursive strategy waleedlatif1 Aug 15, 2026 https://github.com/simstudioai/sim/pull/6735/commits/6f94d58c2e64a632a1a9649c583c9c6982d526b9
Clear filters https://github.com/simstudioai/sim/pull/6735/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/6735/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/6735/files
create-base-modal.tsx https://github.com/simstudioai/sim/pull/6735/files#diff-6dc51f5a2de9f62757e4653bfaa1e26b6b869862d3085e2630c4ef7353c50246
base.test.ts https://github.com/simstudioai/sim/pull/6735/files#diff-30970a782de75a895c7d22e73b00336fddc26080885181d08534cb7c9066ff7d
base.ts https://github.com/simstudioai/sim/pull/6735/files#diff-af4bae723c3963d83aa7976287345a51652d8e76ae08ba51f65c5bfde501c1d8
constants.ts https://github.com/simstudioai/sim/pull/6735/files#diff-8d04724cf9b6fe1731166beb8a0114c3c09a4c03d816956f527109787953fff0
recursive-chunker.test.ts https://github.com/simstudioai/sim/pull/6735/files#diff-ab11626398304ab8cf7d01b6463ff9854a8fd63089c46c64a5dd2d608b926a33
recursive-chunker.ts https://github.com/simstudioai/sim/pull/6735/files#diff-8332c7cf6bd23aaf1569820b84ead0863cf318a79469a068f5ef8a255e95c356
.../app/workspace/[workspaceId]/knowledge/components/create-base-modal/create-base-modal.tsxhttps://github.com/simstudioai/sim/pull/6735/files#diff-6dc51f5a2de9f62757e4653bfaa1e26b6b869862d3085e2630c4ef7353c50246
View file https://github.com/simstudioai/sim/blob/6f94d58c2e64a632a1a9649c583c9c6982d526b9/apps/sim/app/workspace/%5BworkspaceId%5D/knowledge/components/create-base-modal/create-base-modal.tsx
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/6735/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/6735/files#diff-6dc51f5a2de9f62757e4653bfaa1e26b6b869862d3085e2630c4ef7353c50246
https://github.com/simstudioai/sim/pull/6735/files#diff-6dc51f5a2de9f62757e4653bfaa1e26b6b869862d3085e2630c4ef7353c50246
https://github.com/simstudioai/sim/pull/6735/files#diff-6dc51f5a2de9f62757e4653bfaa1e26b6b869862d3085e2630c4ef7353c50246
https://github.com/simstudioai/sim/pull/6735/files#diff-6dc51f5a2de9f62757e4653bfaa1e26b6b869862d3085e2630c4ef7353c50246
https://github.com/simstudioai/sim/pull/6735/files#diff-6dc51f5a2de9f62757e4653bfaa1e26b6b869862d3085e2630c4ef7353c50246
Please reload this pagehttps://github.com/simstudioai/sim/pull/6735/files
https://github.com/simstudioai/sim/pull/6735/files#diff-6dc51f5a2de9f62757e4653bfaa1e26b6b869862d3085e2630c4ef7353c50246
https://github.com/simstudioai/sim/pull/6735/files#diff-6dc51f5a2de9f62757e4653bfaa1e26b6b869862d3085e2630c4ef7353c50246
https://github.com/simstudioai/sim/pull/6735/files#diff-6dc51f5a2de9f62757e4653bfaa1e26b6b869862d3085e2630c4ef7353c50246
https://github.com/simstudioai/sim/pull/6735/files#diff-6dc51f5a2de9f62757e4653bfaa1e26b6b869862d3085e2630c4ef7353c50246
https://github.com/simstudioai/sim/pull/6735/files#diff-6dc51f5a2de9f62757e4653bfaa1e26b6b869862d3085e2630c4ef7353c50246
apps/sim/lib/api/contracts/knowledge/base.test.tshttps://github.com/simstudioai/sim/pull/6735/files#diff-30970a782de75a895c7d22e73b00336fddc26080885181d08534cb7c9066ff7d
View file https://github.com/simstudioai/sim/blob/6f94d58c2e64a632a1a9649c583c9c6982d526b9/apps/sim/lib/api/contracts/knowledge/base.test.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/6735/{{ revealButtonHref }}
apps/sim/lib/api/contracts/knowledge/base.tshttps://github.com/simstudioai/sim/pull/6735/files#diff-af4bae723c3963d83aa7976287345a51652d8e76ae08ba51f65c5bfde501c1d8
View file https://github.com/simstudioai/sim/blob/6f94d58c2e64a632a1a9649c583c9c6982d526b9/apps/sim/lib/api/contracts/knowledge/base.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/6735/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/6735/files#diff-af4bae723c3963d83aa7976287345a51652d8e76ae08ba51f65c5bfde501c1d8
https://github.com/simstudioai/sim/pull/6735/files#diff-af4bae723c3963d83aa7976287345a51652d8e76ae08ba51f65c5bfde501c1d8
https://github.com/simstudioai/sim/pull/6735/files#diff-af4bae723c3963d83aa7976287345a51652d8e76ae08ba51f65c5bfde501c1d8
https://github.com/simstudioai/sim/pull/6735/files#diff-af4bae723c3963d83aa7976287345a51652d8e76ae08ba51f65c5bfde501c1d8
https://github.com/simstudioai/sim/pull/6735/files#diff-af4bae723c3963d83aa7976287345a51652d8e76ae08ba51f65c5bfde501c1d8
https://github.com/simstudioai/sim/pull/6735/files#diff-af4bae723c3963d83aa7976287345a51652d8e76ae08ba51f65c5bfde501c1d8
apps/sim/lib/chunkers/constants.tshttps://github.com/simstudioai/sim/pull/6735/files#diff-8d04724cf9b6fe1731166beb8a0114c3c09a4c03d816956f527109787953fff0
View file https://github.com/simstudioai/sim/blob/6f94d58c2e64a632a1a9649c583c9c6982d526b9/apps/sim/lib/chunkers/constants.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/6735/{{ revealButtonHref }}
apps/sim/lib/chunkers/recursive-chunker.test.tshttps://github.com/simstudioai/sim/pull/6735/files#diff-ab11626398304ab8cf7d01b6463ff9854a8fd63089c46c64a5dd2d608b926a33
View file https://github.com/simstudioai/sim/blob/6f94d58c2e64a632a1a9649c583c9c6982d526b9/apps/sim/lib/chunkers/recursive-chunker.test.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/6735/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/6735/files#diff-ab11626398304ab8cf7d01b6463ff9854a8fd63089c46c64a5dd2d608b926a33
https://github.com/simstudioai/sim/pull/6735/files#diff-ab11626398304ab8cf7d01b6463ff9854a8fd63089c46c64a5dd2d608b926a33
https://github.com/simstudioai/sim/pull/6735/files#diff-ab11626398304ab8cf7d01b6463ff9854a8fd63089c46c64a5dd2d608b926a33
https://github.com/simstudioai/sim/pull/6735/files#diff-ab11626398304ab8cf7d01b6463ff9854a8fd63089c46c64a5dd2d608b926a33
Please reload this pagehttps://github.com/simstudioai/sim/pull/6735/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/6735/files
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.