Title: perf(executor): stop re-parsing the whole structured output on every streamed token by b84330808 · Pull Request #10087 · simstudioai/sim · GitHub
Open Graph Title: perf(executor): stop re-parsing the whole structured output on every streamed token by b84330808 · Pull Request #10087 · simstudioai/sim
X Title: perf(executor): stop re-parsing the whole structured output on every streamed token by b84330808 · Pull Request #10087 · simstudioai/sim
Description: Summary A chat or run can select a response-format field of an Agent block (an output like agent1_answer). When the agent streams, StreamingResponseFormatProcessor calls JSON.parse(buffer.trim()) on the whole buffer for every token delta. When that fails, it also counts braces with two global regexes over the whole buffer. CPU per response grows with the square of the output length. Text can only parse to an object once it ends in } or ]. The processor now tracks the last non-whitespace character of the stream and only parses then. The brace-count branch re-parsed the same buffer.trim() that had just thrown or produced a non-object, so it could never return a result. I removed it. The emitted output does not change. Agents with tools send each turn's text as one chunk, so this mostly affects structured-output agents without tools, which stream token by token. Type of Change Bug fix New feature Breaking change Documentation Other: performance Testing executor/utils.test.ts passes unchanged. Biome is clean. I ran a throwaway fuzz with 2000 random outputs: nested objects with } inside strings, arrays, primitives, invalid JSON, trailing whitespace or text, and multibyte characters. Each was split into random 1 to 12 character chunks, and the old and new processors emitted identical streams. Timing setup: processStream over about 4-character chunks (like token deltas), pretty-printed JSON. Node 24.15 on Windows 11, 7 alternating rounds, median (min to max): output chunks before after 2 KB 516 10.8 ms (7.9 to 11.3) 1.6 ms (1.1 to 1.9) 10 KB 2,563 102.4 ms (96.6 to 105.3) 5.3 ms (4.6 to 13.1) 50 KB 12,814 1683.5 ms (1599.9 to 1753.5) 34.2 ms (28.6 to 39.3) This CPU time is spread over the stream rather than added to latency, since the model's token rate dominates. It still runs on the event loop that serves other requests. Checklist Code follows project style guidelines Self-reviewed my changes Tests added/updated and passing No new warnings introduced I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)
Open Graph Description: Summary A chat or run can select a response-format field of an Agent block (an output like agent1_answer). When the agent streams, StreamingResponseFormatProcessor calls JSON.parse(buffer.trim()) o...
X Description: Summary A chat or run can select a response-format field of an Agent block (an output like agent1_answer). When the agent streams, StreamingResponseFormatProcessor calls JSON.parse(buffer.trim()) o...
Opengraph URL: https://github.com/simstudioai/sim/pull/10087
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:e59a4e51-fe1e-d66f-a063-362da2b5f6f6 |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | BA2C:F138D:15AB3AE:1D93FDD:6AFB79C3 |
| html-safe-nonce | 67b08828cb63f917a820e419d6b10ad844f820618f2c344da9e885e1bf3208a7 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJCQTJDOkYxMzhEOjE1QUIzQUU6MUQ5M0ZERDo2QUZCNzlDMyIsInZpc2l0b3JfaWQiOiIxMDgxNjEwODgwMTkxMDAzMDc1IiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | 22a1c773f5a28c0db0390ccb951c989b44845d37bca91989b4cab08c48b2fcdf |
| hovercard-subject-tag | pull_request:5332972890 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/simstudioai/sim/pull/10087/files |
| twitter:image | https://avatars.githubusercontent.com/u/6395743?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/6395743?s=400&v=4 |
| og:image:alt | Summary A chat or run can select a response-format field of an Agent block (an output like agent1_answer). When the agent streams, StreamingResponseFormatProcessor calls JSON.parse(buffer.trim()) o... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | 0e29e7974fdda2feeaf5c891efd4b23b0734b23480232b62c7bdef13f40b5b48 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/simstudioai/sim git https://github.com/simstudioai/sim.git |
| octolytics-dimension-user_id | 199344406 |
| octolytics-dimension-user_login | simstudioai |
| octolytics-dimension-repository_id | 912559512 |
| octolytics-dimension-repository_nwo | simstudioai/sim |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 912559512 |
| octolytics-dimension-repository_network_root_nwo | simstudioai/sim |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 2e8872b770cc4d5d0cc1f2026afab747a268e2ce |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width