Title: feat(agent): validate structured outputs after generation with one retry by mzxchandra · Pull Request #7553 · simstudioai/sim · GitHub
Open Graph Title: feat(agent): validate structured outputs after generation with one retry by mzxchandra · Pull Request #7553 · simstudioai/sim
X Title: feat(agent): validate structured outputs after generation with one retry by mzxchandra · Pull Request #7553 · simstudioai/sim
Description: Problem
The executor never validates agent-block structured output after generation. processStructuredResponse is a bare JSON.parse(content.trim()), and on parse failure it logs, attaches a _responseFormatWarning that nothing in the repo reads, and returns success via the standard-format fallback. Downstream blocks read the structured fields as undefined, indistinguishable from a legitimate empty answer.
Two enforcement gaps make this worse:
Native structured outputs are validated against a weakened schema. For models with nativeStructuredOutputs, the Anthropic provider passes the schema through the SDK's transformJSONSchema, which strips enum, const, minLength/maxLength, pattern, numeric minimum/maximum, and maxItems out of the enforced grammar and pastes them into description as advisory prose. Outputs are grammar-valid but can violate the authored constraints.
Prompt-based models have zero enforcement — the schema is system-prompt text only, and nothing checked the result.
Observed in production-shaped runs (~130 extraction workflows): enum-violating hollow rows emitted mid-degeneration, importance: 0 against a prose-only "1 to 10" bound, and 128k max_tokens blowups whose truncated JSON failed parse and silently became success.
Change
A new executor/handlers/agent/structured-output.ts validates every completed (non-streaming) structured response in the provider-request seam of the agent handler, treating three failure modes as one failure condition:
Truncation — the final model trace segment's finishReason is max_tokens / length / MAX_TOKENS (case-insensitive). Every provider's trace enricher already records this on the model time segments, so no per-provider wiring is needed.
Unparseable JSON — with a Markdown code-fence rescue before failing, since prompt-based models commonly fence otherwise-valid JSON.
Schema violation — ajv (already a dependency; strict: false so user-authored schemas with unknown keywords still compile) validates against the authored schema, restoring enum, minLength, and numeric bounds to enforced status on both native and prompt-based paths. A schema ajv cannot compile degrades to the parse-only checks rather than failing runs.
On failure: one identical resample, but only for requests without tools — tools execute inside the provider call, so a resample would replay their side effects. The failed attempt's tokens, cost, and trace segments are folded into the final result so the block's reported usage matches actual spend. A second failure (or the first, on a tool-carrying request) fails the block explicitly with the validation reason.
Where the knob lives
No new settings. Enforcement follows the response format's existing strict flag, which the block's input schema has always documented as defaulting to true and which parseResponseFormat already normalizes onto every format — it was previously read by nothing. "strict": false keeps the exact legacy lenient fallback (including the _responseFormatWarning).
Deliberately not model-conditional at the surface: the UI's model value can be an unresolved
Open Graph Description: Problem The executor never validates agent-block structured output after generation. processStructuredResponse is a bare JSON.parse(content.trim()), and on parse failure it logs, attaches a _respon...
X Description: Problem The executor never validates agent-block structured output after generation. processStructuredResponse is a bare JSON.parse(content.trim()), and on parse failure it logs, attaches a _respon...
Opengraph URL: https://github.com/simstudioai/sim/pull/7553
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:9c4cb451-404d-bfd9-7ada-3bd59d8d32fd |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | 985E:6449C:CD735D:11AF383:6AFB5A84 |
| html-safe-nonce | 35559e88f60bc6a63c10ef98699d0c33fd89475aa2a5c76493f8f231446e64b9 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI5ODVFOjY0NDlDOkNENzM1RDoxMUFGMzgzOjZBRkI1QTg0IiwidmlzaXRvcl9pZCI6IjY3NjAwOTM0ODk5MTc4MTEzMzIiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | cafbe9f4530c2355f5f2ff04ab2d13130071eb5a1b931da04d33110c0d5e6619 |
| hovercard-subject-tag | pull_request:4453883629 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/simstudioai/sim/pull/7553/files |
| twitter:image | https://avatars.githubusercontent.com/u/129460234?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/129460234?s=400&v=4 |
| og:image:alt | Problem The executor never validates agent-block structured output after generation. processStructuredResponse is a bare JSON.parse(content.trim()), and on parse failure it logs, attaches a _respon... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | ea33d13c96d0ff66bd8fa906b193d2213536b78cb06b0c50a4632c58b1418f94 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/simstudioai/sim git https://github.com/simstudioai/sim.git |
| octolytics-dimension-user_id | 199344406 |
| octolytics-dimension-user_login | simstudioai |
| octolytics-dimension-repository_id | 912559512 |
| octolytics-dimension-repository_nwo | simstudioai/sim |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 912559512 |
| octolytics-dimension-repository_network_root_nwo | simstudioai/sim |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 59ef97164c52cf59bd10558ddec9c4f482763988 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width