René's URL Explorer Experiment


Title: feat(agent): validate structured outputs after generation with one retry by mzxchandra · Pull Request #7553 · simstudioai/sim · GitHub

Open Graph Title: feat(agent): validate structured outputs after generation with one retry by mzxchandra · Pull Request #7553 · simstudioai/sim

X Title: feat(agent): validate structured outputs after generation with one retry by mzxchandra · Pull Request #7553 · simstudioai/sim

Description: Problem The executor never validates agent-block structured output after generation. processStructuredResponse is a bare JSON.parse(content.trim()), and on parse failure it logs, attaches a _responseFormatWarning that nothing in the repo reads, and returns success via the standard-format fallback. Downstream blocks read the structured fields as undefined, indistinguishable from a legitimate empty answer. Two enforcement gaps make this worse: Native structured outputs are validated against a weakened schema. For models with nativeStructuredOutputs, the Anthropic provider passes the schema through the SDK's transformJSONSchema, which strips enum, const, minLength/maxLength, pattern, numeric minimum/maximum, and maxItems out of the enforced grammar and pastes them into description as advisory prose. Outputs are grammar-valid but can violate the authored constraints. Prompt-based models have zero enforcement — the schema is system-prompt text only, and nothing checked the result. Observed in production-shaped runs (~130 extraction workflows): enum-violating hollow rows emitted mid-degeneration, importance: 0 against a prose-only "1 to 10" bound, and 128k max_tokens blowups whose truncated JSON failed parse and silently became success. Change A new executor/handlers/agent/structured-output.ts validates every completed (non-streaming) structured response in the provider-request seam of the agent handler, treating three failure modes as one failure condition: Truncation — the final model trace segment's finishReason is max_tokens / length / MAX_TOKENS (case-insensitive). Every provider's trace enricher already records this on the model time segments, so no per-provider wiring is needed. Unparseable JSON — with a Markdown code-fence rescue before failing, since prompt-based models commonly fence otherwise-valid JSON. Schema violation — ajv (already a dependency; strict: false so user-authored schemas with unknown keywords still compile) validates against the authored schema, restoring enum, minLength, and numeric bounds to enforced status on both native and prompt-based paths. A schema ajv cannot compile degrades to the parse-only checks rather than failing runs. On failure: one identical resample, but only for requests without tools — tools execute inside the provider call, so a resample would replay their side effects. The failed attempt's tokens, cost, and trace segments are folded into the final result so the block's reported usage matches actual spend. A second failure (or the first, on a tool-carrying request) fails the block explicitly with the validation reason. Where the knob lives No new settings. Enforcement follows the response format's existing strict flag, which the block's input schema has always documented as defaulting to true and which parseResponseFormat already normalizes onto every format — it was previously read by nothing. "strict": false keeps the exact legacy lenient fallback (including the _responseFormatWarning). Deliberately not model-conditional at the surface: the UI's model value can be an unresolved reference or sim-auto (resolved only at execution), so a model-conditional field cannot know the actual model — and prompt-based models need validation strictly more. Deliberately not the block-level retry machinery: that wraps the entire handler and would replay memory reads, MCP discovery, and tool calls. Streaming responses are unchanged — there is no complete content to validate at this seam. Side benefit for schema authors With post-generation validation in place, minLength: 1, minimum/maximum, and maxItems in response format schemas become meaningful for the first time on every model. (Also worth knowing: format: "date" is in the SDK's supported-formats set, so date fields get native grammar enforcement just by declaring it.) Testing New structured-output.test.ts (17 tests): fixtures modeled on the observed corruption — hollow enum-violating rows, truncated JSON, fenced JSON, MAX_TOKENS/length finish reasons, uncompilable schemas, bare-schema formats, usage merging. Handler-level tests: retry-then-succeed with token merging and identical resample request, fail-after-retry, schema-violation retry, strict: false legacy fallback, no-resample-with-tools, max_tokens tripwire. Six existing provenance tests updated to return schema-valid mock content (their 'Mocked response content' default now correctly fails strict validation). bun run type-check, bun run check:api-validation, bun run check:boundaries, and the full agent-handler suite (115 tests) pass. Docs: the agent block's Response Format section now describes enforcement, the retry, and the strict: false opt-out. 🤖 Generated with Claude Code https://claude.ai/code/session_018L7CosQfKB9SGCF8E9fLfY

Open Graph Description: Problem The executor never validates agent-block structured output after generation. processStructuredResponse is a bare JSON.parse(content.trim()), and on parse failure it logs, attaches a _respon...

X Description: Problem The executor never validates agent-block structured output after generation. processStructuredResponse is a bare JSON.parse(content.trim()), and on parse failure it logs, attaches a _respon...

Opengraph URL: https://github.com/simstudioai/sim/pull/7553

X: @github

direct link

Domain: github.com

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf
route-pattern/:user_id/:repository/pull/:id/files(.:format)
route-controllerpull_requests
route-actionfiles
fetch-noncev2:9c4cb451-404d-bfd9-7ada-3bd59d8d32fd
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-id985E:6449C:CD735D:11AF383:6AFB5A84
html-safe-nonce35559e88f60bc6a63c10ef98699d0c33fd89475aa2a5c76493f8f231446e64b9
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI5ODVFOjY0NDlDOkNENzM1RDoxMUFGMzgzOjZBRkI1QTg0IiwidmlzaXRvcl9pZCI6IjY3NjAwOTM0ODk5MTc4MTEzMzIiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ==
visitor-hmaccafbe9f4530c2355f5f2ff04ab2d13130071eb5a1b931da04d33110c0d5e6619
hovercard-subject-tagpull_request:4453883629
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///pull_requests/show/files
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/simstudioai/sim/pull/7553/files
twitter:imagehttps://avatars.githubusercontent.com/u/129460234?s=400&v=4
twitter:cardsummary_large_image
og:imagehttps://avatars.githubusercontent.com/u/129460234?s=400&v=4
og:image:altProblem The executor never validates agent-block structured output after generation. processStructuredResponse is a bare JSON.parse(content.trim()), and on parse failure it logs, attaches a _respon...
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
Noneea33d13c96d0ff66bd8fa906b193d2213536b78cb06b0c50a4632c58b1418f94
turbo-cache-controlno-preview
diff-viewunified
go-importgithub.com/simstudioai/sim git https://github.com/simstudioai/sim.git
octolytics-dimension-user_id199344406
octolytics-dimension-user_loginsimstudioai
octolytics-dimension-repository_id912559512
octolytics-dimension-repository_nwosimstudioai/sim
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id912559512
octolytics-dimension-repository_network_root_nwosimstudioai/sim
turbo-body-classeslogged-out env-production page-responsive full-width
disable-turbotrue
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release59ef97164c52cf59bd10558ddec9c4f482763988
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/simstudioai/sim/pull/7553/files#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fsimstudioai%2Fsim%2Fpull%2F7553%2Ffiles
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fsimstudioai%2Fsim%2Fpull%2F7553%2Ffiles
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Ffiles&source=header-repo&source_repo=simstudioai%2Fsim
Reloadhttps://github.com/simstudioai/sim/pull/7553/files
Reloadhttps://github.com/simstudioai/sim/pull/7553/files
Reloadhttps://github.com/simstudioai/sim/pull/7553/files
simstudioaihttps://github.com/simstudioai
simhttps://github.com/simstudioai/sim
Codehttps://github.com/simstudioai/sim
Issues129 (129)https://github.com/simstudioai/sim/issues
Pull requests197 (197)https://github.com/simstudioai/sim/pulls
Actionshttps://github.com/simstudioai/sim/actions
Projectshttps://github.com/simstudioai/sim/projects
Security and qualityhttps://github.com/simstudioai/sim/security
Insightshttps://github.com/simstudioai/sim/pulse
Sign up for GitHub https://github.com/signup?return_to=%2Fsimstudioai%2Fsim%2Fissues%2Fnew%2Fchoose
terms of servicehttps://docs.github.com/terms
privacy statementhttps://docs.github.com/privacy
Sign inhttps://github.com/login?return_to=%2Fsimstudioai%2Fsim%2Fissues%2Fnew%2Fchoose
mzxchandrahttps://github.com/mzxchandra
staginghttps://github.com/simstudioai/sim/tree/staging
feat/agent-structured-output-validationhttps://github.com/simstudioai/sim/tree/feat/agent-structured-output-validation
Conversation 6 https://github.com/simstudioai/sim/pull/7553
Commits 1 https://github.com/simstudioai/sim/pull/7553/commits
Checks 31 https://github.com/simstudioai/sim/pull/7553/checks
Files changed https://github.com/simstudioai/sim/pull/7553/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/7553/files
feat(agent): validate structured outputs after generation with one retry https://github.com/simstudioai/sim/pull/7553/files#top
Show all changes 1 commit https://github.com/simstudioai/sim/pull/7553/files
a6bff70 feat(agent): validate structured outputs after generation with one retry mzxchandra Sep 6, 2026 https://github.com/simstudioai/sim/pull/7553/commits/a6bff7058fc28e0ffc50dc243e09c2d020e8d707
Clear filters https://github.com/simstudioai/sim/pull/7553/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/7553/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/7553/files
agent.mdx https://github.com/simstudioai/sim/pull/7553/files#diff-822548d545ef391aedc308c13740bef7e4aefe25a2f13be228c37d72418cc387
agent-handler.test.ts https://github.com/simstudioai/sim/pull/7553/files#diff-5fa11051a09b82360438ce6a447b4eeb6faa66fa2f668ce615f81fb4ea906f07
agent-handler.ts https://github.com/simstudioai/sim/pull/7553/files#diff-99cbd7f57b71719ca1561f15e7aecdc9f7a969ae5ea3589aee9fa64b3c9bf614
structured-output.test.ts https://github.com/simstudioai/sim/pull/7553/files#diff-0a16692195a4bb6d6768638ddc55241e1ff0f916bd45c420e23118db88e3a3a8
structured-output.ts https://github.com/simstudioai/sim/pull/7553/files#diff-2557d5c510809a6d35c6cd8329a9d528e7d6fd28a0b83283adf2954d0d3acbb1
apps/docs/content/docs/workflows/blocks/agent.mdxhttps://github.com/simstudioai/sim/pull/7553/files#diff-822548d545ef391aedc308c13740bef7e4aefe25a2f13be228c37d72418cc387
View file https://github.com/simstudioai/sim/blob/a6bff7058fc28e0ffc50dc243e09c2d020e8d707/apps/docs/content/docs/workflows/blocks/agent.mdx
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/7553/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/7553/files#diff-822548d545ef391aedc308c13740bef7e4aefe25a2f13be228c37d72418cc387
https://github.com/simstudioai/sim/pull/7553/files#diff-822548d545ef391aedc308c13740bef7e4aefe25a2f13be228c37d72418cc387
apps/sim/executor/handlers/agent/agent-handler.test.tshttps://github.com/simstudioai/sim/pull/7553/files#diff-5fa11051a09b82360438ce6a447b4eeb6faa66fa2f668ce615f81fb4ea906f07
View file https://github.com/simstudioai/sim/blob/a6bff7058fc28e0ffc50dc243e09c2d020e8d707/apps/sim/executor/handlers/agent/agent-handler.test.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/7553/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/7553/files#diff-5fa11051a09b82360438ce6a447b4eeb6faa66fa2f668ce615f81fb4ea906f07
https://github.com/simstudioai/sim/pull/7553/files#diff-5fa11051a09b82360438ce6a447b4eeb6faa66fa2f668ce615f81fb4ea906f07
https://github.com/simstudioai/sim/pull/7553/files#diff-5fa11051a09b82360438ce6a447b4eeb6faa66fa2f668ce615f81fb4ea906f07
https://github.com/simstudioai/sim/pull/7553/files#diff-5fa11051a09b82360438ce6a447b4eeb6faa66fa2f668ce615f81fb4ea906f07
https://github.com/simstudioai/sim/pull/7553/files#diff-5fa11051a09b82360438ce6a447b4eeb6faa66fa2f668ce615f81fb4ea906f07
https://github.com/simstudioai/sim/pull/7553/files#diff-5fa11051a09b82360438ce6a447b4eeb6faa66fa2f668ce615f81fb4ea906f07
https://github.com/simstudioai/sim/pull/7553/files#diff-5fa11051a09b82360438ce6a447b4eeb6faa66fa2f668ce615f81fb4ea906f07
https://github.com/simstudioai/sim/pull/7553/files#diff-5fa11051a09b82360438ce6a447b4eeb6faa66fa2f668ce615f81fb4ea906f07
https://github.com/simstudioai/sim/pull/7553/files#diff-5fa11051a09b82360438ce6a447b4eeb6faa66fa2f668ce615f81fb4ea906f07
https://github.com/simstudioai/sim/pull/7553/files#diff-5fa11051a09b82360438ce6a447b4eeb6faa66fa2f668ce615f81fb4ea906f07
https://github.com/simstudioai/sim/pull/7553/files#diff-5fa11051a09b82360438ce6a447b4eeb6faa66fa2f668ce615f81fb4ea906f07
https://github.com/simstudioai/sim/pull/7553/files#diff-5fa11051a09b82360438ce6a447b4eeb6faa66fa2f668ce615f81fb4ea906f07
https://github.com/simstudioai/sim/pull/7553/files#diff-5fa11051a09b82360438ce6a447b4eeb6faa66fa2f668ce615f81fb4ea906f07
https://github.com/simstudioai/sim/pull/7553/files#diff-5fa11051a09b82360438ce6a447b4eeb6faa66fa2f668ce615f81fb4ea906f07
https://github.com/simstudioai/sim/pull/7553/files#diff-5fa11051a09b82360438ce6a447b4eeb6faa66fa2f668ce615f81fb4ea906f07
Please reload this pagehttps://github.com/simstudioai/sim/pull/7553/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/7553/files
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.