Title: fix(browser): improve snapshots, native controls and action reliability by waleedlatif1 · Pull Request #7872 · simstudioai/sim · GitHub
Open Graph Title: fix(browser): improve snapshots, native controls and action reliability by waleedlatif1 · Pull Request #7872 · simstudioai/sim
X Title: fix(browser): improve snapshots, native controls and action reliability by waleedlatif1 · Pull Request #7872 · simstudioai/sim
Description: Companion PRs simstudioai/mothership#502 Merge together into staging so the browser tool contracts and desktop implementation stay aligned. Summary Preserve reachable click targets and retain obstruction/stale-target guards; use bounded native Electron capture with hidden-window preservation and no overlapping stalled captures. Preserve mixed inline snapshot text within the existing text budget and expose interactive headings as actionable references, retaining later controls on text-heavy pages. Support validated native structured inputs and atomic complete-set multi-select. Verify all selected values/labels after events, retain credential/focus checks, and never replay uncertain writes. Report unconfirmed effects clearly and remove obsolete capture decoding and nullable-image branches. Uses Playwright text-node traversal, explicit-role precedence, and native fill/complete-set selection. Sim adds prevalidation before mutation and full post-event readback. Native capture keeps the existing five-second deadline and cancellation semantics. Canonical tool descriptions and additive schemas are in Mothership #502; deploy with the updated desktop for the new values argument. Type of Change Bug fix Testing Real Electron against 22 unchanged MiniWoB++ tasks: 91/110 before → 109/110 after; fresh holdout 218/220. Snapshot instruction coverage 90/110 → 110/110. These are deterministic driver scripts, not an LLM benchmark score; episode timeout is 120 seconds. Multi-select uses the new complete-set API. Remaining failures are a bundled jQuery autocomplete defect: Playwright locator.click reproduces the same wrong selection in 10/10 controlled trials. No widget-specific workaround added. 1,644 desktop unit tests, 13 real Electron scenarios, all workspace type checks, lint, 46 audits and generated-artifact checks passed. Regression tests include atomic refusal, duplicate-value swaps, focus changes, malformed native values and uncertain-write handling. Full repository run passed all 19 test tasks: 51,810 web-app tests passed; dedicated database suites remain skipped without their fixture configuration. A separate 31-case generated-schema suite includes the eight new selection-contract cases. All seven structured input types are exercised in Electron. Checklist Code follows project style guidelines Self-reviewed my changes Tests added/updated and passing No new warnings introduced I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)
Open Graph Description: Companion PRs simstudioai/mothership#502 Merge together into staging so the browser tool contracts and desktop implementation stay aligned. Summary Preserve reachable click targets and retain o...
X Description: Companion PRs simstudioai/mothership#502 Merge together into staging so the browser tool contracts and desktop implementation stay aligned. Summary Preserve reachable click targets and retain o...
Opengraph URL: https://github.com/simstudioai/sim/pull/7872
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:3d5547db-b59d-34f1-5fb3-aeab5de23994 |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | 8A12:23CE74:207C49:2D6707:6AFB3041 |
| html-safe-nonce | 2cbbc4002c0509515c56f2dc3a6a1faa172c42531e7f4097f5dff7059ba773b5 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI4QTEyOjIzQ0U3NDoyMDdDNDk6MkQ2NzA3OjZBRkIzMDQxIiwidmlzaXRvcl9pZCI6IjExOTgwODEzMDUxMDc3MDU5MjIiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | 2cac4bce1842490f3dafa05a7cc7c5b996412edc1787e45eccf72b2aeb0d920e |
| hovercard-subject-tag | pull_request:4542679393 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/simstudioai/sim/pull/7872/files |
| twitter:image | https://avatars.githubusercontent.com/u/40672544?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/40672544?s=400&v=4 |
| og:image:alt | Companion PRs simstudioai/mothership#502 Merge together into staging so the browser tool contracts and desktop implementation stay aligned. Summary Preserve reachable click targets and retain o... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | efe184de15c1d8c1107a340cfc27bf7187b458dc07687ee1fa51f035148fb3d1 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/simstudioai/sim git https://github.com/simstudioai/sim.git |
| octolytics-dimension-user_id | 199344406 |
| octolytics-dimension-user_login | simstudioai |
| octolytics-dimension-repository_id | 912559512 |
| octolytics-dimension-repository_nwo | simstudioai/sim |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 912559512 |
| octolytics-dimension-repository_network_root_nwo | simstudioai/sim |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | f567cf3af46b50a3211fb5dab569a9d6a7b58025 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width