Title: fix(files): read a file nobody vouched for, as an untracked one already is by icecrasher321 · Pull Request #6986 · simstudioai/sim · GitHub
Open Graph Title: fix(files): read a file nobody vouched for, as an untracked one already is by icecrasher321 · Pull Request #6986 · simstudioai/sim
X Title: fix(files): read a file nobody vouched for, as an untracked one already is by icecrasher321 · Pull Request #6986 · simstudioai/sim
Description: Summary A workspace file whose provenance was never recorded was refused by every model and runtime boundary at once: attachments, sandbox mounts, and the tool routes that parse, transcribe or describe a file. A file that was never tracked returned exact-empty two branches earlier and worked fine. Both say the same thing about their contents — nothing. So the second was a permanent penalty for having tried to record provenance and failed, with no way back: nothing rewrites a file's provenance but another content write. 107 live files across 10 workspaces are sitting in it, 85 of them images, which is exactly what the vision/OCR/transcription routes consume. This is the pathology the enforcement module already names for the other surfaces — "an aware writer that momentarily could not vouch strictly worse than an unaware one… a workspace could not recover without a data repair." Files were held out of that list pending their own decision; this is that decision. workspace-file is now a real surface in DURABLE_SECRET_PROVENANCE_SURFACES, where the env documentation has always offered it — setting it previously logged "unrecognized" and silently did nothing. The four governed boundaries read an unrecorded file and record the same workspace-visible secret_provenance.unrecorded audit entry the other surfaces record. Only absence is relaxed. The classifier collapsed a missing row, a content-version mismatch, a stale binding, and a malformed sidecar into the same unknown that a recorded absence produces. It now separates them, so the policy cannot reach the four that are not absences. Consumers that have not opted into the surface's policy (ffmpeg, knowledge ingest, the file-manage accumulator) keep refusing exactly as before. Legacy is untouched: secret_provenance_version === null still returns exact-empty on the branch above, unchanged. 0007 clears the files already in the state, which would otherwise report on every read forever. Review fixes Two real defects, both caught in review, both now fixed and pinned by tests that fail against the previous code. An unrecorded contribution was laundered into an exact one. mergeWorkspaceFileSecretProvenance checked only for unknown, so unrecorded fell through to the exact branch — combining bytes nobody vouched for with bytes that were vouched for produced a positive claim that neither input made, and a derived file could hand a later boundary an exact-empty it had no basis for. Absence now survives the merge: unknown > unrecorded > exact. 0007 reversed the live writer's lock order. The content-update path updates workspace_files and then replaces the sidecar, one transaction, parent-first. The repair deleted the sidecar and then updated the parent — opposite order, so an overlapping upload deadlocks and Postgres aborts either the deployment or a tenant's file write. It now takes the parent lock first in id order, matching the writer and 0005 beside it, which is also what makes the status = 'unknown' re-check decisive rather than racy. An initialization could send the database a value it rejects. The union carries three states; the sidecar's CHECK constraint accepts ('exact', 'unknown'). Only the replace path discriminated — initializeWorkspaceFileSecretProvenanceInTx forwarded the status verbatim, so an 'unrecorded' would have hit a constraint violation and aborted the enclosing transaction. Unreachable today (both callers pass exact-empty), but the signature permitted a value the column does not, which is a landmine rather than a bug. Found while auditing every consumer of the widened union. One reported concern was checked and rejected: archive extraction flattens 'unrecorded' to 'unknown' before persisting, which sounded like it would strand extracted files in the unrelaxable state. It does not — replace writes 'unknown' for any non-exact input and that reads back as 'unrecorded', so both values persist identically and both round-trip to the relaxable state. The doc comment claiming there was nothing here to deadlock against was false — written without checking, one PR after the same bug was caught on the table-row repair — and is gone. The ordering is now pinned by a test over both repairs, since reasoning about it has failed twice. The premise this PR started from was wrong Relaxing a stored unknown assumed it always meant "nobody recorded this". It did not. The sidecar collapsed two opposite claims into one stored value: Absence — no registry ran, so nothing was written down. Says exactly what an untracked file says. Taint — a writer knew secrets were present and refused: a child of an archive whose parent held secret entries (archive.ts:327), a generated asset whose decision.safe came back false, a transcode whose scanner had hasSecrets: true and could not locate them. Relaxing the second would have walked known-secret-bearing content through every model boundary at once. Both reviewers caught this independently. The decision type already separated them — a registry that never ran returns safe: true, every refusal returns safe: false — and only storage lost it. Migration 0302 widens the CHECK to accept a third status; writers persist unrecorded and unknown separately; readers relax the first and refuse the second. The attachment test asserts both directions on one call. Only workspace files store this distinction, and that is deliberate. On every other durable surface the non-exact sidecar comes from a single condition — an incomplete incoming bundle or registry — which is always an absence, so two statuses say everything there is to say. Files are the only surface that derives one stored object from another, so they are the only one that can refuse on purpose. The shared policy is unchanged and uniform across all five: read an absence and audit it, refuse a taint. That invariant is now written down on DURABLE_SECRET_PROVENANCE_SURFACES so the asymmetry is not mistaken for drift in either direction. Type of Change Bug fix Testing Tested manually. bun run lint, all 24 CI audits, check:migrations, and type-check pass. 5,836 tests pass across the uploads, provenance, executor, knowledge, copilot and file-tool suites; packages/db 60. Three pre-existing tests encoded the old fail-closed decision and were updated deliberately — the unrecorded attachment is now kept, an unrecorded mount now proceeds without importing, and the mixed egress test keeps refusing tainted, stale and cross-workspace keys with only the unrecorded row removed. Two others were failing wrongly mid-change and caught a real bug: the first cut of this applied the policy to a content-version mismatch and a missing row, which it must not. Each of the two review fixes has a test verified to fail against the reverted fix before being kept — the merge one on precedence, the migration one on statement order across both repairs. Checklist Code follows project style guidelines Self-reviewed my changes Tests added/updated and passing No new warnings introduced I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)
Open Graph Description: Summary A workspace file whose provenance was never recorded was refused by every model and runtime boundary at once: attachments, sandbox mounts, and the tool routes that parse, transcribe or des...
X Description: Summary A workspace file whose provenance was never recorded was refused by every model and runtime boundary at once: attachments, sandbox mounts, and the tool routes that parse, transcribe or des...
Opengraph URL: https://github.com/simstudioai/sim/pull/6986
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:11f48780-d9c0-7e25-edf5-369a7235f340 |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | 9216:3BD2EB:1193CB:18A3CE:6AFB4C8A |
| html-safe-nonce | 7d92dd195dd592b78cb7a81dd26c8d9cd96957769980fac6bade9852bce5ecce |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI5MjE2OjNCRDJFQjoxMTkzQ0I6MThBM0NFOjZBRkI0QzhBIiwidmlzaXRvcl9pZCI6IjMzMzM5NjUxMDcwMzE1NjU4IiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | c335f23dc6a8c813cd7943c98122877c9ae4527c519f079a1a55025ce55918a5 |
| hovercard-subject-tag | pull_request:4339682159 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/simstudioai/sim/pull/6986/files |
| twitter:image | https://avatars.githubusercontent.com/u/17950982?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/17950982?s=400&v=4 |
| og:image:alt | Summary A workspace file whose provenance was never recorded was refused by every model and runtime boundary at once: attachments, sandbox mounts, and the tool routes that parse, transcribe or des... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | acf7616d06cc051644705b379878c9bb13dc5e70ecf4ffd9b34140f8ac01d7e1 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/simstudioai/sim git https://github.com/simstudioai/sim.git |
| octolytics-dimension-user_id | 199344406 |
| octolytics-dimension-user_login | simstudioai |
| octolytics-dimension-repository_id | 912559512 |
| octolytics-dimension-repository_nwo | simstudioai/sim |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 912559512 |
| octolytics-dimension-repository_network_root_nwo | simstudioai/sim |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 26504db649b58525278f77a52b80ba5f10d5f4de |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width