Title: fix(knowledge): give failed documents a grace while their retries are still scheduled by waleedlatif1 · Pull Request #6924 · simstudioai/sim · GitHub
Open Graph Title: fix(knowledge): give failed documents a grace while their retries are still scheduled by waleedlatif1 · Pull Request #6924 · simstudioai/sim
X Title: fix(knowledge): give failed documents a grace while their retries are still scheduled by waleedlatif1 · Pull Request #6924 · simstudioai/sim
Description: Live on staging. Found by an independent audit of the merged connector work — an interaction between #6920 and #6921, each correct alone. The bug The sweep treats failed as terminal, documented as "the run that produced it has ended." That was true when processing ran inline. Since #6920 made dispatch asynchronous: the worker writes processingStatus: 'failed' and then rethrows the Trigger task retries up to maxAttempts (default 3) So between attempts the row reads failed while a live run is scheduled to retry it. The sweep gave failed zero grace, so it reclaimed the document, deleted its embeddings, and re-dispatched with a fresh pass id — two live runs, two indexing passes, two bills. Exactly what #6911 and #6921 exist to prevent, reached through the one state deliberately left unguarded. Backoff between attempts is short, so the window is seconds per document — but it is per document across a 7,730-document corpus, and an OOM-kill reschedule is far longer than the configured backoff. The fix Aged from processingCompletedAt, which every failure write stamps, so it reads exactly when the last attempt ended rather than when the document was dispatched. Sized at the queue grace, not a run-duration bound. This was the one wrong turn available: STALE_PROCESSING_MINUTES bounds a run, but what is being waited on between attempts is a re-queue behind the same global concurrency limit. On a large backlog the next attempt starts hours after the previous one ended, so a 45-minute grace would still reclaim documents mid-retry-chain — just less often, which is the worst kind of fix. The grace is now an expression, not a constant: ceil(LARGEST_OBSERVED_CORPUS / QUEUE_CONCURRENCY × OCCUPANCY × CONTENTION) = ceil(7730 / 20 × 1 × 2) = 773 minutes Each input is documented with where to re-measure it. The previous 240 was derived from a 2,600-document corpus and sat below the drain time of the 7,730-document connector it was written for. Rounding is deliberately up: too small silently re-bills live work; too large only delays recovery of documents nothing is processing. The test pins 773 so changing an input fails loudly rather than being absorbed. Recovery for a genuinely lost dispatch becomes grace + sync interval — free at the 1,440-minute default interval, and still an order of magnitude inside RETRY_WINDOW_DAYS. Also: withdrawing an unearned stamp The queue stamp is written before dispatch, so it records intent rather than fact. That ordering is deliberate and stays — a chunked batch can half-succeed, and stamping after dispatch would strand the documents that genuinely are executing. Before-dispatch fails safe; after-dispatch fails unsafe. But in the total-failure branch we can prove nothing dispatched, so the stamp is withdrawn there before it throws. Scoped three ways: to the batch's own ids, to rows still pending, and by compare-and-set on the exact stamp that call wrote — so a concurrent batch that has already re-stamped a document keeps its grace. The withdrawal is best-effort while the stamp write still throws. Asymmetric on purpose: a failed stamp means the grace cannot be promised and dispatching anyway is the unsafe direction, whereas a failed withdrawal only delays recovery by one grace and must not mask the dispatch error underneath it. Verification type-check clean · check:audits 32/32 · 754 tests across lib/knowledge and all four knowledge API suites. Ten mutations, all red. The flagship — restoring failed to unconditionally sweepable, i.e. exactly today's staging behavior — turns four tests red. Three more separately kill each scoping guarantee on the withdrawal (own ids, pending guard, own-stamp CAS), and two kill the derivation inputs. One thing the tests cannot cover, stated rather than implied: that the withdrawal fires only on total failure. Under test isTriggerAvailable() is false, so every path goes in-process and fails — there is no way to make a dispatch succeed, so a mutation moving the clear outside the dispatched === 0 branch would not be caught. That branch is three lines and reviewable by eye. Follow-up, deliberately not here A processing_attempts column would convert the deterministic-OOM case from an unbounded seven-day re-sweep loop into a real resolution: increment inside the funnel's existing guarded write, reset on successful completion, sweep skips above N, leaving the document at failed with its error visible and the user's retry button still working. The same counter doubles as a dispatch generation — a worker carrying the value it was dispatched with can decline to bill when the row has moved past it, which is the state-based guarantee a timing grace structurally cannot give. Those two belong in one change rather than two columns a month apart, and both deserve their own review rather than riding on a hotfix.
Open Graph Description: Live on staging. Found by an independent audit of the merged connector work — an interaction between #6920 and #6921, each correct alone. The bug The sweep treats failed as terminal, documented as ...
X Description: Live on staging. Found by an independent audit of the merged connector work — an interaction between #6920 and #6921, each correct alone. The bug The sweep treats failed as terminal, documented as ...
Opengraph URL: https://github.com/simstudioai/sim/pull/6924
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:07877e82-f1d3-3119-f4ae-648bc9f000ee |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | D5D4:FBC06:F7F3C8:1556CE0:6AFB657D |
| html-safe-nonce | 94f36004896834b8f142289231510898067f1360ce2bc1b1cf29f79ca95e154e |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJENUQ0OkZCQzA2OkY3RjNDODoxNTU2Q0UwOjZBRkI2NTdEIiwidmlzaXRvcl9pZCI6IjU3NTc0NjA3NzY1NTc5NjI2MjIiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | e441540930e74ab4ba3065ad52debf93fdff49a9ae614590c765aaa969700e0d |
| hovercard-subject-tag | pull_request:4328185050 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/simstudioai/sim/pull/6924/files |
| twitter:image | https://avatars.githubusercontent.com/u/40672544?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/40672544?s=400&v=4 |
| og:image:alt | Live on staging. Found by an independent audit of the merged connector work — an interaction between #6920 and #6921, each correct alone. The bug The sweep treats failed as terminal, documented as ... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | ea33d13c96d0ff66bd8fa906b193d2213536b78cb06b0c50a4632c58b1418f94 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/simstudioai/sim git https://github.com/simstudioai/sim.git |
| octolytics-dimension-user_id | 199344406 |
| octolytics-dimension-user_login | simstudioai |
| octolytics-dimension-repository_id | 912559512 |
| octolytics-dimension-repository_nwo | simstudioai/sim |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 912559512 |
| octolytics-dimension-repository_network_root_nwo | simstudioai/sim |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 59ef97164c52cf59bd10558ddec9c4f482763988 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width