Title: fix(knowledge): stop retrying an embedding key with no credit left by waleedlatif1 · Pull Request #6868 · simstudioai/sim · GitHub
Open Graph Title: fix(knowledge): stop retrying an embedding key with no credit left by waleedlatif1 · Pull Request #6868 · simstudioai/sim
X Title: fix(knowledge): stop retrying an embedding key with no credit left by waleedlatif1 · Pull Request #6868 · simstudioai/sim
Description: Problem A connector sync ran the full hour and hit MAX_DURATION_EXCEEDED. The cause is not rate limiting. OpenAI answers an exhausted balance with 429, the same status as a rate limit: { "error": { "message": "You have no credits remaining. Add credits to continue using the API at ...", "type": "insufficient_quota", "code": "credit_balance_exhausted" } } isTransientEmbeddingError classified every 429 as transient, so these were retried like a rate limit. But a rate limit reopens and a spent account does not — no amount of waiting can make the next attempt succeed. That alone would be wasted work. What makes it persistent is the interaction with the stuck-document sweep: a failed document is re-queued on every sync, so each one burns its full retry budget again, forever. An account with no credit stops being a one-off failure and becomes permanent load, which is what consumed the hour. Measured in production before this change: Documents failed on insufficient_quota 2,286 across 2 knowledge bases Continuously retrying since 2026-07-27 (three weeks) Embedding failures from a genuine rate limit 0 Every terminal embedding 429 in production is an exhausted balance. Not one is a rate limit. Change The rejection body is what distinguishes them, so it is read when the error is constructed and recorded on the error. Retries against a spent key stop immediately. Retrying and failing over are decided separately, and that distinction is the point: Retry asks whether another attempt against the same provider could work. An exhausted balance rules that out. Failover asks whether a different provider could work. An exhausted balance says nothing about the next key in the chain. So the error stops the retries while remaining eligible for failover, rather than being reclassified as fatal. Why this matters for what just shipped The retry budget was widened in the previous PR (5 retries, 30s ceiling). That is correct for a real rate limit, but against a spent account it would have made this worse — 6 attempts per document instead of 4. The regression test reproduces exactly that: it fails at 6 attempts without this change and passes at 1. Verification 632 tests pass; 29/29 audits; type-check clean Three tests, each verified to fail without the change: the quota case stops at one attempt; a rate_limit_exceeded body with the same status still retries and recovers; an exhausted-balance error stays eligible for failover Note The underlying account still has no credit — this stops us from burning an hour per sync discovering that, and surfaces the provider's own actionable message ("add credits at ...") instead of retrying past it. The sweep re-queueing failed documents with no cooldown remains open and is the next thing worth fixing.
Open Graph Description: Problem A connector sync ran the full hour and hit MAX_DURATION_EXCEEDED. The cause is not rate limiting. OpenAI answers an exhausted balance with 429, the same status as a rate limit: { "erro...
X Description: Problem A connector sync ran the full hour and hit MAX_DURATION_EXCEEDED. The cause is not rate limiting. OpenAI answers an exhausted balance with 429, the same status as a rate limit: { "...
Opengraph URL: https://github.com/simstudioai/sim/pull/6868
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:8ae83592-67bd-d1cd-d35c-07a56883c3e0 |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | D38C:277BD5:1549DC1:1D174D8:6AFB783D |
| html-safe-nonce | ad16a87342747dc35f3d25ccec79cb2ec0ad45762825206180fafa792dbc1573 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJEMzhDOjI3N0JENToxNTQ5REMxOjFEMTc0RDg6NkFGQjc4M0QiLCJ2aXNpdG9yX2lkIjoiMzIyMDI2OTIxNjUwNjI3OTk5NyIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | 77ba86971453dc1f0b51248a3b792b38cce095c1917860351cc75c9470f0e507 |
| hovercard-subject-tag | pull_request:4317996991 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/simstudioai/sim/pull/6868/files |
| twitter:image | https://avatars.githubusercontent.com/u/40672544?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/40672544?s=400&v=4 |
| og:image:alt | Problem A connector sync ran the full hour and hit MAX_DURATION_EXCEEDED. The cause is not rate limiting. OpenAI answers an exhausted balance with 429, the same status as a rate limit: { "erro... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | 0e29e7974fdda2feeaf5c891efd4b23b0734b23480232b62c7bdef13f40b5b48 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/simstudioai/sim git https://github.com/simstudioai/sim.git |
| octolytics-dimension-user_id | 199344406 |
| octolytics-dimension-user_login | simstudioai |
| octolytics-dimension-repository_id | 912559512 |
| octolytics-dimension-repository_nwo | simstudioai/sim |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 912559512 |
| octolytics-dimension-repository_network_root_nwo | simstudioai/sim |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 2e8872b770cc4d5d0cc1f2026afab747a268e2ce |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width