Title: fix(knowledge): stop billing a document once per processing attempt by waleedlatif1 · Pull Request #6911 · simstudioai/sim · GitHub
Open Graph Title: fix(knowledge): stop billing a document once per processing attempt by waleedlatif1 · Pull Request #6911 · simstudioai/sim
X Title: fix(knowledge): stop billing a document once per processing attempt by waleedlatif1 · Pull Request #6911 · simstudioai/sim
Description: recordUsage built its sourceReference as knowledge-document:{documentId}:{startTime}, where startTime is captured per attempt. That string is what usage_log hashes into its dedup eventKey, so the timestamp defeated the deduplication by construction. knowledge-process-document runs up to 3 attempts, and the connector sync's stuck-document sweep can re-dispatch on top of that — so one document could be billed four times for a single indexing pass. The key knowledge-document:{documentId}:{indexingPassId}, where indexingPassId is the dispatch requestId. That is already the codebase's own name for one indexing pass: dispatchViaBatchTrigger uses it in doc-process-{documentId}-{requestId} with a comment saying it blocks intra-dispatch retries while later syncs mint a fresh one. It is fixed across all attempts of a run and fresh on every new dispatch — exactly "same work retried" versus "new work". Two alternatives were rejected: documentId alone dedupes retries but would suppress billing for a legitimate re-index — a content change, a rehydrate, a user reprocess. usage_log has no retention (verified: no delete path anywhere; rows only disappear via the user FK cascade), so that under-bill would be permanent, not windowed. Including the token count looks like a sharper discriminator, but OCR- and LLM-backed parsing is not bit-stable across attempts, so a differing count on attempt 2 would defeat the very dedup being added. Falls back to the resolved embedding pricing id where no pass id is threaded — still collapses attempts, and still re-bills a KB whose embedding model changed. The half that would have been missed background/knowledge-processing.ts destructured requestId for logging and dropped it before calling processDocumentAsync. On the Trigger path — the only path that actually retries — the new parameter would have arrived undefined and only the weaker fallback would apply. Now threaded. The existing worker tests pin the exact argument list and caught the signature change, which is what they are for; they now assert the threaded value. Verification type-check clean · check:audits 32/32 · 659 tests across lib/knowledge and background. Mutation-checked: restoring the timestamp turns all 5 new tests red. Collapsing to documentId-only granularity turns 4 red and correctly leaves the stability test green — confirming the stability and discrimination assertions fail independently rather than keying off one condition. Dropping the threaded requestId turns both worker tests red. Swept, not changed Every other sourceReference producer in apps/sim was checked. The knowledge-search ones key on a per-HTTP-request id, which is correct — each search genuinely incurs a fresh embedding cost and searches are not retried under a shared identity. The enrichment producer is content-scoped. None carries a per-attempt value.
Open Graph Description: recordUsage built its sourceReference as knowledge-document:{documentId}:{startTime}, where startTime is captured per attempt. That string is what usage_log hashes into its dedup eventKey, so the t...
X Description: recordUsage built its sourceReference as knowledge-document:{documentId}:{startTime}, where startTime is captured per attempt. That string is what usage_log hashes into its dedup eventKey, so the t...
Opengraph URL: https://github.com/simstudioai/sim/pull/6911
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:1121de51-1dc2-53e4-0721-270a91882a7a |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | BEE8:A82D7:2059:2ABD:6AFB63EE |
| html-safe-nonce | d55a97594fa571f65e091cadf59d3afbaf080a1e686384c2be0ddf672062417d |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJCRUU4OkE4MkQ3OjIwNTk6MkFCRDo2QUZCNjNFRSIsInZpc2l0b3JfaWQiOiI0Mzk1ODI0OTc1NDI0NzQyMzgzIiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | 7f249b2cc759e180576481d90e94c3c47d869e3d54649ccac072f33e80dcea94 |
| hovercard-subject-tag | pull_request:4327501308 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/simstudioai/sim/pull/6911/files |
| twitter:image | https://avatars.githubusercontent.com/u/40672544?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/40672544?s=400&v=4 |
| og:image:alt | recordUsage built its sourceReference as knowledge-document:{documentId}:{startTime}, where startTime is captured per attempt. That string is what usage_log hashes into its dedup eventKey, so the t... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | ea33d13c96d0ff66bd8fa906b193d2213536b78cb06b0c50a4632c58b1418f94 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/simstudioai/sim git https://github.com/simstudioai/sim.git |
| octolytics-dimension-user_id | 199344406 |
| octolytics-dimension-user_login | simstudioai |
| octolytics-dimension-repository_id | 912559512 |
| octolytics-dimension-repository_nwo | simstudioai/sim |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 912559512 |
| octolytics-dimension-repository_network_root_nwo | simstudioai/sim |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 59ef97164c52cf59bd10558ddec9c4f482763988 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width