Title: fix(knowledge): write external group membership as a diff by waleedlatif1 · Pull Request #7943 · simstudioai/sim · GitHub
Open Graph Title: fix(knowledge): write external group membership as a diff by waleedlatif1 · Pull Request #7943 · simstudioai/sim
X Title: fix(knowledge): write external group membership as a diff by waleedlatif1 · Pull Request #7943 · simstudioai/sim
Description: Summary Every directory sync deleted a group's entire external membership and reinserted it, whether or not anything had changed. That took the membership table to ~55M lifetime inserts and ~55M deletes against ~127k live rows — about 430x write amplification — holding it near 91% dead tuples across 2,447 autovacuum cycles, an order of magnitude more churn than any comparable table. That vacuum load competes for the same I/O as every other query on the instance. Membership is now written as a difference: read the group's current tokens, delete only what the enumeration no longer lists, insert only what it newly lists. An unchanged group writes nothing. The group row is locked before that read. The two callers fence on different leases — the directory lease and the connector sync lease — so neither excludes the other, and on the directory path the group upsert commits in a separate transaction, so its row lock is already released. Without the lock the removal set is computed against a snapshot a concurrent pass may have moved past, and a subject that pass inserted would survive an enumeration that never observed it: membership retained rather than revoked. The lock also closes the narrower window the blind delete already had between its delete and its commit. created_at on a member row now means first-observed rather than last-observed. No reader projects it, and the group's lastSyncedAt remains the freshness signal and is still written every pass. Also corrects the processing-queue TSDoc, which argued the per-tenant lane limits were sized rather than inherited, and cited a concurrency figure derived from an unsound measurement. Type of Change Bug fix Testing 3,150 tests pass across lib/knowledge and background; one pre-existing failure that needs DATABASE_URL bun run type-check clean, bun run lint clean, all 46 audits pass, docs manifest in sync Four tests cover it — unchanged group writes nothing, changed group writes only the delta, the group row is locked before the read, and the partial-sync case still writes the accessible group — each verified to fail when the fix is reverted Concurrency reasoning is from the lease and transaction structure, not an executed concurrent test; a Postgres integration test running two passes against one group would be the way to prove it empirically Checklist Code follows project style guidelines Self-reviewed my changes Tests added/updated and passing No new warnings introduced I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)
Open Graph Description: Summary Every directory sync deleted a group's entire external membership and reinserted it, whether or not anything had changed. That took the membership table to ~55M lifetime inserts and ~5...
X Description: Summary Every directory sync deleted a group's entire external membership and reinserted it, whether or not anything had changed. That took the membership table to ~55M lifetime inserts an...
Opengraph URL: https://github.com/simstudioai/sim/pull/7943
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:958365e9-7dc5-9de3-2590-2676a11eb258 |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | A106:1D6C4E:CCFAC:120413:6AFB758A |
| html-safe-nonce | 4439e84c22166c8dcaead3d5f131a0fd533e4c0abc55e416337439ea702d7b37 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJBMTA2OjFENkM0RTpDQ0ZBQzoxMjA0MTM6NkFGQjc1OEEiLCJ2aXNpdG9yX2lkIjoiMzUzMDM4MDg0MDQxODM3NTA1MCIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | d31775687ee005240c13434bc82a1f974663e545c72e8e033d74dd9cfae4d5f5 |
| hovercard-subject-tag | pull_request:4562893364 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/simstudioai/sim/pull/7943/files |
| twitter:image | https://avatars.githubusercontent.com/u/40672544?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/40672544?s=400&v=4 |
| og:image:alt | Summary Every directory sync deleted a group's entire external membership and reinserted it, whether or not anything had changed. That took the membership table to ~55M lifetime inserts and ~5... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | 0e29e7974fdda2feeaf5c891efd4b23b0734b23480232b62c7bdef13f40b5b48 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/simstudioai/sim git https://github.com/simstudioai/sim.git |
| octolytics-dimension-user_id | 199344406 |
| octolytics-dimension-user_login | simstudioai |
| octolytics-dimension-repository_id | 912559512 |
| octolytics-dimension-repository_nwo | simstudioai/sim |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 912559512 |
| octolytics-dimension-repository_network_root_nwo | simstudioai/sim |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 2e8872b770cc4d5d0cc1f2026afab747a268e2ce |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width