René's URL Explorer Experiment


Title: fix(knowledge): replace deletion-safety heuristics with a two-phase tombstone by waleedlatif1 · Pull Request #5884 · simstudioai/sim · GitHub

Open Graph Title: fix(knowledge): replace deletion-safety heuristics with a two-phase tombstone by waleedlatif1 · Pull Request #5884 · simstudioai/sim

X Title: fix(knowledge): replace deletion-safety heuristics with a two-phase tombstone by waleedlatif1 · Pull Request #5884 · simstudioai/sim

Description: Summary Follow-up to #5883, which merged before this redesign was ready. #5883 fixed the immediate bug (trashed Google Sheets tabs never purging on a normal sync) with sourceConfirmedEmpty — a connector-set flag bypassing two static safety heuristics (shouldSkipEmptyListing, exceedsDeletionSafetyThreshold). On reflection that approach was the wrong shape: a boolean escape hatch that any future connector could set to bypass both deletion safety nets at once, with no protection against the source itself being transiently wrong. This PR replaces it with a proper two-phase tombstone, matching how production sync systems actually solve this (Microsoft Entra Connect's deletion threshold, SCIM/Entra ID's deferred-delete deprovisioning, Cassandra/Couchbase tombstones): never let a single observation trigger an irreversible mass deletion, no matter how confident the signal looks. A document missing from a normal sync's listing is now soft-deleted (marked pending-removal), not purged immediately. It's only hard-deleted once a later sync confirms it's still absent. If it reappears in between, it's resurrected automatically — self-heals a transient outage or bad API response without needing to distinguish "real" emptiness from "ambiguous" emptiness at all. A forced full sync still purges everything absent in one pass — the existing "trigger a full sync to force cleanup" escape hatch is preserved. Removes shouldSkipEmptyListing, exceedsDeletionSafetyThreshold, and sourceConfirmedEmpty entirely. Google Sheets no longer needs any connector-specific bypass — it goes through the exact same general path as all 8 connectors, with zero special-casing and zero misuse surface for future connectors. Uses the existing (previously unused for individual documents) document.deletedAt column as the tombstone marker — no schema migration. shouldReconcileDeletions (the isIncremental/listingCapped/listingTruncated gate) is unchanged. Resurrection runs unconditionally even when that gate is closed, since presence is trustworthy evidence regardless of listing completeness. Type of Change Bug fix / design improvement Testing Added 7 new unit tests for partitionSyncReconciliation (resurrect / soft-delete / hard-delete / mixed-batch / fullSync / null-externalId cases), replacing the removed heuristic tests. bun run lint, tsc --noEmit, and the full connectors + lib/knowledge + background/knowledge-connector-sync test suites (508 tests) all pass. Ran /cleanup's full 8-pass audit — no dead code, no stale/redundant comments, nothing to fix. Checklist Code follows project style guidelines Self-reviewed my changes Tests added/updated and passing No new warnings introduced I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

Open Graph Description: Summary Follow-up to #5883, which merged before this redesign was ready. #5883 fixed the immediate bug (trashed Google Sheets tabs never purging on a normal sync) with sourceConfirmedEmpty — a conn...

X Description: Summary Follow-up to #5883, which merged before this redesign was ready. #5883 fixed the immediate bug (trashed Google Sheets tabs never purging on a normal sync) with sourceConfirmedEmpty — a conn...

Opengraph URL: https://github.com/simstudioai/sim/pull/5884

X: @github

direct link

Domain: github.com

route-pattern/:user_id/:repository/pull/:id/files(.:format)
route-controllerpull_requests
route-actionfiles
fetch-noncev2:fe18106e-9709-5f48-700f-e3ac041715b3
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-idC836:19F67D:89811C:C3DDB9:6A631AE3
html-safe-nonceae6072fe8613432ba6eace351acb92adf6afe00001cae89e6708ac5c398ba478
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJDODM2OjE5RjY3RDo4OTgxMUM6QzNEREI5OjZBNjMxQUUzIiwidmlzaXRvcl9pZCI6IjMwNjgwNTExNjQ1OTQ2NDE2MzYiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ==
visitor-hmac38a86977e4e1e7443dc1e38552c7777fa11e02cf51dc8d7cad87172c98108d42
hovercard-subject-tagpull_request:4114357339
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///pull_requests/show/files
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/simstudioai/sim/pull/5884/files
twitter:imagehttps://avatars.githubusercontent.com/u/40672544?s=400&v=4
twitter:cardsummary_large_image
og:imagehttps://avatars.githubusercontent.com/u/40672544?s=400&v=4
og:image:altSummary Follow-up to #5883, which merged before this redesign was ready. #5883 fixed the immediate bug (trashed Google Sheets tabs never purging on a normal sync) with sourceConfirmedEmpty — a conn...
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
None1a6c056e02f174fffc096c521ec0ff6fb83e40a2ec8cb8875466ec1524872dd6
turbo-cache-controlno-preview
diff-viewunified
go-importgithub.com/simstudioai/sim git https://github.com/simstudioai/sim.git
octolytics-dimension-user_id199344406
octolytics-dimension-user_loginsimstudioai
octolytics-dimension-repository_id912559512
octolytics-dimension-repository_nwosimstudioai/sim
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id912559512
octolytics-dimension-repository_network_root_nwosimstudioai/sim
turbo-body-classeslogged-out env-production page-responsive full-width
disable-turbotrue
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release6a93e25585f487ddff9e3996c06d5b869d6e1828
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/simstudioai/sim/pull/5884/files#start-of-content
https://github.com/
Sign in https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fsimstudioai%2Fsim%2Fpull%2F5884%2Ffiles
GitHub CopilotWrite better code with AIhttps://github.com/features/copilot
GitHub Copilot appDirect agents from issue to mergehttps://github.com/features/ai/github-app
MCP RegistryNewIntegrate external toolshttps://github.com/mcp
ActionsAutomate any workflowhttps://github.com/features/actions
CodespacesInstant dev environmentshttps://github.com/features/codespaces
IssuesPlan and track workhttps://github.com/features/issues
Code ReviewManage code changeshttps://github.com/features/code-review
Code QualityEnforce quality at mergehttps://github.com/features/code-quality
GitHub Advanced SecurityFind and fix vulnerabilitieshttps://github.com/security/advanced-security
Code securitySecure your code as you buildhttps://github.com/security/advanced-security/code-security
Secret protectionStop leaks before they starthttps://github.com/security/advanced-security/secret-protection
Why GitHubhttps://github.com/why-github
Documentationhttps://docs.github.com
Bloghttps://github.blog
Changeloghttps://github.blog/changelog
Marketplacehttps://github.com/marketplace
View all featureshttps://github.com/features
Enterpriseshttps://github.com/enterprise
Small and medium teamshttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
App Modernizationhttps://github.com/solutions/use-case/app-modernization
DevSecOpshttps://github.com/solutions/use-case/devsecops
DevOpshttps://github.com/solutions/use-case/devops
CI/CDhttps://github.com/solutions/use-case/ci-cd
View all use caseshttps://github.com/solutions/use-case
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
View all industrieshttps://github.com/solutions/industry
View all solutionshttps://github.com/solutions
AIhttps://github.com/resources/articles?topic=ai
Software Developmenthttps://github.com/resources/articles?topic=software-development
DevOpshttps://github.com/resources/articles?topic=devops
Securityhttps://github.com/resources/articles?topic=security
View all topicshttps://github.com/resources/articles
Customer storieshttps://github.com/customer-stories
Events & webinarshttps://github.com/resources/events
Ebooks & reportshttps://github.com/resources/whitepapers
Business insightshttps://github.com/solutions/executive-insights
GitHub Skillshttps://skills.github.com
Documentationhttps://docs.github.com
Customer supporthttps://support.github.com
Community forumhttps://github.com/orgs/community/discussions
Trust centerhttps://github.com/trust-center
Partnershttps://github.com/partners
View all resourceshttps://github.com/resources
GitHub SponsorsFund open source developershttps://github.com/open-source/sponsors
Security Labhttps://securitylab.github.com
Maintainer Communityhttps://maintainers.github.com
Acceleratorhttps://github.com/open-source/accelerator
GitHub Starshttps://stars.github.com
Archive Programhttps://archiveprogram.github.com
Topicshttps://github.com/topics
Trendinghttps://github.com/trending
Collectionshttps://github.com/collections
Enterprise platformAI-powered developer platformhttps://github.com/enterprise
GitHub Advanced SecurityEnterprise-grade security featureshttps://github.com/security/advanced-security
Copilot for BusinessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Pricinghttps://github.com/pricing
Search syntax tipshttps://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax
documentationhttps://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax
Sign in https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fsimstudioai%2Fsim%2Fpull%2F5884%2Ffiles
Sign up https://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Ffiles&source=header-repo&source_repo=simstudioai%2Fsim
Reloadhttps://github.com/simstudioai/sim/pull/5884/files
Reloadhttps://github.com/simstudioai/sim/pull/5884/files
Reloadhttps://github.com/simstudioai/sim/pull/5884/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/5884/files
simstudioai https://github.com/simstudioai
simhttps://github.com/simstudioai/sim
Notifications https://github.com/login?return_to=%2Fsimstudioai%2Fsim
Fork 3.7k https://github.com/login?return_to=%2Fsimstudioai%2Fsim
Star 29.2k https://github.com/login?return_to=%2Fsimstudioai%2Fsim
Code https://github.com/simstudioai/sim
Issues 111 https://github.com/simstudioai/sim/issues
Pull requests 119 https://github.com/simstudioai/sim/pulls
Actions https://github.com/simstudioai/sim/actions
Projects https://github.com/simstudioai/sim/projects
Security and quality 0 https://github.com/simstudioai/sim/security
Insights https://github.com/simstudioai/sim/pulse
Code https://github.com/simstudioai/sim
Issues https://github.com/simstudioai/sim/issues
Pull requests https://github.com/simstudioai/sim/pulls
Actions https://github.com/simstudioai/sim/actions
Projects https://github.com/simstudioai/sim/projects
Security and quality https://github.com/simstudioai/sim/security
Insights https://github.com/simstudioai/sim/pulse
Sign up for GitHub https://github.com/signup?return_to=%2Fsimstudioai%2Fsim%2Fissues%2Fnew%2Fchoose
terms of servicehttps://docs.github.com/terms
privacy statementhttps://docs.github.com/privacy
Sign inhttps://github.com/login?return_to=%2Fsimstudioai%2Fsim%2Fissues%2Fnew%2Fchoose
waleedlatif1https://github.com/waleedlatif1
staginghttps://github.com/simstudioai/sim/tree/staging
fix/gsheets-tombstone-reconciliationhttps://github.com/simstudioai/sim/tree/fix/gsheets-tombstone-reconciliation
Conversation 67 https://github.com/simstudioai/sim/pull/5884
Commits 13 https://github.com/simstudioai/sim/pull/5884/commits
Checks 19 https://github.com/simstudioai/sim/pull/5884/checks
Files changed 6 https://github.com/simstudioai/sim/pull/5884/files
fix(knowledge): replace deletion-safety heuristics with a two-phase tombstone https://github.com/simstudioai/sim/pull/5884/files#top
Show all changes 13 commits https://github.com/simstudioai/sim/pull/5884/files
f206d38 fix(knowledge): replace deletion-safety heuristics with a two-phase t… waleedlatif1 Jul 23, 2026 https://github.com/simstudioai/sim/pull/5884/commits/f206d38d6d4405f0c8c9e70dd438538504b0343d
368555c fix(knowledge): resurrect atomically on content update, sweep tombsto… waleedlatif1 Jul 23, 2026 https://github.com/simstudioai/sim/pull/5884/commits/368555cda4eab14a2a151b9f544fea3dc65a7c53
0b1d0df fix(knowledge): don't resurrect a tombstoned document whose content r… waleedlatif1 Jul 23, 2026 https://github.com/simstudioai/sim/pull/5884/commits/0b1d0df6820df08cfa1dbd98dcd65bc6f939c1a5
b4d11aa fix(knowledge): exclude fulfilled-but-unverified hydration outcomes f… waleedlatif1 Jul 23, 2026 https://github.com/simstudioai/sim/pull/5884/commits/b4d11aa526637ceb21704bf58367b8b46e8cff65
1423cc3 fix(knowledge): force a full listing when pending-removal documents e… waleedlatif1 Jul 23, 2026 https://github.com/simstudioai/sim/pull/5884/commits/1423cc357436caf044f51e73f3a4abbd66ddc449
16dd2cf fix(knowledge): bound tombstone-forced full syncs, resurrect kept doc… waleedlatif1 Jul 23, 2026 https://github.com/simstudioai/sim/pull/5884/commits/16dd2cf99230a03ed43a0867a63faf876ce7a709
1ef8c25 fix(knowledge): serialize reconciliation writes against a concurrent … waleedlatif1 Jul 23, 2026 https://github.com/simstudioai/sim/pull/5884/commits/1ef8c259f296ab4aadbd769343b9c50517210440
3285239 fix(knowledge): close remaining hard-delete race window, guard listin… waleedlatif1 Jul 23, 2026 https://github.com/simstudioai/sim/pull/5884/commits/328523942b811a36de3768bb0164efe3b5807aa8
3178a27 refactor(knowledge): dedupe retry-cutoff computation, extract ownersh… waleedlatif1 Jul 23, 2026 https://github.com/simstudioai/sim/pull/5884/commits/3178a278c1dc4fbb4a11e0c825250d243309df51
73e767e fix(knowledge): re-verify connectorId at the actual hard-delete, fix … waleedlatif1 Jul 23, 2026 https://github.com/simstudioai/sim/pull/5884/commits/73e767ef1bcdad8fd7715a8edb6f3dcd4157d39e
ddc52a4 fix(knowledge): re-verify connectorId in updateDocument's atomic write waleedlatif1 Jul 23, 2026 https://github.com/simstudioai/sim/pull/5884/commits/ddc52a4c97cf429fdef46aa02241e45d4d76d323
0e160a7 fix(knowledge): close connectorId race in the stuck-document retry path waleedlatif1 Jul 23, 2026 https://github.com/simstudioai/sim/pull/5884/commits/0e160a774f6fb60812a6a36246436391c4db7b8a
a5c804f fix(knowledge): lock the connector row for stuck-doc retry ownership too waleedlatif1 Jul 23, 2026 https://github.com/simstudioai/sim/pull/5884/commits/a5c804fa22ed97c7264ae450c3f1b393202677fe
Clear filters https://github.com/simstudioai/sim/pull/5884/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/5884/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/5884/files
route.ts https://github.com/simstudioai/sim/pull/5884/files#diff-1123ed11307562c4f018c51d26a42b13f893f6351dbda1a01bfb6213179697ea
google-sheets.test.ts https://github.com/simstudioai/sim/pull/5884/files#diff-536acff4d71765c38dc3035ef67a012a11a9608bc45dbfb5603d4192bcc2af65
google-sheets.ts https://github.com/simstudioai/sim/pull/5884/files#diff-59668be6cc3f52dae0af8edbe67ae58d38cc9f17c69d2fb8a430f6b6bb1fd4f0
sync-engine.test.ts https://github.com/simstudioai/sim/pull/5884/files#diff-021afebf7e100833ecdc940e307263436964ce85ef3dfd40fc90307a1d115534
sync-engine.ts https://github.com/simstudioai/sim/pull/5884/files#diff-277f76464f02682e44e6de58ebd215c927f0f6a3a7165af669a0717b593790e9
service.ts https://github.com/simstudioai/sim/pull/5884/files#diff-12adff6ec055d413ec20f1debed4ca207b994d23df6de9aa4ef79bf47f2857b2
apps/sim/app/api/knowledge/[id]/connectors/[connectorId]/route.tshttps://github.com/simstudioai/sim/pull/5884/files#diff-1123ed11307562c4f018c51d26a42b13f893f6351dbda1a01bfb6213179697ea
View file https://github.com/simstudioai/sim/blob/a5c804fa22ed97c7264ae450c3f1b393202677fe/apps/sim/app/api/knowledge/%5Bid%5D/connectors/%5BconnectorId%5D/route.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/5884/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/5884/files#diff-1123ed11307562c4f018c51d26a42b13f893f6351dbda1a01bfb6213179697ea
Please reload this pagehttps://github.com/simstudioai/sim/pull/5884/files
https://github.com/simstudioai/sim/pull/5884/files#diff-1123ed11307562c4f018c51d26a42b13f893f6351dbda1a01bfb6213179697ea
apps/sim/connectors/google-sheets/google-sheets.test.tshttps://github.com/simstudioai/sim/pull/5884/files#diff-536acff4d71765c38dc3035ef67a012a11a9608bc45dbfb5603d4192bcc2af65
View file https://github.com/simstudioai/sim/blob/a5c804fa22ed97c7264ae450c3f1b393202677fe/apps/sim/connectors/google-sheets/google-sheets.test.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/5884/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/5884/files#diff-536acff4d71765c38dc3035ef67a012a11a9608bc45dbfb5603d4192bcc2af65
https://github.com/simstudioai/sim/pull/5884/files#diff-536acff4d71765c38dc3035ef67a012a11a9608bc45dbfb5603d4192bcc2af65
https://github.com/simstudioai/sim/pull/5884/files#diff-536acff4d71765c38dc3035ef67a012a11a9608bc45dbfb5603d4192bcc2af65
https://github.com/simstudioai/sim/pull/5884/files#diff-536acff4d71765c38dc3035ef67a012a11a9608bc45dbfb5603d4192bcc2af65
apps/sim/connectors/google-sheets/google-sheets.tshttps://github.com/simstudioai/sim/pull/5884/files#diff-59668be6cc3f52dae0af8edbe67ae58d38cc9f17c69d2fb8a430f6b6bb1fd4f0
View file https://github.com/simstudioai/sim/blob/a5c804fa22ed97c7264ae450c3f1b393202677fe/apps/sim/connectors/google-sheets/google-sheets.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/simstudioai/sim/pull/5884/{{ revealButtonHref }}
https://github.com/simstudioai/sim/pull/5884/files#diff-59668be6cc3f52dae0af8edbe67ae58d38cc9f17c69d2fb8a430f6b6bb1fd4f0
https://github.com/simstudioai/sim/pull/5884/files#diff-59668be6cc3f52dae0af8edbe67ae58d38cc9f17c69d2fb8a430f6b6bb1fd4f0
https://github.com/simstudioai/sim/pull/5884/files#diff-59668be6cc3f52dae0af8edbe67ae58d38cc9f17c69d2fb8a430f6b6bb1fd4f0
Please reload this pagehttps://github.com/simstudioai/sim/pull/5884/files
Please reload this pagehttps://github.com/simstudioai/sim/pull/5884/files
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width

URLs of crawlers that visited me.