René's URL Explorer Experiment


Title: fix(webapp,clickhouse): keep the rest of a ClickHouse batch when one run or span has un-ingestable JSON by ericallam · Pull Request #4358 · triggerdotdev/trigger.dev · GitHub

Open Graph Title: fix(webapp,clickhouse): keep the rest of a ClickHouse batch when one run or span has un-ingestable JSON by ericallam · Pull Request #4358 · triggerdotdev/trigger.dev

X Title: fix(webapp,clickhouse): keep the rest of a ClickHouse batch when one run or span has un-ingestable JSON by ericallam · Pull Request #4358 · triggerdotdev/trigger.dev

Description: Summary A single run output, trace span, or payload carrying JSON that ClickHouse can't ingest (for example nesting past its depth limit) used to fail the whole insert batch, so unrelated runs and spans silently disappeared from the runs list, traces, and logs. This keeps the rest of the batch and handles the offending row instead of dropping everything around it. Fix Recovery is per-table, matched to what each table needs: Runs (task_runs_v2) keep their status. We follow ClickHouse's failing-row hint to strip just the un-ingestable JSON column(s) so the run still lands (its output reads from Postgres on the detail page), up to a configurable limit (RUN_REPLICATION_MAX_POISON_STRIPS_PER_BATCH, default 1). Past the limit we stop and land the batch with allow_errors in a single pass, skipping the remainder. Cost stays a fixed handful of inserts no matter how large or poisoned a flush is. Trace events and payloads (high volume, append-only) recover with a single allow_errors insert: the good rows land in one pass and only the un-ingestable rows are skipped. Before falling back, a lightweight sanitizer still repairs what it can losslessly (lone UTF-16 surrogates, out-of-range integers) so a repairable row lands in full. To read the failing-row hint we patch @clickhouse/client-common: its error parser truncates the server response and discards the (at row N) position, so the patch preserves the full text for the recovery path to read.

Open Graph Description: Summary A single run output, trace span, or payload carrying JSON that ClickHouse can't ingest (for example nesting past its depth limit) used to fail the whole insert batch, so unrelated runs ...

X Description: Summary A single run output, trace span, or payload carrying JSON that ClickHouse can't ingest (for example nesting past its depth limit) used to fail the whole insert batch, so unrelated r...

Opengraph URL: https://github.com/triggerdotdev/trigger.dev/pull/4358

X: @github

direct link

Domain: github.com

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf
route-pattern/:user_id/:repository/pull/:id/files(.:format)
route-controllerpull_requests
route-actionfiles
fetch-noncev2:17384a75-574d-8697-6479-10b9676edc43
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-idEACC:29CAD0:728EBC:A98C32:6AFAF669
html-safe-nonce6aaea3ef5c02bdd1437478af9884b3ede34ec1d50c5ffcaf7b86010effe09fcb
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJFQUNDOjI5Q0FEMDo3MjhFQkM6QTk4QzMyOjZBRkFGNjY5IiwidmlzaXRvcl9pZCI6IjU1NjAwNjIzNDg3NjE5NTM4OTciLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ==
visitor-hmac4e702d7bbfe68edd3af098ac4aa8bb52396ac4706efeee43cc8c051b2a6afd9a
hovercard-subject-tagpull_request:4120401493
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///pull_requests/show/files
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/triggerdotdev/trigger.dev/pull/4358/files
twitter:imagehttps://avatars.githubusercontent.com/u/534?s=400&v=4
twitter:cardsummary_large_image
og:imagehttps://avatars.githubusercontent.com/u/534?s=400&v=4
og:image:altSummary A single run output, trace span, or payload carrying JSON that ClickHouse can't ingest (for example nesting past its depth limit) used to fail the whole insert batch, so unrelated runs ...
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
None0178036d8230524a18422872393ab7e708443ae4c2047f758560a2db870c1913
turbo-cache-controlno-preview
diff-viewunified
go-importgithub.com/triggerdotdev/trigger.dev git https://github.com/triggerdotdev/trigger.dev.git
octolytics-dimension-user_id95297378
octolytics-dimension-user_logintriggerdotdev
octolytics-dimension-repository_id572570113
octolytics-dimension-repository_nwotriggerdotdev/trigger.dev
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id572570113
octolytics-dimension-repository_network_root_nwotriggerdotdev/trigger.dev
turbo-body-classeslogged-out env-production page-responsive full-width
disable-turbotrue
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
releasea85e96a83bed7871993e6b1ffda24105928eee17
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/triggerdotdev/trigger.dev/pull/4358/files#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Ftriggerdotdev%2Ftrigger.dev%2Fpull%2F4358%2Ffiles
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Ftriggerdotdev%2Ftrigger.dev%2Fpull%2F4358%2Ffiles
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Ffiles&source=header-repo&source_repo=triggerdotdev%2Ftrigger.dev
Reloadhttps://github.com/triggerdotdev/trigger.dev/pull/4358/files
Reloadhttps://github.com/triggerdotdev/trigger.dev/pull/4358/files
Reloadhttps://github.com/triggerdotdev/trigger.dev/pull/4358/files
triggerdotdevhttps://github.com/triggerdotdev
trigger.devhttps://github.com/triggerdotdev/trigger.dev
Codehttps://github.com/triggerdotdev/trigger.dev
Issues203 (203)https://github.com/triggerdotdev/trigger.dev/issues
Pull requests67 (67)https://github.com/triggerdotdev/trigger.dev/pulls
Discussionshttps://github.com/triggerdotdev/trigger.dev/discussions
Actionshttps://github.com/triggerdotdev/trigger.dev/actions
Projectshttps://github.com/triggerdotdev/trigger.dev/projects
Security and quality39 (39)https://github.com/triggerdotdev/trigger.dev/security
Insightshttps://github.com/triggerdotdev/trigger.dev/pulse
Sign up for GitHub https://github.com/signup?return_to=%2Ftriggerdotdev%2Ftrigger.dev%2Fissues%2Fnew%2Fchoose
terms of servicehttps://docs.github.com/terms
privacy statementhttps://docs.github.com/privacy
Sign inhttps://github.com/login?return_to=%2Ftriggerdotdev%2Ftrigger.dev%2Fissues%2Fnew%2Fchoose
ericallamhttps://github.com/ericallam
mainhttps://github.com/triggerdotdev/trigger.dev/tree/main
fix/run-replication-json-parse-batch-drophttps://github.com/triggerdotdev/trigger.dev/tree/fix/run-replication-json-parse-batch-drop
Conversation 44 https://github.com/triggerdotdev/trigger.dev/pull/4358
Commits 13 https://github.com/triggerdotdev/trigger.dev/pull/4358/commits
Checks 67 https://github.com/triggerdotdev/trigger.dev/pull/4358/checks
Files changed https://github.com/triggerdotdev/trigger.dev/pull/4358/files
Please reload this pagehttps://github.com/triggerdotdev/trigger.dev/pull/4358/files
fix(webapp,clickhouse): keep the rest of a ClickHouse batch when one run or span has un-ingestable JSON https://github.com/triggerdotdev/trigger.dev/pull/4358/files#top
Show all changes 13 commits https://github.com/triggerdotdev/trigger.dev/pull/4358/files
be8115b fix(webapp): keep the rest of a batch when one run or span has un-ing… ericallam Jul 23, 2026 https://github.com/triggerdotdev/trigger.dev/pull/4358/commits/be8115b462775f2e10d27bff233e18b509df4224
29ebc5e chore(webapp): rename batch row-isolation metric and emit run-replica… ericallam Jul 24, 2026 https://github.com/triggerdotdev/trigger.dev/pull/4358/commits/29ebc5e16944e201bf1df0740ce00de17decf095
7b5460c refactor(webapp): share JSON-parse recovery and keep poison rows with… ericallam Jul 24, 2026 https://github.com/triggerdotdev/trigger.dev/pull/4358/commits/7b5460c5bfc68be21cadc7f08f68402dd9c5da5a
73c447e feat(webapp): cap row-isolation recovery cost against a poison flood ericallam Jul 24, 2026 https://github.com/triggerdotdev/trigger.dev/pull/4358/commits/73c447e7284cc5282c1fb7afc1cd4d55061a3f8b
7d26d78 fix(webapp): only count a dropped batch when nothing landed, and asse… ericallam Jul 24, 2026 https://github.com/triggerdotdev/trigger.dev/pull/4358/commits/7d26d78a6eeab987934364390682560626bfc63a
c78c5b0 test(webapp): sample benchmark ELU as a per-window delta and build po… ericallam Jul 24, 2026 https://github.com/triggerdotdev/trigger.dev/pull/4358/commits/c78c5b0e3beac1208f3cc1e246d0adfc45991a83
535889f fix(webapp,clickhouse): stop one un-ingestable JSON row dropping its … ericallam Jul 24, 2026 https://github.com/triggerdotdev/trigger.dev/pull/4358/commits/535889fc04fc0e56fc8f46525ef66156e8857440
36df1a1 fix(webapp,clickhouse): keep ClickHouse row snippets out of logs ericallam Jul 25, 2026 https://github.com/triggerdotdev/trigger.dev/pull/4358/commits/36df1a18bf430912548823154c313ecdf6cd9ac3
9da98a3 fix(clickhouse): make the patched rawMessage non-enumerable at the so… ericallam Jul 27, 2026 https://github.com/triggerdotdev/trigger.dev/pull/4358/commits/9da98a35924fe66dcb682baef436b423b2258460
e9ecce0 fix(webapp): strip only a poison run's output, not its error ericallam Jul 31, 2026 https://github.com/triggerdotdev/trigger.dev/pull/4358/commits/e9ecce019ea5d55a14c4f1e60c913571d2110cbc
a20aed2 fix(webapp): anchor the row hint and stop rethrowing a dead batch ericallam Jul 31, 2026 https://github.com/triggerdotdev/trigger.dev/pull/4358/commits/a20aed2f85b3518e2368573604c019cf4d976052
6b32405 fix(webapp): don't notify trace subscribers when nothing landed ericallam Jul 31, 2026 https://github.com/triggerdotdev/trigger.dev/pull/4358/commits/6b324050ad55aea39c324ee72560fb1cf3fa6639
8357958 fix(webapp): log a whole-batch drop at error again, not warn ericallam Jul 31, 2026 https://github.com/triggerdotdev/trigger.dev/pull/4358/commits/8357958e474793d96cabc0f8dcf97fa55baca75c
Clear filters https://github.com/triggerdotdev/trigger.dev/pull/4358/files
Please reload this pagehttps://github.com/triggerdotdev/trigger.dev/pull/4358/files
Please reload this pagehttps://github.com/triggerdotdev/trigger.dev/pull/4358/files
fix-runs-list-batch-drop-on-unparseable-json.md https://github.com/triggerdotdev/trigger.dev/pull/4358/files#diff-e3412a80c364de2e763f9135647cf3d7e4db9a411dfa01c8a0212a777e192a06
env.server.ts https://github.com/triggerdotdev/trigger.dev/pull/4358/files#diff-f8823615ef72da3494e74de9e62b2fa31315ed9360f17bc06ab943f4dc25f4fd
runsReplicationInstance.server.ts https://github.com/triggerdotdev/trigger.dev/pull/4358/files#diff-eb6979ac5d616f99cff1238f08a6773c3d04c2506180675fd1d7bc9f9f22b510
runsReplicationService.server.ts https://github.com/triggerdotdev/trigger.dev/pull/4358/files#diff-57d6170ebedce17e6b4c66542bf89ca786d33ac432459f7e5cfec3bc7391e0b5
clickhouseEventRepository.server.ts https://github.com/triggerdotdev/trigger.dev/pull/4358/files#diff-2feda1843834eeed8bb4b12473f3b851b4c608ca6eacc4d193aca742fd6e30ad
sanitizeRowsOnParseError.server.ts https://github.com/triggerdotdev/trigger.dev/pull/4358/files#diff-4d209459ebc0ff63710c764a97f46e6cfda604ee2aacaf6cc9bb255390da9891
clickhouseEventRepositoryJsonRecovery.test.ts https://github.com/triggerdotdev/trigger.dev/pull/4358/files#diff-8a282d8f868a3500d31f34ef9b7bf867cb19fa05c854367d8750e6a71feb9a65
runsReplicationBenchmark.producer.ts https://github.com/triggerdotdev/trigger.dev/pull/4358/files#diff-278004c9ee42fbd733d39a4520518b0cd31501b65fa96dbd336f2b42c88ded90
runsReplicationJsonRecoveryBenchmark.test.ts https://github.com/triggerdotdev/trigger.dev/pull/4358/files#diff-be31a16314fe6c963c6b5c73656bc1f87459030aaf5bbca5eecd8d280d3f2bf2
runsReplicationService.part10.test.ts https://github.com/triggerdotdev/trigger.dev/pull/4358/files#diff-203c5d384b47f6f1de1dd45f6c38bb2233dc24fb05609e0dc88a233687798240
sanitizeRowsOnParseError.test.ts https://github.com/triggerdotdev/trigger.dev/pull/4358/files#diff-936a199f8ad54e45d02e7068a496c1ec2543b26cbb4b5144729aaf58058cde3f
client.ts https://github.com/triggerdotdev/trigger.dev/pull/4358/files#diff-10741538b9e113836bae6e267758a1fc5794f80a06960ccf4f57b43936fcb94f
errors.test.ts https://github.com/triggerdotdev/trigger.dev/pull/4358/files#diff-807c74ba6d479fa7e602381aea14ebef44537320d191b7ffed62f9baaeec5ec9
errors.ts https://github.com/triggerdotdev/trigger.dev/pull/4358/files#diff-1f14c9c88917c479d3bf1ace66de88c1809c4a5733117821581d5d87da5b08b7
package.json https://github.com/triggerdotdev/trigger.dev/pull/4358/files#diff-7ae45ad102eab3b6d7e7896acd08c427a9b25b346470d7bc6507b6481575d519
@clickhouse__client-common@1.12.1.patch https://github.com/triggerdotdev/trigger.dev/pull/4358/files#diff-4afb058f43fdf9f11b57b1ba2750f9a7d193444147cb9694ea90c95630ad9dcd
pnpm-lock.yaml https://github.com/triggerdotdev/trigger.dev/pull/4358/files#diff-32824c984905bb02bc7ffcef96a77addd1f1602cff71a11fbbfdd7f53ee026bb
.server-changes/fix-runs-list-batch-drop-on-unparseable-json.mdhttps://github.com/triggerdotdev/trigger.dev/pull/4358/files#diff-e3412a80c364de2e763f9135647cf3d7e4db9a411dfa01c8a0212a777e192a06
View file https://github.com/triggerdotdev/trigger.dev/blob/8357958e474793d96cabc0f8dcf97fa55baca75c/.server-changes/fix-runs-list-batch-drop-on-unparseable-json.md
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/triggerdotdev/trigger.dev/pull/4358/{{ revealButtonHref }}
apps/webapp/app/env.server.tshttps://github.com/triggerdotdev/trigger.dev/pull/4358/files#diff-f8823615ef72da3494e74de9e62b2fa31315ed9360f17bc06ab943f4dc25f4fd
View file https://github.com/triggerdotdev/trigger.dev/blob/8357958e474793d96cabc0f8dcf97fa55baca75c/apps/webapp/app/env.server.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/triggerdotdev/trigger.dev/pull/4358/{{ revealButtonHref }}
https://github.com/triggerdotdev/trigger.dev/pull/4358/files#diff-f8823615ef72da3494e74de9e62b2fa31315ed9360f17bc06ab943f4dc25f4fd
https://github.com/triggerdotdev/trigger.dev/pull/4358/files#diff-f8823615ef72da3494e74de9e62b2fa31315ed9360f17bc06ab943f4dc25f4fd
apps/webapp/app/services/runsReplicationInstance.server.tshttps://github.com/triggerdotdev/trigger.dev/pull/4358/files#diff-eb6979ac5d616f99cff1238f08a6773c3d04c2506180675fd1d7bc9f9f22b510
View file https://github.com/triggerdotdev/trigger.dev/blob/8357958e474793d96cabc0f8dcf97fa55baca75c/apps/webapp/app/services/runsReplicationInstance.server.ts
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/triggerdotdev/trigger.dev/pull/4358/{{ revealButtonHref }}
https://github.com/triggerdotdev/trigger.dev/pull/4358/files#diff-eb6979ac5d616f99cff1238f08a6773c3d04c2506180675fd1d7bc9f9f22b510
https://github.com/triggerdotdev/trigger.dev/pull/4358/files#diff-eb6979ac5d616f99cff1238f08a6773c3d04c2506180675fd1d7bc9f9f22b510
Please reload this pagehttps://github.com/triggerdotdev/trigger.dev/pull/4358/files
Please reload this pagehttps://github.com/triggerdotdev/trigger.dev/pull/4358/files
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.