Title: fix(webapp): recover from ClickHouse JSON parse failures on out-of-range integers by matt-aitken · Pull Request #3759 · triggerdotdev/trigger.dev · GitHub
Open Graph Title: fix(webapp): recover from ClickHouse JSON parse failures on out-of-range integers by matt-aitken · Pull Request #3759 · triggerdotdev/trigger.dev
X Title: fix(webapp): recover from ClickHouse JSON parse failures on out-of-range integers by matt-aitken · Pull Request #3759 · triggerdotdev/trigger.dev
Description: Summary Second class of poisoned-row failure in the runs replication path. PR #3708 plugged lone UTF-16 surrogates; this one handles bare JSON integer literals outside ClickHouse's Int64..UInt64 range. Recovery stays purely reactive — the existing sanitizeRows walker just gains an extra branch, so the hot replication path pays nothing on healthy rows. Fixes the still-firing customer-facing symptom from TRI-9755: a small number of runs continued to be stranded in EXECUTING on the Tasks page after #3708 deployed. CloudWatch showed Dropped batch — ClickHouse JSON parse error but sanitizer found nothing to fix firing 8/8 times since the previous deploy (zero successful sanitizations). Root cause: upstream JS Number precision loss on a 21-digit Google Plus ID (117039831458782873093 → 117039831458782870000) — the precision-lossy value still serialises as a bare integer that exceeds UInt64.MAX, which ClickHouse rejects with INCORRECT_DATA. How the bug ships The customer task emits an output containing a Poshmark profile's spec_format: {"key":"gp_id","proper_key":"Gp Id","value":117039831458782870000,"type":"int"} That value is 1.17e20 — comfortably above UInt64.MAX (1.84e19) but comfortably below 1e21. Number.prototype.toString only switches to exponential form at |value| >= 1e21, so JSON.stringify emits the bare token 117039831458782870000 and the ClickHouse JSON(max_dynamic_paths) column fails with: Code: 117. DB::Exception: Cannot parse JSON object here: {…}: (while reading the value of key output): (at row 1) : While executing ParallelParsingBlockInputFormat. (INCORRECT_DATA) (version 25.12.x) Same error verbatim as prod. The same number quoted ("117039831458782870000") inserts fine — ClickHouse's dynamic JSON column accepts a String subtype on the same path. What changed apps/webapp/app/v3/eventRepository/sanitizeRowsOnParseError.server.ts: New private isUnsafeJsonInteger(value) helper — true iff value is a finite integer-valued JS Number where |value| < 1e21 (so JSON.stringify emits integer form, not exponent) and value falls outside [Int64.MIN, UInt64.MAX]. sanitizeUnknownInPlace gains a number-branch: when the predicate holds, replace the Number with String(value). The downstream JSON column dynamic-types the path as String for that row — fine, since the value was already precision-lossy upstream (no JS Number above 2^53 is numerically meaningful anyway). Float-valued numbers, large floats (>= 1e21), NaN and Infinity are left alone — JSON.stringify emits them with exponents or as null, both of which ClickHouse accepts. apps/webapp/test/sanitizeRowsOnParseError.test.ts: four new unit tests + an extension to sanitizeRows covering surrogate + integer fixes counted together across rows. The unit suite now covers: Positive value above UInt64.MAX (117039831458782870000 — the actual prod value) Negative value below Int64.MIN Boundary values pass through (42, Number.MAX_SAFE_INTEGER, 2^63) Non-integer numbers untouched (floats, 1e25, NaN, Infinity) The actual scan-social-profiles nested shape — finds the offending gp_id deep inside output.data.profiles[].spec_format[].platform_variables[].value .server-changes/runs-replication-bigint-recovery.md — release notes entry. Why reactive, not pre-flight #prepareJson runs millions of times per day on the replication hot path. Walking every JSON tree to look for oversized integers would add bounded-but-real CPU on every healthy row. sanitizeRows only fires after a ClickHouse parse-error rejection, which is a few times a day platform-wide. Extending it costs effectively zero on healthy traffic and gains us recovery on the rare poisoned row. Verification Reproduced 1:1 in a throwaway Docker clickhouse/clickhouse-server:25.12.11.4 (closest available to the prod 25.12.1.1579 build). Pre-sanitize JSON fails with the exact prod error; post-sanitize JSON inserts cleanly and the row is readable with gp_id stored as a String subtype. pnpm --filter webapp exec vitest run test/sanitizeRowsOnParseError.test.ts — 22/22 passing (18 existing + 4 new). pnpm run typecheck --filter webapp — clean. Test plan pnpm run typecheck --filter webapp Unit tests pass against new + existing cases End-to-end Docker ClickHouse repro confirms recovery Post-deploy: confirm Sanitizing batch after ClickHouse JSON parse error warns fire instead of Dropped batch … errors when scan-social-profiles outputs trip CH again Post-deploy: confirm permanentlyDroppedBatches counter stops climbing in /stp/trigger-app-prod/ecs/replication/service-container/process-logs What this does NOT do Doesn't backfill the ~120k+ existing stranded EXECUTING rows in production. Same as #3708 — that needs a reconciliation/backfill sweep (separate ticket — TRI-9755 fix #3). Doesn't address the upstream root cause (the customer task emitting a JS-Number-precision-lossy big int). That's a customer-task concern; our replication path needs to be robust to whatever shape arrives. 🤖 Generated with Claude Code
Open Graph Description: Summary Second class of poisoned-row failure in the runs replication path. PR #3708 plugged lone UTF-16 surrogates; this one handles bare JSON integer literals outside ClickHouse's Int64..UInt6...
X Description: Summary Second class of poisoned-row failure in the runs replication path. PR #3708 plugged lone UTF-16 surrogates; this one handles bare JSON integer literals outside ClickHouse's Int64..U...
Opengraph URL: https://github.com/triggerdotdev/trigger.dev/pull/3759
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:6a63ca5e-122f-d0cb-f07a-42db910b7298 |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | 998C:228216:10B504:17B036:6AFAF40D |
| html-safe-nonce | 13bbd2f8dde986b529ec866d929d93bde42b831ae110f0ced1b0cca9db098d3a |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI5OThDOjIyODIxNjoxMEI1MDQ6MTdCMDM2OjZBRkFGNDBEIiwidmlzaXRvcl9pZCI6IjIwNjU1MDYwNTU2OTI2NzgxNTciLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | 5b2ad2e242b4733caeb711567d26ec7fe66ef39453af1aaa8607f62034b10bbf |
| hovercard-subject-tag | pull_request:3753142375 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/triggerdotdev/trigger.dev/pull/3759/files |
| twitter:image | https://avatars.githubusercontent.com/u/10635986?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/10635986?s=400&v=4 |
| og:image:alt | Summary Second class of poisoned-row failure in the runs replication path. PR #3708 plugged lone UTF-16 surrogates; this one handles bare JSON integer literals outside ClickHouse's Int64..UInt6... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | 0178036d8230524a18422872393ab7e708443ae4c2047f758560a2db870c1913 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/triggerdotdev/trigger.dev git https://github.com/triggerdotdev/trigger.dev.git |
| octolytics-dimension-user_id | 95297378 |
| octolytics-dimension-user_login | triggerdotdev |
| octolytics-dimension-repository_id | 572570113 |
| octolytics-dimension-repository_nwo | triggerdotdev/trigger.dev |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 572570113 |
| octolytics-dimension-repository_network_root_nwo | triggerdotdev/trigger.dev |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | a85e96a83bed7871993e6b1ffda24105928eee17 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width