Title: Helm chart pins ClickHouse 25.6.1, which has a memory-tracker overflow that breaks the dashboard until pod restart · Issue #3520 · triggerdotdev/trigger.dev · GitHub
Open Graph Title: Helm chart pins ClickHouse 25.6.1, which has a memory-tracker overflow that breaks the dashboard until pod restart · Issue #3520 · triggerdotdev/trigger.dev
X Title: Helm chart pins ClickHouse 25.6.1, which has a memory-tracker overflow that breaks the dashboard until pod restart · Issue #3520 · triggerdotdev/trigger.dev
Description: Hey guys...this one's a doosy, I'm sorry. Trying to push us to use AI more (whom this issue's author clearly is), and frankly, I never would have found or understood this one otherwise. Airing on the side of too much info rather than too...
Open Graph Description: Hey guys...this one's a doosy, I'm sorry. Trying to push us to use AI more (whom this issue's author clearly is), and frankly, I never would have found or understood this one otherwise. Airing on t...
X Description: Hey guys...this one's a doosy, I'm sorry. Trying to push us to use AI more (whom this issue's author clearly is), and frankly, I never would have found or understood this one otherwise....
Opengraph URL: https://github.com/triggerdotdev/trigger.dev/issues/3520
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Helm chart pins ClickHouse 25.6.1, which has a memory-tracker overflow that breaks the dashboard until pod restart","articleBody":"Hey guys...this one's a doosy, I'm sorry. Trying to push us to use AI more (whom this issue's author clearly is), and frankly, I never would have found or understood this one otherwise. Airing on the side of too much info rather than too little.\n\n## Summary\n\nThe official Helm chart (`oci://ghcr.io/triggerdotdev/charts/trigger`) pins the Bitnami ClickHouse subchart to `clickhouse-9.3.7`, which in turn pins `bitnami/clickhouse:25.6.1-debian-12-r0`. Under sustained ingest, ClickHouse 25.6.1 hits a memory-tracker accounting bug that causes the global memory counter to overflow to ~7 EiB (≈ `2^63`), at which point every query — reads and writes — is rejected by `OvercommitTracker` until the pod is restarted. The trigger.dev webapp dashboard surfaces this as **\"Unable to load your task runs\"**, and event/run telemetry stops being persisted.\n\nThe condition is not self-clearing. The host has plenty of free memory (RSS ~2 GiB out of a 21.6 GiB limit), but ClickHouse's internal accounting is wedged. Only an in-process restart resets it.\n\n## Environment\n\n- Trigger.dev Helm chart: `4.0.5` (also reproduced on `4.4.5` since both pin the same subchart — see below)\n- App image: `ghcr.io/triggerdotdev/trigger.dev:v4.4.4`\n- Kubernetes: EKS, single-shard ClickHouse statefulset (`trigger-clickhouse-shard0-0`), ~20 GiB PVC\n- ClickHouse resources: requests `13.5Gi` / limits `24Gi` (chart values block)\n- Workload: typical Trigger.dev v4 ingest — task runs + trace events flowing through `task_runs_v2`, `task_events_v2`, and the `raw_task_runs_payload_v1` staging table\n\n## Reproduction\n\nThis was a production incident, not an isolated synthetic repro, but the trigger seems to be sustained write pressure with concurrent background merges on `raw_task_runs_payload_v1`. Once the merge tasks start failing with `MEMORY_LIMIT_EXCEEDED`, retries pile up and never recover.\n\n## What we observed\n\nClickHouse pod logs (representative, repeating thousands of times per minute):\n\n```\nCode: 241. DB::Exception: (total) memory limit exceeded:\n would use 7.00 EiB (attempt to allocate chunk of 4.00 MiB bytes),\n current RSS: 2.12 GiB, maximum: 21.60 GiB.\n OvercommitTracker decision: Query was selected to stop by OvercommitTracker.\n (MEMORY_LIMIT_EXCEEDED)\n... while reading from part .../raw_task_runs_payload_v1/...\n... in query: INSERT INTO trigger_dev.task_runs_v2 ...\n... in query: INSERT INTO trigger_dev.task_events_v2 ...\n```\n\nThe `7.00 EiB` figure is the giveaway — that's `~2^63` bytes, i.e. a signed-integer overflow in ClickHouse's global memory tracker. The actual RSS is ~2 GiB.\n\nWebapp logs (during the event):\n\n```\nEventRepo.DynamicFlushScheduler Error attempting to flush batch\n consecutiveFailures: 19438\n table: trigger_dev.task_events_v2\n error: InsertError: (total) memory limit exceeded ... 7.00 EiB ...\n```\n\nThe webapp accumulated ~1.75M backlogged events that had to drain after we restarted ClickHouse. UI showed \"Unable to load your task runs\" because the dashboard's reads against `task_runs_v2` were rejected by the same tracker.\n\n## Root cause (best assessment)\n\nThis is a known class of ClickHouse memory-tracker accounting bug — a `free()` is double-counted (or an `alloc()` is missed) in one of the hot paths (background merges, async inserts, materialized views), the global atomic counter goes negative, the unsigned interpretation looks like ~9 EiB, and `OvercommitTracker` rejects every subsequent allocation. There are multiple upstream ClickHouse commits in 25.7+ touching memory-tracker accuracy and overflow handling.\n\n## Mitigation (workaround)\n\n```\nkubectl rollout restart statefulset/trigger-clickhouse-shard0 -n trigger\n```\n\nRecovery is immediate — error rate goes from `~11k/min` to `~0` within seconds of the pod coming back up. Webapp `consecutiveFailures` drops from `~47k` to single digits as soon as ClickHouse is reachable again.\n\n## Suggested resolution (chart-side)\n\nThe trigger Helm chart's Bitnami CH subchart pin (`charts/clickhouse-9.3.7` → `bitnami/clickhouse:25.6.1-debian-12-r0`) hasn't moved since `trigger@4.0.5` and is still the same in `trigger@4.4.5`. Bumping the Bitnami subchart pin in `hosting/k8s/helm/Chart.yaml` to a current version (latest Bitnami CH chart on `main` ships `25.7.5-debian-12-r0`) would pull in upstream tracker fixes for everyone running self-hosted, without each operator having to override `clickhouse.image.tag` themselves.\n\nOperators currently on `trigger@4.0.5`–`4.4.5` are exposed to this regardless of chart version, since the subchart pin is unchanged.\n\n## Suggested resolution (operational hardening, optional)\n\nA few small additions would make this much less painful even if the underlying bug isn't fully fixed:\n\n1. **A liveness probe that exercises the query path** (e.g. `clickhouse-client -q \"SELECT 1\"`) on the bundled CH StatefulSet. The current TCP/HTTP probe stays green when the pod is wedged — a query-based probe would let Kubernetes auto-restart the pod within a minute. Today an operator has to notice the user-visible failure first.\n2. **Webapp circuit breaker / backlog cap on `EventRepo.DynamicFlushScheduler`.** When `consecutiveFailures` crosses some threshold, sample/drop trace events instead of accumulating millions of items in memory. Losing 5 minutes of partial trace data is preferable to a 1.75M-item backlog that takes hours to drain post-recovery and increases the chance of re-tripping the tracker.\n3. **Document the failure mode and recovery in the self-hosting docs**, so other operators recognize \"Unable to load your task runs\" + `MEMORY_LIMIT_EXCEEDED ... 7 EiB` as a single condition with a known one-line fix.\n\nHappy to test a chart bump on our end and report back, or open a small PR against `hosting/k8s/helm/` for the subchart bump if helpful.\n\n— Filed by an operator running self-hosted Trigger.dev v4 on EKS\n","author":{"url":"https://github.com/brentshulman-silkline","@type":"Person","name":"brentshulman-silkline"},"datePublished":"2026-05-05T02:35:53Z","url":"https://github.com/triggerdotdev/trigger.dev/issues/3520"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:0454db32-09c1-cb64-fc03-07d1e6308d6f |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | E01A:36012D:160072F:1EC5938:6AFAEC67 |
| html-safe-nonce | ec98c04c1b29491fcc0d92f824e5ca0fed6bc429084f89769d927796a82b43fa |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJFMDFBOjM2MDEyRDoxNjAwNzJGOjFFQzU5Mzg6NkFGQUVDNjciLCJ2aXNpdG9yX2lkIjoiNTY1OTIzODAyMjcwOTQzOTU5MSIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | eb1308a48b2c511cbb4292f6346799c69747943f5ae2cb66ad078217061dea0e |
| hovercard-subject-tag | issue:4381089994 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/triggerdotdev/trigger.dev/3520/issue_layout |
| twitter:image | https://opengraph.githubassets.com/05efe5fdfa4fae90f15e6732ca7b215ceb128702eb0e84c1cda90eed64c06f8a/triggerdotdev/trigger.dev/issues/3520 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/05efe5fdfa4fae90f15e6732ca7b215ceb128702eb0e84c1cda90eed64c06f8a/triggerdotdev/trigger.dev/issues/3520 |
| og:image:alt | Hey guys...this one's a doosy, I'm sorry. Trying to push us to use AI more (whom this issue's author clearly is), and frankly, I never would have found or understood this one otherwise. Airing on t... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | brentshulman-silkline |
| hostname | github.com |
| expected-hostname | github.com |
| None | 0178036d8230524a18422872393ab7e708443ae4c2047f758560a2db870c1913 |
| turbo-cache-control | no-preview |
| go-import | github.com/triggerdotdev/trigger.dev git https://github.com/triggerdotdev/trigger.dev.git |
| octolytics-dimension-user_id | 95297378 |
| octolytics-dimension-user_login | triggerdotdev |
| octolytics-dimension-repository_id | 572570113 |
| octolytics-dimension-repository_nwo | triggerdotdev/trigger.dev |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 572570113 |
| octolytics-dimension-repository_network_root_nwo | triggerdotdev/trigger.dev |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 12fff95459b24b0f882a66698ee6a36a5934d247 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width