René's URL Explorer Experiment


Title: Helm chart pins ClickHouse 25.6.1, which has a memory-tracker overflow that breaks the dashboard until pod restart · Issue #3520 · triggerdotdev/trigger.dev · GitHub

Open Graph Title: Helm chart pins ClickHouse 25.6.1, which has a memory-tracker overflow that breaks the dashboard until pod restart · Issue #3520 · triggerdotdev/trigger.dev

X Title: Helm chart pins ClickHouse 25.6.1, which has a memory-tracker overflow that breaks the dashboard until pod restart · Issue #3520 · triggerdotdev/trigger.dev

Description: Hey guys...this one's a doosy, I'm sorry. Trying to push us to use AI more (whom this issue's author clearly is), and frankly, I never would have found or understood this one otherwise. Airing on the side of too much info rather than too...

Open Graph Description: Hey guys...this one's a doosy, I'm sorry. Trying to push us to use AI more (whom this issue's author clearly is), and frankly, I never would have found or understood this one otherwise. Airing on t...

X Description: Hey guys...this one's a doosy, I'm sorry. Trying to push us to use AI more (whom this issue's author clearly is), and frankly, I never would have found or understood this one otherwise....

Opengraph URL: https://github.com/triggerdotdev/trigger.dev/issues/3520

X: @github

direct link

Domain: github.com


Hey, it has json ld scripts:
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Helm chart pins ClickHouse 25.6.1, which has a memory-tracker overflow that breaks the dashboard until pod restart","articleBody":"Hey guys...this one's a doosy, I'm sorry. Trying to push us to use AI more (whom this issue's author clearly is), and frankly, I never would have found or understood this one otherwise. Airing on the side of too much info rather than too little.\n\n## Summary\n\nThe official Helm chart (`oci://ghcr.io/triggerdotdev/charts/trigger`) pins the Bitnami ClickHouse subchart to `clickhouse-9.3.7`, which in turn pins `bitnami/clickhouse:25.6.1-debian-12-r0`. Under sustained ingest, ClickHouse 25.6.1 hits a memory-tracker accounting bug that causes the global memory counter to overflow to ~7 EiB (≈ `2^63`), at which point every query — reads and writes — is rejected by `OvercommitTracker` until the pod is restarted. The trigger.dev webapp dashboard surfaces this as **\"Unable to load your task runs\"**, and event/run telemetry stops being persisted.\n\nThe condition is not self-clearing. The host has plenty of free memory (RSS ~2 GiB out of a 21.6 GiB limit), but ClickHouse's internal accounting is wedged. Only an in-process restart resets it.\n\n## Environment\n\n- Trigger.dev Helm chart: `4.0.5` (also reproduced on `4.4.5` since both pin the same subchart — see below)\n- App image: `ghcr.io/triggerdotdev/trigger.dev:v4.4.4`\n- Kubernetes: EKS, single-shard ClickHouse statefulset (`trigger-clickhouse-shard0-0`), ~20 GiB PVC\n- ClickHouse resources: requests `13.5Gi` / limits `24Gi` (chart values block)\n- Workload: typical Trigger.dev v4 ingest — task runs + trace events flowing through `task_runs_v2`, `task_events_v2`, and the `raw_task_runs_payload_v1` staging table\n\n## Reproduction\n\nThis was a production incident, not an isolated synthetic repro, but the trigger seems to be sustained write pressure with concurrent background merges on `raw_task_runs_payload_v1`. Once the merge tasks start failing with `MEMORY_LIMIT_EXCEEDED`, retries pile up and never recover.\n\n## What we observed\n\nClickHouse pod logs (representative, repeating thousands of times per minute):\n\n```\nCode: 241. DB::Exception: (total) memory limit exceeded:\n  would use 7.00 EiB (attempt to allocate chunk of 4.00 MiB bytes),\n  current RSS: 2.12 GiB, maximum: 21.60 GiB.\n  OvercommitTracker decision: Query was selected to stop by OvercommitTracker.\n  (MEMORY_LIMIT_EXCEEDED)\n... while reading from part .../raw_task_runs_payload_v1/...\n... in query: INSERT INTO trigger_dev.task_runs_v2 ...\n... in query: INSERT INTO trigger_dev.task_events_v2 ...\n```\n\nThe `7.00 EiB` figure is the giveaway — that's `~2^63` bytes, i.e. a signed-integer overflow in ClickHouse's global memory tracker. The actual RSS is ~2 GiB.\n\nWebapp logs (during the event):\n\n```\nEventRepo.DynamicFlushScheduler  Error attempting to flush batch\n  consecutiveFailures: 19438\n  table: trigger_dev.task_events_v2\n  error: InsertError: (total) memory limit exceeded ... 7.00 EiB ...\n```\n\nThe webapp accumulated ~1.75M backlogged events that had to drain after we restarted ClickHouse. UI showed \"Unable to load your task runs\" because the dashboard's reads against `task_runs_v2` were rejected by the same tracker.\n\n## Root cause (best assessment)\n\nThis is a known class of ClickHouse memory-tracker accounting bug — a `free()` is double-counted (or an `alloc()` is missed) in one of the hot paths (background merges, async inserts, materialized views), the global atomic counter goes negative, the unsigned interpretation looks like ~9 EiB, and `OvercommitTracker` rejects every subsequent allocation. There are multiple upstream ClickHouse commits in 25.7+ touching memory-tracker accuracy and overflow handling.\n\n## Mitigation (workaround)\n\n```\nkubectl rollout restart statefulset/trigger-clickhouse-shard0 -n trigger\n```\n\nRecovery is immediate — error rate goes from `~11k/min` to `~0` within seconds of the pod coming back up. Webapp `consecutiveFailures` drops from `~47k` to single digits as soon as ClickHouse is reachable again.\n\n## Suggested resolution (chart-side)\n\nThe trigger Helm chart's Bitnami CH subchart pin (`charts/clickhouse-9.3.7` → `bitnami/clickhouse:25.6.1-debian-12-r0`) hasn't moved since `trigger@4.0.5` and is still the same in `trigger@4.4.5`. Bumping the Bitnami subchart pin in `hosting/k8s/helm/Chart.yaml` to a current version (latest Bitnami CH chart on `main` ships `25.7.5-debian-12-r0`) would pull in upstream tracker fixes for everyone running self-hosted, without each operator having to override `clickhouse.image.tag` themselves.\n\nOperators currently on `trigger@4.0.5`–`4.4.5` are exposed to this regardless of chart version, since the subchart pin is unchanged.\n\n## Suggested resolution (operational hardening, optional)\n\nA few small additions would make this much less painful even if the underlying bug isn't fully fixed:\n\n1. **A liveness probe that exercises the query path** (e.g. `clickhouse-client -q \"SELECT 1\"`) on the bundled CH StatefulSet. The current TCP/HTTP probe stays green when the pod is wedged — a query-based probe would let Kubernetes auto-restart the pod within a minute. Today an operator has to notice the user-visible failure first.\n2. **Webapp circuit breaker / backlog cap on `EventRepo.DynamicFlushScheduler`.** When `consecutiveFailures` crosses some threshold, sample/drop trace events instead of accumulating millions of items in memory. Losing 5 minutes of partial trace data is preferable to a 1.75M-item backlog that takes hours to drain post-recovery and increases the chance of re-tripping the tracker.\n3. **Document the failure mode and recovery in the self-hosting docs**, so other operators recognize \"Unable to load your task runs\" + `MEMORY_LIMIT_EXCEEDED ... 7 EiB` as a single condition with a known one-line fix.\n\nHappy to test a chart bump on our end and report back, or open a small PR against `hosting/k8s/helm/` for the subchart bump if helpful.\n\n— Filed by an operator running self-hosted Trigger.dev v4 on EKS\n","author":{"url":"https://github.com/brentshulman-silkline","@type":"Person","name":"brentshulman-silkline"},"datePublished":"2026-05-05T02:35:53Z","url":"https://github.com/triggerdotdev/trigger.dev/issues/3520"}

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react-reconciler,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@d5750093facf
route-pattern/_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format)
route-controllervoltron_issues_fragments
route-actionissue_layout
fetch-noncev2:0454db32-09c1-cb64-fc03-07d1e6308d6f
current-catalog-service-hash81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114
request-idE01A:36012D:160072F:1EC5938:6AFAEC67
html-safe-nonceec98c04c1b29491fcc0d92f824e5ca0fed6bc429084f89769d927796a82b43fa
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJFMDFBOjM2MDEyRDoxNjAwNzJGOjFFQzU5Mzg6NkFGQUVDNjciLCJ2aXNpdG9yX2lkIjoiNTY1OTIzODAyMjcwOTQzOTU5MSIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9
visitor-hmaceb1308a48b2c511cbb4292f6346799c69747943f5ae2cb66ad078217061dea0e
hovercard-subject-tagissue:4381089994
github-keyboard-shortcutsrepository,issues,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///voltron/issues_fragments/issue_layout
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/triggerdotdev/trigger.dev/3520/issue_layout
twitter:imagehttps://opengraph.githubassets.com/05efe5fdfa4fae90f15e6732ca7b215ceb128702eb0e84c1cda90eed64c06f8a/triggerdotdev/trigger.dev/issues/3520
twitter:cardsummary_large_image
og:imagehttps://opengraph.githubassets.com/05efe5fdfa4fae90f15e6732ca7b215ceb128702eb0e84c1cda90eed64c06f8a/triggerdotdev/trigger.dev/issues/3520
og:image:altHey guys...this one's a doosy, I'm sorry. Trying to push us to use AI more (whom this issue's author clearly is), and frankly, I never would have found or understood this one otherwise. Airing on t...
og:image:width1200
og:image:height600
og:site_nameGitHub
og:typeobject
og:author:usernamebrentshulman-silkline
hostnamegithub.com
expected-hostnamegithub.com
None0178036d8230524a18422872393ab7e708443ae4c2047f758560a2db870c1913
turbo-cache-controlno-preview
go-importgithub.com/triggerdotdev/trigger.dev git https://github.com/triggerdotdev/trigger.dev.git
octolytics-dimension-user_id95297378
octolytics-dimension-user_logintriggerdotdev
octolytics-dimension-repository_id572570113
octolytics-dimension-repository_nwotriggerdotdev/trigger.dev
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id572570113
octolytics-dimension-repository_network_root_nwotriggerdotdev/trigger.dev
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release12fff95459b24b0f882a66698ee6a36a5934d247
ui-targetfull
theme-color#1e2327
color-schemelight dark
react-app-nameissues

Links:

Skip to contenthttps://github.com/triggerdotdev/trigger.dev/issues/3520#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Ftriggerdotdev%2Ftrigger.dev%2Fissues%2F3520
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Ftriggerdotdev%2Ftrigger.dev%2Fissues%2F3520
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fvoltron%2Fissues_fragments%2Fissue_layout&source=header-repo&source_repo=triggerdotdev%2Ftrigger.dev
Reloadhttps://github.com/triggerdotdev/trigger.dev/issues/3520
Reloadhttps://github.com/triggerdotdev/trigger.dev/issues/3520
Reloadhttps://github.com/triggerdotdev/trigger.dev/issues/3520
triggerdotdevhttps://github.com/triggerdotdev
trigger.devhttps://github.com/triggerdotdev/trigger.dev
Codehttps://github.com/triggerdotdev/trigger.dev
Issues203 (203)https://github.com/triggerdotdev/trigger.dev/issues
Pull requests67 (67)https://github.com/triggerdotdev/trigger.dev/pulls
Discussionshttps://github.com/triggerdotdev/trigger.dev/discussions
Actionshttps://github.com/triggerdotdev/trigger.dev/actions
Projectshttps://github.com/triggerdotdev/trigger.dev/projects
Security and quality39 (39)https://github.com/triggerdotdev/trigger.dev/security
Insightshttps://github.com/triggerdotdev/trigger.dev/pulse
https://github.com/triggerdotdev/trigger.dev
New issuehttps://github.com/login?return_to=https://github.com/triggerdotdev/trigger.dev/issues/3520
New issuehttps://github.com/login?return_to=https://github.com/triggerdotdev/trigger.dev/issues/3520
#3524https://github.com/triggerdotdev/trigger.dev/pull/3524
Helm chart pins ClickHouse 25.6.1, which has a memory-tracker overflow that breaks the dashboard until pod restarthttps://github.com/triggerdotdev/trigger.dev/issues/3520#top
#3524https://github.com/triggerdotdev/trigger.dev/pull/3524
https://github.com/brentshulman-silkline
brentshulman-silklinehttps://github.com/brentshulman-silkline
on May 5, 2026https://github.com/triggerdotdev/trigger.dev/issues/3520#issue-4381089994
https://github.com/nicktrn
nicktrnhttps://github.com/nicktrn
on May 5, 2026https://github.com/triggerdotdev/trigger.dev/issues/3520#issuecomment-4378157135
@brentshulman-silklinehttps://github.com/brentshulman-silkline
nicktrnhttps://github.com/nicktrn
on May 5, 2026https://github.com/triggerdotdev/trigger.dev/issues/3520#event-25170258768
fix(helm): bump clickhouse subchart 9.3.7 -> 9.4.4 (clickhouse 25.7.5) (https://github.com/triggerdotdev/trigger.dev/commit/a8966a40aba3bd0e9b4d4513cee65845493e4aa3
https://docs.github.com/articles/closing-issues-via-commit-messages
a8966a4https://github.com/triggerdotdev/trigger.dev/commit/a8966a40aba3bd0e9b4d4513cee65845493e4aa3
nicktrnhttps://github.com/nicktrn
completedhttps://github.com/triggerdotdev/trigger.dev/issues?q=is%3Aissue%20state%3Aclosed%20archived%3Afalse%20reason%3Acompleted
#3524https://github.com/triggerdotdev/trigger.dev/pull/3524
on May 5, 2026https://github.com/triggerdotdev/trigger.dev/issues/3520#event-25170258816
Sign up for freehttps://github.com/signup?return_to=https://github.com/triggerdotdev/trigger.dev/issues/3520
Sign in to commenthttps://github.com/login?return_to=https://github.com/triggerdotdev/trigger.dev/issues/3520
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.