Title: Celery integration strips user custom headers from `task.request.headers` · Issue #5566 · getsentry/sentry-python · GitHub
Open Graph Title: Celery integration strips user custom headers from `task.request.headers` · Issue #5566 · getsentry/sentry-python
X Title: Celery integration strips user custom headers from `task.request.headers` · Issue #5566 · getsentry/sentry-python
Description: How do you use Sentry? Sentry Saas (sentry.io) Version 2.53.0 Steps to Reproduce Celery integration strips user custom headers from task.request.headers Environment Package Version sentry-sdk 2.53.0 celery 5.6.2 python 3.12 Description T...
Open Graph Description: How do you use Sentry? Sentry Saas (sentry.io) Version 2.53.0 Steps to Reproduce Celery integration strips user custom headers from task.request.headers Environment Package Version sentry-sdk 2.53....
X Description: How do you use Sentry? Sentry Saas (sentry.io) Version 2.53.0 Steps to Reproduce Celery integration strips user custom headers from task.request.headers Environment Package Version sentry-sdk 2.53....
Opengraph URL: https://github.com/getsentry/sentry-python/issues/5566
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Celery integration strips user custom headers from `task.request.headers`","articleBody":"### How do you use Sentry?\n\nSentry Saas (sentry.io)\n\n### Version\n\n2.53.0\n\n### Steps to Reproduce\n\n# Celery integration strips user custom headers from `task.request.headers`\n\n## Environment\n\n| Package | Version |\n|------------|---------|\n| sentry-sdk | 2.53.0 |\n| celery | 5.6.2 |\n| python | 3.12 |\n\n## Description\n\nThe Celery integration's `_update_celery_task_headers` function in `sentry_sdk/integrations/celery/__init__.py` strips user-provided custom headers from reaching `task.request.headers` on the worker side.\n\nCelery has a long-standing bug ([celery/celery#4875](https://github.com/celery/celery/issues/4875)) where task headers placed at the top level of the message are not reliably accessible via `task.request.headers`. Instead, only headers nested inside the inner `\"headers\"` dict (i.e., `message[\"headers\"][\"headers\"]`) survive to `task.request.headers` on the worker.\n\nThe Sentry integration is aware of this issue -- there is even a comment referencing `#4875` at line 219. The integration correctly places its own tracing headers (`sentry-trace`, `baggage`, etc.) into the inner `\"headers\"` dict. However, it does **not** copy user-provided custom headers into the inner dict, effectively making them invisible to `task.request.headers` on the worker.\n\n## Root Cause\n\nIn `sentry_sdk/integrations/celery/__init__.py`, the `_update_celery_task_headers` function (lines 162-234):\n\n**Step 1** (line 169): The function copies all original headers (including user custom headers) into `updated_headers`:\n\n```python\nupdated_headers = original_headers.copy()\n```\n\n**Step 2** (line 224): It creates the inner `\"headers\"` dict and populates it with only Sentry's tracing headers (`sentry-trace`, `baggage`, etc.):\n\n```python\nupdated_headers.setdefault(\"headers\", {}).update(headers)\n```\n\nHere, `headers` is the dict returned by `sentry_sdk.get_isolation_scope().iter_trace_propagation_headers()` -- it contains only Sentry tracing headers, not user custom headers.\n\n**Step 3** (lines 230-232): It then copies only keys prefixed with `\"sentry-\"` from the outer dict into the inner dict:\n\n```python\nfor key, value in updated_headers.items():\n if key.startswith(\"sentry-\"):\n updated_headers[\"headers\"][key] = value\n```\n\nThis loop explicitly filters to `\"sentry-\"` prefixed keys. Any user-provided custom headers (e.g., `my_custom_key`, `correlation_id`, `tenant_id`) are left stranded at the top level and never copied into the inner `\"headers\"` dict.\n\n**Result:** Due to celery/celery#4875, the worker's `task.request.headers` only reflects the inner dict contents. User custom headers are silently dropped.\n\n## Steps to Reproduce\n\n### Minimal reproduction script\n\n```python\n# repro.py\nimport sentry_sdk\nfrom sentry_sdk.integrations.celery import CeleryIntegration\nfrom celery import Celery\nfrom celery.signals import task_prerun\n\nsentry_sdk.init(\n dsn=\"https://examplePublicKey@o0.ingest.sentry.io/0\",\n integrations=[CeleryIntegration()],\n traces_sample_rate=1.0,\n)\n\napp = Celery(\"repro\", broker=\"redis://localhost:6379/0\")\n\n\n@app.task(bind=True)\ndef my_task(self):\n # This is what users expect to work:\n value_from_headers = self.request.headers.get(\"my_custom_key\")\n print(f\"self.request.headers.get('my_custom_key') = {value_from_headers}\")\n # ^ Returns None (BUG)\n\n # Workaround -- the value IS present at the top level of the request:\n value_from_attr = getattr(self.request, \"my_custom_key\", None)\n print(f\"getattr(self.request, 'my_custom_key') = {value_from_attr}\")\n # ^ Returns 'my_value'\n\n # Show what's actually in headers:\n print(f\"self.request.headers = {self.request.headers}\")\n # ^ Only contains sentry-trace, baggage, sentry-task-enqueued-time, etc.\n\n\nif __name__ == \"__main__\":\n # Send the task with a custom header\n my_task.apply_async(args=[], headers={\"my_custom_key\": \"my_value\"})\n print(\"Task sent with headers={'my_custom_key': 'my_value'}\")\n```\n\n### Steps\n\n1. Start a Redis instance (or any broker).\n2. Run the worker: `celery -A repro worker --loglevel=info`\n3. Send the task: `python repro.py`\n4. Observe the worker output:\n ```\n self.request.headers.get('my_custom_key') = None\n getattr(self.request, 'my_custom_key') = my_value\n self.request.headers = {'sentry-trace': '...', 'baggage': '...', 'sentry-task-enqueued-time': ...}\n ```\n5. Disable the Sentry integration (remove `sentry_sdk.init(...)`) and repeat.\n6. Observe the worker output now correctly shows:\n ```\n self.request.headers.get('my_custom_key') = my_value\n ```\n\n## Expected Behavior\n\nCustom headers passed via the `headers=` kwarg should be available in `task.request.headers` on the worker, regardless of whether the Sentry integration is enabled. The Sentry integration should not alter the visibility of user-provided headers.\n\n## Actual Behavior\n\nWhen the Sentry integration is enabled, user custom headers are silently dropped from `task.request.headers`. They exist at the top level of the message (accessible via `getattr(self.request, 'key')`) but not in the inner `\"headers\"` dict that Celery exposes as `task.request.headers`.\n\n## Suggested Fix\n\nIn `_update_celery_task_headers`, after copying Sentry's own headers into the inner dict, also copy user-provided custom headers from `original_headers` so they survive the celery/celery#4875 behavior.\n\n### Option A: Copy all non-internal keys from original headers\n\nAfter line 232, add:\n\n```python\n# Preserve user-provided custom headers in the inner \"headers\" dict\n# so they survive to task.request.headers on the worker (celery#4875).\nfor key, value in original_headers.items():\n if key != \"headers\" and key not in updated_headers.get(\"headers\", {}):\n updated_headers[\"headers\"][key] = value\n```\n\n### Option B: Remove the `\"sentry-\"` prefix filter\n\nReplace the loop at lines 230-232:\n\n```python\n# Before (only copies sentry-* keys):\nfor key, value in updated_headers.items():\n if key.startswith(\"sentry-\"):\n updated_headers[\"headers\"][key] = value\n\n# After (copies all non-internal keys):\nfor key, value in updated_headers.items():\n if key != \"headers\" and not isinstance(value, dict):\n updated_headers[\"headers\"][key] = value\n```\n\nOption A is preferred because it uses `original_headers` as the source (avoiding accidentally copying Sentry's top-level additions like `sentry-task-enqueued-time` twice) and respects any keys the inner dict already has via `setdefault`.\n\n## Workaround\n\nUntil this is fixed, users can access custom headers via top-level request attributes instead of `task.request.headers`:\n\n```python\n@app.task(bind=True)\ndef my_task(self):\n # Instead of:\n # value = self.request.headers.get(\"my_custom_key\")\n\n # Use:\n value = getattr(self.request, \"my_custom_key\", None)\n```\n\nNote: this workaround is fragile and relies on Celery's internal behavior of flattening top-level message headers into request attributes.\n\n## References\n\n- [celery/celery#4875](https://github.com/celery/celery/issues/4875) -- the upstream Celery bug that necessitates the inner `\"headers\"` dict workaround\n- [`sentry_sdk/integrations/celery/__init__.py` lines 162-234](https://github.com/getsentry/sentry-python/blob/master/sentry_sdk/integrations/celery/__init__.py) -- the `_update_celery_task_headers` function\n\n\n### Expected Result\n\nThe headers included on the request are preserved.\n\n### Actual Result\n\nThe headers included on the request are stripped out.","author":{"url":"https://github.com/jamesrusso","@type":"Person","name":"jamesrusso"},"datePublished":"2026-03-01T20:24:19Z","url":"https://github.com/getsentry/sentry-python/issues/5566"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:c6239b42-59aa-b999-652c-940c6c43d452 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | D50A:28815B:8A45A2:BA6CDC:6B00E39F |
| html-safe-nonce | 858ee5fad00c6ef3d52f35f97497a01a2bb83cfe36fff11ea25b801143688183 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJENTBBOjI4ODE1Qjo4QTQ1QTI6QkE2Q0RDOjZCMDBFMzlGIiwidmlzaXRvcl9pZCI6IjI0MzI5NzkwNTk0OTIzMTYwNjMiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | 2f8a6afae3a45e4e649499279659dc5fa90dca21e79f3f4e3520cebace428ac3 |
| hovercard-subject-tag | issue:4008192609 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/getsentry/sentry-python/5566/issue_layout |
| twitter:image | https://opengraph.githubassets.com/8e827c21fd3469911fc1cd525a01872c83c12a30d1bf8212a655dcff33a99b2c/getsentry/sentry-python/issues/5566 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/8e827c21fd3469911fc1cd525a01872c83c12a30d1bf8212a655dcff33a99b2c/getsentry/sentry-python/issues/5566 |
| og:image:alt | How do you use Sentry? Sentry Saas (sentry.io) Version 2.53.0 Steps to Reproduce Celery integration strips user custom headers from task.request.headers Environment Package Version sentry-sdk 2.53.... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | jamesrusso |
| hostname | github.com |
| expected-hostname | github.com |
| None | 27088b6977484d0d3321d9a11c907c6d6637b315cfa359ef7952dbf6292f9b14 |
| turbo-cache-control | no-preview |
| go-import | github.com/getsentry/sentry-python git https://github.com/getsentry/sentry-python.git |
| octolytics-dimension-user_id | 1396951 |
| octolytics-dimension-user_login | getsentry |
| octolytics-dimension-repository_id | 137814779 |
| octolytics-dimension-repository_nwo | getsentry/sentry-python |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 137814779 |
| octolytics-dimension-repository_network_root_nwo | getsentry/sentry-python |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 624179fb8b88d26ba5bba08894eb60a1c5a1d93d |
| ui-target | canary-2 |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width