Title: Support caller-supplied `Content-Encoding` for pre-compressed request bodies · Issue #996 · apify/apify-client-python · GitHub
Open Graph Title: Support caller-supplied `Content-Encoding` for pre-compressed request bodies · Issue #996 · apify/apify-client-python
X Title: Support caller-supplied `Content-Encoding` for pre-compressed request bodies · Issue #996 · apify/apify-client-python
Description: Follow-up from a review comment on #987: #987 (comment) Problem There is no way for a caller to hand the client an already-encoded request body. Before #987, _prepare_request_call compressed every bytes-like body and merged its own Conte...
Open Graph Description: Follow-up from a review comment on #987: #987 (comment) Problem There is no way for a caller to hand the client an already-encoded request body. Before #987, _prepare_request_call compressed every ...
X Description: Follow-up from a review comment on #987: #987 (comment) Problem There is no way for a caller to hand the client an already-encoded request body. Before #987, _prepare_request_call compressed every ...
Opengraph URL: https://github.com/apify/apify-client-python/issues/996
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Support caller-supplied `Content-Encoding` for pre-compressed request bodies","articleBody":"Follow-up from a review comment on #987: https://github.com/apify/apify-client-python/pull/987#discussion_r3704749196\n\n## Problem\n\nThere is no way for a caller to hand the client an already-encoded request body.\n\n- Before #987, `_prepare_request_call` compressed every bytes-like body and merged its own `Content-Encoding` last, so a caller-supplied value was silently overwritten.\n- Since #987, a caller-supplied `Content-Encoding` is dropped whenever compression is skipped, so a pre-compressed body goes out labeled as unencoded.\n\nThe API supports this and documents it. [`PUT key-value-store record`](https://docs.apify.com/api/v2/key-value-store-record-put) documents a `Content-Encoding` request header accepting `gzip`, `br`, `deflate`, and `identity`: \"To save bandwidth, storage, and speed up your upload, send the request payload compressed with Gzip compression and add the `Content-Encoding: gzip` header. It is possible to set up another compression type with `Content-Encoding` request header.\" And per the [key-value store docs](https://docs.apify.com/platform/storage/key-value-store), \"records are stored exactly as you upload them\", with the encoding taken from that header — so on that endpoint the header is record metadata, not just a transport hint, and dropping it stores compressed bytes under an unencoded label.\n\nHonoring the header also unlocks encodings the client ships no compressor for (`deflate`, `zstd`, ...) without having to add one.\n\n## How the JS client solves it\n\n`apify-client-js` treats a caller-supplied `content-encoding` as \"hands off\" — [`src/interceptors.ts#L81-L82`](https://github.com/apify/apify-client-js/blob/fd619c056e64743843ece519fe56ae87f0c1e687/src/interceptors.ts#L81-L82):\n\n```ts\nasync function maybeCompressRequest(config: ApifyRequestConfig): Promise\u003cApifyRequestConfig> {\n if (config.headers?.['content-encoding']) return config;\n\n const maybeCompressed = await maybeCompressValue(config.data);\n ...\n}\n```\n\nNo opt-out parameter and no dedicated compressor — the header alone is the signal. (`maybeCompressValue` also skips bodies below `MIN_COMPRESS_BYTES = 1024`, which is #934 on the Python side.)\n\n## Proposal\n\n1. **Honor a caller-supplied `Content-Encoding` in `_prepare_request_call`**, matching JS. Precedence: caller `Content-Encoding` -> content-type heuristic (`is_compressible_content_type`) -> compress. The header is forwarded verbatim, so any encoding the backend accepts works, and `identity` becomes a per-request opt-out for free.\n\n ```python\n caller_encoding = next((v for k, v in headers.items() if k.lower() == 'content-encoding'), None)\n if caller_encoding is None and is_compressible_content_type(content_type):\n data = self._http_compressor.compress(data)\n headers = self._merge_headers(headers, {'Content-Encoding': self._http_compressor.content_encoding})\n ```\n\n Explicitly out of scope: stacking the client's encoding on top of the caller's (`Content-Encoding: gzip, br`) — the backend does not document the list form.\n\n This reverses the drop-the-header behavior added in #987 and flips `test_prepare_request_call_replaces_caller_content_encoding` plus the two `..._drops_caller_content_encoding_...` tests.\n\n2. **Expose `content_encoding` on `KeyValueStoreClient.set_record`** (sync and async). Step 1 on its own is barely reachable: no public resource method takes per-request headers, so the only entry points are the client-wide `ApifyClient(headers=...)`, which would apply the header to every request, and the private `_http_client.call`. `set_record` is where the API documents the header, so that is where it belongs.\n\nRejected: a per-request `compress=False` keyword on `HttpClient.call`. `HttpClient`/`HttpClientAsync` are a public ABC, and once resource clients pass the new keyword, existing custom implementations break. Headers already flow through, so step 1 covers the same need without touching that contract.\n\nDocs: add a pre-compressed bodies section to the HTTP compression concept page.\n\n*✍️ Drafted by Claude Code*\n","author":{"url":"https://github.com/vdusek","@type":"Person","name":"vdusek"},"datePublished":"2026-08-03T14:43:25Z","url":"https://github.com/apify/apify-client-python/issues/996"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:ddc68c97-71e4-9512-3fa5-5831ffcfc0e7 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | AAB6:1DDD34:330B35:47FF12:6B005D4D |
| html-safe-nonce | 68bf13fa85bbbe592335d7a1164d51bd158d00d042e581bc2f61dcdd4552726b |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJBQUI2OjFEREQzNDozMzBCMzU6NDdGRjEyOjZCMDA1RDREIiwidmlzaXRvcl9pZCI6IjIwMDU5ODM1NTU2MzY3MTQ4MjkiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | 778d398b904d50ec97fdcd5b6e268ab3e32d9085ab8bc0c667b14dec207d5eab |
| hovercard-subject-tag | issue:5051035384 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/apify/apify-client-python/996/issue_layout |
| twitter:image | https://opengraph.githubassets.com/30136b19690c97d4920f2e731031c04135c6a2d9c537d4716abd87b80369ae06/apify/apify-client-python/issues/996 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/30136b19690c97d4920f2e731031c04135c6a2d9c537d4716abd87b80369ae06/apify/apify-client-python/issues/996 |
| og:image:alt | Follow-up from a review comment on #987: #987 (comment) Problem There is no way for a caller to hand the client an already-encoded request body. Before #987, _prepare_request_call compressed every ... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | vdusek |
| hostname | github.com |
| expected-hostname | github.com |
| None | 29be975efff6507a5d135de9c7d634cf7ff8f0c55222a64386ab3202209aaf73 |
| turbo-cache-control | no-preview |
| go-import | github.com/apify/apify-client-python git https://github.com/apify/apify-client-python.git |
| octolytics-dimension-user_id | 24586296 |
| octolytics-dimension-user_login | apify |
| octolytics-dimension-repository_id | 164647835 |
| octolytics-dimension-repository_nwo | apify/apify-client-python |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 164647835 |
| octolytics-dimension-repository_network_root_nwo | apify/apify-client-python |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | eacd927ba6112e3b01ba99c762bd699a4a4f5df7 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width