Title: storage: zero-copy BidiReadObjectResponse parsing and off-loop CRC32C verification for async multi-range reads · Issue #18629 · googleapis/google-cloud-python · GitHub
Open Graph Title: storage: zero-copy BidiReadObjectResponse parsing and off-loop CRC32C verification for async multi-range reads · Issue #18629 · googleapis/google-cloud-python
X Title: storage: zero-copy BidiReadObjectResponse parsing and off-loop CRC32C verification for async multi-range reads · Issue #18629 · googleapis/google-cloud-python
Description: Summary of the feature request AsyncMultiRangeDownloader reads from Rapid (zonal) buckets over the BidiReadObject gRPC stream. Per process, throughput is bound by the asyncio event-loop thread: every 2 MiB BidiReadObjectResponse is proce...
Open Graph Description: Summary of the feature request AsyncMultiRangeDownloader reads from Rapid (zonal) buckets over the BidiReadObject gRPC stream. Per process, throughput is bound by the asyncio event-loop thread: eve...
X Description: Summary of the feature request AsyncMultiRangeDownloader reads from Rapid (zonal) buckets over the BidiReadObject gRPC stream. Per process, throughput is bound by the asyncio event-loop thread: eve...
Opengraph URL: https://github.com/googleapis/google-cloud-python/issues/18629
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"storage: zero-copy BidiReadObjectResponse parsing and off-loop CRC32C verification for async multi-range reads","articleBody":"## Summary of the feature request\n\n`AsyncMultiRangeDownloader` reads from Rapid (zonal) buckets over the `BidiReadObject` gRPC stream. Per process, throughput is bound by the asyncio event-loop thread: every 2 MiB `BidiReadObjectResponse` is processed on that thread, under the GIL, before the next message can be received, and all streams of a process queue behind it.\n\nProfiling one process reading 16 streams at 2.0 GB/s (`c4-standard-192`, bucket in the same zone, py-spy on the loop thread) gives 318 ms of loop-thread time per GB:\n\n| Step on the loop thread | ms per GB |\n| :--- | ---: |\n| gRPC receive copy | 123 |\n| `BidiReadObjectResponse.deserialize` (upb copies the payload: copy 1) | 69 |\n| `checksummed_data.content` (proto-plus returns a new `bytes`: copy 2) | 58 |\n| `google_crc32c.value()` inline | 38 |\n| asyncio overhead | 30 |\n\nTwo of the three large items are avoidable:\n\n1. The payload is copied twice by the generated parser. A parser that reads the message in place over the `bytes` object gRPC hands to the deserializer can give the read path a `memoryview` of the received message instead (257 us per 2 MiB message today, 5 us in place).\n2. CRC32C verification runs on the loop thread even though `google-crc32c` already releases the GIL for inputs of 1 MiB or more; hashing on a small worker pool overlaps it with the next receive.\n\nBoth need `google_crc32c.value()` to accept a `memoryview`; today it parses its argument with `y#` and raises `TypeError: argument 1 must be read-only bytes-like object`, so a view into the gRPC buffer cannot be verified without copying it back to `bytes`.\n\nProposed changes (three PRs, independent except that the parser depends on the `google-crc32c` release):\n\n- #18626 `google-crc32c`: `value()` / `extend()` / `Checksum.update()` accept any contiguous bytes-like object; the GIL is released for `bytes`-owned memory of 1 MiB or more (a `bytes` or a `memoryview` of one).\n- #18627 `google-cloud-storage`: opt-in zero-copy parser, `create_mrd(..., zero_copy=True)` (or `GOOGLE_CLOUD_STORAGE_ZERO_COPY_BIDI_READ=1`); only active when the installed `google-crc32c` can hash a `memoryview`, otherwise the generated parser is used. With it on, `buffer.write()` receives a read-only `memoryview` of the received message.\n- #18628 `google-cloud-storage`: chunks of 1 MiB or more are hashed on a 2-thread pool and written to their buffer only after the checksum matches, in arrival order; on by default, `GOOGLE_CLOUD_STORAGE_CRC32C_OFFLOAD_MIN_BYTES=0` restores inline verification.\n\n`fsspec/gcsfs` is the main consumer of this path and would opt in; its sinks copy from the view into the caller's buffer.\n\n## Desired code experience\n\n```python\nimport io\nfrom google.cloud.storage.asyncio.async_grpc_client import AsyncGrpcClient\nfrom google.cloud.storage.asyncio.async_multi_range_downloader import AsyncMultiRangeDownloader\n\nclient = AsyncGrpcClient()\nmrd = await AsyncMultiRangeDownloader.create_mrd(\n client, \"my-rapid-bucket\", \"my-object\", zero_copy=True\n)\nbuffers = [io.BytesIO() for _ in range(4)]\n# buffer.write() receives memoryview chunks that alias the received gRPC\n# message; BytesIO, binary files and bytearray sinks accept them unchanged.\nawait mrd.download_ranges([(i * 16 \u003c\u003c 20, 16 \u003c\u003c 20, buf) for i, buf in enumerate(buffers)])\n```\n\nOff-loop verification needs no code change; it applies to `download_ranges` with `enable_checksum=True`.\n\n## Expected results\n\nSame machine and bucket, 16 streams per process, 4 x 16 MiB per `download_ranges` call into preallocated buffers, CRC32C verification on, aggregate GB/s:\n\n| Processes | `main` | zero-copy parser (#18627, median of 3 interleaved runs) | off-loop CRC32C (#18628, median of 9) |\n| :---: | :---: | :---: | :---: |\n| 1 | 1.55 | 2.62 | about 2.0 for both arms (unchanged) |\n| 8 | 9.76 | 11.29 | 8.17 -> 9.59 (+17%, led in 7 of 9 paired runs) |\n\nWith all three changes together, one process goes from 1.6 to 3.5 GB/s and eight processes from 9.8 to 13.5 GB/s; loop-thread time per GB drops from 318 ms to 195 ms.\n\nEnvironment: `google-cloud-storage` at `053dbc9872ad`, `google-crc32c` 1.9.0 (`main` arm), grpcio 1.84.0, protobuf 7.36 (upb backend), Python 3.13, `c4-standard-192` in `us-central1-c`.\n","author":{"url":"https://github.com/yuxin00j","@type":"Person","name":"yuxin00j"},"datePublished":"2026-10-12T02:27:59Z","url":"https://github.com/googleapis/google-cloud-python/issues/18629"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:ffad11c9-f524-d5ee-9f77-f77b54806732 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | 8F2A:9C1C:7CC6A3:A8F0F1:6AFE587A |
| html-safe-nonce | 4ccc075c09c016002a159516d48f0fd25c55be67414259c09fa1bf6d655341fd |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI4RjJBOjlDMUM6N0NDNkEzOkE4RjBGMTo2QUZFNTg3QSIsInZpc2l0b3JfaWQiOiI1Nzk0NDkwMDU2MTQzOTUxOTk0IiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | bb2d2d890313f653d84ecbce8e96ec8f1197bb078c7bd389a4cb303ba022e9a8 |
| hovercard-subject-tag | issue:5807492596 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/googleapis/google-cloud-python/18629/issue_layout |
| twitter:image | https://opengraph.githubassets.com/966fae37d0d69fe7ecbf6d06cf79bc3a19136663da049c31dea1e6fabc3187dd/googleapis/google-cloud-python/issues/18629 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/966fae37d0d69fe7ecbf6d06cf79bc3a19136663da049c31dea1e6fabc3187dd/googleapis/google-cloud-python/issues/18629 |
| og:image:alt | Summary of the feature request AsyncMultiRangeDownloader reads from Rapid (zonal) buckets over the BidiReadObject gRPC stream. Per process, throughput is bound by the asyncio event-loop thread: eve... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | yuxin00j |
| hostname | github.com |
| expected-hostname | github.com |
| None | b3c0f155eafc27e60db6e254d1ffff0e072e9ab3b4018a5ee518797c19d89f86 |
| turbo-cache-control | no-preview |
| go-import | github.com/googleapis/google-cloud-python git https://github.com/googleapis/google-cloud-python.git |
| octolytics-dimension-user_id | 16785467 |
| octolytics-dimension-user_login | googleapis |
| octolytics-dimension-repository_id | 16316451 |
| octolytics-dimension-repository_nwo | googleapis/google-cloud-python |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 16316451 |
| octolytics-dimension-repository_network_root_nwo | googleapis/google-cloud-python |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 4a10310af51a1bfffbedc4fd81cb582acf0cb0ff |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width