René's URL Explorer Experiment


Title: storage: zero-copy BidiReadObjectResponse parsing and off-loop CRC32C verification for async multi-range reads · Issue #18629 · googleapis/google-cloud-python · GitHub

Open Graph Title: storage: zero-copy BidiReadObjectResponse parsing and off-loop CRC32C verification for async multi-range reads · Issue #18629 · googleapis/google-cloud-python

X Title: storage: zero-copy BidiReadObjectResponse parsing and off-loop CRC32C verification for async multi-range reads · Issue #18629 · googleapis/google-cloud-python

Description: Summary of the feature request AsyncMultiRangeDownloader reads from Rapid (zonal) buckets over the BidiReadObject gRPC stream. Per process, throughput is bound by the asyncio event-loop thread: every 2 MiB BidiReadObjectResponse is proce...

Open Graph Description: Summary of the feature request AsyncMultiRangeDownloader reads from Rapid (zonal) buckets over the BidiReadObject gRPC stream. Per process, throughput is bound by the asyncio event-loop thread: eve...

X Description: Summary of the feature request AsyncMultiRangeDownloader reads from Rapid (zonal) buckets over the BidiReadObject gRPC stream. Per process, throughput is bound by the asyncio event-loop thread: eve...

Opengraph URL: https://github.com/googleapis/google-cloud-python/issues/18629

X: @github

direct link

Domain: github.com


Hey, it has json ld scripts:
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"storage: zero-copy BidiReadObjectResponse parsing and off-loop CRC32C verification for async multi-range reads","articleBody":"## Summary of the feature request\n\n`AsyncMultiRangeDownloader` reads from Rapid (zonal) buckets over the `BidiReadObject` gRPC stream. Per process, throughput is bound by the asyncio event-loop thread: every 2 MiB `BidiReadObjectResponse` is processed on that thread, under the GIL, before the next message can be received, and all streams of a process queue behind it.\n\nProfiling one process reading 16 streams at 2.0 GB/s (`c4-standard-192`, bucket in the same zone, py-spy on the loop thread) gives 318 ms of loop-thread time per GB:\n\n| Step on the loop thread | ms per GB |\n| :--- | ---: |\n| gRPC receive copy | 123 |\n| `BidiReadObjectResponse.deserialize` (upb copies the payload: copy 1) | 69 |\n| `checksummed_data.content` (proto-plus returns a new `bytes`: copy 2) | 58 |\n| `google_crc32c.value()` inline | 38 |\n| asyncio overhead | 30 |\n\nTwo of the three large items are avoidable:\n\n1. The payload is copied twice by the generated parser. A parser that reads the message in place over the `bytes` object gRPC hands to the deserializer can give the read path a `memoryview` of the received message instead (257 us per 2 MiB message today, 5 us in place).\n2. CRC32C verification runs on the loop thread even though `google-crc32c` already releases the GIL for inputs of 1 MiB or more; hashing on a small worker pool overlaps it with the next receive.\n\nBoth need `google_crc32c.value()` to accept a `memoryview`; today it parses its argument with `y#` and raises `TypeError: argument 1 must be read-only bytes-like object`, so a view into the gRPC buffer cannot be verified without copying it back to `bytes`.\n\nProposed changes (three PRs, independent except that the parser depends on the `google-crc32c` release):\n\n- #18626 `google-crc32c`: `value()` / `extend()` / `Checksum.update()` accept any contiguous bytes-like object; the GIL is released for `bytes`-owned memory of 1 MiB or more (a `bytes` or a `memoryview` of one).\n- #18627 `google-cloud-storage`: opt-in zero-copy parser, `create_mrd(..., zero_copy=True)` (or `GOOGLE_CLOUD_STORAGE_ZERO_COPY_BIDI_READ=1`); only active when the installed `google-crc32c` can hash a `memoryview`, otherwise the generated parser is used. With it on, `buffer.write()` receives a read-only `memoryview` of the received message.\n- #18628 `google-cloud-storage`: chunks of 1 MiB or more are hashed on a 2-thread pool and written to their buffer only after the checksum matches, in arrival order; on by default, `GOOGLE_CLOUD_STORAGE_CRC32C_OFFLOAD_MIN_BYTES=0` restores inline verification.\n\n`fsspec/gcsfs` is the main consumer of this path and would opt in; its sinks copy from the view into the caller's buffer.\n\n## Desired code experience\n\n```python\nimport io\nfrom google.cloud.storage.asyncio.async_grpc_client import AsyncGrpcClient\nfrom google.cloud.storage.asyncio.async_multi_range_downloader import AsyncMultiRangeDownloader\n\nclient = AsyncGrpcClient()\nmrd = await AsyncMultiRangeDownloader.create_mrd(\n    client, \"my-rapid-bucket\", \"my-object\", zero_copy=True\n)\nbuffers = [io.BytesIO() for _ in range(4)]\n# buffer.write() receives memoryview chunks that alias the received gRPC\n# message; BytesIO, binary files and bytearray sinks accept them unchanged.\nawait mrd.download_ranges([(i * 16 \u003c\u003c 20, 16 \u003c\u003c 20, buf) for i, buf in enumerate(buffers)])\n```\n\nOff-loop verification needs no code change; it applies to `download_ranges` with `enable_checksum=True`.\n\n## Expected results\n\nSame machine and bucket, 16 streams per process, 4 x 16 MiB per `download_ranges` call into preallocated buffers, CRC32C verification on, aggregate GB/s:\n\n| Processes | `main` | zero-copy parser (#18627, median of 3 interleaved runs) | off-loop CRC32C (#18628, median of 9) |\n| :---: | :---: | :---: | :---: |\n| 1 | 1.55 | 2.62 | about 2.0 for both arms (unchanged) |\n| 8 | 9.76 | 11.29 | 8.17 -> 9.59 (+17%, led in 7 of 9 paired runs) |\n\nWith all three changes together, one process goes from 1.6 to 3.5 GB/s and eight processes from 9.8 to 13.5 GB/s; loop-thread time per GB drops from 318 ms to 195 ms.\n\nEnvironment: `google-cloud-storage` at `053dbc9872ad`, `google-crc32c` 1.9.0 (`main` arm), grpcio 1.84.0, protobuf 7.36 (upb backend), Python 3.13, `c4-standard-192` in `us-central1-c`.\n","author":{"url":"https://github.com/yuxin00j","@type":"Person","name":"yuxin00j"},"datePublished":"2026-10-12T02:27:59Z","url":"https://github.com/googleapis/google-cloud-python/issues/18629"}

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format)
route-controllervoltron_issues_fragments
route-actionissue_layout
fetch-noncev2:ffad11c9-f524-d5ee-9f77-f77b54806732
current-catalog-service-hash81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114
request-id8F2A:9C1C:7CC6A3:A8F0F1:6AFE587A
html-safe-nonce4ccc075c09c016002a159516d48f0fd25c55be67414259c09fa1bf6d655341fd
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI4RjJBOjlDMUM6N0NDNkEzOkE4RjBGMTo2QUZFNTg3QSIsInZpc2l0b3JfaWQiOiI1Nzk0NDkwMDU2MTQzOTUxOTk0IiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0=
visitor-hmacbb2d2d890313f653d84ecbce8e96ec8f1197bb078c7bd389a4cb303ba022e9a8
hovercard-subject-tagissue:5807492596
github-keyboard-shortcutsrepository,issues,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///voltron/issues_fragments/issue_layout
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/googleapis/google-cloud-python/18629/issue_layout
twitter:imagehttps://opengraph.githubassets.com/966fae37d0d69fe7ecbf6d06cf79bc3a19136663da049c31dea1e6fabc3187dd/googleapis/google-cloud-python/issues/18629
twitter:cardsummary_large_image
og:imagehttps://opengraph.githubassets.com/966fae37d0d69fe7ecbf6d06cf79bc3a19136663da049c31dea1e6fabc3187dd/googleapis/google-cloud-python/issues/18629
og:image:altSummary of the feature request AsyncMultiRangeDownloader reads from Rapid (zonal) buckets over the BidiReadObject gRPC stream. Per process, throughput is bound by the asyncio event-loop thread: eve...
og:image:width1200
og:image:height600
og:site_nameGitHub
og:typeobject
og:author:usernameyuxin00j
hostnamegithub.com
expected-hostnamegithub.com
Noneb3c0f155eafc27e60db6e254d1ffff0e072e9ab3b4018a5ee518797c19d89f86
turbo-cache-controlno-preview
go-importgithub.com/googleapis/google-cloud-python git https://github.com/googleapis/google-cloud-python.git
octolytics-dimension-user_id16785467
octolytics-dimension-user_logingoogleapis
octolytics-dimension-repository_id16316451
octolytics-dimension-repository_nwogoogleapis/google-cloud-python
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id16316451
octolytics-dimension-repository_network_root_nwogoogleapis/google-cloud-python
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release4a10310af51a1bfffbedc4fd81cb582acf0cb0ff
ui-targetfull
theme-color#1e2327
color-schemelight dark
react-app-nameissues

Links:

Skip to contenthttps://github.com/googleapis/google-cloud-python/issues/18629#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fgoogleapis%2Fgoogle-cloud-python%2Fissues%2F18629
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fgoogleapis%2Fgoogle-cloud-python%2Fissues%2F18629
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fvoltron%2Fissues_fragments%2Fissue_layout&source=header-repo&source_repo=googleapis%2Fgoogle-cloud-python
Reloadhttps://github.com/googleapis/google-cloud-python/issues/18629
Reloadhttps://github.com/googleapis/google-cloud-python/issues/18629
Reloadhttps://github.com/googleapis/google-cloud-python/issues/18629
googleapishttps://github.com/googleapis
google-cloud-pythonhttps://github.com/googleapis/google-cloud-python
Codehttps://github.com/googleapis/google-cloud-python
Issues450 (450)https://github.com/googleapis/google-cloud-python/issues
Pull requests200 (200)https://github.com/googleapis/google-cloud-python/pulls
Discussionshttps://github.com/googleapis/google-cloud-python/discussions
Actionshttps://github.com/googleapis/google-cloud-python/actions
Projectshttps://github.com/googleapis/google-cloud-python/projects
Security and qualityhttps://github.com/googleapis/google-cloud-python/security
Insightshttps://github.com/googleapis/google-cloud-python/insights
https://github.com/googleapis/google-cloud-python
New issuehttps://github.com/login?return_to=https://github.com/googleapis/google-cloud-python/issues/18629
New issuehttps://github.com/login?return_to=https://github.com/googleapis/google-cloud-python/issues/18629
storage: zero-copy BidiReadObjectResponse parsing and off-loop CRC32C verification for async multi-range readshttps://github.com/googleapis/google-cloud-python/issues/18629#top
https://github.com/chandra-siri
api: storageIssues related to the Cloud Storage API.https://github.com/googleapis/google-cloud-python/issues?q=state%3Aopen%20label%3A%22api%3A%20storage%22
priority: p2Moderately-important priority. Fix may not be included in next release.https://github.com/googleapis/google-cloud-python/issues?q=state%3Aopen%20label%3A%22priority%3A%20p2%22
type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.https://github.com/googleapis/google-cloud-python/issues?q=state%3Aopen%20label%3A%22type%3A%20feature%20request%22
https://github.com/yuxin00j
yuxin00jhttps://github.com/yuxin00j
on Oct 12, 2026https://github.com/googleapis/google-cloud-python/issues/18629#issue-5807492596
fix(crc32c): accept any contiguous bytes-like object in value() and extend() #18626https://github.com/googleapis/google-cloud-python/pull/18626
feat(storage): opt-in zero-copy BidiReadObjectResponse parsing for async reads #18627https://github.com/googleapis/google-cloud-python/pull/18627
perf(storage): verify CRC32C of large read chunks off the event-loop thread #18628https://github.com/googleapis/google-cloud-python/pull/18628
#18627https://github.com/googleapis/google-cloud-python/pull/18627
#18628https://github.com/googleapis/google-cloud-python/pull/18628
yuxin00jhttps://github.com/yuxin00j
type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.https://github.com/googleapis/google-cloud-python/issues?q=state%3Aopen%20label%3A%22type%3A%20feature%20request%22
triage meI really want to be triaged.https://github.com/googleapis/google-cloud-python/issues?q=state%3Aopen%20label%3A%22triage%20me%22
on Oct 12, 2026https://github.com/googleapis/google-cloud-python/issues/18629#event-33014478179
yuxin00jhttps://github.com/yuxin00j
chandra-sirihttps://github.com/chandra-siri
on Oct 12, 2026https://github.com/googleapis/google-cloud-python/issues/18629#event-33017070773
daniel-sanchehttps://github.com/daniel-sanche
api: storageIssues related to the Cloud Storage API.https://github.com/googleapis/google-cloud-python/issues?q=state%3Aopen%20label%3A%22api%3A%20storage%22
priority: p2Moderately-important priority. Fix may not be included in next release.https://github.com/googleapis/google-cloud-python/issues?q=state%3Aopen%20label%3A%22priority%3A%20p2%22
triage meI really want to be triaged.https://github.com/googleapis/google-cloud-python/issues?q=state%3Aopen%20label%3A%22triage%20me%22
on Oct 23, 2026https://github.com/googleapis/google-cloud-python/issues/18629#event-34100037323
Sign up for freehttps://github.com/signup?return_to=https://github.com/googleapis/google-cloud-python/issues/18629
Sign in to commenthttps://github.com/login?return_to=https://github.com/googleapis/google-cloud-python/issues/18629
chandra-sirihttps://github.com/chandra-siri
api: storageIssues related to the Cloud Storage API.https://github.com/googleapis/google-cloud-python/issues?q=state%3Aopen%20label%3A%22api%3A%20storage%22
priority: p2Moderately-important priority. Fix may not be included in next release.https://github.com/googleapis/google-cloud-python/issues?q=state%3Aopen%20label%3A%22priority%3A%20p2%22
type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.https://github.com/googleapis/google-cloud-python/issues?q=state%3Aopen%20label%3A%22type%3A%20feature%20request%22
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.