René's URL Explorer Experiment


Title: Bug: Kafka ProtobufDeserializer reads Confluent message indexes as plain varints, dropping fields or failing for non-first message types · Issue #8665 · aws-powertools/powertools-lambda-python · GitHub

Open Graph Title: Bug: Kafka ProtobufDeserializer reads Confluent message indexes as plain varints, dropping fields or failing for non-first message types · Issue #8665 · aws-powertools/powertools-lambda-python

X Title: Bug: Kafka ProtobufDeserializer reads Confluent message indexes as plain varints, dropping fields or failing for non-first message types · Issue #8665 · aws-powertools/powertools-lambda-python

Description: Expected Behaviour A Protobuf value produced with Confluent Schema Registry decodes into every field, whatever message index the serializer wrote. Confluent's serializers write the message index array as zigzag encoded varints (array len...

Open Graph Description: Expected Behaviour A Protobuf value produced with Confluent Schema Registry decodes into every field, whatever message index the serializer wrote. Confluent's serializers write the message index ar...

X Description: Expected Behaviour A Protobuf value produced with Confluent Schema Registry decodes into every field, whatever message index the serializer wrote. Confluent's serializers write the message inde...

Opengraph URL: https://github.com/aws-powertools/powertools-lambda-python/issues/8665

X: @github

direct link

Domain: github.com


Hey, it has json ld scripts:
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Bug: Kafka ProtobufDeserializer reads Confluent message indexes as plain varints, dropping fields or failing for non-first message types","articleBody":"### Expected Behaviour\n\nA Protobuf value produced with Confluent Schema Registry decodes into every field, whatever message index the serializer wrote. Confluent's serializers write the message index array as zigzag encoded varints (array length, then each index), with `[0]` as a single `0x00` byte; see `_encode_varints` / `_read_index_array` in `confluent_kafka.schema_registry.protobuf`.\n\n### Current Behaviour\n\n`ProtobufDeserializer._remove_message_index` (`kafka/deserializer/protobuf.py`) reads the array with `_DecodeSignedVarint`, so it reads twice the real length. Only index `[0]` works. For any other index:\n\n- it skips payload bytes as if they were indexes, and the record either fails with `DecodeError: Wire format was corrupt`, or\n- it decodes without error but with fields silently missing.\n\nThe second case is already in the repo's fixture: record offset 17 in `tests/functional/kafka_consumer/_protobuf/schemas/complex_schema_with_confuent.py` has the prefix `04 02 00` (zigzag for `[1, 0]`) and decodes without `userId` and `name`. `test_confluent_schema_registry_complex_schema` only asserts `email` and `age`, so it passes.\n\n### Code snippet\n\n```python\nimport base64\n\nfrom google.protobuf.descriptor_pb2 import FieldDescriptorProto\n\nfrom aws_lambda_powertools.utilities.kafka import ConsumerRecords, SchemaConfig, kafka_consumer\n\n\ndef confluent_framed(message, indexes):\n    # Message index array as confluent_kafka's ProtobufSerializer writes it: zigzag varints, [0] as a single 0x00\n    def zigzag(v):\n        v, out = (v \u003c\u003c 1) ^ (v >> 63), bytearray()\n        while v & ~0x7F:\n            out.append((v & 0x7F) | 0x80)\n            v >>= 7\n        return bytes(out + bytes([v]))\n\n    prefix = b\"\\x00\" if indexes == [0] else zigzag(len(indexes)) + b\"\".join(zigzag(i) for i in indexes)\n    return base64.b64encode(prefix + message.SerializeToString()).decode()\n\n\nmessage = FieldDescriptorProto(name=\"order_id\", number=1, json_name=\"orderId\")\n\n\n@kafka_consumer(schema_config=SchemaConfig(value_schema_type=\"PROTOBUF\", value_schema=FieldDescriptorProto))\ndef handler(event: ConsumerRecords, context):\n    return next(event.records).value\n\n\nfor indexes in ([0], [1, 0], [2]):\n    event = {\n        \"eventSource\": \"aws:kafka\",\n        \"records\": {\n            \"orders-0\": [\n                {\n                    \"topic\": \"orders\", \"partition\": 0, \"offset\": 1, \"timestamp\": 1, \"timestampType\": \"CREATE_TIME\",\n                    \"key\": None, \"headers\": [],\n                    \"value\": confluent_framed(message, indexes),\n                    \"valueSchemaMetadata\": {\"schemaId\": \"1\", \"dataFormat\": \"PROTOBUF\"},\n                },\n            ],\n        },\n    }\n    try:\n        print(f\"message index {indexes}: {handler(event, None)}\")\n    except Exception as exc:\n        print(f\"message index {indexes}: {type(exc).__name__}: {exc}\")\n```\n\n### Possible Solution\n\nDecode the array as zigzag varints: read the length with `_DecodeVarint` and apply `(n >> 1) ^ -(n & 1)`, then skip that many zigzag varints. Also assert `userId` and `name` in `test_confluent_schema_registry_complex_schema`, and add a parametrised test for indexes `[0]`, `[1]`, `[2]`, `[1, 0]` and a multi-byte index. I have this ready with tests (all Kafka tests pass) and am happy to open the PR.\n\nTrade-off to flag: producers still on Confluent's deprecated format (`use.deprecated.format=True`, plain varints) decode correctly today, and after this change only index `[0]` would keep working for them. Confluent's client warns that format is slated for removal and its default is zigzag, so I'd fix the default. If you want to keep the deprecated format supported, an opt-in flag on `SchemaConfig` could select it. Happy to add that.\n\n### Steps to Reproduce\n\nRun the snippet above. No Kafka or AWS access needed.\n\n### Powertools for AWS Lambda (Python) version\n\nlatest (develop @ 5785377)\n\n### AWS Lambda function runtime\n\n3.13\n\n### Packaging format used\n\nPyPi\n\n### Debugging logs\n\n```python\nOutput of the snippet on develop:\n\nmessage index [0]: {'name': 'order_id', 'number': 1, 'json_name': 'orderId'}\nmessage index [1, 0]: KafkaConsumerDeserializationError: Error trying to deserialize protobuf data - DecodeError: Error parsing message with type 'google.protobuf.FieldDescriptorProto': Wire format was corrupt\nmessage index [2]: KafkaConsumerDeserializationError: Error trying to deserialize protobuf data - DecodeError: Error parsing message with type 'google.protobuf.FieldDescriptorProto': Wire format was corrupt\n```","author":{"url":"https://github.com/alphasafal","@type":"Person","name":"alphasafal"},"datePublished":"2026-10-25T18:53:38Z","url":"https://github.com/aws-powertools/powertools-lambda-python/issues/8665"}

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format)
route-controllervoltron_issues_fragments
route-actionissue_layout
fetch-noncev2:ef4cbcb5-7c63-639a-2aec-513311d46020
current-catalog-service-hash81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114
request-idB7B4:27CEF:132713:1A6D39:6B0081CA
html-safe-nonce28d5a04694b0e9914c67e31043bc3cb0cb4d4e08693f588295d76cea5f569ba5
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJCN0I0OjI3Q0VGOjEzMjcxMzoxQTZEMzk6NkIwMDgxQ0EiLCJ2aXNpdG9yX2lkIjoiNDE2MzQ5NTIyMjEyODQ0Mzg1MCIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9
visitor-hmacfbd898a0dc6c4d6bafdb232ea7c7f46bb755cf64a4236ecac8ee2bcacec65154
hovercard-subject-tagissue:6028043911
github-keyboard-shortcutsrepository,issues,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///voltron/issues_fragments/issue_layout
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/aws-powertools/powertools-lambda-python/8665/issue_layout
twitter:imagehttps://opengraph.githubassets.com/1f9a0349abb93083191cdb71f1ebe522cc6c45f44fec2f8362cd306665c82cf7/aws-powertools/powertools-lambda-python/issues/8665
twitter:cardsummary_large_image
og:imagehttps://opengraph.githubassets.com/1f9a0349abb93083191cdb71f1ebe522cc6c45f44fec2f8362cd306665c82cf7/aws-powertools/powertools-lambda-python/issues/8665
og:image:altExpected Behaviour A Protobuf value produced with Confluent Schema Registry decodes into every field, whatever message index the serializer wrote. Confluent's serializers write the message index ar...
og:image:width1200
og:image:height600
og:site_nameGitHub
og:typeobject
og:author:usernamealphasafal
hostnamegithub.com
expected-hostnamegithub.com
None82c186f13e104004ab7bdb779f3195e59dd38bfde531aaa74b76470fc2462b83
turbo-cache-controlno-preview
go-importgithub.com/aws-powertools/powertools-lambda-python git https://github.com/aws-powertools/powertools-lambda-python.git
octolytics-dimension-user_id129127638
octolytics-dimension-user_loginaws-powertools
octolytics-dimension-repository_id221919379
octolytics-dimension-repository_nwoaws-powertools/powertools-lambda-python
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id221919379
octolytics-dimension-repository_network_root_nwoaws-powertools/powertools-lambda-python
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
released1bc2aa1939001689f5ce935982c10a6138f745c
ui-targetfull
theme-color#1e2327
color-schemelight dark
react-app-nameissues

Links:

Skip to contenthttps://github.com/aws-powertools/powertools-lambda-python/issues/8665#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Faws-powertools%2Fpowertools-lambda-python%2Fissues%2F8665
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Faws-powertools%2Fpowertools-lambda-python%2Fissues%2F8665
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fvoltron%2Fissues_fragments%2Fissue_layout&source=header-repo&source_repo=aws-powertools%2Fpowertools-lambda-python
Reloadhttps://github.com/aws-powertools/powertools-lambda-python/issues/8665
Reloadhttps://github.com/aws-powertools/powertools-lambda-python/issues/8665
Reloadhttps://github.com/aws-powertools/powertools-lambda-python/issues/8665
aws-powertoolshttps://github.com/aws-powertools
powertools-lambda-pythonhttps://github.com/aws-powertools/powertools-lambda-python
Codehttps://github.com/aws-powertools/powertools-lambda-python
Issues40 (40)https://github.com/aws-powertools/powertools-lambda-python/issues
Pull requests20 (20)https://github.com/aws-powertools/powertools-lambda-python/pulls
Discussionshttps://github.com/aws-powertools/powertools-lambda-python/discussions
Actionshttps://github.com/aws-powertools/powertools-lambda-python/actions
Projectshttps://github.com/aws-powertools/powertools-lambda-python/projects
Security and quality1 (1)https://github.com/aws-powertools/powertools-lambda-python/security
Insightshttps://github.com/aws-powertools/powertools-lambda-python/insights
https://github.com/aws-powertools/powertools-lambda-python
New issuehttps://github.com/login?return_to=https://github.com/aws-powertools/powertools-lambda-python/issues/8665
New issuehttps://github.com/login?return_to=https://github.com/aws-powertools/powertools-lambda-python/issues/8665
#8670https://github.com/aws-powertools/powertools-lambda-python/pull/8670
Bug: Kafka ProtobufDeserializer reads Confluent message indexes as plain varints, dropping fields or failing for non-first message typeshttps://github.com/aws-powertools/powertools-lambda-python/issues/8665#top
#8670https://github.com/aws-powertools/powertools-lambda-python/pull/8670
https://github.com/alphasafal
bugSomething isn't workinghttps://github.com/aws-powertools/powertools-lambda-python/issues?q=state%3Aopen%20label%3A%22bug%22
event_sourcesEvent Source Data Class utilityhttps://github.com/aws-powertools/powertools-lambda-python/issues?q=state%3Aopen%20label%3A%22event_sources%22
3.40.0https://github.com/aws-powertools/powertools-lambda-python/milestone/25
https://github.com/alphasafal
alphasafalhttps://github.com/alphasafal
on Oct 25, 2026https://github.com/aws-powertools/powertools-lambda-python/issues/8665#issue-6028043911
5785377https://github.com/aws-powertools/powertools-lambda-python/commit/57853775090ffe9a9995022372e8651ea0cdcb18
alphasafalhttps://github.com/alphasafal
bugSomething isn't workinghttps://github.com/aws-powertools/powertools-lambda-python/issues?q=state%3Aopen%20label%3A%22bug%22
triagePending triage from maintainershttps://github.com/aws-powertools/powertools-lambda-python/issues?q=state%3Aopen%20label%3A%22triage%22
on Oct 25, 2026https://github.com/aws-powertools/powertools-lambda-python/issues/8665#event-34198083211
github-project-automationhttps://github.com/apps/github-project-automation
Powertools for AWS Lambda (Python)https://github.com/orgs/aws-powertools/projects/3
on Oct 25, 2026https://github.com/aws-powertools/powertools-lambda-python/issues/8665#event-34198084223
https://github.com/apps/boring-cyborg
boring-cyborghttps://github.com/apps/boring-cyborg
on Oct 25, 2026https://github.com/aws-powertools/powertools-lambda-python/issues/8665#issuecomment-6401748845
boring-cyborghttps://github.com/kaxil/boring-cyborg
Invite linkhttps://discord.gg/B8zZKbbyET
github-project-automationhttps://github.com/apps/github-project-automation
Powertools for AWS Lambda (Python)https://github.com/orgs/aws-powertools/projects/3
on Oct 25, 2026https://github.com/aws-powertools/powertools-lambda-python/issues/8665
https://github.com/apps/powertools-for-aws-oss-automation
powertools-for-aws-oss-automationhttps://github.com/apps/powertools-for-aws-oss-automation
on Oct 25, 2026https://github.com/aws-powertools/powertools-lambda-python/issues/8665#issuecomment-6401749409
Powertools for AWS - OSS Automationhttps://github.com/aws-powertools/powertools-lambda-oss-automation
Join the conversationhttps://github.com/aws-powertools/powertools-lambda-python/discussions
https://github.com/leandrodamascena
leandrodamascenahttps://github.com/leandrodamascena
on Oct 25, 2026https://github.com/aws-powertools/powertools-lambda-python/issues/8665#issuecomment-6401915442
@alphasafalhttps://github.com/alphasafal
leandrodamascenahttps://github.com/leandrodamascena
triagePending triage from maintainershttps://github.com/aws-powertools/powertools-lambda-python/issues?q=state%3Aopen%20label%3A%22triage%22
on Oct 25, 2026https://github.com/aws-powertools/powertools-lambda-python/issues/8665#event-34198767100
leandrodamascenahttps://github.com/leandrodamascena
alphasafalhttps://github.com/alphasafal
on Oct 25, 2026https://github.com/aws-powertools/powertools-lambda-python/issues/8665#event-34198767366
leandrodamascenahttps://github.com/leandrodamascena
3.40.0https://github.com/aws-powertools/powertools-lambda-python/milestone/25
on Oct 25, 2026https://github.com/aws-powertools/powertools-lambda-python/issues/8665#event-34198768198
leandrodamascenahttps://github.com/leandrodamascena
event_sourcesEvent Source Data Class utilityhttps://github.com/aws-powertools/powertools-lambda-python/issues?q=state%3Aopen%20label%3A%22event_sources%22
on Oct 25, 2026https://github.com/aws-powertools/powertools-lambda-python/issues/8665#event-34198791561
https://github.com/alphasafal
alphasafalhttps://github.com/alphasafal
on Oct 25, 2026https://github.com/aws-powertools/powertools-lambda-python/issues/8665#issuecomment-6402254192
@leandrodamascenahttps://github.com/leandrodamascena
#8670https://github.com/aws-powertools/powertools-lambda-python/pull/8670
leandrodamascenahttps://github.com/leandrodamascena
completedhttps://github.com/aws-powertools/powertools-lambda-python/issues?q=is%3Aissue%20state%3Aclosed%20archived%3Afalse%20reason%3Acompleted
db2f8f0https://github.com/aws-powertools/powertools-lambda-python/commit/db2f8f06b6acaa2727dfd8a9b95a598256577ea5
on Nov 4, 2026https://github.com/aws-powertools/powertools-lambda-python/issues/8665#event-35153808440
github-project-automationhttps://github.com/apps/github-project-automation
Powertools for AWS Lambda (Python)https://github.com/orgs/aws-powertools/projects/3
on Nov 4, 2026https://github.com/aws-powertools/powertools-lambda-python/issues/8665
https://github.com/apps/powertools-for-aws-oss-automation
powertools-for-aws-oss-automationhttps://github.com/apps/powertools-for-aws-oss-automation
on Nov 4, 2026https://github.com/aws-powertools/powertools-lambda-python/issues/8665#issuecomment-6636875432
Powertools for AWS - OSS Automationhttps://github.com/aws-powertools/powertools-lambda-oss-automation
powertools-for-aws-oss-automationhttps://github.com/apps/powertools-for-aws-oss-automation
pending-releaseFix or implementation already in dev waiting to be releasedhttps://github.com/aws-powertools/powertools-lambda-python/issues?q=state%3Aopen%20label%3A%22pending-release%22
on Nov 4, 2026https://github.com/aws-powertools/powertools-lambda-python/issues/8665#event-35153834490
leandrodamascenahttps://github.com/leandrodamascena
on Nov 4, 2026https://github.com/aws-powertools/powertools-lambda-python/issues/8665#event-35153867125
fix(kafka): decode Confluent Protobuf message indexes as zigzag varin…https://github.com/ASHMEET555/powertools-lambda-python/commit/db2f8f06b6acaa2727dfd8a9b95a598256577ea5
https://docs.github.com/articles/closing-issues-via-commit-messages
db2f8f0https://github.com/ASHMEET555/powertools-lambda-python/commit/db2f8f06b6acaa2727dfd8a9b95a598256577ea5
https://github.com/apps/github-actions
github-actionshttps://github.com/apps/github-actions
on Nov 10, 2026https://github.com/aws-powertools/powertools-lambda-python/issues/8665#issuecomment-6766538677
GitHub Actionshttps://help.github.com/en/actions
3.40.0https://github.com/aws-powertools/powertools-lambda-python/releases/tag/v3.40.0
github-actionshttps://github.com/apps/github-actions
pending-releaseFix or implementation already in dev waiting to be releasedhttps://github.com/aws-powertools/powertools-lambda-python/issues?q=state%3Aopen%20label%3A%22pending-release%22
on Nov 10, 2026https://github.com/aws-powertools/powertools-lambda-python/issues/8665#event-35676366434
Sign up for freehttps://github.com/signup?return_to=https://github.com/aws-powertools/powertools-lambda-python/issues/8665
Sign in to commenthttps://github.com/login?return_to=https://github.com/aws-powertools/powertools-lambda-python/issues/8665
alphasafalhttps://github.com/alphasafal
bugSomething isn't workinghttps://github.com/aws-powertools/powertools-lambda-python/issues?q=state%3Aopen%20label%3A%22bug%22
event_sourcesEvent Source Data Class utilityhttps://github.com/aws-powertools/powertools-lambda-python/issues?q=state%3Aopen%20label%3A%22event_sources%22
Powertools for AWS Lambda (Python)https://github.com/orgs/aws-powertools/projects/3
3.40.0https://github.com/aws-powertools/powertools-lambda-python/milestone/25
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.