Title: Bug: Kafka ProtobufDeserializer reads Confluent message indexes as plain varints, dropping fields or failing for non-first message types · Issue #8665 · aws-powertools/powertools-lambda-python · GitHub
Open Graph Title: Bug: Kafka ProtobufDeserializer reads Confluent message indexes as plain varints, dropping fields or failing for non-first message types · Issue #8665 · aws-powertools/powertools-lambda-python
X Title: Bug: Kafka ProtobufDeserializer reads Confluent message indexes as plain varints, dropping fields or failing for non-first message types · Issue #8665 · aws-powertools/powertools-lambda-python
Description: Expected Behaviour A Protobuf value produced with Confluent Schema Registry decodes into every field, whatever message index the serializer wrote. Confluent's serializers write the message index array as zigzag encoded varints (array len...
Open Graph Description: Expected Behaviour A Protobuf value produced with Confluent Schema Registry decodes into every field, whatever message index the serializer wrote. Confluent's serializers write the message index ar...
X Description: Expected Behaviour A Protobuf value produced with Confluent Schema Registry decodes into every field, whatever message index the serializer wrote. Confluent's serializers write the message inde...
Opengraph URL: https://github.com/aws-powertools/powertools-lambda-python/issues/8665
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Bug: Kafka ProtobufDeserializer reads Confluent message indexes as plain varints, dropping fields or failing for non-first message types","articleBody":"### Expected Behaviour\n\nA Protobuf value produced with Confluent Schema Registry decodes into every field, whatever message index the serializer wrote. Confluent's serializers write the message index array as zigzag encoded varints (array length, then each index), with `[0]` as a single `0x00` byte; see `_encode_varints` / `_read_index_array` in `confluent_kafka.schema_registry.protobuf`.\n\n### Current Behaviour\n\n`ProtobufDeserializer._remove_message_index` (`kafka/deserializer/protobuf.py`) reads the array with `_DecodeSignedVarint`, so it reads twice the real length. Only index `[0]` works. For any other index:\n\n- it skips payload bytes as if they were indexes, and the record either fails with `DecodeError: Wire format was corrupt`, or\n- it decodes without error but with fields silently missing.\n\nThe second case is already in the repo's fixture: record offset 17 in `tests/functional/kafka_consumer/_protobuf/schemas/complex_schema_with_confuent.py` has the prefix `04 02 00` (zigzag for `[1, 0]`) and decodes without `userId` and `name`. `test_confluent_schema_registry_complex_schema` only asserts `email` and `age`, so it passes.\n\n### Code snippet\n\n```python\nimport base64\n\nfrom google.protobuf.descriptor_pb2 import FieldDescriptorProto\n\nfrom aws_lambda_powertools.utilities.kafka import ConsumerRecords, SchemaConfig, kafka_consumer\n\n\ndef confluent_framed(message, indexes):\n # Message index array as confluent_kafka's ProtobufSerializer writes it: zigzag varints, [0] as a single 0x00\n def zigzag(v):\n v, out = (v \u003c\u003c 1) ^ (v >> 63), bytearray()\n while v & ~0x7F:\n out.append((v & 0x7F) | 0x80)\n v >>= 7\n return bytes(out + bytes([v]))\n\n prefix = b\"\\x00\" if indexes == [0] else zigzag(len(indexes)) + b\"\".join(zigzag(i) for i in indexes)\n return base64.b64encode(prefix + message.SerializeToString()).decode()\n\n\nmessage = FieldDescriptorProto(name=\"order_id\", number=1, json_name=\"orderId\")\n\n\n@kafka_consumer(schema_config=SchemaConfig(value_schema_type=\"PROTOBUF\", value_schema=FieldDescriptorProto))\ndef handler(event: ConsumerRecords, context):\n return next(event.records).value\n\n\nfor indexes in ([0], [1, 0], [2]):\n event = {\n \"eventSource\": \"aws:kafka\",\n \"records\": {\n \"orders-0\": [\n {\n \"topic\": \"orders\", \"partition\": 0, \"offset\": 1, \"timestamp\": 1, \"timestampType\": \"CREATE_TIME\",\n \"key\": None, \"headers\": [],\n \"value\": confluent_framed(message, indexes),\n \"valueSchemaMetadata\": {\"schemaId\": \"1\", \"dataFormat\": \"PROTOBUF\"},\n },\n ],\n },\n }\n try:\n print(f\"message index {indexes}: {handler(event, None)}\")\n except Exception as exc:\n print(f\"message index {indexes}: {type(exc).__name__}: {exc}\")\n```\n\n### Possible Solution\n\nDecode the array as zigzag varints: read the length with `_DecodeVarint` and apply `(n >> 1) ^ -(n & 1)`, then skip that many zigzag varints. Also assert `userId` and `name` in `test_confluent_schema_registry_complex_schema`, and add a parametrised test for indexes `[0]`, `[1]`, `[2]`, `[1, 0]` and a multi-byte index. I have this ready with tests (all Kafka tests pass) and am happy to open the PR.\n\nTrade-off to flag: producers still on Confluent's deprecated format (`use.deprecated.format=True`, plain varints) decode correctly today, and after this change only index `[0]` would keep working for them. Confluent's client warns that format is slated for removal and its default is zigzag, so I'd fix the default. If you want to keep the deprecated format supported, an opt-in flag on `SchemaConfig` could select it. Happy to add that.\n\n### Steps to Reproduce\n\nRun the snippet above. No Kafka or AWS access needed.\n\n### Powertools for AWS Lambda (Python) version\n\nlatest (develop @ 5785377)\n\n### AWS Lambda function runtime\n\n3.13\n\n### Packaging format used\n\nPyPi\n\n### Debugging logs\n\n```python\nOutput of the snippet on develop:\n\nmessage index [0]: {'name': 'order_id', 'number': 1, 'json_name': 'orderId'}\nmessage index [1, 0]: KafkaConsumerDeserializationError: Error trying to deserialize protobuf data - DecodeError: Error parsing message with type 'google.protobuf.FieldDescriptorProto': Wire format was corrupt\nmessage index [2]: KafkaConsumerDeserializationError: Error trying to deserialize protobuf data - DecodeError: Error parsing message with type 'google.protobuf.FieldDescriptorProto': Wire format was corrupt\n```","author":{"url":"https://github.com/alphasafal","@type":"Person","name":"alphasafal"},"datePublished":"2026-10-25T18:53:38Z","url":"https://github.com/aws-powertools/powertools-lambda-python/issues/8665"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:ef4cbcb5-7c63-639a-2aec-513311d46020 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | B7B4:27CEF:132713:1A6D39:6B0081CA |
| html-safe-nonce | 28d5a04694b0e9914c67e31043bc3cb0cb4d4e08693f588295d76cea5f569ba5 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJCN0I0OjI3Q0VGOjEzMjcxMzoxQTZEMzk6NkIwMDgxQ0EiLCJ2aXNpdG9yX2lkIjoiNDE2MzQ5NTIyMjEyODQ0Mzg1MCIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | fbd898a0dc6c4d6bafdb232ea7c7f46bb755cf64a4236ecac8ee2bcacec65154 |
| hovercard-subject-tag | issue:6028043911 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/aws-powertools/powertools-lambda-python/8665/issue_layout |
| twitter:image | https://opengraph.githubassets.com/1f9a0349abb93083191cdb71f1ebe522cc6c45f44fec2f8362cd306665c82cf7/aws-powertools/powertools-lambda-python/issues/8665 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/1f9a0349abb93083191cdb71f1ebe522cc6c45f44fec2f8362cd306665c82cf7/aws-powertools/powertools-lambda-python/issues/8665 |
| og:image:alt | Expected Behaviour A Protobuf value produced with Confluent Schema Registry decodes into every field, whatever message index the serializer wrote. Confluent's serializers write the message index ar... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | alphasafal |
| hostname | github.com |
| expected-hostname | github.com |
| None | 82c186f13e104004ab7bdb779f3195e59dd38bfde531aaa74b76470fc2462b83 |
| turbo-cache-control | no-preview |
| go-import | github.com/aws-powertools/powertools-lambda-python git https://github.com/aws-powertools/powertools-lambda-python.git |
| octolytics-dimension-user_id | 129127638 |
| octolytics-dimension-user_login | aws-powertools |
| octolytics-dimension-repository_id | 221919379 |
| octolytics-dimension-repository_nwo | aws-powertools/powertools-lambda-python |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 221919379 |
| octolytics-dimension-repository_network_root_nwo | aws-powertools/powertools-lambda-python |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | d1bc2aa1939001689f5ce935982c10a6138f745c |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width