René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Feat: New version of entity_key serDe","articleBody":"**Is your feature request related to a problem? Please describe.**\r\nA clear and concise description of what the problem is. Ex. I'm always frustrated when [...]\r\n\r\nThe current entity_key serDe (version 2) is below:\r\n```\r\ndef serialize_entity_key(\r\n    entity_key: EntityKeyProto, entity_key_serialization_version=1\r\n) -\u003e bytes:\r\n    \"\"\"\r\n    Serialize entity key to a bytestring so it can be used as a lookup key in a hash table.\r\n\r\n    We need this encoding to be stable; therefore we cannot just use protobuf serialization\r\n    here since it does not guarantee that two proto messages containing the same data will\r\n    serialize to the same byte string[1].\r\n\r\n    [1] https://developers.google.com/protocol-buffers/docs/encoding\r\n    \"\"\"\r\n    sorted_keys, sorted_values = zip(\r\n        *sorted(zip(entity_key.join_keys, entity_key.entity_values))\r\n    )\r\n\r\n    output: List[bytes] = []\r\n    for k in sorted_keys:\r\n        output.append(struct.pack(\"\u003cI\", ValueType.STRING))\r\n        output.append(k.encode(\"utf8\"))\r\n    for v in sorted_values:\r\n        val_bytes, value_type = _serialize_val(\r\n            v.WhichOneof(\"val\"),\r\n            v,\r\n            entity_key_serialization_version=entity_key_serialization_version,\r\n        )\r\n\r\n        output.append(struct.pack(\"\u003cI\", value_type))\r\n\r\n        output.append(struct.pack(\"\u003cI\", len(val_bytes)))\r\n        output.append(val_bytes)\r\n\r\n    return b\"\".join(output)\r\n```\r\n\r\ne.g, for `sorted_keys = {tuple: 1} item_id` and `sorted_values = {tuple: 1} int64_val: 1\\n` will give output:\r\n`[b'\\x02\\x00\\x00\\x00', b'item_id', b'\\x04\\x00\\x00\\x00', b'\\x08\\x00\\x00\\x00', b'\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x00']`\r\n\r\nThis makes deserialization not doable. In order to deserialize we can append the \"length\" of value to the `join_key`, such as for the same test key and value we can get the output:\r\n`[b'\\x02\\x00\\x00\\x00', b'\\x07\\x00\\x00\\x00', b'item_id', b'\\x04\\x00\\x00\\x00', b'\\x08\\x00\\x00\\x00', b'\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x00']`\r\n\r\nThen we can deserialize the bytes to proto.\r\n\r\n\r\n\r\n**Describe the solution you'd like**\r\nA clear and concise description of what you want to happen.\r\n\r\n\r\n\r\n**Describe alternatives you've considered**\r\nA clear and concise description of any alternative solutions or features you've considered.\r\n\r\n**Additional context**\r\nAdd any other context or screenshots about the feature request here.\r\n","author":{"url":"https://github.com/HaoXuAI","@type":"Person","name":"HaoXuAI"},"datePublished":"2024-06-16T19:00:04.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/4283/feast/issues/4283"}