René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Unclear type mapping with both packb/unpackb, especially with mixed type sequences","articleBody":"## Problem:\r\n\r\nThe current/packing unpacking situation is confusing and complex when it comes to dealing with the different binary and string types that will be packed to, or unpacked from, the MessagePack \"_str format family_\" and \"_bin format family_\" data types. It is difficult to determine the correct combination for a satisfactory type mapping in all situations.\r\n\r\nIn addition, the current `msgpack-python` (now `msgpack`) implementations do not have a solution (in either direction) for dealing with data containers that contain both string and binary data types.\r\n\r\nFor example (on the unpacking/deserialization side), the following byte sequence defines a MessagePack array that contains two elements: a unicode snowman character in utf-8, and an arbitrary byte sequence of [0x00, 0x01, 0x02]:\r\n\r\n```\r\ndata = b'\\x92\\xa3\\xe2\\x98\\x83\\xc4\\x03\\x00\\x01\\x02'\r\n```\r\nWhat possible combination of `msgpack.unpackb` kwargs can _properly_ unpack this to a two element list containing a python string and a suitable binary type (like `bytearray` in Python 2, and `bytes` in Python 3)? Conversely, how would you generate such a MessagePack structure from python (aside from direct generation as above)?\r\n\r\n## Proposal:\r\n\r\nRather than having a collection of effectivey global switches that can be sent to `packb`/`unpackb` (for example: `raw_as_bytes` and `use_bin_type`), it would be better if there were a method for defining an explicit typemap that would be used at a per-element level, and which defined the type mappings to use for both `packb` (from python to the MessagePack protocol) and `unpackb` (from MessagePack to python).\r\n\r\nFor example, it would be great if `msgpack.unpackb(data_bytes, typemap='ideal'`) would get the \"ideal\" behaviour I outline in the tables further below. When using the `typemap` switch, packing/unpacking could then work in a per-element way, rather than having issues with mixed-type sequences like the global switches currently do. Possible `typemap` values could be similar to in the columns defined further below: `('ideal', '0.4', '0.5', 'default')`, or somesuch, where `default` would be the default value, and currently equate to `typemap='0.5'`. It would also be illegal (raising `ValueError`) to specify kwargs like `raw_as_bytes` together with a `typemap` specification). I think this proposal will resolve any potential compatibility issues.\r\n\r\nThis `typemap` kwarg behaviour should be bidirectional. Specifically, there should also be a similar possibility for `msgpack.packb(data, typemap=`ideal`).\r\n\r\nIn addition (with the exception of python 2 `str` and `bytearray` ambiguity) it should _always_ be the case that `unpackb(packb(v)) == v`. This is currently _not_ true with available `msgpack` python versions.\r\n\r\n## Explicit Type Mapping Tables\r\n\r\nThe type mapping situation for msgpack bin and str (current, and \"ideal\") are covered in the tables below, covering the current situation for different `msgpack` versions, as well as my proposed `ideal` mapping.\r\n\r\nNote that, in the tables below:\r\n* PackType == `mp_str` refers to the \"str format family\", with leading (`101XXXXX`, `0xd9`, `0xda`, `0xdb`)\r\n* PackType == `mp_bin` refers to the \"bin format family\", with lead bits in (`0xc4`, `0xc5`, `0xc6`)\r\n* The \"PackType (_ideal_)\" column indicates what I personally think the accurate pack/unpack targets should be for each type.\r\n* `packb`/`unpackb` results for versions \"\u003c 0.5.x\" and \"\u003e= 0.5.x\" are what you get with default values for all global kwarg switches like `raw_as_bytes`\r\n* I have intentionally not referenced the types where the mapping is (in my opinion) extremely clear, for example:\r\n  * msgpack `nil format` ⇔ `None`\r\n  * msgpack `bool format` ⇔ `bool`\r\n  * msgpack `int format family` ⇔ `int`\r\n  * msgpack `float format family` ⇔ `float`\r\n  * msgpack `array format family` ⇔ `list`\r\n    * aside: I'd prefer an immutable tuple here, although map/dict targets have to be mutable so consistency is an issue\r\n  * msgpack `map format family` ⇔ `dict`\r\n\r\n### Python 2 packing/serialization behaviour\r\n| Python2 type | PackType (_ideal_) | PackType (\u003c0.5.x) | PackType (\u003e=0.5.x) | Comment |\r\n| --- | --- | --- | --- | --- |\r\n| `str`           | `mp_bin` | `mp_str` | `mp_str` | `str` is really bytes in python 2 |\r\n| `unicode`   | `mp_str` | `mp_str` | `mp_str` | Should always encode to utf8 |\r\n| `bytearray` | `mp_bin` | ERROR | `mp_str` | If this doesn't go to `mp_bin`, what does? See notes below on the ERROR case (which was good) |\r\n\r\n\\* In the case of the ERROR for `msgpack` \u003c 0.5.x, this was actually extremely useful, since it resulted an the `default` callback being invoked where you could specifically manage `bytearray` (since msgpack-python itself did not). Now that 0.5.x silently encodes `bytearray` to `mp_str`, this is no longer possible. This is actually the problem that triggered me to raise this issue (after an update to 0.5.x broke a build).\r\n\r\n### Python 2 unpacking/deserialization behaviour\r\n| Msgpack type | Python2 type (_ideal_) | Python2 type (\u003c0.5.x) | Python2 type (\u003e=0.5.x) | Comment |\r\n| --- | --- | --- | --- | --- |\r\n| `mp_str` | `unicode` | `str` | `str` | Should always decode assuming utf8 |\r\n| `mp_bin` | `str` | `str` | `str` |  `bytearray` would actually be a more literal/ideal unpack target, but is unfamiliar to most (and is mutable) |\r\n\r\n### Python 3 packing/serialization behaviour\r\n| Python3 type | PackType (_ideal_) | PackType (\u003c0.5.x) | PackType (\u003e=0.5.x) | Comment |\r\n| --- | --- | --- | --- | --- |\r\n| `str`        | `mp_str` | `mp_str` | `mp_str` | Always encode with utf8 |\r\n| `bytes`   | `mp_bin` | `mp_str` | `mp_str` |  |\r\n\r\n### Python 3 unpacking/deserialization behaviour\r\n| Python3 type | PackType (ideal) | PackType (\u003c0.5.x) | PackType (\u003e=0.5.x) | Comment |\r\n| --- | --- | --- | --- | --- |\r\n| `mp_str` | `str` | `bytes` | `bytes` | The default conversion to `bytes` is particularly confusing |\r\n| `mp_bin` | `bytes` | `bytes` | `bytes` |  |\r\n\r\n## References\r\nThere are some other issues related to this that are worth referencing here:\r\n* #191 -- regarding backwards compatibility issues moving towards 1.0\r\n* #99 -- an old issue about properly decoding string types\r\n* #224 -- an issue specifically about serializing `bytearray`\r\n* [msgpack #121](https://github.com/msgpack/msgpack/issues/121) -- an old/length issue (now closed) about differentiating between raw binary data and strings\r\n\r\n\r\nI decide to make a new issue, since this is a general proposal about A) explicitly clarifying type mapping in both directions between msgpack and python, and B) it explicitly covers both bin and str msgpack formats (in msgpack terminology) as well as both python str/bytes cases.\r\n \r\n","author":{"url":"https://github.com/rwarren","@type":"Person","name":"rwarren"},"datePublished":"2018-02-06T21:53:10.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":3},"url":"https://github.com/281/msgpack-python/issues/281"}