René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"How to handle common, non-primitive data types","articleBody":"This is likely going to be an open question for a while, but there are my current thoughts. All input is welcome.\r\n\r\nI feel like, by and large, data collected from lab instruments can sensibly be converted to primitive data types. The most common types I have in mind are Numpy arrays, and Pandas data frames. Both of these can be represented easily with primitive data types.\r\n\r\nThere are however cases where data will be collected that cannot be converted to a primitive type.\r\n\r\nIn the new `cbor` branch, I've added a section to the JSON encoder that will base64 encode `bytes` Python objects. I've correspondingly included a Marshmallow `Bytes` field to handle validating binary data in this format. It populates the documentation with information about the string values being a base64 encoded block of binary data. Everything is fine on that front.\r\n\r\nHowever, as @rwb27 has mentioned in the past, sometimes the binary data collected will be big enough that the b64 encoding overhead could become problematic. To handle these cases, I've included support for clients to accept `application/cbor` responses instead of `application/json`.\r\n\r\nCBOR has built in support for binary encoded data, so if a client requests a CBOR response, no encoding overhead is introduced. The data gets passed directly to the CBOR response, otherwise identical to the JSON response, but with the binary section unencoded.\r\n\r\nThis solution isn't perfect though. The Thing Description is required to be JSON. This is fine in most cases as it accurately describes the base64 encoded binary blobs. However, it means that the CBOR response will deviate from the Thing Description, receiving a `bytes` type value where the Description says a `string` will be returned.\r\n\r\nI currently feel however that the cases where large, non-primitive data files are being collected with such high frequency that CBOR encoding is required are infrequent enough that, given proper documentation, this solution could still be fine.\r\n\r\nAgain, thoughts are welcome.\r\n\r\n**Note:** The CBOR branch is useful even aside from this. It's a much more compact data format that JSON, so for many cases it may be beneficial to actually communicate over BSON even without needing to transfer `bytes` objects. It was easy to add support, and doesn't affect the JSON functionality at all.","author":{"url":"https://github.com/jtc42","@type":"Person","name":"jtc42"},"datePublished":"2020-05-02T18:51:33.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":2},"url":"https://github.com/70/python-labthings/issues/70"}