René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"More flexible extension support","articleBody":"**TLDR:**\r\n\r\nI'd like to suggest a tweak to the extension API. As well as the current process where an extension returns a blob which the encoder copies, I propose allowing an extension to use the \"private\" API of the encoder to directly perform writes.\r\n\r\nI have prototyped this extension mechanism in a fork. The changes are minimal. See comparison [here](https://github.com/msgpack/msgpack-javascript/compare/main...crabmusket:typed-arrays-plugin#diff-2b07fa9965c82a4be8829b5cdcaff0dbd4e71fe8e601e353efcd72eee2e0c243).\r\n\r\nIf you agree that this idea is desirable I'm happy to clean up my branch, add tests and make a PR.\r\n\r\n---\r\n\r\nI came across this need when I was trying to design an extension to handle TypedArrays natively, but also to _align_ them to the correct byte indexes so that they can be used efficiently after decoding. You can see my extension [here](https://github.com/crabmusket/msgpack-javascript/tree/typed-arrays-plugin/example/typed-arrays) - the readme explains the alignment situation. (And my original comment [here](https://github.com/msgpack/msgpack-javascript/issues/76#issuecomment-922411931).)\r\n\r\n```js\r\n// Encoding and decoding a TypedArray with no boilerplate:\r\nconst floatArray = new Float32Array([1, 2, 3, 4, 5]);\r\nconst encoded = encode({ floatArray }, { extensionCodec });\r\nassert.deepStrictEqual(decode(encoded, { extensionCodec }), { floatArray });\r\n```\r\n\r\nUsing the _current_ extension API to do this, I would need to:\r\n\r\n1. Do an extra copy, from the TypedArray to a Uint8Array containing my extra wrapper and alignment bytes\r\n2. Predict which of MsgPack's extension headers the base encoder will use when writing my extension data, and adjust alignment accordingly\r\n3. Use a reference to the encoder _anyway_, to get the current value of `pos`. I'd have to send this circular reference into the extension codec's context object\r\n\r\nProblem 1 isn't such a big deal; there's a lot of copying during encoding anyway, though I minimise it by choosing appropriate initial buffer sizes. And if copying can be avoided, why not?\r\n\r\nProblem 2 is the main issue I am interested in; I was frankly too lazy to do the maths required to predict the ext header size and then modify the alignment accordingly. My fork allows extensions to directly call the encoder's `write` methods, which means that I was able to always choose an `ext 32` header. This makes it easy to predict the alignment requirements. A small amount of bytes are wasted, but my use-case is for large data arrays where a handful of bytes will make almost no difference to the final payload size.\r\n\r\n---\r\n\r\nIn order to implement this extension efficiently, I added a [write method](https://github.com/crabmusket/msgpack-javascript/blob/7f84239463b162358ec11a39ced36fdb878b35a4/src/ExtData.ts#L9-L11) to ExtData. My plugin returns a [subclass](https://github.com/crabmusket/msgpack-javascript/blob/7f84239463b162358ec11a39ced36fdb878b35a4/example/typed-arrays/plugin.ts#L78) of ExtData which has its own implementation of `write`.\r\n\r\nThis was just a rough cut to see if the concept worked. If I were designing this properly, I'd replace the use of the `ExtData` class with an interface to avoid mandatory inheritance hierarchies. You can see in my `TypedArrayExtData` I'm already being silly by calling the parent constructor with a `new Uint8Array()` which is completely unused.\r\n\r\n---\r\n\r\nPros of this idea:\r\n\r\n- Allows fewer copies during encoding\r\n- Allows me to implement alignment much more easily by controlling which `ext` header is written\r\n\r\nCons of this idea:\r\n\r\n- Introduces an additional method call during the encoding process (`extData.write`, which currently calls straight back to `encoder.encodeExtension`. I think this is an example of \"double dispatch\" in OOP parlance?)\r\n- Extensions using this approach rely on the private API of the Encoder\r\n- I haven't thought about recursive encoding 🤔 \r\n\r\nFor my use case, I didn't mind using the private API. This library seems stable enough. But for my use-case, I would _also_ have to depend on internal implementation details when guessing alignment: if the library ever used different ext headers for the same data size, then my alignment guesses would be wrong. I see no reason why that logic would ever change, but it's still an implementation detail.\r\n\r\nHowever, you might not want to allow clients to depend on this behaviour. It could be worked around, e.g. by providing an adapter with stable public methods which call the encoder's `write` methods.","author":{"url":"https://github.com/crabmusket","@type":"Person","name":"crabmusket"},"datePublished":"2021-09-21T02:11:47.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/192/msgpack-javascript/issues/192"}