René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"vortex.zstd: dictionary-compressed segments not decodable (pure-Java backend lacks dict support)","articleBody":"## Summary\n\n`vortex.zstd` segments compressed **with a trained dictionary** cannot be decoded. `ZstdEncodingDecoder` fails fast when `dictionary_size != 0`:\n\n```\nio.github.dfa1.vortex.core.VortexException:\n  vortex.zstd: dictionary-compressed Zstd segments are not supported (pure-Java decoder)\n```\n\nThis is hit by the upstream compatibility fixture `zstd.vortex` (v0.75.0).\n\n## Why\n\nThe decode backend (`io.airlift:aircompressor-v3`) has **no Zstd dictionary support**:\n- `ZstdFrameDecompressor` exposes no API to preload a dictionary's entropy tables or seed the back-reference window; `reset()` only clears state.\n- Rust uses `zstd::bulk::Decompressor::with_dictionary(dict)` (native libzstd).\n\nThe published `zstd.vortex` fixture genuinely requires it:\n- dict buffer begins with `0xEC30A437` → **trained** dictionary (preset Huffman + 3 FSE tables + content), Dictionary_ID `0x22a28c3d`.\n- frames carry the Dictionary_ID flag referencing that same ID — preset entropy tables **and** window seeding are mandatory.\n\nBuffer layout (per Rust `encodings/zstd/src/array.rs`): `buffer[0]` = dictionary, `buffer[1..]` = frames.\n\n## Options considered\n\n1. **Pure-Java dict decoder from scratch** on `MemorySegment` (~1500–2500 LOC: dict parse, Huffman incl. Treeless, 3 FSE tables incl. Repeat mode, sequence execution with window-prefix matches). Verifiable against one fixture only — high correctness risk.\n2. **Vendor/extend aircompressor's frame decompressor** — rejected: its core uses `sun.misc.Unsafe` (~67 refs), violating the project's no-Unsafe / FFM-only rule.\n3. **zstd-jni (native)** — rejected: violates the no-JNI rule.\n\n## Current state\n\nFail-fast guard retained; covered by `VortexHttpReaderIT#scan_zstdVortex_rejectsDictionaryCompression`, which asserts the clear error. `zstd.vortex` is excluded from the encoding smoke test. Re-enable once a pure-Java dictionary path exists (or aircompressor ships dict support).\n\n## References\n\n- Rust decode: `encodings/zstd/src/array.rs` (`decompress`, line ~888: `Decompressor::with_dictionary`)\n- Buffer layout: same file, `deserialize` line ~204","author":{"url":"https://github.com/dfa1","@type":"Person","name":"dfa1"},"datePublished":"2026-06-20T19:26:36.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":1},"url":"https://github.com/104/vortex-java/issues/104"}