René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Remove pyarrow as required dependency, relying on Arrow PyCapsule Interface","articleBody":"**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**\n\nPyArrow is a massive dependency. Unpacked, it tends to be \u003e100MB in size, and, until the latest versions (I think?) also required numpy as its own non-optional dependency.\n\nIt's also, in effect the only current dependency\nhttps://github.com/apache/datafusion-python/blob/f0bbad7543717c5f08ba2acb92d42c9d30fd2355/pyproject.toml#L46\n\nIt would be great if we could remove it, and that would greatly lessen the minimal environment size for datafusion python.\n\n[Many other Python Arrow libraries](https://github.com/apache/arrow/issues/39195#issuecomment-2245718008) implement the PyCapsule Interface, so the user can use nanoarrow, arro3, Polars, DuckDB, etc, or pyarrow. Whatever is best for them.\n\n**Describe the solution you'd like**\n\nThe Arrow PyCapsule Interface is a lightweight, decentralized protocol for sharing Arrow data between Python libraries. We already implement the PyCapsule Interface, so it's just a matter of removing places where we hard-code use of pyarrow.\n\n**Describe alternatives you've considered**\n\nKeep pyarrow dependency.\n\n**Additional context**\n","author":{"url":"https://github.com/kylebarron","@type":"Person","name":"kylebarron"},"datePublished":"2025-09-03T17:16:12.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":14},"url":"https://github.com/1227/datafusion-python/issues/1227"}