René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Add support for scalar values with extension types","articleBody":"**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**\n\nSuppose I have a pyarrow scalar value that contains an extension type. If I try turning that into a literal expression in datafusion, we should get the associated metadata transparently to the user.\n\nConsider this minimal example:\n\n```python\nimport pyarrow as pa\nimport uuid\nfrom datafusion import lit\n\nvalue = pa.scalar(uuid.uuid4().bytes, pa.uuid())\n\nprint(lit(value))\n```\n\nThis currently fails with `ArrowTypeError: Expected bytes, got a 'UUID' object`. That can be overcome with the simple patch\n\n```patch\n--- a/src/pyarrow_util.rs\n+++ b/src/pyarrow_util.rs\n@@ -30,7 +30,11 @@ impl FromPyArrow for PyScalarValue {\n     fn from_pyarrow_bound(value: \u0026Bound\u003c'_, PyAny\u003e) -\u003e PyResult\u003cSelf\u003e {\n         let py = value.py();\n         let typ = value.getattr(\"type\")?;\n-        let val = value.call_method0(\"as_py\")?;\n+        let val = if value.hasattr(\"value\")? {\n+            value.getattr(\"value\")?\n+        } else {\n+            value.call_method0(\"as_py\")?\n+        };\n```\n\nBut then we still don't have the metadata. It is lost and we get a bare fixed sized binary.\n\n**Describe the solution you'd like**\n\nThe above code should *just work*. I have done a little investigation and using the pycapsule interface we *can* get the schema of the array we generate inside `PyScalarValue::from_pyarrow_bound`. We can then plumb this through when calling `lit()`.\n\nIdeally we would take this opportunity to ensure that when we call `PyScalarValue::from_pyarrow_bound` we are also supporting other libraries besides just `pyarrow`. There has been a complaint a few times that we are too tightly coupled to `pyarrow`. In particular it would be good to demonstrate that when converting a Python object that is a scalar value it works for:\n\n- pyarrow\n- nanoarrow\n- arro3\n- polars\n\nI don't think we necessarily need to support pandas since they are not an Arrow library.\n\n**Describe alternatives you've considered**\n\nAlternatively the user can manually turn their data into the underlying storage and then attach the metadata from their extension type. This feels like a poor user experience.\n\n**Additional context**\n\nThis came up during a different investigation:\n\n\u003e Also worth evaluating while we're doing this: For scalar values, is it possible for them to contain metadata? If I do `pa.scalar(uuid.uuid4().bytes, type=pa.uuid())` and I check the `type` I should have the extension data. Maybe this is already supported, but as part of this PR I want to evaluate that as well.\n\n_Originally posted by @timsaucer in https://github.com/apache/datafusion-python/issues/1299#issuecomment-3497558869_\n","author":{"url":"https://github.com/timsaucer","@type":"Person","name":"timsaucer"},"datePublished":"2025-11-06T22:19:42.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/1301/datafusion-python/issues/1301"}