René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Cannot do udaf that returns list of timestamps","articleBody":"**Describe the bug**\nI'm trying to generate a udaf that returns multiple timestamps for each partition id.\n\n**To Reproduce**\n```python\nimport datafusion as dfn\nfrom datafusion import udf, udaf, Accumulator, col\nimport pyarrow as pa\nimport pyarrow.compute as pc\nimport numpy as np\n\nclass ResampleAccumulator(Accumulator):\n    def __init__(self):\n        self._min = float('inf')\n        self._max = 0\n        # 10 Hz\n        self._timestep = 100 # ms\n\n    def update(self, array):\n        # Logic to update the sum and count from an input array\n        # In a real implementation, you would process the pyarrow array efficiently\n        print(\"Enter update\")\n        local_min, local_max = pc.min_max(array).values()\n        local_min_ns = local_min.cast(pa.timestamp('ns')).value\n        local_max_ns = local_max.cast(pa.timestamp('ns')).value\n\n        self._min = min(local_min_ns, self._min)\n        self._max = max(local_max_ns, self._max)\n        print(f\"update {self._min=}, {self._max=}\")\n\n    def merge(self, states_array):\n        print(\"Enter merge\")\n        # Is there a better way to do this with pc?\n        # or maybe just throw it into numpy\n        self._min = min(states_array[0][0].as_py(), self._min)\n        self._max = max(states_array[1][0].as_py(), self._max)\n        print(f\"merge {self._min=}, {self._max=}\")\n\n\n    def state(self):\n        print(\"Enter state\")\n        # Return the current state as a list of scalars\n        return pa.array([self._min, self._max], type=pa.int64())\n\n    def evaluate(self):\n        print(\"Enter evaluate\")\n        desired_timestamps = np.arange(np.datetime64(self._min, 'ns'), np.datetime64(self._max, 'ns'), np.timedelta64(self._timestep, \"ms\"))\n        print(f\"{len(desired_timestamps)=}\")\n        array_result = pa.array(desired_timestamps, type=pa.timestamp('ns'))\n        print(array_result)\n        return array_result\nresample_udaf = udaf(ResampleAccumulator, [pa.timestamp('ns')], pa.list_(pa.timestamp('ns')), [pa.int64(), pa.int64()], volatility=\"stable\")\n\nctx = dfn.SessionContext()\n\ndf = ctx.from_pydict({\"id\": [0,1], \"time\": [np.datetime64(0, 'ns'), np.datetime64(1_000_000_000, 'ns')]})\nprint(df)\n\nresult = df.aggregate(\n    \"id\",\n    [resample_udaf(col(\"time\"))]\n)\nprint(result.schema())\nresult.collect()\n```\nOutput\n```bash\nTraceback (most recent call last):\n  File \"\u003cpath\u003e/\u003cfile\u003e.py\", line 60, in \u003cmodule\u003e\n    result.collect()\n  File \"\u003cpath\u003e/.venv/lib/python3.12/site-packages/datafusion/dataframe.py\", line 729, in collect\n    return self.df.collect()\n           ^^^^^^^^^^^^^^^^^\nException: DataFusion error: Execution(\"ArrowTypeError: object of type \u003cclass 'pyarrow.lib.TimestampArray'\u003e cannot be converted to int\")\n```\n\n**Expected behavior**\nThis works or provides a clearer error.\n\n**Additional context**\nFails on datafusion 51. If I return just a single timestamp and update the udaf call then this works.","author":{"url":"https://github.com/ntjohnson1","@type":"Person","name":"ntjohnson1"},"datePublished":"2026-01-15T15:53:22.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/1339/datafusion-python/issues/1339"}