René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Materializes all rows as Python proto objects — high memory usage at scale","articleBody":"**Is your feature request related to a problem? Please describe.**\nWhen materializing features to an online store via LocalOutputNode, the current implementation converts the entire Arrow Table into a Python list of ValueProto objects before any writing occurs. At hundreds of thousands of rows, this causes severe memory pressure and can OOM in practice.\n\n**Root Cause**\n\nThe call chain in [`LocalOutputNode.execute()`](sdk/python/feast/infra/compute_engines/local/nodes.py#L369):\n\n```python\nrows_to_write = _convert_arrow_to_proto(\n    input_table, self.feature_view, join_key_to_value_type\n)\nonline_store.online_write_batch(..., data=rows_to_write, ...)\n```\n\n`_convert_arrow_to_proto` ([utils.py:325](sdk/python/feast/utils.py#L325)) performs three full-data copies sequentially:\n\n1. **Arrow → NumPy** (`to_numpy(zero_copy_only=False)`) — necessary to bridge Arrow nulls to Python type system\n2. **NumPy → `List[ValueProto]`** — each scalar becomes an independent Python protobuf heap object (~200 bytes overhead per value vs 4–8 bytes raw)\n3. **Column-wise → row-wise** (`list(zip(...))`) — full materialization into a Python list\n\n\n**Describe the solution you'd like**\n\n**Chunk iteration in `LocalOutputNode` (minimal, low-risk):**\n\n```python\nBATCH_SIZE = 10_000\nfor batch in input_table.to_batches(max_chunksize=BATCH_SIZE):\n    rows_to_write = _convert_arrow_to_proto(\n        batch, self.feature_view, join_key_to_value_type\n    )\n    online_store.online_write_batch(\n        config=context.repo_config,\n        table=self.feature_view,\n        data=rows_to_write,\n        progress=lambda x: None,\n    )\n    # rows_to_write eligible for GC after each iteration\n```\n\n**Describe alternatives you've considered**\nA clear and concise description of any alternative solutions or features you've considered.\n\n**Additional context**\nAdd any other context or screenshots about the feature request here.\n","author":{"url":"https://github.com/cutoutsy","@type":"Person","name":"cutoutsy"},"datePublished":"2026-03-26T12:39:20.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/6160/feast/issues/6160"}