René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"BigQuery offline store loses array data when pushing features with list types","articleBody":"\n\n\n## Summary\n\n  When using `store.push()` with `PushMode.OFFLINE` or `PushMode.ONLINE_AND_OFFLINE`, array/list type columns (e.g., STRING_LIST) are written as empty arrays [] to BigQuery, even though the data is correct in the DataFrame and PyArrow table.\n\n## Root Cause\n\n  The BigQuery LoadJobConfig in offline_write_batch() is missing parquet_options.enable_list_inference = True. Without this option, BigQuery's parquet loader doesn't correctly interpret PyArrow's list format.\n\n  Related issue: https://github.com/googleapis/google-cloud-python/issues/15705#issuecomment-2541921785\n\n## Steps to Reproduce\n```\n  from feast import FeatureStore\n  from feast.data_source import PushMode\n  import pandas as pd\n  from datetime import datetime, timezone\n\n  # Assuming feature view with STRING_LIST field is configured\n  data = {\n      \"entity_id\": \"test_123\",\n      \"tags\": [\"category_a\", \"category_b\"],  # STRING_LIST type\n      \"event_time\": datetime.now(timezone.utc),\n  }\n\n  df = pd.DataFrame([data])\n  store = FeatureStore(repo_path=\".\")\n  store.push(\"my_push_source\", df, to=PushMode.ONLINE_AND_OFFLINE)\n\n  # Result in BigQuery: tags = [] (empty array)\n  # Expected: tags = [\"category_a\", \"category_b\"]\n```\n## Expected Behavior\n\n  Array data should be correctly written to BigQuery with values preserved.\n\n## Actual Behavior\n\n  Array columns are written as empty arrays [] in BigQuery, while the online store receives correct data.\n\n## Proposed Fix\n\n  In feast/infra/offline_stores/bigquery.py, update offline_write_batch() (~line 428):\n```\n  @staticmethod\n  def offline_write_batch(\n      config: RepoConfig,\n      feature_view: FeatureView,\n      table: pyarrow.Table,\n      progress: Optional[Callable[[int], Any]],\n  ):\n      # ... existing code ...\n\n      parquet_options = bigquery.ParquetOptions()\n      parquet_options.enable_list_inference = True\n\n      job_config = bigquery.LoadJobConfig(\n          source_format=bigquery.SourceFormat.PARQUET,\n          schema=arrow_schema_to_bq_schema(pa_schema),\n          create_disposition=config.offline_store.table_create_disposition,\n          write_disposition=\"WRITE_APPEND\",\n          parquet_options=parquet_options,  # Add this line\n      )\n      # ... rest of code ...\n```\n\n## Environment\n\n  - Feast version: 0.58.0\n  - Python version: 3.12\n  - BigQuery client version: (latest)\n\n## Additional Context\n\n  - Online store (PostgreSQL) receives array data correctly\n  - The PyArrow table contains correct array data before parquet write\n  - Parquet file contains correct data when read locally\n  - Only BigQuery load loses the array content\n  - Using load_table_from_json instead of parquet works correctly\n  - Adding enable_list_inference=True to ParquetOptions fixes the issue","author":{"url":"https://github.com/max36067","@type":"Person","name":"max36067"},"datePublished":"2026-01-13T07:55:03.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/5845/feast/issues/5845"}