Title: Ray integration tests are broken on master: list-typed features lose object dtype and Ray materialization writes nothing to the online store · Issue #6912 · feast-dev/feast · GitHub
Open Graph Title: Ray integration tests are broken on master: list-typed features lose object dtype and Ray materialization writes nothing to the online store · Issue #6912 · feast-dev/feast
X Title: Ray integration tests are broken on master: list-typed features lose object dtype and Ray materialization writes nothing to the online store · Issue #6912 · feast-dev/feast
Description: Problem The pr-ray-integration-tests workflow (integration-test-ray job) is failing on master for every PR, regardless of what the PR changes. 12 failed, 32 passed, 4 skipped. Example run: https://github.com/feast-dev/feast/actions/runs/...
Open Graph Description: Problem The pr-ray-integration-tests workflow (integration-test-ray job) is failing on master for every PR, regardless of what the PR changes. 12 failed, 32 passed, 4 skipped. Example run: https://...
X Description: Problem The pr-ray-integration-tests workflow (integration-test-ray job) is failing on master for every PR, regardless of what the PR changes. 12 failed, 32 passed, 4 skipped. Example run: https://...
Opengraph URL: https://github.com/feast-dev/feast/issues/6912
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Ray integration tests are broken on master: list-typed features lose object dtype and Ray materialization writes nothing to the online store","articleBody":"## Problem\n\nThe `pr-ray-integration-tests` workflow (`integration-test-ray` job) is failing on master for every PR, regardless of what the PR changes. 12 failed, 32 passed, 4 skipped.\n\nExample run: https://github.com/feast-dev/feast/actions/runs/36757976337/job/110032974407 (on PR #6909, which only touches `sdk/python/feast/feature_store.py`, `sdk/python/feast/utils.py` and a new unit test — nothing Ray-related).\n\nThe same job is failing on unrelated branches on the same day, which confirms it is a master-level breakage rather than a per-PR regression:\n\n- `fix/offline-server-bounded-batches`\n- `fix/sdk-dual-stack-binds`\n- `perf/decouple-metrics-server`\n\nAll failures are on the single `LOCAL:Ray:sqlite:python_fs:False` configuration.\n\n## Failure 1 — list-typed features lose `object` dtype (10 cases)\n\n`sdk/python/tests/integration/offline_store/test_universal_types.py::test_feature_get_historical_features_types_match`\n\n```\nsdk/python/tests/integration/offline_store/test_universal_types.py:202: in assert_feature_list_types\n assert pd.api.types.is_object_dtype(historical_features_df.dtypes[\"value\"])\nE AssertionError: assert False\n```\n\nFailing parametrizations — every `feature_is_list=True` case:\n\n| feature_dtype | has_empty_list |\n|---|---|\n| int32 | True / False |\n| int64 | True / False |\n| float | True / False |\n| bool | True / False |\n| datetime | True / False |\n\n`feature_is_list=False` cases pass. So `get_historical_features()` on the Ray offline store now returns list columns as a native Arrow/pandas list dtype instead of `object`.\n\n## Failure 2 — Ray materialization writes nothing to the online store (2 cases)\n\n`sdk/python/tests/integration/offline_store/test_universal_materialization.py::test_universal_materialization_consistency` (both `True` and `False` parametrizations)\n\n```\nsdk/python/tests/integration/offline_store/test_universal_materialization.py:39: in test_universal_materialization_consistency\n validate_offline_online_store_consistency(fs, driver_stats_fv, split_dt)\nsdk/python/tests/utils/e2e_test_validation.py:52: in validate_offline_online_store_consistency\n _check_offline_and_online_features(\nsdk/python/tests/utils/e2e_test_validation.py:162: in _check_offline_and_online_features\n assert not df.to_dict(orient=\"list\")[f\"{fv.name}__value\"] or math.isnan(\nE TypeError: must be real number, not NoneType\n```\n\nThe online read returns `None`, i.e. the Ray compute engine materialization did not land rows in the sqlite online store. The `TypeError` is a secondary symptom — `e2e_test_validation.py:162` also doesn't guard against `None` before calling `math.isnan`, so the real assertion never gets a chance to report.\n\n## Environment (from the CI run, `pixi -e ray-tests`)\n\n- ray 2.58.0\n- pandas 2.3.3\n- pyarrow 25.0.1\n\nGiven the dtype nature of failure 1, a recent Ray / pyarrow / pandas bump in the `ray-tests` pixi environment is the most likely trigger.\n\n## Expected\n\n`integration-test-ray` is green on master so it is a meaningful signal on PRs.\n\n## Suggested work\n\n1. Find the commit / lockfile bump that first turned this job red and confirm the trigger.\n2. Fix the Ray offline store list-column conversion so list features come back as `object` dtype (or update the universal type contract deliberately, if the new dtype is intended).\n3. Fix Ray compute engine materialization so rows actually reach the online store.\n4. Harden `_check_offline_and_online_features` in `sdk/python/tests/utils/e2e_test_validation.py:162` to handle `None` so a missing online value fails with a clear assertion instead of a `TypeError`.\n","author":{"url":"https://github.com/haoxu0","@type":"Person","name":"haoxu0"},"datePublished":"2026-09-30T19:54:48Z","url":"https://github.com/feast-dev/feast/issues/6912"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:2e3364e1-a7c5-2087-3f28-ecdc7126d150 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | D6FE:150F38:2C1FC0:3CE0D0:6AFE981C |
| html-safe-nonce | ef9a557333962c490e4cb302a6c68df134bf311555a19d3446283342a9c499e1 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJENkZFOjE1MEYzODoyQzFGQzA6M0NFMEQwOjZBRkU5ODFDIiwidmlzaXRvcl9pZCI6IjU0MzM1NDg0MDQ4MTY3NzkyOTIiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | f2b5881e2f1a9488d9b34ce55080b48d59d71c0a3fcd9dcd9f135a339d4f9443 |
| hovercard-subject-tag | issue:5652210836 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/feast-dev/feast/6912/issue_layout |
| twitter:image | https://opengraph.githubassets.com/3038ed0648e53a6ec02a23bb10e7f79b4cbeffeb0d0cac95c9929c40bca9375f/feast-dev/feast/issues/6912 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/3038ed0648e53a6ec02a23bb10e7f79b4cbeffeb0d0cac95c9929c40bca9375f/feast-dev/feast/issues/6912 |
| og:image:alt | Problem The pr-ray-integration-tests workflow (integration-test-ray job) is failing on master for every PR, regardless of what the PR changes. 12 failed, 32 passed, 4 skipped. Example run: https://... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | haoxu0 |
| hostname | github.com |
| expected-hostname | github.com |
| None | b3c0f155eafc27e60db6e254d1ffff0e072e9ab3b4018a5ee518797c19d89f86 |
| turbo-cache-control | no-preview |
| go-import | github.com/feast-dev/feast git https://github.com/feast-dev/feast.git |
| octolytics-dimension-user_id | 57027613 |
| octolytics-dimension-user_login | feast-dev |
| octolytics-dimension-repository_id | 161133770 |
| octolytics-dimension-repository_nwo | feast-dev/feast |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 161133770 |
| octolytics-dimension-repository_network_root_nwo | feast-dev/feast |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 4a10310af51a1bfffbedc4fd81cb582acf0cb0ff |
| ui-target | canary-2 |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width