Title: Ray compute engine reports a failed online-store write as a successful materialization · Issue #7024 · feast-dev/feast · GitHub
Open Graph Title: Ray compute engine reports a failed online-store write as a successful materialization · Issue #7024 · feast-dev/feast
X Title: Ray compute engine reports a failed online-store write as a successful materialization · Issue #7024 · feast-dev/feast
Description: Expected Behavior When the online store refuses a write during feast materialize with the Ray compute engine, the materialization fails (job status ERROR and FeatureStore.materialize raises), the way it does with the Local engine on the ...
Open Graph Description: Expected Behavior When the online store refuses a write during feast materialize with the Ray compute engine, the materialization fails (job status ERROR and FeatureStore.materialize raises), the w...
X Description: Expected Behavior When the online store refuses a write during feast materialize with the Ray compute engine, the materialization fails (job status ERROR and FeatureStore.materialize raises), the w...
Opengraph URL: https://github.com/feast-dev/feast/issues/7024
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Ray compute engine reports a failed online-store write as a successful materialization","articleBody":"## Expected Behavior\n\nWhen the online store refuses a write during `feast materialize` with the Ray compute engine, the materialization fails (job status ERROR and `FeatureStore.materialize` raises), the way it does with the Local engine on the same setup.\n\n## Current Behavior\n\nWith `batch_engine: ray.engine`, the job reports SUCCEEDED, `materialize` returns normally, and `get_online_features` keeps serving the previous values. Three handlers absorb the error:\n\n- `RayWriteNode.execute` (`sdk/python/feast/infra/compute_engines/ray/nodes.py`, 864): the batch writer is decorated with `@safe_batch_processor`, which logs and returns the batch on any exception (`ray/utils.py` 210-212). This is the path `FeatureStore.materialize` takes.\n- `write_to_online_store` (`ray/utils.py`, handler 82-83) and `write_to_online_store_from_ray_ds` (177-180) log the error and do not raise. These are on the legacy `from_offline_store` path.\n\nWhen Feast starts Ray itself, `log_to_driver` is off unless `enable_ray_logging` is set (`sdk/python/feast/infra/ray_initializer.py` 668-682), so the error line does not reach the console either; it only appears in a Ray worker log under `/tmp/ray`.\n\n`RayReadNode` in the same file re-raises after logging, and the Local, Spark and Flink engines let the error through.\n\n### Something this is hiding\n\n`tests/component/ray/test_compute.py::test_ray_compute_engine_materialize` passes `online_store=MagicMock()` and asserts SUCCEEDED. Run as-is with a real SQLite online store in place of the mock, it still reports SUCCEEDED, but the online table holds 0 rows: the aggregated BatchFeatureView's write raises `ValueError: Column conv_rate not found in table` inside `_convert_arrow_to_proto`, and the handlers above absorb it. With the aggregate column names in the schema instead, the next error is `KeyError: 'Field \"event_timestamp\" does not exist in schema'`. So it looks like the Ray engine cannot currently write an aggregated BatchFeatureView to the online store at all, which matches the `TODO` at `feast/utils.py` 577.\n\n## Steps to reproduce\n\nOn master (386759c), a feature repo with a parquet `FileSource`, the file offline store, the SQLite online store and `batch_engine: {type: ray.engine, max_workers: 1}`:\n\n1. `materialize` once with the online database writable: values 0.8 / 0.7 are served.\n2. Make the online database file read-only (`chmod 0444`), add newer rows (0.1 / 0.2) to the source, and `materialize` again. SQLite raises `attempt to write a readonly database` inside the worker.\n3. The Ray job status is SUCCEEDED, `materialize` returns normally, and `get_online_features` still returns 0.8 / 0.7.\n4. The Local engine on the same read-only database reports ERROR and `materialize` raises.\n\nThe read-only file is a deliberate stand-in for an online store that refuses writes; I have not seen this in a production deployment.\n\n### Specifications\n\n- Version: master at 386759c (15 Oct 2026)\n- Platform: macOS, Python 3.10.7, Ray 2.58.0\n- Subsystem: Ray compute engine (`infra/compute_engines/ray`)\n\n## Possible Solution\n\nA small change (+9/-2 across `ray/utils.py` and `ray/nodes.py`): re-raise after the existing log in the two write functions, and take the write node's batch function out of `safe_batch_processor`, keeping its empty-batch early return. `safe_batch_processor` itself is unchanged (its swallow is pinned by `test_exception_in_func_returns_original_batch`, and other nodes use it). With it, five new tests fail on master and pass, four controls pass both ways, and the existing Ray component tests stay at 75 passed (non-integration) and 13 of 14 (integration).\n\nThe one integration test that turns red is `test_ray_compute_engine_materialize`, for the reason above. That seems like your call, so I have not changed it. A few options:\n\n1. assert ERROR there and track aggregated BatchFeatureView online writes on Ray as their own issue,\n2. drop the aggregations from that test so it tests a plain materialization, or\n3. fix the aggregated write first.\n\nThe intended behaviour change is that a failed online write fails the materialization instead of reporting success, so a pipeline that has been writing nothing will start failing. Happy to open a PR with the change and tests once you have a preference on the test.\n\nThis issue was written with AI assistance (Claude Code); the tests above were run before filing.\n","author":{"url":"https://github.com/ericdelorefice","@type":"Person","name":"ericdelorefice"},"datePublished":"2026-10-15T21:45:11Z","url":"https://github.com/feast-dev/feast/issues/7024"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:3d355d8f-4673-0a32-344f-bfea6ca57c37 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | 9AE6:3C86C1:D5EF79:119026D:6AFE8F79 |
| html-safe-nonce | c5531a7d4fdcf9883958403aa7b80087ee3d9647928ad448c6efbb6d31f38cd3 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI5QUU2OjNDODZDMTpENUVGNzk6MTE5MDI2RDo2QUZFOEY3OSIsInZpc2l0b3JfaWQiOiI0MDc1NzQ4NDU2MTU1NDg0MDI1IiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | 94cc5091b582b15de7e3deb37730658ffe740e8da90fa204c5449eeeb8f55ed2 |
| hovercard-subject-tag | issue:5874430095 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/feast-dev/feast/7024/issue_layout |
| twitter:image | https://opengraph.githubassets.com/3744b01a46b5eaee5344e357dc08b9e6760d9a03425a6fffbf68a6a9505940f8/feast-dev/feast/issues/7024 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/3744b01a46b5eaee5344e357dc08b9e6760d9a03425a6fffbf68a6a9505940f8/feast-dev/feast/issues/7024 |
| og:image:alt | Expected Behavior When the online store refuses a write during feast materialize with the Ray compute engine, the materialization fails (job status ERROR and FeatureStore.materialize raises), the w... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | ericdelorefice |
| hostname | github.com |
| expected-hostname | github.com |
| None | b3c0f155eafc27e60db6e254d1ffff0e072e9ab3b4018a5ee518797c19d89f86 |
| turbo-cache-control | no-preview |
| go-import | github.com/feast-dev/feast git https://github.com/feast-dev/feast.git |
| octolytics-dimension-user_id | 57027613 |
| octolytics-dimension-user_login | feast-dev |
| octolytics-dimension-repository_id | 161133770 |
| octolytics-dimension-repository_nwo | feast-dev/feast |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 161133770 |
| octolytics-dimension-repository_network_root_nwo | feast-dev/feast |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 4a10310af51a1bfffbedc4fd81cb582acf0cb0ff |
| ui-target | canary-1 |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width