Title: File offline store returns wrong or future feature values when the entity dataframe has a column named like the source timestamp field · Issue #7176 · feast-dev/feast · GitHub
Open Graph Title: File offline store returns wrong or future feature values when the entity dataframe has a column named like the source timestamp field · Issue #7176 · feast-dev/feast
X Title: File offline store returns wrong or future feature values when the entity dataframe has a column named like the source timestamp field · Issue #7176 · feast-dev/feast
Description: Expected Behavior get_historical_features returns, for each entity row, the latest feature value at or before the row's event timestamp, and returns the entity dataframe's own columns unchanged, whatever they are called. The DuckDB offli...
Open Graph Description: Expected Behavior get_historical_features returns, for each entity row, the latest feature value at or before the row's event timestamp, and returns the entity dataframe's own columns unchanged, wh...
X Description: Expected Behavior get_historical_features returns, for each entity row, the latest feature value at or before the row's event timestamp, and returns the entity dataframe's own columns uncha...
Opengraph URL: https://github.com/feast-dev/feast/issues/7176
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"File offline store returns wrong or future feature values when the entity dataframe has a column named like the source timestamp field","articleBody":"## Expected Behavior\n\n`get_historical_features` returns, for each entity row, the latest feature value at or before the row's event timestamp, and returns the entity dataframe's own columns unchanged, whatever they are called. The DuckDB offline store does this.\n\n## Current Behavior\n\nWith the file (Dask) offline store, if the entity dataframe has a column with the same name as a feature view's source `timestamp_field` (other than the event timestamp column itself), the point-in-time join reads the entity dataframe's column instead of the source's:\n\n- The TTL filter and the latest-row selection use the request's value, so the result can be a value from after the event timestamp, or missing. In the example below, the expected value is 20 for every row; the file store returns `NA`, `NA` and `30` (a value from the day after the request).\n- The request's own column is dropped from the result, and the source column comes back under an internal name, `ts__`.\n- If the request's column is not a timestamp (a string, integer or float), the call fails with `AttributeError: 'str' object has no attribute 'tzinfo'`.\n- If the entity dataframe has both `ts` and `ts__`, the call fails with `ValueError: Duplicate column names found: ['driver_id', 'event_timestamp', 'ts__', 'ts__', ...]`. With two feature views whose timestamp field is `ts`, memory use keeps growing instead (past 2.4 GB within about 20 seconds for a one-row entity dataframe, when I stopped it).\n\n#7071 fixed the same problem for the source's `created_timestamp_column`. The event `timestamp_field` is still affected. The existing rename in `_field_mapping` only covers a `timestamp_field` equal to the entity dataframe's event timestamp column, and its `__` alias can itself collide with request data: with a source whose `timestamp_field` is `event_timestamp`, a request column named `__event_timestamp` makes the feature come back missing.\n\n## Steps to reproduce\n\n```python\nimport tempfile\nfrom datetime import timedelta\nfrom pathlib import Path\n\nimport pandas as pd\n\nfrom feast import Entity, FeatureStore, FeatureView, Field, FileSource, RepoConfig\nfrom feast.types import Int64\nfrom feast.value_type import ValueType\n\nrepo = Path(tempfile.mkdtemp())\npd.DataFrame(\n {\n \"driver_id\": [1, 1, 1, 2, 2, 2, 3, 3, 3],\n \"ts\": pd.to_datetime([\"2024-01-01\", \"2024-01-02\", \"2024-01-03\"] * 3, utc=True),\n \"trips\": [10, 20, 30] * 3,\n }\n).to_parquet(repo / \"trips.parquet\")\n\ndriver = Entity(name=\"driver\", join_keys=[\"driver_id\"], value_type=ValueType.INT64)\ntrips = FeatureView(\n name=\"driver_trips\",\n entities=[driver],\n schema=[Field(name=\"trips\", dtype=Int64)],\n source=FileSource(path=str(repo / \"trips.parquet\"), timestamp_field=\"ts\"),\n ttl=timedelta(days=7),\n)\n# Every driver has trips 10, 20 and 30 on 2024-01-01, -02 and -03. The requests are\n# at 2024-01-02 12:00, so the expected value is 20. The entity dataframe also has\n# its own column called \"ts\" (any request data that happens to use that name).\nentity_df = pd.DataFrame(\n {\n \"driver_id\": [1, 2, 3],\n \"event_timestamp\": pd.to_datetime([\"2024-01-02 12:00\"] * 3, utc=True),\n \"ts\": [\n pd.Timestamp(\"2023-12-25\", tz=\"UTC\"),\n pd.Timestamp(\"2024-01-10\", tz=\"UTC\"),\n pd.Timestamp(\"2024-01-02 01:00\", tz=\"UTC\"),\n ],\n }\n)\n\nfor offline_store in [\"file\", \"duckdb\"]:\n store = FeatureStore(\n config=RepoConfig(\n project=\"repro\",\n provider=\"local\",\n registry=str(repo / f\"registry_{offline_store}.db\"),\n offline_store={\"type\": offline_store},\n online_store={\"type\": \"sqlite\", \"path\": str(repo / \"online.db\")},\n entity_key_serialization_version=3,\n repo_path=str(repo),\n )\n )\n store.apply([driver, trips])\n df = store.get_historical_features(\n entity_df=entity_df, features=[\"driver_trips:trips\"]\n ).to_df()\n print(f\"# {offline_store}\")\n print(df.sort_values(\"driver_id\").to_string(index=False))\n```\n\nOutput on master (69ac7b0) and on 0.68.0 from PyPI:\n\n```\n# file\n driver_id event_timestamp ts__ trips\n 1 2024-01-02 12:00:00+00:00 NaT \u003cNA>\n 2 2024-01-02 12:00:00+00:00 NaT \u003cNA>\n 3 2024-01-02 12:00:00+00:00 2024-01-03 00:00:00+00:00 30\n# duckdb\n driver_id event_timestamp ts trips\n 1 2024-01-02 12:00:00+00:00 2023-12-25 00:00:00+00:00 20\n 2 2024-01-02 12:00:00+00:00 2024-01-10 00:00:00+00:00 20\n 3 2024-01-02 12:00:00+00:00 2024-01-02 01:00:00+00:00 20\n```\n\n### Specifications\n\n- Version: master (69ac7b0) and 0.68.0\n- Platform: macOS, Python 3.12\n- Subsystem: file (Dask) offline store, `get_historical_features`\n\n## Possible Solution\n\nIn `DaskOfflineStore.get_historical_features`, `_merge` joins the source data to the entity dataframe with `suffixes=(\"\", \"__\")`. When both have a column called `timestamp_field`, the entity dataframe's column keeps the name and the source's becomes `ts__`, but `_normalize_timestamp`, `_filter_ttl`, `_drop_duplicates` and `_drop_columns` all still use `timestamp_field`.\n\nThe fix: rename the source's timestamp column to an unused temporary name before the merge whenever the entity dataframe already has that name, as #7071 does for the created timestamp column. I have a PR ready with regression tests.\n","author":{"url":"https://github.com/iammanoj807","@type":"Person","name":"iammanoj807"},"datePublished":"2026-11-18T07:31:35Z","url":"https://github.com/feast-dev/feast/issues/7176"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:6ce4b2d6-593e-8aa1-c1cf-e74df37559db |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | 94F6:1085C1:D21D96:1145D6A:6AFE8F6C |
| html-safe-nonce | 27352223c60d40947fe3eb7e84b41629ea6b8c703a5e9efd95f5504625e02a98 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI5NEY2OjEwODVDMTpEMjFEOTY6MTE0NUQ2QTo2QUZFOEY2QyIsInZpc2l0b3JfaWQiOiIzMzMyMTY2OTAwMzg4MDQzMzIiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | 692c22184464516ccec1eb3fdc2a471522ff4f916c7f10ec531b6759cba6448b |
| hovercard-subject-tag | issue:6462261971 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/feast-dev/feast/7176/issue_layout |
| twitter:image | https://opengraph.githubassets.com/7ccfa0d121d28d33ec1403d3669d3e6a4f0bfaf5da5652653f183b5ffb2bd38d/feast-dev/feast/issues/7176 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/7ccfa0d121d28d33ec1403d3669d3e6a4f0bfaf5da5652653f183b5ffb2bd38d/feast-dev/feast/issues/7176 |
| og:image:alt | Expected Behavior get_historical_features returns, for each entity row, the latest feature value at or before the row's event timestamp, and returns the entity dataframe's own columns unchanged, wh... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | iammanoj807 |
| hostname | github.com |
| expected-hostname | github.com |
| None | b3c0f155eafc27e60db6e254d1ffff0e072e9ab3b4018a5ee518797c19d89f86 |
| turbo-cache-control | no-preview |
| go-import | github.com/feast-dev/feast git https://github.com/feast-dev/feast.git |
| octolytics-dimension-user_id | 57027613 |
| octolytics-dimension-user_login | feast-dev |
| octolytics-dimension-repository_id | 161133770 |
| octolytics-dimension-repository_nwo | feast-dev/feast |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 161133770 |
| octolytics-dimension-repository_network_root_nwo | feast-dev/feast |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 4a10310af51a1bfffbedc4fd81cb582acf0cb0ff |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width