Title: fix: Break event timestamp ties by created timestamp in the Dask offline store by LuisFigueroaG · Pull Request #6952 · feast-dev/feast · GitHub
Open Graph Title: fix: Break event timestamp ties by created timestamp in the Dask offline store by LuisFigueroaG · Pull Request #6952 · feast-dev/feast
X Title: fix: Break event timestamp ties by created timestamp in the Dask offline store by LuisFigueroaG · Pull Request #6952 · feast-dev/feast
Description: What this PR does / why we need it: When two rows share an event timestamp, Feast is supposed to keep the one with the latest created_timestamp_column. The Dask offline store did this with two consecutive sorts, first by the created timestamp and then by the event timestamp, and relied on the second sort preserving the order of the first: df_to_join = df_to_join.sort_values(by=created_timestamp_column, na_position="first") df_to_join = df_to_join.sort_values(by=timestamp_field, na_position="first") Dask's sort_values (and pandas' default quicksort) is not stable, so for tied event timestamps the older row can end up last and win drop_duplicates(keep="last"). With 300 entities, each having two rows with the same event timestamp, 291 came back with the older row. A single partition with 20 entities already returned 12 wrong rows. This affects get_historical_features (_drop_duplicates) and pull_latest_from_table_or_query (evaluate_offline_job). This sorts once by [timestamp_field, created_timestamp_column] in both places, including the ZeroDivisionError fallbacks. Multi-column sort_values is supported by dask since well before the dask>=2024.2.1 minimum. Which issue(s) this PR fixes: No existing issue. Checks I've made sure the tests are passing. My commits are signed off (git commit -s) My PR title follows conventional commits format Testing Strategy Unit tests Integration tests Manual tests Testing is not required for this change Added test_dask_created_timestamp_tiebreak.py, covering get_historical_features and pull_latest_from_table_or_query with 100 entities across 4 partitions. Both tests failed in 5 out of 5 runs on master and pass with this change. ruff format --check, ruff check and mypy pass on the changed files, and sdk/python/tests/unit/infra/offline_stores/ passes (291 passed).
Open Graph Description: What this PR does / why we need it: When two rows share an event timestamp, Feast is supposed to keep the one with the latest created_timestamp_column. The Dask offline store did this with two cons...
X Description: What this PR does / why we need it: When two rows share an event timestamp, Feast is supposed to keep the one with the latest created_timestamp_column. The Dask offline store did this with two cons...
Opengraph URL: https://github.com/feast-dev/feast/pull/6952
X: @github
Domain: github.com
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:d9bef1ca-139e-e6da-37c4-5b52aec398b1 |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | 97B6:37D9F:E2E7FC:1361BF9:6AFE9261 |
| html-safe-nonce | 9164f22ac7e639f2d0272f3c4d418f31073651636c44d64e3caece04ef595f94 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI5N0I2OjM3RDlGOkUyRTdGQzoxMzYxQkY5OjZBRkU5MjYxIiwidmlzaXRvcl9pZCI6IjM5ODYxNDUxOTkxNTg4OTExMDUiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | f37c497e5fa101bbec6fa9056e5fb40e2c0ac5d5ff145fd45804181424d2b117 |
| hovercard-subject-tag | pull_request:4752905566 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/feast-dev/feast/pull/6952/files |
| twitter:image | https://avatars.githubusercontent.com/u/62305361?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/62305361?s=400&v=4 |
| og:image:alt | What this PR does / why we need it: When two rows share an event timestamp, Feast is supposed to keep the one with the latest created_timestamp_column. The Dask offline store did this with two cons... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | b3c0f155eafc27e60db6e254d1ffff0e072e9ab3b4018a5ee518797c19d89f86 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/feast-dev/feast git https://github.com/feast-dev/feast.git |
| octolytics-dimension-user_id | 57027613 |
| octolytics-dimension-user_login | feast-dev |
| octolytics-dimension-repository_id | 161133770 |
| octolytics-dimension-repository_nwo | feast-dev/feast |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 161133770 |
| octolytics-dimension-repository_network_root_nwo | feast-dev/feast |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 4a10310af51a1bfffbedc4fd81cb582acf0cb0ff |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width