Title: Move communication with IngestionJob to JobCoordinator by pyalex · Pull Request #800 · feast-dev/feast · GitHub
Open Graph Title: Move communication with IngestionJob to JobCoordinator by pyalex · Pull Request #800 · feast-dev/feast
X Title: Move communication with IngestionJob to JobCoordinator by pyalex · Pull Request #800 · feast-dev/feast
Description: What this PR does / why we need it:
To resolve some concerns related to overloading responsibilities in SpecService in my previous PR #792 I moved all communications with IngestionJob to JobCoordinatorService
Implemented flow
New FeatureSet:
FeatureSet being created through SpecService.applyFeatureSet with status PENDING
JobCoordinator.Poll allocates this Set to Job(s) by creating FeatureSetJobStatus (see JobUpdateTask.updateFeatureSets) with version=0 and deliveryStatus=in-progress
JobCoordinator.notifyJobs detects that there's pending FeatureSet with allocated job(s) and send FeatureSetSpec to kafka (topic is shared between all jobs)
JobCoordinator.listenAckFromJobs receives ack from job and update deliveryStatus in FeatureSetJobStatus
As soon as all allocated jobs acknowledged FeatureSet status set to READY
Existing FeatureSet:
FeatureSet being updated through SpecService.applyFeatureSet. Status changed to PENDING. Version incremented.
FeatureSet is already allocated to job(s) so JobCoordinator.notifyJobs picks it up and send to kafka with updating FeatureSetJobStatus to the latest version
... same flow as previous
New job spawned (bc previous canceled, eg):
JobUpdateTask creates FeatureSetJobStatuses for all allocated FeatureSets with version =
Open Graph Description: What this PR does / why we need it: To resolve some concerns related to overloading responsibilities in SpecService in my previous PR #792 I moved all communications with IngestionJob to JobCoordin...
X Description: What this PR does / why we need it: To resolve some concerns related to overloading responsibilities in SpecService in my previous PR #792 I moved all communications with IngestionJob to JobCoordin...
Opengraph URL: https://github.com/feast-dev/feast/pull/800
X: @github
Domain: github.com
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:68002d47-39d5-2174-78d5-488e5146c0a6 |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | C088:99395:54CFC:67265:697F98B5 |
| html-safe-nonce | d7f2a8cd413ebeb98e7696c9eddd616c0d7980f7d30760f25d3897ee6e72865d |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJDMDg4Ojk5Mzk1OjU0Q0ZDOjY3MjY1OjY5N0Y5OEI1IiwidmlzaXRvcl9pZCI6IjkxMzQ3NDA3NjI3OTIwMDc4NjEiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | 52509904ea59a64171a132a29f25bac53045abe7bf5429c32cabfda6c2a0d6bf |
| hovercard-subject-tag | pull_request:435679877 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/feast-dev/feast/pull/800/files |
| twitter:image | https://avatars.githubusercontent.com/u/1303659?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/1303659?s=400&v=4 |
| og:image:alt | What this PR does / why we need it: To resolve some concerns related to overloading responsibilities in SpecService in my previous PR #792 I moved all communications with IngestionJob to JobCoordin... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | 60279d4097367e16897439d16d6bbe4180663db828c666eeed2656988ffe59f6 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/feast-dev/feast git https://github.com/feast-dev/feast.git |
| octolytics-dimension-user_id | 57027613 |
| octolytics-dimension-user_login | feast-dev |
| octolytics-dimension-repository_id | 161133770 |
| octolytics-dimension-repository_nwo | feast-dev/feast |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 161133770 |
| octolytics-dimension-repository_network_root_nwo | feast-dev/feast |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 7c85641c598ad130c74f7bcc27f58575cac69551 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width