René's URL Explorer Experiment


Title: Redis sink flushes only rows that have more recent eventTimestamp by pyalex · Pull Request #913 · feast-dev/feast · GitHub

Open Graph Title: Redis sink flushes only rows that have more recent eventTimestamp by pyalex · Pull Request #913 · feast-dev/feast

X Title: Redis sink flushes only rows that have more recent eventTimestamp by pyalex · Pull Request #913 · feast-dev/feast

Description: What this PR does / why we need it: Since we keep only one row per entity in Redis storage previously it was unpredictable which row will be written, especially in batch ingestion. In this PR we read existing value from redis first to compare eventTimestamps and only if row from stream has more recent eventTimestamp it will be written to store. This guarantees that more recent row won't be overwritten with older value. Now redis will have the latest value for entity. Also, I refactored tests for redis sink to remove duplicated parts of code and use @Parametrized tests instead. Which issue(s) this PR fixes: Fixes # Does this PR introduce a user-facing change?: Redis sink has now more predictable order of writes. Value is being overwritten only by more recent row from stream.

Open Graph Description: What this PR does / why we need it: Since we keep only one row per entity in Redis storage previously it was unpredictable which row will be written, especially in batch ingestion. In this PR we re...

X Description: What this PR does / why we need it: Since we keep only one row per entity in Redis storage previously it was unpredictable which row will be written, especially in batch ingestion. In this PR we re...

Opengraph URL: https://github.com/feast-dev/feast/pull/913

X: @github

direct link

Domain: github.com

route-pattern/:user_id/:repository/pull/:id/files(.:format)
route-controllerpull_requests
route-actionfiles
fetch-noncev2:8a58d2b9-a1b2-91a2-2dca-8bd713c8aa0b
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-id9200:109507:2C491:38393:697DBD8D
html-safe-nonce0839bed2b55bfae1923a605c64ebef501ddb542939faf28d8d2edfda37e148b8
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI5MjAwOjEwOTUwNzoyQzQ5MTozODM5Mzo2OTdEQkQ4RCIsInZpc2l0b3JfaWQiOiI3MTM1OTQxMjAzMTc3ODgwOTczIiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0=
visitor-hmac93a210d5d5afebf0ce1503823bec41634b721d11431caa615b9bdd1113850fc7
hovercard-subject-tagpull_request:460716776
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///pull_requests/show/files
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/feast-dev/feast/pull/913/files
twitter:imagehttps://avatars.githubusercontent.com/u/1303659?s=400&v=4
twitter:cardsummary_large_image
og:imagehttps://avatars.githubusercontent.com/u/1303659?s=400&v=4
og:image:altWhat this PR does / why we need it: Since we keep only one row per entity in Redis storage previously it was unpredictable which row will be written, especially in batch ingestion. In this PR we re...
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
None60279d4097367e16897439d16d6bbe4180663db828c666eeed2656988ffe59f6
turbo-cache-controlno-preview
diff-viewunified
go-importgithub.com/feast-dev/feast git https://github.com/feast-dev/feast.git
octolytics-dimension-user_id57027613
octolytics-dimension-user_loginfeast-dev
octolytics-dimension-repository_id161133770
octolytics-dimension-repository_nwofeast-dev/feast
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id161133770
octolytics-dimension-repository_network_root_nwofeast-dev/feast
turbo-body-classeslogged-out env-production page-responsive full-width
disable-turbotrue
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release7c85641c598ad130c74f7bcc27f58575cac69551
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/feast-dev/feast/pull/913/files#start-of-content
https://github.com/
Sign in https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Ffeast-dev%2Ffeast%2Fpull%2F913%2Ffiles
GitHub CopilotWrite better code with AIhttps://github.com/features/copilot
GitHub SparkBuild and deploy intelligent appshttps://github.com/features/spark
GitHub ModelsManage and compare promptshttps://github.com/features/models
MCP RegistryNewIntegrate external toolshttps://github.com/mcp
ActionsAutomate any workflowhttps://github.com/features/actions
CodespacesInstant dev environmentshttps://github.com/features/codespaces
IssuesPlan and track workhttps://github.com/features/issues
Code ReviewManage code changeshttps://github.com/features/code-review
GitHub Advanced SecurityFind and fix vulnerabilitieshttps://github.com/security/advanced-security
Code securitySecure your code as you buildhttps://github.com/security/advanced-security/code-security
Secret protectionStop leaks before they starthttps://github.com/security/advanced-security/secret-protection
Why GitHubhttps://github.com/why-github
Documentationhttps://docs.github.com
Bloghttps://github.blog
Changeloghttps://github.blog/changelog
Marketplacehttps://github.com/marketplace
View all featureshttps://github.com/features
Enterpriseshttps://github.com/enterprise
Small and medium teamshttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
App Modernizationhttps://github.com/solutions/use-case/app-modernization
DevSecOpshttps://github.com/solutions/use-case/devsecops
DevOpshttps://github.com/solutions/use-case/devops
CI/CDhttps://github.com/solutions/use-case/ci-cd
View all use caseshttps://github.com/solutions/use-case
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
View all industrieshttps://github.com/solutions/industry
View all solutionshttps://github.com/solutions
AIhttps://github.com/resources/articles?topic=ai
Software Developmenthttps://github.com/resources/articles?topic=software-development
DevOpshttps://github.com/resources/articles?topic=devops
Securityhttps://github.com/resources/articles?topic=security
View all topicshttps://github.com/resources/articles
Customer storieshttps://github.com/customer-stories
Events & webinarshttps://github.com/resources/events
Ebooks & reportshttps://github.com/resources/whitepapers
Business insightshttps://github.com/solutions/executive-insights
GitHub Skillshttps://skills.github.com
Documentationhttps://docs.github.com
Customer supporthttps://support.github.com
Community forumhttps://github.com/orgs/community/discussions
Trust centerhttps://github.com/trust-center
Partnershttps://github.com/partners
GitHub SponsorsFund open source developershttps://github.com/sponsors
Security Labhttps://securitylab.github.com
Maintainer Communityhttps://maintainers.github.com
Acceleratorhttps://github.com/accelerator
Archive Programhttps://archiveprogram.github.com
Topicshttps://github.com/topics
Trendinghttps://github.com/trending
Collectionshttps://github.com/collections
Enterprise platformAI-powered developer platformhttps://github.com/enterprise
GitHub Advanced SecurityEnterprise-grade security featureshttps://github.com/security/advanced-security
Copilot for BusinessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/premium-support
Pricinghttps://github.com/pricing
Search syntax tipshttps://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax
documentationhttps://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax
Sign in https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Ffeast-dev%2Ffeast%2Fpull%2F913%2Ffiles
Sign up https://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Ffiles&source=header-repo&source_repo=feast-dev%2Ffeast
Reloadhttps://github.com/feast-dev/feast/pull/913/files
Reloadhttps://github.com/feast-dev/feast/pull/913/files
Reloadhttps://github.com/feast-dev/feast/pull/913/files
feast-dev https://github.com/feast-dev
feasthttps://github.com/feast-dev/feast
Notifications https://github.com/login?return_to=%2Ffeast-dev%2Ffeast
Fork 1.2k https://github.com/login?return_to=%2Ffeast-dev%2Ffeast
Star 6.7k https://github.com/login?return_to=%2Ffeast-dev%2Ffeast
Code https://github.com/feast-dev/feast
Issues 183 https://github.com/feast-dev/feast/issues
Pull requests 68 https://github.com/feast-dev/feast/pulls
Discussions https://github.com/feast-dev/feast/discussions
Actions https://github.com/feast-dev/feast/actions
Security 0 https://github.com/feast-dev/feast/security
Insights https://github.com/feast-dev/feast/pulse
Code https://github.com/feast-dev/feast
Issues https://github.com/feast-dev/feast/issues
Pull requests https://github.com/feast-dev/feast/pulls
Discussions https://github.com/feast-dev/feast/discussions
Actions https://github.com/feast-dev/feast/actions
Security https://github.com/feast-dev/feast/security
Insights https://github.com/feast-dev/feast/pulse
Sign up for GitHub https://github.com/signup?return_to=%2Ffeast-dev%2Ffeast%2Fissues%2Fnew%2Fchoose
terms of servicehttps://docs.github.com/terms
privacy statementhttps://docs.github.com/privacy
Sign inhttps://github.com/login?return_to=%2Ffeast-dev%2Ffeast%2Fissues%2Fnew%2Fchoose
feast-ci-bothttps://github.com/feast-ci-bot
feast-dev:masterhttps://github.com/feast-dev/feast/tree/master
pyalex:redis-read-before-writehttps://github.com/pyalex/feast/tree/redis-read-before-write
Conversation 13 https://github.com/feast-dev/feast/pull/913
Commits 11 https://github.com/feast-dev/feast/pull/913/commits
Checks 0 https://github.com/feast-dev/feast/pull/913/checks
Files changed https://github.com/feast-dev/feast/pull/913/files
Please reload this pagehttps://github.com/feast-dev/feast/pull/913/files
Redis sink flushes only rows that have more recent eventTimestamp https://github.com/feast-dev/feast/pull/913/files#top
Show all changes 11 commits https://github.com/feast-dev/feast/pull/913/files
1d1c048 redis sink read then write pyalex Aug 1, 2020 https://github.com/feast-dev/feast/pull/913/commits/1d1c0484cba9019c33b2051454521dbef958bf88
24d83fd fix load tests pyalex Aug 1, 2020 https://github.com/feast-dev/feast/pull/913/commits/24d83fdabeeb9642e725c2af7d047313cb98b940
4e91993 e2e pyalex Aug 1, 2020 https://github.com/feast-dev/feast/pull/913/commits/4e919939086fed57027dcc2e2dd3ceb707a09b32
fc8e019 fix pyalex Aug 1, 2020 https://github.com/feast-dev/feast/pull/913/commits/fc8e019881f5d39e8ec14cd6668e2a601ad72d06
982f700 specify feature ref pyalex Aug 1, 2020 https://github.com/feast-dev/feast/pull/913/commits/982f7003aedb9237557d89cc75fce3822f4735dd
07fc09c move test up in order pyalex Aug 1, 2020 https://github.com/feast-dev/feast/pull/913/commits/07fc09cb68d3ace7b6def1ab639080e47e56a402
c1ecd03 set project default pyalex Aug 1, 2020 https://github.com/feast-dev/feast/pull/913/commits/c1ecd03c8531b1db23bbbf3c40fb387dab628c88
2c83445 some docs pyalex Aug 1, 2020 https://github.com/feast-dev/feast/pull/913/commits/2c83445ca7efbc88c6418d5e1b7e2792fa56dda1
cf9f274 reorder e2e tests pyalex Aug 1, 2020 https://github.com/feast-dev/feast/pull/913/commits/cf9f27404f666bb244f99cbfe075c5ba54db323a
db1b76a reorder e2e tests pyalex Aug 1, 2020 https://github.com/feast-dev/feast/pull/913/commits/db1b76a3488b67692e732e7080fd6e0ad83eacc0
19a41e7 reorder e2e tests pyalex Aug 1, 2020 https://github.com/feast-dev/feast/pull/913/commits/19a41e78bf58394a08a5d15e6fd6b7d535b2630d
Clear filters https://github.com/feast-dev/feast/pull/913/files
Please reload this pagehttps://github.com/feast-dev/feast/pull/913/files
Please reload this pagehttps://github.com/feast-dev/feast/pull/913/files
pom.xml https://github.com/feast-dev/feast/pull/913/files#diff-8d226dc9b56c36431b6c6c37500e8cf95822dae3325df4cefa035b74a25caec9
BatchDoFnWithRedis.java https://github.com/feast-dev/feast/pull/913/files#diff-6310cb9dab1e3a202619164f1543956e17a3401c7983a39735d704a003f1bdd9
RedisClusterIngestionClient.java https://github.com/feast-dev/feast/pull/913/files#diff-9a3873628a5bd70c15419195f0d1c4e59aa4b3aad7be439b0ecd760a40dac713
RedisCustomIO.java https://github.com/feast-dev/feast/pull/913/files#diff-c3a620361ed2ef289eee2ef4f7413198f84d267797bebed25072635034a4d0b6
RedisIngestionClient.java https://github.com/feast-dev/feast/pull/913/files#diff-ca5d77aa7b980155e26a8dd3a58ff35e724afd8f6c8cc8dda7c5c126f201d0f2
RedisStandaloneIngestionClient.java https://github.com/feast-dev/feast/pull/913/files#diff-7582ca662792e407b7a21e123673363b2fd6814064a4f569b51c84fe39b0aadf
RedisClusterFeatureSinkTest.java https://github.com/feast-dev/feast/pull/913/files#diff-f84d40ca1ed8ea7dfcf43e635870367fc17adf5cdebb0186f1264649db16b472
RedisFeatureSinkTest.java https://github.com/feast-dev/feast/pull/913/files#diff-81e5590bcd0bcc4358ce6ab2fef09dddd76ec5ef62e13219b6c282d682ab744f
basic-ingest-redis-serving.py https://github.com/feast-dev/feast/pull/913/files#diff-d39afd313de0770dd750e6d26f0165fcb9edc5b396b074bc4b41645cff572d77
storage/connectors/redis/pom.xmlhttps://github.com/feast-dev/feast/pull/913/files#diff-8d226dc9b56c36431b6c6c37500e8cf95822dae3325df4cefa035b74a25caec9
View file https://github.com/pyalex/feast/blob/19a41e78bf58394a08a5d15e6fd6b7d535b2630d/storage/connectors/redis/pom.xml
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/feast-dev/feast/pull/913/{{ revealButtonHref }}
https://github.com/feast-dev/feast/pull/913/files#diff-8d226dc9b56c36431b6c6c37500e8cf95822dae3325df4cefa035b74a25caec9
https://github.com/feast-dev/feast/pull/913/files#diff-8d226dc9b56c36431b6c6c37500e8cf95822dae3325df4cefa035b74a25caec9
...nectors/redis/src/main/java/feast/storage/connectors/redis/writer/BatchDoFnWithRedis.javahttps://github.com/feast-dev/feast/pull/913/files#diff-6310cb9dab1e3a202619164f1543956e17a3401c7983a39735d704a003f1bdd9
View file https://github.com/pyalex/feast/blob/19a41e78bf58394a08a5d15e6fd6b7d535b2630d/storage/connectors/redis/src/main/java/feast/storage/connectors/redis/writer/BatchDoFnWithRedis.java
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/feast-dev/feast/pull/913/{{ revealButtonHref }}
mrzzyhttps://github.com/mrzzy
Aug 1, 2020https://github.com/feast-dev/feast/pull/913/files#r463943198
Learn morehttps://docs.github.com/articles/managing-disruptive-comments/#hiding-a-comment
Please reload this pagehttps://github.com/feast-dev/feast/pull/913/files
pyalexhttps://github.com/pyalex
Aug 1, 2020https://github.com/feast-dev/feast/pull/913/files#r463945307
Learn morehttps://docs.github.com/articles/managing-disruptive-comments/#hiding-a-comment
Please reload this pagehttps://github.com/feast-dev/feast/pull/913/files
...edis/src/main/java/feast/storage/connectors/redis/writer/RedisClusterIngestionClient.javahttps://github.com/feast-dev/feast/pull/913/files#diff-9a3873628a5bd70c15419195f0d1c4e59aa4b3aad7be439b0ecd760a40dac713
View file https://github.com/pyalex/feast/blob/19a41e78bf58394a08a5d15e6fd6b7d535b2630d/storage/connectors/redis/src/main/java/feast/storage/connectors/redis/writer/RedisClusterIngestionClient.java
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/feast-dev/feast/pull/913/{{ revealButtonHref }}
https://github.com/feast-dev/feast/pull/913/files#diff-9a3873628a5bd70c15419195f0d1c4e59aa4b3aad7be439b0ecd760a40dac713
https://github.com/feast-dev/feast/pull/913/files#diff-9a3873628a5bd70c15419195f0d1c4e59aa4b3aad7be439b0ecd760a40dac713
https://github.com/feast-dev/feast/pull/913/files#diff-9a3873628a5bd70c15419195f0d1c4e59aa4b3aad7be439b0ecd760a40dac713
https://github.com/feast-dev/feast/pull/913/files#diff-9a3873628a5bd70c15419195f0d1c4e59aa4b3aad7be439b0ecd760a40dac713
https://github.com/feast-dev/feast/pull/913/files#diff-9a3873628a5bd70c15419195f0d1c4e59aa4b3aad7be439b0ecd760a40dac713
Please reload this pagehttps://github.com/feast-dev/feast/pull/913/files
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width

URLs of crawlers that visited me.