René's URL Explorer Experiment


Title: perf: Avoid pandas copy in Spark reads by Pranjulcr7 · Pull Request #6878 · feast-dev/feast · GitHub

Open Graph Title: perf: Avoid pandas copy in Spark reads by Pranjulcr7 · Pull Request #6878 · feast-dev/feast

X Title: perf: Avoid pandas copy in Spark reads by Pranjulcr7 · Pull Request #6878 · feast-dev/feast

Description: What this PR does Pass the retrieved Arrow table directly to Spark, removing the full pandas intermediate in SparkReadNode and Iceberg temp-view registration. Feast already requires PySpark >= 4.0, which supports native Arrow table ingestion. Adds coverage for multiple chunks, nulls, column types, UTC timestamps, empty tables, Iceberg temp views, and native Spark retrieval jobs, with Arrow optimization enabled and disabled. Related issue Addresses the Spark part of #6863. The Flight and Trino work is separate. Testing Spark 4.2.0: all 9 node tests and 2 Iceberg registration cases pass, with both TZ=UTC and TZ=Pacific/Honolulu. Spark 4.0.0: all 7 read-node/Iceberg cases pass with TZ=Pacific/Honolulu. Ruff lint, formatting, and git diff --check pass. Local runs used Python 3.12.14, PyArrow 25.0.1, Java 17, and spark.python.use.daemon=false. Three existing worker-based tests crash in this runner on Spark 4.0; the full node suite passes on 4.2. A broader test_spark.py run also hit two staging mock assertions (filesystem=None) and missing optional boto3, outside the changed paths. Memory comparison 500,000 rows x 16 float64 columns, 8 Arrow chunks, 10% nulls: a 62 MiB table. Median of 3 fresh processes per path, Spark 4.2.0 in local[2], pandas 2.3.3. Arrow optimization enabled and fallback disabled for both paths; each DataFrame was materialized with count(). Python driver RSS Via pandas Direct Arrow Peak 259.0 MiB 193.3 MiB Increase over loaded-table baseline 66.6 MiB 1.3 MiB RSS was sampled every 5 ms during conversion and count. These figures exclude JVM memory and are specific to this synthetic input. Release note Spark materialization uses native Arrow ingestion to avoid the pandas copy and preserve Arrow column types. Commits signed off Conventional commit title

Open Graph Description: What this PR does Pass the retrieved Arrow table directly to Spark, removing the full pandas intermediate in SparkReadNode and Iceberg temp-view registration. Feast already requires PySpark >= 4...

X Description: What this PR does Pass the retrieved Arrow table directly to Spark, removing the full pandas intermediate in SparkReadNode and Iceberg temp-view registration. Feast already requires PySpark &gt...

Opengraph URL: https://github.com/feast-dev/feast/pull/6878

X: @github

direct link

Domain: github.com

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/:user_id/:repository/pull/:id/files(.:format)
route-controllerpull_requests
route-actionfiles
fetch-noncev2:c01a59e9-bf32-7a4d-1acf-be204e9833ff
current-catalog-service-hashae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b
request-id99C6:9D4BF:EC1A25:137AC53:6AFE9896
html-safe-nonceff2c7efc0ab2dbf4cfcac5d706f9a76b12fedf9dc0b12f87f66697847fb65bc1
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI5OUM2OjlENEJGOkVDMUEyNToxMzdBQzUzOjZBRkU5ODk2IiwidmlzaXRvcl9pZCI6IjIyMTIzNjk3NDc5NjAzMDU4MTUiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ==
visitor-hmac6766d5c275222d8ac385f991cb88c7fe63ecc374dec7f92898a6532e29f1fc6e
hovercard-subject-tagpull_request:4659208679
github-keyboard-shortcutsrepository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///pull_requests/show/files
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/feast-dev/feast/pull/6878/files
twitter:imagehttps://avatars.githubusercontent.com/u/31792249?s=400&v=4
twitter:cardsummary_large_image
og:imagehttps://avatars.githubusercontent.com/u/31792249?s=400&v=4
og:image:altWhat this PR does Pass the retrieved Arrow table directly to Spark, removing the full pandas intermediate in SparkReadNode and Iceberg temp-view registration. Feast already requires PySpark >= 4...
og:site_nameGitHub
og:typeobject
hostnamegithub.com
expected-hostnamegithub.com
Noneb3c0f155eafc27e60db6e254d1ffff0e072e9ab3b4018a5ee518797c19d89f86
turbo-cache-controlno-preview
diff-viewunified
go-importgithub.com/feast-dev/feast git https://github.com/feast-dev/feast.git
octolytics-dimension-user_id57027613
octolytics-dimension-user_loginfeast-dev
octolytics-dimension-repository_id161133770
octolytics-dimension-repository_nwofeast-dev/feast
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id161133770
octolytics-dimension-repository_network_root_nwofeast-dev/feast
turbo-body-classeslogged-out env-production page-responsive full-width
disable-turbotrue
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release4a10310af51a1bfffbedc4fd81cb582acf0cb0ff
ui-targetfull
theme-color#1e2327
color-schemelight dark

Links:

Skip to contenthttps://github.com/feast-dev/feast/pull/6878/files#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Ffeast-dev%2Ffeast%2Fpull%2F6878%2Ffiles
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Ffeast-dev%2Ffeast%2Fpull%2F6878%2Ffiles
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fpull_requests%2Fshow%2Ffiles&source=header-repo&source_repo=feast-dev%2Ffeast
Reloadhttps://github.com/feast-dev/feast/pull/6878/files
Reloadhttps://github.com/feast-dev/feast/pull/6878/files
Reloadhttps://github.com/feast-dev/feast/pull/6878/files
feast-devhttps://github.com/feast-dev
feasthttps://github.com/feast-dev/feast
Codehttps://github.com/feast-dev/feast
Issues223 (223)https://github.com/feast-dev/feast/issues
Pull requests143 (143)https://github.com/feast-dev/feast/pulls
Discussionshttps://github.com/feast-dev/feast/discussions
Actionshttps://github.com/feast-dev/feast/actions
Security and quality1 (1)https://github.com/feast-dev/feast/security
Insightshttps://github.com/feast-dev/feast/insights
Sign up for GitHub https://github.com/signup?return_to=%2Ffeast-dev%2Ffeast%2Fissues%2Fnew%2Fchoose
terms of servicehttps://docs.github.com/terms
privacy statementhttps://docs.github.com/privacy
Sign inhttps://github.com/login?return_to=%2Ffeast-dev%2Ffeast%2Fissues%2Fnew%2Fchoose
ntkatholehttps://github.com/ntkathole
feast-dev:masterhttps://github.com/feast-dev/feast/tree/master
Pranjulcr7:perf/spark-arrow-ingestionhttps://github.com/Pranjulcr7/feast/tree/perf/spark-arrow-ingestion
Conversation 9 https://github.com/feast-dev/feast/pull/6878
Commits 2 https://github.com/feast-dev/feast/pull/6878/commits
Checks 1 https://github.com/feast-dev/feast/pull/6878/checks
Files changed https://github.com/feast-dev/feast/pull/6878/files
Please reload this pagehttps://github.com/feast-dev/feast/pull/6878/files
perf: Avoid pandas copy in Spark reads https://github.com/feast-dev/feast/pull/6878/files#top
Show all changes 2 commits https://github.com/feast-dev/feast/pull/6878/files
fd61d4f perf: Avoid pandas copy in Spark reads Pranjulcr7 Sep 28, 2026 https://github.com/feast-dev/feast/pull/6878/commits/fd61d4f8afeaf216417240e90f9bb6d2a06d5e40
d0a03b2 fix: Complete Spark Arrow ingestion Pranjulcr7 Sep 28, 2026 https://github.com/feast-dev/feast/pull/6878/commits/d0a03b25411ad673bf1e19991de2b6599b0ae773
Clear filters https://github.com/feast-dev/feast/pull/6878/files
Please reload this pagehttps://github.com/feast-dev/feast/pull/6878/files
Please reload this pagehttps://github.com/feast-dev/feast/pull/6878/files
nodes.py https://github.com/feast-dev/feast/pull/6878/files#diff-ef3c9f42859de83e0118dbc9b53b79509853421c612aa0cf630f2f894ecd3538
spark.py https://github.com/feast-dev/feast/pull/6878/files#diff-ef3fce408c702d00aa891015376e83ff1084907c69c95cafacf5fae657fca486
test_nodes.py https://github.com/feast-dev/feast/pull/6878/files#diff-7d5ceaa851d400c2592d5fb605d2e2d475eaac7758534c23aeb28890b0f1658f
test_spark.py https://github.com/feast-dev/feast/pull/6878/files#diff-64a2d9d6d7262639e74ee108efbf3957b9ffae33f16deefe5a57e85b77b656e4
https://github.com/feast-dev/feast/blob/master/CODEOWNERS#L10
sdk/python/feast/infra/compute_engines/spark/nodes.pyhttps://github.com/feast-dev/feast/pull/6878/files#diff-ef3c9f42859de83e0118dbc9b53b79509853421c612aa0cf630f2f894ecd3538
View file https://github.com/Pranjulcr7/feast/blob/d0a03b25411ad673bf1e19991de2b6599b0ae773/sdk/python/feast/infra/compute_engines/spark/nodes.py
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/feast-dev/feast/pull/6878/{{ revealButtonHref }}
https://github.com/feast-dev/feast/pull/6878/files#diff-ef3c9f42859de83e0118dbc9b53b79509853421c612aa0cf630f2f894ecd3538
https://github.com/feast-dev/feast/pull/6878/files#diff-ef3c9f42859de83e0118dbc9b53b79509853421c612aa0cf630f2f894ecd3538
https://github.com/feast-dev/feast/pull/6878/files#diff-ef3c9f42859de83e0118dbc9b53b79509853421c612aa0cf630f2f894ecd3538
https://github.com/feast-dev/feast/pull/6878/files#diff-ef3c9f42859de83e0118dbc9b53b79509853421c612aa0cf630f2f894ecd3538
https://github.com/feast-dev/feast/blob/master/CODEOWNERS#L10
sdk/python/feast/infra/offline_stores/contrib/spark_offline_store/spark.pyhttps://github.com/feast-dev/feast/pull/6878/files#diff-ef3fce408c702d00aa891015376e83ff1084907c69c95cafacf5fae657fca486
View file https://github.com/Pranjulcr7/feast/blob/d0a03b25411ad673bf1e19991de2b6599b0ae773/sdk/python/feast/infra/offline_stores/contrib/spark_offline_store/spark.py
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/feast-dev/feast/pull/6878/{{ revealButtonHref }}
https://github.com/feast-dev/feast/pull/6878/files#diff-ef3fce408c702d00aa891015376e83ff1084907c69c95cafacf5fae657fca486
https://github.com/feast-dev/feast/pull/6878/files#diff-ef3fce408c702d00aa891015376e83ff1084907c69c95cafacf5fae657fca486
https://github.com/feast-dev/feast/blob/master/CODEOWNERS#L10
sdk/python/tests/component/spark/test_nodes.pyhttps://github.com/feast-dev/feast/pull/6878/files#diff-7d5ceaa851d400c2592d5fb605d2e2d475eaac7758534c23aeb28890b0f1658f
View file https://github.com/Pranjulcr7/feast/blob/d0a03b25411ad673bf1e19991de2b6599b0ae773/sdk/python/tests/component/spark/test_nodes.py
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/feast-dev/feast/pull/6878/{{ revealButtonHref }}
https://github.com/feast-dev/feast/pull/6878/files#diff-7d5ceaa851d400c2592d5fb605d2e2d475eaac7758534c23aeb28890b0f1658f
ntkatholehttps://github.com/ntkathole
Sep 28, 2026https://github.com/feast-dev/feast/pull/6878/files#r4119519546
Learn morehttps://docs.github.com/articles/managing-disruptive-comments/#hiding-a-comment
Please reload this pagehttps://github.com/feast-dev/feast/pull/6878/files
Pranjulcr7https://github.com/Pranjulcr7
Sep 28, 2026https://github.com/feast-dev/feast/pull/6878/files#r4126274520
Learn morehttps://docs.github.com/articles/managing-disruptive-comments/#hiding-a-comment
Please reload this pagehttps://github.com/feast-dev/feast/pull/6878/files
https://github.com/feast-dev/feast/pull/6878/files#diff-7d5ceaa851d400c2592d5fb605d2e2d475eaac7758534c23aeb28890b0f1658f
https://github.com/feast-dev/feast/blob/master/CODEOWNERS#L10
sdk/python/tests/component/spark/test_spark.pyhttps://github.com/feast-dev/feast/pull/6878/files#diff-64a2d9d6d7262639e74ee108efbf3957b9ffae33f16deefe5a57e85b77b656e4
View file https://github.com/Pranjulcr7/feast/blob/d0a03b25411ad673bf1e19991de2b6599b0ae773/sdk/python/tests/component/spark/test_spark.py
Open in desktop https://desktop.github.com
https://github.co/hiddenchars
https://github.com/feast-dev/feast/pull/6878/{{ revealButtonHref }}
https://github.com/feast-dev/feast/pull/6878/files#diff-64a2d9d6d7262639e74ee108efbf3957b9ffae33f16deefe5a57e85b77b656e4
https://github.com/feast-dev/feast/pull/6878/files#diff-64a2d9d6d7262639e74ee108efbf3957b9ffae33f16deefe5a57e85b77b656e4
https://github.com/feast-dev/feast/pull/6878/files#diff-64a2d9d6d7262639e74ee108efbf3957b9ffae33f16deefe5a57e85b77b656e4
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.