René's URL Explorer Experiment


Title: read_hdf is much slower than it should be · Issue #563 · larray-project/larray · GitHub

Open Graph Title: read_hdf is much slower than it should be · Issue #563 · larray-project/larray

X Title: read_hdf is much slower than it should be · Issue #563 · larray-project/larray

Description: >>> from larray_eurostat import * >>> arr = eurostat_get('migr_pop3ctb') >>> arr.to_hdf('test_arr.h5', 'arr') >>> import pandas as pd >>> %timeit pd.read_hdf('test_arr.h5', 'arr') 114 ms ± 8.29 ms per loop (mean ± std. dev. of 7 runs, 10...

Open Graph Description: >>> from larray_eurostat import * >>> arr = eurostat_get('migr_pop3ctb') >>> arr.to_hdf('test_arr.h5', 'arr') >>> import pandas as pd >>> %timeit pd.read_hdf('test_arr.h5', 'arr') 114 ms ± 8.29 ms ...

X Description: >>> from larray_eurostat import * >>> arr = eurostat_get('migr_pop3ctb') >>> arr.to_hdf('test_arr.h5', 'arr') >>> import pandas as pd >...

Opengraph URL: https://github.com/larray-project/larray/issues/563

X: @github

direct link

Domain: github.com


Hey, it has json ld scripts:
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"read_hdf is much slower than it should be","articleBody":"```python\r\n>>> from larray_eurostat import *\r\n>>> arr = eurostat_get('migr_pop3ctb')\r\n>>> arr.to_hdf('test_arr.h5', 'arr')\r\n>>> import pandas as pd\r\n>>> %timeit pd.read_hdf('test_arr.h5', 'arr')\r\n114 ms ± 8.29 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)\r\n>>> %timeit read_hdf('test_arr.h5', 'arr')\r\n1.24 s ± 35.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\r\n```\r\nThe problem is mostly in inout.array.cartesian_product_df which is very expensive while it should do nothing for .h5 arrays saved from an LArray (it was already \"densified\").\r\n\r\n```python\r\n>>> from larray.inout.array import cartesian_product_df\r\n>>> df = pd.read_hdf('test_arr.h5', 'arr')\r\n>>> %timeit cartesian_product_df(df)\r\n968 ms ± 20.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\r\n\r\n>>> from larray.inout.array import df_labels\r\n>>> %timeit labels = df_labels(df, False)\r\n288 ms ± 3.85 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\r\n>>> labels = df_labels(df, False)\r\n>>> from itertools import product\r\n>>> %timeit new_index = pd.MultiIndex.from_tuples(list(product(*labels)))\r\n440 ms ± 3.83 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\r\n>>> new_index = pd.MultiIndex.from_tuples(list(product(*labels)))\r\n>>> columns = list(df.columns)\r\n>>> %timeit np.array_equal(df.index.values, new_index.values)\r\n23.3 ms ± 638 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)\r\n```\r\nI think I had already fixed this (or at least partially) in the very old pandasbased3 branch.","author":{"url":"https://github.com/gdementen","@type":"Person","name":"gdementen"},"datePublished":"2018-01-03T12:01:48Z","url":"https://github.com/larray-project/larray/issues/563"}

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format)
route-controllervoltron_issues_fragments
route-actionissue_layout
fetch-noncev2:7f21d975-c0c1-4546-6238-6e9531701113
current-catalog-service-hash81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114
request-idB2E2:3CBF1A:1F7E4E1:2B19214:6B0023B6
html-safe-nonce7baf513e722387772028745d324c7864284774376bcabd49fd8b634bf266e9ed
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJCMkUyOjNDQkYxQToxRjdFNEUxOjJCMTkyMTQ6NkIwMDIzQjYiLCJ2aXNpdG9yX2lkIjoiNzg3MzYyNjQxNjM4OTAzOTAzMCIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9
visitor-hmac420b87e98bee901a526df32c054c4aca1c42453cec25e289ac05fd3c8b7e787d
hovercard-subject-tagissue:285667996
github-keyboard-shortcutsrepository,issues,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///voltron/issues_fragments/issue_layout
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/larray-project/larray/563/issue_layout
twitter:imagehttps://opengraph.githubassets.com/97167aae8f8e881f06dfd9bd847fcb1e1236dd23cfa84b534da6a3e6e2840879/larray-project/larray/issues/563
twitter:cardsummary_large_image
og:imagehttps://opengraph.githubassets.com/97167aae8f8e881f06dfd9bd847fcb1e1236dd23cfa84b534da6a3e6e2840879/larray-project/larray/issues/563
og:image:alt>>> from larray_eurostat import * >>> arr = eurostat_get('migr_pop3ctb') >>> arr.to_hdf('test_arr.h5', 'arr') >>> import pandas as pd >>> %timeit pd.read_hdf('test_arr.h5', 'arr') 114 ms ± 8.29 ms ...
og:image:width1200
og:image:height600
og:site_nameGitHub
og:typeobject
og:author:usernamegdementen
hostnamegithub.com
expected-hostnamegithub.com
None4eb4cba0d69f045374ac15de9666d22b0961e1d555cc3824f36b7734c00c78e0
turbo-cache-controlno-preview
go-importgithub.com/larray-project/larray git https://github.com/larray-project/larray.git
octolytics-dimension-user_id26460981
octolytics-dimension-user_loginlarray-project
octolytics-dimension-repository_id16981067
octolytics-dimension-repository_nwolarray-project/larray
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id16981067
octolytics-dimension-repository_network_root_nwolarray-project/larray
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release44c658b24abf00491fb7ae5f56d42c9f38cbeab2
ui-targetfull
theme-color#1e2327
color-schemelight dark
react-app-nameissues

Links:

Skip to contenthttps://github.com/larray-project/larray/issues/563#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Flarray-project%2Flarray%2Fissues%2F563
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Flarray-project%2Flarray%2Fissues%2F563
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fvoltron%2Fissues_fragments%2Fissue_layout&source=header-repo&source_repo=larray-project%2Flarray
Reloadhttps://github.com/larray-project/larray/issues/563
Reloadhttps://github.com/larray-project/larray/issues/563
Reloadhttps://github.com/larray-project/larray/issues/563
larray-projecthttps://github.com/larray-project
larrayhttps://github.com/larray-project/larray
Codehttps://github.com/larray-project/larray
Issues279 (279)https://github.com/larray-project/larray/issues
Pull requests6 (6)https://github.com/larray-project/larray/pulls
Actionshttps://github.com/larray-project/larray/actions
Security and qualityhttps://github.com/larray-project/larray/security
Insightshttps://github.com/larray-project/larray/insights
https://github.com/larray-project/larray
New issuehttps://github.com/login?return_to=https://github.com/larray-project/larray/issues/563
New issuehttps://github.com/login?return_to=https://github.com/larray-project/larray/issues/563
read_hdf is much slower than it should behttps://github.com/larray-project/larray/issues/563#top
https://github.com/alixdamman
bughttps://github.com/larray-project/larray/issues?q=state%3Aopen%20label%3A%22bug%22
performancehttps://github.com/larray-project/larray/issues?q=state%3Aopen%20label%3A%22performance%22
priority: highhttps://github.com/larray-project/larray/issues?q=state%3Aopen%20label%3A%22priority%3A%20high%22
0.30https://github.com/larray-project/larray/milestone/15
https://github.com/gdementen
gdementenhttps://github.com/gdementen
on Jan 3, 2018https://github.com/larray-project/larray/issues/563#issue-285667996
gdementenhttps://github.com/gdementen
bughttps://github.com/larray-project/larray/issues?q=state%3Aopen%20label%3A%22bug%22
performancehttps://github.com/larray-project/larray/issues?q=state%3Aopen%20label%3A%22performance%22
on Jan 3, 2018https://github.com/larray-project/larray/issues/563#event-1407667492
alixdammanhttps://github.com/alixdamman
0.30https://github.com/larray-project/larray/milestone/15
on Mar 7, 2018https://github.com/larray-project/larray/issues/563#event-1508460314
alixdammanhttps://github.com/alixdamman
0.30https://github.com/larray-project/larray/milestone/15
0.31https://github.com/larray-project/larray/milestone/18
on Jul 18, 2018https://github.com/larray-project/larray/issues/563#event-1739415539
alixdammanhttps://github.com/alixdamman
0.31https://github.com/larray-project/larray/milestone/18
0.30https://github.com/larray-project/larray/milestone/15
on Sep 6, 2018https://github.com/larray-project/larray/issues/563#event-1829633762
alixdammanhttps://github.com/alixdamman
priority: highhttps://github.com/larray-project/larray/issues?q=state%3Aopen%20label%3A%22priority%3A%20high%22
on Sep 6, 2018https://github.com/larray-project/larray/issues/563#event-1829634099
https://github.com/gdementen
gdementenhttps://github.com/gdementen
on Nov 27, 2018https://github.com/larray-project/larray/issues/563#issuecomment-441977000
https://github.com/alixdamman
alixdammanhttps://github.com/alixdamman
on Nov 27, 2018https://github.com/larray-project/larray/issues/563#issuecomment-441987328
gdementenhttps://github.com/gdementen
0.30https://github.com/larray-project/larray/milestone/15
0.31https://github.com/larray-project/larray/milestone/18
on Jan 29, 2019https://github.com/larray-project/larray/issues/563#event-2103870950
alixdammanhttps://github.com/alixdamman
work in progresshttps://github.com/larray-project/larray/issues?q=state%3Aopen%20label%3A%22work%20in%20progress%22
on Apr 9, 2019https://github.com/larray-project/larray/issues/563#event-2262764317
alixdammanhttps://github.com/alixdamman
on Apr 25, 2019https://github.com/larray-project/larray/issues/563#event-2299297799
alixdammanhttps://github.com/alixdamman
on May 23, 2019https://github.com/larray-project/larray/issues/563#event-2362509812
fix larray-project#563 : improved speed of read_hdf() when stored obj…https://github.com/alixdamman/larray/commit/22fc174c8a741b490c93fb4854f4b562ea835740
larray-project#563https://github.com/larray-project/larray/issues/563
https://docs.github.com/articles/closing-issues-via-commit-messages
22fc174https://github.com/alixdamman/larray/commit/22fc174c8a741b490c93fb4854f4b562ea835740
alixdammanhttps://github.com/alixdamman
completedhttps://github.com/larray-project/larray/issues?q=is%3Aissue%20state%3Aclosed%20archived%3Afalse%20reason%3Acompleted
05573eahttps://github.com/larray-project/larray/commit/05573eadaa5dee1fc7c1a32fec059aed62db2bc2
on May 23, 2019https://github.com/larray-project/larray/issues/563#event-2362525754
gdementenhttps://github.com/gdementen
work in progresshttps://github.com/larray-project/larray/issues?q=state%3Aopen%20label%3A%22work%20in%20progress%22
on Aug 1, 2019https://github.com/larray-project/larray/issues/563#event-2526092590
gdementenhttps://github.com/gdementen
0.31https://github.com/larray-project/larray/milestone/18
0.30https://github.com/larray-project/larray/milestone/15
on Aug 1, 2019https://github.com/larray-project/larray/issues/563#event-2526094013
Sign up for freehttps://github.com/signup?return_to=https://github.com/larray-project/larray/issues/563
Sign in to commenthttps://github.com/login?return_to=https://github.com/larray-project/larray/issues/563
alixdammanhttps://github.com/alixdamman
bughttps://github.com/larray-project/larray/issues?q=state%3Aopen%20label%3A%22bug%22
performancehttps://github.com/larray-project/larray/issues?q=state%3Aopen%20label%3A%22performance%22
priority: highhttps://github.com/larray-project/larray/issues?q=state%3Aopen%20label%3A%22priority%3A%20high%22
0.30https://github.com/larray-project/larray/milestone/15
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.