René's URL Explorer Experiment


Title: rethink our recommended storage formats · Issue #1191 · larray-project/larray · GitHub

Open Graph Title: rethink our recommended storage formats · Issue #1191 · larray-project/larray

X Title: rethink our recommended storage formats · Issue #1191 · larray-project/larray

Description: Given the mess and ticking time bomb that H5 files via Pandas is (see #1178, #1184 and #1190), we should avoid them. We should define one (or several) storage format(s) for the various uses cases we have: data exchange with other institu...

Open Graph Description: Given the mess and ticking time bomb that H5 files via Pandas is (see #1178, #1184 and #1190), we should avoid them. We should define one (or several) storage format(s) for the various uses cases w...

X Description: Given the mess and ticking time bomb that H5 files via Pandas is (see #1178, #1184 and #1190), we should avoid them. We should define one (or several) storage format(s) for the various uses cases w...

Opengraph URL: https://github.com/larray-project/larray/issues/1191

X: @github

direct link

Domain: github.com


Hey, it has json ld scripts:
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"rethink our recommended storage formats","articleBody":"Given the mess and ticking time bomb that H5 files via Pandas is (see #1178, #1184 and #1190), we should avoid them.\n\nWe should define one (or several) storage format(s) for the various uses cases we have:\n  - data exchange with other institutions (will probably use Parquet and Excel depending on the case)\n  - data exchange between teams\n  - model output (should stay valid in the long term)\n  - caching\n\nA few thoughts:   \n  - I currently think the best option is to keep H5 for output/model storage but use h5py instead of pytables to avoid the pickles but...\n  - the first step is to do a speed test for arrays (only) of various sizes and types between H5 (via pytables and via h5py), feather and parquet\n    to see if feather is faster and if the difference is enough to be worth going non-standard for other use cases than caching\n  - for caching the tradeoffs are different because no need for long term storage and only full arrays.\n    There, everything is fair game: feather/laf or msgpack or flatbuffers (but I would rather avoid pickles).\n  - could finish and recommend .laf but non-Array objects will make it a big project anyway (metadata, object arrays, mixed type axis labels, \n    sessions with axis, scalars and groups, etc.)\nFWIW, msgpack seems promising *but* the 2 Python implementations I have seen only support loading the message entirely.\n\nI have some more thoughts in a .txt file on my computer I should copy here 😉.\n\nI put this in the 0.36 milestone but the goal is not to implement everything by then but at least know where we are going so that we can then decide how and when to get there.","author":{"url":"https://github.com/gdementen","@type":"Person","name":"gdementen"},"datePublished":"2026-04-09T10:50:43Z","url":"https://github.com/larray-project/larray/issues/1191"}

react-profiling0
react-import-mapreact,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322
route-pattern/_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format)
route-controllervoltron_issues_fragments
route-actionissue_layout
fetch-noncev2:da2b7dba-259e-146a-5fe4-35efa91b8ee5
current-catalog-service-hash81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114
request-id91BC:3A7693:21BB359:2DBA0BD:6B002121
html-safe-nonceeda3de64216a9dee37ebcaea235704121e758de31ad45e0938006aca74810623
visitor-payloadeyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI5MUJDOjNBNzY5MzoyMUJCMzU5OjJEQkEwQkQ6NkIwMDIxMjEiLCJ2aXNpdG9yX2lkIjoiNTE4MDU1MTY4ODc3NTM1NTIxIiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0=
visitor-hmacd29db15fe3bb287bb26d6c50a1d32d50e4a958bb164bc362362554e0a318b626
hovercard-subject-tagissue:4231563859
github-keyboard-shortcutsrepository,issues,copilot
google-site-verificationApib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I
octolytics-urlhttps://collector.github.com/github/collect
analytics-location///voltron/issues_fragments/issue_layout
fb:app_id1401488693436528
apple-itunes-appapp-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/larray-project/larray/1191/issue_layout
twitter:imagehttps://opengraph.githubassets.com/0af2fc9bdb9b6df333e851f5d1db2aa20b1196abea499b847e1e5046c7da1344/larray-project/larray/issues/1191
twitter:cardsummary_large_image
og:imagehttps://opengraph.githubassets.com/0af2fc9bdb9b6df333e851f5d1db2aa20b1196abea499b847e1e5046c7da1344/larray-project/larray/issues/1191
og:image:altGiven the mess and ticking time bomb that H5 files via Pandas is (see #1178, #1184 and #1190), we should avoid them. We should define one (or several) storage format(s) for the various uses cases w...
og:image:width1200
og:image:height600
og:site_nameGitHub
og:typeobject
og:author:usernamegdementen
hostnamegithub.com
expected-hostnamegithub.com
None4eb4cba0d69f045374ac15de9666d22b0961e1d555cc3824f36b7734c00c78e0
turbo-cache-controlno-preview
go-importgithub.com/larray-project/larray git https://github.com/larray-project/larray.git
octolytics-dimension-user_id26460981
octolytics-dimension-user_loginlarray-project
octolytics-dimension-repository_id16981067
octolytics-dimension-repository_nwolarray-project/larray
octolytics-dimension-repository_publictrue
octolytics-dimension-repository_is_forkfalse
octolytics-dimension-repository_network_root_id16981067
octolytics-dimension-repository_network_root_nwolarray-project/larray
turbo-body-classeslogged-out env-production page-responsive
disable-turbofalse
browser-stats-urlhttps://api.github.com/_private/browser/stats
browser-errors-urlhttps://api.github.com/_private/browser/errors
release44c658b24abf00491fb7ae5f56d42c9f38cbeab2
ui-targetfull
theme-color#1e2327
color-schemelight dark
react-app-nameissues

Links:

Skip to contenthttps://github.com/larray-project/larray/issues/1191#start-of-content
https://github.com/
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Flarray-project%2Flarray%2Fissues%2F1191
GitHubRepository hosting, code review, and collaborationhttps://github.com/platform
CopilotAI coding agent for tasks, projects, and automated workhttps://github.com/features/copilot
ActionsCI/CD and automationhttps://github.com/features/actions
Code QualityQuality analysis, fixes, and enforcementhttps://github.com/features/code-quality
CodespacesCloud development environmentshttps://github.com/features/codespaces
Code SecurityVulnerability detection and remediationhttps://github.com/security/advanced-security/code-security
Secret ProtectionSecret scanning and leak preventionhttps://github.com/security/advanced-security/secret-protection
All Featureshttps://github.com/features
Documentationhttps://docs.github.com/en
Traininghttps://learn.github.com/skills
Certificationshttps://learn.github.com/credentials
Build with Copilothttps://github.com/features/ai
Customize Copilothttps://docs.github.com/en/copilot/how-tos/copilot-on-github/customize-copilot/customize-copilot-overview
Integrationshttps://github.com/marketplace
Trending reposhttps://github.com/trending
GitHub Sponsorshttps://github.com/open-source/sponsors
Developer articleshttps://github.com/resources/articles
ChangelogYour source for the latest features, improvements, and fixes across GitHub products, all in one place.https://github.blog/changelog/
Enterpriseshttps://github.com/enterprise
Teamhttps://github.com/team
Startupshttps://github.com/enterprise/startups
Nonprofitshttps://github.com/solutions/industry/nonprofits
Healthcarehttps://github.com/solutions/industry/healthcare
Financial serviceshttps://github.com/solutions/industry/financial-services
Manufacturinghttps://github.com/solutions/industry/manufacturing
Governmenthttps://github.com/solutions/industry/government
See all industrieshttps://github.com/solutions/industry
Why GitHubhttps://github.com/why-github
Customer Storieshttps://github.com/customer-stories
Business insightshttps://github.com/solutions/executive-insights
Trust Centerhttps://github.com/trust-center
GitHub Advanced SecurityEnterprise-grade code securityhttps://github.com/security/advanced-security
Copilot for businessEnterprise-grade AI featureshttps://github.com/features/copilot/copilot-business
Premium SupportEnterprise-grade 24/7 supporthttps://github.com/enterprise/premium-support
Documentationhttps://docs.github.com/en
Resource centerhttps://github.com/resources
Bloghttps://github.blog/
Events & webinarshttps://github.com/resources/events
Community Forumhttps://github.com/orgs/community/discussions
Customer Storieshttps://github.com/customer-stories
Integrationshttps://github.com/marketplace
Customer supporthttps://help.github.com/
Traininghttps://learn.github.com/skills
GitHub Statushttps://www.githubstatus.com/
Abouthttps://github.com/about
Partnershttps://github.com/partners
Changeloghttps://github.blog/changelog/
GitHub UniverseThe event for developers, by developers — talks, workshops, and what’s next for GitHub.https://githubuniverse.com/
Pricinghttps://github.com/pricing
Sign inhttps://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Flarray-project%2Flarray%2Fissues%2F1191
Sign uphttps://github.com/signup?ref_cta=Sign+up&ref_loc=header+logged+out&ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Fvoltron%2Fissues_fragments%2Fissue_layout&source=header-repo&source_repo=larray-project%2Flarray
Reloadhttps://github.com/larray-project/larray/issues/1191
Reloadhttps://github.com/larray-project/larray/issues/1191
Reloadhttps://github.com/larray-project/larray/issues/1191
larray-projecthttps://github.com/larray-project
larrayhttps://github.com/larray-project/larray
Codehttps://github.com/larray-project/larray
Issues279 (279)https://github.com/larray-project/larray/issues
Pull requests6 (6)https://github.com/larray-project/larray/pulls
Actionshttps://github.com/larray-project/larray/actions
Security and qualityhttps://github.com/larray-project/larray/security
Insightshttps://github.com/larray-project/larray/insights
https://github.com/larray-project/larray
New issuehttps://github.com/login?return_to=https://github.com/larray-project/larray/issues/1191
New issuehttps://github.com/login?return_to=https://github.com/larray-project/larray/issues/1191
rethink our recommended storage formatshttps://github.com/larray-project/larray/issues/1191#top
enhancementhttps://github.com/larray-project/larray/issues?q=state%3Aopen%20label%3A%22enhancement%22
priority: very highhttps://github.com/larray-project/larray/issues?q=state%3Aopen%20label%3A%22priority%3A%20very%20high%22
0.36https://github.com/larray-project/larray/milestone/28
https://github.com/gdementen
gdementenhttps://github.com/gdementen
on Apr 9, 2026https://github.com/larray-project/larray/issues/1191#issue-4231563859
#1178https://github.com/larray-project/larray/issues/1178
#1184https://github.com/larray-project/larray/issues/1184
#1190https://github.com/larray-project/larray/issues/1190
gdementenhttps://github.com/gdementen
0.36https://github.com/larray-project/larray/milestone/28
on Apr 9, 2026https://github.com/larray-project/larray/issues/1191#event-24332454032
gdementenhttps://github.com/gdementen
enhancementhttps://github.com/larray-project/larray/issues?q=state%3Aopen%20label%3A%22enhancement%22
priority: very highhttps://github.com/larray-project/larray/issues?q=state%3Aopen%20label%3A%22priority%3A%20very%20high%22
on Apr 9, 2026https://github.com/larray-project/larray/issues/1191#event-24332455104
Sign up for freehttps://github.com/signup?return_to=https://github.com/larray-project/larray/issues/1191
Sign in to commenthttps://github.com/login?return_to=https://github.com/larray-project/larray/issues/1191
enhancementhttps://github.com/larray-project/larray/issues?q=state%3Aopen%20label%3A%22enhancement%22
priority: very highhttps://github.com/larray-project/larray/issues?q=state%3Aopen%20label%3A%22priority%3A%20very%20high%22
0.36No due datehttps://github.com/larray-project/larray/milestone/28
https://github.com
Termshttps://docs.github.com/site-policy/github-terms/github-terms-of-service
Privacyhttps://docs.github.com/site-policy/privacy-policies/github-privacy-statement
Securityhttps://github.com/security
Statushttps://www.githubstatus.com/
Communityhttps://github.community/
Docshttps://docs.github.com/
Contacthttps://support.github.com?tags=dotcom-footer

Viewport: width=device-width


URLs of crawlers that visited me.