Title: rethink our recommended storage formats · Issue #1191 · larray-project/larray · GitHub
Open Graph Title: rethink our recommended storage formats · Issue #1191 · larray-project/larray
X Title: rethink our recommended storage formats · Issue #1191 · larray-project/larray
Description: Given the mess and ticking time bomb that H5 files via Pandas is (see #1178, #1184 and #1190), we should avoid them. We should define one (or several) storage format(s) for the various uses cases we have: data exchange with other institu...
Open Graph Description: Given the mess and ticking time bomb that H5 files via Pandas is (see #1178, #1184 and #1190), we should avoid them. We should define one (or several) storage format(s) for the various uses cases w...
X Description: Given the mess and ticking time bomb that H5 files via Pandas is (see #1178, #1184 and #1190), we should avoid them. We should define one (or several) storage format(s) for the various uses cases w...
Opengraph URL: https://github.com/larray-project/larray/issues/1191
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"rethink our recommended storage formats","articleBody":"Given the mess and ticking time bomb that H5 files via Pandas is (see #1178, #1184 and #1190), we should avoid them.\n\nWe should define one (or several) storage format(s) for the various uses cases we have:\n - data exchange with other institutions (will probably use Parquet and Excel depending on the case)\n - data exchange between teams\n - model output (should stay valid in the long term)\n - caching\n\nA few thoughts: \n - I currently think the best option is to keep H5 for output/model storage but use h5py instead of pytables to avoid the pickles but...\n - the first step is to do a speed test for arrays (only) of various sizes and types between H5 (via pytables and via h5py), feather and parquet\n to see if feather is faster and if the difference is enough to be worth going non-standard for other use cases than caching\n - for caching the tradeoffs are different because no need for long term storage and only full arrays.\n There, everything is fair game: feather/laf or msgpack or flatbuffers (but I would rather avoid pickles).\n - could finish and recommend .laf but non-Array objects will make it a big project anyway (metadata, object arrays, mixed type axis labels, \n sessions with axis, scalars and groups, etc.)\nFWIW, msgpack seems promising *but* the 2 Python implementations I have seen only support loading the message entirely.\n\nI have some more thoughts in a .txt file on my computer I should copy here 😉.\n\nI put this in the 0.36 milestone but the goal is not to implement everything by then but at least know where we are going so that we can then decide how and when to get there.","author":{"url":"https://github.com/gdementen","@type":"Person","name":"gdementen"},"datePublished":"2026-04-09T10:50:43Z","url":"https://github.com/larray-project/larray/issues/1191"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:da2b7dba-259e-146a-5fe4-35efa91b8ee5 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | 91BC:3A7693:21BB359:2DBA0BD:6B002121 |
| html-safe-nonce | eda3de64216a9dee37ebcaea235704121e758de31ad45e0938006aca74810623 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI5MUJDOjNBNzY5MzoyMUJCMzU5OjJEQkEwQkQ6NkIwMDIxMjEiLCJ2aXNpdG9yX2lkIjoiNTE4MDU1MTY4ODc3NTM1NTIxIiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | d29db15fe3bb287bb26d6c50a1d32d50e4a958bb164bc362362554e0a318b626 |
| hovercard-subject-tag | issue:4231563859 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/larray-project/larray/1191/issue_layout |
| twitter:image | https://opengraph.githubassets.com/0af2fc9bdb9b6df333e851f5d1db2aa20b1196abea499b847e1e5046c7da1344/larray-project/larray/issues/1191 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/0af2fc9bdb9b6df333e851f5d1db2aa20b1196abea499b847e1e5046c7da1344/larray-project/larray/issues/1191 |
| og:image:alt | Given the mess and ticking time bomb that H5 files via Pandas is (see #1178, #1184 and #1190), we should avoid them. We should define one (or several) storage format(s) for the various uses cases w... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | gdementen |
| hostname | github.com |
| expected-hostname | github.com |
| None | 4eb4cba0d69f045374ac15de9666d22b0961e1d555cc3824f36b7734c00c78e0 |
| turbo-cache-control | no-preview |
| go-import | github.com/larray-project/larray git https://github.com/larray-project/larray.git |
| octolytics-dimension-user_id | 26460981 |
| octolytics-dimension-user_login | larray-project |
| octolytics-dimension-repository_id | 16981067 |
| octolytics-dimension-repository_nwo | larray-project/larray |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 16981067 |
| octolytics-dimension-repository_network_root_nwo | larray-project/larray |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 44c658b24abf00491fb7ae5f56d42c9f38cbeab2 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width