Title: Thinking about improving data updates · Issue #647 · plotly/plotly.js · GitHub
Open Graph Title: Thinking about improving data updates · Issue #647 · plotly/plotly.js
X Title: Thinking about improving data updates · Issue #647 · plotly/plotly.js
Description: tl;dr There's more and more code that couples the aspect of plotting logic with the aspect of incrementally propagating changes, e.g. see all things going on in Plotly.restyle. Would be good to discuss ways to improve on the situation. M...
Open Graph Description: tl;dr There's more and more code that couples the aspect of plotting logic with the aspect of incrementally propagating changes, e.g. see all things going on in Plotly.restyle. Would be good to dis...
X Description: tl;dr There's more and more code that couples the aspect of plotting logic with the aspect of incrementally propagating changes, e.g. see all things going on in Plotly.restyle. Would be good to...
Opengraph URL: https://github.com/plotly/plotly.js/issues/647
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Thinking about improving data updates","articleBody":"**tl;dr**\nThere's more and more code that couples the aspect of plotting logic with the aspect of incrementally propagating changes, e.g. see all things going on in `Plotly.restyle`. Would be good to discuss ways to improve on the situation. Manual code leads to a tangle and some small, simple library focused on change propagation e.g. [MobX](https://docs.google.com/presentation/d/1tP0VWjproXsZex3R8zWCA7Bra7RtMcu4daJXCUbs8pc/edit#slide=id.g1102c17e6b_0_31) would be worth looking into.\n\n**Plotting turns a stream of user intent into a stream of side effects such as DOM updates**\n\nPlotting can be conceived of as a black box:\n- input streams are plot specifications, typically the payloads in `Plotly.plot`, `Plotly.restyle`, `Plotly.relayout`, animation inducing user calls as well as DOM events such as `window.resize` and `mousedown`\n- output is a stream of side effecting operations, e.g. DOM mutations, WebGL API calls, and sometimes event callbacks\n- currently, some output is provided by encouraging users to read directly from internal object state but it's something to move away from, by providing a query API and/or event callbacks with meaningful data, so I'll ignore this\n\nThe use of 'stream' highlights the fact that with user pointer operations, restyle/relayout, animation etc. _generally_ make plotting a temporal process, rather than something that can be modeled with a function with some input JSON and an output SVG - even if some of the uses are as simple as this special case.\n\n**Plotting logic is a directed acyclic graph of computation nodes**\n\nWe have multiple pieces of input (e.g. `data[0].x`) at the input and DOM mutating calls as the output. However there's complex calculation in the middle that can be thought of as a DAG. For example, \n- the above `x` vector serves as the basis for calculating a `[min, max]` domain that will determine the bounds of the X axis\n- the `x` vector is also trivially input to scatter point positions, however, a `scale` transform converts domain values to e.g. pixel coordinates\n- for things like the boxplot, there may be various aggregations building atop of the `x` vector\n- aesthetics might depend on things like how long the `x` vector is; maybe defaulting from scatterplot to a density plot at some threshold\n\nAll such calculations themselves can be input to downstream calculations.\n\n**Plotting needs to be economical**\n\nWhile it would be possible to make a single function whose inputs are `{domRoot, userIntentHistory}`, it's impractical: response times with a naive implementation would be too high (keeping `userIntentHistory is merely of modest size impact). There's no way to recompute everything from scratch and expect a 60FPS frame rate when turning a WebGL plot or animating something.\n\nThis means that there needs to be some kind of caching, therefore state management. The sole purpose of maintaining state is caching (besides this, we may retain `userIntentHistory` to allow time travel, and of course the output streams are linked to calls that modify the DOM).\n\n**Means of reducing recomputation costs**\n\nIdeally we'd like to \n- Only recompute what's strictly needed. For example, if I add a new highest value to vector `x` it needs to lead to an increased visible X axis domain, provided it's set to automatic. However, if the newly inserted value is inside the bounds, there's no need to recalculate anything that depends only on the `[min, max]` domain. Sure, sometimes there's no harm due to speed of recalculation or lack of need for speed, but there _are_ cases when it's useful to be fairly granular about recalculations due to some specific performance need. Solving these specific performance needs one by one, without a formal change propagation approach is brittle.\n- It may even be useful, necessary and easy to pick calculation algorithms to be incremental. For example, a newly arriving X value can be directly used to update the `[min, max]` bounds, as opposed to inserting it in the preexisting large vector and applying the vector `extent` calculation. Similarly, many types of aggregates can be calculated _on-line_ as well as _batch_. For example, mean, variance and standard deviation.\n\n**Some possible tools**\n\nHandwritten userland JavaScript isn't quite good for managing a dependency graph, because given enough nodes and optimization rounds, there will be inevitable cache invalidation issues, and potentially, memory leaks. Keeping things consistent and in in sync is also a challenge especially in the presence of asynchronous events. Most importantly, coupling the plot logic aspect with the incremental recalculation aspect makes both aspects hard to decipher, debug and further develop.\n\nThere are a lot of tools that provide some kind of framework for calculating and propagating values that can change over time, responding to input, inspired by Functional Reactive Programming. Without endorsing any of these excellent libraries ([xstream](https://github.com/staltz/xstream), [most.js](https://github.com/cujojs/most) etc.) perhaps [MobX](https://github.com/mobxjs/mobx) would feel closest to the current architecture in that it gives you objects that have properties acting like calculated spreadsheet cells, and as @etpinard suggested in the 2.0 wishlist, object-oriented, but investigation would be needed to see how it fits. All these libs are around 10k compressed.\n\n**History**\n\nWe've touched on related topics in the past; a few inspirations:\n- Wishlist suggestions such as using more OO; not storing data in the DOM; using pure functions; more complete data in callbacks: https://github.com/plotly/plotly.js/issues/420\n- Customer filed PRs for faster (re)calculation and way faster incremental calculation, e.g. on large WebGL meshes\n- Most of the Plotly.restyle function, whose 500 lines do heavy amounts of manual work, e.g. https://github.com/plotly/plotly.js/blob/master/src/plot_api/plot_api.js#L1736-L1814 and https://github.com/plotly/plotly.js/pull/617#discussion_r67130816\n- Previous discussions e.g. on the animation topic\n","author":{"url":"https://github.com/monfera","@type":"Person","name":"monfera"},"datePublished":"2016-06-15T11:39:19Z","url":"https://github.com/plotly/plotly.js/issues/647"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:32b7ce73-8090-8139-bee7-21e3e5760e42 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | 8C8A:2D40DE:1C0ABC4:23A9285:6B019728 |
| html-safe-nonce | 742288643e39521a2b0a8c7ae1962479cb077f8fe724538ef080f166f22c8647 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiI4QzhBOjJENDBERToxQzBBQkM0OjIzQTkyODU6NkIwMTk3MjgiLCJ2aXNpdG9yX2lkIjoiMTI3OTc5MTc4MTI4Njc2NjQ4IiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | 6390cf1e688cac6f8ca14b6632dd7b0db8b2b73a0b4167cedbff8c81c075bd0b |
| hovercard-subject-tag | issue:160401302 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/plotly/plotly.js/647/issue_layout |
| twitter:image | https://opengraph.githubassets.com/f8926cf609a7bd64895cec2012d5e4f4fdef0478237274d13b5cd2e2f1f738be/plotly/plotly.js/issues/647 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/f8926cf609a7bd64895cec2012d5e4f4fdef0478237274d13b5cd2e2f1f738be/plotly/plotly.js/issues/647 |
| og:image:alt | tl;dr There's more and more code that couples the aspect of plotting logic with the aspect of incrementally propagating changes, e.g. see all things going on in Plotly.restyle. Would be good to dis... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | monfera |
| hostname | github.com |
| expected-hostname | github.com |
| None | cc5dfdd32f62ebd3223cab3af3aa2b268d3605218237326a48a99d1efd89495c |
| turbo-cache-control | no-preview |
| go-import | github.com/plotly/plotly.js git https://github.com/plotly/plotly.js.git |
| octolytics-dimension-user_id | 5997976 |
| octolytics-dimension-user_login | plotly |
| octolytics-dimension-repository_id | 45646037 |
| octolytics-dimension-repository_nwo | plotly/plotly.js |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 45646037 |
| octolytics-dimension-repository_network_root_nwo | plotly/plotly.js |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 8c77c9e8439031ad54c75a250694842892f693e1 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width