René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Orca integration for static image export","articleBody":"## Overview\r\nThis is design proposal for the integration of orca into plotly.py in order to support the programmatic export of high quality static images.\r\n\r\n## Related issues:\r\n - https://github.com/plotly/plotly.py/issues/880\r\n - https://github.com/plotly/plotly.py/issues/596\r\n - https://github.com/plotly/plotly.py/issues/564\r\n\r\n## Background\r\nThe programmatic export of static raster and vector images from JavaScript-based data visualization libraries is a notoriously complicated problem.  One common solution is to combine selenium with a driver for a headless web browser like phantomjs or headless firefox/chrome. This approach is used by [Bokeh](https://bokeh.pydata.org/en/latest/docs/user_guide/export.html) and [Altair](https://altair-viz.github.io/user_guide/saving_charts.html) for example.  One challenge with this approach is that it requires the installation of dependencies that are not managed by a Python environment friendly package manager like `conda` (Although [phantomjs](https://github.com/ariya/phantomjs) is available through conda, its development has been suspended and it does not support WebGL).  This presents challenges in terms of portability and reproducibility.\r\n\r\nThe plotly.js team has taken a different approach with the [Orca project](https://github.com/plotly/orca). Orca is a standalone Electron application that can run as a command line image export tool, or it can run in a server mode and respond to image export requests interactively.  Orca is the backbone of the plot.ly image export service, and it was open sourced earlier this year.\r\n\r\nBecause Orca can be built into a standalone executable that does not depend on a system web browser, it is possible to package Orca as a conda package, and we've had [recent success](https://github.com/plotly/orca/pull/113) towards this goal.\r\n\r\nThis issue is for the discussion of how to build the best plotly.py image export experience on top of Orca.\r\n\r\n## Goals\r\n - Users shouldn't need to be aware of how complicated static image export is. At most it should require a single additional conda installation command.\r\n - It should be as easy and reliable to use as matplotlib's image export.\r\n - Nothing should flash on the screen or dock or taskbar during export.\r\n - For raster formats, it should support png, jpg, and webp with configurable resolution.\r\n - For vector formats it should support svg, pdf, and eps.\r\n - It should be possible to save images directly to the local filesystem, or to a writable file object.\r\n - It should be possible to return a byte string containing the image data without specifying filenames (and ideally without actually writing anything to temp files).\r\n - It should be fast enough to support use as an interactive plotting backend (See #1098)\r\n - It should provide really helpful error messages if the orca executable isn't found.\r\n\r\n## Potential Approaches\r\n\r\n### 1. Use command-line interface with figure as arg\r\nThe current Python instructions in the [Orca README](https://github.com/plotly/orca) suggest the following usage:\r\n\r\n```python\r\nfrom subprocess import call\r\nimport json\r\nimport plotly\r\n\r\nfig = {\"data\": [{\"y\": [1,2,1]}]}\r\ncall(['orca', 'graph', json.dumps(fig, cls=plotly.utils.PlotlyJSONEncoder)])\r\n```\r\n\r\nHere the figure is serialized to a JSON string and passed as a command line argument to orca.  This is nice because it avoids the need to create a temporary file.  Unfortunately, there's a limit to how large the command line arguments can be, and large figures cross that boundary, resulting in an exception.\r\n\r\n### 2. Use command-line interface with figure as tmp file\r\n\r\nAn alternative that doesn't run into this scaling problem is to first write the figure to a temporary file and then call orca with the path to the file.  Furthermore, if a collection of figures needs to converted at once, the paths can all be passed to orca at once and orca will convert them in a batch mode.  This is much faster on average because the orca executable only has to start up and shut down once per batch, rather than once per figure.\r\n\r\n### 3. Use orca in server mode\r\n\r\nAnother approach would be to launch orca as a subprocess in server mode.  The Python library would send individual image export requests to the server on an agreed upon port. The server would respond with the byte string of the converted image.  This approach has several advantages, but also some increased complexity.\r\n\r\n#### 3.1 Advantages\r\n**Response time:** Launch orca as a command line program or as a server process takes roughly 2 seconds to complete.  However requests to an already running server process are much faster. I've seen round trip request to response times of under 50ms.  2 seconds is acceptable in the context of exporting figures to images on the filesystem, but it is not acceptable for interactive use as a static backend.  50ms feels as fast as matplotlib.\r\n\r\n**No temp files:** This approach doesn't involve the use of any temporary files, and it makes it much simpler to support the non-file image use cases, like returning a bytes string or `PIL.Image.Image` object to the user.\r\n\r\n\r\n#### 3.2 Complications\r\nThere are some additional complications to this approach.  First, the long runner server process would need to be managed by the Python library.  It's too resource intensive to run all the time by default, so the user would need to start it explicitly, or we would need to start it the first time an export is requested.\r\n\r\nThen there's the question of whether we leave the server process running indefinitely. Or do we implement some kind of timeout that would shut the process down after a (configurable) period of inactivity?\r\n\r\nFinally, the communication between the Python process and the server requires an open local port, so there's the potential for restrictive firewalls to be a problem. (But, on the other hand, this is also true of the Jupyter Notebook and most applications that interact with an ipython kernel.)\r\n\r\n## What's next\r\nNext we're going to work on testing and releasing conda packages for orca version 1.1.0. \r\n\r\nMethod 2 above (temp files) is probably the least risky approach, but I really want the advantages that come with Method 3 (server process), so I'd like to give this a shot first. I've already developed a prototype of the server mode approach, with automatic startup and timeout shutdown, and I have it working on OS X, Linux, and Windows.  So far I've found it to be very reliable, and the responsiveness is really exciting. \r\n\r\nSo, I'm quite hopeful that we'll be able to build a solid user experience on top of the server mode. But I would like hear some other perspectives here. \r\n\r\n@chriddyp @jackparmer @cldougl @nicolaskruchten @etpinard @Kully ","author":{"url":"https://github.com/jonmmease","@type":"Person","name":"jonmmease"},"datePublished":"2018-08-10T14:51:17.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":7},"url":"https://github.com/1105/plotly.py/issues/1105"}