René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"SciJava Location API Improvements","articleBody":"During a conversation with @ctrueden we created the following plan for further improvements of the scijava-location API:\r\n\r\n## Abstraction structure:\r\n\r\n1. __Location:__\r\n    Contains the metadata, where a file is located, what credentials are needed for access, etc.\r\n2. __Session:__\r\n    Provides access to data sources on a host.\r\n3. __DataHandle:__\r\n   Provides access to the bytes in a data source, is used e.g. by scifio formats to read images.\r\n\r\n## SessionService\r\n- can create a Session for a remote, takes Location as input\r\n    - \"remote\" = the host/non-path part of a location URI)\r\n- caches Sessions for each remote\r\n- whenever a Session is handed out, increment a usage ref\r\n- whenever a DataHandle is closed, it decrements its session's usage ref\r\n    - if applicable -- some DataHandles don't need Sessions\r\n\r\n## Protocols\r\n- List of protocols we want:\r\n   - [ ] SSH SCP\r\n   - [ ] (S)FTP\r\n   - [ ] HTTP/HTTPS:\r\n       - with resume, to avoid excessive rereading\r\n       - support PUT for uploads?\r\n   - [ ] OMERO\r\n   - [ ] HDFS\r\n   - [ ] Cloud block storage:\r\n      - [ ]  Amazon S3\r\n      - [ ] Openstack Swift\r\n      - [ ] Azure Block Storage\r\n      - [ ] Google Cloud Storage\r\n\r\n## Service/plugin architecture\r\n- Session will need to be a new Plugin type\r\n- SessionService will be a HandlerService\r\n    - Model it after DataHandleService; it is very similar\r\n    but with additional API, obviously -- e.g., fetchOrCreate or some such\r\n- Update the DataHandleService to have a new method:\r\n    - `create(L extends Location, Session\u003cL\u003e)`\r\n    - This method allows reusing a specific session.\r\n    - This session may not be closed when the handle is closed \r\n- The existing DataHandleService method `create(Location)` will simply ask the SessionService to `fetchOrCreate` a session for that Location.\r\n- Naively, it might seem like we need a \"Remote\" interface or some such, but I\r\n  actually think we won't need it. I think each Session will be able to extract\r\n  the information it needs from its associated type of Location, and that will be\r\n  good enough, and simpler.\r\n- Finish the StreamHandle interface(?) for DataHandles that are built on InputStream/OutputStreams.\r\n    - Some location types like URLHandle would probably(?) benefit from extending StreamHandle\r\n    - But if none of the protocols would actually benefit from extending StreamHandle, we could decide not to do it.\r\n\r\n### Protocol specific notes\r\n\r\n- Native HTTP / HTTPS support in Java seems to lack support for resume, looks like we need an external library for that.\r\n- We need to make sure that we only cache sessions that support concurrent access. -\u003e Need method `public boolean isConcurent()` in interface.\r\n- Sessions might only allow for limited concurrent access, if we encounter this we will need a `public int concurentAccessLimit()` method.\r\n- When do we close automatically created sessions?\r\n  - Need a cache eviction strategy like LRU\r\n  - Don't immediately close Session when reference count reaches 0\r\n  - Maybe use a connection pool with a modifiable limit?\r\n\r\n\r\n### Implementation specific notes\r\n- calling `dispose()` on the SessionService will close all connections.\r\n- Currently working on this on the ![more-handles-gabriel branch](https://github.com/scijava/scijava-common/commits/more-handles-gabriel)\r\n","author":{"url":"https://github.com/gab1one","@type":"Person","name":"gab1one"},"datePublished":"2016-12-15T12:11:25.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":2},"url":"https://github.com/251/scijava-common/issues/251"}