René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"File attachments","articleBody":"# Purpose\r\n\r\nEnable storage of entire files intact, without having to read and serialize them, similar to attachments.  Enable the use of these attachments through GUI drag-and-drop features.  The attached files are not browsable outside DataJoint.  DataJoint must maintain data integrity.\r\n\r\n# Operation\r\n\r\nAttachments use a mechanism similar to external storage using file hashes as the filenames in a storage repository.   The datatype should be either attach or attach-suffix where the suffix can be up to eight character long.  The storage protocol and location (filesystem, S3, globus) is specified similar to external storage. \r\n\r\nThe table `~attach` is created and maintained similar to how `~external` is used for external storage.  The only difference is that it also stores the original filename of the stored file.\r\n\r\nJust as with external, this mechanism ensures that inserting the same file multiple times within the same schema does not result in duplicated storage.\r\n\r\n## Insert \r\nInsert works by inserting the full path to the file to be inserted.  The insert is asynchronous and commits the insert into the table into ~attach.\r\nIn its initial implementation, the entire file will be first read into memory for hashing and writing into storage. \r\n\r\n## Fetch\r\nFetch downloads the attached files.  The download will be implemented asynchronously in the initial implementation.  While the file is being downloaded, it is given the extension .download.   When the download is complete the file is renamed into its target name.\r\n\r\nThe destination folder can be specified in the configuration.  That can be overridden by the optional argument download_path of fetch.  If neither is identified the download is performed into the current directory. \r\n\r\nIn all cases, the full path to the downloaded file is returned in the fetch output as attribute values.\r\n\r\nSimilar to external storage, fetch can be configured to use local caching. \r\n\r\nIf the same file retrieved multiple time as part of the same fetch, the file is only retrieved once and the same path in returned for each requested attribute value.\r\n\r\nIf the file already exists, the downloaded file is given an incrementing suffix.\r\n\r\n## Delete\r\n\r\nSimilar to external, delete only operates on the host tables and does not remove entries from ~attach or from the file repository.\r\n\r\nCleanup of ~attach and of storage folder work identically to the corresponding mechanisms of external storage.\r\n\r\nCleanup commands:\r\n```python\r\nschema.attachment.delete_garbage()   # transaction-safe\r\nschema.attachment.cleanup_store()   # not transaction-safe\r\n```\r\n","author":{"url":"https://github.com/dimitri-yatsenko","@type":"Person","name":"dimitri-yatsenko"},"datePublished":"2018-08-06T18:20:05.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":2},"url":"https://github.com/480/datajoint-python/issues/480"}