René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Support allow_reuse in repo","articleBody":"Currently all of the pipeline steps have allow_reuse=False. As a developer, it would be great to enable reuse of steps so that only my changes run.\r\n\r\nThe allow_reuse=True is not working in the repo because of 2 reasons:\r\n\t\r\n1. The repo would need to not pass build_id as a parameter to all the steps (or allow user to build and run with a static/fake build id for iterating on code). Updating any parameter value or parameter default means no reuse of steps.\r\n\t \r\n2. All of the pipeline steps also share the same hashed directory, which causes a snapshot rebuild if any of the files change in that directory changes. All the steps in the train pipeline currently all use: source_directory=e.sources_directory_train. In the repo, it seems like train.py is a standalone script. If the repo wanted to optimize more for reuse, it could put scripts into isolated directories for each step or point to the file instead of the directory. As long as the snapshot is not forced to rebuild, then reuse should be able to happen.","author":{"url":"https://github.com/xinyi-joffre","@type":"Person","name":"xinyi-joffre"},"datePublished":"2020-01-17T21:58:38.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":4},"url":"https://github.com/140/MLOpsPython/issues/140"}