René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Job Coordination Improvement Proposal","articleBody":"**Is your feature request related to a problem? Please describe.**\r\n\r\nThere's some technical debt in ingestion part mostly related to Job Coordination: job is being restarted on schema update, this restart can take arbitrary amount of time, hence we had to split jobs as much as possible to minimize mutual disruption (on restart). This negatively affects our UX and also restrict us from using Dataflow resources more efficiently, which currently leads to very high spendings on Dataflow.\r\n\r\n**Describe the solution you'd like**\r\n\r\nI drafter proposal that addresses those issues and suggest new design in communications between core service and ingestion pipeline\r\nhttps://docs.google.com/document/d/1gqkCWZUyVBIU8OKhxIhIf1BBd3JrbOx2WnXrpGLFCAc/edit#heading=h.wigpvke4im4g\r\n","author":{"url":"https://github.com/pyalex","@type":"Person","name":"pyalex"},"datePublished":"2020-06-01T05:30:21.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":1},"url":"https://github.com/761/feast/issues/761"}