René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Shift ingestion feature metrics to after store write ","articleBody":"**Is your feature request related to a problem? Please describe.**\r\nExisting metrics collection for ingestion is done in parallel to store writes:\r\n```\r\n               +\r\n               |\r\n       +-------+--------+\r\n       |                |\r\n+------+-----+   +------+-----+\r\n|            |   |            |\r\n|   store    |   |  metrics   |\r\n|   write    |   |   write    |\r\n|            |   |            |\r\n+------------+   +------------+\r\n```\r\n\r\nWhile this does give a rough gauge as to what data looks like in flight, it doesn't actually reflect the status of the data in the store, which is ultimately what we are concerned about, since write failures are common. \r\n\r\n**Describe the solution you'd like**\r\nIdeally we want to write metrics POST store writes, as so:\r\n```\r\n              ...\r\n               +\r\n               |\r\n         +-----+-----+\r\n         |           |\r\n         |   store   |\r\n         |   write   |\r\n         |           |\r\n         +-----+-----+\r\n               |\r\n      +--------+--------+\r\n      +                 +\r\n   success           failure\r\n      +                 +\r\n      |                 |\r\n+-----+-----+     +-----+-----+\r\n|  write    |     |  write    |\r\n|  sucess   |     |  failure  |\r\n|  metrics  |     |  metrics  |\r\n+-----------+     +-----------+\r\n```\r\n\r\nMetrics collected post store writes would more accurately reflect the data in the stores. \r\n","author":{"url":"https://github.com/zhilingc","@type":"Person","name":"zhilingc"},"datePublished":"2020-02-24T04:22:18.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/489/feast/issues/489"}