René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Modularize ingestion distributed compute engine support","articleBody":"This is a companion to #402 and the larger topic of storage engine modularization which was realized in #529 and subsequent PRs that implemented the new interfaces.\r\n\r\nJust as adding support for new storage engines tends to cause a dependency explosion for Feast `ingestion` \u0026 `serving`, the same is true for Beam Runner / job management adapter glue in `core` (this all could move to `serving` with future plans, but that won't change the fundamental problem this issue is about).\r\n\r\nSo for both storage and compute engines, I feel that some modularity strategy is needed for loose binding at build time, configurable for runtime. The goals would be to:\r\n\r\n- Minimize dependency pains that developers and contributors to Feast need to deal with if they are not actively working on a particular stack. The dependency trees are often large and fragile, especially in the Hadoop ecosystem, such as Hive and Spark.\r\n- Reduce deployment bloat if operators wish to package Feast internally with only the module JARs they need to support their organization's stack. (IIRC last I checked, `hadoop-common` or `hadoop-client` leave you with close to 200MB of jars, and `beam-runners-spark` and `beam-sdks-java-io-hcatalog` among others have these deps [as provided scope, but the point stands I believe]).\r\n\r\nPossibilities might be OSGi or `java.util.ServiceLoader` (and Spring integration or alternatives thereof). Open to other ideas!\r\n\r\nRelates to #362 ","author":{"url":"https://github.com/ches","@type":"Person","name":"ches"},"datePublished":"2020-01-27T05:28:04.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":4},"url":"https://github.com/444/feast/issues/444"}