René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Replace string references used in non user facing components in Feast","articleBody":"## Background\r\n\r\nCurrently Feast uses string references in 3 areas:\r\n- Feature Set References\r\n- Feature References\r\n- Subscriptions\r\n\r\nAs a result many implementations of string reference parsing and formatting has cropped up all over Feast's code:\r\n- Feast Serving's [`RefUtil`](https://github.com/gojek/feast/blob/e4f8fe905d3c182416e1b69bf60bfbc0e8348f8a/serving/src/main/java/feast/serving/util/RefUtil.java)\r\n- Feast Core's [`Store`](https://github.com/gojek/feast/blob/7a0ff91839757233d3af957b84096f9e2836386b/core/src/main/java/feast/core/model/Store.java#L127)\r\n- Feast Ingestion's [`SpecUtil`](https://github.com/gojek/feast/blob/7a0ff91839757233d3af957b84096f9e2836386b/ingestion/src/main/java/feast/ingestion/utils/SpecUtil.java#L34)\r\n- Feast Storage Writers: [`Redis`](https://github.com/gojek/feast/blob/7a0ff91839757233d3af957b84096f9e2836386b/storage/connectors/redis/src/main/java/feast/storage/connectors/redis/writer/RedisCustomIO.java#L185) [`RedisCluster`](https://github.com/gojek/feast/blob/7a0ff91839757233d3af957b84096f9e2836386b/storage/connectors/rediscluster/src/main/java/feast/storage/connectors/rediscluster/writer/RedisClusterCustomIO.java#L187).\r\n- Feast Storage Retrievers [`Redis`](https://github.com/gojek/feast/blob/7a0ff91839757233d3af957b84096f9e2836386b/storage/connectors/redis/src/main/java/feast/storage/connectors/redis/retriever/RedisOnlineRetriever.java#L214) [`RedisCluster`](https://github.com/gojek/feast/blob/7a0ff91839757233d3af957b84096f9e2836386b/storage/connectors/rediscluster/src/main/java/feast/storage/connectors/rediscluster/retriever/RedisClusterOnlineRetriever.java#L218)\r\n- etc...\r\n\r\n## Problem\r\n\r\nChanges to the feature/feature set references/subscriptions are difficult as we have to track down all the different string parsing/formatting implementations. Changes to these areas should not require a comb over the entire codebase to rectify all these parsing and formatting implementations.\r\n\r\nExamples  of currently WIP and potential changes that is currently affected by this and why the problem is relevant:\r\n- Feature Set version removal #386 \r\n- Changes to the Feast concept \u0026 hierarchy and data model #479 \r\n- Removal of Projects from Feature References when in the string form #631 \r\n\r\n### Scope \r\n\r\nString references that should still be kept around:\r\n- those exposed by the user facing API (ie the SDKs, config files) as they need to be humanly readable.\r\n- those used by interface with components outside of feast (ie PostgreSQL, BigQuery, etc.), although non string references should be preferred if possible. \r\n\r\n## Possible Solution\r\nReplace string references in non user facing components with their Protobuf equivalents.  \r\nExamples of this include:\r\n- Using message types directly in protobuf definitions:\r\n```protobuf\r\nmessage FeatureRow {\r\n    repeated Field fields = 2;\r\n    google.protobuf.Timestamp event_timestamp = 3;\r\n    FeatureSetReference feature_set_ref = 8;\r\n    string ingestion_id = 7;\r\n}\r\n```\r\n\u003e\r\n\r\n- Using references as keys in maps directly.\r\n```java\r\nHashMap\u003cFeatureSetReference, FeatureSetSpec\u003e featureSetMap = new HashMap\u003c\u003e();\r\n```\r\n\r\n\u003e :warning: There might be possible [caveats to this](https://stackoverflow.com/questions/34729968/protobuf-objects-as-keys-in-maps)\r\n- Serialize references without having to extract its internals:\r\n```java\r\nString refStr = Base64.getEncoder().encodeToString(featureSetReference.toByteArray());\r\n```\r\n","author":{"url":"https://github.com/mrzzy","@type":"Person","name":"mrzzy"},"datePublished":"2020-05-03T09:47:45.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":4},"url":"https://github.com/674/feast/issues/674"}