René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Optimize Redis memory foot print for ingestion / serving","articleBody":"**Is your feature request related to a problem? Please describe.**\r\nAs of now, the Redis Value size is directly proportional to the length of the feature name strings. This becomes more significant when there are several features. As a result the memory foot print can be significantly higher than the equivalent data stored in CSV / parquet / avro.\r\n\r\n**Describe the solution you'd like**\r\nInstead of writing the byte representation of Feature Row, which has a list of Fields, it might be suffice if we simply store the list of Values instead. Feast Online Serving is already aware of the FeatureSetSpec, and hence the field names. This information can be used to reconstruct the Feature Row from the list of Values. Without storing the feature names, the Redis value size can be drastically reduced.\r\n\r\n**Describe alternatives you've considered**\r\nAnother alternative would be, instead of storing the feature names in the Redis, we can store the hash code of the feature name instead. Again, the corresponding Feature Row can be reconstructed on the Feast online serving side.\r\n\r\nLast alternative would be to apply some compression algorithm (eg Gzip/Bzip) on the byte array representation of Feature Row, though it is unlikely to surpass the above alternatives in terms of efficiency. There will also be impact in terms of latency due to additional processing required during retrieval.\r\n\r\n**Additional context**\r\nAny approach to resolve this issue will likely resulted in backward incompatible changes: the existing Redis value will not longer be interpretable once the way we stored Redis value changes. Therefore, we might have to include a Store level configuration, to toggle this feature on only when the user is ready to migrate.\r\n","author":{"url":"https://github.com/khorshuheng","@type":"Person","name":"khorshuheng"},"datePublished":"2020-03-06T02:53:33.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":15},"url":"https://github.com/515/feast/issues/515"}