René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Redis High Availability","articleBody":"**Is your feature request related to a problem? Please describe.**\r\n\r\nCurrently we do not natively support Redis in high availability mode for online serving. \r\n- Redis is a single point of failure\r\n- Redis cannot currently scale out more than one node, leading to resource utilization problems (disk, cpu).\r\n\r\nAdditionally, there is a problem with the current implementation of the Redis client in Apache Beam where it establishes too many connections to Redis.\r\n\r\n**Describe the solution you'd like**\r\nThe goal is to add support for HA Redis as part of Feast. This would require\r\n1. Users should be able to configure the addresses to these stores as part of their serving deployment. This may necessitate an update the Redis store in the [store.proto](https://github.com/gojek/feast/blob/master/protos/feast/core/Store.proto) (or the creation of a new type of store like `RedisCluster`.\r\n2. Refactoring the [Feast Redis IO](https://github.com/gojek/feast/blob/master/ingestion/src/main/java/feast/store/serving/redis/RedisCustomIO.java#L302) in the Apache Beam ingestion code to support Redis clusters.\r\n3. Refactoring the [Feast Serving client](https://github.com/gojek/feast/blob/master/serving/src/main/java/feast/serving/service/RedisServingService.java) to support Redis clusters.\r\n4. Ensuring that the Redis clients (both for cluster and non-cluster mode) do not create an excessive amount of connections. Ideally each dataflow or beam node should be sharing a single connection pool between all bundles.\r\n5. EDIT: We should try to use [Lettuce](https://github.com/lettuce-io/lettuce-core) as the Redis Client for both serving and ingestion. \r\nhttps://github.com/spring-projects/spring-session/issues/789\r\nhttps://github.com/lettuce-io/lettuce-core","author":{"url":"https://github.com/woop","@type":"Person","name":"woop"},"datePublished":"2020-02-18T01:00:22.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":3},"url":"https://github.com/478/feast/issues/478"}