René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"gRPC error on `get_online_features` (NOAUTH authentication required)","articleBody":"## Expected Behavior \r\n`get_online_features` should return the correct result:\r\n\r\n```\r\nfeatures = client.get_online_features(\r\n    feature_refs=[\"driver_statistics:avg_daily_trips\"],\r\n    entity_rows=entities_with_timestamp[[\"driver_id\"]].T.to_dict().values()).to_dict()\r\n\r\nprint(\"\\nFeatures Output:\\n\")\r\nprint(pd.DataFrame.from_dict(features))\r\n```\r\n## Current Behavior\r\n\r\nGetting a gRPC error when calling `get_online_features` which points to an issue in `feast-online-serving`.\r\nMaterialization features is already completed and redis keys are present in redis store (`offline_to_online_ingestion`).\r\nCalling `get_online_features` works initially after `offline_to_online_ingestion`, but not predictably in a standalone script.\r\n\r\nError:\r\n```\r\ndetails = \"Unexpected error when pulling data from from Redis.\"\r\ndebug_error_string = \"{\"created\":\"@1644025055.432554000\",\"description\":\"Error received from peer ipv6:[::1]:6566\",\"file\":\"src/core/lib/surface/call.cc\",\"file_line\":1062,\"grpc_message\":\"Unexpected error when pulling data from from Redis.\",\"grpc_status\":2}\"\r\n\r\nTraceback (most recent call last):\r\nFile \"test_ingestion_and_get_features.py\", line 220, in \u003cmodule\u003e\r\nfeatures = client.get_online_features(\r\nFile \"/usr/local/anaconda3/envs/feast_poc/lib/python3.8/site-packages/feast/client.py\", line 1022, in get_online_features\r\nraise grpc.RpcError(e.details())\r\ngrpc.RpcError: Unexpected error when pulling data from from Redis.\r\n```\r\n\r\nError in `feast-online-serving` pod logs:\r\n\r\n```\r\n2022-02-04 16:35:24.823 WARN feast-release-feast-online-serving-7f65995d4d-7mghv --- [lt-executor-203] f.s.c.ServingServiceGRpcController : Failed to get Online Features\r\nio.grpc.StatusRuntimeException: UNKNOWN: Unexpected error when pulling data from from Redis.\r\nat io.grpc.Status.asRuntimeException(Status.java:524)\r\nat feast.storage.connectors.redis.retriever.OnlineRetriever.lambda$getFeaturesFromRedis$1(OnlineRetriever.java:114)\r\nat java.base/java.util.ArrayList.forEach(ArrayList.java:1541)\r\nat feast.storage.connectors.redis.retriever.OnlineRetriever.getFeaturesFromRedis(OnlineRetriever.java:101)\r\nat feast.storage.connectors.redis.retriever.OnlineRetriever.getOnlineFeatures(OnlineRetriever.java:56)\r\nat feast.serving.service.OnlineServingServiceV2.getOnlineFeatures(OnlineServingServiceV2.java:126)\r\nat feast.serving.controller.ServingServiceGRpcController.getOnlineFeaturesV2(ServingServiceGRpcController.java:96)\r\nat feast.proto.serving.ServingServiceGrpc$MethodHandlers.invoke(ServingServiceGrpc.java:314)\r\nat io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:172)\r\nat io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35)\r\nat io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23)\r\nat io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40)\r\nat io.grpc.Contexts$ContextualizedServerCallListener.onHalfClose(Contexts.java:86)\r\nat io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35)\r\nat io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23)\r\nat io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40)\r\nat io.opentracing.contrib.grpc.TracingServerInterceptor$2.onHalfClose(TracingServerInterceptor.java:235)\r\nat io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35)\r\nat io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23)\r\nat io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40)\r\nat io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35)\r\nat io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23)\r\nat io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40)\r\nat io.grpc.Contexts$ContextualizedServerCallListener.onHalfClose(Contexts.java:86)\r\nat io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:331)\r\nat io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:820)\r\nat io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)\r\nat io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)\r\nat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\r\nat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\r\nat java.base/java.lang.Thread.run(Thread.java:829)\r\nCaused by: java.util.concurrent.ExecutionException: io.lettuce.core.RedisCommandExecutionException: NOAUTH Authentication required.\r\nat java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)\r\nat java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)\r\nat feast.storage.connectors.redis.retriever.OnlineRetriever.lambda$getFeaturesFromRedis$1(OnlineRetriever.java:104)\r\n... 29 more\r\nCaused by: io.lettuce.core.RedisCommandExecutionException: NOAUTH Authentication required.\r\nat io.lettuce.core.ExceptionFactory.createExecutionException(ExceptionFactory.java:135)\r\nat io.lettuce.core.ExceptionFactory.createExecutionException(ExceptionFactory.java:108)\r\nat io.lettuce.core.protocol.AsyncCommand.completeResult(AsyncCommand.java:120)\r\nat io.lettuce.core.protocol.AsyncCommand.complete(AsyncCommand.java:111)\r\nat io.lettuce.core.protocol.CommandWrapper.complete(CommandWrapper.java:59)\r\nat io.lettuce.core.protocol.CommandHandler.complete(CommandHandler.java:654)\r\nat io.lettuce.core.protocol.CommandHandler.decode(CommandHandler.java:614)\r\nat io.lettuce.core.protocol.CommandHandler.channelRead(CommandHandler.java:565)\r\nat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\r\nat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\r\nat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\r\nat io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1518)\r\nat io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267)\r\nat io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314)\r\nat io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)\r\nat io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)\r\nat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)\r\nat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\r\nat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\r\nat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\r\nat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)\r\nat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\r\nat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\r\nat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)\r\nat io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)\r\nat io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)\r\nat io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)\r\nat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)\r\nat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\r\nat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\r\n... 1 more\r\n```\r\n\r\n## Steps to reproduce\r\n\r\nInstall helm chart on AKS v1.22:\r\nhttps://github.com/Azure/feast-azure/blob/rijai/feastchart/cluster/setup/feast-0.9.5-helmchart/README.md\r\n\r\nUpdate `feast-onilne-serving` deployment image to use latest image tag which supports auth (0.26.3):\r\nhttps://github.com/feast-dev/feast-java-old/pull/43\r\n\r\nSet `feast-release-feast-online-serving` configmap to use external redis:\r\n```\r\nfeast:\r\n  core-host: \"feast-release-feast-core\"\"\r\n  core-grpc-port: 6565\r\n  active_store: online\r\n  stores:\r\n    - name; online\r\n      type: REDIS\r\n      config:\r\n        host: \"external-redis.net\"\r\n        port: 6380\r\n        ssl: true\r\n        password: \"password\"\r\n      subscriptions:\r\n        - name: \"*\"\r\n          project: \"*\"\r\ngrpc:\r\n  server:\r\n    port: 6566\r\nserver:\r\n  port: 18099\r\n```\r\n\r\nDefine `FeatureTable` and run `offline_to_online_ingestion`, then in a separate script run `get_online_features`.\r\n\r\n### Specifications\r\n\r\n- Version: 0.9.5\r\n- Platform: Azure Kubernetes Service (v1.22.4)\r\n- Subsystem:\r\n- Redis version: 4.1.14\r\n\r\n## Possible Solution\r\n\r\nInstead of relying on auto-discovery feature of lettuce.io redis client, use `RESP2` protocol on lower versions of Redis.\r\nhttps://github.com/lettuce-io/lettuce-core/issues/1543#issuecomment-741635020\r\n\r\nTried making a change in `RedisClient.java`:\r\n\r\n```\r\n  private RedisClient(StatefulRedisConnection\u003cbyte[], byte[]\u003e connection) {\r\n    this.asyncCommands = connection.async();\r\n\r\n    // Disable auto-flushing\r\n    this.asyncCommands.setAutoFlushCommands(false);\r\n  }\r\n\r\n  public static RedisClientAdapter create(RedisStoreConfig config) {\r\n\r\n    RedisURI uri = RedisURI.create(config.getHost(), config.getPort());\r\n\r\n    if (config.getSsl()) {\r\n      uri.setSsl(true);\r\n    }\r\n\r\n    if (!config.getPassword().isEmpty()) {\r\n      uri.setPassword(config.getPassword());\r\n    }\r\n\r\n    io.lettuce.core.RedisClient redisClient = io.lettuce.core.RedisClient.create(uri);\r\n    redisClient.setOptions(\r\n        io.lettuce.core.ClientOptions.builder()\r\n            .protocolVersion(ProtocolVersion.RESP2)\r\n            .build());\r\n\r\n    StatefulRedisConnection\u003cbyte[], byte[]\u003e connection = redisClient.connect(new ByteArrayCodec());\r\n\r\n    return new RedisClient(connection);\r\n  }\r\n```\r\n\r\nHowever I run into the following issue at runtime:\r\n\r\n```\r\nCaused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [feast.serving.service.ServingServiceV2]: Factory method 'servingServiceV2' threw exception; nested exception is java.lang.NoClassDefFoundError: io/lettuce/core/protocol/ProtocolVersion\r\nat org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185)\r\nat org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:650)\r\n```","author":{"url":"https://github.com/andrijaperovic","@type":"Person","name":"andrijaperovic"},"datePublished":"2022-02-05T17:59:11.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":6},"url":"https://github.com/2285/feast/issues/2285"}