René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"PubSub: Subscriber keeps failing with 503","articleBody":"Hey guys, \r\n\r\nI am seeing the pubsub client receive a 503 unavailable and instead of retrying it just exits. And it can't receive any more messages. The subscriber starts and works for a few minutes before throwing the error. I can reproduce it consistently with multiple different pubsub subscriptions. \r\n\r\n**Package**: google-cloud-pubsub==1.4.2\r\n**OS**: Mac Catalina, also tried on an Ubuntu VM. \r\n**Python**: Python 3.7.3\r\n\r\n\r\nHere is an example test case where I see the issue followed by a stack trace: \r\n\r\n```\r\nimport os\r\nimport json\r\nfrom time import sleep\r\nfrom datetime import datetime\r\nimport logging\r\nfrom google.cloud import pubsub_v1\r\nfrom config import Config\r\n\r\nlogging.basicConfig()\r\nlogging.getLogger().setLevel(logging.DEBUG)\r\n\r\ndef _callback(message):\r\n    try:\r\n        print('message', message)\r\n    except Exception:\r\n        print('exception')\r\n\r\nif __name__ == '__main__':\r\n\r\n    subscription = pubsub_v1.SubscriberClient()\r\n    subscription_path = subscription.subscription_path(\r\n        Config.PROJECT_ID,\r\n        Config.PUBSUB_SUB_NAME)\r\n\r\n\r\n    future = subscription.subscribe(\r\n        subscription_path,\r\n        _callback,\r\n        pubsub_v1.types.FlowControl(max_messages=Config.MAX_MESSAGES))\r\n\r\n    try:\r\n        future.result()\r\n    \r\n    except KeyboardInterrupt:\r\n        future.cancel()\r\n        logging.error(\"Keyboard Interrupt\")\r\n    \r\n    except Exception:\r\n        raise\r\n\r\n```\r\n\r\nAnd stack trace - \r\n\r\n```\r\nDEBUG:google.cloud.pubsub_v1.subscriber._protocol.leaser:Snoozing lease management for 8.538204 seconds.\r\nDEBUG:google.api_core.bidi:Thread-ConsumeBidirectionalStream caught error 503 The service was unable to fulfill your request. Please try again. [code=8a75] and will exit. Generally this is due to the RPC itself being cancelled and the error will be surfaced to the calling code.\r\nTraceback (most recent call last):\r\n  File \"/Users/tameem/pubsub-test/env/lib/python3.7/site-packages/google/api_core/grpc_helpers.py\", line 144, in error_remapped_callable\r\n    return _StreamingResponseIterator(result)\r\n  File \"/Users/tameem/pubsub-test/env/lib/python3.7/site-packages/google/api_core/grpc_helpers.py\", line 72, in __init__\r\n    self._stored_first_result = six.next(self._wrapped)\r\n  File \"/Users/tameem/pubsub-test/env/lib/python3.7/site-packages/grpc/_channel.py\", line 416, in __next__\r\n    return self._next()\r\n  File \"/Users/tameem/pubsub-test/env/lib/python3.7/site-packages/grpc/_channel.py\", line 706, in _next\r\n    raise self\r\ngrpc._channel._MultiThreadedRendezvous: \u003c_MultiThreadedRendezvous of RPC that terminated with:\r\n\tstatus = StatusCode.UNAVAILABLE\r\n\tdetails = \"The service was unable to fulfill your request. Please try again. [code=8a75]\"\r\n\tdebug_error_string = \"{\"created\":\"@1587006128.267570000\",\"description\":\"Error received from peer ipv6:[]:443\",\"file\":\"src/core/lib/surface/call.cc\",\"file_line\":1056,\"grpc_message\":\"The service was unable to fulfill your request. Please try again. [code=8a75]\",\"grpc_status\":14}\"\r\n\u003e\r\n\r\nThe above exception was the direct cause of the following exception:\r\n\r\nTraceback (most recent call last):\r\n  File \"/Users/tameem/pubsub-test/env/lib/python3.7/site-packages/google/api_core/bidi.py\", line 637, in _thread_main\r\n    self._bidi_rpc.open()\r\n  File \"/Users/tameem/pubsub-test/env/lib/python3.7/site-packages/google/api_core/bidi.py\", line 280, in open\r\n    call = self._start_rpc(iter(request_generator), metadata=self._rpc_metadata)\r\n  File \"/Users/tameem/pubsub-test/env/lib/python3.7/site-packages/google/cloud/pubsub_v1/gapic/subscriber_client.py\", line 1059, in streaming_pull\r\n    requests, retry=retry, timeout=timeout, metadata=metadata\r\n  File \"/Users/tameem/pubsub-test/env/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py\", line 143, in __call__\r\n    return wrapped_func(*args, **kwargs)\r\n  File \"/Users/tameem/pubsub-test/env/lib/python3.7/site-packages/google/api_core/retry.py\", line 286, in retry_wrapped_func\r\n    on_error=on_error,\r\n  File \"/Users/tameem/pubsub-test/env/lib/python3.7/site-packages/google/api_core/retry.py\", line 184, in retry_target\r\n    return target()\r\n  File \"/Users/tameem/pubsub-test/env/lib/python3.7/site-packages/google/api_core/timeout.py\", line 214, in func_with_timeout\r\n    return func(*args, **kwargs)\r\n  File \"/Users/tameem/pubsub-test/env/lib/python3.7/site-packages/google/api_core/grpc_helpers.py\", line 146, in error_remapped_callable\r\n    six.raise_from(exceptions.from_grpc_error(exc), exc)\r\n  File \"\u003cstring\u003e\", line 3, in raise_from\r\ngoogle.api_core.exceptions.ServiceUnavailable: 503 The service was unable to fulfill your request. Please try again. [code=8a75]\r\nINFO:google.api_core.bidi:Thread-ConsumeBidirectionalStream exiting\r\nINFO:google.cloud.pubsub_v1.subscriber._protocol.leaser:Thread-LeaseMaintainer exiting.\r\nINFO:google.cloud.pubsub_v1.subscriber._protocol.heartbeater:Thread-Heartbeater exiting.\r\n```\r\n","author":{"url":"https://github.com/tameemiftikhar","@type":"Person","name":"tameemiftikhar"},"datePublished":"2020-04-16T03:04:41.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":3},"url":"https://github.com/75/python-pubsub/issues/75"}