René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Regression in v2.13.0: ConsumeBidirectionalStream caught unexpected exception  and will exit","articleBody":"#### tl;dr: Cause and fix\r\n\r\nI believe v2.13.0 introduced a regression here: https://github.com/googleapis/python-api-core/compare/v2.12.0...v2.13.0#diff-769a9f2c04700ff355d77a3292b7ff3cea5ff9bda91d0b31138eeb89fcd4d02dL95-L98\r\n\r\nv2.12.0 code:\r\n```\r\n    def _is_active(self):\r\n        # Note: there is a possibility that this starts *before* the call\r\n        # property is set. So we have to check if self.call is set before\r\n        # seeing if it's active.\r\n        if self.call is not None and not self.call.is_active():\r\n            return False\r\n        else:\r\n            return True\r\n```\r\n\r\nv2.13.0 code:\r\n\r\n```\r\n    def _is_active(self):\r\n        # Note: there is a possibility that this starts *before* the call\r\n        # property is set. So we have to check if self.call is set before\r\n        # seeing if it's active.\r\n        return self.call is not None and self.call.is_active()\r\n```\r\n\r\nNote that in `v2.12.0` version `_is_active()` will return True when `self.call is None` but in `v2.13.0` this will return False.\r\n\r\nThis seems to very directly violate the behaviour highlighted in the comment:\r\n\r\n```\r\n        # Note: there is a possibility that this starts *before* the call\r\n        # property is set. So we have to check if self.call is set before\r\n        # seeing if it's active.\r\n```\r\n\r\nAs such the fix should be as simple as\r\n\r\n```\r\nreturn self.call is None or self.call.is_active()\r\n```\r\n\r\n#### Environment details\r\n\r\n  - OS type and version: Mac 14.1.1 \r\n  - Python version: `python --version`: 3.9.18\r\n  - pip version: `pip --version`: 22.3\r\n  - `google-api-core` version: `pip show google-api-core`\r\n\r\n```\r\nName: google-api-core\r\nVersion: 2.14.0\r\nSummary: Google API client core library\r\nHome-page: https://github.com/googleapis/python-api-core\r\nAuthor: Google LLC\r\nAuthor-email: googleapis-packages@google.com\r\nLicense: Apache 2.0\r\nLocation: /Users/dhendry/code/perpetua-pybackend-common/.venv/lib/python3.9/site-packages\r\nRequires: google-auth, googleapis-common-protos, protobuf, requests\r\nRequired-by: firebase-admin, google-api-python-client, google-cloud-appengine-logging, google-cloud-bigquery, google-cloud-bigquery-storage, google-cloud-core, google-cloud-firestore, google-cloud-logging, google-cloud-pubsub, google-cloud-secret-manager, google-cloud-storage, google-cloud-tasks\r\n```\r\n\r\nAlso relevant here: \r\n\r\n```\r\nName: google-cloud-bigquery-storage\r\nVersion: 2.22.0\r\nSummary: Google Cloud Bigquery Storage API client library\r\nHome-page: https://github.com/googleapis/python-bigquery-storage\r\nAuthor: Google LLC\r\nAuthor-email: googleapis-packages@google.com\r\nLicense: Apache 2.0\r\nLocation: /Users/dhendry/code/perpetua-pybackend-common/.venv/lib/python3.9/site-packages\r\nRequires: google-api-core, proto-plus, protobuf\r\nRequired-by: perpetua-pybackend-common\r\n```\r\n\r\n#### Steps to reproduce\r\n\r\nAfter upgrading `google-api-core` from `2.12.0` to `2.13.0` we are seeing an `ConsumeBidirectionalStream caught unexpected exception  and will exit.` error when using the `BigQueryWriteClient`. I have narrowed down the issue to this library, NOT the google-cloud-bigquery-storage library\r\n\r\nNote: you many need to be on a slow-ish network connection (\u003c= ~50 MBit/sec) to reproduce reliably.\r\n\r\n  1. Create an `AppendRowsStream`\r\n  2. Call `.send()` with a large `AppendRowsRequest` request (close to the maximum, say 9 MB). Very specifically, it seems that the request must be large enough to take more than 1 second to complete (1 second because that is the default value for `google.api_core.bidi._RequestQueueGenerator._period`)\r\n  3. An exception will be raised\r\n\r\n#### Code example\r\n\r\n```python\r\nfrom google.cloud.bigquery_storage_v1 import BigQueryWriteClient\r\nfrom google.cloud.bigquery_storage_v1.types import ProtoRows, ProtoSchema\r\nfrom google.cloud.bigquery_storage_v1.types.storage import AppendRowsRequest\r\nfrom google.cloud.bigquery_storage_v1.writer import AppendRowsStream\r\n\r\ns = AppendRowsStream(\r\n    BigQueryWriteClient(),\r\n    AppendRowsRequest(\r\n        write_stream=\"STREAM NAME\",\r\n        proto_rows=AppendRowsRequest.ProtoData(\r\n            writer_schema=ProtoSchema(proto_descriptor=\"ADD PROTO DESCRIPTOR\"),\r\n        ),\r\n    ),\r\n)\r\n\r\ns.send(\r\n    AppendRowsRequest(\r\n        proto_rows=AppendRowsRequest.ProtoData(\r\n            rows=ProtoRows(serialized_rows=[SERIALIZED ROWS TOTALLING ~9mb])\r\n        )\r\n    )\r\n)\r\n```\r\n\r\n#### Stack trace\r\n```\r\n2023-11-30 17:33:07,189 [DEBUG] Started helper thread Thread-ConsumeBidirectionalStream\r\n2023-11-30 17:33:14,180 [DEBUG] Empty queue and inactive call, exiting request generator.\r\n2023-11-30 17:33:14,266 [DEBUG] waiting for recv.\r\n2023-11-30 17:33:14,266 [DEBUG] recved response.\r\n2023-11-30 17:33:14,266 [DEBUG] waiting for recv.\r\n2023-11-30 17:33:14,268 [INFO] RPC termination has signaled streaming pull manager shutdown.\r\n2023-11-30 17:33:14,268 [ERROR] Thread-ConsumeBidirectionalStream caught unexpected exception  and will exit.\r\nTraceback (most recent call last):\r\n  File \".venv/lib/python3.9/site-packages/google/api_core/bidi.py\", line 662, in _thread_main\r\n    response = self._bidi_rpc.recv()\r\n  File \".venv/lib/python3.9/site-packages/google/api_core/bidi.py\", line 345, in recv\r\n    return next(self.call)\r\n  File \".venv/lib/python3.9/site-packages/google/api_core/grpc_helpers.py\", line 115, in __next__\r\n    return next(self._wrapped)\r\n  File \".venv/lib/python3.9/site-packages/grpc/_channel.py\", line 541, in __next__\r\n    return self._next()\r\n  File \".venv/lib/python3.9/site-packages/grpc/_channel.py\", line 965, in _next\r\n    raise StopIteration()\r\nStopIteration\r\n2023-11-30 17:33:14,268 [DEBUG] Stopping consumer.\r\n2023-11-30 17:33:14,269 [INFO] Thread-ConsumeBidirectionalStream exiting\r\n```\r\n\r\n\r\n","author":{"url":"https://github.com/dhendry","@type":"Person","name":"dhendry"},"datePublished":"2023-11-30T18:16:13.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":1},"url":"https://github.com/560/python-api-core/issues/560"}