René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Bigtable: read_rows: no deadline causing stuck clients; no way to change it","articleBody":"`Table.read_rows` does not set any deadline, so it can hang forever if the Bigtable server connection hangs. We see this happening once every week or two when running inside GCP, which causes our server to get stuck indefinitely. There appears to be no way in the API to set a deadline, even though [the documentation](https://googleapis.github.io/google-cloud-python/latest/bigtable/table.html#google.cloud.bigtable.table.Table.read_rows) says that the `retry` parameter should do this. Due to a bug, it does not.\r\n\r\n### Details:\r\n\r\nWe are calling `Table.read_rows` to read ~2 rows from BigTable. Using pyflame on a stuck process, both worker threads were waiting on Bigtable, with the stack trace below. I believe the bug is the following:\r\n\r\n1. Call `Table.read_rows`. [This calls `PartialRowsData`](https://github.com/googleapis/google-cloud-python/blob/master/bigtable/google/cloud/bigtable/table.py#L436), passing the `retry` argument which defaults to `DEFAULT_RETRY_READ_ROWS`. The default misleadingly sets `deadline=60.0`. ; It also passes `read_method=self._instance._client.table_data_client.transport.read_rows` to `PartialRowsData`, which is a method on `BigtableGrpcTransport`.\r\n2. [`PartialRowsData.__init__` calls `read_method()`](https://github.com/googleapis/google-cloud-python/blob/master/bigtable/google/cloud/bigtable/row_data.py#L398); this is actually raw gRPC `_UnaryStreamMultiCallable`, *not* the gapic `BigtableClient.read_rows`, which AFAICS, is never called. Hence, this gRPC streaming call is started with any deadline. \r\n3. `PartialRowsData.__iter__` calls `self._read_next_response`, which [calls `return self.retry(self._read_next, on_error=self._on_error)()`](https://github.com/googleapis/google-cloud-python/blob/master/bigtable/google/cloud/bigtable/row_data.py#L454). This gives the impression that `retry` is used, but if I understand gRPC streams correctly, I'm not sure that even makes sense. I think even if the gRPC stream return some error, calling `next` won't actually retry the gRPC, it will just immediately raise the same exception. To retry, I believe you need to actually restart it by calling `read_rows` again.\r\n4. If the Bigtable server now \"hangs\", the client hangs forever.\r\n\r\n### Possible fix:\r\n\r\nChange `Table.read_rows` call the gapic `BigtableClient.read_rows` with the `retry` parameter., and change `PartialRowsData.__init__` to take this response iterator, and not take a `retry` parameter at all. This would at least allow setting the gRPC streaming call deadline, although I don't think it will make retrying work (since I *think* the gRPC streaming client will just immediately returns an iterator without actually waiting for a response from the server?)\r\n\r\nI haven't actually tried implementing this to see if it works. For now, we will probably just make a raw gRPC read_rows call so we can set an appropriate timeout. \r\n\r\n\r\n#### Environment details\r\n\r\n*OS*: Linux, ContainerOS (GKE), Container is Debian9 (using distroless)\r\n*Python*: 3.5.3\r\n*API*: google-cloud-bigtable 0.33.0\r\n\r\n\r\n#### Steps to reproduce\r\n\r\nThis program loads the Bigtable emulator with 1000 rows, calls `read_rows(retry=DEFAULT.with_deadline(5.0))`, then sends `SIGSTOP` to pause the emulator. This SHOULD cause a `DeadlineExceeded` exception to be raised after 5 seconds. Instead, it hangs forever.\r\n\r\n1. Start the Bigtable emulator: `gcloud beta emulators bigtable start`\r\n2. Find the PID: `ps ax | grep cbtemulator`\r\n3. Run the following program with `BIGTABLE_EMULATOR_HOST=localhost:8086 python3 bug.py $PID`\r\n\r\n```python\r\nfrom google.api_core import exceptions\r\nfrom google.cloud import bigtable\r\nfrom google.rpc import code_pb2\r\nfrom google.rpc import status_pb2\r\nimport os\r\nimport signal\r\nimport sys\r\n\r\nCOLUMN_FAMILY_ID = 'column_family_id'\r\n\r\ndef main():\r\n    emulator_pid = int(sys.argv[1])\r\n    client = bigtable.Client(project=\"testing\", admin=True)\r\n    instance = client.instance(\"emulator\")\r\n\r\n    # create/open a table\r\n    table = instance.table(\"emulator_table\")\r\n    column_family = table.column_family(COLUMN_FAMILY_ID)\r\n    try:\r\n        table.create()\r\n        column_family.create()\r\n    except exceptions.AlreadyExists:\r\n        print('table exists')\r\n\r\n    # write a bunch of data\r\n    for i in range(1000):\r\n        k = 'some_key_{:04d}'.format(i)\r\n        print(k)\r\n        row = table.row(k)\r\n        row.set_cell(COLUMN_FAMILY_ID, 'column', 'some_value{:d}'.format(i) * 1000)\r\n        result = table.mutate_rows([row])\r\n        assert len(result) == 1 and result[0].code == code_pb2.OK\r\n        assert table.read_row(k) is not None\r\n\r\n    print('starting read')\r\n    rows = table.read_rows(retry=bigtable.table.DEFAULT_RETRY_READ_ROWS.with_deadline(5.0))\r\n    rows_iter = iter(rows)\r\n    r1 = next(rows_iter)\r\n    print('read', r1)\r\n    os.kill(emulator_pid, signal.SIGSTOP)\r\n    print('sent sigstop')\r\n    for r in rows_iter:\r\n        print(r)\r\n    print('done')\r\n\r\n\r\nif __name__ == '__main__':\r\n    main()\r\n```\r\n\r\n#### Stack trace of hung server (using slightly older version of the google-cloud-bigtable library\r\n\r\n```\r\n/usr/local/lib/python2.7/threading.py:wait:340\r\n/usr/local/lib/python2.7/site-packages/grpc/_channel.py:_next:348\r\n/usr/local/lib/python2.7/site-packages/grpc/_channel.py:next:366\r\n/usr/local/lib/python2.7/site-packages/google/cloud/bigtable/row_data.py:_read_next:426\r\n/usr/local/lib/python2.7/site-packages/google/api_core/retry.py:retry_target:179\r\n/usr/local/lib/python2.7/site-packages/google/api_core/retry.py:retry_wrapped_func:270\r\n/usr/local/lib/python2.7/site-packages/google/cloud/bigtable/row_data.py:_read_next_response:430\r\n/usr/local/lib/python2.7/site-packages/google/cloud/bigtable/row_data.py:__iter__:441\r\n```\r\n","author":{"url":"https://github.com/evanj","@type":"Person","name":"evanj"},"datePublished":"2019-05-29T00:22:56.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":6},"url":"https://github.com/6/python-bigtable/issues/6"}