René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"BigTable: Memory usage abnormally increasing when using table.read_rows method","articleBody":"Hi,\r\n\r\nWhile using the python google-cloud-bigtable to read a large collection of data (10M to 90M rows of at least 4MB to 5MB per row), I noticed a strange behavior: The used RAM was at lease twice the size of the data I was reading from the table.\r\n\r\nHere is the code I did to reproduce :\r\n\r\n```python\r\nfrom google.cloud import bigtable\r\n\r\ninstance_name = \"the-instance\"\r\ntable_name = \"the-table\"\r\n\r\nclient = bigtable.Client(admin=False)\r\ninstance = client.instance(instance_name)\r\ntable = instance.table(table_name)\r\n\r\nlast_readall_key = None\r\nrows = table.read_rows(start_key=last_readall_key)\r\n\r\nwhile True:\r\n    try:\r\n        rows.consume_next()\r\n    except StopIteration as e:\r\n        break\r\n    except Exception as e:\r\n        print 'Got exception of type %s : %s' % (type(e), str(e)) # mostly grpc._channel._Rendezvous exceptions are thrown here\r\n\r\n        table = instance.table(table_name)\r\n        rows = table.read_rows(start_key=last_readall_key)\r\n\r\n        rows.consume_next()\r\n\r\n    for key in rows.rows:\r\n        last_readall_key = key\r\n\r\n    rows._rows.clear() #this was an attempt to fix, but it doesn't have any effect\r\n```\r\n\r\nWhen running the above code, my instance RAM fills itself until full.\r\n\r\nAnother weird thing, is that the problem does not happen on my MacOS Sierra 10.12.6 machine. I only witness the problem on Linux machines.\r\n\r\nI saw this problem on Linux Ubuntu 14.04 LTS and Linux Ubuntu 10.04.5 LTS. I did not try with other Linux distributions or others versions of Ubuntu though.\r\n\r\nAm I doing something wrong ? Does anyone knows what may cause this problem ?\r\nWould anyone have any trick to avoid this problem ?\r\n\r\nFind below the pip requirements to reproduce :\r\n\r\n```\r\ncachetools==2.0.1\r\ncertifi==2017.11.5\r\nchardet==3.0.4\r\ndill==0.2.7.1\r\nenum34==1.1.6\r\nfuture==0.16.0\r\nfutures==3.2.0\r\ngoogle-api-core==0.1.2\r\ngoogle-auth==1.2.1\r\ngoogle-cloud-bigtable==0.28.1\r\ngoogle-cloud-core==0.28.0\r\ngoogle-gax==0.15.16\r\ngoogleapis-common-protos==1.5.3\r\ngrpcio==1.7.3\r\nidna==2.6\r\nply==3.8\r\nprotobuf==3.5.0.post1\r\npyasn1==0.4.2\r\npyasn1-modules==0.2.1\r\npytz==2017.3\r\nrequests==2.18.4\r\nrsa==3.4.2\r\nsix==1.11.0\r\nurllib3==1.22\r\n```\r\n(these dependancies are installed automatically when using `pip install google-cloud-bigtable` )\r\n\r\nThanks for your time.","author":{"url":"https://github.com/Rafff","@type":"Person","name":"Rafff"},"datePublished":"2017-12-13T22:13:31.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":6},"url":"https://github.com/4586/google-cloud-python/issues/4586"}