René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"increased memory usage in 2.4.0","articleBody":"Version 2.4.0 of the library is allocating much more memory that the previous version, 2.3.1, when running multiple queries.\r\nIn particular, it seems that the `QueryJob` object is retaining the results of the query internally, and that memory is not deallocated.\r\n\r\nI think that the problem is related to https://github.com/googleapis/python-bigquery/pull/374.\r\n\r\n#### Environment details\r\n\r\n  - macOS 11.0.1 (also observing this on Linux in a production environment)\r\n  - Python version: 3.8.6\r\n  - pip version: 20.1.1 \r\n  - `google-cloud-bigquery` version: 2.4.0\r\n\r\n#### Steps to reproduce\r\n\r\nRun the script in the code example with `google-cloud-bigquery` 2.4.0 and 2.3.1 versions.\r\nYou will also need to install:\r\n```\r\ngoogle-cloud-bigquery-storage==2.1.0\r\npandas==1.1.4\r\npsutil==5.7.3\r\n```\r\n\r\nThe outputs on my machine are:\r\n\r\nWith 2.4.0:\r\n```\r\nInitial memory used: 77 MB\r\nMemory used: 642 MB\r\nMemory used: 875 MB\r\nMemory used: 1117 MB\r\nMemory used: 1342 MB\r\nMemory used: 1568 MB\r\nMemory used: 1792 MB\r\nMemory used: 2039 MB\r\nMemory used: 2265 MB\r\nMemory used: 2505 MB\r\nMemory used: 2725 MB\r\n```\r\n\r\nWith 2.3.1:\r\n\r\n```\r\nInitial memory used: 77 MB\r\nMemory used: 97 MB\r\nMemory used: 98 MB\r\nMemory used: 99 MB\r\nMemory used: 99 MB\r\nMemory used: 99 MB\r\nMemory used: 99 MB\r\nMemory used: 100 MB\r\nMemory used: 101 MB\r\nMemory used: 101 MB\r\nMemory used: 101 MB\r\n```\r\n\r\n#### Code example\r\n\r\nPlease note that we are storing a reference to the `QueryJob` objects, but **not** to the resulting DataFrames.\r\n\r\n```python\r\nimport os\r\n\r\nimport psutil\r\nfrom google.cloud import bigquery\r\n\r\nif __name__ == '__main__':\r\n    client = bigquery.Client()\r\n\r\n    process = psutil.Process(os.getpid())\r\n    print(f\"Initial memory used: {process.memory_info().rss / 1e6:.0f} MB\")\r\n\r\n    jobs = []\r\n\r\n    for i in range(10):\r\n        job = client.query(\"SELECT x FROM UNNEST(GENERATE_ARRAY(1, 1000000)) AS x\")\r\n        job.result().to_dataframe()\r\n        jobs.append(job)\r\n        print(f\"Memory used: {process.memory_info().rss / 1e6:.0f} MB\")\r\n```\r\n\r\n\r\n","author":{"url":"https://github.com/pietrodn","@type":"Person","name":"pietrodn"},"datePublished":"2020-11-19T08:05:36.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":4},"url":"https://github.com/394/python-bigquery/issues/394"}