Title: Possible memory leak when using Google Sheets API · Issue #535 · googleapis/google-api-python-client · GitHub
Open Graph Title: Possible memory leak when using Google Sheets API · Issue #535 · googleapis/google-api-python-client
X Title: Possible memory leak when using Google Sheets API · Issue #535 · googleapis/google-api-python-client
Description: There seems to be a memory leak when using the google-api-client with GSheets. Environment: $ python --version Python 3.6.4 $ pip show google-api-python-client Name: google-api-python-client Version: 1.7.3 Here's a simple reproducer (wit...
Open Graph Description: There seems to be a memory leak when using the google-api-client with GSheets. Environment: $ python --version Python 3.6.4 $ pip show google-api-python-client Name: google-api-python-client Versio...
X Description: There seems to be a memory leak when using the google-api-client with GSheets. Environment: $ python --version Python 3.6.4 $ pip show google-api-python-client Name: google-api-python-client Versio...
Opengraph URL: https://github.com/googleapis/google-api-python-client/issues/535
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Possible memory leak when using Google Sheets API","articleBody":"There seems to be a memory leak when using the google-api-client with GSheets.\r\n\r\nEnvironment:\r\n```\r\n$ python --version\r\nPython 3.6.4\r\n$ pip show google-api-python-client\r\nName: google-api-python-client\r\nVersion: 1.7.3\r\n```\r\n\r\nHere's a simple reproducer (without a `.client_secret.json`):\r\n```python\r\n#!/usr/bin/env python3\r\n\r\nimport httplib2\r\nimport os\r\nfrom apiclient import discovery\r\nfrom memory_profiler import profile\r\nfrom oauth2client import client, tools\r\nfrom oauth2client.file import Storage\r\nfrom time import sleep\r\n\r\nSCOPES = \"https://www.googleapis.com/auth/spreadsheets.readonly\"\r\n# See https://cloud.google.com/docs/authentication/getting-started\r\nCLIENT_SECRET_FILE = \".client_secret.json\"\r\nAPPLICATION_NAME = \"ClientDebug\"\r\nDISCOVERY_URL = \"https://sheets.googleapis.com/$discovery/rest?version=v4\"\r\n\r\ndef get_credentials():\r\n home_dir = os.path.expanduser(\"~\")\r\n credential_dir = os.path.join(home_dir, \".credentials\")\r\n flags = None\r\n if not os.path.exists(credential_dir):\r\n os.makedirs(credential_dir)\r\n credential_path = os.path.join(credential_dir,\r\n \"sheets.googleapis.com-clientdebug.json\")\r\n\r\n store = Storage(credential_path)\r\n credentials = store.get()\r\n if not credentials or credentials.invalid:\r\n flow = client.flow_from_clientsecrets(CLIENT_SECRET_FILE, SCOPES)\r\n flow.user_agent = APPLICATION_NAME\r\n credentials = tools.run_flow(flow, store, flags)\r\n\r\n return credentials\r\n\r\n@profile(precision=4)\r\ndef get_responses(creds):\r\n \"\"\"Fetch spreadsheet data.\"\"\"\r\n sheet_id = \"1TowKJrFVbT4Bfp-HFcMh_CZ5anfH0CLfmoqCz9SUr9c\"\r\n\r\n http = creds.authorize(httplib2.Http())\r\n service = discovery.build(\"sheets\", \"v4\", http=http,\r\n discoveryServiceUrl=(DISCOVERY_URL), cache_discovery=False)\r\n result = service.spreadsheets().values().get(\r\n spreadsheetId=sheet_id, range=\"A1:O\").execute()\r\n values = result.get(\"values\", [])\r\n\r\n print(\"Got {} rows\".format(len(values)))\r\n\r\nif __name__ == \"__main__\":\r\n creds = get_credentials()\r\n for i in range(0, 50):\r\n get_responses(creds)\r\n sleep(2)\r\n```\r\n\r\nFor measurements I used `memory_profiler` module with following results:\r\n## First and second iteration\r\n```\r\nGot 760 rows\r\nFilename: ./main.py\r\n\r\nLine # Mem usage Increment Line Contents\r\n================================================\r\n 35 26.5195 MiB 26.5195 MiB @profile(precision=4)\r\n 36 def get_responses(creds):\r\n 37 \"\"\"Fetch spreadsheet data.\"\"\"\r\n 38 26.5195 MiB 0.0000 MiB sheet_id = \"1TowKJrFVbT4Bfp-HFcMh_CZ5anfH0CLfmoqCz9SUr9c\"\r\n 39 \r\n 40 26.5195 MiB 0.0000 MiB http = creds.authorize(httplib2.Http())\r\n 41 26.5195 MiB 0.0000 MiB service = discovery.build(\"sheets\", \"v4\", http=http,\r\n 42 29.2891 MiB 2.7695 MiB discoveryServiceUrl=(DISCOVERY_URL), cache_discovery=False)\r\n 43 49.5742 MiB 20.2852 MiB result = service.spreadsheets().values().get(\r\n 44 49.5742 MiB 0.0000 MiB spreadsheetId=sheet_id, range=\"A1:O\").execute()\r\n 45 49.5742 MiB 0.0000 MiB values = result.get(\"values\", [])\r\n 46 \r\n 47 49.5742 MiB 0.0000 MiB print(\"Got {} rows\".format(len(values)))\r\n\r\n\r\nGot 760 rows\r\nFilename: ./main.py\r\n\r\nLine # Mem usage Increment Line Contents\r\n================================================\r\n 35 49.5742 MiB 49.5742 MiB @profile(precision=4)\r\n 36 def get_responses(creds):\r\n 37 \"\"\"Fetch spreadsheet data.\"\"\"\r\n 38 49.5742 MiB 0.0000 MiB sheet_id = \"1TowKJrFVbT4Bfp-HFcMh_CZ5anfH0CLfmoqCz9SUr9c\"\r\n 39 \r\n 40 49.5742 MiB 0.0000 MiB http = creds.authorize(httplib2.Http())\r\n 41 49.5742 MiB 0.0000 MiB service = discovery.build(\"sheets\", \"v4\", http=http,\r\n 42 49.5742 MiB 0.0000 MiB discoveryServiceUrl=(DISCOVERY_URL), cache_discovery=False)\r\n 43 67.9922 MiB 18.4180 MiB result = service.spreadsheets().values().get(\r\n 44 67.9922 MiB 0.0000 MiB spreadsheetId=sheet_id, range=\"A1:O\").execute()\r\n 45 67.9922 MiB 0.0000 MiB values = result.get(\"values\", [])\r\n 46 \r\n 47 67.9922 MiB 0.0000 MiB print(\"Got {} rows\".format(len(values)))\r\n```\r\n\r\n## Last iteration\r\n```\r\nGot 760 rows\r\nFilename: ./main.py\r\n\r\nLine # Mem usage Increment Line Contents\r\n================================================\r\n 35 229.6055 MiB 229.6055 MiB @profile(precision=4)\r\n 36 def get_responses(creds):\r\n 37 \"\"\"Fetch spreadsheet data.\"\"\"\r\n 38 229.6055 MiB 0.0000 MiB sheet_id = \"1TowKJrFVbT4Bfp-HFcMh_CZ5anfH0CLfmoqCz9SUr9c\"\r\n 39 \r\n 40 229.6055 MiB 0.0000 MiB http = creds.authorize(httplib2.Http())\r\n 41 229.6055 MiB 0.0000 MiB service = discovery.build(\"sheets\", \"v4\", http=http,\r\n 42 229.6055 MiB 0.0000 MiB discoveryServiceUrl=(DISCOVERY_URL), cache_discovery=False)\r\n 43 229.6055 MiB 0.0000 MiB result = service.spreadsheets().values().get(\r\n 44 229.6055 MiB 0.0000 MiB spreadsheetId=sheet_id, range=\"A1:O\").execute()\r\n 45 229.6055 MiB 0.0000 MiB values = result.get(\"values\", [])\r\n 46 \r\n 47 229.6055 MiB 0.0000 MiB print(\"Got {} rows\".format(len(values)))\r\n```\r\n\r\nThere's clearly a memory leak, as the reproducer fetches the same data over and over again, yet the memory consumption keeps rising. Full log can be found [here](https://gist.github.com/mrc0mmand/dd306a69e4732a5c1e186d2ce50913fb).\r\n\r\nAs a temporary workaround for one of my long-running applications I use an explicit garbage collector call, which mitigates this issue, at least for now:\r\n\r\n```python\r\n...\r\nimport gc\r\n...\r\n result = service.spreadsheets().values().get(\r\n spreadsheetId=sheet_id, range=\"A1:O\").execute()\r\n values = result.get(\"values\", [])\r\n\r\n gc.collect()\r\n...\r\n```\r\n----\r\n\r\nI went a little deeper, and the main culprit seems to be in the `createMethod` function when creating dynamic method `batchUpdate`:\r\n```\r\nMethod 'batchUpdate, approx. __doc__ size: 2886834 \r\n\u003cclass 'function'> \r\nFilename: /home/fsumsal/venv/googleapiclient/lib64/python3.6/site-packages/googleapiclient/discovery.py \r\n \r\nLine # Mem usage Increment Line Contents \r\n================================================ \r\n 1064 48.7 MiB 48.7 MiB @profile \r\n 1065 def _add_basic_methods(self, resourceDesc, rootDesc, schema): \r\n 1066 # If this is the root Resource, add a new_batch_http_request() method. \r\n 1067 48.7 MiB 0.0 MiB if resourceDesc == rootDesc: \r\n... \r\n 1086 \r\n 1087 # Add basic methods to Resource \r\n 1088 48.7 MiB 0.0 MiB if 'methods' in resourceDesc: \r\n 1089 66.8 MiB 0.0 MiB for methodName, methodDesc in six.iteritems(resourceDesc['methods']): \r\n 1090 56.0 MiB 0.0 MiB fixedMethodName, method = createMethod( \r\n 1091 66.8 MiB 18.1 MiB methodName, methodDesc, rootDesc, schema) \r\n 1092 66.8 MiB 0.0 MiB print(type(method)) \r\n 1093 66.8 MiB 0.0 MiB self._set_dynamic_attr(fixedMethodName, \r\n 1094 66.8 MiB 0.0 MiB method.__get__(self, self.__class__)) \r\n 1095 # Add in _media methods. The functionality of the attached method will \r\n 1096 # change when it sees that the method name ends in _media. \r\n 1097 66.8 MiB 0.0 MiB if methodDesc.get('supportsMediaDownload', False): \r\n 1098 fixedMethodName, method = createMethod( \r\n 1099 methodName + '_media', methodDesc, rootDesc, schema) \r\n 1100 self._set_dynamic_attr(fixedMethodName, \r\n 1101 method.__get__(self, self.__class__))\r\n```\r\n\r\n(This method has a *huge* docstring.)\r\n\r\nNevertheless, there is probably a reference loop somewhere, as the `gc.collect()` call manages to collect all those unreachable objects.","author":{"url":"https://github.com/mrc0mmand","@type":"Person","name":"mrc0mmand"},"datePublished":"2018-06-29T12:50:39Z","url":"https://github.com/googleapis/google-api-python-client/issues/535"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:b4aeabd0-ec8f-4877-82eb-72959202bcb5 |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | BBE2:30C16C:4FFA34:6BC991:6B00A76F |
| html-safe-nonce | bb644e19b2babd1967dcbf96473ee606229fe7c4df0a7c796be9ee50e2008aef |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJCQkUyOjMwQzE2Qzo0RkZBMzQ6NkJDOTkxOjZCMDBBNzZGIiwidmlzaXRvcl9pZCI6Ijc3OTUwMzQ1NDkzMDEzMjM2MzEiLCJyZWdpb25fZWRnZSI6ImlhZCIsInJlZ2lvbl9yZW5kZXIiOiJpYWQifQ== |
| visitor-hmac | d8e59b67820e1ca52be54622b59f4256946b7156cc72834892dcff9c3a8a6851 |
| hovercard-subject-tag | issue:336986812 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/googleapis/google-api-python-client/535/issue_layout |
| twitter:image | https://opengraph.githubassets.com/d51999fa234d6497a87887a98d79a4c237d58f29ecc466b87a3e037afdf5b6fb/googleapis/google-api-python-client/issues/535 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/d51999fa234d6497a87887a98d79a4c237d58f29ecc466b87a3e037afdf5b6fb/googleapis/google-api-python-client/issues/535 |
| og:image:alt | There seems to be a memory leak when using the google-api-client with GSheets. Environment: $ python --version Python 3.6.4 $ pip show google-api-python-client Name: google-api-python-client Versio... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | mrc0mmand |
| hostname | github.com |
| expected-hostname | github.com |
| None | 455b493e3402a3a3a06beac5d20bb7db3da060808b220aa48d6a7eca7c59863a |
| turbo-cache-control | no-preview |
| go-import | github.com/googleapis/google-api-python-client git https://github.com/googleapis/google-api-python-client.git |
| octolytics-dimension-user_id | 16785467 |
| octolytics-dimension-user_login | googleapis |
| octolytics-dimension-repository_id | 15749269 |
| octolytics-dimension-repository_nwo | googleapis/google-api-python-client |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 15749269 |
| octolytics-dimension-repository_network_root_nwo | googleapis/google-api-python-client |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 2f575159976167242d488c0b062eb355c4e198b7 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width