René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"blob.download_to_filename fails with google-cloud-storage==1.3.0","articleBody":"Due to the notable implementation changes in **google-cloud-storage 1.3.0**, the **_blob.download_to_filename_** fails whatever **google-cloud** from **0.24.0** to **0.27.0** (latest)\r\n\r\nOur prod server which using **google-cloud==0.24.0** has been broken since last Saturday. I analyzed a bit the error. The problem is come from the upgraded package **google-cloud-storage==1.3.0**. We are using the **google-cloud-storage==1.2.0**.\r\n\r\nI did several tests which figured out the following problems:\r\n\r\n**Problem 1:** _blob.download_to_filename failed when using google-cloud==0.27.0_ (included **google-cloud-storage==1.3.0** as default, defined in **setup.py**). Here is a part of stacktrace:\r\n```python\r\nblob.download_to_filename(blob_local_path)\r\n  File \"/root/.local/lib/python2.7/site-packages/google/cloud/storage/blob.py\", line 482, in download_to_filename\r\n    self.download_to_file(file_obj, client=client)\r\n  File \"/root/.local/lib/python2.7/site-packages/google/cloud/storage/blob.py\", line 464, in download_to_file\r\n    self._do_download(transport, file_obj, download_url, headers)\r\n  File \"/root/.local/lib/python2.7/site-packages/google/cloud/storage/blob.py\", line 418, in _do_download\r\n    download.consume(transport)\r\n  File \"/root/.local/lib/python2.7/site-packages/google/resumable_media/requests/download.py\", line 101, in consume\r\n    self._write_to_stream(result)\r\n  File \"/root/.local/lib/python2.7/site-packages/google/resumable_media/requests/download.py\", line 62, in _write_to_stream\r\n    with response:\r\nAttributeError: __exit__\r\n```\r\n\r\n**Problem 2:** _blob.download_to_filename failed when using google-cloud==0.24.0_ (included **google-cloud-storage==1.3.0** as default, defined in **setup.py**). Here is a part of stacktrace:\r\n```python\r\nFile \"dev_env/python_venv/local/lib/python2.7/site-packages/google/cloud/storage/blob.py\", line 482, in download_to_filename\r\n    self.download_to_file(file_obj, client=client)\r\n  File \"dev_env/python_venv/local/lib/python2.7/site-packages/google/cloud/storage/blob.py\", line 464, in download_to_file\r\n    self._do_download(transport, file_obj, download_url, headers)\r\n  File \"dev_env/python_venv/local/lib/python2.7/site-packages/google/cloud/storage/blob.py\", line 418, in _do_download\r\n    download.consume(transport)\r\n  File \"dev_env/python_venv/local/lib/python2.7/site-packages/google/resumable_media/requests/download.py\", line 96, in consume\r\n    transport, method, url, **request_kwargs)\r\n  File \"dev_env/python_venv/local/lib/python2.7/site-packages/google/resumable_media/requests/_helpers.py\", line 101, in http_request\r\n    func, RequestsMixin._get_status_code, retry_strategy)\r\n  File \"dev_env/python_venv/local/lib/python2.7/site-packages/google/resumable_media/_helpers.py\", line 146, in wait_and_retry\r\n    response = func()\r\n  File \"dev_env/python_venv/local/lib/python2.7/site-packages/google_auth_httplib2.py\", line 198, in request\r\n    uri, method, body=body, headers=request_headers, **kwargs)\r\nTypeError: request() got an unexpected keyword argument 'data'\r\n```\r\n\r\nSo, why it happened? I realized that in the _REQUIREMENTS_ of the package **google-cloud** which is defined in the file **setup.py**. An example for **google-cloud==0.27.0** https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/setup.py#L67\r\n\r\n- **google-cloud==0.24.0** requires google-cloud-storage \u003e= 1.0.0 and **\u003c 2.0dev**\r\n- **google-cloud==0.25.0** requires google-cloud-storage \u003e= 1.1.0 and **\u003c 2.0dev**  =\u003e https://github.com/GoogleCloudPlatform/google-cloud-python/commit/bd9fdfba07fac63a91847628613928a569250c0f#diff-2eeaed663bd0d25b7e608891384b7298R66\r\n- **google-cloud==0.25.1** requires google-cloud-storage \u003e= 1.1.0 and **\u003c 1.2dev**  =\u003e https://github.com/GoogleCloudPlatform/google-cloud-python/commit/ad4b02c88088666e298aa53cd61687b461c09191#diff-2eeaed663bd0d25b7e608891384b7298R66\r\n- **google-cloud==0.26.0** requires google-cloud-storage \u003e= 1.2.0 and **\u003c 2.0dev** =\u003e https://github.com/GoogleCloudPlatform/google-cloud-python/commit/214aba604fbcaa9e4936fa1798efde050389992a#diff-2eeaed663bd0d25b7e608891384b7298R67\r\n- **google-cloud==0.26.1** requires google-cloud-storage \u003e= 1.2.0 and **\u003c 1.3dev** =\u003e https://github.com/GoogleCloudPlatform/google-cloud-python/commit/ea5245ad27e777c654448a259a9fa5f6bf5d758e#diff-2eeaed663bd0d25b7e608891384b7298R67\r\n- **google-cloud==0.27.0** requires google-cloud-storage \u003e= 1.3.0 and **\u003c 1.4dev** =\u003e https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/setup.py#L67\r\n\r\nWe can see that if we only define **google-cloud==[0.24.0 to 0.27.0]**, \"pip install\" always try to install **google-cloud-storage 1.3.0** which requires the dependencies **google-cloud-core ~= 0.26**. That's why the bug happens.\r\n\r\n**Solution:**\r\n- Downgrade to **google-cloud \u003c 0.27.0** and force the requirements **google-cloud-storage \u003c 1.3.0**\r\n- The problem seems still there with **google-cloud==0.27.0** and **google-cloud-storage==1.3.0** (to check but at least it has been broken for my case, see **Problem 2** above)\r\n\r\n**Suggestion:** better manage the deps for **google-cloud** in **setup.py**.\r\n\r\nThanks.\r\n\r\n","author":{"url":"https://github.com/sonlac","@type":"Person","name":"sonlac"},"datePublished":"2017-08-07T15:21:06.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":30},"url":"https://github.com/3736/google-cloud-python/issues/3736"}