René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Vision: Google Cloud Storage Client project=None bootstrapping problem","articleBody":"Hello,\r\n\r\nIn using the Google Cloud API's, I noticed a small discrepancy in the documentation's expected behavior, and the actual behavior. \r\nThe documentation and Google cloud examples code typically assumes that credentials should be created using a JSON key store saved on persistent storage. However, this isn't always an option, as with deployments using ephemeral storage. Luckily, a Credentials object can be created directly from a dictionary, which houses the same mapping that the JSON keystore would hold. This is the procedure that I use in my project (the specifics of which, is shown below).\r\n\r\n\r\nThe [Storage.Client documentation](https://googlecloudplatform.github.io/google-cloud-python/latest/storage/client.html#google.cloud.storage.client.Client) suggests that the `project` parameter is optional, and thus may be None. We found, however, that when it is not specified, an error is thrown, as a result of a sort of bootstrapping problem in which the Google Cloud APIs check to see if the project is None, and if so, attempt to create it (which is impossible, without first having the keys).\r\nTo elaborate:\r\n\r\n1. When we make a call to Storage.Client(), we step into [storage/google/cloud/storage/client.py](https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/). \r\n2. This class calls `super(Client, self).__init__(project=project, credentials=credentials, _http=_http)`, which takes us to [core/google/cloud/client.py](https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/core/google/cloud/client.py) .\r\n3. This calls `_ClientProjectMixin.__init__(self, project=project)`, where we see the following call:\r\n```\r\n        project = self._determine_default(project)\r\n```\r\n4. Following this call, we see:\r\n```\r\n    if project is None:\r\n        _, project = google.auth.default()\r\n```\r\n\r\n\r\nSo, there is an attempt to infer the project parameter based on the environment credentials. However, as discussed above, we are using a dictionary-created Credentials object as our credentials,  so no such credentials exist in the environment. **This is a problem because the documentation claims that project is optional, when, in fact, it is not if the project is not stored in the environment.**\r\n\r\nNote, however, that this seems to be a problem specific with Storage.Client(), and not other API clients. Trying this procedure (see the code below) for `google.cloud.vision.ImageAnnotatorClient` seems to work perfectly fine with just a credentials object, without the project argument. \r\n\r\n\r\n----\r\n\r\n\r\n\r\n1. **OS:** Ubuntu 16.04\r\n2. **Python Version:** Python 2.7.10\r\n3. **Google APIs:**\r\ngoogle-auth==1.1.1\r\ngoogle-cloud-core==0.27.1\r\ngoogle-cloud-storage==1.4.0\r\ngoogle-cloud-vision==0.27.0\r\ngoogle-gax==0.15.15\r\ngoogle-resumable-media==0.2.3\r\ngoogleapis-common-protos==1.5.3\r\noauth2client==4.1.2\r\n\r\n\r\n4. **Stacktrace:**\r\n\r\n\r\n```\r\nTraceback (most recent call last):\r\n  File \"storage_client.py\", line 23, in \u003cmodule\u003e\r\n    client = _get_client()\r\n  File \"storage_client.py\", line 18, in _get_client\r\n    client = storage.Client(credentials=creds)\r\n  File \"/Users/admin/testproject/venv/lib/python2.7/site-packages/google/cloud/storage/client.py\", line 59, in __init__\r\n    _http=_http)\r\n  File \"/Users/admin/testproject/venv/lib/python2.7/site-packages/google/cloud/client.py\", line 211, in __init__\r\n    _ClientProjectMixin.__init__(self, project=project)\r\n  File \"/Users/admin/testproject/venv/lib/python2.7/site-packages/google/cloud/client.py\", line 165, in __init__\r\n    project = self._determine_default(project)\r\n  File \"/Users/admin/testproject/venv/lib/python2.7/site-packages/google/cloud/client.py\", line 178, in _determine_default\r\n    return _determine_default_project(project)\r\n  File \"/Users/admin/testproject/venv/lib/python2.7/site-packages/google/cloud/_helpers.py\", line 179, in _determine_default_project\r\n    _, project = google.auth.default()\r\n  File \"/Users/admin/testproject/venv/lib/python2.7/site-packages/google/auth/_default.py\", line 286, in default\r\n    raise exceptions.DefaultCredentialsError(_HELP_MESSAGE)\r\ngoogle.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or\r\nexplicitly create credential and re-run the application. For more\r\ninformation, please see\r\nhttps://developers.google.com/accounts/docs/application-default-credentials.\r\n\r\n```\r\n\r\n\r\n5. **Steps to reproduce:**\r\n\r\n\r\n\r\n- So, create an OAuth2 credentials object from a python dictionary (let's call it `ketfile_dict` using `google.oauth2.service_account.Credentials.from_service_account_info(keyfile_dict)`.\r\n\r\n- Create an instance of a storage client using `google.cloud.storage.Client(credentials=creds)`\r\n\r\n- Although the documentation states that the \"project\" argument is optional, the API will in fact throw an error when no project argument is presented.\r\n\r\n6. Code example\r\n\r\nAssume that in the same directory, you create a file, `constants.py`, with the following fields defined from the appropriate fields of the service account key JSON.\r\n\r\nconstants.GCP_SCOPE\r\nconstants.GCP_TYPE\r\nconstants.GCP_CLIENT_EMAIL\r\nconstants.GCP_PRIVATE_KEY\r\nconstants.GCP_PRIVATE_KEY_ID\r\nconstants.GCP_CLIENT_ID\r\nconstants.GCP_TOKEN_URI\r\nconstants.GCP_PROJECT_ID\r\n\r\n\r\n\r\n```\r\nfrom google.cloud import storage\r\nfrom google.oauth2 import service_account\r\nimport constants\r\n\r\n\r\n# Return an Oauth2 credentials instance\r\ndef get_credentials(with_keyfile=False):\r\n    keyfile_dict = _get_keyfile_dict()\r\n    scope = [constants.GCP_SCOPE]\r\n    creds = service_account.Credentials.from_service_account_info(keyfile_dict).with_scopes(scope)\r\n\r\n    if not with_keyfile:\r\n        return creds\r\n    else:\r\n        # Return creds and the keyfile.\r\n        return creds, keyfile_dict\r\n\r\n\r\n# Construct a dictionary of the required key:value pairs.\r\ndef _get_keyfile_dict():\r\n    keyfile_dict = {}\r\n    keyfile_dict['type'] = constants.GCP_TYPE\r\n    keyfile_dict['client_email'] = constants.GCP_CLIENT_EMAIL\r\n    keyfile_dict['private_key'] = constants.GCP_PRIVATE_KEY\r\n    keyfile_dict['private_key_id'] = constants.GCP_PRIVATE_KEY_ID\r\n    keyfile_dict['client_id'] = constants.GCP_CLIENT_ID\r\n    keyfile_dict['token_uri'] = constants.GCP_TOKEN_URI\r\n    keyfile_dict['project_id'] = constants.GCP_PROJECT_ID\r\n    return keyfile_dict\r\n\r\n\r\ndef _get_client():\r\n    creds, keyfile_dict = get_credentials(with_keyfile=True)\r\n    project = keyfile_dict['project_id']\r\n    #client = storage.Client(project=project, credentials=creds) # This WILL work because a project is passed in.\r\n    #client = vision.ImageAnnotatorClient(credentials=creds) #Note that the Vision client will work without a project argument\r\n    client = storage.Client(credentials=creds) # This will throw an error because project=None.\r\n    return client\r\n\r\n\r\n# Instantiates a client\r\nclient = _get_client()\r\n```\r\n\r\n\r\nLet me know if you have any other questions. :)\r\n\r\n\r\n\r\n\r\n","author":{"url":"https://github.com/safi-manar","@type":"Person","name":"safi-manar"},"datePublished":"2017-10-23T09:04:00.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":4},"url":"https://github.com/4239/google-cloud-python/issues/4239"}