René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Batch commit throws ServiceUnavailable: 503 Backend Error","articleBody":"See below:\n\nOPERATING SYSTEM\n\n$ uname -mrs\nLinux 3.19.5-100.fc20.x86_64 x86_64\n\nENVIRONMENT VARIABLES\n# PYTHON 2.7\n# export PYTHONHOME=/usr/bin/python2.7\n\nexport PYTHONPATH=/usr/lib/python2.7/site-packages\n# PYTHON modules for GCP\n\nexport PYTHONPATH=$PYTHONPATH:/home/charles/google-cloud-sdkgoogle-cloud-sdk/bin\nexport PYTHONPATH=$PYTHONPATH:/home/charles/google-cloud-sdkgoogle-cloud-sdk/lib\nexport PYTHONPATH=$PYTHONPATH:/home/charles/google-cloud-sdkgoogle-cloud-sdk/lib/googlecloudsdk\nexport PYTHONPATH=$PYTHONPATH:/home/charles/google-cloud-sdkgoogle-cloud-sdk/lib/googlecloudsdk/gcloud\nexport PYTHONPATH=$PYTHONPATH:/home/charles/google-cloud-sdkgoogle-cloud-sdk/lib/googlecloudapis\n# export PYTHONPATH=$PYTHONPATH:/home/charles/google-cloud-sdkgoogle-cloud-sdk/platform\n# export PYTHONPATH=$PYTHONPATH:/home/charles/google-cloud-sdkgoogle-cloud-sdk/platform/bq\n# export PYTHONPATH=$PYTHONPATH:/home/charles/google-cloud-sdkgoogle-cloud-sdk/platform/gcutil\n# export PYTHONPATH=$PYTHONPATH:/home/charles/google-cloud-sdkgoogle-cloud-sdk/platform/gcutil/lib\n# export PYTHONPATH=$PYTHONPATH:/home/charles/google-cloud-sdkgoogle-cloud-sdk/platform/gsutil\n# export PYTHONPATH=$PYTHONPATH:/home/charles/google-cloud-sdkgoogle-cloud-sdk/platform/gsutil/gslib\n# export PYTHONPATH=$PYTHONPATH:/home/charles/google-cloud-sdkgoogle-cloud-sdk/platform/gsutil/third_party\n# export PATH=$PATH:$PYTHONHOME\n\nexport PATH=$PATH:$PYTHONPATH\n# Google AppEngine SDK\n\nexport PATH=$PATH:/home/charles/google_appengine/\n# Google Cloud SDK\n\nexport CLOUDSDK_CONFIG=/home/charles/.config/gcloud\nexport CLOUDSDK_PYTHON=/usr/bin/python2.7\nexport CLOUDSDK_PYTHON_SITEPACKAGES=1\n# BOTO\n# export BOTO_CONFIG=/home/charles/.boto\n\nexport NVM_DIR=\"/home/charles/.nvm\"\n[ -s \"$NVM_DIR/nvm.sh\" ] \u0026\u0026 . \"$NVM_DIR/nvm.sh\"  # This loads nvm\n# The next line updates PATH for the Google Cloud SDK.\n\nsource '/home/charles/google-cloud-sdk/path.bash.inc'\n# The next line enables bash completion for gcloud.\n\nsource '/home/charles/google-cloud-sdk/completion.bash.inc'\n\nFORM OF AUTHENTICATION\n\nThe credentialed account being used is the owner of the Google project, for which Cloud Datastore API is enabled.\n\nPYTHON CODE\n\nfrom gcloud import storage\nstorage.set_default_bucket('')\nstorage.set_default_project(_PROJECT_NAME)\nfrom gcloud import datastore\ndatastore.set_default_dataset_id(_DATA_SET_ID)\n# datastore.set_defaults(namespace=_NAMESPACE)\n\nfrom gcloud.datastore.batch import Batch\n\nimport traceback\n\ndef Load_Datastore():\n    try:\n        Get_Bucket_Metadata()\n        #set configuration variables  \n        storage_conn = storage.get_connection()\n        storage.set_default_connection(storage_conn)  \n        datastore_conn = datastore.get_connection()\n        datastore.set_default_connection(datastore_conn)\n\n```\n    for _BUCKET_NAME in _BUCKET_LIST:\n        print \"Bucket Name: %s\" % (_BUCKET_NAME)\n    .\n    .\n    .\n\n                #an empty list is false\n                if not _RECORDS:\n                    print \"_RECORDS is empty\"\n                else:\n                    print \"there are %s records to write into Datastore\"%(str(len(_RECORDS)))\n                    with datastore.batch.Batch(connection=datastore_conn) as batch:\n                        _ENTITIES=[]                                           \n                        for _RECORD in _RECORDS:\n                            _PROPERTIES=[]\n\n                            if _RECORD:\n                                for _KEY_VALUES in _RECORD:\n                                    _KEY_VALUE_PAIR=_KEY_VALUES.split('|')\n                                    if str(_KEY_VALUE_PAIR[0])=='ROW_KEY':\n                                        _ENTITY_KEY=str(_KEY_VALUE_PAIR[1])\n                                    else:\n                                        _PROPERTIES.append(\"%s|%s\"%(str(_KEY_VALUE_PAIR[0]),str(_KEY_VALUE_PAIR[1])))\n\n                            if not _PROPERTIES:   \n                                print \"_PROPERTIES is empty\"\n                            else:                                              \n                                try:\n                                    key = datastore.Key(_SOURCE_TABLE_NAME.upper(),_ENTITY_KEY,namespace=_NAMESPACE,dataset_id=_DATA_SET_ID)\n\n                                    try:\n                                        DS_ENTITY = datastore.Entity(key)\n                                        for _PROPERTY in _PROPERTIES:\n                                            _NAME_VALUE=_PROPERTY.split('|')\n                                            _COLUMN_NAME=str(_NAME_VALUE[0])\n                                            _COLUMN_VALUE=str(_NAME_VALUE[1])\n                                            DS_ENTITY.update({_COLUMN_NAME:_COLUMN_VALUE})\n                                        # Now let's save it to our datastore:\n                                        _ENTITIES.append(DS_ENTITY)                                 \n                                    except Exception, inst:\n                                        print \"datastore.Entity(%s) Unexpected error: %s\"%(str(_ENTITY_KEY),str(sys.exc_info()[0]))\n                                        if isinstance(inst,ValueError):\n                                            print \"ValueError: %s\"%(inst.message)\n                                        if isinstance(inst,AttributeError):\n                                            print \"AttributeError: %s\"%(inst.message)\n                                except Exception, inst:\n                                    print \"datastore.Key(_SOURCE_TABLE_NAME,_ENTITY_KEY,_NAMESPACE,_DATASET_ID Unexpected error: %s\"%(str(sys.exc_info()[0]))\n                                    if isinstance(inst,ValueError):\n                                        print \"ValueError: %s\"%(inst.message)\n                                    if isinstance(inst,AttributeError):\n                                        print \"AttributeError: %s\"%(inst.message)\n\n                        for _ENTITY in _ENTITIES:\n                            batch.put(_ENTITY)\n                            print 'batch.put(_ENTITY)'\n\n\nexcept Exception, inst:\n    print '\\nLoad_Datastore() Unexpected error: ' + str(sys.exc_info()[0])\n    if isinstance(inst,ValueError):\n        print \"ValueError: %s\\n\"%(inst.message)\n    if isinstance(inst,AttributeError):\n        print \"AttributeError: %s\\n\"%(inst.message)\n    if isinstance(inst,TypeError):\n        print \"TypeError: %s\\n\"%(inst.message)\n    print '-'*60\n    traceback.print_exc(file=sys.stdout)\n    print '-'*60                            \n    raise\n```\n\nSTACK TRACE\n## Load_Datastore() Unexpected error: \u003cclass 'gcloud.exceptions.ServiceUnavailable'\u003e\n\nTraceback (most recent call last):\n  File \"/home/charles/workspace_Python/HBaseSchemaFromXML/pkg1/datastore_loader.py\", line 326, in Load_Datastore\n    print 'batch.put(_ENTITY)'\n  File \"/usr/lib/python2.7/site-packages/gcloud-0.5.0-py2.7.egg/gcloud/datastore/batch.py\", line 232, in __exit__\n    self.commit()\n  File \"/usr/lib/python2.7/site-packages/gcloud-0.5.0-py2.7.egg/gcloud/datastore/batch.py\", line 207, in commit\n    response = self.connection.commit(self._dataset_id, self.mutation)\n  File \"/usr/lib/python2.7/site-packages/gcloud-0.5.0-py2.7.egg/gcloud/datastore/connection.py\", line 324, in commit\n    datastore_pb.CommitResponse)\n  File \"/usr/lib/python2.7/site-packages/gcloud-0.5.0-py2.7.egg/gcloud/datastore/connection.py\", line 108, in _rpc\n    data=request_pb.SerializeToString())\n  File \"/usr/lib/python2.7/site-packages/gcloud-0.5.0-py2.7.egg/gcloud/datastore/connection.py\", line 85, in _request\n    raise make_exception(headers, content, use_json=False)\n## ServiceUnavailable: 503 Backend Error\n\nUnexpected error: \u003cclass 'gcloud.exceptions.ServiceUnavailable'\u003e\nException instance type: \u003cclass 'gcloud.exceptions.ServiceUnavailable'\u003e\ndone.\n\n---\n\nRan 0 tests in 0.000s\n\nNOTES\n\nThe script executes all of the batch.put() statements, and then throws the above error after the 'when' clause finishes.\n\nAny suggestions on how to fix this bug are appreciated\n","author":{"url":"https://github.com/charlesbclifford","@type":"Person","name":"charlesbclifford"},"datePublished":"2015-05-09T00:38:03.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":22},"url":"https://github.com/866/google-cloud-python/issues/866"}