René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"feat: add ability to retry query job automatically if the job fails","articleBody":"\r\n#### Environment details\r\n\r\n  - OS type and version: Mac (10.15.7) and Python Base Docker Image\r\n  - Python version: `Python 3.7.8`\r\n  - Poetry version: `Poetry version 1.0.10`\r\n  - `google-cloud-bigquery`:  `1.28.0              Google BigQuery API client library`\r\n\r\n#### Issue\r\n\r\nI am using the `retry` parameter in `biquery.Client.query` but it isn't retrying my request.  \r\n\r\n#### Code example\r\n\r\nSetup code: \r\n\r\n```python\r\nimport structlog\r\nfrom google.api_core import retry, exceptions\r\nfrom google.api_core.retry import if_exception_type, Retry\r\nfrom google.cloud import bigquery\r\n\r\nfrom common.logging.setup_logging import set_logger_config\r\n\r\nset_logger_config() # Enables debug level logging for retry\r\n\r\nproject = \"myproject\"\r\nsource_table = f\"{project}.mydataset.TABLE_NOT_REALLY_HERE\"\r\nstatement = f\"SELECT * FROM `{source_table}`\"\r\nclient = bigquery.Client(project=project)\r\n```\r\n\r\nNow if you run: \r\n\r\n```python\r\nretry_policy = retry.Retry(predicate=if_exception_type(exceptions.NotFound))\r\nquery_job = client.query(statement, retry=retry_policy)\r\nresult = list(query_job.result(retry=retry_policy))\r\nprint(result)\r\n```\r\n\r\nOnly one request is made: \r\n\r\n```\r\nTraceback (most recent call last):\r\n  File \"/Users/.../lib/python3.7/site-packages/google/cloud/bigquery/job.py\", line 3230, in result\r\n    super(QueryJob, self).result(retry=retry, timeout=timeout)\r\n  File \"/Users/.../lib/python3.7/site-packages/google/cloud/bigquery/job.py\", line 835, in result\r\n    return super(_AsyncJob, self).result(timeout=timeout)\r\n  File \"/Users/.../lib/python3.7/site-packages/google/api_core/future/polling.py\", line 134, in result\r\n    raise self._exception\r\ngoogle.api_core.exceptions.NotFound: 404 Not found: Table \u003cTABLE_NAME\u003e was not found in location US\r\n\r\n(job ID: 5aeb5865-7bde-4795-ae80-75410f8b9282)\r\n\r\n                                              -----Query Job SQL Follows-----                                              \r\n\r\n    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |\r\n   1:SELECT *\r\n   2:                FROM `\u003cTABLE_NAME\u003e`\r\n   3:                \r\n    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |    .    |\r\npython-BaseException\r\n\r\nProcess finished with exit code 1\r\n```\r\n\r\n#### Workaround\r\n\r\nIf instead, I run: \r\n\r\n```python\r\n@Retry(predicate=if_exception_type(exceptions.NotFound))\r\ndef do():\r\n    query_job = client.query(statement)\r\n    result = list(query_job.result())\r\n    print(result)\r\ndo()\r\n```\r\n\r\nThis works as expected and retries the request: \r\n\r\n```\r\n{\"logger\": \"google.api_core.retry\", \"timestamp\": \"2021-03-02T19:07:48.000317Z\", \"severity\": \"debug\", \"message\": \"Retrying due to 404 Not found: Table \u003ctable_name\u003e was not found in location US\\n\\n(job ID: \u003cREMOVED\u003e, sleeping 1.0s ...\"}\r\n{\"logger\": \"google.api_core.retry\", \"timestamp\": \"2021-03-02T19:07:49.363842Z\", \"severity\": \"debug\", \"message\": \"Retrying due to 404 Not found: Table \u003ctable_name\u003e was not found in location US\\n\\n(job ID: \u003cREMOVED\u003e, sleeping 3.2s ...\"}\r\n{\"logger\": \"google.api_core.retry\", \"timestamp\": \"2021-03-02T19:07:52.741042Z\", \"severity\": \"debug\", \"message\": \"Retrying due to 404 Not found: Table \u003ctable_name\u003e was not found in location US\\n\\n(job ID: \u003cREMOVED\u003e, sleeping 2.8s ...\"}\r\n{\"logger\": \"google.api_core.retry\", \"timestamp\": \"2021-03-02T19:07:55.969272Z\", \"severity\": \"debug\", \"message\": \"Retrying due to 404 Not found: Table \u003ctable_name\u003e was not found in location US\\n\\n(job ID: \u003cREMOVED\u003e, sleeping 9.2s ...\"}\r\n{\"logger\": \"google.api_core.retry\", \"timestamp\": \"2021-03-02T19:08:05.493544Z\", \"severity\": \"debug\", \"message\": \"Retrying due to 404 Not found: Table \u003ctable_name\u003e was not found in location US\\n\\n(job ID: \u003cREMOVED\u003e, sleeping 4.2s ...\"}\r\n{\"logger\": \"google.api_core.retry\", \"timestamp\": \"2021-03-02T19:08:09.960265Z\", \"severity\": \"debug\", \"message\": \"Retrying due to 404 Not found: Table \u003ctable_name\u003e was not found in location US\\n\\n(job ID: \u003cREMOVED\u003e, sleeping 44.0s ...\"}\r\n```\r\n\r\n\r\n","author":{"url":"https://github.com/JakeSummers","@type":"Person","name":"JakeSummers"},"datePublished":"2021-03-02T19:11:55.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":7},"url":"https://github.com/539/python-bigquery/issues/539"}